Back to Learning Matrix
M2

Multimodal AI Interaction

Based on physical AI terminals, integrating edge vision, voice, and business-system APIs to enable multimodal spatial interaction.

In warehouse management, showroom tours, and smart reception desks, retrieving business data requires pausing manual work to search on phones or keyboards, limiting efficiency. Conventional terminals lack visual context and cannot proactively detect approaching visitors or abnormal actions. Many smart terminals belong to closed ecosystems, making integration with existing WMS, ERP, and CRM systems difficult; certain industrial and government scenarios prohibit uploading audio and business data to public clouds.

Partnership Inquiry
SenseCAP Watcher (XiaoZhi English Edition)
Difficulty
Advanced
Duration
L1 1 day / L2 2–3 days / L3 3–5 days
Shortest Format
1 day (Taster Session · L1)
Teaching Format
3 tiers: Taster / Workshop / Bootcamp
Core Protocols
MCP / REST API / Wi-Fi
L3 Offline Compute
100 TOPS（Jetson Orin NX 16GB）
Track
Build AI Products
Who This Course Is For
Solution consultants and business sales · Hands-on instructors and curriculum development teams · Enterprise informatization and intelligence engineers · Faculty and students at vocational colleges and applied universities
Prerequisites
L1 suits beginners or those new to edge AI devices; L2 requires basic Docker and REST API experience; L3 requires Linux, PyTorch / Jetson fundamentals, and Shell skills

Typical Scenarios

  • Smart warehousing and workshop management: hands-free inventory lookup, voice-based stock-in/out logging, visual alerts for anomalous materials
  • Exhibition halls and public guidance: proactive visitor detection, multilingual voice narration, booth linkage control
  • Smart reception and meeting spaces: visitor reception, voice control of room devices, automated schedule queries
  • Assisted care and spatial services: specific-behavior detection alerts, voice help requests, and remote linkage
  • Voice interaction in industrial isolated zones: pure LAN offline voice queries and device status announcements

Three depths and where each one gets you

  1. L1 Demo Level 1 day

    Multimodal Interaction Capability Experience

    Query data and control devices by speaking to the Watcher

    • Understand the technical architecture combining edge vision and LLM Agents
    • Master the role of MCP in bridging on-device AI to business workflows
    • Master architectural criteria for choosing between cloud-edge coordination and local deployment across scenarios
  2. L2 Consultant Level 3 days

    Business System Integration and Linkage Configuration

    Connect voice to warehouse and other business systems through MCP

    • Independently configure Watcher vision and voice Agent parameters
    • Master Docker-based deployment of local business systems and MCP bridge services
    • Master extending new business APIs via MCP protocol
  3. L3 Design Level 5 days

    End-to-End Local Offline Voice AI Pipeline Deployment

    Run the whole voice pipeline locally: works offline, data never leaves the intranet

    • Master end-to-end local voice AI pipeline architecture (VAD → ASR → LLM → TTS)
    • Master quantizing, deploying, and optimizing large language models on Jetson edge computing hardware
    • Capability to deliver AI interaction solutions in privacy-sensitive and air-gapped industrial environments

What learners can do afterwards

  • On-device vision object detection and event triggering
  • Natural language voice queries and Agent role configuration
  • Business-system tool calling based on MCP protocol
  • Local WMS system Docker deployment and API integration
  • Purely local offline voice AI pipeline deployment (VAD → ASR → LLM → TTS)

Hardware and toolchain

This course centers on "visible AI terminals + local inference compute".

  • WATCHER SenseCAP Watcher (XiaoZhi English Edition) On-Device Multimodal Interaction Terminal · Voice Capture & Vision Recognition Entry
  • GW reComputer RK3588-40 Edge smart controller · Business system & MCP bridge host
  • HMI reTerminal D1001 8-inch industrial smart touch screen · Workstation HMI
  • GPU reComputer J4012 (Jetson Orin NX 16GB) L3 Advanced compute host · Purely local offline voice pipeline
Illustrated list of 4 devices
  • SenseCAP Watcher (XiaoZhi English Edition)

    SenseCAP Watcher (XiaoZhi English Edition)

    On-device multimodal interaction terminal, voice capture and vision recognition entry point

    Integrates audio/video capture and screen display, supporting object detection, human presence sensing and natural language voice interaction via Wi-Fi to SenseCraft AI; 2 units per team.

  • reComputer RK3588-40 Edge Intelligence Controller

    reComputer RK3588-40 Edge Intelligence Controller

    Edge host running business systems and MCP bridge service

    Equipped with 16GB RAM and 6 TOPS computing power, running containerized WMS warehouse management and MCP Bridge services to connect local business data with LLM tool calling; includes independent power adapter.

  • reTerminal D1001 8-inch Smart Touchscreen

    reTerminal D1001 8-inch Smart Touchscreen

    Workstation HMI, warehouse business data entry and status monitoring

    8-inch industrial smart touch terminal integrating camera and dual microphones. Serves as workstation HMI for warehouse data entry and status monitoring, directly connecting to the host to display WMS consoles and logs.

  • reComputer J4012 Jetson Orin NX 16GB

    reComputer J4012（Jetson Orin NX 16GB）

    L3 Advanced edge compute host, deploying purely local offline voice pipeline

    Delivers 100 TOPS edge computing power with preconfigured JetPack, CUDA, and TensorRT environments, deploying a completely local offline voice AI pipeline that functions reliably without internet access.

Also supplied: a portable on-site display (13.3" 1080P), a CUDY AX3000 Wi-Fi 6 router, a power strip, Cat 6 gigabit cables, a smart-warehouse WMS practice pack (barcode labels, bin-location labels, physical sample boxes), a Watcher desktop stand and other general accessories.

From Cloud Multimodal Setup to Local Offline Deployment

SenseCraft AI handles fast configuration and validation, the MCP bridge keeps business data on the LAN, and the Jetson offline pipeline removes public-internet dependency entirely.

  1. SenseCraft AI + Watcher

    Cloud LLM + On-Device Multimodal Terminal · Zero-Code Configuration

    1. Watcher Network Setup & Binding
    2. Vision Model & Agent Role Configuration
    3. Voice Query & MCP Tool Calling Experience

  2. MCP Bridge + Docker + OpenClaw

    Standard protocol bridge · Local business data never leaves the intranet

    1. Local WMS Docker Deployment
    2. MCP Bridge config.yml Configuration
    3. OpenClaw Automation Tool Registration & Integration Testing

  3. Jetson Orin NX + Offline Voice Pipeline

    VAD → ASR → LLM → TTS fully local workflow · Zero public internet dependency

    1. JetPack Environment Verification
    2. Quantized Model Deployment & VRAM Tuning
    3. Offline Integration Testing & Latency Optimization

Key Turning Point · From Cloud Collaboration to Local Offline Private Deployment SenseCraft AI cloud solution rapidly validates prototypes; MCP bridge keeps inventory and records within the local network; Jetson offline pipeline eliminates internet dependencies for air-gapped, privacy-first environments.

L1/L2 require an internet connection for LLM APIs; L3 requires 100 TOPS edge compute (Jetson Orin NX 16GB); RK3588-40 (6 TOPS) cannot run local LLM inference.

Syllabus and formats

15 teaching units in total. The delivery formats come first; the full syllabus expands below.

  • Taster Session

    No final project

    1 Day · 6–8h · L1 Presentation Layer · Focused on edge experience and MCP tool invocation demo

  • Hands-On Course

    Includes final project

    2–3 Days · 14–20h · L1+L2 · Watcher config + Local WMS deployment + MCP bridge + Automation linkage

  • Delivery Course

    Includes final project

    3–5 Days · 24–35h · L1+L2+L3 · Full coverage including Jetson offline voice pipeline deployment and air-gapped testing

L1/L2 cloud collaboration solutions require stable uplink internet bandwidth on site for SenseCraft AI platform and LLM service calls; L1/L2 cannot run in external-network-free environments and must switch to the L3 offline solution.

Speech recognition accuracy is affected by on-site ambient noise, dialect accents, and specialized industry vocabulary; high-noise industrial sites require directional audio capture equipment, and far-field blind capture effects are not promised.

The L3 pure local solution relies on 100 TOPS-class edge GPU hardware (Jetson Orin NX 16GB and above), the 6 TOPS RK3588-40 cannot host local LLM inference.

The taster session does not include local business system deployment or offline voice pipelines. Do not promise clients that taster session students can independently complete WMS deployment or offline pipeline setup — that is the workshop and bootcamp delivery standard.

One of the lessons · 01

Pre-class Preparation and Environment Pre-check

Hardware bench inventory, network connectivity checks, Watcher firmware pre-checks, RK3588-40 Docker & WMS image setup, J4012 JetPack loading, and training material readiness

—

Show the full syllabus of 15 units and the schedule for each format
15teaching modules coverage across formats
Module / OutputTaster1 dayWorkshop2–3 daysBootcamp3–5 days
01Pre-class Preparation and Environment Pre-checkHardware bench inventory, network connectivity checks, Watcher firmware pre-checks, RK3588-40 Docker & WMS image setup, J4012 JetPack loading, and training material readiness—FullFullFull
02Multimodal Interaction Architecture & Core ConceptsArchitecture deep dive: edge vision, voice Agents, MCP protocol, and cloud-edge collaboration; architectural differences and selection criteria for cloud vs. local offline deploymentsSenseCraft AIFullFullFull
03On-Device Lightweight Vision Inference ExperienceHands-on with Watcher object detection (item recognition, proximity sensing, gesture detection), no-code model adaptation via SenseCraft AI, and UART/network output parsingSenseCAP WatcherFullFullFull
04Scenario-Based Voice Q&A & Agent MechanismReal-time NLP queries for warehousing/retail, end-to-end voice loop (input → inference → TTS playback), Agent prompts & role definitions, and memory comparisons (none / short / long-term)SenseCraft AI AgentFullFullFull
05MCP Tool Calling & Business Integration DemoMCP-based tool invocation demo (live database queries), end-to-end smart warehousing demo (voice-based inventory lookups / check-in / check-out), and OpenClaw automation linkageMCP / OpenClawFullFullFull
06Watcher Vision AI & Edge Network ConfigurationWatcher networking setup and SenseCraft binding, vision model selection and threshold tuning, and event reporting rule configuration (target appearance, zone detection)SenseCraft AI / WatcherNoneFullFull
07Voice Agent & Role Prompt ConfigurationDefine Agent roles (warehouse assistant / showroom docent), configure system prompts and interaction styles, and validate conversational memory modesSenseCraft AI AgentNoneFullFull
08Local Business Management System Docker DeploymentDeploy sample WMS (suharvest/warehouse_system) on RK3588-40 using Docker and Git, initialize admin account and generate API keys, and import demo inventory and storage-location datasetsDocker / WMSNoneFullFull
09MCP Bridge Service Configuration & IntegrationRetrieve Watcher MCP endpoints and authentication credentials, edit config.yml with local API URLs and API keys, launch MCP Bridge service, and verify endpoint handshake statusMCP BridgeNoneFullFull
10OpenClaw Automation Task IntegrationSet up OpenClaw automation environment, register automation scripts as MCP tools, and configure voice triggers for automated queries and scheduled workflowsOpenClawNoneFullFull
11Full-Pipeline Integration Testing & TroubleshootingValidate representative commands (stock lookups, receiving submissions, logistics tracking) and troubleshoot common timeouts, auth failures, and port conflicts—NoneFullFull
12Local Offline Voice Pipeline Architecture AnalysisModule responsibilities and latency profiling for VAD/ASR/LLM/TTS; benchmark comparison between offline and cloud architectures (latency, concurrency, VRAM, and privacy compliance)—NoneNoneFull
13Jetson Runtime Environment & Model Deployment TuningVerify JetPack/CUDA/TensorRT/PyTorch runtime, deploy ASR models (Whisper/FunASR), deploy 4-bit quantized local LLM (Qwen2.5-7B-Instruct) and TTS engines (ChatTTS/Piper), and route Watcher audio streams locallyJetson Orin NXNoneNoneFull
14Offline Integration Testing & Latency OptimizationPhysically disconnect external network to verify LAN self-closed-loop operation, test each stage latency, tune model context length and sampling parameters—NoneNoneFull
15Solution Review and Delivery SummaryTeam presentations and use-case defense, cost and architectural debrief (Cloud SaaS vs. Local Edge), API extension standards, and handover documentation archiving—PartialFullFull

● Full◐ Partial— None●＋ Extended

Taster focuses on L1 on-device experience and MCP demos, excluding local WMS deployment and offline pipelines; Workshop covers L1+L2 with full Watcher configs, WMS setup, and MCP bridges; Bootcamp delivers L1+L2+L3 including Jetson offline voice pipeline deployment.

  • Taster Session

    • Day 1 MorningModules 01 + 02 + 03

      Environment pre-check → Multimodal architecture concepts → On-device vision inference experience

    • Day 1 AfternoonModules 04 + 05 + 15 (abbreviated)

      Voice Q&A & Agent mechanisms → MCP tool calling & business demo → Review & wrap-up

    Does not include local business system deployment or offline voice pipelines.

  • Hands-On Course

    • Day 1Modules 01–05

      Environment pre-check → Multimodal architecture → On-device vision → Voice Agent → MCP tool calling demo

    • Day 2Modules 06–09

      Watcher vision config → Agent role prompts → WMS Docker deployment → MCP bridge integration

    • Day 3 (optional)Modules 10 + 11 + 15

      OpenClaw automation linkage → Full-pipeline integration testing → Solution review and delivery summary

    Workshop delivers 1 working multi-modal system integration covering visual sensing, voice Agent, local WMS, and automation tools. Prerequisites: basic Docker and REST API experience.

  • Delivery Course

    • Day 1–2Modules 01–11

      Full L1+L2 content (on-device experience + Watcher configuration + WMS deployment + MCP bridging + full-pipeline integration testing)

    • Day 3Module 12

      Local Offline Voice Pipeline Architecture Analysis

    • Day 4Module 13

      Jetson Runtime Environment & ASR/LLM/TTS Model Deployment Tuning

    • Day 5Modules 14 + 15

      Offline integration testing and latency optimization → Solution review and delivery archiving

    Bootcamp prepares learners to deliver AI interaction solutions in privacy-sensitive, air-gapped industrial environments. Prerequisites: Linux, PyTorch / Jetson fundamentals, Shell skills, and L1–L2 proficiency.

The taster course takes one day, needs no business-system deployment, and stops at "voice can query and vision can trigger". It suits trade shows, open days and first meetings with a client.

Workshop Day 3 is an optional flexible day: if students have a strong foundation, it can be compressed to 2 days (Day 2 afternoon merged with OpenClaw and full-pipeline integration testing); if more MCP bridge tuning time is needed, use the full 3 days.

What You Take Away

The following are full-version (bootcamp) deliverables; the workshop delivers the first 4 items; the taster session delivers abbreviated versions of items 1 and 2.

  1. 01

    Watcher hardware configuration and vision model specification

    Includes Watcher device network setup records, vision model selection and confidence/trigger threshold parameters, event reporting rule configuration checklist.

  2. 02

    Multimodal Agent system prompts and memory strategy configuration files

    Includes Agent role System Prompt, interaction style settings, dialogue memory mode (no memory/short-term/long-term) configuration and effect verification records.

  3. 03

    Deployment guide for local business system and MCP bridge service

    Includes Docker Compose deployment files, WMS management console initialization steps, MCP Bridge config.yml configuration template, and API Key management specifications.

  4. 04

    OpenClaw automation configuration scripts

    Includes OpenClaw environment deployment steps, configuration for registering automation scripts as MCP-callable tools, and configuration examples for voice-command-triggered automated queries and scheduled tasks.

  5. 05

    Local offline voice AI pipeline deployment and tuning handbook (L3)

    Includes VAD→ASR→LLM→TTS module deployment steps, Jetson VRAM allocation and quantized model optimization parameters, offline integration test records, and end-to-end latency test report.

Open-source projects you can reproduce

These projects use the same hardware and methods as the course, and what we learn in class is fed back into them.

  • Voice-operated warehouse

    Speak to a Watcher to stock in, stock out or check inventory; the screen reports the quantity and the running total. The whole stack can be deployed locally.

  • Local voice service

    Streaming speech recognition and synthesis on Jetson Orin, RK3576, RK3588 and Raspberry Pi, with under 180 ms latency on Jetson and no cloud dependency.

Bridge physical on-device interaction with enterprise business logic

M2 focuses on deeply connecting visual perception, voice interaction, and legacy business systems (such as WMS and ERP). The curriculum guides learners through physical device integration, MCP tool invocation, and fully offline LAN deployments, providing instructors with a ready-to-verify hardware and software environment.

  • Extension 01

    Adapt to your site conditions

    The voice query logic in Unit 04 is fully customizable. Instructors can replace inventory queries with exhibition guiding, equipment status patrols, or meeting room scheduling, guiding students to design multi-turn dialogue tailored to real operations.

  • Extension 02

    Connect Systems

    Following Unit 08, instructors can replace the sample warehouse database with campus lab inventories or real enterprise APIs, demonstrating live queries and updates via natural spoken dialogue.

  • Extension 03

    Integrate Teaching Experience

    Explain real-world constraints — ambient background noise, far-field pickup boundaries, and hallucination guardrails in tool-calling — so learners grasp how production realities differ from lab demos.

Scope Boundaries & Compliance

L1/L2 business data flows within the LAN via local MCP bridging, core data stays in-domain; L3 runs purely local offline, zero public-network dependency.

In Scope

  • On-Site Target Perception & Structured Business Voice Q&A
  • MCP-Protocol-Based WMS/ERP/CRM System Tool Calling
  • Edge Vision Event Triggering & Automation Task Linkage
  • Multimodal Interaction for Smart Warehousing, Exhibition Guidance, Smart Reception, and Other Scenarios
  • Pure Local Offline Voice Interaction in High-Privacy & Industrial Isolated-Network Environments (L3)

Out of Scope

  • Does not replace high-concurrency, long-chain dedicated human customer service systems
  • Does not guarantee 100% reasoning accuracy for vague, subjective, multi-turn logic
  • Excludes reverse-engineering legacy systems that lack public or documented APIs
  • Not applicable to far-field blind capture in high-noise industrial sites (directional audio capture required)
  • L1/L2 solutions rely on the internet for LLM APIs; without external networks, switch to the L3 offline solution
  • L3 offline solution requires 100 TOPS edge compute (Jetson Orin NX 16GB+); RK3588-40 (6 TOPS) cannot support local LLM inference
  • Speech recognition accuracy is affected by on-site ambient noise, dialect accents, and specialized industry vocabulary; specific recognition accuracy metrics for particular scenarios are not promised

Put M2 on your timetable

Budgets are based on participant count and hardware requirements. Email us your expected group size and target depth to receive a detailed syllabus and kit plan within 3 business days.

Partnership Inquiry View Courses