hailin
ce63ece340
Enable users to send text messages, images, and files to the Agent while an active voice call is in progress. This addresses the case where spoken instructions are unclear or screenshots/documents need to be shared for analysis. ## Architecture Data flows through LiveKit data channel (not direct HTTP): Flutter → publishData(topic='text_inject') → voice-agent → llm.inject_text_message() → POST /api/v1/agent/tasks (same session) → collect streamed response → session.say() → TTS playback This preserves the constraint that voice-agent owns the agent-service sessionId — Flutter never contacts agent-service directly. ## Flutter UI (agent_call_page.dart) - Add keyboard toggle button to active call controls (4-button row) - Collapsible text input area with attachment picker (+) and send button - Attachment support: gallery multi-select, camera, file picker (images max 1024x1024 quality 80%, PDF supported, max 5 attachments) - Horizontal scrolling attachment preview with delete buttons - 200KB payload size check before LiveKit data channel send - Layout adapts: Spacer flex 1/3 toggle, reduced bottom padding ## voice-agent (agent.py) - Register data_received event listener after session.start() - Filter for topic='text_inject', parse JSON payload - Call llm.inject_text_message(text, attachments) and TTS via session.say() - Use asyncio.ensure_future() wrapper for async handler (matches existing disconnect handler pattern for sync EventEmitter) ## AgentServiceLLM (agent_llm.py) - New inject_text_message(text, attachments) method on AgentServiceLLM - Reuses same _agent_session_id for conversation context continuity - WS+HTTP streaming: connect, pre-subscribe, POST /tasks with attachments field, collect full text response, return string - _injecting flag prevents concurrent _do_stream from clearing session ID on abort errors while inject is in progress - Same systemPrompt/voiceMode/engineType as voice pipeline No agent-service changes required — attachments already supported end-to-end (JSONB storage → multimodal content blocks → Claude). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| android | ||
| assets | ||
| ios/Runner/Assets.xcassets/AppIcon.appiconset | ||
| lib | ||
| test | ||
| .gitignore | ||
| .metadata | ||
| README.md | ||
| analysis_options.yaml | ||
| pubspec.lock | ||
| pubspec.yaml | ||
README.md
it0_app
A new Flutter project.
Getting Started
This project is a starting point for a Flutter application.
A few resources to get you started if this is your first Flutter project:
For help getting started with Flutter development, view the online documentation, which offers tutorials, samples, guidance on mobile development, and a full API reference.