Commit Graph

5 Commits

Author SHA1 Message Date
hailin 64499a5d86 feat(dingtalk): 小龙虾招募全语音/文字引导流程 + OAuth 一键授权卡片
## 功能说明
用户通过语音或文字说「帮我招募一只小龙虾」,iAgent 全程引导完成
OpenClaw 实例创建 + 钉钉 OAuth 一键授权绑定。

## 核心设计
- 语音场景 (claude_agent_sdk): Claude 通过 Bash/wget 调用内部 HTTP
  端点触发 OAuth,绕开 ToolExecutor 限制,两引擎均兼容
- 文字场景 (claude_api): 使用 initiate_dingtalk_binding 自定义工具,
  通过 uiEvent 机制传递 OAuth URL

## agent-service 变更
- agent-engine.port.ts: EngineStreamEvent 联合类型新增 oauth_prompt
- allowed-tools-resolver.service.ts: initiate_dingtalk_binding 加入
  ALL_SDK_TOOLS / admin / operator 工具白名单
- tool-executor.ts: 新增 executeInitiateDingTalkBinding(),调用内部
  oauth/init 端点获取 OAuth URL,返回 uiEvent
- claude-api-engine.ts: 在 tool_result 之后检查 result.uiEvent 并
  yield 出去;buildToolDefinitions 注册 initiate_dingtalk_binding schema
- system-prompt-builder.ts:
  - SystemPromptContext 新增 sessionId? 字段
  - 语音 session (sessionId 存在) → Step 3 使用 wget 调用
    POST /sessions/{sessionId}/dingtalk/oauth-trigger(两引擎通用)
  - 文字 session (无 sessionId) → Step 3 调用 initiate_dingtalk_binding
    工具(claude_api 专用)
- voice-session.controller.ts:
  - 注入 AgentStreamGateway / DingTalkRouterService / AgentInstanceRepository
  - startVoiceSession: 提前确定 sessionId,在 build() 前传入,使系统
    提示能内嵌正确的端点 URL
  - 新增 POST :sessionId/dingtalk/oauth-trigger — 无 JWT(内部端点,
    由 Claude Bash 工具调用),sessionId 作为能力令牌;生成 OAuth URL
    并通过 gateway.emitStreamEvent 直接推送 oauth_prompt 事件到 WS 流

## voice-agent 变更
- agent.py: 构造 AgentServiceLLM 时传入 room=ctx.room
- agent_llm.py:
  - __init__ 增加 room 参数,存储为 self._room
  - 新增 _publish_oauth_prompt(evt_data): null-safe,通过 LiveKit
    publish_data(topic="oauth_prompt") 推送到 Flutter
  - _do_inject_voice / _do_inject / _do_stream_voice / _do_stream:
    处理 oauth_prompt 事件 → asyncio.create_task(_publish_oauth_prompt)
  - 替换已弃用的 asyncio.ensure_future / get_event_loop().create_task
    → asyncio.create_task(Python 3.10+ 兼容)

## Flutter 变更
- agent_call_page.dart: DataReceivedEvent 监听 topic="oauth_prompt",
  解析 url/instanceName,弹出 _showOAuthBottomSheet(深色主题,🦞
  图标,「立即授权」按钮 launchUrl externalApplication)
- stream_event.dart: 新增 OAuthPromptEvent(url, instanceId, instanceName)
- stream_event_model.dart: toEntity() 新增 'oauth_prompt' case
- chat_message.dart: MessageType 枚举新增 oauthPrompt
- chat_providers.dart: _handleStreamEvent 新增 OAuthPromptEvent case,
  生成 type=oauthPrompt 的 ChatMessage(metadata 含 url/instanceName)
- chat_page.dart: 新增 oauthPrompt 时间线节点 + _OAuthPromptCard 组件
  (「立即授权」按钮,launchUrl externalApplication);import url_launcher

## 修复的关键 Bug
1. [严重] initiate_dingtalk_binding 只对 claude_api 有效,语音默认用
   claude_agent_sdk → 新 wget 端点两引擎均可用
2. [严重] 文字聊天页面不处理 oauth_prompt 事件(静默丢弃)→ 补全
   Flutter 4 处代码(entity/model/provider/page)
3. [中]   _publish_oauth_prompt 缺 local_participant null 检查 → 已修复
4. [轻]   asyncio.ensure_future / get_event_loop() 弃用警告 → 已修复

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-08 11:22:06 -07:00
hailin e4c2505048 feat: add multimodal image input with streaming markdown optimization
Two major features in this commit:

1. Streaming Markdown Rendering Optimization
   - Replace deprecated flutter_markdown with gpt_markdown (active, AI-optimized)
   - Real-time markdown rendering during streaming (was showing raw syntax)
   - Solid block cursor (█) instead of AnimationController blink
   - 80ms token throttle buffer reducing rebuilds from per-token to ~12.5/sec
   - RepaintBoundary isolation for markdown widget repaints
   - StreamTextWidget simplified from StatefulWidget to StatelessWidget

2. Multimodal Image Input (camera + gallery + display)
   - Flutter: image_picker for gallery/camera, base64 encoding, attachment
     preview strip with delete, thumbnails in sent messages
   - Data layer: List<String>? → List<Map<String, dynamic>>? for structured
     attachment payloads through datasource/repository/usecase
   - ChatAttachment model with base64Data, mediaType, fileName
   - ChatMessage entity + ChatMessageModel both support attachments field
   - Backend DTO, Entity (JSONB), Controller, ConversationContextService
     all extended to receive, store, and reconstruct Anthropic image
     content blocks in loadContext()
   - Claude API engine skips duplicate user message when history already
     ends with multimodal content blocks
   - NestJS body parser limit raised to 10MB for base64 image payloads
   - Android CAMERA permission added to manifest
   - Image.memory uses cacheWidth/cacheHeight for memory efficiency
   - Max 5 images per message enforced in UI

Data flow:
  ImagePicker → base64Encode → ChatAttachment → POST body →
  DB (JSONB) → loadContext → Anthropic image content blocks → Claude API

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 03:24:17 -08:00
hailin d5f663f7af feat: inject-message support for mid-stream task interruption
Backend (agent-engine.port.ts):
- Add `cancelled` event type: emitted when a task is cancelled (user-initiated
  or injection), so Flutter can close the old stream cleanly
- Add `task_info` event type: emitted after inject to pass the new taskId to
  the client, enabling cancel/re-inject on the replacement task

Flutter (features/chat/):
- ChatState: track current `taskId` alongside `sessionId`; clear on completion
  or error
- Handle `TaskInfoEvent`: update taskId in state when server issues a new task
- Handle `CancelledEvent`: treat as stream termination (agentStatus → idle)
- MessageType.interrupted: new UI node (warning style) for mid-stream cancels
- _inject(): send text as an inject request while streaming; backend cancels
  the current task and starts a new one with the injected message
- Input area: during streaming, hint changes to "追加指令...", Enter key calls
  _inject() instead of _send(), and both inject-send + stop buttons are shown
- isAwaitingApproval kept separate from isStreaming so approval flow is not
  blocked by inject mode

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-27 21:33:50 -08:00
hailin 2403ce5636 feat: multi-turn conversation context management with session history UI
Implement DB-based conversation message storage (engine-agnostic) that
works across both Claude API and Agent SDK engines. Add ChatGPT/Claude-style
conversation history drawer in Flutter with date-grouped session list,
session switching, and new chat functionality.

Backend: entity, repository, context service, migration 004, session/message
API endpoints. Flutter: ConversationDrawer, sessionId flow from backend
response via SessionInfoEvent, session list/switch/delete support.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:04:35 -08:00
hailin 00f8801d51 Initial commit: IT0 AI-powered server cluster operations platform
Full-stack monorepo with DDD + Clean Architecture:
- Backend: 7 NestJS microservices + 5 shared libraries (TypeScript)
- Mobile: Flutter app with Riverpod (Dart)
- Web Admin: Next.js dashboard with Zustand + React Query
- Voice: Python voice service (STT/TTS/VAD)
- Infra: Docker Compose, K8s manifests, Turborepo build

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:54:37 -08:00