Commit Graph

13 Commits

Author SHA1 Message Date
hailin 8d2fd3335a feat(telemetry): add presence-service + Flutter telemetry module
## Backend — packages/services/presence-service (新微服务)

完整的 DDD + Clean Architecture 实现,移植自 RWADurian presence-service,
针对 IT0 架构做了以下适配:

### 核心功能
- 心跳接口: POST /api/v1/presence/heartbeat(JWT 验证,60s 间隔)
  → Redis Sorted Set `presence:online_users` 记录在线时间戳
  → 默认 5 分钟窗口判断在线(PRESENCE_WINDOW_SECONDS=300)
- 事件上报: POST /api/v1/analytics/events(批量,最多 50 条)
  → 写入 presence_event_log 表 + 更新 presence_device_profile
  → Redis HyperLogLog `presence:dau:{date}` 实时 DAU 估算
- 查询接口(需 AdminGuard):
  - GET /api/v1/analytics/online-count  — 实时在线人数
  - GET /api/v1/analytics/online-history — 历史在线快照
  - GET /api/v1/analytics/dau — DAU 统计

### IT0 适配要点
- JWT payload: `sub` = UUID userId(非 RWADurian 的 userSerialNum)
  → JwtAuthGuard: request.user = { userId: payload.sub, roles, tenantId }
- AdminGuard: 改为检查 `roles.includes('admin')`(非 type==='admin')
- 移除 Kafka EventPublisherService(IT0 无 Kafka)
- 移除 Prometheus MetricsService(IT0 无 Prometheus)
- 表前缀改为 `presence_`(避免与其他服务冲突)
- userId 字段 VarChar(36)(UUID 格式,非原来的 VarChar(20))
- Redis DB=10 隔离(独立 key 空间)

### 数据库表(public schema)
- presence_event_log       — 事件流水(append-only)
- presence_device_profile  — 设备快照(upsert,每台设备一行)
- presence_daily_active_users — DAU 日统计
- presence_online_snapshots   — 在线人数每分钟快照

### 定时任务(@nestjs/schedule)
- 每分钟: 采集在线人数快照 → presence_online_snapshots
- 每天 01:05 (UTC+8): 计算前一天 DAU → presence_daily_active_users

---

## Flutter — it0_app/lib/core/telemetry (新模块)

### 文件结构
- telemetry_service.dart      — 单例入口,统筹所有组件
- models/telemetry_event.dart — 事件模型,toServerJson() 将设备字段提升为顶层列
- models/device_context.dart  — 设备上下文(Android/iOS 信息)
- models/telemetry_config.dart — 远程配置(采样率/开关,支持远端同步)
- collectors/device_info_collector.dart — 采集 device_info_plus 设备信息
- storage/telemetry_storage.dart  — SharedPreferences 队列(最多 500 条)
- uploader/telemetry_uploader.dart — 批量上传到 /api/v1/analytics/events
- session/session_manager.dart    — WidgetsBindingObserver 监听前后台切换
- session/session_events.dart     — 会话事件常量
- presence/heartbeat_service.dart — 定时心跳 POST /api/v1/presence/heartbeat
- presence/presence_config.dart   — 心跳配置(间隔/requiresAuth)
- telemetry.dart                  — barrel 导出

### 集成点
- app_router.dart _tryRestore(): TelemetryService().initialize() 在 auth 之前
- auth_provider.dart login/loginWithOtp: setUserId + setAccessToken + resumeAfterLogin
- auth_provider.dart tryRestoreSession: 恢复 userId + accessToken
- auth_provider.dart logout: pauseForLogout + clearUserId + clearAccessToken

### 新增依赖
- device_info_plus: ^10.1.0
- equatable: ^2.0.5

---

## 基础设施

### Dockerfile.service
- 在 builder 和 production 阶段均添加 presence-service/package.json 的 COPY

### docker-compose.yml
- 新增 presence-service 容器(端口 3011/13011)
  - DATABASE_URL: postgresql://... (Prisma 所需连接串格式)
  - REDIS_HOST/PORT/DB: 10(presence 独立 Redis DB)
  - APP_PORT=3011, JWT_SECRET, PRESENCE_WINDOW_SECONDS=300
- api-gateway depends_on 新增 presence-service

### kong.yml (dbless 声明式)
- 新增 presence-service 服务(http://presence-service:3011)
  - presence-routes: /api/v1/presence
  - analytics-routes: /api/v1/analytics
- 对整个 presence-service 启用 JWT 插件(Kong 层鉴权)

### DB 迁移
- packages/shared/database/src/migrations/010-create-presence-tables.sql
  — 4 张 presence_ 前缀表 + 完整索引(IF NOT EXISTS 幂等)
- run-migrations.ts: runSharedSchema() 中新增执行 010-create-presence-tables.sql

---

## 部署步骤(服务器)

1. git pull
2. 执行 presence 表迁移(首次):
   docker exec it0-postgres psql -U it0 -d it0 \
     -f /path/to/010-create-presence-tables.sql
   或通过 migration runner:
   cd /home/ceshi/it0 && node packages/shared/database/dist/run-migrations.js
3. 重建并启动 presence-service:
   docker compose build presence-service api-gateway
   docker compose up -d presence-service api-gateway

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 17:44:01 -08:00
hailin 94a14b3104 feat: migrate voice call from WebSocket/PCM to LiveKit WebRTC
实时语音对话架构迁移:WebSocket → LiveKit WebRTC

## 背景
原语音通话架构基于 FastAPI WebSocket 传输原始 PCM,管道串行执行
(VAD → 批量STT → Agent → 攒句 → 批量TTS),首音频延迟约 6 秒。
迁移到 LiveKit Agents 框架后,利用 WebRTC 传输 + 流水线并行,
预期延迟降至 1.5-2 秒。

## 架构
Flutter App ←── WebRTC (Opus/UDP) ──→ LiveKit Server ←──→ Voice Agent
  livekit_client                      (自部署, Go)       (Python, LiveKit Agents SDK)
                                                          ├─ VAD (Silero)
                                                          ├─ STT (faster-whisper / OpenAI)
                                                          ├─ LLM (自定义插件 → agent-service)
                                                          └─ TTS (Kokoro / OpenAI)

关键设计:LLM 不直接调用 Claude API,而是通过自定义插件代理到现有
agent-service,保留 Tool Use、会话历史、租户隔离等能力。

## 新增服务

### voice-agent (packages/services/voice-agent/)
LiveKit Agent Worker,包含:
- agent.py: 入口,prewarm() 预加载模型,entrypoint() 编排会话
- plugins/agent_llm.py: 自定义 LLM 插件,代理 agent-service API
  - POST /api/v1/agent/tasks 创建任务
  - WS /ws/agent 订阅流式事件 (stream_event)
  - 跨轮复用 session_id 保持对话上下文
- plugins/whisper_stt.py: 本地 faster-whisper STT (批量识别)
- plugins/kokoro_tts.py: 本地 Kokoro-82M TTS (24kHz PCM)
- config.py: pydantic-settings 配置

### LiveKit Server (deploy/docker/)
- livekit.yaml: 信令端口 7880, RTC TCP 7881, UDP 50000-50200
- docker-compose.yml: 新增 livekit-server + voice-agent 容器

### LiveKit Token 端点
- voice-service/src/api/livekit_token.py:
  POST /api/v1/voice/livekit/token
  生成 Room JWT,嵌入 auth_header 到 AgentDispatch metadata

## Flutter 客户端改造
- agent_call_page.dart: 从 ~814 行简化到 ~380 行
  - 替换: WebSocketChannel, AudioRecorder, PcmPlayer, 手动心跳/重连
  - 使用: Room.connect(), setMicrophoneEnabled(true), LiveKit 事件监听
  - 波形动画改用 participant.audioLevel
- pubspec.yaml: 添加 livekit_client: ^2.3.0
- app_config.dart: 增加 livekitUrl 字段
- api_endpoints.dart: 增加 livekitToken 端点

## 配置说明 (环境变量)
- STT_PROVIDER: local (默认, faster-whisper) / openai
- TTS_PROVIDER: local (默认, Kokoro) / openai
- WHISPER_MODEL: base (默认) / small / medium / large
- WHISPER_LANGUAGE: zh (默认)
- KOKORO_VOICE: zf_xiaoxiao (默认)
- DEVICE: cpu (默认) / cuda

## 不变的部分
- agent-service: 完全不改,voice-agent 通过现有 API 调用
- voice-service 核心: pipeline/STT/TTS/VAD 保留 (Twilio 备用)
- Kong 网关: 现有路由不变
- 数据库: 无 schema 变更

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 08:55:33 -08:00
hailin 1f1bf18a75 fix: remove clipboard paste menu item, fix timeline line overlap, dim input placeholder
- Remove redundant "从剪贴板粘贴" option from attachment menu (long-press to paste natively)
- Remove super_clipboard dependency (no longer needed)
- Fix timeline vertical line overlapping icon nodes by using dynamic dotRadius
- Dim input field placeholder color to AppColors.textMuted

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 05:05:27 -08:00
hailin 5f28605e13 feat: add clipboard paste, multi-image select, and file picker
- Add super_clipboard and file_picker dependencies
- Clipboard paste: reads PNG/JPEG image data from system clipboard
- Multi-image: pickMultiImage with remaining count limit
- File picker: supports images (jpg/png/gif/webp) and PDF files
- Updated attachment preview to show file icon for non-image types
- Bottom sheet now shows 4 options: gallery, camera, clipboard, file

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 04:32:16 -08:00
hailin e4c2505048 feat: add multimodal image input with streaming markdown optimization
Two major features in this commit:

1. Streaming Markdown Rendering Optimization
   - Replace deprecated flutter_markdown with gpt_markdown (active, AI-optimized)
   - Real-time markdown rendering during streaming (was showing raw syntax)
   - Solid block cursor (█) instead of AnimationController blink
   - 80ms token throttle buffer reducing rebuilds from per-token to ~12.5/sec
   - RepaintBoundary isolation for markdown widget repaints
   - StreamTextWidget simplified from StatefulWidget to StatelessWidget

2. Multimodal Image Input (camera + gallery + display)
   - Flutter: image_picker for gallery/camera, base64 encoding, attachment
     preview strip with delete, thumbnails in sent messages
   - Data layer: List<String>? → List<Map<String, dynamic>>? for structured
     attachment payloads through datasource/repository/usecase
   - ChatAttachment model with base64Data, mediaType, fileName
   - ChatMessage entity + ChatMessageModel both support attachments field
   - Backend DTO, Entity (JSONB), Controller, ConversationContextService
     all extended to receive, store, and reconstruct Anthropic image
     content blocks in loadContext()
   - Claude API engine skips duplicate user message when history already
     ends with multimodal content blocks
   - NestJS body parser limit raised to 10MB for base64 image payloads
   - Android CAMERA permission added to manifest
   - Image.memory uses cacheWidth/cacheHeight for memory efficiency
   - Max 5 images per message enforced in UI

Data flow:
  ImagePicker → base64Encode → ChatAttachment → POST body →
  DB (JSONB) → loadContext → Anthropic image content blocks → Claude API

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 03:24:17 -08:00
hailin f5d9b1f04f feat: add app upgrade system with self-hosted APK update support
- Add core/updater module: version checker, download manager (resumable + SHA-256),
  APK installer, app market detector, self-hosted updater with progress dialogs
- Add Android native MethodChannels for APK installation and market detection
- Add FileProvider config and REQUEST_INSTALL_PACKAGES permission
- Wire UpdateService singleton into main.dart initialization
- Add auto-check on home entry with cooldown + app resume detection
- Add manual "检查更新" button and dynamic version display in settings
- Fix chat page: bottom overflow, bash spinner persistence, collapsible results
- Merge standing orders into tasks page as second tab

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 22:35:01 -08:00
hailin e706a4cdc7 fix: enable simultaneous playback + recording in voice call
Root cause: PcmPlayer called openPlayer() without audio session config,
so Android defaulted to earpiece-only mode. When the mic was actively
recording, playback was silently suppressed — the agent's TTS audio was
sent successfully over WebSocket but never reached the speaker.

Changes:

1. PcmPlayer (pcm_player.dart):
   - Added audio_session package for proper audio session management
   - Configure AudioSession with playAndRecord category so mic + speaker
     work simultaneously
   - Set voiceCommunication usage to enable Android hardware AEC (echo
     cancellation) — prevents feedback loops when speaker is active
   - defaultToSpeaker routes output to loudspeaker instead of earpiece
   - Restored setSpeakerOn() method stub (used by UI toggle)

2. AgentCallPage (agent_call_page.dart):
   - Fixed fire-and-forget bug: _pcmPlayer.feed() returns Future but was
     called without await, causing interleaved feedUint8FromStream calls
   - Added _feedChain serializer to guarantee sequential audio feeding

3. Dependencies:
   - Added audio_session package to pubspec.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 06:48:16 -08:00
hailin 8015154a3e feat: replace default Flutter icon with iAgent robot logo
使用项目自有的绿色机器人 SVG logo 生成各分辨率 Android 启动图标,
替换默认的 Flutter 蓝色纸飞机。支持 Android Adaptive Icon(白底 + 机器人前景)。
同时生成 iOS 图标。

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:41:36 -08:00
hailin 092a561867 feat: 完成 iAgent App 三大功能 + 修复租户上下文
## 功能一:设置页(完整实现)
- 新增浅色主题(lightTheme),支持深色/浅色/跟随系统三种模式
- app.dart 接入 themeMode 动态切换
- 设置页完整重写:个人信息编辑、修改密码、主题切换、通知开关
- 新增 settings_remote_datasource 对接后端 admin/settings API
- settings_providers 新增 AccountProfileNotifier 管理远程个人资料

## 功能二:语音通话(音频集成)
- 添加 flutter_sound 依赖,创建 PcmPlayer 流式 PCM 播放器
- agent_call_page 替换空壳:真实麦克风采集(record + GTCRN 降噪)
- 真实 PCM 16kHz 流式播放,基于 RMS 能量驱动波形动画
- 修复 WebSocket URL 路径:/ws/voice/ → /api/v1/voice/ws/
- voice_repository_impl 支持后端返回相对路径自动拼接

## 功能三:推送通知(WebSocket MVP)
- 添加 flutter_local_notifications + socket_io_client 依赖
- 创建 AppNotification 实体、NotificationService(Socket.IO 连接 comm-service)
- 通知 providers:列表管理 + 未读计数
- 登录后自动连接通知服务,登出断开
- 底部导航 Alerts 标签添加未读角标(Badge)
- AndroidManifest 添加 POST_NOTIFICATIONS 权限
- main.dart 初始化本地通知插件

## 修复:租户上下文未初始化(500错误)
- 根因:登录后未设置 currentTenantIdProvider,导致 X-Tenant-Id 头缺失
- Flutter 端:login() 成功后从 JWT 设置 tenantId,logout 时清除
- 后端:tenant-context.middleware 增加 JWT tenantId 回退逻辑
- AuthUser 模型新增 tenantId 字段解析

新增 5 个文件,修改 16 个文件,添加 3 个依赖包

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:10:52 -08:00
hailin a568558585 feat: replace speech_to_text with GTCRN ML noise reduction + backend STT
Replace traditional on-device speech_to_text with a modern pipeline:
- Record audio via `record` package with hardware noise suppression
- Apply GTCRN neural denoising (sherpa-onnx, ICASSP 2024, 48K params)
- Trim silence, POST to backend /voice/transcribe (faster-whisper)

Changes:
- Add /transcribe endpoint to voice-service for audio file upload
- Add SpeechEnhancer wrapper for sherpa-onnx GTCRN model (523KB)
- Rewrite chat_page.dart voice input: record → denoise → transcribe
- Keep NoiseReducer.trimSilence for silence removal only
- Upgrade record to v6.2.0, add sherpa_onnx, path_provider

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 07:59:15 -08:00
hailin 39c0d83424 feat: rename app from IT0 to iAgent (我智能体)
Web Admin:
- Update browser title to "iAgent Admin Console"
- Translation: en appTitle "iAgent", zh appTitle "我智能体"
- Sidebar: en "iAgent Admin", zh "我智能体"
- Settings placeholder: "iAgent Platform" / "我智能体平台"
- Update alt tags on logo images

Flutter:
- MaterialApp title: "iAgent"
- Chat: "Ask iAgent..." / "Start a conversation with iAgent"
- Terminal: "iAgent Remote Terminal"
- Agent call: "iAgent Calling" / "iAgent"

Logo SVG: text changed from "AI AGENT" to "iAgent"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 06:39:40 -08:00
hailin 2f228e4591 feat: integrate robot logo into web admin and Flutter app
- Add logo.svg (green robot character) to project root
- Web admin: replace text "IT" badge with SVG logo in sidebar
- Web admin: add logo image to login, register, invite pages
- Web admin: add SVG favicon and apple-touch-icon metadata
- Flutter: add flutter_svg dependency, replace text "IT0" with logo on login page

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 06:17:08 -08:00
hailin 00f8801d51 Initial commit: IT0 AI-powered server cluster operations platform
Full-stack monorepo with DDD + Clean Architecture:
- Backend: 7 NestJS microservices + 5 shared libraries (TypeScript)
- Mobile: Flutter app with Riverpod (Dart)
- Web Admin: Next.js dashboard with Zustand + React Query
- Voice: Python voice service (STT/TTS/VAD)
- Infra: Docker Compose, K8s manifests, Turborepo build

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 22:54:37 -08:00