Skip to content
PLANOVI Docs

Jarvis AI Voice Assistant

Jarvis AI Voice Assistant Module

The Jarvis module (lib/features/jarvis/) provides hands-free conversational voice assistance and operational automation within PLANOVI. It allows users to ask questions regarding energy yields, create calendar appointments, search cooperative documentation, and set up reminders using natural language speech or chat.

Architecture & Data Flow

Jarvis uses a hybrid client/edge architecture. Microphone capture and text-to-speech rendering occur natively on the Flutter client, while NLP intent parsing, LLM generation, and database queries execute inside a serverless Supabase Edge Function (supabase/functions/jarvis/).

sequenceDiagram
    autonumber
    actor User
    participant App as JarvisVoiceScreen
    participant Rec as "VoiceRecorderProvider (record)"
    participant API as JarvisApiService
    participant Edge as "Supabase Edge Function (/jarvis)"
    participant LLM as "OpenAI / AI Model"
    participant DB as Postgres Database
    participant TTS as "JarvisTtsService (flutter_tts)"

    User->>App: Press & hold microphone button
    App->>Rec: startRecording()
    User->>App: Speaks command ("Jaki mamy stan magazynu energii?")
    User->>App: Release microphone button
    App->>Rec: stopRecording() -> yields audioPath (.m4a/.aac)
    
    App->>API: sendToJarvis(audioPath, userTimezone, requiresAudio=true)
    API->>Edge: POST multipart/form-data with Bearer Token
    
    Edge->>LLM: Speech-To-Text / Whisper Transcription
    Edge->>DB: Query tenant energy telemetry & balance
    Edge->>LLM: Formulate contextual response
    Edge-->>API: JSON Response (text, audio_url, pending_action, reminder_data)
    
    API-->>App: ChatMessage added to JarvisChatProvider
    App->>TTS: Play synthesized speech response
    TTS-->>User: Audio playback + animated UI waveform

File Structure & Responsibilities

lib/features/jarvis/
├── models/
│   └── chat_message.dart          # Data model for user/assistant messages & actions
├── providers/
│   ├── jarvis_chat_provider.dart  # Manages active conversation, streaming & actions
│   └── voice_recorder_provider.dart # Audio recording lifecycle, amplitude meters
├── screens/
│   ├── jarvis_screen.dart         # Standard chat interface view
│   ├── jarvis_voice_screen.dart   # Immersive fullscreen voice assistant UI
│   └── jarvis_settings_screen.dart # Voice speed, pitch, language & AI parameters
├── services/
│   ├── jarvis_api_service.dart    # HTTP/Multipart client calling /functions/v1/jarvis
│   ├── jarvis_document_service.dart # Auto-generates summary documents from queries
│   ├── jarvis_tts_service.dart    # Text-To-Speech wrapper using flutter_tts
│   ├── offer_pdf_service.dart     # Generates PDF quotes from assistant recommendations
│   └── reminder_service.dart      # Background timer triggers and alarm player
├── theme/
│   └── jarvis_colors.dart         # Futuristic cyan/neon dark styling tokens
└── widgets/
    ├── chat/                      # Chat bubble widgets, bottom input, quick actions
    └── waveform_visualizer.dart   # Real-time microphone audio visualizer

Key Components

1. JarvisApiService

  • Located at: lib/features/jarvis/services/jarvis_api_service.dart
  • Handles authentication with Supabase:
    • Automatically checks currentSession.isExpired and invokes _supabase.auth.refreshSession() before calling edge endpoints.
    • Dispatches multipart/form-data when an audio recording file is provided.
    • Dispatches standard application/json when sending typed text queries.
  • Parameters transmitted:
    • company_id: Active tenant identifier for RLS enforcement.
    • user_timezone: e.g., Europe/Warsaw, ensuring accurate relative calendar scheduling.
    • requires_audio: Whether the client requests pre-synthesized audio back from the backend.
    • calendar_save_pref: User preference for internal vs Google Calendar storage.

2. VoiceRecorderProvider

  • Uses the record package to capture microphone audio.
  • Emits real-time amplitude values (getAmplitude()) consumed by WaveformVisualizer to render interactive speech waves.
  • Produces compressed M4A/AAC files saved to the temporary application directory.

3. JarvisTtsService

  • Configures flutter_tts for responsive local audio synthesis:
    • Default language: pl-PL (Polish), falling back to en-US.
    • Controls speech rate, pitch, and audio ducking so ambient sounds don’t collide.

4. ReminderService

  • Listens for timed notifications returned in assistant payloads.
  • Uses audioplayers to trigger audible chime tones and displays modal alerts when scheduled reminders trigger.

Intent Actions & Interactive Payloads

When Jarvis determines that a user’s prompt requires a platform action (e.g. creating a meeting, adding an energy device, or submitting a vote), the Edge Function returns a pending_action dictionary.

graph LR
    Intent[NLP Intent Detected] --> Action{Requires Confirmation?}
    Action -->|Yes| Prompt[Render Confirmation Card in Chat]
    Action -->|No| DirectExec[Execute Database Mutation]
    
    Prompt --> UserConfirm[User Clicks 'Zatwierdź']
    UserConfirm --> API_Execute[API Dispatches Confirmed Action]
    API_Execute --> DB[(Postgres Database)]

Examples of supported pending_action intents:

  • schedule_meeting: Prepopulates title, start/end timestamps, and participant emails into the Scheduler module.
  • device_power_limit: Submits a power limitation command to an active inverter via DeviceControlProvider.
  • generate_report: Triggers the generate-report edge function and downloads the generated PDF.

Troubleshooting & Support Guide

1. Microphone Permission Denied

  • Symptom: User taps the microphone, but recording immediately aborts or throws RecordException.
  • Remedy:
    • Android: Verify RECORD_AUDIO permission is declared in AndroidManifest.xml and granted in App Settings.
    • iOS: Check NSMicrophoneUsageDescription in Info.plist.
    • Web: Ensure the browser was accessed over https:// (browsers block microphone APIs on insecure HTTP origins).

2. “Brak aktywnej sesji” / Session Expired

  • Symptom: Red snackbar indicating missing or expired session.
  • Remedy: The JWT token could not be automatically refreshed. The user should log out and log back in to clear invalid cached credentials in FlutterSecureStorage.

3. Edge Function 504 Timeout

  • Symptom: Chat shows indefinite loading spinner or Function timeout error.
  • Remedy: Complex queries involving large PDF context analysis may exceed Deno’s 60-second limit. Advise users to narrow down specific queries or check Edge Function logs in the Supabase Dashboard.