Support & Operational Troubleshooting Runbook

This runbook is designed for on-call engineers, developers, and support ticket staff managing the Planovi Backend in production.

---

Triage Quick Reference

Issue / Incident	High Probability Cause	Primary Investigation Step	Fast Mitigation
Telemetry stops updating	InCharge IoT token expired or remote API down	Check VPS logs for ingest-telemetry	Invalidate Deno KV token cache & force sign-in
OCR extraction errors	Blurry upload or Gemini rate limit	Check system_alerts for OCR error codes	Re-upload higher-contrast document or manual review
Invoice calculations incorrect	Missing hourly rollup or tariff override	Verify telemetry_hourly completeness	Run aggregate-hourly on specific window
VPS Webhook fails to deploy	GitHub secret mismatch or Deno syntax error	Inspect /var/log/webhook/deploy.log	Check git log -1 on VPS and re-trigger manually

---

1. Runbook: Telemetry Ingest Failure (ingest-telemetry)

Symptoms

• Live dashboard shows frozen values.
• Flutter app displays “Inverter Offline” warnings.

Diagnosis Steps

1. Check VPS Edge Function Logs:

   ssh root@191.218.165.149
   docker logs --tail 100 supabase-edge-runtime | grep ingest-telemetry

2. Inspect Deno KV Cache: If the log outputs AUTH: using cached token followed immediately by upstream 401 Unauthorized, the cached token was invalidated remotely by the hardware provider before its 24-hour expiry.

Resolution

• Flush the Deno KV cache key:

  deno eval 'const kv = await Deno.openKv(); await kv.delete(["incharge", "token"]); kv.close();'

• Invoke ingest-telemetry manually to trigger a fresh login and verify new records appear in telemetry_raw.

---

2. Runbook: Gemini OCR Declaration Failure (process-declaration-ocr)

Symptoms

• Declaration submission remains stuck in PROCESSING state.
• Confidence score in member_declarations is flagged below 0.85.

Diagnosis Steps

1. Check image resolution and file type in Supabase Storage bucket declarations.
2. Inspect the JSON extraction error in system_alerts:

   SELECT error_message, payload FROM system_alerts
   WHERE component = 'process-declaration-ocr'
   ORDER BY created_at DESC LIMIT 5;

Resolution

• If the PESEL checksum failed due to handwriting ambiguity:
  • Open the management console in the Flutter App.
  • Navigate to Declarations → Manual Review Queue.
  • Enter the verified PESEL and click Approve Extraction.

---

3. Runbook: Billing Settlement Discrepancy (billing-calculator)

Symptoms

• Member reports unexpected grid consumption charges despite solar surplus.

Diagnosis Steps

1. Run a validation query on telemetry_hourly to check if all 24 intervals exist for each day in the billing cycle:

   SELECT date_trunc('day', timestamp) as day, count(*)
   FROM telemetry_hourly
   WHERE coop_id = '<coop_uuid>' AND timestamp >= '2026-08-01' AND timestamp < '2026-09-01'
   GROUP BY 1 HAVING count(*) < 24;

2. If gaps exist:
   • Re-run aggregate-hourly across the missing timestamp range.
   • Re-execute billing-calculator with the force_recalculate: true flag.

---

4. Runbook: VPS Webhook Deployment Debugging (191.218.165.149)

Diagnosis

Check the webhook daemon status on the VPS:

ssh root@191.218.165.149
systemctl status planovi-webhook.service
tail -n 50 /var/log/planovi/deploy.log

Manual Force Deployment

If the webhook failed to trigger due to network interruption:

cd /opt/planovi/planovi-backend
git fetch origin main && git reset --hard origin/main
supabase functions deploy --all