Fix receipt scanner AI response parsing #11

Merged

danshtu merged 1 commit from fix/receipt-scanner-parsing into master

2026-03-01 10:46:09 -05:00

theoassistant commented

2026-03-01 10:21:05 -05:00

Collaborator

Fixes the receipt scanner that was failing to populate form fields after AI processing.

Problem

The receipt scanner AI was working correctly but the form wasn't being populated. Looking at the console output, the Firebase AI SDK returns responses in a different format than the standard Gemini SDK:

[
  {
    "event.name": "gen_ai.choice",
    "message": {
      "content": [{"text": "{...JSON data...}"}]
    }
  }
]

The code was expecting result.response.text() but the actual structure wraps the response in telemetry events.

Changes

1. Add Firebase AI response format detection

Check if response is an array (Firebase AI format with telemetry)
Find the gen_ai.choice event and extract text from message.content[0].text
Fall back to standard Gemini format if not array
Add error logging for unexpected formats

2. Add "beverages" category to AI prompt

Added guidance for lemonade, juices, drinks → "beverages" category
Also added oat milk to the milk category examples

Testing

The user uploaded a Food Lion receipt with:

FL Straw Lemonade (x2)
Oatly Oat Milk (x1)
Whole Milk Gallon (x5)
Bag Lemons (x1)
Strawberries 1lb (x1)

The AI successfully parsed it but the form didn't populate. With this fix, the response will now be properly extracted and the form will auto-fill.

Result

✅ Receipt scanner now works with Firebase AI SDK
✅ Handles both Firebase AI and standard Gemini response formats
✅ Better error messages for debugging
✅ Added beverages category for drinks/ingredients

Fixes the receipt scanner that was failing to populate form fields after AI processing. ## Problem The receipt scanner AI was working correctly but the form wasn't being populated. Looking at the console output, the Firebase AI SDK returns responses in a different format than the standard Gemini SDK: ```json [ { "event.name": "gen_ai.choice", "message": { "content": [{"text": "{...JSON data...}"}] } } ] ``` The code was expecting `result.response.text()` but the actual structure wraps the response in telemetry events. ## Changes ### 1. Add Firebase AI response format detection - Check if response is an array (Firebase AI format with telemetry) - Find the `gen_ai.choice` event and extract text from `message.content[0].text` - Fall back to standard Gemini format if not array - Add error logging for unexpected formats ### 2. Add "beverages" category to AI prompt - Added guidance for lemonade, juices, drinks → "beverages" category - Also added oat milk to the milk category examples ## Testing The user uploaded a Food Lion receipt with: - FL Straw Lemonade (x2) - Oatly Oat Milk (x1) - Whole Milk Gallon (x5) - Bag Lemons (x1) - Strawberries 1lb (x1) The AI successfully parsed it but the form didn't populate. With this fix, the response will now be properly extracted and the form will auto-fill. ## Result ✅ Receipt scanner now works with Firebase AI SDK ✅ Handles both Firebase AI and standard Gemini response formats ✅ Better error messages for debugging ✅ Added beverages category for drinks/ingredients

theoassistant added 1 commit

2026-03-01 10:21:07 -05:00

Fix receipt scanner AI response parsing 3916b62bd5

The Firebase AI SDK returns responses in a different format than the standard
Gemini SDK. It wraps responses in telemetry events with this structure:

[
  {
    "event.name": "gen_ai.choice",
    "message": {
      "content": [{"text": "..."}]
    }
  }
]

This was causing the scanner to fail silently - the AI was working correctly
but the response wasn't being parsed.

Changes:
- Add response format detection to handle both Firebase AI and standard Gemini formats
- Extract text from telemetry events when present
- Add "beverages" category to AI prompt (for lemonade, juices, etc.)
- Add better error logging for unexpected response formats

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>