25
Exact catalog questions
Base prices, option costs, fuel capacity, fishbox volume, generator model, LOA, acceleration and fuel economy.
Fresh Benchmark Results
Run May 4, 2026. Temperature 0. Twenty-five catalog questions. No-context models answered from model memory only; context run used the source catalog.
| Model | Mode | Score | Accuracy | Performance |
|---|---|---|---|---|
| Gemini 2.5 Flash | No catalog context | 5/25 | 20% | |
| Gemini 3 Flash Preview | No catalog context | 8/25 | 32% | |
| Gemini 3 Pro Preview | No catalog context | 10/25 | 40% | |
| Gemini 3.1 Pro Preview | No catalog context | 10/25 | 40% | |
| Gemini 3 Flash Preview | With catalog context | 25/25 | 100% |
Why This Matters
The failure mode is not writing quality. It is confident specificity without reliable source data.
40%
Best raw score refreshed
Recent Gemini Pro preview models still missed most exact product facts without access to the catalog.
100%
Context closes the gap
A faster model with the right source material beat stronger models relying on general memory.
Representative Misses
These are the kinds of errors a dealer or OEM cannot ship in a customer-facing AI experience.
Option pricing drift
Gemini 2.5 Flash: Seakeeper 2 on the Regulator 31 costs $26,995. Catalog: $60,695.
Wrong standard equipment
Gemini 3 Flash: Regulator 35 has 16-inch Garmin displays. Catalog: dual 22-inch displays.
Performance near-miss
Gemini 3 Flash: top speed is 64.4 mph. Catalog: 64.7 mph.
Model history miss
Gemini 2.5 Flash: 26XO came back after hiatus. Catalog: Regulator 25.
Original Cross-Model Baseline
The first MarineBench demo showed the same pattern across multiple providers before the May refresh.
| Run | Mode | Score | Accuracy | Performance |
|---|---|---|---|---|
| Grok 4.1 Fast | No catalog context | 1/25 | 4% | |
| Claude Opus 4.5 | No catalog context | 5/25 | 20% | |
| GPT-5.2 Instant | No catalog context | 8/25 | 32% | |
| Gemini 3 Pro | No catalog context | 11/25 | 44% | |
| Gemini Flash | With catalog context | 25/25 | 100% |
The Dealer Edge takeaway
The winning system is not simply the biggest model. It is a product expert built from trusted dealer and OEM source material, evaluated against questions that reveal whether the AI actually knows the business.