On the night of September 4th I decided to move the AI chat on our Redhat MT5 panel from the cloud to a model we host ourselves. On paper it was a clean win: our fresh 50-case benchmark scored the local model higher on quality (7.24 vs 6.02 average) and far faster (p50 of 5.2s vs 35.1s), with zero per-token cost.
The catch: that benchmark ran on roughly 310-token prompts. Real panel questions are never that short.
What the benchmark didn't say
Before flipping production, we cracked open the 7-day usage records. The numbers made the whole plan wobble:
| 7-day measured value (n = 330) | Measured |
|---|---|
| Average prompt per question | 6,928 tokens |
| Longest prompt | 27,402 tokens |
| Requests predicted over 85s | 153 of 330 = 46% |
| EA hard receive cap | 90 seconds (fixed) |
The panel wraps market context, signal summaries, and chart history into every request. A user's question can be one line; what the model must read is routinely 7,000+ tokens.
The prefill rule
We measured the Mac Mini M4 Pro's prefill — the speed at which it reads the prompt — at 115–128 tokens/second. A 6,928-token prompt therefore spends ~55–60 seconds being read before the model writes a single character of reply.
Direct probes confirmed it: a 7,044-token prompt producing a 153-character answer took 130.2 seconds locally, while the identical question through the cloud finished in 14.7 seconds (7,019 in / 446 out). The cloud path was never the bottleneck — our own prefill was.
Pulling the plug before customers felt it
We flipped the real path at 03:23 on September 5th and tested with our own QC account. The first panel question timed out at 35 seconds. That is the only failed row in the entire migration record, because we rolled the whole path back 30 minutes later — before the first real user ever hit it.
Post-rollback, a confirmation question came back clean: model glm-4.5, a proper Thai answer on support/resistance with reasoning. The system was byte-identical to before.
Lessons for anyone running AI against trading tools
- Measure from production records, not the benchmark. Short-prompt benchmarks will mislead you if your production prompts are 20x longer.
- Know your machine's prefill number. Every token you send has a price in seconds, even when the model is "free" on your own hardware.
- A fast rollback is half the win. We kept the old path wired in parallel, so the entire retreat took 30 minutes and no real user noticed.
The next step is a prompt diet: trim unnecessary history from 7K down under 3K. If that lands, the local model becomes genuinely cheaper and faster. But that's a future post — written after the new numbers come out of the real system.
