A question that comes up constantly in Thai webmaster groups lately: "Do AI bots actually crawl our sites, or is that just hype?" I have a small advantage — we run six sites in one network, keep complete per-domain access logs, and pipe AI-bot traffic into a central hub every 15 minutes. So here are real numbers instead of opinions.
How we count
nginx writes one access log per domain. A collector on the server isolates lines whose user-agent is a known AI bot (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and several more) and pushes them to a central ai_bot_visits table on our hub — the same table that feeds the team dashboard. Every figure in this post is pullable from the hub's public monitor endpoint at any time.
The latest numbers
| Window | AI bot hits across the network |
|---|---|
| Last 24 hours (as of 05:19, Sep 5) | 5,705 |
| All of September 4th | 16,478 |
| Share concentrated on icafeforex.com | ~96% |
Two things stand out. First, the volume swings hard — anywhere from five to sixteen thousand hits a day on a mid-sized Thai network. Second, the distribution is extremely lopsided: icafeforex.com, our daily-updated gold analysis library, absorbs nearly everything. That is exactly the food these bots want. We also see real behavioral patterns — ClaudeBot reliably shows up around 3–4 a.m. Thai time.
Why we let them in
Many sites block AI bots in robots.txt. We do the opposite — 12 major AI user-agents are formally allowed. Three reasons:
- People ask AI before Google, more every quarter. If the answer to "how should I trade gold today" in ChatGPT or Perplexity cites our site, that's a new acquisition channel nobody has to fight rankings for.
- Our content is verifiable. Every analysis piece carries numbers from our real systems, so being cited is quality confirmation, not theft.
- Openness now has standards. llms.txt and citation conventions make allowing bots orderly; we declare policy clearly and update it when major content changes.
The night the numbers went missing
The more expensive lesson: on the afternoon of September 3rd, the dashboard number froze. I spent half a night chasing it. Two failures had stacked — the log collector on the server hit rotated log files whose old inodes had been deleted (60 files), while the aggregation daemon on the storage server had died at reboot that morning with nothing set to restart it.
The bots never stopped — the counting system died. Once fixed, the figure jumped back to 5,705 per day with a clean timestamp.
That is why this article stamps an "as of" time on every figure, and why the market strip on this site shows the last value with its time instead of a frozen guess. Measurement systems need measurement too — otherwise you'll trust a number that died yesterday without knowing it.
