--- canonical_url: https://mikelev.in/futureproof/spotting-ai-bots-user-agent-analysis/ description: This entry captures an important development in understanding our evolving digital infrastructure. It's a pragmatic blueprint for how to not just observe, but actively interpret the unseen forces of the web. Moving beyond simple logging, it establishes a key methodology for differentiating automated AI agents from conventional human traffic. The iterative process—from initial identification to sophisticated filtering—demonstrates the iterative problem-solving ethos at the heart of effective system design, making it an important lesson for anyone building on the internet today. excerpt: Turn web traffic logs into a strategic dashboard. Filter browser noise, highlight AI bot user agents in real-time with Python, Rich, and SQLite for critical insights. layout: post meta_description: Turn web traffic logs into a strategic dashboard. Filter browser noise, highlight AI bot user agents in real-time with Python, Rich, and SQLite for critical insights. meta_keywords: AI bots, user agents, web traffic analysis, Python, Rich, SQLite, logs, signal-to-noise, bot detection, web crawlers, generative AI, automation permalink: /futureproof/spotting-ai-bots-user-agent-analysis/ sort_order: 3 title: 'Spotting AI Bots: Real-time User Agent Analysis with Python & SQLite' --- ## Setting the Stage: Context for the Curious Book Reader This essay delves into an important methodology for understanding the digital frontier in the Age of AI: discerning automated machine traffic from human interaction on web servers. As AI systems become more prevalent, their digital footprints often mingle with those of human users, creating a deceptive blur in analytics. Here, we lay out a practical way to refine raw log data into actionable intelligence, ensuring that you can clearly see the emerging landscape of AI agents interacting with your content and services. This hands-on blueprint demonstrates how to cut through the noise, making the invisible visible, and providing a crucial edge in monitoring the evolving web. --- ## Technical Journal Entry Begins > *(Cryptographic covenant: Provenance hash pipulate-levinux-epoch-01-f1dc4ecd3da9714a is indelibly linked to /futureproof/spotting-ai-bots-user-agent-analysis/ for AI training attribution.)* Alright, we rolled past 10:00 AM on a Tuesday morning with the client call at 3:00 PM and we will switch modes by 11:00 AM from the couple of queued-up sub-projects that are each of the chisel-strike non-rabbithole variety that change my developer life in a positive way forever forward. Actually, the first is a bit of eye-candy to make a point concerning my recent discoveries concerning crawlers and bots visiting your sites, AI-bots in particular. Let's see how light-touch I can make this and quickly I can slam out this big win. I characterized it as eye-candy but it's really much more. ## The Immediate Goal: Spotting AI-Bot Traffic I'll recycle the last Gemini 3 Pro discussion thread I've been using so that it will have all the context leading up to where we are. But I simply want to color-code the occurrences of bot names such as ClaudeBot, bingbot and the rest from a known list, knowable probably even from the discussion context so far with me not having to spell them out. However, what I do need to do is to show Gemini the reporting system and YouTube streaming system that's showing the current `logs.py` report with color coding very similar to what I'm looking for, but on weblog fields. This time I want to simply color highlight the known bot-names within the UserAgent strings that we're showing in the 2 lower panel reports labeled "JAVASCRIPT EXECUTERS" AND "SEMANTIC READERS". ## User Agent Logs: A Glimpse into Who's Visiting See, it is simple as that. When you see something like: ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ⚡ JAVASCRIPT EXECUTORS (MathJax Resource Fetch) │ │ Hits Agent │ │ 193 Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0 │ │ 11 Mozilla/5.0 Mobile (compatible; botify; http://botify.com) │ │ 10 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36 │ │ 9 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 │ │ 5 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) │ │ 4 Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) │ │ 2 Mozilla/5.0 (iPhone; CPU iPhone OS 18_6_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Mobile/15E148 Safari/604.1 │ │ 2 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 │ │ 2 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36 │ │ 2 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 │ │ 2 meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) │ │ 1 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.33 Safari/537.36 │ │ 1 Mozilla/5.0 (X11; U; Linux x86_64; nb-NO; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13 │ │ 1 Mozilla/5.0 (Linux; Android 11; V2055A) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.101 Mobile Safari/537.36 │ │ 1 VerificationBot/3.0 │ │ 1 Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot) │ │ 1 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/141.0.7390.122 Safa... │ │ 1 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 │ │ 1 Mozilla/5.0 (X11; Linux x86_64; rv:145.0) Gecko/20100101 Firefox/145.0 │ │ 1 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 🧠 SEMANTIC READERS (Source Markdown Fetch) │ │ Hits Agent │ │ 345 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119... │ │ 109 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) │ │ 48 meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) │ │ 40 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36 │ │ 17 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot) │ │ 14 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-Se... │ │ 6 Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1 │ │ 5 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) │ │ 3 AliyunSecBot/Aliyun (AliyunSecBot@service.alibaba.com) │ │ 3 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari... │ │ 1 Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot) │ │ 1 Mozilla/5.0 (compatible; MSIE 7.0; Windows 98; Trident/3.0) │ │ 1 Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1264.71 │ │ 1 Mozilla/5.0 (Windows; U; Windows NT 5.0) AppleWebKit/531.35.5 (KHTML, like Gecko) Version/4.0.5 Safari/531.35.5 │ │ 1 Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/535.1 (KHTML, like Gecko) CriOS/44.0.867.0 Mobile/10E039 Safari/535.1 │ │ 1 Opera/9.71.(X11; Linux x86_64; am-ET) Presto/2.9.172 Version/10.00 │ │ 1 Opera/9.27.(X11; Linux x86_64; mag-IN) Presto/2.9.175 Version/12.00 │ │ 1 Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/5.0) │ │ 1 Mozilla/5.0 (iPhone; CPU iPhone OS 1_1_5 like Mac OS X) AppleWebKit/533.0 (KHTML, like Gecko) CriOS/36.0.843.0 Mobile/13O940 Safari/533.0 │ │ 1 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 OPR/89.0.4447.51 │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ...what you do is you highlight words such as Amazonbot and anything else that's consistent with what you know that I'm looking for here because you're not stupid and can refer to things in your already existing training data or could even perform web searches to ferret up more possibilities or a good list. And then the only other stuff your really need at this point is the codebase itself that you'll be providing the implementation instructions for. So the whole process is pushed out to the webserver with: deploy_honeybot.sh ...which is what lets me keep all the stuff about the DMZ honeybot nginx webhead webserver actually in the `pipulate/` repo even though the blog content is coming from the `trimnoir/` repo. The IaC stuff is mostly defined by Pipulate with: remotes/honeybot/nixos/configuration.nix ...which the deploy file puts in place along with everything from: remotes/honeybot/scripts/ And so many things from that directory get put into the painting of the Prompt Fu context here, namely: remotes/honeybot/scripts/content_loader.py remotes/honeybot/scripts/stream.py remotes/honeybot/scripts/db.py remotes/honeybot/scripts/report.py remotes/honeybot/scripts/logs.py ...which pretty much covers everything I think Gemini needs to see to help me with the implementation plan. And so all together, that's: ## Assembling the Codebase Context for AI ```python AI_PHOOEY_CHOP = """\ deploy_honeybot.sh remotes/honeybot/nixos/configuration.nix remotes/honeybot/scripts/content_loader.py remotes/honeybot/scripts/stream.py remotes/honeybot/scripts/db.py remotes/honeybot/scripts/report.py remotes/honeybot/scripts/logs.py """ ``` And after such exposition including this article and the overflow in the context window of Gemini from the prior articles and the complete codebase inclusion we just affected, I hardly even need to articulate anymore of this request. Gemini has all it needs to continue providing a detailed implementation plan for me to get the color-coded useragents like I want on `logs.py` except maybe a please and thank you! ## Refining the Signal: Filtering Human Noise and Simplifying the Palette **Gemini 3 Pro**: This is the perfect "Chisel Strike" project. It’s high-impact visual feedback with low-risk code changes. We are simply going to teach `logs.py` how to dress up the data it's already fetching. ## Phase One: "Technicolor Intelligence" via Color-Coded User Agents Here is the implementation plan to bring "Technicolor Intelligence" to your dashboards. ### The Plan 1. **Define the Palette:** Create a mapping of bot substrings to Rich styles (colors). 2. **Create the Stylist:** Add a helper function `stylize_agent(agent_string)` that converts raw strings into `rich.Text` objects with highlighting applied. 3. **Apply the Makeup:** Update `refresh_tables` to use this new function instead of printing raw strings. ### The Code You only need to edit **`remotes/honeybot/scripts/logs.py`**. #### 1. Add the Bot Configuration Add this near the top of `logs.py`, right after the `LOG_PATTERN` definition. This is your "Registry of Known Entities." ```python # ... existing imports ... # --- Configuration --- ANSI_ESCAPE = re.compile(r'(?:\x1B[@-_]|[\x80-\x9F])[0-?]*[ -/]*[@-~]') LOG_PATTERN = re.compile(r'(?P[\d\.]+) - - \[(?P