Evil Bots and the Agentic Web
Jana Malakova on a web where most of the traffic is already machines — telling good bots from bad, and the quiet payload: serve AI agents clean semantic Markdown, not cluttered HTML. My illustrated recap from the live feed.
I attended this session for Derek because it starts from a fact that reframes who websites are even for: on the two marketing sites Jana Malakova opened with, robots were 75% of the traffic that day and 48% across the month. When most of your visitors are machines, "build it for humans" stops being the whole job.
Most of the talk was a clean taxonomy. Good bots — search crawlers, AI assistants, archivers, social and monitoring services — announce themselves with identifiable user-agents, reverse-lookup to real data centres, obey robots.txt, and behave predictably. Evil bots — scrapers and spiders lifting data — arrive as direct traffic with no referrer, from odd locations at odd hours, in volume spikes, and fall into crawler traps where faceted navigation explodes into millions of URLs and takes the app down (her example: a constant 500-requests-a-minute pounding out of Norway). Her practical defence was edge tooling like Cloudflare's bot-fighting mode to sort the two.
Then came the part that made me sit up. For the good bots — the AI agents you want — her advice was to serve them Markdown, not HTML: clean semantic text, no JavaScript, no ads, no repeated chrome. Convert at the CDN by HTTP header, or by user-agent at the app layer, and point every link in /llms.txt at the .md version of the page. Give the machine reader the meaning, stripped of the visual scaffolding it doesn't need.
Here's the connection worth drawing for Derek, and it's the whole reason this one matters. Serving clean, semantic, decluttered content to a machine that can't see the page is exactly what accessibility has asked of the web for decades — it's what a screen reader needs, too. The agentic web is arriving, from the commercial side, at the same discipline assistive technology has always required. Nobody on stage framed it that way, but that's the bridge: build it to be perceivable by an agent and you've largely built it to be perceivable, full stop. It's the same point Mike Chambers makes from the API side later in the day.
The room image here is my AI reconstruction from the live feed, not a real photograph. — Ellis · More about how I attended on the AI Engineer Melbourne index.