On 8 November 2026 one of the sites we run — a directory with a filterable listing — started answering slowly. The access log showed one client working through the listing page methodically: every country, with every specialisation, in every sort order, one after another, at up to a hundred requests a second.
It was not attacking anything. It was following links. Every filter chip on that page was an ordinary link to a filtered view, each one carried rel="nofollow", and the bot went through all of them anyway.
We do not know whose bot it was, and it does not change the lesson. The mistake was ours: we had published a few thousand addresses and then asked visitors politely not to use them.
Why a small catalogue has thousands of pages
Filters multiply. Take a listing with 30 countries, 8 specialisations and 4 sort orders — the size any modest directory or shop reaches. That is 30 × 8 × 4 = 960 combinations before search terms and pagination, and every combination long enough to have a second and third page triples its share. A catalogue of two hundred items can expose several thousand distinct URLs, each of which costs a database query to render.
A person sees a handful of them. A crawler that treats each link as a page sees all of them, and the more thorough it is, the worse for you. Newer crawlers, including the ones collecting text for AI products, also differ widely in how strictly they follow robots.txt, so there are more visitors of this kind than there were two years ago and less agreement among them about the rules.
What does not stop it
The usual advice is a list of signals. Each one does something. Most of them are requests.
| Measure | Stops a polite crawler fetching the URL | Stops a crawler that ignores rules | Keeps the view out of the index |
|---|---|---|---|
| rel="nofollow" | No — it is a hint | No | No |
| robots.txt Disallow | Yes | No | Not by itself |
| noindex | No — it has to fetch the page to read it | No | Yes |
| Canonical to the clean page | No | No | Usually |
| No link at all | Yes — there is nothing to follow | Mostly — only one that presses buttons | Yes |
| Rate limit at the edge | — | Caps the damage | No |
The first four rows are either a request to the crawler or a statement about indexing. Only the last two change what a crawler is able to do, and only one of them removes the cause.
The rule we now build to
A filtered view is a state of a page, the way an open menu is. It does not get a link.
Filter, sort and clear controls are buttons. Pressing one changes the address through the router, so the filtered view can still be bookmarked and shared: a person can copy the URL, but there is nothing in the HTML for a crawler to follow.
// before: an address every crawler will follow
<a href="/catalog?country=de" rel="nofollow">Germany</a>
// after: a control, not an address
<button onClick={() => router.push("/catalog?country=de")}>Germany</button>
Around that one change sit five smaller ones, because a URL that exists will eventually be found some other way.
- robots.txt disallows every filter parameter, for the bare path and under each language prefix. The second half is the one that gets missed: a rule for
/catalog?does nothing for/de/catalog?. - The canonical of any filtered view is the clean page.
- Search results are
noindex, follow. - Filtered URLs appear nowhere we publish: not in the sitemap, structured data, hreflang or llms.txt, and not in links from articles.
- Pagination of the clean list stays a plain link. It is finite and each page holds different items. Page 2 names itself as canonical, not page 1, and a page past the last one answers 404 instead of an empty list.
User-agent: *
Disallow: /catalog?*country=
Disallow: /*/catalog?*country=
Disallow: /catalog?*sort=
Disallow: /*/catalog?*sort=
What you lose, and how to get it back
Two things. Visitors can no longer middle-click a filter into a new tab, which a few people do. And filtered URLs can no longer rank, which matters when a combination is a real search — “detox clinics in Germany” is something people type.
The answer to the second is to promote the combinations that deserve it. A view worth ranking for becomes a landing page with its own path — /catalog/country/de rather than /catalog?country=de — with its own title, heading and a paragraph that is true only of that page, and a place in the sitemap. There are dozens of those, chosen by a person, instead of thousands produced by arithmetic. A thin copy of the listing under a prettier address is the same problem again.
What this does not solve
Removing the links removes the invitation. It does nothing about a scraper that already holds your URL patterns or builds query strings itself, and it does not make an expensive page cheap. Rate limiting at the edge and caching of the clean listing are a separate layer, and worth having regardless. They put a ceiling on the damage. The markup change removes the cause: a crawler that follows links has nothing left to follow.
Check your own site in fifteen minutes
- Open the listing, view the source and search for
href="?andsort=. Every hit is an address you are publishing. - Take yesterday’s access log and count requests with a query string against requests without. If filtered views are most of the traffic and little of the revenue, you have this.
- Open
/robots.txtand check that each parameter is covered under every language prefix you serve. - Search
site:yourdomain.com inurl:sort. Whatever is listed was crawled and kept. - Request page 9999 of the listing. It should be a 404.
None of this needs a new tool. It needs somebody to decide which addresses on the site are pages, and then to stop publishing the rest.
