To block crawlers or not? The dilemma Norwegian online stores don't know they're in

How many stores actually block, and who?

Of the 99 stores that received a report, 21 turned our crawler away: 16 with HTTP 403 ("forbidden"), 2 with 429 ("too many requests") and 3 with no status code at all. Fashion and apparel is worst, with half the stores blocking. Electronics follows close behind at 44 percent.

The clearest single pattern is the Varner group. Dressmann, Cubus, Bik Bok and Carlings, four brands on four different domains, all block, with an identical failure pattern right down to individual test level. That isn't four random, local decisions. It's one group-wide setting applied across every brand.

What can it cost to shut the door?

Take Bik Bok. Google showed an AI Overview, the AI-generated answer at the top of the search results, for all five of Bik Bok's own buyer questions. Bik Bok was cited zero times. On two of the questions, the AI Overview cited Swedish retailer Nelly instead.

Think about that for a moment: the AI answer is there, customers see it, and a competitor from next door gets the spot.

One honest caveat: we can't prove blocking alone is why Bik Bok isn't cited. Visibility in AI answers depends on many things, and blocked stores can only be tested on a fifth of our technical checks, so they aren't directly comparable to stores we could read in full on the audit numbers. But one thing is certain: a closed door doesn't make the job any easier for the machines that would have recommended you.

But there are good reasons to block

Let's be fair to the stores too, because this isn't a dumb choice. Large online stores carry real costs tied to bot traffic:

1. Scraping costs. Content gets harvested at scale, and that traffic isn't free.
2. Competitors harvesting prices. Automated price monitoring by competitors is an everyday reality in e-commerce.
3. Server load. Aggressive bots can eat up capacity that real customers need.
4. Content theft. Product copy and images someone paid to create get copied.

The problem isn't that stores protect themselves. The problem is that the protection doesn't tell friend from foe, and that almost nobody has checked who it's actually shutting out.

Does blocking make you invisible? No, and that's almost more unsettling

This one surprised us. Komplett blocks crawlers harder than anyone else in the study, using a so-called tarpit, a trap that deliberately holds robots in place. Yet Komplett has the highest AI citation rate in its segment: cited in 3 of its 5 buyer questions, with an AI Overview shown on 4 of 5.

How is that possible? Because when the AI can't get in, it pulls information about you from third-party sources instead: forums, directories, the press. A large, widely covered brand doesn't disappear from AI answers just because the door is shut.

But notice what's actually happening: you give up control over which sources speak for you. The AI describes your store based on what others have written, not what you've published yourself. For Komplett, that's apparently working out fine today. For a smaller brand, or one with mixed coverage out there, it's a gamble.

The opposite extreme: XXL has built doors almost nobody else has

At the other end of the scale is XXL. Of the 75 stores we measured for agent readiness, XXL comes out on top. "Agent-Native," with an MCP server card, agent skills, an API catalog and Markdown content negotiation in place, and 11 of 15 checks passed. The category average is 2.5 out of 100. Almost nobody else has even started.

So let's be honest here too: XXL is still cited 0 of 3 times in AI Overview today. The investment hasn't paid off in visibility yet. Building infrastructure for AI agents is a bet on what's coming, not a quick win. It's worth knowing that before you start.

What should you do? Four concrete pieces of advice

Whether you end up blocking a lot or a little, do it knowingly and on purpose. Here's how we'd approach it:

1. Find out what your bot protection actually does. Don't assume, test it yourself. Try fetching your homepage the way a crawler would, with no browser identity, and see what happens. Plenty of people are in for a surprise at their own 403.
2. Make sure policy and practice actually match. A robots.txt that says yes while your WAF says no is the worst of both worlds: you signal openness without delivering it, and take on the downsides of both choices.
3. Distinguish between types of bots. Let in the AI crawlers you want, and block what you don't, deliberately. Modern bot protection can be configured far more precisely than "all or nothing."
4. Keep an eye on agent protocols. MCP and similar standards are still immature today, but they're defining how AI agents will talk to websites going forward. You don't need to be XXL, but you should know what's moving.

Buying agents are coming, and then this becomes an active choice

What makes this dilemma more important every year is that "robots" no longer just means scrapers and search engines. AI assistants that answer buyer questions are robots too. And soon come the buying agents, software that researches, compares and purchases on the customer's behalf.

The day your customer's agent shows up and knocks, today's unconscious blocking is no longer a technical footnote. It becomes an active choice to shut them out.

Closing thoughts

21 of 99 stores block robots, even though every one we could read says it wants AI on its side, on paper. We don't think that's a deliberate strategy. We think nobody has asked the question. Bot protection gets bought as security, robots.txt comes with the platform, and nobody has looked at the two together.

The good news is that this is easy to fix. You don't need to build world-class agent infrastructure tomorrow. You just need to find out what your gatekeeper is actually doing today, and decide whether that's what you want. It's an afternoon's work, and a decision that deserves to be made by someone who knows it exists.

Good luck, and go test your own 403!