It is a business decision disguised as a technical one. And plenty of sites have made it accidentally, by copying someone else's robots.txt.
What you gain by allowing crawling
- The chance to be cited and recommended in generative answers.
- Visibility in discovery queries you used to fight for in search.
- Brand traffic: the user sees your name, then looks you up directly.
What you gain by blocking
- Control over premium content, courses or proprietary databases.
- You avoid your content being summarised without a visit when your model depends on ad impressions.
- Negotiating leverage if you later license content.
The decision, by business model
- SaaS, professional services, B2B, ecommerce: allowing is almost always right. Your content is a catalogue, not the product.
- Ad-funded media: mixed. Many allow search-grounded assistants (which link out) and block training crawlers.
- Courses, paid reports, databases: always block the private area and consider blocking long previews.
How it is configured in practice
robots.txt distinguishes by user agent. It is worth separating two uses: training bots and live search bots that do link back. You can allow the latter and block the former. Add explicit rules for your private and app routes, and leave open the public content you want cited.
Then verify the file loads correctly and that you have not accidentally blocked traditional search engines: the most expensive and most common mistake.
What does not work
- Blocking everything "just in case" and expecting to still appear in answers: those are incompatible.
- Blocking in robots.txt while publishing an llms.txt: contradictory.
- Changing the policy monthly: you lose any chance of measuring the effect.
How to check the impact
Whatever you decide, measure. Record your mention rate on a fixed question set before the change and four and eight weeks after. It is the only way to know whether you protected something valuable or switched yourself off.