Robots.txt Generator

What robots.txt controls

A robots.txt file sits at the root of a domain and tells cooperating crawlers which paths they may or may not request. It is a set of rules grouped by user-agent, each with Allow and Disallow lines, and optionally a Sitemap pointer. It is a crawling directive, not a security boundary; well-behaved bots obey it, but nothing enforces it.

When you reach for it

You write one to keep crawlers out of admin panels, search result pages, or staging areas, to point search engines at your sitemap, or to throttle aggressive bots away from expensive endpoints. It is one of the first files to review when a site indexes pages it should not.

Example

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/public/

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml

Rules are grouped per user-agent, and a crawler obeys only the most specific matching group, not the union of all groups.

Common pitfalls

The dangerous misunderstanding is treating Disallow as a way to hide a page. Disallow stops crawling, not indexing: if other sites link to a disallowed URL, search engines can still list it, just without the content. To actually keep a page out of results, allow crawling and use a noindex meta tag, or protect it behind auth. The other trap is scope: a single robots.txt only governs its own host and protocol, so http and https, and each subdomain, need their own file.

FAQ

Does Disallow remove a page from Google? No. It stops crawling. The URL can still appear in results. Use noindex to drop it.

Can I block all bots at once? Yes, with User-agent: * and Disallow: /, but malicious scrapers simply ignore the file.

Where must the file live? Exactly at the domain root, /robots.txt. A copy in a subfolder is ignored.