Robots.txt Generator with URL Tester
Choose a preset, add the paths you want crawlers to skip, include your sitemap, and copy or download a ready robots.txt file. Then test any URL to see whether your rules allow or block it. Everything runs in your browser.
1. Start from a preset
2. Rules for all crawlers
3. Block specific crawlers (optional)
Blocking is a request. Compliant crawlers follow it, but it is not enforced.
Your robots.txt
Test a URL
What robots.txt does
A robots.txt file is a small text file in the root of your website. Before a crawler such as Googlebot visits your pages, it asks for this file and reads which parts of the site it should skip. It is mainly used to stop crawlers from wasting time on pages that have no search value, such as admin areas, shopping carts, internal search results and duplicate filtered pages.
The syntax in plain words
- User-agent: says which crawler the rules below apply to. An asterisk means all crawlers.
- Disallow: a path the crawler should not request.
Disallow: /blocks the whole site, and an emptyDisallow:blocks nothing. - Allow: an exception inside a blocked path.
- Sitemap: the full address of your XML sitemap. This line is not tied to a user agent.
- Wildcards:
*matches any characters, and$marks the end of a URL, for exampleDisallow: /*.pdf$.
When two rules match the same URL, Google follows the most specific one, which means the longest matching path. If an Allow and a Disallow match equally, the Allow wins. The tester above uses the same logic.
Common mistakes to avoid
- Leaving "Disallow: /" on a live site. This is often copied over from a staging site and quietly blocks everything.
- Blocking CSS and JavaScript. Search engines need these files to display your pages the way visitors see them.
- Using it to hide pages. The file is public and does not keep a page out of search results. Use noindex or a login for that.
- Putting it in the wrong place. It only works at the root, for example
/robots.txt, and each subdomain needs its own file. - Forgetting that paths are case-sensitive.
/Admin/and/admin/are different.
How to publish and check it
Save the file as robots.txt and upload it to your site root. In an ASP.NET Core project, put it in the wwwroot folder. Then open yourdomain.com/robots.txt in a browser to confirm it loads, and review it in the robots.txt report in Google Search Console. Changes can take a day or so to be picked up.
Frequently asked questions
What is a robots.txt file?
It is a plain text file at the root of your website that tells crawlers which parts of the site they may or may not request. Well-behaved crawlers, including Googlebot and Bingbot, read it before crawling.
Where do I put the robots.txt file?
It must be available at the root of your domain, for example https://example.com/robots.txt. In an ASP.NET Core project, place the file in the wwwroot folder so the static files middleware serves it. On WordPress you can upload it to the site root or edit it through an SEO plugin.
Does robots.txt remove a page from Google?
No. It controls crawling, not indexing. A blocked URL can still appear in search results if other sites link to it. To keep a page out of search results, use a noindex meta tag or header, and do not block that page in robots.txt, because crawlers need to be able to see the noindex instruction.
Is robots.txt a security tool?
No. The file is public and only polite crawlers follow it. Never rely on it to hide private pages. Protect them with a login or server rules instead.
Does Google support Crawl-delay?
No, Google ignores the Crawl-delay line. Some other crawlers, such as Bing, may respect it. Use it only if you have a specific reason.
Should I block AI crawlers?
That is your decision. Blocking tells compliant AI crawlers not to use your pages, but it is voluntary on their side, and it does not remove content that was collected earlier. This tool lets you add blocks for well-known AI user agents with one click.
Is my data sent to your server?
No. The file is built in your browser and nothing you type is uploaded or stored by us.
Not sure what search engines should see on your site?
A technical SEO review checks your robots.txt, sitemap, redirects and indexing so nothing important is blocked by mistake.