Robots.txt Generator: Free Online Creator

This robots.txt generator builds a ready-to-use robots.txt file without making you write the syntax by hand. Pick a preset or set rules for Google, Bing and AI crawlers, add disallow paths, sitemaps and a crawl delay, then copy the result or save it as robots.txt. It is free and needs no sign-in.

What Is a Robots.txt File?

A robots.txt file is a plain text file at the root of your domain that tells crawlers which URLs they may request. It follows the Robots Exclusion Protocol, and well-behaved bots read it before they crawl anything else on your site.

The file is built from groups. Each group starts with a User-agent line that names a crawler (or * for all of them), followed by Disallow and Allow lines that list URL paths. An empty Disallow: means nothing is blocked. Disallow: / means the whole site is blocked. You can also add a Sitemap line with the full URL of your XML sitemap.

Robots.txt controls crawling, not indexing. According to Google's Search Central documentation, a blocked URL can still appear in results (without a description) if other pages link to it. To keep a page out of search results, use a noindex directive on a page that crawlers are allowed to fetch.

How to Use the Robots.txt Generator

You can create a robots.txt file in a few minutes by choosing a starting point and then adjusting each rule. Follow these steps:

1. Start with a button under Quick Presets: Allow All Bots, Block All Bots, Block AI Bots or WordPress Default. Each one fills the form so you only have to adjust the details.
2. Choose the default rule for all robots. The first option allows everything, and the second refuses everything.
3. Optionally pick a crawl delay: 5, 10, 20, 60 or 120 seconds. Leave it on the default for no delay.
4. Click Add Sitemap and paste the full address, such as https://www.example.com/sitemap.xml. You can add several sitemaps.
5. Open the crawler list and set individual bots to default, allowed or refused. The list covers search engines (Google, Google Image, Bing, Yahoo, Baidu, Yandex, Naver, DuckDuckGo, Google Ads) and AI bots (GPTBot, ChatGPT User, Google AI (Gemini), ClaudeBot, Common Crawl, PerplexityBot, Bytespider, Meta AI, Apple AI, Cohere AI).
6. Under the disallow section, click Add Path for each folder or URL pattern to block, for example /cgi-bin/. Use the allowed paths section to make exceptions.
7. Generate the file, check the Generated robots.txt box, then click Copy or Save as robots.txt. Reset clears the form.

Worked Example: A WordPress Blog That Blocks One AI Crawler

Say you run a WordPress blog and want search engines to crawl everything except the admin area, but you do not want GPTBot to use your content. Here is how you would set it up.

• Default rule: allow all robots.
• Disallow path: /wp-admin/
• Allowed path: /wp-admin/admin-ajax.php (some themes and plugins load content through this file)
• Sitemap: https://www.example.com/sitemap.xml
• GPTBot (OpenAI): refused

The output contains a User-agent: GPTBot group with Disallow: /, a User-agent: * group with an empty Disallow: and your allow rule, your /wp-admin/ disallow line, and the sitemap line. Google, Bing and everyone else can still crawl your posts. Only GPTBot is turned away. Upload the file so it loads at https://www.example.com/robots.txt, then verify it with a robots.txt tester.

How Do You Read and Act on the Generated File?

The generated text is the exact file crawlers will read, so each line maps to one setting you chose. The counters beside the output show the number of lines and directives, which helps you spot a file that is much longer or shorter than you expected.

What a good result looks like

A healthy file for most sites is short: an open default rule, a handful of disallowed folders (admin, cart, internal search, staging paths) and one or more sitemap lines. If you see Disallow: / under User-agent: * on a site you want in search results, stop and fix it before uploading.

Clean-param and crawl delay

The Advanced Options section holds a Clean-param field, written as param1&param2 /path/. For example, utm_source&utm_medium&utm_campaign /page/ tells supporting crawlers, such as Yandex, to ignore tracking parameters on that path. Google does not support Clean-param or Crawl-delay, so do not rely on them to manage Googlebot. Bing and some other crawlers do read Crawl-delay.

Should You Block AI Crawlers in Robots.txt?

Blocking AI crawlers is a business decision, and the generator lets you make it one bot at a time. The Block AI Bots preset refuses the AI group in one click, while the allowed paths section lets you open specific content, such as a public guides folder, while the rest stays closed.

Know what each choice means. Refusing Google AI (Gemini) uses the Google-Extended token, which Google describes as a control for AI use of your content, separate from Googlebot crawling for Search. Refusing Applebot-Extended works in a similar way for Apple. If you want to be quoted in AI answers, keep the crawlers behind those products allowed. If you want to opt out of training, refuse them. Either way, review the list again every few months because new crawlers appear.

Who Is This Robots.txt Maker For?

Anyone who manages a website and needs a valid file without memorizing directive rules will find it useful.

• Site owners and bloggers who need a starter file for WordPress, Blogger or a custom site.
• SEO specialists who want to keep crawlers out of filtered URLs, internal search results and checkout paths so crawl time goes to pages that matter.
• Developers who need to block a staging site or a folder like /cgi-bin/ quickly.
• Publishers who want different rules for search engines and AI crawlers.

Some hosted platforms create robots.txt for you or limit what you can edit, so check your platform's settings before replacing the file. If you manage related technical files, the .htaccess redirect generator covers the server side of crawl management, and the On-Page SEO and Meta Tags tools cover page-level controls.

Best Practices, Common Mistakes and Troubleshooting

Most robots.txt problems come from a few repeated mistakes, and each has a quick fix.

• Blocking the whole site by accident. Setting the default rule to refuse writes Disallow: /. Switch it back to allow, or generate again after pressing Reset.
• Blocking CSS and JavaScript folders. Google needs those files to render pages. Remove the rule or add an allowed path for the assets.
• Using robots.txt to hide private pages. The file is public and anyone can read it, so listing a secret folder advertises it. Protect private content with a login.
• Forgetting the trailing slash. /blog/ blocks the folder, while /blog matches any path that begins with those characters.
• Wrong upload location. The file must be named robots.txt and sit at the root of the host, such as example.com/robots.txt. A copy in a subfolder is ignored.
• Relative sitemap URLs. Enter the full address starting with https.

After uploading, test the important URLs with the robots.txt rule tester to confirm that Googlebot, Bingbot or an AI crawler is allowed or blocked as you intended. If a blocked page still shows in search, check its canonical setup with the canonical tag checker and use noindex rather than a disallow rule.

Limitations to Keep in Mind

Robots.txt is a request, not a lock. Reputable crawlers follow it, but malicious bots can ignore it, and different crawlers interpret edge cases slightly differently. The generator lets you add up to 20 disallow paths, 20 allowed paths and 10 sitemaps, which is enough for most sites. It builds the file from your choices and does not crawl your site or check whether the paths you enter exist, so confirm that every path matches a real URL structure.

Ready to build yours? Choose a preset above, adjust the bots and paths, then copy or save the file and upload it to your site's root. For more utilities in the same area, browse the Web Developer Tools hub.