The Robots.txt Generator helps you draft a robots.txt file from visible form controls instead of writing each crawler section by hand. Choose whether all robots are allowed or refused by default, adjust individual crawler rules, optionally add a crawl delay, enter a sitemap URL, add comma-separated disallowed directories, and generate a copy-ready text block for review.
A robots.txt file is crawler guidance, not access control. Well-behaved search crawlers use it to decide which URLs they should request, but the file does not hide private content, require authentication, remove indexed URLs by itself, or stop non-compliant bots from visiting a path. Use server permissions, login requirements, noindex directives, and removal tools where those are the appropriate controls. Use this generator when you already know the crawling policy you want and need a consistent starting file.
Verified Tool Summary
The current tool is a browser-side Alpine component named window.bitflanRobotstxtComponent(). It stores the selected options in the page, builds a plain text string when Generate Robots.txt is selected, and places the result in an Ace editor. The visible Copy button copies the editor contents through the site's clipboard helper.
The generated output always includes crawler-specific sections for Googlebot, Googlebot Image, Googlebot Mobile, Robozilla, Slurp, Gigabot, MSNBot, Teoma, Nutch, Baiduspider, Naverbot, Yeti, Yahoo MM crawler, psbot, Yahoo Blogs, and the Internet Archive user agent shown in the interface as Alexa / Wayback. It then adds a default User-agent: * section. For each allow/refuse selector, Allowed creates an empty Disallow: line, while Refused creates Disallow: /.
The prior MSN PicSearch wiring issue has been inspected in the local implementation. The MSN PicSearch selector is stored as msn_pic, and the generated User-agent: psbot rule now reads that same property. With MSN PicSearch allowed, the expected line is an empty Disallow:, not Disallow: /. That matters because an accidental slash would tell compliant crawlers matching that user agent not to crawl the entire site.
The tool does not fetch your website, test whether a path exists, validate a sitemap URL, check whether a crawler still honors a legacy user-agent name, lint the final syntax, upload files, write to your server, or publish the result. No tool-specific server request is required to generate the text, although the page may still load normal site assets, analytics, and external libraries such as Ace.
How to Use the Robots.txt Generator
- Choose Default for All Robots: Allowed produces an empty default
Disallow:; Refused producesDisallow: /in theUser-agent: *section. - Select a Crawl Delay value, or leave it at None. Available values are 5, 10, 20, 60, and 120 seconds.
- Enter a sitemap URL in Sitemap if you want the generated file to include a
Sitemap:line. - Review each crawler selector, including Google, Google Images, MSN Search, Yahoo, Ask/Teeoma, Baidu, Naver, and MSN PicSearch, and set each one to Allowed or Refused.
- Add private or low-value crawl paths in Disallowed Directories as a comma-separated list, such as
/admin, /cart, /internal-search. - Select Generate Robots.txt to create the output in the editor.
- Read the generated text carefully, copy it, save it as
robots.txt, and publish it at the root of the correct host, such ashttps://example.com/robots.txt.
Inputs, Outputs, and Limitations
| Area | Verified behavior | Review before using |
|---|---|---|
| Default crawler rule | Creates a User-agent: * section with either an empty Disallow: or Disallow: /. |
Blocking * can prevent many compliant crawlers from crawling the site. Use it only when that is intentional. |
| Individual crawler selectors | Generates fixed user-agent sections for the crawler labels shown in the interface. | Some names are legacy or vendor-specific. Confirm whether the crawlers you care about still use those user-agent tokens. |
| Crawl delay | Adds one Crawl-delay: line after the default User-agent: * section when a non-default value is selected. |
Support varies by crawler. Google Search does not use Crawl-delay in robots.txt; use search engine tooling where available. |
| Sitemap | Appends Sitemap: followed by exactly the text entered in the sitemap field. |
Use an absolute sitemap URL and check it separately. The generator does not validate sitemap availability or XML format. |
| Disallowed directories | Splits the field on commas, trims blank entries, and appends each non-empty item as a Disallow: line. |
Enter path prefixes that start with /. The tool does not normalize paths, escape spaces, or warn about accidental broad matches. |
| Output | Writes plain text into an Ace editor and offers a copy button. | There is no download or publish action. You are responsible for saving the file as UTF-8 plain text at the site root. |
Verified Example
For a typical public marketing site, you might leave most crawler selectors allowed, add a sitemap, and keep internal utility paths out of crawl queues. With Default for All Robots set to Allowed, Crawl Delay set to None, MSN PicSearch set to Allowed, Sitemap set to https://example.com/sitemap.xml, and Disallowed Directories set to /admin, /cart, /internal-search, the relevant parts of the generated output include:
User-agent: psbot
Disallow:
User-agent: *
Disallow:
Disallow: /admin
Disallow: /cart
Disallow: /internal-search
Sitemap: https://example.com/sitemap.xmlThis example shows two important details from the implementation. First, the MSN PicSearch option controls the psbot section, and the allowed setting leaves Disallow: empty. Second, directory entries are appended after the default section as additional Disallow: lines. If you want the directory rules to apply broadly, that placement is useful. If you need a more complex file with separate rules per crawler, edit the generated text manually before publishing.
Robots.txt Guidance, Not Security
A robots file can reduce unwanted crawling of duplicate pages, internal search results, filter combinations, cart routes, staging-like public folders, or generated URLs that waste crawl budget. It can also point crawlers toward a sitemap. It should not be used as the only protection for admin panels, paid content, customer records, unpublished assets, source files, backups, or anything that must remain private.
There are three reasons for that caution. First, robots.txt is publicly readable, so listing a sensitive directory may reveal that the path exists. Second, crawlers that ignore the Robots Exclusion Protocol may still request the URL. Third, a blocked URL can sometimes remain known through links or historical discovery even when its content is not crawled. Sensitive areas need authentication, authorization, server rules, and careful deployment practice.
Backlink and Collaboration Opportunities
A well-maintained robots.txt file is also a small trust signal for technical partners. When you work with agencies, developers, marketplace vendors, affiliate programs, or data providers, include the file in your launch checklist and ask them to link to the canonical resources they expect crawlers to find. Useful opportunities include adding your XML sitemap to documentation tickets, asking partner sites to link to canonical landing pages rather than parameter-heavy URLs, and giving SEO consultants a copy of the generated draft before release.
For content partnerships, pair robots review with backlink cleanup. If a partner links to an old campaign path that you plan to disallow, request an updated link to the current page instead. If your site has downloadable resources, publish a clear index page and earn links to that page rather than to temporary files. Robots directives should support discoverability decisions; they do not replace strong internal links, clean canonical URLs, or relevant backlinks from trusted sites.
Troubleshooting
| Problem | Likely cause | What to do |
|---|---|---|
| The generated file blocks the whole site. | Default for All Robots or an individual crawler selector is set to Refused, producing Disallow: /. |
Change the selector to Allowed unless full-site blocking is intentional, then regenerate and reread every Disallow: / line. |
| Directory rules look wrong. | The directories field is split only by commas; unusual spacing or missing leading slashes can create unclear directives. | Use simple path prefixes such as /admin and /search. Avoid pasting full prose, multiple lines, or unreviewed spreadsheet cells. |
| A crawler ignores crawl delay. | Not every crawler supports the non-standard Crawl-delay directive. |
Check the crawler's own documentation and use webmaster tools or server controls if request rate is the real issue. |
| The file is not being discovered. | robots.txt is not published at the root of the exact protocol and host being crawled. |
Verify the final URL directly, for example https://example.com/robots.txt, and repeat for each important subdomain. |
| Search results still show a URL. | Robots rules control crawling, not necessarily indexing or removal of already-known URLs. | Use the appropriate noindex, canonical, redirect, or search-console removal workflow for the situation. |
Related Tools
Use the Sitemap Generator when you need a crawlable URL list to reference from the Sitemap: line. Use the Canonical URL Generator when duplicate or parameterized URLs need a preferred version rather than a crawl block. Use the Hreflang Generator for localized page sets that should be discoverable in the right language or region. Use the HTTP Status Code Checker to confirm important URLs return the expected response before blocking, redirecting, or submitting them. Use the SEO Tags Generator when you are preparing page-level metadata alongside crawler directives.
Technical References and Review Notes
The Robots Exclusion Protocol is described in RFC 9309. Google also publishes practical documentation for robots.txt behavior in Google Search. Those references are worth reading before publishing broad blocks, crawl-delay assumptions, or rules for complex URL patterns.
This Digital Domain Kit tool was last reviewed on September 16, 2026. The verified technical method is client-side string generation from Alpine state into an Ace editor, with no website crawl or server-side syntax validation in the generator itself. Report problems or mismatches through the contact page, especially if a visible option creates output that does not match the selected crawler rule.