Robots.txt Generator
Compile formatted crawl instructions and sitemap directives for search spiders. Runs in your browser with no account required.
Drag & drop your image here
Supports JPG and PNG • Up to 50MB
Supports JPG, PNG, and WebP • Up to 50MB and 40 megapixels
Upload PDF Document
Select multiple PDF files to combine them
Select a PDF file to edit pages
Search engines truncate snippets to fit the result layout; these counts are preview guidance, not enforced Google limits.
{{ activeTool.name.split(' ')[0] }}
Document ID: {{ invoiceNumber }}| Description | Qty | Price | Subtotal |
|---|---|---|---|
| {{ item.desc }} | {{ item.qty }} | ${{ item.price }} | ${{ item.price * item.qty }} |
| Keyword | Count | Density |
|---|---|---|
| {{ item.word }} | {{ item.count }} | {{ item.percent }}% |
| No significant keywords extracted. | ||
How to use Robots.txt Generator
Compile formatted crawl instructions and sitemap directives for search spiders.
-
01
Choose crawl directives
Define the user agent and the paths that should be allowed or disallowed.
-
02
Add the sitemap location
Use the absolute HTTPS URL of the sitemap you want crawlers to discover.
-
03
Review before publishing
Copy the result to /robots.txt and test important URLs with the tools supplied by your search platform.
- Output
- Plain-text robots directives
- Location
- Site-root /robots.txt
- Purpose
- Crawler access guidance
Example: block a private utility path
- Input
- User-agent: * and Disallow: /private/
- Result
- Compliant crawlers are asked not to crawl URLs beginning with /private/.
Limits and checks
- robots.txt controls crawling; it is not access control and cannot protect confidential content.
- A disallowed URL can still be discovered or indexed from external links in some circumstances.
- Directive support varies by crawler, so verify rules with each important search platform.
Robots.txt Generator questions
Where should robots.txt be uploaded?
Place it at the root of the exact host, such as https://example.com/robots.txt.
Does Disallow remove a page from search?
Not reliably. Use appropriate authentication or noindex handling for content that must not appear in search.
Can robots.txt contain more than one user-agent group?
Yes. Separate groups can provide different rules for different crawlers.