Robots.txt for Beginners: What It Does and What It Does Not Do
Robots.txt is a small file with a big job: guiding crawlers. Learn the syntax, common mistakes, and why it is not a security tool.
What is the Robots.txt Generator?
Robots.txt tells search engine crawlers which parts of your site to skip. This generator builds a standard file with your disallow paths and sitemap location.
How to use it
- List the paths to block, one per line.
- Add your sitemap URL.
- Click Generate and save the file as robots.txt at your site root.
Worked example
Disallowing /admin/ and /private/ with a sitemap line produces a ready-to-upload robots.txt.
Should I block my admin pages in robots.txt?
Blocking /admin/ is common, but remember robots.txt is public — never list genuinely secret paths there. Protect sensitive areas with passwords instead.
Frequently asked questions
Does robots.txt hide pages from search?
It asks crawlers not to crawl them, but it does not remove already-indexed pages or block determined visitors.
Where does robots.txt live?
At your site root, e.g. example.com/robots.txt.