Free tools · Building and launching
Build a robots.txt you understand
A robots.txt file tells crawlers where not to go. It is four lines of text that can quietly remove your entire site from Google if you get one of them wrong, so every line here is explained rather than just generated.
Only used to write the sitemap line.
Save as robots.txt
User-agent: *
Disallow: /admin
Disallow: /wp-admin
Disallow: /loginWhere it goes
At the root of your domain, reachable at /robots.txt. Crawlers look there and nowhere else, so a copy in a subfolder does nothing.
Blocking a page stops Google reading it, not listing it. To keep a page out of search you need a noindex tag, which means the page has to stay crawlable.
Robots.txt generator · FAQ
Common questions about robots.txt generators
What is robots.txt for?
It tells crawlers which parts of your site not to request. It is a traffic instruction, not a lock: it keeps well-behaved crawlers out of your admin pages and away from pointless URLs, and it is ignored entirely by anything malicious.
Does robots.txt stop a page appearing in Google?
No, and this catches people out constantly. Blocking a page stops Google reading it, not listing it, so a blocked page can still appear with no description. To actually keep a page out of search you need a noindex tag, which means the page must stay crawlable.
Where does the robots.txt file go?
At the root of your domain, reachable at yoursite.co.nz/robots.txt and nowhere else. Crawlers only look in that one place, so a robots.txt in a subfolder does nothing at all.
Should I include my sitemap in robots.txt?
Yes. It costs one line and it is one of the easiest ways for a crawler to find every page you want indexed, especially on a new site with few links pointing at it.
Rather have someone do it for you?
We build websites for New Zealand businesses at a fixed price from $2,500, live in four weeks.
Get a fixed-price quote