Two files, two different jobs
A sitemap is a list of the pages you want Google to know about. A robots.txt file is a note to crawlers about which URLs they may visit. People often mix them up, and the mix-up can hide a site from search.
What robots.txt is, and is not
Google describes robots.txt as a file that tells crawlers which URLs they can access on your site, mainly to avoid overloading it with requests. It is not a way to keep a page out of Google. A blocked page can still show up in results, without a description, and a page you disallow can still be indexed if other sites link to it. To keep a page out of search, use noindex or password protection.
It is also not a security tool: crawlers that ignore it are not bound by it, so never rely on it to protect private pages. And do not block the script or style files your pages need, because that can stop Google understanding the page properly.
What a sitemap is, and is not
Google supports sitemaps as XML, as RSS or Atom feeds, or as a plain text file with one full URL per line. One sitemap can hold up to 50,000 URLs or 50 MB uncompressed, and it must be UTF-8. Put it at the root of your site. Google ignores the priority and changefreq values, and uses lastmod only if your dates are consistently accurate.
Submitting a sitemap is a hint. Google does not promise to download it or crawl every URL in it, so a sitemap helps discovery but does not decide indexing.
How to submit
Use the Sitemaps report in Search Console, or add a Sitemap line with the full address to your robots.txt. If your site is built with a platform or plugin, it probably makes the sitemap for you; check that the address opens in a browser.
A safe default for a small site (our advice)
- One sitemap with your real pages, kept up to date.
- A short robots.txt that allows everything and points to the sitemap.
- No Disallow lines unless you know why they are there.
- If a developer asks to block your whole site while building it, ask them to remove the block on launch day, and check it yourself.
Open yourdomain/sitemap.xml and yourdomain/robots.txt in a browser and read them.
How this post was made: an AI drafted it from the official pages listed under Sources, every figure was checked against those pages, and the Reveaxa team approved it before it was published. Our own advice is marked as such. If a source changes, we update the post and its date.
Reveaxa guide reference: RVX-B-robots-txt-and-sitemap-for-small-sites