Robots.txt and Sitemaps Explained for Small Business Owners

First, what is a crawler?

A crawler, also called a bot or spider, is a program that visits web pages and reads them. Google has one. So do Bing and many AI companies. They follow links from page to page, like a very fast visitor. What they find gets stored, and later it can show up in search results or in AI answers.

What robots.txt does

Robots.txt is a plain text file in the top folder of your website. When a good crawler arrives, it checks this file first. The file can say things like:

• Everyone may visit the whole site.
• Please skip the admin area.
• This specific crawler may not visit at all.

You can see yours by typing your domain followed by /robots.txt into your browser, for example example.com/robots.txt.

A very simple file has a line naming the crawler (an asterisk means "all of them"), a line that allows or blocks a folder, and a line pointing to your sitemap.

What robots.txt does not do

• It does not secure anything. Bad bots can ignore it, and anyone can read it. Never use it to hide private information.
• It does not guarantee a page stays out of Google. A blocked page can still show up if other sites link to it. To keep a page out, use a "noindex" setting or put it behind a password.
• It can break your visibility if used wrongly. One bad line can block your whole site.

The scary one-line mistake

A line that blocks everything is sometimes added while a site is being built, so that Google does not see an unfinished site. If it stays after launch, your site can disappear from search. Check this after any redesign. See redesign without losing Google rankings.

What a sitemap does

A sitemap is a file, usually at yourdomain.com/sitemap.xml, that lists your pages. It helps search tools find new pages quickly and understand which ones matter. It can also include the date each page last changed.

Good sitemaps:

• List only pages you want people to find.
• Leave out duplicates, test pages and thank-you pages.
• Use full, correct addresses that actually work.
• Update when you add or remove pages.

Most website platforms, including WordPress, Shopify and Wix, create a sitemap automatically. Check that it exists and looks right.

Telling Google about your sitemap

You can add your sitemap to Google Search Console, a free tool from Google that shows how your site appears in search. It will tell you if Google has trouble reading the file or any listed pages. You can also add a line to robots.txt that points to your sitemap.

AI crawlers and robots.txt

Many AI companies run their own crawlers, and each has a name. Robots.txt can allow or block them one by one. Some businesses block them, some allow them. If you want AI assistants to be able to read and mention your business, make sure you have not blocked them by accident. Your web developer can check the file for you. It also pairs well with an llms.txt file, a short summary written for AI tools.

A simple 10-minute check

1. Open yourdomain.com/robots.txt. Does it load? Does it contain a line that blocks your whole site?
2. Open yourdomain.com/sitemap.xml. Does it list your main pages?
3. Make sure no important page is missing, and no private or test page is listed.
4. Add your sitemap to Google Search Console if you have not already.
5. Recheck after any big change to your site.

When to ask for help

Ask a specialist if you see any of these: your site is missing from Google, the files return errors, you run several sites or languages, or you have an online shop with thousands of pages. Mistakes here can be costly, but they are also usually quick to fix.

Where this fits

These files are part of what is called technical SEO, the behind-the-scenes setup. For the whole picture, see what is SEO in plain English.

How Social Buzz can help

We check and correct robots.txt and sitemaps, set up Search Console, and make sure the right crawlers can read your site. Details are on our SEO and AI search page, or email info@social-buzz.app.

Questions people ask