What is robots.txt?

Short answer

Robots.txt is a small file on your website. It tells search engines and AI bots which parts they may not visit.

Every website can have a robots.txt at the address /robots.txt. It says which bots may not visit which folders. It often also has the link to your sitemap.

Note: robots.txt does not keep a page out of Google. Google can still list the address but without content. Do you really not want to be found? Then use a noindex or a password.

Also check that your robots.txt does not block AI bots. Otherwise ChatGPT can never mention you.

Real-life example

A test version of a website was blocked with robots.txt. Still the addresses showed up in Google. A password on the test version solved it.

How we help with this

At Marketing Friends we take care of this for businesses across Europe. Read more about Website maintenance and hosting or start with the free scan.

Also good to know

Curious how your website scores? You get your score and your three biggest competitors within a minute.

Start the free scan