Please stop crawling my LIVE SITE (SOLVED)

Dear Community

I have spent a little bit of time researching this on the forum and found patches of information but I felt I would like to open a thread.

I want to completely, to the best I possibly can block any search engines from finding my application.

Under the SEO tools in the box it currently says this

User-agent: *
Disallow: /version-test/

I assume that there is something else I can type in here to not allow Bots to crawl my live site- can anyone suggest how i can achieve this?

Many thanks in advance

1 Like

That’s usually what one writes in the robots.txt file to prevent bots from crawling your site.


To exclude all robots from the entire server

User-agent: * 
Disallow: /

Add another disallow.

Disallow: /

Another option is to use meta tags.

So it really is that simple?
Far too easy guys thanks a lot!

1 Like

Yep, be sure to mark as solved. Others may have this question down the road. So good to flag it accordingly.

This topic was automatically closed after 70 days. New replies are no longer allowed.