This tool (technical seo.com/tools/robots-txt/) will indeed test the live robots.txt file on your site, or really any site. In this example I’m studying Amazon here, best to follow in the context of an app site and getting the most out of your content.
Bubble has this located in the Settings Section under SEO.
The tool can go both ways, little jittery, but it works and then of course paste them into the robots.txt field. Then I tested them on Google Search Console and those RegEx expressions work. I found a few websites that goes over how these work, Google is tricky, but there’s nothing better than testing live for yourself.
- On a side note, something I noticed in the last year Google seems to be really devaluing websites that silo too deep. I think it’s time to rethink the long-tail and go with the root for these categories or search features of these web apps.
** The biggest SEO growth hack today is Adwords and this Big Bert algo seems to really figure out quickly all the other related organic (long tail keywords) to rank you for. So once again Paid Drives all, with these apps being high powered like this, Content is King Model will hit eventually!
But I still don’t want Google making up urls and that goes for any other bots too for server load performance, but I will report back on this and the log files to see what Google tries next.