Using CloudFlare for my robots.txt - need help with dissallows

Need help
1

Within cloudflare I have enabled managed robots.txt. However, when I view mywebsite.com/robots.txt I still see the standard bubble robots text.

User-agent: *
Disallow: /version-test/

Sitemap: https://mywebsite/sitemap.xml

Question 1. How do I edit the CF robots.txt

Question 2. Any additional dissallows I should add? This is an active website/app with pages, profiles, sign ups, posts, etc.

2

Your Bubble app settings has a robot.txt section already. Use that. Bubble already reverse proxies through Cloudflare.

If you need to enable your own reverse proxy, park your app in a subdomain and then use domain rules to direct traffic to the subdomain. I assume your domain is managed by CF.

3

Hey @brad11 , here’s what’s going on.

Why the Cloudflare rules aren’t showing up

Cloudflare’s managed robots.txt is supposed to prepend its rules before your origin’s robots.txt. Right now there’s a known bug where that merge just doesn’t happen for some domains, other people are hitting this exact same issue on the Cloudflare community forum. First thing to try: purge your Cloudflare cache for /robots.txt specifically (Caching > Configuration > Purge Cache > Custom Purge, enter the exact URL). If that doesn’t fix it, it’s a bug on Cloudflare’s side, not a config mistake you’re making.

Q1: How do I edit the CF robots.txt

You actually can’t edit Cloudflare’s managed robots.txt content directly, it only works through toggles (which AI bots to block, whether to show the content signals block). Since you need custom rules specific to your app anyway, and given the merge bug, I’d just turn off Cloudflare’s managed robots.txt and manage it directly in Bubble instead. Bubble has its own robots.txt editor under Settings > SEO / Metatags in the editor. That’s the more reliable place to control this.

Q2: Additional disallows

For an active app with profiles, signups, and posts, here’s a solid starting point:

User-agent: *
Disallow: /version-test/
Disallow: /reset_pw/
Disallow: /email_confirmed/
Disallow: /change_email/
Disallow: /verify_email/
Disallow: /api/1.1/
Disallow: /admin/

A few calls to make based on your specific app:

  • Block signup and login pages, no SEO value in having those indexed
  • Block any logged in dashboard, settings, or account pages
  • Only block profile pages if they’re private, otherwise leave them open since public profiles can drive discovery
  • Keep the sitemap line at the bottom no matter what

One thing worth keeping in mind: robots.txt is a request, not an enforcement mechanism, bots can choose to ignore it, and it won’t pull pages that are already indexed. For anything actually sensitive, pair the disallow with a noindex meta tag on the page itself.

4

This is great! Thank you for the clear explination.