SEO question (junk pages)

I have deployed my app live and users can access only about 4 pages through the app (this is how it’s supposed to be). I have dozens of other pages of the app that are in development and i am now wondering if these junk pages that i’m working on will in anyway read by Goolgle crawlers and displayed to searchers or even worse, accessible. ??

If so, should i delete all these pages then deploy live then undo the deletions and keep them?

Or do i just modify the robots.txt file to exclude them?

Unless you put measures in place to stop them, then yes, both of those things are possible….

So add any pages you don’t want Google to index to the robots.txt with ‘disallow’ or add a noindex tag on the pages.

To stop people from being able to access the pages, make sure you have conditions on the pages to navigate users away if they don’t meet certain conditions (e.g. if they don’t have the correct user type).

1 Like

I have similar pages under development, but I took the approach of not including them in robots.txt because I didn’t want anyone to be able to find them (anyone can read the robots.txt file). My approach was:

  • Make sure pages aren’t listed in the sitemap
  • Make sure there are no links from your accessible pages that lead to these pages
  • Add some random numbers and characters to the page URL so that it would be difficult to guess
  • Put all the content in a group that is hidden to everyone except for the admin user (you) and also put on a redirect, in the off chance that Google does come across it

This should remove ways for anyone (including Google) to find your pages.

I had to figure this out myself, so if I’m missing anything I’d welcome correction.

It would be useful if Bubble had a feature like Webflow, where you can lock individual pages behind passwords while you are working on them. Currently, Bubble only offers that at the site level.

Ok and what types of pages do we want on the site map? obviously home page and blogs, but what about pages that are only accessible when a users is sign-up normally an user not signed in cannot get to this page and if they do and they redirected to home page. but does this mean i should or should’t have this page on my site map?

Also can someone explain the difference between using robot.txt disallow and site map excluding? they sound like they do the same thing, no?

Sitemap is the directions, robots.txt is the stop signs.

Also, for the pages i “expose” on the site map, are they going to be displayed on search engines with the title i see here (on the list on the SEO / metatags page) or with the “Page title” text or with the “title tag” text?

Screen Shot 2022-04-14 at 6.31.57 pm

Soooo should i do both? So excluding something from a site map will tell google not to show it in search engines results (e.g., probably for my reset password page, terms and condition page) and the robot.txt will do what? prevent a page from being accessed in any way even through using the app?

Robots will be the stop sign that says ‘oh hey crawler you may have found this page, still don’t index it’. So even if a page is not in your site map, if the crawler finds it somehow, robots will stop it from being indexed.

1 Like

Thanks, that clears it up.

Now i just need an answer to this question.

Ok gang, here’s where i’m up to with working through all this site map / robot.txt stuff:

I’ve made the site map exclusions the same as the robot.txt disallows (any objections? this look right?):

However, it appears that Google is still showing these pages that i excluded from site map and disallowed on robot.txt:
Screen Shot 2022-04-14 at 11.02.03 pm

Just another note, i think i might have to upload a custom site map instead of the standard bubble tick box one. I saw on youtube someone say that if you have dynamic pages (e.g. blog type pages) the standard bubble tick box site map won’t cut it. If that’s true it’ll be extra work for me that’s for sure.

That’s incorrect - Bubble autogenerates dynamic pages on the sitemap for you (up to 50,000 URLs per page).

However, it appears that Google is still showing these pages that i excluded from site map and disallowed on robot.txt:

Don’t expect changes you make in your app to take effect in Google straight away. It can take days/weeks (or in some cases months) for Google to update their listings… you can re-submit your sitemap, request recrawls, or (if necessary) request removals, directly from your Google search console if you need to which can help speed things up.

2 Likes

Sweet. For reference, here’s the 5 minute site map video i watched https://www.youtube.com/watch?v=8tUxBTuF7CM

Ok, but hypothetically, if i did nothing about it, it would fix itself right? just take a while uh?

Ok, but hypothetically, if i did nothing about it, it would fix itself right? just take a while uh?

Eventually it will, yes…

1 Like