Memory Issues: What Data is Loaded when a Page Element has a Type?

So, here’s something that seems to be affecting performance on certain of my pages. I have a site/app with "Listing"s for rentals. The Listing data type object is, of course, quite complex. It contains text info (descriptive stuff), links to other objects (like calendars, which in turn have event objects within them), lists of photos, etc., etc.

I would have thought, given the Bubble metaphor of pages being able to have a “type” (and this being the main and easiest way to create dynamic pages that present certain data from objects of that type) that Bubble would be “smart” (or at least “smart”-ish) about what data gets received by the page.

What I’m finding is that, if I build a page is of type “Listing” that only makes use of a few of the Listing’s attributes (in my specific case, I’m displaying a calendar associated with the Listing in question, which really only accesses the Listing’s associated Calendars and the events within those Calendars), there’s a whole ton of data that is still received by the page.

This became obvious to me when comparing performance on this particular page when a user is logged in, versus when they are not logged in. In the Logged In case, the page receives much more data as Privacy Rules “unlock” access to more data. When one is Logged Out and views the page, a much smaller amount of data is loaded.

Similarly, some Listings are larger and more complete than others. A Listing for which there are no photos or descriptive data (but which might have a calendar associated with it) will load less “phantom”/“unreferenced” data than a more complete Listing.

Because the “Listing” type is so expansive, what I’m finding is that my “simple” calendar display page can, in certain cases, be loading HUNDREDS of Megabytes of data – none of which I’m actually “using” or referencing in the page.

This just strikes me as really strange, as the Bubble development paradigm encourages you to build pages based on object types. So that (for example), I have many different kinds of pages that are of type “Listing”, but that perform different functions. For example:

  • A “view_listing” page that shows photos, descriptive text, calendar availability, etc. to prospective renters.

  • Administrative pages where the owner of a listing enters information, uploads photos, links calendars, etc.

  • Pages like a “booking-widget” page (what I’m talking about above) that show calendar availability, but none of the other stuff.

And then the URLs for these pages, of course, all end with the same unique ID (the unique ID of the Listing in question).

Here’s my question/issue: When one “sends data to a page” (that is the page object has a data type like Listing) I would have assumed that “sending data” was kind of metaphorical. That what one is actually doing, is sending a POINTER to potential data used in the page. (Like, Listing is just a handle that one can conveniently grab to access data: e.g., “Current Page’s Listing’s Short Title”.)

But this seems to not be the case. Simply referring to a large data object in this way, really DOES cause a bunch of potentially unneeded stuff to load. This doesn’t make much sense to me.

How do longtime Bubbler’s deal with this?

I guess the workaround is NOT to use objects of that data type. Instead of “send data Listing to page X” one would “send data Listing’s uniqueID to page X” and only grab what one needs? This seems to make things much harder than they should be…

2 Likes

A somewhat related post by @josh, lists vs searches …

I’ve had similar experiences to yours, where items linked to are loaded as well, but @josh can give an explanation of what is loading when.

The more I look at this, the more I feel like perhaps I have stumbled upon an odd Bubble bug.

Look at this snip of the page in question:

This not-incredibly-complex page (of type Listing) is ballooning at times to more than 300 MB used, if the user is logged in and the user is this Listing’s creator. When that happens, this calendar widget becomes extremely sluggish and (for example) navigating to the next/previous month by clicking the arrows takes a long time.

In comparison, a more data-intensive page (also of type Listing), which is displaying photos, a bunch of textual information, etc, etc. stays around 100 Mb (still big but not half a GIG as in the above screenshot).

Crazy-making, no?

This MIGHT be moment-timezone’s timezone info, but it’s weird that this giant memory inflation doesn’t happen for EVERY version of this page… just some Listing’s… can’t explain it.

Hi @mishav, yeah I’ve seen that post and found it pretty helpful. The odd thing about what I’m seeing with this particular page is shown in the reply I just posted. For some weird reason, this page will balloon to half a gig of memory consumption for certain Listings (mostly my own “test” listing which represents the oldest and most complete listing profile in my system).

OK, well, I think I found the issue and it is an ODD one, but perhaps not a “bug” per se:

After much debugging, it seems that there’s something unique about this “problem” Listing (which is my main testing sort of Listing). It has attached to it a bunch of data that I can no longer access.

It BELIEVE the “phantom” data is a bunch of API Connector-retrieved data that was stored on this particular Listing record. This is probably the result of earlier development phases when I was building and testing my API for parsing external iCal files.

The only “evidence” for this was a bunch of “api_c2_…_blah_de_blah” data that shows up in View Source when I’m logged in as myself (or another account that has Admin rights).

The PROBLEM with that is – like any list-type retrieved API data – those “things” are stuck over in what I call the “API Ghetto”. They have no top-level data types and so cannot be searched and cannot be deleted.

I must have – at one time – had a field on Listing that held those values. But field no longer exists. However, it seems these second-class data objects are still around and somehow attached to the listing… Even though there is no “handle” to access that list of archaic data objects.

So the only solution is to either delete that particular Listing or copy it. I’ve created a copy of it (using Make a new thing… set all of the zillion fields to the original version of the Listing). The copy DOES NOT seem to have the weird leftover data types attached. And this copy performs rather better than it’s original.

HOWEVER, I still find it strange how much data seems to get loaded via default simply when referencing an object. In my app, everything is pretty much LISTING-CENTRIC. So, referring to a Listing is just as common as referring to “Current User”, for example.

On the page I’ve been investigating here, I only require and refer to a few aspects of the Listing. Things like:

  • Listing’s “Calendar List” (and the Calendars within that list, and the events inside of those calendars, and the calendars’ last modified dates)
  • Several booleans that describe behavior/licensing of the listing
  • The Timezone the Listing resides in (this is not a sub-field of Location, I pull it up into its own field elsewhere in the app.)

But I DO NOT reference things like:

  • Any descriptive text about the listing (but anyone logged in gets this info pushed to the page)
  • The listing’s Location (but this shows up too)
  • The listing’s list of Photos (but pointers to the unique IDs of these show up in view source)
  • Many many other fields

And yet Bubble decides to download these anyway. This seems really strange to me because what it means, is that a page that is designed to function just fine whether the User is Logged In or Not Logged In, will always performs BETTER for the NOT LOGGED IN User, because less data gets loaded.

I GUESS the workaround is PRIVACY RULES at the page level? But what a hassle… and can those even be defined that way? (I think they probably can be, but the Expression field on the Privacy tab seems pretty buggy – it often doesn’t allow the building of complex expressions. I find one often has to build the expression in some other element and then copy and paste it to the “When” field. See below:

(^^^ you can’t build this expression in this field – but you can build it somewhere else and paste it here… bug or feature??)

The data loading thing just seems really strange. If Bubble can’t automagically detect that the page in question doesn’t need certain fields of the page’s data type, it would be helpful to have an interface for controlling that at the Page Element level – setting that manually.

Like, when you select the Page’s “Type of content”, you should be able to force a field to pre-load or not, perhaps? Like here:

Anybody ever run into all of this and find it all a bit odd? This kind of behavior seems to explain a lot of the perception about Bubble apps being “slow”. They often are… because of this behavior… and it seems overly hard to control. Just some thoughts!

3 Likes

Quick follow up Q:

Perhaps an interface for this (page level data loading behavior) exists somewhere in Bubble, but I don’t know where it might be…

:thinking:

Interesting analysis and a good tip about the When field in privacy rules @keith. I’m going to keep tracking this post because I think it will shortly be a concern for me too. I have jobs rather than listings but they will operate in pretty much the same way and I will only ever be working with a few fields, not all the possible fields.

As for the data remnants left over from old deleted fields … that really is a bummer. I don’t like the way bubble keeps deleted fields and tables. I get the reason and it is fine but there should also be a “I know what I’m doing and I absolutely want this gone’ option”. My early experimentation led to all sorts of problems because deleted fields don’t show on issues and I was encountering weird behaviour until I would happen to see the word ‘deleted’.

2 Likes

I think that being able to specify which fields are being used for a thing, on retrieval in general, would be amazing for performance.

Doing this automatically would be problematic for the way data is passed to elements / plugins, i.e. it would be difficult to determine in advance which fields are used.

Thanks for sharing your findings, @keith.

1 Like

Well, I’m cool with expressions retaining references to now-deleted data types/fields as otherwise you’d have a hard time locating them, and those are often logic-breaking situations.

The data that is seemingly “stuck” to the problem listing is a very special case, but isn’t uncommon if you work with external APIS:

They are lists of custom objects: multi-field/key things.

As I’ve pointed out in other posts, the individual data items in such lists come across to Bubble in such a way that they are primitives. But Bubble does not provide you with a matching datatype at the database level and so it is possible to “lose” any way to access such items which is what may have happened here.

I’ve not deleted the “problem” listing so that I can experiment with this a bit more.

Today I’m definitely going to futz around with whether it’s possible to use page-specific data privacy rules to better control the loading of irrelevant data on this and similar pages.

A quick update on this (@mishav) : I had mused that it might be possible to control data loading by using Privacy rules that incorporate something like “and Current Page doesn’t contain ‘some string’” or similar.

In general this approach does not work as Privacy rules are implicitly "OR"ed and unlike Conditions on a Conditional tab, do not have an order of precedence. (Per the reference, “When multiple rules apply, the user has access to an object if any one rule grants access to it.”)

So while you could do this, the conditions become inordinately complex and I don’t think this is a smart approach at the current time (it is very hard to maintain).

(FEATURE IDEA: It would be very interesting to have a page-level Privacy rule option that trumps other defined Privacy Roles. THAT would make this completely doable.)

The other option (and kind of the only one) that occurs to me for controlling data loading is to in fact construct a “lighter” version of the complex data type in question and use that. (Either literally a different data type OR basically a clone of the data object that only has essential fields populated.)

The only issue one runs into there is keeping such things synchronized. Not sure that’s quite worth the effort in my own use case… basically, im starting to see the need for an easy interface to DELAY or prevent loading of certain data fields when we know they are not needed.

Did the privacy rules limit the fields sent to the page?

Privacy rules feels like the wrong place to put this logic, but if they do restrict the data, there could be some logic like Current user’s list of permissions contains “calendar” to allow access to Listing’s calendar. The list of permissions can be set by the page.

Of course this falls down pretty quickly when considering multiple pages or the same page multiply open at the same time by the same user, and might also have API workflows running with the user authentication that don’t have a page context.

Would “Listing Lite” store a reference to a calendar as a unique_id in a text? So loading is not automatic, but there is still a searchable connection.
One downside, if the calendar is deleted, the unique_id would be left as an orphan, instead of being cleaned up automatically (by becoming empty).

Yeah: couple of observations:

The issue with page level privacy is the interface simply isn’t designed to enable that. You can craft these expressions (somewhere else) and paste them, but then ANY condition that passes unlocks a permitted value so you wind up with some very wonky expressions (and lots of them to cover all the cases). Implicit OR having to be worked around!

Not a good approach. (Though it DOES prohibit data loading… so it’s an option, but a fragile one.) it’s fairly easy to test for oneself — and yes it does work, but you’re fighting against other conditions if you are using Privacy Rules as intended.

Another fun fact, Privacy Rules CAN incorporate ANY conditional expression, but you cannot create just ANY conditional expression on that page… like if you need “Current Page contains…” you’ll have to construct that on a Reusable Element page and Copy/paste the expression INTO the Privacy Page…

Second Q:

The CALENDAR(s) are actually the thing I want, plus a few fields of their Parent Listing so that’s not the issue for me. The issue is limiting OTHER large objects on the Listing which are not needed in the page.

(Which is probably the right way to think about this: Let Bubble do what Bubble needs, but Stop/Throttle unneeded data.)

But yes, absolutely understood on what happens should one choose to reference unique IDs vs the object itself! :+1:

That was a nice find, but I’m worried it will get patched if it’s considered a bug.

@josh @emmanuel

Can we use this or will it get patched eventually?

image

Isn’t this more or less implemented in the DB views?

image

I haven’t done tests on this to check if unchecked fields are retrieved or not.

Yes, anything that involves working around our UI via copy / paste or similar tricks is a bug and cannot be relied on. If something doesn’t appear in a drop down, it’s almost always because we need to do work on the backend for it to function properly. Occasionally you might find things that look like they work, but in practice there’s usually some kind of issue that will either cause you unexpected problems or lead to us unexpectedly changing it. Sorry to be a buzzkill, I like Easter eggs too, but want to make clear that that kind of trick will almost certainly break on you at some point.

3 Likes

Thanks for the clarification, @josh! I figured as much.

BTW, this “pathological” data object I describe in this thread is pretty interesting – it has a bunch of data on it (from API Connector experiments) that it clearly in the database, but I have no way of deleting anymore.

This sort of problem is related to what I think is a very weird “feature”/behavior of the API Connector – specifically:

When an API returns a list of complex values (multiple-field/key objects), like in my case a “list of events” where events have things like event: {type, start_date, end_date, description}… These things DO NOT get a datatype in the database.

So, they are not searchable and not saveable in Bubble terms.

This seems like a bug to me. Or is that intentional?

Thanks,
Keith

Unrelated to the Topic here, but you bring up a valid point about fixing ‘workarounds’.

There is one workaround I find myself using regularly and was curious if it would be ‘fixed’ one day and break some app functionality I have relied on. Specifically it is using a Current Cell’s Index Number. I often find myself using text, inputs or groups in a Repeating Group cell and using the Current Cell’s Index. Then I move this element into a Group still inside of this same cell, but within a Group in the Cell. This is because you cannot access the Cell’s Index directly from a inside a group

Do you anticipate this specific example to be fixed one day, and thus breaking the current functionality, or would it be handled in a way to still work?

This one hasn’t been on our radar. More likely that we’d “fix” by making it an official feature, since off the top of my head I don’t think it is problematic — I’m adding it to our list of things to look at. But no promises until someone on the team investigates, which could be a while.

1 Like

I seemed to have buzzkilled this discussion with my off topic comment, but now I find myself also interested in this topic of not loading data fields that I will not need on a specific page (but will on a different page). Very similar use case of the Images for an Event, as an organizer I do not always want to load these when I load a list of Events, but on the Event description page I do want to load these.

Did you find a good solution to this?

Well @gf_wolfer, as noted by @josh, my hacky experiment in using conditional expressions which are not buildable directly on the Privacy Rules page is unsupported, fragile and prone to being removed. So, I don’t do that in my live app and don’t recommend doing it.

There’s no real workaround that I’ve discovered in terms of being able to take more control over what data from a complex object might be downloaded to the browser, outside of “normal” (non-hacked, if you will) Privacy Rules.

But I continue to feel that some sort of facility like this SHOULD be made available in Bubble. If not on the Privacy Tab itself, maybe as something like “Data Access Conditionals” that would allow you to compose any conditions that would (when the condition is met) add additional constraints to whatever Privacy Rule is in force.

This would allow us to force unused things (which the author of the app knows are not needed, but that Bubble cannot algorithmically know) to be blocked from downloading to the browser.

The only potential workaround I can see at the moment is to segregate one’s data types in what I would describe as a very un-Bubble-like way. For example, on my property listing data type, I have a list of photos and a list of amenities and a list of calendars (which each themselves have a list of events), etc., etc.

The only way to ensure/force that Bubble wouldn’t download those things is to just NOT have those lists on the Listing object itself. Instead, those objects (like photos and amenities and calendars, etc.) would all have to have a field on them that’s either of Listing type (if Listing is suitably lightweight as an object) or that is a text representing the Listing’s unique ID.

Then, one would only access those things explicitly when needed via Do a search for… the thing(s) you need… constraint that field’s value is this Listing (or this listing’s unique ID).

And while that might be more performant in some cases, it’s (as I said before) very un-Bubble-like. You should have Listing’s Photo List, Listing’s Amenity List, Listing’s Calendars List’s Calendar:item #whatever… etc. That’s the Bubble way. To have pages of a type Listing and then do searches for this and that and the other thing rather undoes many of the “rapid development environment” benefits of Bubble.

Basically, it’s like conceptually Bubble wants you to (encourages you to) create these pretty hefty data objects. But then performance-wise you can get yourself into a pickle and then various people will chime in saying, “well, you shouldn’t have done it that way.”

(To which I say, “Yeah, but then my app isn’t very Bubble-like at all.” So give me a way to pre-filter/prohibit various objects [more correctly: fields on objects] conditionally. Let me give Bubble some hints about what I need at the moment for those times when there are performance advantages to doing so.)

2 Likes

Agree here! Maybe to propose an #Idea to bubble, to have option with “App search tool” to search if there are any “deleted” fields in usage?