Better performance: less entries and more lists, or more entries and no lists?

I’m wondering what’s best in terms of performance: having less DB entries (lines) and using “list fields”, or just creating more entries?

For example:
A) When a user completes a task, write TaskId inside the CompletedTasks list field inside SomeEntry of SomeTable.
B) When a user completes a task, create new line with UserName under UsernameField, and TaskId under TaskField, inside WhateverTable.

How Many Users Can Bubble Apps Support? (Critical for Scale) (youtube.com)

This one

1 Like

Welcome back to the community @luuustuuucruuu !

Using lists is certainly better for performance. However, as the app scales these make the entries heavier and this slows down the app. Flat data types scale better.

Having said this … many MVPs focus on doing one thing great. Once the concept has gained meaningful traction … further considerations to scaling come into play … and the app likely will need a re-build. So … short list fields can take you a long way without spending too much WUs and performing as best as possible.

:+1:

1 Like

It depends entirely on what you mean by ‘performance’…

In terms of what is faster to load, lists have always been, and still are, faster to load than doing a search for the same things, and cost less in WU - up to a point…

I ran some extensive tests on this early last year (after WUs were introduced), and loading a list of 600 things was always faster (and less costly in WU) than doing a search for the same 600 things… whereas loading a list of 2000 things was slower than doing a search…

I never established the exact point where it changes from one to the other, but it must be somewhere between those numbers (although it’s probably far less to do with the number of things, and much more to do with the size of the data, so in that respect numbers like those are meaningless on their own).

Although the actual perceived difference in speed is probably negligible, the difference in WU cost can be significant.

But that’s purely in terms of loading the data to the page… there are way more things to consider (both from performance/WU perspective, and other database/functionality aspects, than just that).

You have to look at the bigger, overall picture of your app, and not just one specific page or RG to establish what is truly more efficient.

So, as always, the exact answer will depend entirely on your specific use case, how and where you’re loading the data, and the amount and size of the data.

That said, I would generally advise against using lists whenever:

  1. you know there the list will contain more than 10 things (some will say 30, others 100, but in any case, you know there will be more than a small number of things)

  2. you don’t know how many things the list will contain (even if in most cases it might be a small number, if it’s possible for the list to contain hundreds or thousands of things - i.e. you’re not limiting the number of things which can be added to the lists)

There are still exceptions to the above - where storing several hundred things on a list field can be a better option, in both load speed and WU cost terms, (and there can be cases where using lists of texts, rather than things, can offer a good compromise between performance and WU cost), but generally I’d stick to the above rule.

For your specific use-case, where there is an indeterminate number of completed tasks (plus, you may well need to store additional information about the completed tasks), creating separate database objects is definitely the way I would advise.

2 Likes

Hey @adamhholmes

Wise advice!!!

I opted to suggest lists because most projects if approached as simple MVPs end up being rebuilt.

You are absolutely right though. Just looking at an app … and planning all aspects of it … I agree 100% that flat objects scale significantly better.

On another note: I once read @keith saying that the hard limit of a list field is 10k entries. I read that your take is that there is no hard limit. Any comments on this?

1 Like

I agree… the truth is, in most cases, even using lists ineffectively/inefficiently probably won’t cause that much of an issue at small scale.

That’s correct, there is a hard limit in Bubble of 10k items for list fields.

But in practice (at least in my own testing) anything over about 3k becomes unusable on the font-end, so even in the specific cases where larger lists can be more efficient, they still can’t be that big.

1 Like

I thought that was the case, too, but I was recently told otherwise.

1 Like

Thanks Gentlemen @adamhholmes @mikeloc for chiming in!

Just checked the Bubble docs … which by the way … have improved substantially I should say and they do indicate a hard limit of 10k

Just fyi which I know is nothing new to neither one of you :wink:

Thanks again!

1 Like

My own testing (just now) still shows 10k as the hard limit for adding items to list fields:

(in fact, it seems the limit is actually 9999… trying to add one more item to a list that already has 9999 items results in the error above)

2 Likes

:eyes: :+1:

With all due respect to Keith, I didn’t necessarily think he was right (although, he did say he tested it), but I honestly didn’t care because I would never use a list field to hold more than a couple hundred items.

2 Likes

Yeah… exactly… whether it’s 10k, 100k, or a million, it’s hardly important… as the real practical limit is much much lower, and for most reasonable use-cases lists are going to be used for 10s or 100s items… not thousands.

note: the limit of 10k list items only applies to database list fields… not other Bubble lists, such as custom states (which seem to run into trouble around 130k items).

2 Likes

Cool find. I remember @chris.williamson1996 (I think) tested what happens if you reach the size limit of a single field and it was just a silent failure, so interesting that this at least throws an error.

3 Likes

Yeap … Iit is nice that these error notifications are more prevalent across the platform now

1 Like

This might be true for Things and it might be true for operations like “workflow on a list”. But the number of entries that can be in a List field look what you can do with numbers:

List Math Object is a Thing with a numeric list field on it. You can add up to 100,000 numbers to such a list:

(Attempting to create a list larger than that will fail. Attempting to add another item to a list field that is already at 100,000 items will fail.)

3 Likes

I figured that you would come at us with valuable info @keith :wink: :+1:

1 Like

Thanks a lot @adamhholmes and others for taking the time to write.

Think of my app this way: each user must review employee profiles; or at least read them (that’s what I need to store—did userA read employeeX profile?.. and so on for each employee). At the end of the day, my app may end up having just a hundred user, but of course I’m building as if it’ll get a million :wink: So let’s assume we’re talking big numbers of users and of employees.

To add to the complexity, each employee is nested in a team, each team in a department, each department in a division, and each division in a company. I don’t need to display all this hierarchical data at once (only teams and employees that belong to the same department at once), but of course the backend connections must be in place.

One thing to be noted is that each employee will have a unique number (from 1 to whatever).

So I can either use lists:

Or go with tables:

That “Reviewed employees” list could therefore potentially be very long (although probably not thousands), and according to what you said, that could be an issue:

Now what you said about WUs worries me a bit, because since multiple employees are displayed at once on a page, the user could “review” their profile pretty rapidly (click, click, click… and voilà, already 3 database writes/updates!). Currently that’s what I have in place (not lists).

That being said, I’d rather pay more in WUs than have issues with performance and scalability… Now if database entries is still the way to go, then I guess that would raise the question of how to structure it. Currently, I have separate tables for companies, divisions, departments, teams and employees, each being referenced to one another with ‘data type’ fields, and a “Reviews” table (similar to what’s described above).

@cmarchan @mikeloc … upon further investigation, @keith is 100% correct, in that list fields can indeed contain up to 100k items… (not 10k as the Bubble manual states). And not just for Numbers, but for Things as well.

The issue is how you create the list in the database.

There is a difference between using ‘Set List’, and adding items using ‘Add’ or ‘Add List’

Adding items to a list, using ‘Add’ or ‘Add List’ will not allow you to have more than 10k items on the list.

So trying to add one item to a list that already contains 10k will result in the ‘List at maximum length: 10k’ error.

So will trying to create a list using ‘Add List’…

…on an empty list, adding anything more than 10k items will not work and will produce the above error.

But, using ‘Set List’, will allow up to 100k items to be set on the list.

However, nothing can be added to a list that already has 10k items.

So, if you set a list of 35k items using ‘Set List’ then try to add one more item using ‘Add’ (or 10 more items using 'Add List), you’ll get the max list error.

So the Bubble docs are indeed wrong on this (at least in one sense).

4 Likes

How in the world can you test this so fast Adam … hehehe Thanks for doing and for sharing :+1:

1 Like

Great additional info from @adamhholmes, as always, sir! It wouldn’t be obvious from my screenshots, but the very large lists of numbers that I generated were being made from the outputs of my Make Numeric List and List Math server-side action plugins (from Floppy) and, indeed, I attached those lists to a Thing using “Set List”.

I don’t think I’d realized what you discovered about Add and Add List.

1 Like