Hi guys, I’m planning to use a table with 15 million rows and 3 columns (origin, destination, distance). People usually use distance_matrix from Google but this was not an option in this project. What would be the best approach to this? Import this to Bubble? If yes, how? Or request it on demand via API? At the moment this table is in a vps DB (which I’m planning to close if I can use Bubble store that). Although it’s huge, the requests are going to be quite simple, having both origin and destination passed to return the distance. Thanks in advance.
Wow…
Never heard of any database that big inside Bubble, even knowing that Bubble have no limit for this. I, particularly, would keep it outside for personal reasons…
Why you wish to bring it in instead of load it thru an API call?
Hi Rafael, save the money I’m currently paying for that vps and keeping everything inside Bubble might reduce the risk of lack of connectivity and make things fast… (not sure my assumptions are right tough… fairly new to Bubble and no programming exp here… lol)
I see…
You could try importing it (partly) into Bubble and see how it works… keep importing and running benchmarks in parallel…
I am curious to see the result…
@lottemint.md @petter @nocodeventure @NigelG @Jici @mikeloc @adamhholmes @boston85719
Care to chime in?
1 Like
Thanks Rafael, what would be the best way to import it? I think CSV can’t support these many rows right? (I think excel’s limit is 1 mi…). Here are the options I have…
The biggest table I have actually in one of my client app is 1 310 000 items with 5 column. We use this in a Bubble search box and this work very well. It really depend of what you want to do with data and how you will display them (get using a search in RG, or using a searchbox to select item…).
So first question, how will you query Data from the DB (or API…)?
For import, we had to import this data set too, and to avoid any issue with CSV importer timed out, we choose to split the file into smaller chunk. Someone take a few hours to import all files but we didn’t get any problem. There’s a limit on how many item you can import In one file if I remember.
There’s a few plugin also that might help to work with CSV or JSON file. I don’t know them all but don’t hesitate to ask plugin author if this can work for your case.
2 Likes
You are right, CSV limit is 1M rows. 
But anyway, it is better to split it in 15 or 16 different files instead of be surprised with a timeout issue after many hours importing one giant file 
… And this also gives you the opportunity to keep running performances tests gradually…
Cool, I agree. I will test it and let you know how it goes.
1 Like
hey guys, just an update. I’ve managed to split that into 1m rows files but the first one took me 12 hours to be uploaded. 3 columns, text, text, number. I might have a look at the API option and create a dynamic way to save the data requested into a smaller table. I hope the time to learn how to do it takes less than the 16 days it would take me to upload all this lol… Anyway, I won’t need ALL the distances between every city to every city in Brazil so this new table will grow as my user base (in different locations) grows. It might delay some internal calcs due to the API requests but It looks better than having a that have table there with useless data in it…let see how it goes. Thanks for the time to look into it!
two weeks uploading CSV…
1 Like
As it is fixed I would think having it external (Firebase/GBQ) might be easier.
4 Likes
For fixed data, I agree. Lightning fast results!
1 Like