I’m struggling with how to setup the backend/database structure of my webapp, and it seems like Bubble is not a good fit for storing and working with large amounts of data, hence I wanted to hear if you have recommendations for how I can store data outside of Bubble?
The webapp is a weekend travel booking & discovery site.
In the webapp you can browse all possible destinations suitable for weekend travel (short direct flights) from your departure airport, and you can see all the flights per weekend (Fri-Sun) and associated prices per destination.
To do that, I need a fairly sizable database of all possible direct flights on a Fri-Sun Schedule for the next 6 months (around 50k flights for one departure airport). I get this data outside of bubble, by running a script that does 1500+ searches for different destinations and dates, and generates around 50k+ results. I upload those results on a weekly basis into a flight prices data type in bubble. Since past flights uploaded may be unavailable and may have changed in price, I delete the entire flight price database before I reupload the new search results.
That allows me to see all flights for all weekends in next 6 months, filter on it, and display e.g. minimum prices.
You see this data in a few different ways:
- On a repeating group where I show all possible destinations (a different datatype) and look up the minimum price in the flight prices data type.
- Om the destination deep-dive page where I show all possible flights for that destination, show analytics of how those prices evolve over different weekends, and allow you to browse all flights and see details of each of them.
Check the work-in-progress here: www.weego.dk
As you may already be thinking, this process is very painful in bubble. It takes a lot of time to delete a database of 50k, and a lot of time to reupload a new set of 50k entries. As I scale the platform to cover other destinations, this will quickly scale to include 500k+ data entries, and thus the operations will be too time demanding. I am therefore looking at a different place to store and update this flight price database, that’s easy to manage (ideally no-code), and where I can then call APIs to get the data I’m using above.
I’ve explored tools like backendless, but they vary pricing with number of items, so it very quickly becomes very expensive for my use case.
Any good ideas of how and where to manage this database?