Hey everyone, I’m new to Bubble and have very little experience with coding and APIs.
I’m building a project where I have a python web scraper going to crawl URLs that users enter. The URLs will be saved to their account and the goal is to display the updated price on a daily basis.
My question is, after I scrape the price and send it back to Bubble to store, is there a best way to save it to the database? Ideally, I want to be able to store historical data to be able to display pricing trends as well.
My thinking is to have 2 data types, one for the current most up-to-date price for specific URLs and the second to store the historical data. Is this needed or is there a better way to go about this?
Thanks in advance!
1 Like
Hey @mstlouis6595 - welcome to Bubble!
This sounds good to me at a high-level. I would personally have a “URL” data type with a “Current Price” field (type number), and a “Price History” data type with 2 fields: “URL” (type URL) and “Price” (type number). Everytime you scrape a URL, you create a new Price History, and update the URL’s Current Price if necessary. You can also delete old Price Histories to avoid overloading your database unnecessarily overtime.
How to best structure it for optimal performance would depend on your exact use-case - for eg how long you would want to keep Price History records for each URL; if you want to put a limit on the number of Price History records you store; or how many URLs you’ll have in total.
For eg - if you’ll have lots of URLs, but very few Price Histories for each one, you could have a field “Price Histories” (list of Price History) in the URL data type, and list these as This URL's Price Histories.
However - if you have very few URLs, but lots of Price Histories for each one, you would be better off with using Search for Price Histories.
