Building an app for football stats. It will handle multiple API calls but the main one that I have is bringing in NFL Rosters and updating them. While it wont be bringing in all players, it will be bringing in certain positions so anywhere from 400-800 player worth of data at a time including name, player id, age, weight, height, teamid and position. The API call has limitations to how often it can be pulled per day so displaying the data straight from the API isnt really an option. My thoughts are these 3 options
Option 1 - do a recursive workflow where it delete all of the players in my bubble database each time, and then re-uploads the database. This wouldnt require any searching of the database but would entail a lot of deletion and creating of new data each time which is obviously workload intensive.
Option 2A - Do another recursive worklflow but have it only create new players when there is no player in my database with the same playerid as the piece of data coming in. Then the next step in the workflow would be to update a player but only update that data if something in that players data is not the same. So id also have to do a search for each field of data for each player to make sure its identical. If it isnt identical, it would update the player
Option 2B - Same process as 2A except just automatically “making changes” to every player in the database without doing a search for even if nothing has changed.
Is it more workload intensive deleting and creating large amounts of data like option 1? Or would I be better off only creating new players when they show up and then updating the ones that are already in my database? Realistically that wouldnt involve alot of creation or data changes but would the workload from just making sure nothing has changed be worse?
If it helps at all, a datatype can have a field with the same type as your API call type. So you can create a “cache” datatype that has one field with all the API data for the day. Then just pull from the latest cached data as needed.
@thedraftprofile my paid plugin Data Jedi has a server side action and a client side element that can compare two lists of api objects. On the server side the action will have returned values (for use in subsequent actions as data source for dynamic expressions), these are 4 returned values, new items (things in new list not found in original list), modified items (things in both original and new list but with field differences), unchanged items (no changes and found in original and new list) and full list (list of all items modified if different, plus new items and original unchanged items).
It is built to do exactly what you are trying to do, which is to compare two lists of api data to determine which to update and which to leave alone.
Depending on your approach WU can become very costly with any of the approaches your mentioned, except for Option 2B keep in mind that even if you run ‘makes changes’ if no changes were needed it costs a bit less (about 0.5 WUs less) than if it had to make the changes into the DB.
Another thing as pointed out, you can save api data into the bubble database as a field on custom data types. What you could do is just have the ‘team’ have a field that is a list of ‘players’ with each being the api data type.
If I was building for a client, I’d use the team data type with api objects for players as the list field, run the server side action with my plugin to determine which need to be added and which need to be updated and would run a recursive backend workflow on the list of teams to do the comparison for the list of players.
