I have an Airtable database and I have some API calls set up in Bubble to pull records from Airtable and add them to my Bubble data base. In this workflow I have a trigger set up to repeat the process again at a later time.
I avoid duplicate entries by setting up a step in the workflow to ignore them. But is this the best solution for me to be using?
Each time this workflow runs, it pulls every single record in this particular Airtable database, and then filters it to avoid adding the ones it already got on the last workflow. Should I have something set up to somehow not have to get the whole database each time and just start where I left off? If so, any tips on how to do that?
There’s less than 100 records right now so I don’t think it’s an issue, but this base could get above 10,000 records in the near future, seems like a lot to process.
Could you setup your airtable data to segment the new records that need to be synced to bubble
(Data Stage A) and those that have already been added (Data Stage B)? After your process runs, the added records get moved from A to B to reduce the processing time each time that bubble runs your api call.
My recommendation is a sentinel value. Something you can filter on against the source airtable. Be it row number, modified date, etc. Store the last value you used as metadata on the Bubble-side.
On next call of the API, your Bubble workflow checks for the last sentinel value and then supplies it as a filter criteria to the API to only fetch newer rows than the sentinel. Finally, update the sentinel based on the highest value of the data you retrieved from airtable.
Agree w/ the sentinel value idea. You could also just write a flag to Airtable rows that have been sync’d. Add a checkbox field and mark it checked when sync’d. Or, compare a timestamp of some kind.
@jackwabbit - Would you mind sharing how your workflow looks? I have a need to do something similar and am curious how you’re getting data read-from-Airtable-and-written-to-Bubble cleanly.
Thanks for the help everyone. I’m a bit confused on the sentinel value thing, I’ve never heard of that. @ALB Yeah that makes more sense to me, I’m a bit new to this stuff. I’ll try the checkbox method or some kind of date/time-stamp set up.
Also, @ALB did you still need info on the workflow?
Sorry, this is a side project and I just come to work on it every now and then that’s why I replied so late lol.
First I do this “airtable-api-call”. I use the Bubble API plug in and have a GET call set up to pull fields from Airtable. I just used the Airtable API Doc to get that call set up.
