You can do this easily with Data Jedi Plugin
This is not 100% true. You can, if you want, return 129,000 records. But should you, or rather, do you need to? The real question would be do you need to. In your case, with Data Jedi, you do not need to. You can run this conversion in a backend workflow and return only the records that changed.
@ian11 I do not know what the data structure is of the API call, but based on what you wrote that I read, this statement is not necessarily applicable to your use case since your data count comes back as delta and not an aggregated count, requiring you to add or subtract those delta values to arrive at the ‘new’ values. I could be wrong, but that is how I read your use case.
No real need if you are using Data Jedi or some plugin that allows you to easily modify values on front end or backend. You can instead just take the incoming api call data, filter out all delta values of 0 and then just modify the values whose delta is greater than or less than 0.
How is the user going to actually interact with this. Is this some dashboard that is open all day long that they continuously view to have minute by minute updates or is this a dashboard they will go to view once a day or week to just check up on impressions?
Is the user needing ALL values, or can it be restricted to fetch from api only the values relevant to the current user?
Is the delta value going to be returned from the API such that if you do not request the data for 24 hours, will the delta be accurate based on changes since your last API request. Same concept, if you request every 10 minutes, is the delta value accurate. For me, the question is, if you request every 24 hours, does your delta value accurately assess how much has this value changed since the last request 24 hours previously. OR do you NEED to have it every minute in order for delta values to be accurate?
Is the 129K rows of data for 30 day period? Or would you have the same 129K rows of data over a 3 minute period? What I am wondering is what does this data from the API represent in real terms. If you have a ‘impressions’ with an unique ID to represent the impressions for a one minute interval on an advert with unique ID so we can say, call the API anytime, every 1 minute or every 24 hours, you will likely see the same unique ID of an advert listed more than once, with multiple unique IDs of the ‘impressions’ delta. So I fetch via API would I be able to filter by the ID of the advert to see all ‘updates’ of impression delta since my last api fetch request regardless of duration between fetches.
If you would see the same advert unique ID multiple times, each time having attached a unique id for impressions over the time span, that therefore has multiple delta values (+1, +4, -6), this would be easy to put into the arbitrary text operator in the screen shot below as a list of deltas because the + and - in front of each will result in the math returned properly.
Does you api provider allow for constraints, especially date constraints.
In order to really solve the query, we need to understand clearly, what is the data structure, how is the data returned, what constraints can be used when fetching the data, how is the data going to actually be used by the user.
When knowing all those very important details in coming up with an optimized approach (there are so many ways you can do one thing), you can land on the best solution based on your data and use case.