Workflow for tracking car dealer listings with external api

Need help
1

Hi guys, I need help with a current project I am working on. I am building a tracking tool for a used car marketplace website. I am using their API to get the info about the listings of specific dealears. I make an API call looking for the listing ID’s that are available on that day, then substract the ID’s that were available yesterday and get the ID’s of the cars that were sold.
Step 1:

  1. I manually enter a dealer ID.
  2. A new Dealer data type is created with the provided ID.
  3. A new Date data type is created to track the current scan’s date.
  4. The Date is linked to the Dealer.

Step 2: API Workflow (Step1)

External API has a limit of 25 listing per API call

  1. Fetch up to 25 AdIDs (listing IDs) from the external API for the specified dealer.
  • Each AdID is added to the Date record created in the setup step.
  1. If the dealer has more than 25 listings, I schedule the same workflow (Step1) recursively, incrementing the page parameter to fetch the next set of AdIDs.
  2. Once all pages are processed, I move to the next workflow (Step2).

Step 3: API Workflow (Step2)

  1. For each AdID fetched in Step1, I schedule a recursive workflow to:
  • Fetch detailed data about the listing (e.g., price, description, etc.).
  • Create or update a Listing data type in my database.

Step 4:
I schedule the workflow for 24hrs to check for the availabe AdID’s if there are any new ID’s those are assigned to the new_cars field on the Date type. Missing AdID’s are assigned to the sold_cars field on the Date type.

I would love to get some feedback on how would you approach this, I feel like my workflow is too complicated. Also another thing that is bothering me is that I can’t confirm if the car is actually sold or just removed from the dealer page for a few days because of the malfunction.

2

I assume they have no webhooks for when they update a listing such as selling the car?

The ingestion process seems pretty standard if you have to manually check yourself. Also when you’re getting the IDs from the dealer; can you filter based on any other information?

Side note: In the UI it may help to put a small text element that says “last updated 12/09/2024 7:00am” so that users know there may be changes.