Remove duplicates when uploading CSV

Has anyone found a good WU-optimized solution to check and remove duplicates after a CSV upload via the Upload CSV action?

What I would probably do usually is run a workflow to check all the uploaded items and remove duplicates, but I don’t think that’s very workload-efficient :sweat_smile:

Depends on the data volume ? And duplicate are based on one field ? More ?

If one text field, and not that much data, what about do a search/each item xyz minus list do a search xyz uniques elements ?

What I do for users uploading csv, is looping on each row uploaded… and check if duplicate… not that much wu as it is a do a search.

For my use-case, more than one field with thousands of rows :smiling_face_with_tear:

I’m thinking about going the route of using a Cloudflare Worker that parses through the CSV and checks the duplicates, removes the duplicates then pushes the new data to my Bubble database via the Data API

1 Like

That is a good solution.

1 Like

Yep, I’d do the same, or use n8n for potential WU heavy operations

1 Like