Hi,
What is the best way to allow a user to upload a CSV file from the frontend and send that CSV file to an API configured in the API Connector through a workflow (or backend workflow)? The second part of the question is how we can have the user map the header fields of the CSV to a defined header definition of the API to ease the transformation of the CSV data?
Context:
We have an external backend execution workflow running in Azure that takes the CSV file, transforms and moves its contents into a database also in Azure (1-3 million rows per CSV file), and kicks off additional backend workflows based on the data in the Azure backend. We are not storing the contents of the CSV in Bubble due to the sheer number of rows of data, processed data, etc. However, the final data is exposed through an API for Bubble to retrieve and use in the broader solution.
If I understood your question correctly, the first part is fairly straightforward. You just need to add a File Uploader element on the frontend. When the user uploads the CSV, Bubble will automatically save the file to the server and generate a URL for it. After that, you can simply send the generated URL via an API call using the API Connector. Very simple.
Now, regarding the second part of your question — I’m not sure it’s possible using only Bubble’s native features. It likely requires a plugin or some external tool to handle that part.
Read through this post It outlines how to save a csv file into your bubble database (the example was through an option set attribute of type file, but can be done on a data type). It was for an example of 1,000 entries, but since you would only need to consider the size of the file for its ability to be uploaded via the file uploader or built in uploader on attributes and fields, I think you’d be fine so long as the .csv file is less than the max size allowed via uploaders.
Once you do that, you can use the data in bubble and don’t need it sent anywhere else so long as the data wouldn’t change at all. And with this you do not need to map any columns to fields as that is handled automatically when you initialize the API call in API connector.
This is available just after initializing the api call in api connector.
One thing that is nice about this as well, is from my testing, the Repeating Group ‘lazy loading’ features work too…on a repeating group if you leave blank the checkbox to ‘show all items immediately’ bubble does some kind of ‘lazy loading’, so even if you have millions of rows of data, I’d expect you’d still see some of the results show up pretty quickly on the page, and as the user scrolls down more will be added in.
Even though this will likely work fine for millions of rows of data as long as the .csv file is small enough to be uploaded through the file uploaders, if you can find a way to segment the data to break it up in the event that not all of it is needed at any one time, it might serve you well to do so.
