Complex CSV/Sheet import to DB (Recreate table in Bubble)

Hi there :grin:
I have a pretty complex task to achieve in Bubble. Here’s the problem:

  • I want my user to be able to upload sheet files (such as CSV or XLS)
  • I want to import the rows and columns in my database
  • I want to be able to display the data exactly like the original file in Bubble using RG

So the goal is to upload and recreate the table in Bubble.

The thing here is that the CSV file will never have the same data structure (columns will changes, …) so I can’t use the native CSV upload and third-party such as EasyCSV.

One thing that could work will be to receive the data of the CSV file as a list of columns or rows on an API workflow and then create a thing for each item using the Schedule API on a list but I can’t figure out how to do that either…

If someone here have an idea I’d love to hear it :heart:
Cheers

1 Like

Hi!

This is indeed complex, particularly when the uploaded CSV’s are not a predictable structure.
I have been tackling a similar challenge, so I can share some knowledge…

The hardest part is mapping whatever the uploaded column names are to the field names of your database.

I’ve found some various tools that enable this process, but it involves a number of steps and can get complex. Here are what I’ve found to be the simplest:

  • I’ve found the first step is to translate/parse your uploaded CSV data into JSON. This makes it somewhat easier to manipulate, and also easier for eventually sending the data to the Bubble Database. There are a few plugins that do this well:
    https://bubble.io/plugin/data-parser-1512303845588x451639973002608640
    There’s another plugin called “CSV File to JSON” (which you can search for in the Add Plugin window)
    These plugins can return both the Row data, as well as a list of the “Column Names”.
    You’ll then need to temporarily store this data into a text field either in a custom state or in a database record.
  • You’ll need to design a specific popup or page that will take the list of “Columns Names/Headers” that were parsed from your JSON data and allow the user to “Map” those columns to the field names in your database.
    This can get tricky, and there are several ways to do it depending on how many columns there are. It could involve creating a new DataType soley for matching “old column name” to “new column name”, and displaying that in a repeating group. Or it could be done entirely on the page using custom states, but again this can be tricky, and is a bit too complex to explain in detail.
  • Either way, the ultimate goal is to perform a Find & Replace on the raw JSON text string to essentially rename the “key/column names” to match those in your database, based on how the user chose to map them. This is necessary in order for Bubble to properly import the data.
  • Once the key/column names have been renamed, then your best bet is to make use of the Bubble Data API to import the data to your database.
    https://bubble.io/reference#API.get_api
  • There is another plugin that can simplify this a bit:
    https://bubble.io/plugin/bubble-data-api-1512305317944x713478546001494000
    And conveniently, your data will already be in JSON format, which is what the Data API expects.

Unfortunately, it is not an easy process lol. Again, the biggest struggle is mapping unknown column names to match your DB.

Here are a few other resources that I’ve discovered on my journey that you can check out that may or may not help:

  • Parabola https://parabola.io/
    Can be helpful for renaming/parsing csv data. However, they don’t natively allow API webhooks.
  • Integromat https://www.integromat.com/
    Integromat is somewhat advanced but EXTREMELY powerful for processing and editing raw data. It’s Parabola on steroids, with native API webhooks. I’ve used it also for renaming column names. It has a bit of a learning curve though…
  • Orchestra Plugin https://bubble.io/plugin/orchestra-repeating-group-addon-1581095540181x938333534159634400
    This may prove useful when trying to have a user “map” column names to DB fields within a repeating group. It basically allows you to access the data WITHIN a RG cell, and create a new list out of that data.

I’ve been playing around with a test import page that I might be able to share at some point, TBD.
Hope this helps!

2 Likes

@brianabbottlight I owe you one for this long and detailed answer.

I tried with Integromat to get the CSV file in input and have an API call as an output with the columns as lists.

This is what I tried so far:

So basically it takes the CSV file sent by Bubble to recreate one in Integromat (which work)

Then I try to convert it using the JSON converter:

And there’s my problem, the body content seems to not be formatted in a good way, so Bubble return “error parsing the JSON”

Thanks for your time :grin:

I might have a solution/hack… will try to put it together in a test app and see if it works. The only constraint would be that you would need to set a maximum numbers of column which a user can have.

After re-reading your OP, I think I may have slightly misinterpreted your overall goal, though most of what I suggested is still applicable…

To clarify: do you want your users to be able to have custom names for their columns/field in your app? Or is a user uploading sets of data into a pre-determined set of fields (for example: name, address, phone number, etc.)?

What is your app’s general function, if I may ask? Just to try and get a better understanding of the goal.

If your goal is the first option (having users have custom names for their columns/fields), then this is possible, but it will be limited to a maximum number of columns/fields that a user can upload (as @shawnmi6 also mentions) .
Bubble does not allow the creation of “columns” or “fields” by a user. That must be predefined by you in the editor.

1 Like

@brianabbottlight

First option. Users will be able to upload different CSV files with a different format of columns and rows. (Max 20 columns and 500 rows).

The final goal is to build an accountancy app where an accountant can upload XLS files and their customer can see them in a custom way in Bubble (using RG to recreate the table)

I don’t really know what we can do with code here, but I think this could work if I was able to get the table as lists.

1 Like

@vnihoul77

Were you able to figure this out by chance? We are trying to do something very similar right now. Allow user to upload different excel files and then have the be displayed in a custom way. If this is not possible, it would be good to know.

Thank you!