Hi,
I am looking to build an ETL that loads a large CSV file with around 40 columns and 200 K rows , store it in database and transform the data to be used elsewhere. for each transformation I have a predefined time I can’t exceed (15 min to load and map the data)
Different CSV files come every 20-30 minutes and I need to pick-them up one after the other and manipulate them. So I need kind of a chron job to do that.
I searched Bubble forum for CSV upload and did not find the help I need (saw examples using Etsy, but not good for what I want).
Any help is appreciated
Best,
Gerard
Are you using the Bubble CSV upload, or going via the Data API ?
Hi
Would prefer Data API since this should be an automated function
Best,
Gerard
Makes sense. Someone has used Talend Open Studio to do this, successfully it seems.
connected to Bubble? Is there any doc or video explaining how to combine Talend Open Studio with Bubble.is?
Thanks
No, don’t think so.
Looking at TOS is on my “to do list” for Bubble.
If you do a search for Talend, there was someone who had used it.
Will try this. Thank you Nigel
In Talend OS, you can use tRESTClient component to send or receive data from Bubble’s Data API. Other components are there to prepare JSON’s, and read or write to CSV files. If you need an example, I can prepare one.
Thank you Tomaz,
I will appreciate a small example if it is not too much too ask.
Best
Gerard
It will take a while for me to reach my computer, please allow me a day or two.
I see now that you have 200K rows - bubble is relatively slow, I did my migrations record by record and it took approx. 1 day for 300K. There is an option to do a bulk upload of 1000 records at a time in Data API, I haven’t use it though. 15 minutes might be too optimistic…
OK good to know, one day to manage 300K is not a production level programing. i will check on the bulk upload for the 1000 records. I appreciate your help
Yeah, an option is also to use external db and db connector plugin.
Here’s a simple Talend job which writes a row of data (a thing) into a Bubble table named myThings:
I suggest you look in the Talend Help what is the purpose, functioning and properties of specific components, e.g. tWriteJSONField, tRESTClient.
The above job firstly prepares a row of data with tFixedFlowInput, you can use tInputFileDelimited to read CSV data. tWriteJSONField transforms the row of data into a JSON structure, and tMap routes this structure into a string part of a REST request (these requests have a body and string parts). tRESTClient actually calls the RESTfull service, and for Bubble Data API can be set as:
tRESTClient has two output flows, its output depends from the answer received from the server (e.g. 200, 404 …), so you can control the situation. Content types can be set under Advanced settings | HTTP Headers of tRESTClient
I hope this helps.
Thank you very much. I will surely help. I appreciate your effort.
Gerard