Building w/ Existing Database

ian.goldberg · 2020-06-11T14:03:26.166Z

Hey everybody,

I’m new to bubble, and am looking to build some sort of internal CRM for our company and our franchise partners. We maintain a large database of clients, and subscriptions and it has around 200k records in it, and powers our business connecting to various different SaaS tools (Zendesk, stripe, ActiveCampaign for emails, etc).

We started exploring some of the database features of Bubble. We run our organization on Azure with SQL Server. In order to use Bubble, we would need to be able to integrate with our existing database.

I was able to set up the SQL Plugin to pull data from our database, but based on the docs and community, it looks like we are limited to 200 results in a single select query. Given the volume of data we have, this is a roadblock.

Therefore, the first question I have is how can we, if we can, overcome that limit? Have others run into similar issues?

An example of something we would want to do is build out a searchable customer table with 2000 entries that are connected with our Azure database. What is the best way to go about trying to build something like this?

The second thing I have noticed with the SQL Plugin is that the IP address that needs to be whitelisted on our database seems to change every so often, which as you can imagine, would quickly get out of hand. Thoughts or ideas on what might be happening here?

Would love to hear some feedback around this.

Thanks community!

ian.goldberg · 2020-06-12T13:22:39.204Z

friendly bump

shpak.serhiy · 2020-06-12T13:57:07.596Z

Hi Ian,
have you already participated in alpha testing of the new CSV-uploader?
It seems it is the only viable option for your project to import the DB not via SQL-connector but via CSV-uploader, that in alpha version has no limitation on the number of imported records.

Just in case - here is the link [Recruiting for Alpha] Large CSV data imports

ian.goldberg · 2020-06-12T14:22:23.705Z

We are trying to see if there are any options to avoid maintaining two databases. We have systems (zendesk/chargify/ActiveCampaign) that result in changes in our client data that we don’t want to have to continuously sync with another database in bubble. Trying to figure out if there is a good method to use an external database as our main bubble database.

ian.goldberg · 2020-06-14T17:20:11.431Z

Any other thoughts?

shpak.serhiy · 2020-06-14T20:00:17.493Z

Hi Ian,
If you do not want to maintain two databases you actually do not need to import your existing base into Bubble - obviously you will create a second DB.

Why don’t you query your database directly from Bubble via SQL plugin?

If you have already tried it - why didn’t it work for you?

Best,

Serhiy

ian.goldberg · 2020-06-15T13:29:57.241Z

Hey @shpak.serhiy we are running into the issue where the SQL plugin can only pull in 200 records at once. Is there a good workaround for this? We have thousands of clients in our database.

Ian

shpak.serhiy · 2020-06-16T06:08:17.005Z

It seems like generic limitation put on all data imports into Bubble. I faced similar limitation when getting data via API - 100 records on Personal plan. If you want to get more records you have to implement iterative workflow and repeat data importing step. I didn’t played with this workaround, but if it is possible to specify in the request from which record to start and how many to import, this approach could feasible.