Has anyone developed a really clear best practice guide to managing changes to your database between dev and live? I’ve found a heap of old forum threads that give pieces of information, ranging from “CSV’s are the way to do it”, “CSV’s suck”, “recursive workflows” … but nothing that goes into enough detail to really step a relatively new developer through those process. (or at least not from what I can find).
For context:
I have a live app with live user data in it
I have now created new features in the dev environment, including new fields on existing data types, that need to be migrated to live.
I want to get the live data (old data type structures) into the dev environment (with the new data type structures) and then upload all of that back into live.
I have tried using the Export/Upload CSV workflow with many issues, including errors on User data types because the “user already exists”.
Would really appreciate some detailed, expert advice on this!
Hi Adam, I’m trying to find the right workflow for me to continuously improve the live application, after making changes in the development environment that include modifying fields in data types. It’s not obvious to me as a relatively new Bubble dev as to what process to follow to ensure that I avoid having to completely “copy from development to live” each time to make major changes to the database (changes to the app front end is fine).
Hi George, you make it sound very straight forward Are there any published guides on HOW to implement that process? I’m looking for some detail. Cheers.
Build a backend workflow to run these modifications
Schedule this backend workflow in development using a button on an admin page to run it on the development database and make sure it works on development data
Deploy to live
Schedule the backend workflow in live to update the live data
Yeah, as @georgecollier suggests, you don’t want to be copying live data to the dev app, and then back to the live app (too much chance of data inconsistencies doing that).
Just make whatever changes you need directly to the data in the live database.
So be sure to include whatever workflows you need for data manipulation in your build, then once deployed to live, run those workflows on the live data.
You can use the Bulk update button in the app data section of the editor
This eliminates a need to create any extra work like adding buttons onto an admin page.
Just remember about Bulk is that it is only used to basically trigger a backend workflow on a list from the editor directly. You have to select the type of data first while in the app data section of the editor, then you are provided a list of backend workflows that can be run for that type of data selected. The only backend workflows that can be run are those that have just one parameter and that parameter needs to be of the same type of data selected.
What is great is that you can use the ‘views’ feature of the app data section to create the list of data entries to run on, since when you click bulk and select your backend workflow it will run on the list of the current ‘view’ from the app data section.
Makes it very simple to make changes to specific entries when at times that is needed and there would be no need to constantly update the search on an admin page to get the right list of entries to run on.
@dalerankine, also bear in mind that you sometimes need to deploy database changes in stages. For example, you may be deprecating Field A and adding Field B and Field C. If you need to reference Field A’s values in order to populate the new Fields B and C for your live users (using @boston85719’s procedure), don’t delete Field A until you’ve deployed Fields B and C and populated them in live.
Thanks David, makes sense. I’d love to see these steps fleshed out as a step-by-step - I’m not very familiar with using Bulk or backend workflows, or how they’re used to add/deprecate database fields. I’m pretty sure with that extra help I’ll get familiar with this quickly.
Imagine your app currently allows users to include a single phone number in their profile and you make changes to allow them to add multiple phone numbers to their profile. The changes you implement might include the following modifications to the database schema:
Create a new Phone data type with a User field and a Phone field.
In the User data type, add a new Phones field and delete the existing Phone field.
Data type
Fields before change
Fields after change
User
Phone (text data type)
Phones (list of Phones data type)
Phone
n/a
User (User data type) Phone (text data type)
Here’s a high-level order of operations you could follow.
Create the new Phone data type with its fields.
Add the new Phones field to the User data type.
Create a backend API workflow that includes:
An input parameter to specify a User
An action to create a new Phone thing using the User and the User’s Phone
An action to add the new Phone thing to the User’s Phones field
Run a bulk operation using that workflow for all the users in your dev database.
Deploy to live.
Run the same bulk operation in your live database.
No, not A/B testing the live site. The original post was about how to efficiently change the live database when the structure of the dev database is different/updated.
Are there any published guides on HOW to implement that process? I’m looking for some detail. Cheers.