Forum Academy Marketplace Showcase Pricing Features

Getting all records from Bubble's Data API

If you’re trying to pull records from your Bubble database, you may have found the following limitations:

  • Workflow API = 50 records per request, one request per workflow
  • Data API = 100 records per request

Luckily the Data API supports pagination but requires a bit of code to be able to pull all records. Using the async library in Javascript, I wrote the code snippet below to fetch all records using multiple requests in the data API.

Hope it can be of use to anyone else who needs it!

    const axios = require("axios");
    const { doWhilst } = require("async");


    (async () => {
      // set variables for how Bubble handles Data API calls
      const pageCount = 100;

  // set allRecords to receive data, start from beginning of call
  let allRecords = [],
    noOfPages = 0,
    cursor = 0;
  await doWhilst(
    async () => {
      try {
        // get first batch of data, and if there are remaining records, set noOfPages to that number / 100
        const { data } = await axios.get(
          `BUBBLE_DATA_API_URL?cursor=${cursor}`
        );
        const { remaining, results } = data.response;
        if (allRecords.length) {
          noOfPages -= 1;
        } else {
          noOfPages = Math.ceil(remaining / pageCount);
        }
        // set cursor for additional calls to the length of results to not get any more
        cursor += results.length;

        // update allRecords with new results
        allRecords = [...allRecords, ...results];

        // see how many pages left we have
        return Boolean(noOfPages);
      } catch (err) {
        console.log("error: ", err);
      }
    },
    (hasMorePages, cb2) => {
      cb2(null, hasMorePages);
    }
  );
9 Likes

This is great!

Will need to do some testing, but could this be run from the Bubble Backend to have a complete list of Data returned and then sent to another service?

Seems likely! I’ve only used this externally to pull in data from an app, but my guess is you could trigger a workflow that calls a plugin which runs this code from the server side

1 Like

Hi
how can i use this code to integrate with powerbi?
how do you advise me to do

Tanks

1 Like

If you’re trying to pull data from your Bubble app to PowerBI, you’d first run that code to get all objects into JSON, and then run a converter to get the data into whatever format PowerBI accepts.

For example, here’s an npm package that converts JSON to CSV, similar packages exist for XML, other formats. https://www.npmjs.com/package/json2csv

1 Like

My friend thanks for your reply.

With your first post I understood your logic, and managed to create a script in the M language that pulls the data from the bubble into my power bi data flow.

Thank you very much

2 Likes

great to hear!

Hi Fernando, I’m also interested in how to pull data from Bubble app to PowerBi.
I’m concerned how it will perform for datasets with more than 100K rows.
How can we contact you for understand your solution? if possible, can you please share more details of your solution.

Hi @brentsum

This looks awesome. I have no idea how to use it though. :disappointed: