Cant download full database with GET

Need help Database
1

I am using this set of features to download my database to a local file (and converting to .CSV)

on the 50001 attempt it timesout (I have another 15K rows to go). I have no explanation. Obviously the code works to gather data, it just stops working. The test I get in the request body is:

The origin web server timed out responding to this request.

\n \n \n
\n

What can I do?

\n

If you’re a visitor of this website:

\n

Please try again in a few minutes.

\n\n

If you’re the owner of this website:

\n

The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.

along with more html.

I don’t see any documentation of a 50,000 entry limit for downloading via the API interface.

any ideas would b e helpful.

2

@hirscr

Recommend to send a ticket. Please share the results :slight_smile:

Very interested in knowing about possible API limits or rules

3

Here is what i got back. So it seems i need to query the database

———
Hello Robert,

Thanks for reaching out about this issue with your app’s Data API. I can see how this may be a bit confusing and frustrating! There is a Bubble limitation here causing the error that I can provide more details about. Our team, however, is unable to run or troubleshoot custom code on behalf of users.

There is a limitation in Bubble’s codebase with how items past the 50,000th item of a specific type are indexed and returned. This means, running a “straight” query that includes the 50,000th item or greater will produce the error. We understand that many users have a lot of data and, in your case, you’ll need to access the remaining 15,000 items.

Our engineering team reports that a workaround is to ascertain the created or modified date of the 50,000th item then using the date to query for items with a date greater than the 50,000th item.

Again, I apologize for that you ran into this error. If you have any questions or run into any other issues, feel free to reach back out to our team.

4

Interesting. Thank you for sharing.

5

I believe the instructions for downloading data are both inadequate and wrong.

Its says that you need to URL encode the constraints. But when I did that it complained about the parameters not being json encoded when I used json.dumps as the constraint, it worked perfectly. Some working examples in python would be really helpful.

My working code looks like this:

url='https://yourapp.bubbleapps.io/api/1.1/obj/yourThing'

    constraints = [
        {
            'key': 'Created Date',
            'constraint_type': 'greater than',
            'value': back
        }
        ,
        {
            'key': 'Created Date',
            'constraint_type': 'less than',
            'value': front
        }
    ]

    params = {'cursor': cursor, 'limit': 100, 'constraints' :json.dumps(constraints) }
    resp = requests.get(url, params)