Forum Academy Marketplace Showcase Pricing Features

How to obtain record details from Data API bulk upload operation?

I’ve been able to successfully bulk upload thousands of records containing addresses via the Data API using Python, but occasionally I’ll receive an error message for a given record indicating that there was an issue with the Google Geocoding API:

{"status":"success","id":"1588966554792x231280202360638180"}
{"status":"success","id":"1588966555212x572269708680598600"}
{"status":"success","id":"1588966555438x366702563142093250"}
{"status":"error","message":"Error hitting Google Geocode API: You have exceeded your daily request quota for this API. If you did not set a custom daily request quota, verify your project has an active billing account: http://g.co/dev/maps-no-account"}
{"status":"success","id":"1588966555680x154248749868121540"}
{"status":"success","id":"1588966555906x653839323256888400"}
{"status":"success","id":"1588966556009x682643926351009800"}

Each of the records in my bulk upload POST request includes a card_primary_key value, and I’d like to be able to match that value to the status message and Bubble id value that get returned for each record in the API response, but I’m not sure how to do that.

As far as I can tell, the response object returned by the API does not include any properties that would enable me to retrieve details about the underlying records represented in each response.

Not sure if this is something that can be solved via the Bubble API, or whether this is something I’d need to address in my Python code, but I’ve been researching it and have not found any obvious / straightforward answers. If anyone can point me in the right direction, I’d very much appreciate it.

Not sure if this is the most elegant approach, but I ended up creating a custom Python function to generate the report I needed. Posting the code here in case it proves useful for anyone else.

In the function below, upload_results is the response object from the Bubble Data API.

import requests
import json
import pandas as pd

def make_bulk_upload_report(upload_results):
    """
    Record the API reponse for each individual record uploaded to the Bubble Data API bulk endpoint.
    
    """    
    
    record_keys = [] # Create an empty list to store parsed record keys
    for result in upload_results: # Iterate through each item in the response object from the POST call
        payload = upload_results.request.body # access all data sent via POST request
        parsed_payload = json.loads(json.dumps(payload.decode("utf-8"))) # decode bytes, dump to string, load as JSON string
        parsed_records = [json.loads(parsed_record) for parsed_record in parsed_payload.splitlines()] # separate JSON string by newline and load each newline-delimited record into a list as a new JSON string
        for uploaded_record in parsed_records: # iterate through each item in the list of JSON string records
            record_key = uploaded_record["card_primary_key"] # access the item's primary key and store it as a variable
            record_keys.append(record_key) # append the variable to the list of record keys

    status_reports = [] # Create an empty list to store parsed status reports
    parsed_post_response = json.loads(json.dumps(upload_results.text))  # dump response object as string and load as JSON string
    parsed_responses = [json.loads(parsed_response) for parsed_response in parsed_post_response.splitlines()] # separate JSON string by newline and load each newline-delimited record into a list as a new JSON string 
    for parsed_response in parsed_responses: # iterate through each item in the list of JSON string records
        response_status = parsed_response # store the item's value as a variable
        status_reports.append(response_status) # append variable to the list of status reports

    full_report = dict(zip(record_keys, status_reports)) # combine card keys list and status report list into a dictionary of dictionaries
    for individual_report in full_report.items(): # Iterate through the dictionary of dictionaries
        card_key, upload_status = individual_report # Assign variables names to each element in each dictionary 
        d = {} # Create a new empty dictionary
        d["card_primary_key"] = card_key # Add a new key to the newly created dict and pass in the appropriate element
        d["upload_status"] = upload_status # Add a new key to the newly created dict and pass in the appropriate element
        with open("/Main/Outputs/upload_results_all_{}.txt".format(pd.to_datetime('today').strftime("%Y%m%d")), 'a', encoding="utf-8") as f:
            f.write(str(d) + '\n') # Write each new dictionary to file and separate each by newline
1 Like