I’ve been able to successfully bulk upload thousands of records containing addresses via the Data API using Python, but occasionally I’ll receive an error message for a given record indicating that there was an issue with the Google Geocoding API:
{"status":"success","id":"1588966554792x231280202360638180"}
{"status":"success","id":"1588966555212x572269708680598600"}
{"status":"success","id":"1588966555438x366702563142093250"}
{"status":"error","message":"Error hitting Google Geocode API: You have exceeded your daily request quota for this API. If you did not set a custom daily request quota, verify your project has an active billing account: http://g.co/dev/maps-no-account"}
{"status":"success","id":"1588966555680x154248749868121540"}
{"status":"success","id":"1588966555906x653839323256888400"}
{"status":"success","id":"1588966556009x682643926351009800"}
Each of the records in my bulk upload POST request includes a card_primary_key value, and I’d like to be able to match that value to the status message and Bubble id value that get returned for each record in the API response, but I’m not sure how to do that.
As far as I can tell, the response object returned by the API does not include any properties that would enable me to retrieve details about the underlying records represented in each response.
Not sure if this is something that can be solved via the Bubble API, or whether this is something I’d need to address in my Python code, but I’ve been researching it and have not found any obvious / straightforward answers. If anyone can point me in the right direction, I’d very much appreciate it.
Not sure if this is the most elegant approach, but I ended up creating a custom Python function to generate the report I needed. Posting the code here in case it proves useful for anyone else.
In the function below, upload_results is the response object from the Bubble Data API.
import requests
import json
import pandas as pd
def make_bulk_upload_report(upload_results):
"""
Record the API reponse for each individual record uploaded to the Bubble Data API bulk endpoint.
"""
record_keys = [] # Create an empty list to store parsed record keys
for result in upload_results: # Iterate through each item in the response object from the POST call
payload = upload_results.request.body # access all data sent via POST request
parsed_payload = json.loads(json.dumps(payload.decode("utf-8"))) # decode bytes, dump to string, load as JSON string
parsed_records = [json.loads(parsed_record) for parsed_record in parsed_payload.splitlines()] # separate JSON string by newline and load each newline-delimited record into a list as a new JSON string
for uploaded_record in parsed_records: # iterate through each item in the list of JSON string records
record_key = uploaded_record["card_primary_key"] # access the item's primary key and store it as a variable
record_keys.append(record_key) # append the variable to the list of record keys
status_reports = [] # Create an empty list to store parsed status reports
parsed_post_response = json.loads(json.dumps(upload_results.text)) # dump response object as string and load as JSON string
parsed_responses = [json.loads(parsed_response) for parsed_response in parsed_post_response.splitlines()] # separate JSON string by newline and load each newline-delimited record into a list as a new JSON string
for parsed_response in parsed_responses: # iterate through each item in the list of JSON string records
response_status = parsed_response # store the item's value as a variable
status_reports.append(response_status) # append variable to the list of status reports
full_report = dict(zip(record_keys, status_reports)) # combine card keys list and status report list into a dictionary of dictionaries
for individual_report in full_report.items(): # Iterate through the dictionary of dictionaries
card_key, upload_status = individual_report # Assign variables names to each element in each dictionary
d = {} # Create a new empty dictionary
d["card_primary_key"] = card_key # Add a new key to the newly created dict and pass in the appropriate element
d["upload_status"] = upload_status # Add a new key to the newly created dict and pass in the appropriate element
with open("/Main/Outputs/upload_results_all_{}.txt".format(pd.to_datetime('today').strftime("%Y%m%d")), 'a', encoding="utf-8") as f:
f.write(str(d) + '\n') # Write each new dictionary to file and separate each by newline
1 Like
