AWS bucket (non bubble bucket) upload binaries

[rowntreerob]

Addresses gaps in bubble’s general capability for cloud file utilities that are apparent in issues such as [SOLVED] How to upload an image using POST API:

Critique of current bubble process ( binary file xfer )

File transfers require targeting and attachment of binaries directly in the body of FETCH POST operation that any browser supports. Any file in the filesys on the client AND any file publicly accessible via a URL transferred to any other API endpoint in a 2 step ( client piece , server-side ) process. Bubble currently does a poor job, attested to by forum activity on topics like Apis and binaries or uploading binaries.

Instead of a plugin, i built my own open components - client & server pieces with eye to HTML5 & node capabities ( streams , files , https ) that are easily adapted to bubble idiom.
Acting as a proxy, the server piece forwards any POSTED binary calling a 2nd POST ( dest = desired API ) with the uploaded request.body set to the request object ( binary file ) provided to the proxy.

Example function ( select a binary img from local fs and upload to a personal AWS S3 bucket ) . Response from server is JSON properties that can be passed into bubble.DB to record state of media files uploaded to AWS. If the AWS bucket is public then the DB url can be constructed from the response json.

I’ve seen zerocode’s plugin and its kinda complex re configuration. Although this approach uses native html tag and a seperate server process, its coded implementation is pretty simple…

select file to upload

post the selected file on http

server-side forwards to AWS Upload
with just small block of code that calls the AWS SDK details below

Once you transfer the client code from ( fiddle to bubble html native tag ) you have…

Running the bubble app with devtools, you have following after an upload to S3 bucket

free code ( fiddle for client, github for server ) . help yourself.

client - copy/paste from jsfiddle to native bubble html widget

server piece on github

General issues is the ease with which binary files can be moved around using mix of current tools ( native html block & node.express server ) permitting easy file handling , transfers among API endpoints.

[gx1994]

Hi, @rowntreerob .
Thanks for sharing a nice idea!
Would this idea work for uploading files to OneDrive?
(I need to attach binaries in the body of the PUT request as in the topic case…)

[rowntreerob]

Yes And… for short answer
not a oneDrv expert but its a REST API expecting a binary stream in http.request.body…

from docs

Focused as we are here on http ( bin file obj in req.body ) , recalling how good html 5 & browser is a streaming said obj over an http(s) connection , leaving bubble aside just for a moment, its trivial process to POST a file to oneDrv. DOM/HTML side selecting the file is straightforward.

Back to bubble where im no expert and where a number of solutions ( alternates ) exist…

The native html option works fine but really kind of subverts the overall bubble approach when you consider the stack & devtools n such. Yes it will work on onedrive just the way it works on AWS. And as this js.fiddle shows for an AWS example, you can abstract out of the cloud storage platform, low-level details, leaving you with a client side service that just works with any cloud bucket.

Bubble ecosystem wise, what they seem to want you todo is eat their dogfood for this. Implement their file upload ( file mgr OR image upload ) to their S3 bucket on the cloud where you are gonna be charged 30 or 50 X for cloud store capacity. From there, using the bubble API infrastructure you can forward the uploaded binary to its finale -oneDrive- destination ( dont forget to delete the file from bubble’s S3.bucket )

these 2 pics show bubble ecosystem view of setup and use of the “proxy, forward the file” focused on bubble tools ( API config, API connector calls from workflow )

in the above example using bubble’s tools, at the end of the workflow, the uploaded files binary would be in the AWS cloud bucket included in the API connector’s parm list. Following a prelim hop to the bubble bucket, an outside call to outside server GETS from Bubble S3 and PUTS to the destination cloud bucket ( my own S3 in the example) . The final bubble.wkflow step is to insert into bubble.DB the link (public.READ.access) so bubble app can GET from oneDrive the upload object. Your implementation would use oneDrive as the destination.

sample server code below:

My wild guess would be that your better remaining in the bubble ecosystem rather than loading up a Native , bubble, html tag with all that javascript as in the OP. Remaining in the system means you have to PUT to AND delete from the built in bubble bucket on S3. It also means you use their toolset to the extent possible. ( widgets for uploads, workflows& api connectors for calls )

The external service is just a proxy built on node streams ( Fetch.GET pipes to Fetch.POST )

The AWS version of service endpt : https://infinite-mountain-20609.herokuapp.com

[gx1994]

Thank you for your kindness at the end of the year.
But, I’m not familiar with Bubble and coding so it’s taking me a while to understand your detailed explanation.
I have a couple of questions.

1. By including binaries directly in the endpoint URL, can I request with them?
   I’m confused because API Connector shouldn’t allow binary streams in the request body.

2. How do I convert the file to binaries?
   From your screen shots, I don’t think you manually converted the file to binaries. Does the file automatically convert to binaries in the process of getting it from S3?

Thanks.

[rowntreerob]

@gx1994

• By including binaries directly in the endpoint URL, can I request with them?
  I’m confused because API Connector shouldn’t allow binary streams in the request body.
• How do I convert the file to binaries?
  From your screen shots, I don’t think you manually converted the file to binaries. Does the file automatically convert to binaries in the process of getting it from S3?

An endpoint ( the path that an api handles is just informational line referring to resource or resources in the cloud ). So , a single endpoint , utilized in a from-to copy process , may have BOTH input side AND output side resources of type File. Look at post url in the connector config noted above …

awsfs/[link]     where link refers to the FILE on bubble's aws S3. 

• the input files place on the web is here (input above)
• destination , per the api docs for cloud store being used , is the aws-bucket or microsoft place

The handler just reads input and writes output by Fetching From the bubble /S3 url where an earlier file-upload placed it. Input process reads the binary from bubble, placing binary bytes available into the request.body of the next step.

you are correct on limitations of api connector, but in this case the endpoint has both input/output information within it. Connectors req.body is null because the endpoint tells the handler where to fetch the file from - from bubble’s S3 link.

Let the internet / browser / javascript do the work for you and the management of conversions is just handled ( it is automagic ) . Step one fetch at (sample code L93-4 ) of the code GETS binary bytes from bubble ( the file after the upload by bubble front-end )

L104 of the code basically says “use a buffered stream for bytes” to place whatever is input from the sourceFile into the request.body for the next steps PUT call to microsoft cloud.

Body.stream means request.body

[gx1994]

Hi. @rowntreerob .
That’s great!
I’m not an expert in bubbles and coding, but your kind explanation helped me understand the API better. I really appreciate your help.
Thank you.