Hey everyone. I am currently struggling a bit conceptually (and indeed practically) with implementing a way for my app to retrieve multiple images from an external API (in this case, an S3 bucket) and display them to the users.
About my app:
Think AirBnB. My users upload between 5 and 10 images to showcase each of their listed properties.
These images are uploaded directly to my own personal S3 bucket - This is already implemented through this fantastic Dropzone plugin.
I now need to retrieve these images again in order to display them to the users.
My current solution:
- Each User has one or many Listing objects associated with it.
- Each Listing object has a âimage urlsâ field that holds a list of the paths of its images in the s3 bucket.
- When the images are required, each image is retrieved separately by making an external API call to my API Gateway endpoint with the âimage urlâ that is specified for that image in that DB.
Though this works, the loading times are quite slow and it also means I will have to do an API call in every instance that a property listing image needs to be shown.
When it comes to the home page, where I need to showcase about 16 listings at any given time, this will mean I will have to make about 128 external API calls (16 listings x 8 images on average per listing) before the page is fully loaded. I have not implemented this yet but it sounds dramatically inefficient.
My questions:
-
First of all, am I even doing the right thing by storing these images in my own S3 bucket? I originally went down this path because when uploading these images directly to bubble my loading times seemed pretty slow, and also I was made aware that bubble storage can be quite costly when your app scales up. However, after going through the whole process of learning how to use AWS and setting this all up, it doesnât seem to give much significant speed improvement, and when I retrieve images from S3, they are uploaded to the bubble file manager anyways (see next question).
-
When I do retrieve the images from S3, they appear on my bubble file manager, which means they are getting uploaded into my bubble storage anyway, which was exactly what I was trying to avoid in the first place. Is there any way to avoid this and just directly access my files from S3?
-
With my current implementation, as described above, I am making an external API call for each of the individual images for each of the individual listings. Would it be better for me to try and retrieve all the images for a given listing in one API call, using some sort of Multipart/form-data payload to get them all at one time? And if so, is it even possible to process such an API call from bubbleâs perspective, i.e., can bubble take in multipart/form-data and separate the individual images into a list of images to be displayed for the listing?
Final notes
I think these are my only questions for now, but I will keep updating as I go along.
If anyone would like any more details of my implementation in either my bubble app or in AWS I am happy to share this.
This is my first bubble forum post, so please forgive me if anything is unclear, I am still learning a lot!
Thanks!!
Not sure on your database structure but if keep with the Airbnb theme, you would probably want to have a Properties thing in your database. This should have a list of texts which are the S3 URLs,
Then when you show this on the frontend, where you want the images to appear, you are using a repeating group with Properties > Images (the list of texts which is the S3 URLs). You then display the image as current cells properties images in that repeating group. This completely bypasses Bubbleâs storage and wonât have them uploading to Bubble.
If you have some caching settings on the S3 storage, it will load them the first time someone loads the page but if they return to the page later, they wonât need to load again as they will be cached.
Hope that helps give you a rabbit hole to go down.
2 Likes
I think the design you want is to have your images Public. And not to âretrieveâ the images into your Bubble app - just directly âloadâ the images from the Bucket.
Then you have an unchanging URL to access the images. Depending on the storage provider there are different ways to make Files (or Bucket) public in your bucket (and there are Bucket level permission and file (object) level permissions).
One way to simplify the design is to set permissions either at a Bucket level or âfolderâ level - rather than try to manage each objectâs permission.
There is information for each storage provider here https://dropzone-demo.bubbleapps.io/version-test#how-do-i-get-a-permanent,-non-expiring-links-to-public-files-in-a-bucket?
2 Likes
Thank you @lindsay_knowcode and @stuart8 for your suggestions. Loading the images directly from the URL was a great solution to this and resulted in a drastic improvement in load times.
I actually ended up using AWS Cloudfront, which kind of combines the suggestion of caching from @stuart8 and that of public access from @lindsay_knowcode, without actually having to make my bucket publicly accessible.
@lindsay_knowcode I now have an additional question for you specifically regarding the dropzone plugin if thatâs okay.
Basically everything is working great now except for one little detail. When I am in the âdropzone file accepted loopâ, I am uploading the image using a custom generated file path and then adding that same file path to my list of urls for the given property. See the image below:
The idea was that this would allow the file to upload, and then add the s3 file path to the list of urls for that property (done with the âMake Changes to Image Holderâ action you can see in the image). This would then cause an update to the repeating group on my page since it uses the Image Holder as input, and thus cause the image to load in.
However, it seems that the âMake Changes to Image Holderâ action that adds the url to my db is running before the image has completed itâs upload, meaning the repeating group updates but the image cannot yet be loaded, so it does not show up until I refresh the page.
Ideally I would want to add the image url to the db only when the image has actually uploaded to s3, but I couldnât find a way in your plugin docs to do a task after each individual file is successfully uploaded, rather only after all files are successfully uploaded.
Any suggestions regarding this?
1 Like
Awesome! Iâd just advise that if you plan to reach any scale, I can recommend Cloudflareâs R2 service over Amazonâs S3. Cloudflare is fully compatible with the S3 API so although I know nothing about this dropzone plugin, I canât see why it wouldnât work the same.
Running an AI image generator SaaS via Bubble, we initially used Amazon and the bill was getting crazy, with R2, there are no egress fees and this means our bill with many TB of data and over 2.5 million images in our database is still less than $50 per month.
Of course having so much data then goes into a whole other conversation of database structure and optimizing for Bubble pricing but I donât want to distract you further now! Good luck!
2 Likes
I know exactly what you mean. What I do is when I add the image meta data to the DB I set a flag on that image meta data like âis uploaded to S3 = Noâ and save the UUID with that metatdata . Then implement a workflow action something like this in the Workflow âAll uploads are completeâ and set the âis uploaded to S3 = Yesâ.
Then use this âis uploaded to S3â flag to signal on the UI that the upload is not complete. I also use this technique for post-processing you want to do after upload eg transcription.
and nice tip @stuart8 
Okay great, I will check it out. Thank you so much for the help!
Ah okay, that sounds like a good workaround.
When you say you âadd a flagâ, do you just mean you add a new yes/no field to the data type that holds your images? Or is there some other meaning of flag that I am missing?
And when do you reset this flag if you anticipate multiple uploads on the same instance of the dropzone element? At the start of the Accepted File Loop?
Exactly - sorry for using jargon.
Re "when do you reset this flag " - that yes/no flag is set to mean âthis file is completely uploadedâ on the event âall files uploadedâ event.
1 Like
@peterjbrink Iâve added an event so that you can âknowâ when each file has completed upload. Rather than âall files uploadedâ. Iâve wanted this personally for a long time. In the example I am just using toaster notifications so the events are visible - but you could implement changing the status of the uploaded files. Useful if your users are uploading a lot of files.
2 Likes
Apologies for the late response here @lindsay_knowcode. I moved on to some other parts of my application and have only circled back to this now.
This is such a great addition to the plugin - it has added the last piece to the puzzle that I have been working on regarding these image uploads. Itâs amazing to see your dedication to keeping the plugin up to date and implementing new features for its users.
Thank you so much again for your help in all of this, it has been great to interact with someone so knowledgable and helpful.
1 Like
