I have created a photo sharing app. I basically offer to users the possibility to create and reference photo albums, sharing it to other users (for sale or donation)
I use AWS S3 dropzone plugin to upload and retrieve images, which is very efficient.
When users upload pictures (lets say 50), the 50 pictures get uploaded in my S3 bucket one by one.
Each time a picture is successfully uploaded, I send an API call to my personnal Imgix account, to create and save a thumbnail version of this pic.
Then, for each “photo album” created, I have a line in my database being a list of images (in this case 50 images). Those images are all the thumbnails, used to display one the album page instead of the actual full HD pic that would take too much resources to load on every page visit.
My issue is that, for some reason, some thumbnails dont get saved to the album’s “thumbnails” list. It end ups with a list of 46/47 images. The missing ones are random position, its not recurring on a specific or extreme position such as first or last.
I first though that this could come from the API call returning an error, but the needed images actually get saved to my bubble files manager, proving that the image was actually returned and generated.
I tried using front end worklow, and programming backend API workflow, but same thing happens. Here is how its saved :
Do you have any idea where this could come from ? And how to fix it ? It is pretty annoying as almost all albums created have 3 or 4 thumbnails “unsaved” creating frustration for users.
You problably have a race issue where too many backend WF trigger at the same time (or almost).
I’m not sure exactly how you have set your process. Can you share the whole process steps ?
Is it possible for a user to drop multiple file at once? if yes, this mean your loop will trigger on all files in the dropzone. Instead of using loop of the plugin, I think you shoul schedule on a list with 2-3 second delay between them (but because you are calling an API that could have different processing time… you may increase this delay)
Yes users basically drop as many files as they want. Then the pluging uploads them one by one to my S3 bucket after checking if files fit restrictions I set up.
There is also an “accepted files loop” that I use to save other elements to database (such as file name or file bites size) and works perfectly. I can’t use this loop to trigger my “thumbnail” workflow as I need the file to be already uploaded to my S3 bucket.
I tried scheduling the call on a list with 2 sec delay. It works fine for an already uploaded list of images. In my case I want the thumbnails to be generated during the upload process to reduce waiting time.
Adding a process that takes 2 sec per image on an upload of 500 images makes my users wait 16 minutes more just for thumbnails.
Maybe you already have this, but what if you create an image db and store each of them in a different thing and store the thumbnail in this db thing with the original file? At this moment, no race issue and you can use search to retrieve them (with another field that link them all together).
That could work yes, I will try this and see how it goes ! But this would mean a db restructuration so I am not sure if all already existing albums could still be displayed properly …
At the very end of the upload, (lets say when 50 photos are all uploaded), I schedule worflow on a list, the list being : List of files uploaded filtered : this file name is not in thumbnails list: each items file name. I add 1 sec delay to avoid the “race issue”.
This way I can schedule the API on a list of 5 to 10 files max and reduce wait from users. This completely avoids having missing thumbnails. I added a custom state for the page “importing pictures” that gets changed when all thumbails have been generated to avoid user running actions such as saving album before it being all generated.
