Amazon S3, Vimeo for Storage

[DiggDang]

Hi Guys… at the risk of looking silly for asking such basic questions, my frustration level are outweighing my ego, so here I am… hahaha.

I am trying to get my app to allow a user to send a video message to other users that the receiver should be able to view on the app too, like a WhatsApp messenger. I am using the File Uploader for the Sender to achieve this. When a sender creates a video, I see that it uploads to Bubble’s Data and gives an S3 URL. I have read that Bubble storage will quickly run out and they will then threaten to take my first-born, so I have signed up with Amazon S3 AND a Vimeo Plan. I will go with whichever is fastest, most flexible and hopefully keeping in mind that I don’t have an endless budget.

1. Could someone who potentially has done something like this before, advise me if I should do Amazon S3 or Vimeo? I have read that going the S3 route would lead to long delays in playback, and possibly not playing at all. I know that .mov files (recorded on iPhone) don’t work very well, but also read that videos can be converted on the fly, in the process??? Vimeo would look like a good option if doing all that S3 stuff is difficult, but would this get stupid expensive? In both cases, I feel like upload times would be crazy as well.

2. Once I decide on who to go with, I’m then stuck in trying to understand something crazy basic… how to get the uploaded video to S3 and/or Vimeo, which will then likely give me a link to the video that I can then send to the receiving user of my app. It seems that all of the documentation I’m reading is about what to do with it when it’s in S3/Vimeo, but I see nothing about how to GET it there in the first place.

I’m looking for the Video Message to be quick… as if I was sending a WhatsApp video to a friend and just recording a few seconds of me talking. I whip out my phone, start recording, hit send, it compresses, gets sent, receiver gets it, watches it, done. I know so much goes into that.

If anyone can help, I would be forever grateful.

[adamhholmes]

Video storing and streaming is a complex topic and there are a lot of considerations, based on factors such as the volume of video uploads, the amount of storage required, who will be viewing the videos, who will be uploading videos, transcoding, the amount of views etc. All these things will affect the costs, and therefore every use case will have different requirements.

Vimeo charges a fixed fee, and handles everything for you (transcoding, compression, streaming, bandwidth, storage etc.) - so you always know exactly what you’ll be paying, and you’ll know that your videos will work for everyone.

With AWS you have to set up and handle all those things yourself through their services (which is much more complex to set up).

AWS charges you only for what you use, and their storage is very cheap, but you also have to pay for uploads, transcoding, and data transfers. So if your overall volume of video uploads and views is low, then with AWS you’ll pay very little (less than you’d pay for Vimeo) but if your volume is high AWS can get expensive (especially for transcoding), very quickly. If you don’t require transcoding then the costs are fairly low, but you may run into compatibility and viewing issues with your videos and users.

Helpfully, AWS has a pretty good calculator on their site you can use to estimate the costs of using their services - just make sure you factor in everything (it takes a bit of time to read all their documentation so you’ll know exactly which services you’ll need).

Personally, I’d suggest only using AWS if either you’re in complete control of the number of uploads per month (so you can calculate the costs accurately) OR you’re charging your users directly for their own storage or uploads, so you can be sure you’ll be able to cover the cost of that from the revenue. Otherwise, if you start getting thousands of video uploads per month the bill can easily start to get very high (especially if you’re using their transcoding services).

If you don’t need transcoding and are only paying for storage and data transfer then the costs will be much lower.

Some other alternatives include Ziggeo and Wistia, which offer various pricing options whist handling all of the technical side for you.

In terms of how to get your videos into the various platforms - there’s a Bubble plugin for uploading to AWS. If you’re using Vimeo or Wistia you’ll need to look at setting up your own API calls using the Bubble API connector.

For Ziggeo there’s also a Bubble plugin that lets you upload to you Ziggeo account.

So, my advice would be to look into the various options, estimate your usage, and try to calculate the costs and work from there. The final decision will really depend on your exact use case and what you need/what you don’t need etc.

[DiggDang]

Thanks, Adam! You helped me in another post, so I appreciate you helping again! This is a very detailed answer and I’m so grateful for it, and it helps unmuddy my mind on this.

I think I’m settling on Vimeo to do it all for me and keep it simple. If I may pick your brain for a bit more:

• I installed the plugin for Vimeo.
• I then went to Vimeo’s Developer page to “set up an App” (is this what I’m supposed to do??? Am I barking up the wrong tree with this???)
• On that Developer page, I put the Client Identifier (in Vimeo) into the Vimeo Bubble Plugin’s App ID Key. I also put the Client Secrets string (in Vimeo) into Vimeo Bubble Plugin’s App Secret Key.

At this point, I have no idea if what I’m doing is remotely correct.

Then, when someone uploads a file, I’m trying to set up a workflow item that when the send button is pressed, that it checks if it is a video file, and then does a “Vimeo - Upload a Video from Link”, but then it asks for a “(param.)link”…

At this point, I’m wondering “Isn’t this supposed to be no-code?”.

Could I trouble you for some guidance here? Thanks, Adam… if you’re ever near Toronto, Canada, beers are on my tab!

[adamhholmes]

No problem, I’ll try and help as best I can…

Uploading to Vimeo is not the easiest thing to set up, but once it’s set up it works well. Here’s a brief outline of the way I do it…

There are 3 ways you can upload files to Vimeo - perhaps the simplest way is to first upload the video file to another destination, then ‘pull’ that file from that destination to Vimeo. This method takes the least amount of custom coding to get working, and is very simple and quick. Personally this is the method I use for my current apps that are linked to Vimeo (it was the method both Vimeo and Bubble suggested to me when I contacted them to ask about the various methods some time ago).

• I installed the plugin for Vimeo.

I assume you mean the Vimeo plugin by Coaching No Code Apps

I haven’t used this plugin personally so can’t give too much guidance on using it, but having looked at it’s API I know it also uses the ‘Pull’ method to upload videos to Vimeo.

• I then went to Vimeo’s Developer page to “set up an App” (is this what I’m supposed to do??? Am I barking up the wrong tree with this???)

Yes, this is correct. Before you can access the Vimeo API you need to register your app, and request ‘Upload’ scope, meaning your app has permission to allow uploads via API.

• On that Developer page, I put the Client Identifier (in Vimeo) into the Vimeo Bubble Plugin’s App ID Key. I also put the Client Secrets string (in Vimeo) into Vimeo Bubble Plugin’s App Secret Key.

Again, I haven’t used that plugin, but from what you’ve described that sounds correct to me.

Then, when someone uploads a file, I’m trying to set up a workflow item that when the send button is pressed, that it checks if it is a video file, and then does a “Vimeo - Upload a Video from Link”, but then it asks for a “(param.)link”…

As far as setting up your workflows and upload process, there are various ways to go about it. It’s fairly involved, but I’d suggest something like this:

• As this plugin uses the ‘Pull’ method to upload to Vimeo, you’ll first need to upload the Video file somewhere. This can either be to your App’s Bubble storage, an AWS S3 bucket, or any other storage.

• But let’s assume you upload to your App’s Bubble storage, as this will be the simplest way to do it.

• So add a file uploader element to you page. From memory, I’m not sure if the standard Bubble uploader allows you to restrict file types - but if not there are some third party plugin file uploaders that do, so it would be best to only allow video files to be uploaded. If you use AWS and the Zeroqode AWS uploader plugin (which I do) you can restrict the file types on that.

• You can add some kind of progress bar or percentage display to you app to show your user’s the file is being uploaded, then let them know once it’s complete.

• Once the file upload is complete you’ll need to trigger the plugin to ‘Pull’ the file from its uploaded location into Vimeo (again I haven’t used this plugin, but presumably there’s an Action you can use to do it) - you’ll have to point it to the URL of the uploaded file to ‘pull’ into your Vimeo account.

• Ideally, you’ll then need some way to monitor the processing/transcoding of the file in Vimeo before sending your message - unfortunately Vimeo don’t have any webhooks to tell when it’s complete so you’ll need to monitor it yourself to check the transcode status. It looks like the Vimeo plugin has plenty of data API calls, so you can use the ‘Get Specific Video Call’ to check the Transcode Status of the video, every few seconds until the transcode status is complete - You’ll need to use a recursive backend workflow for this.

• You may, or may not, want to create a ‘Thing’ in your database for the uploaded video (I’d recommend that you do) so as soon as the video is uploaded (you don’t need to wait until it’s transcode status is Complete), create a new Thing (Video) in your database, and store all the relevant info such as the user who uploaded it, the message it’s attached to etc. (you can even store the Vimeo API data). You’ll also need to attach it to the message.

• Once the transcoding is complete you can send the message to the recipient (you could do this before the transcoding is complete, but they may have to wait before they can view it depending on how soon after receiving it they try to view it).

• If you uploaded the original video file to your Bubble storage, then you’ll probably want to delete the original uploaded file at this point to avoid using all your storage space - so run a workflow to delete the file. (file transfer between the original upload location and Vimeo is extremely fast, but it’s still probably safest to wait until the Vimeo transcode status is Complete, or longer, before deleting it, just to make sure the file has been transferred ok)

As I said, the ‘Pull’ method is one of the 3 ways you can upload to Vimeo, it’s what the plugin uses, and what I use, but there are 2 other ways that you might want to consider instead. You’ll need to create your own API calls for those, and there is more custom coding involved, so for simplicity, I’d suggest sticking with this method, but if you have the time/money/inclination to investigate further then the TUS approach is also worth looking into (I’ll probably be looking to switch to this method myself for my video app at some point, but it’s more involved - so for now I’m sticking with the pull approach as it works perfectly for our app once it’s all set up).

[DiggDang]

OMG… hahaha… Wish me luck! I’ll need some time to absorb this and play with it. I’ll let you know how it goes. Thanks for the explanation. Be in touch.

[DiggDang]

Hi Adam… I now have the API going, but something is going wrong. When a video is uploaded on the app, my workflow saves it to Bubble and I have an action that does the API call… but when I check Vimeo, My Videos just show stuck in “Uploading” and just never stops.

Any ideas?

[adamhholmes]

That sounds like it might be a problem with the plugin (as I said I’ve never used it so can’t say whether it works correctly or not)…

I noticed that the plugin doesn’t include a parameter for file size in the Upload Video API call (which I think is needed for Vimeo to determine when the upload is complete - although I’m not 100% certain on that).

Probably best to contact the plugin creator to let them know there’s a problem.

Alternatively you could look at setting up your own API call to do it, being sure to send the URL and file size, (The Vimeo Plugin is open-source, so you can see the plugin code - so if you’re not sure how to set up the API call you can just copy the settings from the plugin, and be sure to add another parameter for ‘file size’).

It’s actually quite simple once you know what you’re doing, so if you need any guidance just ask.

[DiggDang]

Sorry, I guess I wasn’t clear… I got rid of the Vimeo plugin and did it through the API Connector… Are you able to copy the parameters you have used for me so I can see what you have? I did use the approach and file size ones. For the Redirect URL, what would that be?

Thanks again, Adam. I see you trying to help on other posts for similar issues, so you seem like a good guy with a good heart. Thanks for all of your contributions to the forum.

[DiggDang]

Also, the way I have it set up, I think it’s taking it straight from the file upload and not the S3 link in Bubble… how can I tell Vimeo to pull it from the Bubble/S3 location? I have the location link in S3 currently recorded in my database, but thinking about it, the same workflow is asking for the API call, which I don’t think actually references the Bubble/S3 link anywhere…

[adamhholmes]

Maybe if you share some screenshots of your workflows and current API call setup I’ll be able to assist better.

[DiggDang]

Sure… have a look at this:

This is Vimeo not liking what’s uploaded:

Not sure if those keys matter, but in case someone in the future needs to review these screens, I’ve blacked it out.

[adamhholmes]

Yeah, you don’t want to share keys in public forums…

Ok, so you’re using the wrong ‘Approach’ in your API call. (You’re using Post instead of Pull).

Your JSON body should be:

All fields apart from the ‘approach’ should be dynamic, and set in your workflow.

The ‘name’ field can just be set to the uploaded file name (or you can set some other name for it)

The ‘size’’ needs to be the size of the uploaded file in kb (you should be able to access that from the upload element)

And the ‘link’ should be the URL of the uploaded file (again you can access that from the upload element’s value’s URL)

[DiggDang]

Still no luck over here… for some reason, when I set it up like you have, the workflow changes, such that the Vimeo API action in my workflow gets reset and I don’t have the same options.

I’m still getting the videos sent to Vimeo, but it doesn’t actually show the video. Could it be a privacy setting I have? I currently have the videos set to private with URL.

[adamhholmes]

If you’re still using the ‘form-based’ approach (post) then no, your Bubble privacy settings won’t have anything to do with it, as you’ll be uploading the video directly to Vimeo, rather than to your Bubble storage first.

What happens when you make a ‘post’ approach request is a new (empty) Video is created in Vimeo, and then it waits for the file to be uploaded (you need to create an HTML form and upload the file via that). In your case you’re not doing that, so Vimeo will be waiting forever to receive the upload.

You can use the form-based method if you like (it’s not that difficult to get working) but there are a lot of limitations in terms of UI/UX that will require some custom coding to get around. That’s why I prefer the ‘pull’ approach, even if it does require the extra step of uploading somewhere else first.

If you’re using the ‘Pull’ method, then it could be something to do with your privacy settings for the file, but I think that would stop the transfer from happening entirely, so the new video would never be created in Vimeo in the first place.

I can’t tell what the issue is from the screenshots (although it looks like you’re not sending any dynamic data about the video to the API call, which could be the issue), but if you want to share a read-only link to your editor I can take a quick look to see if I can see what the issue may be.

[DiggDang]

Thanks for sticking with me through this, Adam. I think I’m ready to give up on this. Before I do, the whole reason I was using Vimeo in the first place was that I wanted to encode it for playback on the receiver’s end of a video message. I found a way to play the S3 link in a player, which seems like a much easier solution. However, right now, it’s hooked up to Bubble’s file storage with S3, so I’m looking into the costs of using my own S3 and perhaps solving one problem with another, when I can’t get my files to S3! Hahaha… Anyway, I know you mentioned that it gets costly, but when I see the prices for S3 storage, it actually looks very affordable ($0.023/GB). I don’t think I require the encoding anymore, since using the New Video Player Plugin seems to play any video format just fine.

Thanks again for sticking with me through this, and I wish I had a better conclusion for you. I wish you the best.

[richard10]

I can tell you having built our storage requirements into s3 that it is non trivial to set up in bubble.

We took the following approach

Build some test pages with everything you need on the page to do all the securitisation of the Amazon request

Move as much of that off page as possible so you can set up some really simple endpoints to pass things to in bubble “pass a file name into the delete endpoint”

Handle all your tough stuff off page, but you have to learn how Amazon does things first. Their documentation is thorough. But their implementation is complex.

You might want to get someone to help you with a small plugin for the security bits so that you have actions in bubble to encrypt your credential with etc. The strong probability is that none of the available plugins will meet your needs well.

I am pretty good at this, but still I spent a couple of weeks kissing frogs before I figured it out. But the satisfaction of doing it is a big bonus!

Richard

[DiggDang]

Thanks for this @richard10, and thanks again @adamhholmes for sticking with me through the above. I was able to set up AWS S3 and it is working very nicely now. I used the AWS Uploader Plugin which had pretty clear documentation and my videos now write to S3. I also noticed on upload that it compresses videos and images, and then uploads pretty quickly. I then use the Video Player to play the videos, and it will play pretty much any file format (.mov, .mp4). All in all, I’m pretty happy with this setup. Thanks again!

[richard10]

Oh super news! Good luck!

[aneeshsinha01]

can you guide me as to how to setup using amazon s3,basically just how you did it would be great ,also does s3 automatically compress the video size or do we need to do some changes for that?

[DiggDang]

Hi @aneeshsinha01 … I initially thought S3 was compressing things, but it turns out that by default, in what I’m doing (S3 - Simple Storage Solutions), it’s not actually compressing, I don’t think… Amazon is kinda a mess of many amazing services and I find they do a lousy job in showing them all. I even had a call with them and it was no better… they tried and I give them credit for that… but they have 3-4-5 different variations of storage that they explained and they all do slightly different things. Anyway, I’ll let you get into that. S3 on it’s own looks like it is fairly reasonably priced for what I’m using it for, so I’m continuing with it for now. However, you do pay for calls to the info, and that is what I’m surprised at, rather than just storage. It may be something to think about as I did not know this beforehand. They have services though, that you can take what you have and compress them later, but I will deal with that when I have that need.

As for how to integrate it, I don’t remember exact steps as every piece of my app has been an adventure. What I DO remember is that I used AWS File Uploader by Zeroqode… It is a paid plugin, I believe, but the instructions they provide were easy to follow and is a file uploader that uploads directly to S3. I remember wishing all plugins were that easy, as many tend to be ones you have to figure out on your own.

I hope that helps get you on the right path. Feel free to ask questions and I’ll do my best to answer.