What would your architectural design approach be for a high throughput API for media files.
If you were to design a web API backend to server photos, possibly videos as well, what would your approach be? What would you use to build a service like this?
So imagine this service will require to handle a lot of of requests per second and on top of that to serve all those media files. It wouldn't be just a response with some data, but also the media file. I am interested in the technical details of the setup and build approach you will take to design and build a scalable service like this.
For example which web server you would use and why? What language, database, etc. Or if you would use a framework, or maybe a cloud service to combine your API with some storage.
If you decide to build or request as a architecture specification using the Netflix example - you might have your product in few years. People often forget that it took Netlfix years to come to this.
Having said that I absolutely support Netflix architecture or just broadly Microservices (though there are dangers in that too) where those fit for what you are trying to build.
We have chosen to go with Serverless approach - and this is something I'd chose in your case:
AWS Lambdas for API handling and S3 for content store.
Any language that AWS Lambda runtime supports (Node.js, Python, Java)
This should give you fast enough development cycle and enough of scalability. Even though it is 'built' on AWS the dependency may be very low (checkout https://serverless.com/). But be sure this setup will go up in price as the size of content store goes up, as ell as the amount of calls (which should be fine if charge per size)
If you just need to mock something up and are willing to
trow awayrewrite later just make quick 'monolith' app using Rails like frameworks.But best is to find a Tech co-founder ;)
Well... for reference here's how Netflix does it
https://youtu.be/CZ3wIuvmHeM
I was pretty in awe at this. Fun to watch.
Hi James. Thank you for sharing this. Awesome indeed!
For videos - do you mean streaming, or "just" downloading...?
Anyway, why do you want to build it yourself? Is there any reason not to use something like AWS?
Downloading, not streaming. I don't want to build the infrastructure myself :) By AWS you mean using their S3 to store the uploads there? Yeah this is an option, why not. But imagine the API because it's purpose will be to serve media, it will provide some other minimal options as well. Though limited, but imagine uploading an image, and then be able to create a thumbnail version by specifying the dimensions in the request. Or maybe request some metadata of the photo etc..
Of course, for a service like this if the requests are not crazy high it wouldn't be a problem. But imagine having to serve a lot of requests and for each request their might be a some image processing involved.
I think that without an in-depth knowledge of your service we can give you only limited advises. My assumption is that your files will be too big (or there will be too many of them) to use a faster block storage (like Amazon EBS, DigitalOcean Block Storage) and your service is not too big to use a custom infrastructure. The API options will be dynamic - e.g. you don't know the requested thumbnail size - so you can't preprocess your files. So I think your only option is to use Amazon S3, DigitalOcean Spaces, Backblaze B2 or other object storage solution and to process the files on the fly. User requests a thumbnail, your app will download the file, process it, upload the result back and respond with a link to the final image. You can use an object storage provider and a VPS or you can build it using the Amazon AWS stack - S3, API Gateway and Lambda