I'm getting feedback on what developers use object storage for and collecting concerns and potential improvements.
Almost every cloud vendor offers S3 object storage - AWS, Google, Azure, DigitalOcean, Linode, etc.
There are even startups such as BackBlaze and Wasabi that offer it much cheaper ($.005 and $.0059 respectively per GB at the time of this post.
There is even a decentralized S3 compatible object storage Tardigrade.
What object storage do you use?
Is it with the same cloud provider where you host your compute and other services?
If you use multiple providers, what is your reasoning for doing so?
What do you mainly store in S3? Blobs, logs, images, video, etc
What is your access patterns? e.g. write once read rarely (backups), write once read many (static website), etc
What is your biggest cost? Your biggest cost might not be storage but API requests (e.g. AWS is very expensive).
What is your biggest concern? Cost, performance, etc
Pros/Cons?
What would you like to see changed?
I appreciate your input!
I try to use a CDN to solve any scaling issue I can -- vs. databases and memcached layers.
We use BunnyCDN and Backblaze B2.
These both a surprisingly fast TTL that lets you use your cheap CDN a lot more like a slow database than a fast CDN.
They're cheap, fast, and awesome. And I can remember each bit of downtime they've had and what caused it for the past 2 years.
So, we store objects in both to give us a failover option (that we can enable via DNS very quickly).
I store images, JSON docs, and a small amount of video.
These are not great solutions for video though (you have to build a lot of tooling on top of them).
There's two reasons why you'd use AWS or Google instead of these:
You care a lot about nine nines of availability. Neither of these are perfect, but they will have better uptime than BunnyCDN or Backblaze B2.
Your objects would do a lot of ingress or egress from other instances in AWS or GC. The total cost might be lower for you if you have to connect to a lot of instances in either of these cloud services.
No one is measuring storage costs. The costs are all transfer for these, and if you don't have enough transfer to track, then you're never really going to have to worry about the cost of these services.
Transfer to odd regions (China, Africa) end up costing more than they should in all providers.
DigitalOcean spaces isn't great. I don't like Wasabi's business model, although it worked fine when I tried it, they obviously price below cost.
Thanks for the reply! I thought DO Spaces was a little slow as well. What don't you like about Wasabi's business model?
Wasabi appears to be trading venture capital for short-term user growth by providing an s3-compatible service below cost. And they have weird "It's unlimited but you can't do X" restrictions. Not super trustworthy.
I completely agree. I almost consider it false advertising. They said unlimited download but their terms say otherwise. Either allow unlimited downloads or not.
1- S3.
2- Yes
3- I don't, but I would due to transfer costs that are insanely high at AWS. I'd go to DigitalOcean, which is far cheaper.
4- Audio, video, images, some JS, and CSS.
5- Write once per deploy, read a lot - static website
6- Transfer costs
7- Latency/BW is a concern, as it affects the TTFB of my page.
Thanks for the feedback. So you are thinking about switching to DigitalOcean? How much raw data do you store?
I am seriously considering moving to DO. They are far simpler to use, have predictable costs, and are much cheaper. I'll just wait for the AWS credits expire and that's what I'll do.
Not much. A couple hundred GB. Most of the costs are in the transfer.
Several different uses for different projects.
Project 1 .
User avatar images, user asset images, other images uploaded to application. Access pattern. Write once, read thousands of times.
Project 2.
Document storage. Dropbox is often blocked by corporate firewalls, so we utilize s3 to share marketing materials, presentations etc.. as part of marketing and sales processes. Write once, read thousands of times.
Project 3.
Long-term DB backup storage. Write once., never or rarely read.
In additional to AWS, I've also used the GCP equivalent, but only for DB backups. AWS is generally the default for me because the libraries are already mature and I'm familiar with them.
Thanks for the feedback!
Hi Nick,
I use AWS and Digital Ocean completely to run my web Scraping API - www.scrapingproject.com. AWS Lambda, S3, etc.
I hope it helps.
Hey EVR,
I've noticed you analyze your logs. Do you Athena for that?
If so, would you mind providing me feedback on this website? We are building something that could be helpful in your use case, especially for saving some bucks.
https://www.storagequery.com/
Thanks for the feedback!