ObjectStorage is a mess

In my experience, object storage is the one of the most ad-hoc designed and (un)managed component of such applications. Specially in startups, usually the developers are trusted to make the best decisions in the absence of a foresight. At times, object storage is used a crap-closet to store whatever is generated or uploaded.

To make matters more complicated, as your product grows, you find yourself (or your team) deploying more and more buckets (AWS-S3/EBS) and containers (Azure Blob/File Storage). It is almost inevitable to provision more storage e.g., for new endpoints, new environments, and performance/cost optimization. However, the ad-hoc management makes it difficult to manage data life-cycle, access control and compliance requirements. Other low-level management tasks, such as separation of environments, backup, clone, erase, transfer, analytics etc., are also cumbersome. To make things further complicated, you may also have to keep up with new and upcoming regulations e.g., GDPR, and EBA regulations through single customer views, location affinity, detecting private data, anonymization etc.

I am working on this problem but having trouble figuring out the MVP. If you are in a technical leadership role e.g., product manager/owner, team-lead, solution architect, CTO/CDO/CIO and have similar problems, I would love to have a chat with you to learn if I am making the right thing.