Secrets management for services is an awful experience to setup
First time poster, long time lurker, sorry for any noob mistakes.
I'm a fullstack engineer with around a decade and a half of experience from startups to big tech. But each time I start a personal side project, I get stuck when thinking of the best way of managing secrets.
I start off by thinking about how I would deploy the project. I vaguely know where I would host them (Front-end goes to Netlify usually, but backend varies, usually DigitalOcean). Right after that, I start thinking about the database, and that leads me to think about secrets. How would I store them?
My usual choices are:
- A cloud provider's service (AWS has Secrets Manager, Parameter Store, and you can fashion your own using KMS keys)
- The CICD usually has a secrets manager
- Use .env and then store a .env file for each stage in your password manager, copy / pasting them after deployment (I opt for this in the start a lot)
- Use a dedicated service like Hashicorp Vault
I'm sure there are more, but these are the general categories in my mind. Let me know what I missed.
This time I thought 'why' I struggle with secrets management on each project? Each time I usually start off with .env, and then find a proper way of deploying it and so on. The best experience I've had so far where I wasn't thinking about it was with fly.io's CLI where it has first class support for secrets.
I started thinking that maybe there's a tiny space for a service just for secrets, and one that's not complicated and works across cloud platforms and front end and backend and CI/CD pipelines. One that works for indies and small teams.
Here are some of the high-level tenets of my approach:
- Secrets should not be standalone, there should be a way of storing metadata (like service, environment, regions, etc.) for each secret
- Changing secrets should allow for side-effects, like Webhooks to restart your service, slack messages, etc.
- Secrets should be end-to-end encrypted, and clients will only get secrets that are within their scopes
I have a solution in mind, and have started developing it. Before I go too far into it, I wanted to check if some of you shared the annoyance (or had more points to add), or am I just over thinking things?
I agree, secrets management is painful. Very painful. I've even googled for solutions. What makes it worse is that secrets are one of the only things that changes as you move code to production. So, you can't know ahead of time that your secrets are "correct" and have a test that validates that.
Although, it becomes less annoying as a project becomes more developed. You change secrets really infrequently once your product is more matured. Regardless, in my head it always feels "duct-taped" and that bugs me.
Indeed, once you've set it up, it gets much less painful. However, the overhead is usually not gone. In order to add a secret, you first update your .env file. For deploying it, you place it in your CICD pipeline, build, ensure staging works (if you have one), update the secret for production, build, ensure that works.
It's fine as long as your project isn't using more resources as time goes on, but there's always going to be something that keeps making you modify (usually just add to) the existing set of secrets.
We've been quietly building https://github.com/dotenv-org/dotenv-vault to solve these problems - from the same people that pioneered dotenv.
I've been building envsecrets.com for a couple of months. It is an open-source end-to-end encrypted tool to manage your environment secrets. It essentially wages a war on your .env files and supercharges your secrets management.
I've love for you to create a free-forever account and take it for a spin. Let me know how you feel about it and ping me in case you need any help! ❤️
I made a self-host project allowing to manage secrets easily for many projects for the reasons you mentions in your post, see https://github.com/martinlevesque/secretsfly
Personally I just encrypt env secrets using a node-package called secure-env. It uses a single encryption key to decrypt the .env file at runtime. This way i'm able to actually push all env files directly to github with the repo. At runtime I just specify the environment. You can also have a separate key for each environment depending on your team structure. Any environment then only needs the encryption key to load in all the env variables / secrets.
I like it because i'm less likely to forget an env variable. It's pushed up with the rest of the code. It 's possible to forget to add the same key to environments like production though. Now that i'm thinking about it, I could have my preprocessor compare keys in all env files to ensure parity before I can push the code up.
This may not fit your need though as it would require a new deployment to take affect, rather than just a webhook to restart. Not too much of an issue though if you're using CI/CD.
This is the Rails approach also (it's built-in) and works very well.
What do you do when you need to maintain different values for the same environment variable across different environments? Do you preface them STAGING_, PRODUCTION_, etc?
I create a different env file for each environment. Then the file name that matches the environment passed in at run time will be used as the env file.
You could also create a default file with values that will be common across all environments. Then each environment specific value could simply be used as an override or addition to that file.
Definitely not overthinking things, I currently work as a developer for a digital consultancy, so always on different projects. It's a real hassle managing secrets, and have searched for solutions in the past.
Did you find something that fit your needs? If not, what was the biggest thing that wasn’t a fit?
I did, but personally its not currently enough of a problem for me to actually use one of the solutions available like Hashicorp Vault etc.
Hey @mutahhir,
Yeah, I think this is a common problem most devs face.
I've used doppler in certain projects and gets the work done.
Ooh! Thanks for the comment. Yeah, Doppler seems to be pretty close to what I had in mind. I’ll check it out!
@sabirahmed follow up question: is there anything that you don’t like with Doppler?
@mutahhir: Honestly I've not used doppler to such a large extent, just the basic stuff. I'm sure it can do a lot of things, but it was mostly for secret management. Since I work with a small team size it wasn't such a big issue.