Simple CLI for cloud deployments.
I am not sure if the format of this post is proper or in the right place as I am a first time user of IndieHackers. I want to share a thought I had the other day.
I am a DevOps cloud engineer working with a team of designers, frontend and data engineers. I am seeing that the cloud stuff goes over their heads, and they hate the deployment process. For example; spinning up a group of databases to test an ETL job, deploying a site, or putting some script on a schedule. Even stuff like TerraForm seems hard for them. They use AWS Lambdas for simple setups, but when the codebase gets complex they come running to my team to help them deploy, or create new layers for their Lambdas, or other resources that their Lambdas need. Oh, and no-one I know seems to use Heroku anymore. A few frontend guys use zeit.co, but they still need to create the database elsewhere.
Lately, I am spending time creating and maintaining scripts at work to help developers and designers deploy common stuff quickly. These scripts basically abstract security, scaling, DNS, etc, and gets their resources talking to each other, like a database or a queue talking to a container running JavaScript.
I was surprised to learn how much of a pain this cloud deployment stuff is for the designer/frontend developer group, even for a data engineer. I am wondering if there are others that fit this category.
Launched Lemonvite on Product Hunt today: $5 per event, no ads, no subscription.
If you can identify a few typical cases like (setup jupyter notebook with database access), you might be able to create an abstraction layer on top of, for example, Terraform or Ansible. If, however, would like to provide similar flexibility that the underlying infrastructure allows, then you will end up reimplementing terraform or similar tools itself. At least this was my experience with similar situations. I end up creating a simple web site where people pick form a few common options added one or two parameters then the appropriate terraform script was run and the important state parameters (IP, password, etc) was collected and presented in a table form. Almost zero flexibility but it worked but it was project and company-specific very much.
Its interesting that you mentioned Jupyter. I am working with a friend of mine on a consulting company around AI. I've setup some scripts and an infrastructure that the data scientists use to get different Jupyter/Tensorflow/Database environments running on AWS. Each is client specific, so when they need to create a new environment they do as you said, fill in some parameters and deploy.
The problem with Paas like Heroku is it become expensive quiet quickly.
So I think people find a way to do it themselves or pay someone to do it. Heroku is there to fill the gap in between.
Personally, has a full-stack dev, I don't mind doing a bit of sysadmin or CI/CD (I did it all for Colofon.io) but I do it my way. I cannot wrap my head around Firebase, AWS or Azure has it seems to be a full time job to understand how to use it. And it's prone to changes so if I come back 3 months later and UI is different, I become crazy.
I use a cloud VM with debian and (even as a Windows user) I think it's better if I need to scale or switch. Finding someone who know his way around Debian/Linux is easy. For Azure ? Not so much...
My only nightmare is SSL certificates. I don't know why, it's always a headache for me.
The problem with simple setups like this is that in many cases, if you work with real data science team, they really would like/need to use Spark, BigQuery, DataFlow, etc. Managing such tools by yourself is expensive, so it is better to use IaaS that already have these tools. Then you have to learn about the environment. Heroku and Zenit do not suit these kinds of teams/applications understandably, that is not their focus. So it is always a tradeoff.
Regarding your certificate issues: Have you tried https://letsencrypt.org/ ? They offer free certificates and scripts that do certificate renewal automatically for you.
I was thinking of an easy way for developers/designers to create and destroy short lived services, like testing environments, demos, static sites, or scheduled scripts. Like why build out complex setups for a demo site, POC project, a landing page, or database test. If things go well with the "test/POC" then move it into its own less abstracted setup. I think Heroku was for this purpose, to deploy/test/move. To get something up quickly, and if it gets traction then build it solid elsewhere.
Hey @corpulent
I am the founder of Smoothy.cloud, a startup that exists to solve exactly this problem.
If you could miss a few minutes of your day, I would love to hear you out about the exact problems that your team faces. I love talking about this stuff and maybe we can come up with a solution that can benefit both our companies.
Hope to hear from you soon!
So with Smoothy I have to give access to my AWS account, and Smoothy runs deployments on it on my behalf? Or this part is abstracted and Smoothy controls its own cloud environment?
Actually I dont have a company, I am an employee of one.
Also, the people that I work with, most of them don't even know the technical details of Docker. All they know is common stuff, like databases, programming languages, and the web/mobile frontend stack. Cloud/servers/deployments, is out of their league. So more often than not, I find myself just helping them with the "end result". Like, I need a Jupyter/Tensorflow with DB access, or a PHP runtime with a MySQL behind it, how you get there I dont care, just make this setup and deploy my code. I dont know, maybe its just a few people that I know who are like this.
"So with Smoothy I have to give access to my AWS account, and Smoothy runs deployments on it on my behalf?"
Correct! This is exactly what Smoothy does. It takes all the manual configuration out of your hands, while you retain full control over your deployments.
"The people that I work with, most of them don't even know the technical details of Docker."
This is actually a very common situation in my experience. For this very reason, we abstracted all the Docker complexity in Smoothy. This allows developers to benefit from the advantages of Docker, without them even having to know that they are using it.
The goal of Smoothy is to close, or at least tighten the gap between Dev and Ops, by automating and abstracting all the complexity of the cloud and offering it through a developer-proof interface.
If you would be interested, I'm always looking for experienced developers/cloud engineers who are willing to test out Smoothy and help shape the future of cloud hosting. 🚀☁️
I will test out out Smoothy this month.
In the mean time, I still have to finish some sort of a CLI for my co-workers/friends to generate a config file that is used as a blueprint for launching containerized environments. Instead of them having to write it. Kind of like drone.io but for more general purposes.