I built an end-to-end data pipeline tool in Go called Bruin
Hi all, I have been pretty frustrated with how I had to bring together bunch of different tools together, so I built a CLI tool that brings together data ingestion, data transformation using SQL and Python and data quality in a single tool called Bruin:
https://github.com/bruin-data/bruin
Bruin is written in Golang, and has quite a few features that makes it a daily driver:
it can ingest data from many different sources using ingestr
it can run SQL & Python transformations with built-in materialization & Jinja templating
it runs Python fully locally using the amazing uv, setting up isolated environments locally, mix and match Python versions even within the same pipeline
it can run data quality checks against the data assets
it has an open-source VS Code extension that can do things like syntax highlighting, lineage, and more.
We had a small pool of beta testers for quite some time and I am really excited to launch Bruin CLI to the rest of the world and get feedback from you all. I know it is not often to build data tooling in Go but I believe we found ourselves in a nice spot in terms of features, speed, and stability.
Looking forward to hearing your feedback!
This is really impressive! Love the local-first design, especially the ability to mix Python versions per pipeline.
Go isn’t a common pick for data tools, but I bet the speed and stability benefits are huge here. Can’t wait to give it a spin—congrats on the launch!
hey @denizhd, my first concern would be scale of data it can run if i use bruin. Most of the daily ingestion tasks of companies can finish on single small ec2 machine but it is hard to understand how much resource it will need. So they use spark which can dynamically allocate resource. I have worked on project to estimate resource need of the tasks before migrating to new platform and it is very hard to be precise.