When writing too much code kills your business
About a year ago I shipped Fizbuz v1. It was built on AWS Lambda, DynamoDB and Auth0. I had discovered a new, but really interesting, open source tool called Architect that made it really easy to spin-up AWS resources and build fast, and for a while I was really productive. And it worked well... for a bit.
Starting Fast
After doing customer discovery and building a prototype on Rails, I had clear backend requirements:
- Serverless architecture
- REST API for simple read/writes
- GraphQL API for complex reads
- APIs & the data layer bootstrapped by GraphQL type definitions
I had never built a web application using Lambda or DynamoDB, so I was in full "learn" mode. I soaked up everything I could about how to build these kinds of applications and ended up building a ton of API and data layer tooling from scratch.
This felt very productive at the time and I was incredibly proud of how slick it was that I could define a new business object in GraphQL and that matching REST API endpoints and data layer CRUD operations would be automatically provisioned with minimal additional work.
Slowing Down
After a furious few months of coding, I shipped v1 of Fizbuz and started onboarding users. As I came in contact with users, I started to collect more and more feedback on how to improve the product.
This is where I started noticing a slow-down in my ability to ship new features. As I would context switch from non-technical tasks (talking to customers, doing discovery, networking, etc) back to the code, I would sometimes have a hard time understanding how the code worked. Since I wasn't using third-party tools for commons tasks like client-side data fetching or API request processing or database interaction, I couldn't rely on documentation or Google to get unstuck.
The issue came into much sharper focus when I would try to onboard other developers for contract work. Since I wasn't using tooling that they were familiar with, too much time was wasted getting them up to speed on how the system worked, rather than spending time on shipping new customer-facing features.
Screeching to a Halt
A few days ago a user alerted me that when they were using Fizbuz, they noticed that the app seemed to be caught in an infinite loop and was making an endless series of HTTP requests to the server. This wasn't visible in the UI, which is why I hadn't noticed, so I dove into the codebase to figure out what was wrong.
It turned out that I had refactored some client-side data fetching code in order to be more compliant with React best practices around Hooks and Exhaustive Deps. I unwittingly broke perfectly good code and created an infinite loop. In this case, the requests were idempotent reads, but next time who knows? What if they had been writes?
Moving Forward
I love making things, I think all developers do. And I'm proud of much of the code I wrote. It solved a real problem for me and did so in an elegant way. But I had to look myself in the mirror and ask myself this:
Do I really care about building tools to layer REST and GraphQL APIs on top of DynamoDB?
I decided that I care more help developers build better networks and find better opportunities, and my engineering cycles for the rest of the year will be spent on building product and features that deliver on this.
I am currently in the process of evaluating stacks that meet my original requirements and talking to developers who have experience with these stacks and are available for contract work. Right now it looks like AWS AppSync, Prisma and Hasura are strong contenders for the backend and data layer.
I'll be writing more about my journey to ship a v2 of Fizbuz, so please feel free to follow along!
If you have any suggestions on a stack I should look at, please feel free to leave a comment.
Unpopular opinion: React is the 80% problem here.
If you weren't using React, would you have:
Does your app really need React? Could you provide the same functionality with Rails views, turbolinks, and actioncable?
The more I code in this decade the more I want to go back to how it was circa 2005. The sweet spot when mvc became popular for the server side and js was used to validate a form or two.
Interesting. I've started ditching React also for some projects, but I still hate MVC. I can't get over the fact that there's this magic way that files are bound together by names and if you want something a little different - you're screwed. I mean if it works and oyu like it, that could be the best option of all, but I personally can't stand it :D
I really love this comment. I remember discovering Rails 1.0 back in 2005 and feeling like I had been granted productivity super powers. It was because of Rails that I was able to prototype, iterate and ship a web application entirely by myself that ended up supporting tens of thousands of customers.
That being said, I decided to move on from Rails (and Ruby) for a few reasons. So I'm definitely in the market for a toolchain/product/service that enables me to write code exclusively in JavaScript (personal choice) and that provides the kind of productivity benefits that I got from Rails.
One tool that might fit the bill is RedwoodJS.
I had that same experience with Rails and also went away mostly because I'm a Dev and I don't want to be doing devops.
Also, imo, SPAs provide a much superior user experience and coupled with Firebase you get something that is painless to maintain, scalable and blazing fast.
But I feel like I have lost part of my superpowers in this transition.
I think it's probably just that this way of building apps is still young so there aren't solutions that allow Rails like productivity yet.
This is right. I'm using good old KnockoutJS for frontend coding on productfruits.com.
It's not a trendy JS framework, but it works and it allows me to do things quick and fast. And that should be the main goal.
My whole tech stack is .NET core, Postgres database and a Linux virtual machine with GitHub driven deployments.
Not cool, but it works :-)
Works > Cool
I'm not trying to optimize for trendy, but one thing I am considering is mindshare. I want to build on a toolchain that makes it easy for me to hire (FT or contract) people, which is one of the reasons I went with React. There are just a ton of folks who know how to use it.
You can always find good developers if your product is good, no matter what technology you use (OK, it shouldn't be super out-dated). I wouldn't think this way, you're still building it for your users, for your revenue.
I went through a few startups (now big global companies) and I can say all good developers worked for these companies because of the product itself and the vibe of something growing from zero to infinity.
Love this mindset @frix. Its like getting the right people on the bus with your vision then allowing people to contribute their magic. When you find those that are more motivated by the challenge than the $ then you know you got something good going. Thx for sharing.
Actually, considering the current situation, it shouldn’t be a problem to find a developer for any kind of project for a long time. Regardless the tech stack. :-/
Yes, my friends working in the automotive industry (VW) have their SW projects stopped. It doesn't seem good...
Are you a developer?
Not everyone is ok dealing with the mess.
Also, it’s not adding any value for a developer to work with an outdated tech stack.
If you want to find GOOD developers to work on such product - you’ll have to pay extra at least
Yes, I am.
I explicitly stated that it should not be outdated, also I didn't write about writing a messy code.
Under messy code I meant that frameworks such as KnockoutJS doesn’t force/encourage you to architect code properly or don’t give an opportunity to do so in some cases.
Just take a look at Magento frontend code which uses KnockoutJS
You can write messy code with React, you can do a lot of mess with serverless, you can even use Git in an awful way :) I mean it's always about what you know and what you are able to do.
What was the source of those requirements? Were potential customers actually saying, "I'll pay for this but only if it's a serverless architecture and has a complex hybrid REST/GraphQL API"?
This was also what I thought :) sounds mighty technical for customer discovery insight.
Super fair question! The customer discovery led to a product that had a very rich schema. Rendering a single user profile would require fetching data from possibly dozens of records. This was similar to the use case that drove FB to invent GraphQL in the first place.
So as the developer of the product, I chose the architecture that I thought best fit the what was being built.
Have you looked at AWS Amplify ? It pretty much does what you were building.
You can define your configuration and its able to provision the appropriate resources automatically on AWS.
For example, to add GraphQL API you do "amplify add api" (let's you configure it)
Then you can edit a schema with annotations like "@model" (that tells AWS to create a DynamoDB table)
Then you do "amplify push" (creates resources in the cloud)
And that's it !
It also includes predefined queries and CRUD mutations and they have helper components for React, Vue and Angular.
Authentication is also very easy.
I'm freelance dev if you need :)
I don't have that much experience with Amplify though, I was looking into it to maybe migrate a personal project from Firebase.
I decided against it though, since this would most likely take me weeks to do, time where I'm not implementing useful features instead.
One problem for me also was that I wanted to use Vuex-ORM and the GraphQL plugin but it doesn't expect the same schema, I abandoned the idea at that point. I could probably have spent time opening github issues etc to figure it out, but again, that's not time spent building valuable features
I am definitely looking into Amplify! But a bunch of people, who have used in for real applications in production, have warned me that it has a ton of rough edges.
It feels like kind of a "whole kitchen sink" of things, and what I'm mostly looking for is a data layer product/service that works well in a serverless context and provides a rich GraphQL interface out of the box.
What about FaunaDB ?
They provide exactly that.
They also have a Netlify plugin so you can use their authentication system and other serverless services.
It's been a while since I looked into Fauna + Netlify, but if I remember correctly, the main reason I didn't stick with it was the pricing of Netlify.
They have multiple plugins for each service and they each have their own cost.
I thought this would add up quickly.
Fauna seemed really powerful and not the kind of tech where you get stuck because they just don't do what you need.
The learning curve might be steep though.
But other than that it seemed like a great developer experience.
I'm interested in knowing what you end up choosing as I'm interested in this kind of stack too.
Thanks for the pointer! FaunaDB is really interesting. I'm going to dig into it more.
One thing that might stop me from using it is the learning curve and lock-in of FQL (Fauna Query Language).
Yes, that's the price you pay for flexibility I guess.
They do have very good documentation and a lot of YouTube videos that go in depth into specific use cases.
I'm a bit of a noob when it comes to the backend but outside of AWS tools, I've found Google's firebase backend both reliable and expanding.
The right tool for the job is always the popular tool that everyone knows, for which there are a million libraries already written, security bugs are patched within hours, and instead of writing code you can just copy and paste off stackoverflow. Beyond that, none of your "requirements" matter. You're building a web app. Just my 2cents.
This is actually a problem I'm trying to solve with Nodewood. I'm making a SaaS starter kit that makes it easy to get right to writing business logic and easy to stay writing business logic. It's all Javascript, so libraries can be shared between frontend and backend, and I went with Vue for the frontend framework specifically because it's a lot easier to work with as a small team.
It's probably a bit late for your use case, especially since I'm still in development, but it's good to know that there's at least similar use cases out there for what I'm working on.
Your landing page makes me want to use this, especially the "Designed to be extendable" part.
As someone that would be interested, this is the first thing I want to know. Is it modular, opt-in of any functionality and am I going to be trapped at one point because you don't support my use case.
But also, does it come serverless ? Because the last thing I want to do is have to configure and maintain servers.
Well, it depends on what you mean by modular. Some things are pretty deeply baked in, like the use of Vue. Nodewood provides a lot of functionality out of the box, and that's through a whole bunch of Vue components working together. If you wanted to swap for React, you'd be throwing away most of the value of using Nodewood.
However, the
appfolder (where you place your code) is a mirror of thewoodfolder (where the Nodewood code lives) and any file you place in theappfolder overrides the one the in thewoodone. This means you can easily override any file in the framework, so extending or even replacing the built-in functionality is easy.As for managing servers, I haven't settled on a strategy. In my day job, I've managed fleets of standard servers and serverless apps, and I might just end up offering both. Either way, it will come with a deploy script that will deploy to production for you. It'll be a command-line command, not a long manual process.
If you're interested, I'd love to get your feedback when it starts getting closer to release, for example when the documentation starts coming together a little better. Even if you end up going with something else in the meantime, good feedback is always useful.
I would love to, I've been thinking about building something like this for a while now. I'm curious how you implemented this as well.
My gold principle is that code (or design) has to make sense at first glance. It doesn't mean you can understand all the details in 2 seconds. But you know where to continue when looking for something.
Separating front and back is a good idea as far as my experience goes. It just makes things simpler to reason about. I've got (only) integration tests at the back so I can develop without having to go to the front. Once that's done, building the front in any framework is a joy.
My major gain that I got was when I started using functional languages (my favorite: Elixir). I don't know how to explain it.. it just seems to force you to think more to complete the simplest tasks, but once you're done they just effin work. I take a months long break from my code, come back, and everything just makes sense. I don't know if it's factually true, but it seems to me that I end up with much less code in general, and the code i do have is easy to reason about. Now when I have to go back to an imperative language and am forced to use a for loop - I choke. It's just bad. How can I possible reason about my code if I have to keep going up and down in a single function.
Actually - that's a great way to explain functional languages: it's harder to understand a single row, but it's much easier to understand a whole function. Ah whatever, blabbering here.
All in all I would highly suggest you throw away all of that stack, don't look for new ones, and think about what is it that your app actually needs to do in principle. Only take helpers on board if there's something that you need to do that's been done better than you can. But cut away all the fat first.
I once took Dynamo out for a run also. I thought it'd be great for performance and it's easy to use, just like the landing said. In reality I now needed to learn a whole new technology and it came with a whole new set of problems - all the while what I needed was the been-to-war sql-is-easy performance-built-in PostgreSQL.
lol, I was wondering why people were writing such walls of text :D
Please keep this discussion coming our way. I am working on learning aws serverless/reactjs at this very moment for a b2b side project. Much appreciated!
Hey, this is really funny because I was nodding my head all the way along to your post, saying "Yea my app feels like this! It's super frustrating!"
Then I got to the end of the post where you are considering Hasura.
I'm on Hasura :D
To your post's point, I think that the second that an app gets larger than what you can fit into your head, this problem is inevitable. I'm starting to hit that with codernotes.io, and I'm starting to see that complexity.
(Of course - shameless plug here - codernotes was written to help developers cache that complexity so they don't have to keep it in their brain, and I think it's one of the reasons I'm only reaching this point now, after 99% of the features have been implemented)
So, about Hasura:
I really, really liked it at first. As someone who has never worked with a real backend before (my dev experience is in c# and java services, not web-facing), I was blown away by how fast I could spin up tables and a permission-driven API for it. However, as the app got bigger, I have found a few pain points you should be aware of.
The main one is Authentication. Here's the current Sign-up / Auth flow in my app:
That's where it was at for a while, until I needed "teams", and I wanted the source of truth for those teams to be Hasura. To get that, I needed more auth.
Yuck.
Things like this have been the biggest regret w/Hasura. I don't feel like I've reduced complexity in the app - I feel like I've moved it. I wish I had spent the time to dive deeper into Phoenix / Rails, where session tokens are simple, a REST api is easy, and migrations are practically automatic.
That being said, for a first-time web developer, I would still definitely recommend Hasura.
If someone is super comfortable with cloud-native technology like Firebase, Hasura seems really powerful.
So, this is just a single opinion. I still think there's some use cases for it.
Also, I've been throwing a bunch of the notes for Hasura on codernotes, so it's now a great resource for some common sticking points. Especially if you're using svelte on the front-end :)
Re: complexity and development tools
I’ve half jokingly theorised more than once over last few years:
Complexity is a constant - it just gets moved around.
Svelte and Hasura.
I am looking into this stack now for the next side project.
Are your codernotes you mentioned, public ?
Unfortunately not, but I'm happy to talk through any questions or issues you may have!
Wow, thanks Kevin! I really appreciate this thoughtful reply.
Based on what I understand about Hasura, I follow what you're saying.
Do you think this is a by-product of trying to 100% of your backend through Hasura? Could you solve some of this complicated back-and-forth by implementing a dedicated sign-up service (HTTP endpoint) that is separate from Hasura?
That's one of the things I'll be looking at with any persistence layer or backend service: the ability to escape hatch out for the 5-10% of cases that aren't neatly solved by their abstraction.
I think this is smart.
Yes, I definitely think this is a by-product of trying to abstract away the backend, and not treating Hasura as simply a graphql generator and GUI. I'd highly recommend them for those two use cases.
Auth has been the main sticking point, followed by telemetry (everything is abstracted so it's difficult to get useful user data without writing a serverless function for each database change you care about).
Hasura does have a webhook mode, allowing you to 'opt out', but you'll be hard-pressed to find any examples online.
https://hasura.io/docs/1.0/graphql/manual/auth/authentication/webhook.html.
Hey all, I thought I'd pile on here with some of my Hasura experience.
I chose not to use Hasura as a one stop backend shop and side stepped the above issues that Kevcon80 was referring to. I’ve got a dedicated API for all mutations, and I manage my Postgres instance via TypeORM. Hasura’s convenient for a quick and easy read-only GQL interface, and its basic out-of-the-box auth(orization) solutions work reasonably well to get started, but I fully anticipate layering on with custom auth as the complexity of my app grows.
All in all? Hasura's been great to start! I'll see how it holds up in the coming months for my company. Right now I sort of view it like a super convenient, batteries included
join-monster.Won't Firebase Auth + Hasura integration suffice for most common-auth scenarios?
nice!