1
10 Comments

What to use for a backend tech stack?

Hey IH!

I'm currently building a writing app similar to google docs. Similar in the sense that there is an autosave feature, and eventually there will be the ability for multiple editors at once.

I'm currently using firebase as a backend because of how easy it is to set up and begin developing.

I was testing out autosave and found that in one 30 minute writing session My session executed about 2k writes to firestore (free tier quota is 20,000/mo). Some optimization would definitely help, but I didn't want to do that prematurely.

So my question is, what do you use for a backend? Are there options out there that are as easy and fast as firebase? Is it better to build your own?

on August 21, 2019
  1. 2

    This doesn't answer your question about backends, just adding to the discussion. I've been dealing with auto-save on StormJournal, I'm just using react/redux/laravel/mysql for now. It was quite a hard problem to optimize for because I have three different data types that I wanted to auto save differently (because otherwise I would be saving lots of data unnecessarily) and had to deal with different states of the app (new user, load on logins, page refresh, etc). To autosave every single change would overload the system with massive http requests and race conditions, as you'll be aware. I know what you mean about premature optimizations but I'm glad I sorted it out before launch because I dread the thought of implementing it while people are actually using the system. I put the different data types onto different timers that only send http requests when there has been a change to that data type within the last X seconds - which might be a simple enough optimization for you to implement.

    1. 2

      Thanks for your insight. One nice thing about NoSQL (firestore) is there doesn't have to be a distinction of data type. I'm just storing what I have in my redux state directly into firestore. I can see how that would get tough to manage though with different save frequencies.

      I definitely need to implement something to check if data has actually changed or not and only send an update for that one piece. Firestore is partitioned in way that made me design it so each paragraph (block element) a user has is 1 write (1 http request). I'm currently rewriting a user's entire document every time. So if there are 20 paragraphs, autosave is firing 20 http requests every few seconds. There is definitely an optimization there where I only update the paragraph that changed.

      1. 2

        I hear ya man, I started with a nosql model (implemented via json fields in sql) and was saving the entire state which included all text documents, all search terms, and all their API search results, plus the state of the page. It wasn't long before I realised just how much unnecessary data I was hitting my server with. I ended up pulling out the search results and documents into their own table and referenced them from keys in my page state json, which I kept in the same field. Now I only get and store whatever is needed for displaying at any one time. I wondered at the time if I was overdoing it, but the sheer number and size of requests that hit the server soon add up. Plus it was an interesting problem to wrestle with as a developer and something I learned a bit from. Your situation sounds different in that one document is open at a time, right? Volume probably wouldn't as much of a problem for you, so much as frequency, I'd imagine? Fun stuff to work on anyway man. Good luck.

        I'll be keen to check in with you every now and then, especially when you start thinking about websockets. I've got a plan for v2 involving an overhaul of the saving system to use websockets. I ran into another problem of worrying about a user opening and editing the same document on different devices and how much of a nightmare tangle that would cause. I put in a temporary fix of only allowing the user to log in on one computer at a time, but I'd rather use websockets so it doesn't matter, each device always has the latest changes - this will also open up the door for collaborative docs, like you were mentioning.

        1. 1

          Yeah, these are very cool problems you don't really get to see building a common thing like an e-commerce site.

          I have a single top-level document open at a time right now, but that has many documents referenced by it. Firestore's data model has concepts of documents and collections. The root of the db is your base collections. For me, that is /users and /spaces. A space is a users workspace that can have many pages, and each page can have many blocks (paragraphs). A firestore document under /spaces holds the top-level data, with a collection /pages that references all of the firestore documents representing pages in my app. Each firestore document holding a page holds a reference to another collection /elements. Firestore documents under elements hold the text and metadata for each paragraph. This data model creates a tree structure:

          // Sorry for this ill-formatted code, I thought it would be monospaced
          /spaces <= collection
              abdc123 (spaceId) <= document
                  /pages <= collection
                      fghj456 (pageId) <= document
                          /elements <= collection
                               uiyt789  (elementId) <= document
          

          The document ids can be generated by firebase, or you can set them yourself. The reason I had to make it so nested is that each firestore document has a size limit of 1Mb. A paragraph would likely never get there, but a single page could.

          Right now, on save, I'm essentially just doing a full replace starting at the space id. A user is really only updating one element at a time, though. Unless they delete a lot at once. I just need to be smarter about managing the database references down to each element, identifying what actually changed, and saving only that. That will significantly cut down on writes to the DB.

          I'm excited about websockets! I'd love to hear how that implementation goes for you. I think I'm going to stick with firebase for this (for a while at least). The js SDK provides an implementation where you can subscribe to real-time updates of any firestore document. It uses websockets under the hood, but you don't have to do any channel implementation. Just have to manage subscriptions which play nicely with a redux model.

          As for the problem of having the same user logged in on multiple devices, you could maybe handle it how it seems google docs does. Generate some deviceId fingerprint and combine that with the user name. If you log into google docs on two devices, you'll see a collaborator and the name is yours. On only one device, there is no other collaborator. Each device always having the latest via websockets definitely helps this be less of a problem, too.

          Sorry for the book! I can quickly get out of hand when talking about implementation details. Thanks for the back and forth

  2. 2

    I have used firebase and node and I have also used mongo db and node with express. I like using mongo db and express.

    1. 1

      Have you ever implemented websockets with mongo db and express?

      One thing I like about firebase is you just get that for free with the sdk. I'm going to need it if I want to implement multiple people editing at once.

  3. 1

    This is an area where Elxir, the tech stack I teach, is particularly well suited. It's got unique concurrency and soft real-time capabilities due to the Erlang VM it runs on. In addition, its primary web framework, Phoenix, has channels as a first class citizen. Phoenix also has built-in tooling for presence, which is very difficult to get right if you're building it yourself.

    If you know how to use it, it's easy to get millions of writes per month on a bottom-tier $5/month VPS.

  4. 1

    For something like that, you could also use a GraphQL server that has subscriptions enabled. That way you have a real-time connecting with data via a websocket to do read/writes.

    Hasura is one open-source option. We also help get that setup (along with typing you DB and migrations) at Midtype.

  5. 1

    I'm running PixxiBook also on Firebase but the realtime database. Is billed by bandwidth rather than transactions and we use much more than I anticipated but the costs are not unreasonable and so I haven't put any effort into optimising it.

    In your case, you can probably decimate the number of writes but even at the current rate and pricing that session probably cost a quarter of one cent - not worth bothering to optimize IMHO.

    The benefit of Firebase DBs are that you can put much of the logic in the web client and the DB access rules. We use firebase functions for a couple of server side ops: just payments and webhooks from our print providers.

    1. 1

      Ahh, interesting. I never noticed that the realtime database didn't count transactions. I wonder if that was their motivation for building firestore so that they had a way to count transactions and bill for it. Firestore bills for data stored and bandwidth on top of transactions.

      You're right though, it is still insanely cheap. It's a wonderful time to be a developer.

      Yeah, I definitely love being able to put most of the logic on the client. Enables me to iterate so much faster.