7
19 Comments

Tech stack for 100k static pages

What JS stack would you use for a site with about 100k pages, mostly reads, very infrequent adds and updates.

The data source is available in 3 formats, CSV, Postgres db and Rest API.

  1. Nextjs + my own DB, API
    Even though I'm a fan of Gatsby, I'm ruling it out because of the long build time.
    Which leads me to the next candidate - Next.js. I'm thinking I can explicitly at build-time pregenerate, let's say 10k most popular pages, and rely on SSR and caching for the rest 90K.

  2. NetlifyCMS + Nextjs
    I have little real world experience with Netlify CMS but it seems like a viable solution than #1 as it comes freebies such as auth, editing .

  3. Others?

  1. 4

    I would certainly go with Next.js (especially if you'll need some backend for dynamic data), the question seems more indeed where/how to deploy it.

    If the mutations are really infrequent and can be automated to trigger a new build, I'd suggest deploying through Vercel or Serverless Framework (with their next.js component, deployed to AWS, though you need to have your domain on Route53), so you can take advantage of their optimizations on the deploy process and reduce build time while keeping the dynamic pages efficient and performant for end-users.

    Be aware that if you have a lot of traffic, Vercel or Netlify can become expensive really quick, which doesn't happen as quickly for Serverless deployed to AWS.

    1. 1

      What advantages do Vercel offer vs deploying on your own VM? I'm thinking out loud here - most popular pages will be pre-generated and cache at the CDN level. The server will only take a hit every time a new page has to be built.

      1. 2

        Great question! Basically they just manage all DevOps/infra for you, you just have a simple vercel.json file to deploy it (serverless.yml for Serverless Framework), and the backend is works like an AWS lambda call (it's not a server always on and ready, it boots up to respond, and shuts down after 5 minutes of inactivity).

        With your own VM you need to make sure it's always up, etc.

        I hope that makes it clear, even though this is a very short explanation.

  2. 3

    I would use a static site generator like Eleventy.
    It is really simple to use and you can mix sources (API, static markdown, custom DB connection...)

    1. 1

      Very interesting! It seems like I would have to generate all 100k at build time.

  3. 3

    I have used react-static. It was pretty straightforward to load a huge amount of data (I believe I had 1000s of pages) in the router config to create the static site. The build was pretty fast too.

  4. 2

    I have a GatsbyJS V2.0 site with about 94,000 pages. It has both static pages as well as SPA. Happy with the current build time of about 25 min (and 15 min rsync to Digital Ocean) and looking forward further reduction with V3.0 incremental build support (Did not work for me in V2).
    This is a consumer facing site (emert.org) listing all product recalls (USA Only).

    1. 1

      Thanks so much for sharing your numbers. If update is very infrequent, 40 mins isn't too bad! One follow up question: In development mode, does Gatsby build do a decent job of caching? ie do you have to wait 25 mins to see a minor CSS change?

      1. 1

        In dev mode, Gatsby is well designed to do decent HMR and caching. It is very quick with reloading your changes.
        I used Gatsby because of it's React integration. If need arises, you can always add a React SPA using Gatsby's router.

  5. 2

    I'm running on Gatsby + AWS Amplify Console for deployments. It's fully automated and pretty feature complete. Worth looking into it!

    1. 1

      Roughly how many pages do you have? How long is the build time?

      1. 1

        It's a full fledged app, so it includes a lot of testing and dependencies. If it was just a Gatsby blog it would take around 2-3 minutes.

  6. 1

    I am a fan of Jekyll. What you would you even use a JS framework for static content ? It's not suitable for things like indexing by browsers.

    1. 1

      I was thinking staying in JS in order to add make the site interactive and possibly protect it behind a login.

  7. 1

    Any Jamstack tool with a good CMS !
    Regarding GatsbyJS, you can use Incremental build to speed up the build process.
    It will rebuild only what's changed...
    Another example is Smashing magazine :)
    https://www.netlify.com/blog/2017/03/16/smashing-magazine-just-got-10x-faster/

    1. 1

      Can you use Gatsby incremental build outside of their paid services?

      1. 1

        You can do it using experimental feature but if you need it right now maybe it is not safe enough...
        https://www.netlify.com/blog/2020/04/23/enable-gatsby-incremental-builds-on-netlify/

Trending on Indie Hackers
Feedback on my (not yet published) about page 30 comments Vegans, vegetarians, and anyone with an allergy, food intolerance, or just a preference, I need you! 26 comments Open Sourcing my SAAS Starter Kit 13 comments Songbox breaks £1000mrr ($1378) 6 comments Racket - easy short form podcasting 6 comments Nerdogram - A photo sharing app for Github nerds 5 comments