Our struggle With Puppeteer

While revamping our Landing Page builder to enhance its User Experience, we wanted to create a preview image of landing pages on users' dashboard in order to help them easily differentiate between their landing pages. Below is a before and after image for you to gain more context about what we're talking about here.

So, to get to the final result of displaying a preview image on the users' landing pages as shown in the new UI, we went through a lot of iterations. It wasn't about how to create the preview image, that was the easiest part, but where to host the generator function. Let me tell you the full story…

On the first iteration, the preview image was being rendered on demand using Puppeteer. We were calling Puppeteer every time the dashboard is being loaded to create the preview images. Functionality wise it was working fine. However, performance wise it was horrible. Imagine calling Puppeteer each time you load the dashboard just to display a preview image of each landing page, and then imagine a user having multiple landing pages waiting for Puppeteer to get their preview images. That was hard to tolerate. And that's when the second iteration took place…

On the second iteration, we decided to create the preview image once and then attach it to its respective landing page and store it on our backend which is built with Ruby on Rails. Our frontend is built with Next.js and hosted on Vercel; hence, in order to perform that action, we had to create a Serverless Function on Vercel for Puppeteer in order to generate the preview image once the landing page is created or updated. Our logic was going to be as follows:

1. Frontend sends the created/updated landing page data to backend.
2. Backend validates the landing page data and then creates a Sidekiq job to call the Serverless Function on Vercel.
3. Serverless Function receives the landing page URL and calls Puppeteer to create the preview image.
4. Backend receives the preview image and saves it.

All was going smooth and we really liked that solution and were excited to implement it. But, happy ending is not yet to come. While deploying our Puppeteer function to Vercel's Serverless Function, we found out that the maximum size of the function must be 50 MB; however, our Puppeteer function was 59 MB. We tried to downgrade the Puppeteer version to get it below 50 MB and we managed to do so. But, again, happy ending is not yet to come. The downgraded version was deprecated, hence not functional. And then comes the third iteration…

On the third iteration, we asked ourselves: what if we create a Puppeteer service on our Rails app? After some research we found that doing so is possible by using Grover gem along with Puppeteer. So we started the process by installing the gem and Puppeteer then we created the service. So now, instead of calling a serverless function to create the preview image, everything can be done from within. We tested everything on local and it was going as expected. So we deployed to staging for testing. But, again and again, happy ending is not yet to come. Puppeteer service was failing on staging.

After some investigation we found that Puppeteer requires higher privileges to launch the browser which we don't have on our host, which is Digital Ocean's App Platform. Initially, we chose App Platform because we wanted to put our full focus on building a high quality and premium product; hence, we didn't want to spend much time on managing servers. But, seems like it has its pros and cons as well. We tried to search for solutions everywhere until we managed to find one. Someone on Digital Ocean's community suggested deploying the app using Docker which can help us get the required access.

We started the process of dockerizing our Rails app and we managed to complete it in a short period of time. We started the deployment process on App Platform using Docker and it built successfully; however, deployment failed… We tried to check the logs to debug, but, for our surprise, the logs didn't have any useful information. Actually, it didn't have any information at all; it only contained empty square brackets, [ ]. That was a bit frustrating because we couldn't figure out where the issue was in order to debug. We tried opening a support ticket but then we were in a rush to complete this feature because we already spent more time than expected on it. So, by the time Digital Ocean's support replied to our ticket, we were already in our fourth and final iteration…

On the fourth and final iteration, we've had enough already and decided to create a microservice for our Puppeteer function. Initially, we had the thought of using a microservice but we didn't proceed with it as we were trying to cut costs as much as we can; because, after all, we're not profitable yet. But, it seems that we don't have any other choice. We created the microservice and everything is working smoothly now. Finally, happy ending has come.

Discussion:
From your point of view, what do you think of our approach? Kindly share with us if you have any feedback. Your feedback will help us grow :)