Patching Next.js to add production grade logging

Premise

I am not a frontend engineer. Claiming that would be a disservice to all the people who write code to move pixels to perfection, these days extremely dynamically and with finesse. However, I've worked closely with frontend engineers and had to troubleshoot applications they wrote in production under the pressure of losing thousands of dollars for every minute the pixels weren't in the right place.

Recently, I've been involved in a project where the framework of choice was Next.js. I read on Twitter that it is a very popular framework among frontend engineers. From Vercel's website, I read:

Used by some of the world's largest companies, Next.js enables you to create high-quality web applications with the power of React components.

Well, if its core maintainer is Vercel, a $3.25 billion company (at the time of writing) and used by many other large companies, it certainly has some merit. Indeed, the UIs I've seen created are really nice - smooth and fast. I cannot speak for Next.js in terms of its ergonomics since I've only modified a small part of the code once, and that was simply an authentication guard.

However, what I can speak about, and not just speak about but rather try to improve, is its logging system, or lack thereof.

What you get out of the box

I am not sure what you get out of the box if you deploy your Next.js app on Vercel. I was never interested in locking myself into a vendor when developing software. It is unnecessary complexity tucked into a veil of features and simplicity that becomes very expensive if you end up creating successful software. But, I can tell you what you get if you decide to host your Next application on a bare-metal server, a container, or a container orchestration platform. For me, it's Kubernetes.

alt text

This is it. And I am lucky since the code we're running has some deprecation warnings, it feels like the code is actually communicating with me about what's wrong. Most people simply see the first part of this image, down to the boot time. This is bad in more ways than I would like to describe in this post.

I hope that everyone reading my blog understands why logging is important. If not, I promise to make another post in the future to thoroughly explain why good logging (not just logging) is really helpful for modern software.

What you can do

We want to integrate a log-it-all solution for Next.js, which unfortunately means creating a persistent patch for the server since the framework uses its own internal server and doesn't expose the response and request objects.

The most sustainable way to do this, that I could think of, is using patch-package to modify node_modules/next/dist/server/lib/start-server.js automatically. The packages we are using to off-load the formatting and actual logging using JSON (because eventually, we are going to ingest these logs in an ELK stack) are pino-http and pino-pretty.

Including a postinstall script

In the package.json file, add a postinstall script.

  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint",
    "ws": "node -r esm ./websockets-server.ts",
    "postinstall": "patch-package" // this one here
  },

Then, you want to install patch-package as a dev dependency since it's only needed at the package install time, as well as pino-http and pino-pretty as a regular dependency.

yarn add --dev patch-package
yarn add pino-http
yarn add pino-pretty

Modifying the server

In node_modules/next/dist/server/lib/start-server.js add the following consts after the _ispostpone.

const pino = require('pino');
const pinoHttp = require('pino-http');
const pretty = require('pino-pretty');

We also want to create a new prettyStream that we can use in order to only include information that we want and in a way that makes reading the logs easier.

const prettyStream = pretty({
  colorize: true,
  translateTime: 'SYS:standard',
  ignore: 'pid,hostname'
});

We create a prettyStream that will colorize the output of the logging as well as use a standard human readable time format (SYS:standard) and ignore the pid and hostname attirbutes that we don't care about (I don't, you might).

Then, finally, we want to create our logger. Here's one I wrote and it works well for my use case.

const logger = pinoHttp({
  logger: pino({
    level: 'info', 
    base: null,
  }, prettyStream),
  serializers: {
    req: (req) => {
      return {
        id: req.id,
        method: req.method,
        url: req.url,
        headers: req.headers,
      };
    },
    res: (res) => {
      return {
        statusCode: res.statusCode,
        headers: res.headers,
      };
    }
  }
});

Now, search and find the requestListener method that will give us access to the request and response objects. And the logger in the first line of the method.

async function requestListener(req, res) {
    if (!/^(\/_next\/static|\/_next\/image|\/favicon.ico|\/static\/|\/.*\.(css|js|png|jpg|jpeg|gif|svg|ico))/.test(req.url)) {
        logger(req, res);
    }
    ...

Instead of logging every request, which would quickly turn out to be extremely noisy and expensive to ingest, we filter which requests we want to log by regexing on the req.url, dropping all requests that are equesting static files.

The logging happens after res.end, so we get the full request/response objects logged.

Create the patch

Now, we have to create the patch that is going to be applied during the package install.

npx patch-package next

patch-package will try to apply the patch even if the version of Next.js changed, and in most cases it will succeed.

What you achieved

This.

alt text

Conclusion

It's a 34-line diff that will help you immensely if your web application turns out to be a hit. Go patch your Next.js server and advocate in the GitHub threads for Vercel to add proper logging support.

alt text