How can we dynamically modify the LogLevel of a Go program at runtime?

A few days ago, we accidentally discovered that our WebPLB logs were flooded by a barrage of requests from our own control.

The name of our program that actually handles the traffic is WebPLB. Our API Service is named WebPPT. Can you guess what our other services are called? (Hint: They all follow the format WebP[A-Z][A-Z].)

Having too many logs can cause a lot of trouble for online debugging. In addition to changing the corresponding unimportant logs to Debug, we are also considering how to change the logLevel at runtime. This way, when needed, we can change the log level, and the application can see the required logs without restarting.

Our logs use logrus, so you can easily change the log level with the following code:

log.SetLevel(log.DebugLevel)

But the question is, how do we call this code at runtime? The most naive idea is environment variables, but if we use environment variables, we need to consider the following issues:

1. The application needs to run a task regularly to read environment variables and update the log level, so there will be a certain delay.
2. The effectiveness of environment variables. The environment variables set in the current shell are not visible to already running processes.

Suddenly, I thought of a very important concept in UNIX - signals. Signals are a method of interprocess communication. When a process receives a signal, it can choose to handle and execute it. This happens to be our requirement, and using signals has the following advantages:

1. No need for periodic tasks, no delay.
2. Convenient to adjust at any time, no need to restart.
3. Native support, maybe even for Windows.

By using kill -L, we can see which signals are available, summarized as follows:

We can use the custom signals SIGUSR1 and SIGUSR2. We just need to run them in the main through a goroutine. Here’s an example code:

package main

import (
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/sirupsen/logrus"
)

func main() {
	log := logrus.New()
	log.SetLevel(logrus.InfoLevel)

	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGUSR1, syscall.SIGUSR2)

	go func() {
		for {
			sig := <-sigChan
			switch sig {
			case syscall.SIGUSR1:
				log.SetLevel(logrus.DebugLevel)
				log.Warnln("Log level changed to DEBUG")
			case syscall.SIGUSR2:
				log.SetLevel(logrus.InfoLevel)
				log.Warnln("Log level changed to INFO")
			}
		}
	}()

	// rest of your application
	for {
		log.Debugln("debug log")
		log.Infoln("info log")
		time.Sleep(time.Second * 2)
	}

}

In another window, you can send signals through htop or kill -s SIGUSR1 <pid>. As shown in the image, you can see that the program’s log level is changed at runtime.