WebP Cloud Services Blog

How can we dynamically modify the LogLevel of a Go program at runtime?

· Benny Think

A few days ago, we accidentally discovered that our WebPLB logs were flooded by a barrage of requests from our own control.

The name of our program that actually handles the traffic is WebPLB. Our API Service is named WebPPT. Can you guess what our other services are called? (Hint: They all follow the format WebP[A-Z][A-Z].)

Having too many logs can cause a lot of trouble for online debugging. In addition to changing the corresponding unimportant logs to Debug, we are also considering how to change the logLevel at runtime. This way, when needed, we can change the log level, and the application can see the required logs without restarting.

Our logs use logrus, so you can easily change the log level with the following code:

log.SetLevel(log.DebugLevel)

But the question is, how do we call this code at runtime? The most naive idea is environment variables, but if we use environment variables, we need to consider the following issues:

  1. The application needs to run a task regularly to read environment variables and update the log level, so there will be a certain delay.
  2. The effectiveness of environment variables. The environment variables set in the current shell are not visible to already running processes.

Suddenly, I thought of a very important concept in UNIX - signals. Signals are a method of interprocess communication. When a process receives a signal, it can choose to handle and execute it. This happens to be our requirement, and using signals has the following advantages:

  1. No need for periodic tasks, no delay.
  2. Convenient to adjust at any time, no need to restart.
  3. Native support, maybe even for Windows.

By using kill -L, we can see which signals are available, summarized as follows:

Signal NumberSignal NameDefault Behavior
1SIGHUPTerminate process; can be handled, ignored, or caught
2SIGINTInterrupt process (like Ctrl+C); can be caught or ignored
3SIGQUITQuit process and generate core dump
4SIGILLIllegal Instruction; generally leads to a core dump
5SIGTRAPTrap for debugging
6SIGABRTAbort signal from abort() system call
7SIGBUSBus error (bad memory access)
8SIGFPEFloating-point exception
9SIGKILLKill signal; cannot be caught, blocked, or ignored
10SIGUSR1User-defined signal 1
11SIGSEGVInvalid memory reference (segmentation fault)
12SIGUSR2User-defined signal 2
13SIGPIPEWrite to a pipe with no reader; can be caught or ignored
14SIGALRMTimer signal from alarm() system call
15SIGTERMTermination signal; can be caught or ignored
16SIGSTKFLTStack fault (not common on most Unix systems)
17SIGCHLDChild process terminated, stopped (or resumed)
18SIGCONTContinue executing, if stopped; cannot be blocked
19SIGSTOPStop process; cannot be caught, blocked, or ignored
20SIGTSTPTerminal stop signal (like Ctrl+Z)
21SIGTTINBackground process attempting read
22SIGTTOUBackground process attempting write
23SIGURGUrgent condition on socket (like out-of-band data)
24SIGXCPUCPU time limit exceeded (see setrlimit())
25SIGXFSZFile size limit exceeded (see setrlimit())
26SIGVTALRMVirtual alarm clock (from setitimer())
27SIGPROFProfiling timer expired
28SIGWINCHWindow resize signal
29SIGIOI/O now possible (from fcntl())
30SIGPWRPower failure
31SIGSYSBad system call
34-64SIGRTMIN to SIGRTMAXReal-time signals; behavior is application-specific

We can use the custom signals SIGUSR1 and SIGUSR2. We just need to run them in the main through a goroutine. Here’s an example code:

package main

import (
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/sirupsen/logrus"
)

func main() {
	log := logrus.New()
	log.SetLevel(logrus.InfoLevel)

	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGUSR1, syscall.SIGUSR2)

	go func() {
		for {
			sig := <-sigChan
			switch sig {
			case syscall.SIGUSR1:
				log.SetLevel(logrus.DebugLevel)
				log.Warnln("Log level changed to DEBUG")
			case syscall.SIGUSR2:
				log.SetLevel(logrus.InfoLevel)
				log.Warnln("Log level changed to INFO")
			}
		}
	}()

	// rest of your application
	for {
		log.Debugln("debug log")
		log.Infoln("info log")
		time.Sleep(time.Second * 2)
	}

}

In another window, you can send signals through htop or kill -s SIGUSR1 <pid>. As shown in the image, you can see that the program’s log level is changed at runtime.


The WebP Cloud Services team is a small team of three individuals from Shanghai and Helsingborg. Since we are not funded and have no profit pressure, we remain committed to doing what we believe is right. We strive to do our best within the scope of our resources and capabilities. We also engage in various activities without affecting the services we provide to the public, and we continuously explore novel ideas in our products.

If you find this service interesting, feel free to log in to the WebP Cloud Dashboard to experience it. If you’re curious about other magical features it offers, take a look at our WebP Cloud Services Docs. We hope everyone enjoys using it!


Discuss on Hacker News