Hot-swapping a log file on a running process w/ GDB w/o stopping it

General description – what the hell will we try to do here:

When you find yourself chasing memory leaks, or Heisenbugs for a long time chances are you will have a huge debug log. If the disk gets full and you need only the logs at the end when the problem raises it’s ugly head, use the technique described here.

 

The technique involves:

• closing the file handle where the process writes the logs currently – int creat(const char *path, mode_t mode);
• creating a new log file and passing to the process for writing – creat(filename, mode) which is equivalent to open(path, O_WRONLY|O_CREAT|O_TRUNC, mode);

Step-by-step walk-through:

Verify that your suspicions about the log file are correct and it is indeed bloated:

[[email protected] ~]# cd /var/log/
[[email protected] log]# ls -alht
total 90G
-rw-r--r--.  1 root   root     90G Nov 17 04:05 mindaugas_debug.log
-rw-------.  1 root   root    412K Nov 17 04:05 secure
-rw-------.  1 root   root     20K Nov 17 04:04 maillog
-rw-r--r--.  1 root   root     16M Nov 17 04:04 lastlog
-rw-rw-r--.  1 root   utmp    990K Nov 17 04:04 wtmp

Delete the miscreant:

[[email protected] log]# rm -f mindaugas_debug.log
[[email protected] log]#

Check process file descriptors (fd’s):

[[email protected] ~]# ls -alht /proc/1368/fd
total 0
lrwx------. 1 root root 64 Nov 17 04:11 0 -> /dev/pts/2
l-wx------. 1 root root 64 Nov 17 04:11 1 -> /var/log/mindaugas_debug.log (deleted)
lrwx------. 1 root root 64 Nov 17 04:11 10 -> socket:[36693705]
...

Attach gdb to the process, close the file handle to the old file and attach the new one:

$ gdb -p 1368
... [skipped outout]
(gdb) p close(1)
$1 = 0
(gdb) p creat("/var/log/mindaugas_debug.log", 0600)
$2 = 1
(gdb) q
The program is running. Quit anyway (and detach it)? (y or n) y
...

This, my friends, is all.