General description – what the hell will we try to do here:
When you find yourself chasing memory leaks, or Heisenbugs for a long time chances are you will have a huge debug log. If the disk gets full and you need only the logs at the end when the problem raises it’s ugly head, use the technique described here.
The technique involves:
- closing the file handle where the process writes the logs currently –
int creat(const char *path, mode_t mode)
; - creating a new log file and passing to the process for writing –
creat(filename, mode)
which is equivalent toopen(path, O_WRONLY|O_CREAT|O_TRUNC, mode)
;
Step-by-step walk-through:
Verify that your suspicions about the log file are correct and it is indeed bloated:
[[email protected] ~]# cd /var/log/ [[email protected] log]# ls -alht total 90G -rw-r--r--. 1 root root 90G Nov 17 04:05 mindaugas_debug.log -rw-------. 1 root root 412K Nov 17 04:05 secure -rw-------. 1 root root 20K Nov 17 04:04 maillog -rw-r--r--. 1 root root 16M Nov 17 04:04 lastlog -rw-rw-r--. 1 root utmp 990K Nov 17 04:04 wtmp
Delete the miscreant:
[[email protected] log]# rm -f mindaugas_debug.log [[email protected] log]#
Check process file descriptors (fd’s):
[[email protected] ~]# ls -alht /proc/1368/fd total 0 lrwx------. 1 root root 64 Nov 17 04:11 0 -> /dev/pts/2 l-wx------. 1 root root 64 Nov 17 04:11 1 -> /var/log/mindaugas_debug.log (deleted) lrwx------. 1 root root 64 Nov 17 04:11 10 -> socket:[36693705] ...
Attach gdb to the process, close the file handle to the old file and attach the new one:
$ gdb -p 1368 ... [skipped outout] (gdb) p close(1) $1 = 0 (gdb) p creat("/var/log/mindaugas_debug.log", 0600) $2 = 1 (gdb) q The program is running. Quit anyway (and detach it)? (y or n) y ...
This, my friends, is all.