Your logrotate config is probably doing nothing
The default logrotate strategy is to rename(2) the log file and then
create a fresh one in its place. This works because most well-behaved daemons
either accept a SIGHUP and reopen their log, or use a library that
checks the inode. The rotation config carries a postrotate block to
send that signal.
Plenty of software does neither. It opens the file once at startup, keeps the
file descriptor forever, and writes to it with O_APPEND. Rename the
file out from under it and nothing breaks — the descriptor still points at the
same inode, now living at a new path. The daemon keeps writing to
app.log.1. The newly created app.log stays at zero
bytes forever.
The failure is completely silent. logrotate exits 0. The directory
listing looks healthy — you get your seven rotated files on schedule. It is only
when you go looking for last Tuesday's error that you notice six of the seven
are empty and the seventh is 4 GB.
How to tell
Ask the kernel which file the process actually has open. If the target has
(deleted) after it, or points at a rotated name, you have the bug:
ls -l /proc/$(pgrep -f myapp)/fd | grep '\.log'
The other tell is size distribution. A healthy rotation set has roughly comparable file sizes. A broken one has one enormous file and a pile of zero-byte ones.
The fix
Use copytruncate. Instead of renaming, logrotate copies the contents
to the rotated name and then truncates the original file in place. The inode
never changes, so the daemon's file descriptor stays valid and it keeps writing
to the same, now-empty, file.
/var/log/myapp/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
copytruncate
su myapp myapp
}
Two notes on that block. delaycompress matters more than it looks:
with copytruncate there is a brief window between the copy and the
truncate, and compressing immediately can catch a partially-written line.
Delaying compression by one cycle avoids it.
And su is not optional once the daemon runs as an unprivileged user.
logrotate refuses to rotate a directory it does not own unless you tell it which
identity to assume — another failure that is quiet, because it shows up in the
logrotate log rather than anywhere you look.
The tradeoff
copytruncate is not free. Any line written between the copy and the
truncate is lost. For an access log at a few hundred lines per second, that is a
handful of records per day, which is usually acceptable. For an audit log where
every record matters, it is not — there you need to fix the daemon or put a real
log shipper in front of it.
Check rotation the same way you check backups: not by confirming the job ran, but by opening the output and finding data in it.