logrotate shooting itself in the foot

[Ralph Grothe]

Hello Linux Experts,

I have a heartbeat cluster made of RH FC3 nodes that serve a HA webserver.
The webserver's logs need to be rotated exactly at the start of a new month.
I know that my logrotate config file for httpd works as soon as I tidy up logrotate's own status file.
The damned thing is that logrotate itself is producing a broken status file that it is rejecting on the next cron.daily triggered rotation.
I already replaced the logrotate RPM that came with FC3 with a newer release that I compiled from the sources.
But this didn't help.
I cannot find out why logrotate is producing a status file with syntax errors.
Here's what I mean

# logrotate -d /etc/logrotate.d/httpd
reading config file /etc/logrotate.d/httpd
reading config info for /var/www/apache/httpd/logs/*log \
/app/sbc/aDREG.d/aDREG.log
error: bad line 20 in state file /var/lib/logrotate.status
*** glibc detected *** double free or corruption: 0x081294f8 ***
Aborted


# cat -n /var/lib/logrotate.status|sed -n 18,22p
18 "/var/www/apache/httpd/logs/ssl_error_log" 2006-10-31
19 "/var/www/apache/httpd/logs/suexec.log" 2006-10-31
20 "
21 /app/sbc/aDREG.d/aDREG.log" 2006-10-31
22 "/var/log/mgetty.log.tty[^.]" 2006-10-31


As you can see in line 20 there is a single opening double quote, then a new line with the rest of the entry which is causing logrotate to reject the broken status file.
I never touched the status file,
and as soon as I remove it and rerun logrotate without the -d option it will diligently be rotating the webserver logs.

Any ideas?

Regards
Ralph

[Ivan Ferreira]

It's extrange the line 20. What the configuration of the httpd logrotate file and the full listing of the directories being rotated? Are you using a "\" in your logrotate file? As I see in the other logrotate files, a \ is not used even when the list of directories is very large, for example the mgetty config file.

Try removing the \ if you have it.

[Ralph Grothe]

Hi Ivan,

as far as I can see I haven't backslash escaped a single double quote.
The logrotate config file for the webservices is pretty simple.
I only used a backslash to escape the newline.
But maybe that's the problem, and logrotate doesn't understand this common Unix flatfile nl escape idiom?



$ cat -v /etc/logrotate.d/httpd
compress

/var/www/apache/httpd/logs/*log \
/app/sbc/aDREG.d/aDREG.log {
missingok
notifempty
sharedscripts
postrotate
/bin/kill -USR1 `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || tru
e
endscript
}

[Ivan Ferreira]

It could be because the problem start just between logs/*.log and aDREG.log

[Ralph Grothe]

Yes, probably that's the cause.
I changed the logrotate.d/httpd config file to join the separate target lines to one line like such:
(damned, ITRC style sheets seem to separate them again in my posting)

# head -4 /etc/logrotate.d/httpd
compress

/var/www/apache/httpd/logs/*log /app/bms/aDREG.d/aDREG.log {
missingok


Then I fixed the old broken logrotate.status file in line 20, so that the webservers' logs aren't deemed due for rotation on next cron.daily execution of logrotate (remember I need to rotate on month boundaries exactly, and I already manually rotated the webserver logs today)

When I now dry-run logrotate
# logrotate -d /etc/logrotate.d/httpd

the spit out comments look ok.
I hope this fixed it, and you don't mind me not assigning points until tomorrow, after I made sure that cron.daily/logrotate not again got a hick up.

[Ivan Ferreira]

You can use logrotate -f to force the rotation of the logs if you want to test your modifications.

[Ralph Grothe]

No, no, that's what I aim to prevent for now.
Otherwise I once again would have to manually grep and concat and zip the logs.
Well, now at the beginning of the month they are still manageable.
But on a 4 GB logfile by the end of month the grep and gzip run quite a while, taking uneccessarily CPU scheduling slices.
Besides, I'm confident that the rotation will work next time.
The dryrun looked promissing.

[Steven E. Protter]

Shalom Ralph,

I've encountered similar results in HA environments.

I build an NFS server to hold my logs. Since dealing with hardware issues, my HA cluster environment, centos has not had a single minute of downtime.

The problem with that is its being used and logrotate can't disrupt the service that has a file handle on those log files so it can rotate and clean them.

I literally had to create a 10 second service outage in order to enable logrotate to get a hold on my log files and rotate them.

Note that Q&A issues, which are are also experiencing here are common with Fedora Releases. Thats why I went Centos which is a recompile (Like Oracle Enterprise Linux) of the Red Hat Enterprise Code. This reduces but does not eliminate Q&A issues.

SEP

[Ralph Grothe]

Hello again,

logrotate is really misbehaving.
On cron.daily's last invocation it at least didn't produce the nasty error anymore,
but now it rotated the webserver logs though they weren't due for rotation.
At least that was what I had thought after having mended the broken status file yesterday.
Now I know that I should have appended 2006-11-1 instead of 2006-10-31, as it read before, to the affected filenames.
However, by now I think the whole fuss of getting logrotate treat those files as I wish isn't worth the effort.
It would have been quicker if I had put my manual grep, split, rejoin and zip in a wee script and released those webserver logs altogether from logrotate reign.


SEP,

privately I also have moved to CentOS
because they offer binary compatiple RPMs with RHEL, and better, have many yum repository mirrors wherefrom I can run a weekly rsync.
The only problem is that this cluster was once set up on Fedora, and you know how troublesome it is to upgrade a running system.
I also have thought about a centralized log solution with maybe syslog-ng.
But here the same applies,
let alone DMZ restrictions.