will non-system file system full cause server reboot?

[Gary Yu]

Hi all,

I have a A500 server with only VG00 on it. There's a cron job doing "tar/gzip" backup every night to a file system "/mybackup" on /dev/vg00/lvol09. lastnight, the gzip after tar failed because there's no free space left( I would assume gzip will allocate some temp space for itself), then the server rebooted! there's no entries at all in the OLDsyslog, I'm just wondering, will 100% full on such file system (non /etc, /opt/, /var etc.) cause server reboot? I had file system full problems before on other servers, but never saw it caused reboot...

thanks,
Gary

[Gary Yu]

a correction, I mean there's no entries at all [ related to this file system full problem ] in the OLDsyslog.log

[Matt Hearn]

I've never seen it happen myself, but that doesn't mean it's not possible. UNIX is a strange critter.

And no, gzip uses whatever space is in the filesystem you're currently using. So if you've got an fs with 100GB, and you need to compress a file of 90GB, you probably won't be able to do it (unless the compressed file somehow shrinks down to under 10GB).

On the other hand, isn't there an option in tar that does compression on the fly and produces a "tgz" file? Check the manpage for tar and see how the "z" option works.

[RAC_1]

What does the /etc/shutdownlog says?

Also do you get anything in /var/adm/crash or in
/var/tomstones/ts99 file?

Also check with lastb command.

[Bernhard Mueller · Accepted solution]

Gary,

you are right, it *should* not reboot. However gzip drags a lot of CPU and if the disk constantly complained... you never know.

However, I take it /dev/vg00/lvol9 is on the boot disk(s).

Therefore, if you can move this backup directory to any other disk, do it.

You may want to enhance your backup script so that gzip is not started if the tarfile is larger than 60% of lvol9 (gzip removes the uncompressed version only after zipping succeeded.)

Or consider zipping into a second (smaller) directory by redirection (and maybe move zipped file back after success).
gzip /something/tarfile > /other_fs/tarfile.gz
mv /other_fs/tarfile.gz /something

If you only have one CPU start the gzip with a "nice value" so that other processes get a chance.
nice -n 19 gzip /something/tarfile > /other_fs/tarfile.gz

Regards,
Bernhard

[Bernhard Mueller]

Matt,

for -z you need to download/compile GNU tar.

Regards,
Bernhard

[Gary Yu]

thanks all for the quick response, I have already updated the script to avoid this from happenning again.

the /etc/shutdownlog says:
" Reboot after panic: , isr.ior = 0'340002.0'3d8f7ef8"

and I don't have enough knowledge to interpret the /var/tombstones/ts99 and files in /var/crash/crash.0, any instruction?

thanks,
Gary

[Jean-Louis Phelix]

Hi,

You should contact HP because in most case this kind of panic is an HPMC (hardware failure).

Regards.

[Bernhard Mueller]

Gary,

contact HP and open a HW call, they will instruct you how to proceed with the crashdump.

Regards,
Bernhard

[RAC_1]

Seems that it was a crash due to system panic.

You can check it as follows.

Check what your have in /var/adm/crash.
(I assume savecrash has been configured to save crash files)

adb -m /var/adm/crash/vmunix /vrad/adm/crash

Once on adb prompt do msgbuf +8/s

Check for stack trace.

Also you can crashinfo utility for this.

[Gary Yu]

Hi all,

I have opened a service call with HP, and from their analysis on the ts99 file, it seemed that one of the CPU didn't reported to the hardware monitor (GSP I guess?) which caused the reboot.

But both CPUs are functioning normally right now, maybe like Bernhard said, the gzip are dragging lots of CPU and I/O, which triggered an implicit defect on that processor. Anyway, HP will come and replace that processor soon.

thanks all!
Gary