LSM+ADVFS problem

[boris_16]

i have lsm volume with mirror in advfs domain.
df -k is domain 50% Capacity.
showfset domain 160000 files

can't create file in this domain.

System log write: file system is full.

DS10,Digital UNIX V4.0F, patch kit 8

[Hein van den Heuvel]

Boris,

Welcome to the Tru64 Forum.

Please look around with 'man advfs'
or online at:
http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51B_HTML/MAN/MAN4/0124____.HTM

AdvFS is log based file system, allowing for transactional meta data write and thus for quick recovery on crashes.
So there has to be space for the real user data, as well as for the transaction log.
Sound like you transaction log is full. It is more or less a circular log, but I suppose certain usages can fill it up and you may want to configure a large one.

Suggestions:

Use:
/sbin/showfdmn [-k] domain
This will show you how big the log is for a given domain, how much is used, and where thelog is.

Try:
/sbin/advfs/fixfdmn [-n]
This may well 'clear' the log.
Check again with showfdmn

Consider:
/sbin/advfs/switchlog domain_name vol_id

Consider:
/sbin/advfs/nvlogpg
... to see how the log was used, what keeps it in use.

You can set the log size on mkfdmn, but I do not think you can re-size it after the fact.


Good luck,
Hein.




fixfdmn

[jim owens_1]

I believe you mislead Hein by saying "System log write: file system is full." As I'm not aware of any case where advfs prints a message that begins "System log write:", I expect you meant you saw a file system full message in the system log.

A too-small log can cause problems in advfs, but create failures are not one of them. I've copied this from the advfs quick reference:

Domains containing a large number of files created on versions prior to Version 4.0D (and very rarely on later Version 4 operating systems) may have inadequate metadata allocation. This causes ENOSPC errors to be returned to applications even though running the df or showfdmn command shows there is still space available in the domain. To avoid this for large files, preallocate metadata with the -x and -p options of the mkfdmn command.

Your 160000 files have filled the BMT. The only fix is to create a new domain. Unless you have enough spare storage to create a new domain and copy directly to it from your existing domain, you will have to backup your data, destroy the existing domain, and create it again with more metadata preallocated.

Assuming you have to do it the hard way, I recommend you backup all your data and upgrade the system to V5.1B, then create new domains and restore your data. The domain versions created in V5.1B (Domain Version 4) have improved BMT allocation plus other improvements and do not require the preallocation by mkfdmn to support large numbers of files.

[Hein van den Heuvel]

Thanks Jim!

Yeah, I could not find that error message either. Boris may have given a transcription instead of teh exact message text/place.

I also had a hard time in my mind to make a full log create a permanent error, but though that maybe with the old V4.0F there might have been an isseu.

If Boris can at all upgrade to a 5.1(B), then he really should.

With the LSM mirrored data, maybe recreating the domain is less tricky than otherwise?
- make extra good backup
- split volume into 2 plexes
- re-create new domain on 1
- copy files
- recreate mirror (be sure to go the right way!)

Also, maybe using ADDVOL some (temporary) relieve can be found after a subsequent 'balance' command.

The post-5.0 addvol manpage addresses pre-5.0 BMT concerns. For example online in:

http://btrcx1.cip.uni-bayreuth.de/cgi-bin/manpages/addvol/8

http://btrcx1.cip.uni-bayreuth.de/cgi-bin/manpages/balance/8


Good luck,
Hein.

[jim owens_1]

Either of Hein's solutions might work but I have not tried them. There are a few good and bad things I know about the addvol option:

Each advfs volume has a unique BMT so the "-x" and "-p" can be used on the addvol to preallocate a larger metadata space without making a new domain.

However, an interesting problem can occur with the original (source) volume. Because advfs often needs BMT space on the source to record a move or delete operation, it can fail to perform the "rm", "balance", "migrate", "rmvol", etc.

I suspect only files that are 1 extent can be migrated between volumes if the source BMT is full. The advfs commands don't have a way to continue past an error so my only guess how to get past this problem would be to write a script to find single-extent files and call /usr/sbin/migrate for each file. That should free enough source BMT space to allow multi-extent files to be moved by balance.

[jim owens_1]

Oh, I forgot to say the most important thing to do is follow Hein's advice before you try any fixes...

- make a good backup