Operating System - HP-UX
1748158 Members
4088 Online
108758 Solutions
New Discussion юеВ

Question regarding refcount of deleted but still open files

 
Knut Eckstein
Advisor

Question regarding refcount of deleted but still open files

Dear all,

during the last weeks I've been working on a port of The Coroners Toolkit (TCT) to HP-UX 10.20. Details about the port can be found in a message that I posted to the security forum last week.

During the port the following problem turned up:

When deleting a file that is still opened by a process, HP-UX does
delete the directory entry but does not decrement the refcount to zero
in the on-disk inode like other Unices do. Therefore, the ils program
cannot report such a file as it looks like a normal file due to the
refcount being not zero, but one.

Why does this behaviour differ from other Unix implementations? As far
as I know, all other platforms that TCT is available for (Solaris,
*BSD, Linux), do set the refcount to zero for a file that has been
deleted, but is still opened by a process. There the ils program can
be used to report such files, that are typically generated by an
intruders who want to cover their tracks.

Any light on this issue would be greatly appreciated,

Knut

PS: As has been discussed in the TCT thread in the security forum, a workaround could be constructed using a combination of lsof and some scripts, but I'm more interested in why HP-UX behaves in this way and what was the reasoning behind implementing file deletion in this way...
5 REPLIES 5
harry d brown jr
Honored Contributor

Re: Question regarding refcount of deleted but still open files

You would want a deleted file's inodes released to the free inode table while it's still open?

live free or die
harry
Live Free or Die
harry d brown jr
Honored Contributor

Re: Question regarding refcount of deleted but still open files


And the most IMPORTANT difference between Solaris/BSD/Linux and HPux/AIX, is that HPux and AIX have Journaled File Systems, and Solaris & BSD & Linux do not.

live free or die
harry
Live Free or Die
Knut Eckstein
Advisor

Re: Question regarding refcount of deleted but still open files

Harry,

let's first clarify a potential misunderstanding. I'm talking about the behaviour of the Berkeley FFS, that is in case of HP-UX hfs. On Irix its called efs, on Solaris and *BSD it's called ufs. I'm not talking at all about journaled filesystems. You are right that beginning with 10.20, HP also offered a journaling filesystem called vxfs. This is technology from Veritas that you can buy as an add-on for Solaris and nowadays also for Linux. AFAIK the full Veritas functionality has to be bought as an add-on for HP-UX too, my 10.20 manuals tell me the product is called OnlineJFS.

I forumulated this more clearly in my other post in the security forum, which I didn't want to repeat here because I thought that might be considered an impolite form of "cross-posting".

Regarding your first answer, I'm not "asking" for a change in behaviour, I'm asking for an explanation for an observation that I made. The observation is that HP's FFS implementation differs from the *BSD and Solaris implementation with respect to the treatment of files that are deleted while still being open. With *BSD and Solaris, the directory entry is removed and the refcount of the on-disk inode is decremented to zero. The inode is still marked "in use", so it's not put on the "free list". The same behaviour can be observed with Linux' ext2 filesystem.

This behaviour is used by TCT to easily detect files that are still open, but don't show up in any directory listings because the intruder already deleted them.

Regards,

Knut
Frank Slootweg
Honored Contributor

Re: Question regarding refcount of deleted but still open files

I find it rather strange that the BSD behaviour is different from that of HP-UX's HFS (I don't *know* that it is, I am just taking your word for it.). After all, HP-UX's HFS is just a port of the Mc Kusick (sp?) Fast File System (FFS), and I assume that is what BSD is also (still) using.

Having said that, I consider the HP-UX behaviour to be the 'right' behaviour. If you do decrement the link count to zero while the file is still open, then how do you recognize an 'UNREF'/'orphan' file, i.e. a file with an inode and (a) data block(s) but without a directory? After all, fsck(1M), again a port of the FFS code, *does* recognize UNREFs.

If you have access to Usenet Newsgroups, you may want to (re)ask this question in one of the generic UNIX groups, with crossposts to comp.sys.hp.hpux et al.
Knut Eckstein
Advisor

Re: Question regarding refcount of deleted but still open files

Frank,

first of all thanks a lot for answering my question.

yes, *BSD ufs filesystem is nearly identical to the McKusick FFS in the same way as HP's hfs. The only difference that I found during my port is that hfs uses different magic numbers, added support for 64 bit UID/GID fields and that one or two convenience macros that were missing in the HP-UX include files. I compared the include files of FreeBSD and HP-UX 10.20 and they were almost identical, so I could safely take over those macros and everything compiled cleanly.

AFIAK, the recognition of orphaned files is done by going through all directories of a filesystem counting the occurrance of inode numbers and comparing that to the refcount on the inodes. In the beginning I thought, the fact that the link count is zero but the data block field is not would we used for recognition of orphaned files, which would not have worked in the case of HP-UX. To find out what really happens, I removed a file which was still open, verified that the refcount was still 1 and then switched off my HP-UX box. Upon restart, fsck correctly identified the file as orphaned. On the other FFS implementations, it's probably the same, as a system could crash after a direntry has been removed, but before the refcount had been adjusted, so *relying* on refcount==0 for detecting orphaned files as I had originally thought, is certainly not a good idea. If you watch fsck output carefully, you will see output similar "Phase 2: Scanning directories..."

I personally think that *BSD and Solaris do it the right way, because the refcount should be strictly tied to the number of directory entries. I.e. HP-UX creates a slight inconsistency by removing the (last) direntry but keeping the refcount at 1.

When I discussed my findings with Wietse Venema, the author of this part of TCT, he was also very surprised, as it is contrary to what he has experienced on all other TCT platforms: SunOS, Solaris, *BSD and BSDI. He urged me to make sure that I typed sync repeatedly before determining the refcount with icat, because he initially didn't believe in the validity of my test results...

Thanks also for you Usenet suggestion. I already posted this question in comp.sys.hp.hp-ux but didn't get back a response. The Google Groups archive shows that HP engineers have been very active and helpful in the hpux groups in the early nineties, but now I assume they are concentrating on these forums here...

Again, thanks a lot for taking your time,

Knut