vxfs: inode points to old file

[ppmnt]

Recently we encountered some very strange behavior: after implementing changes to our web site jsp code base we usually touch all files. By modifying the last mod time it is ensured that the new file is being accessed when customers go to the site.

However, on two of our four servers when you visit the site in the browser, the old code was still being executed. A cksum showed that the files were identical on all 4 servers. When we copied all scripts to a different location and then back to their original location, thus overwriting all files (and assigning a new inode) the problem was fixed.

Has anybody run into this behavior? We can't quite figure out if it is OS related or a vxfs issue. Any ideas?

Stats:
OS - HP-UX 11.11
FS - vxfs (journaled)
Web server - Apache 1.3.27
Front end application (servicing jsp code) - Broadvision 6.0

Thanks!

[Bill Hassell]

If the program was never stopped, you can remove the file from the directory and it will appear to be gone, yet the inode is still busy and the code will be unchanged. By touching this particular filename or copying a new versions to the directory. a new file is created but the executing process in RAM is still the old code. Once the process finishes, the inode will truly be free and restarting the process will use the new code.

[Ross Zubritski]

Hi EVA,

Are you current on all LVM patches?

Regards,

RZ

[Jeff Schussele]

Hi Eva,

I'd be more inclined to believe the old code may have been cached somewhere by the application SW than I would believe this to be a vxfs/inode problem.
But nonetheless, you should insure that you are up to date on vxfs patch levels.

Was Apache & Broadvision completely stopped when you initially changed the code?
How about during the file "move"?

Rgds,
Jeff

[Jeff Schussele]

Hi Eva,

I wouldn't bother digging deeper on this. I think Bill has pegged it. IF the application or any SW that accesses this file was still running when it was replaced, you'll get the exact symptom you saw - it was still resident in memory. It will look like it's the right code in an ls or even a cksum, but that doesn't matter - the old code will still be in memory & *that's* what will be executed.

Moral => Stop ALL applications before updating anything that becomes memory resident.

You can get away with this for stuff that's read occasionally & not resident, but not in this case.

Rgds,
Jeff

[ppmnt]

Jeff,

Stopping all applications would imply taking the whole site offline everytime we fix a typo. The site is almost exclusively dynamic, hardly any static html content.

We have been running for several years with the touch being part of our release process and have never run into this issue since we started touching.

Also, that doesn't explain why it would still be in memory after the application was restarted, right? If it was the Broadvision that was caching it, it would be cleared once it is restarted, wouldn't it?

[Jeff Schussele]

OK, I can understand that position.
Then what you must do is use either the standard HP command fuser or the superior 3rd party command lsof to insure that this file in NOT in use at the time of update.
If you don't know for sure whether it's in use or not, it's going to be hit or miss. And apparently in the past you've hit....until this last time.

Rgds,
Jeff

[ppmnt]

lsof would be a good idea. Unfortunately I'm afraid it would be be nearly impossible to implement as part of the release process.

There are four Broadvision Interaction Manager servers. Let's say a releasa is a tar ball with 10 jsp files that will be overwritten. We use lsof to make sure they are not in use, then untar, touch, done.

With an average of about 130,000 visitors and 1.1 million page views per day, (1) it will be hard to have find a time when 10 files are not in use, and (2) it is very unlikely that they will not be accessed in the time between the lsof command and executing the untar.

This is turning into a headache.

Thanks for your input!
~Eva

[Jeff Schussele]

This is a good example of where MC/SG can help.
You can have an active/standby cluster & make the changes to the standby & when all is correct, fail the package over to the standby & start to upgrade the active-now-standby. If they're identical systems, you wouldn't have to fail it back until the next upgrade.
Downtime could easily be kept in the single-digit minutes - depending on pkg shutdown/startup times as low as 2-3 minutes.

My $0.02,
Jeff

[ppmnt]

Yes, we actually use MC/SG for a different part of the environment. With these servers however it is critical that at least during the day all 4 are available at all times. They are pegged as it is.

What I'm really wondering, though: Are you sure that resident memory is the root cause of our problem? Why did restarting BV not fix it?

[ppmnt]

The reason I'm asking is because above you said you thought Bill had "pegged it" with his reply. But one thing Bill said was: "A new file is created but the executing process in RAM is still the old code. Once the process finishes, the inode will truly be free and restarting the process will use the new code."

I just want to make sure I'm going in the right direction with revamping our release process instead of investigating the root cause...

Thanks!
~Eva

[Jeff Schussele]

Hi,

Couple of reasons:

1) As you stated - the old code was executing, but the ls & cksums showed the new code in the directory. The only way this could happen is if A) old code was still resident in memory or B) An application had the data cached.

2) inode table trouble VERY rarely just affects just 1 or 2 inodes. If you truly had an inode table problem, old code running would be the least of your problems.

Of course I can't be 100% sure, but I'm fairly certain that's what happened here. I don't know your application at all, of course, but maybe it wasn't BV that had the file open - maybe it was Apache. Have you run lsof against that file lately to see just what has it open?

To me it sounds like a procedural problem & knowing just *what* open the files in question is a valuable piece of info to properly plan the process.

Rgds,
Jeff

[ppmnt]

Jeff,

Thanks for being patient with me and still providing input.

Nothing but Broadvision runs on these servers. A user request goes from one of the web servers to one of the BV IM servers, where the jsps are, to a BV core server, to the database (and back). The IM servers have nothing else running on them.

I tried running lsof on the machine but just get:
lsof: can't read proc table info
I don't know my way around lsof that well, I'll read up on this error.

It does make sense that inode trouble would cause a lot more issues than what we are seeing. That's why I was wondering if it was cached somewhere else. Not in the application (that cached got completely flushed), but maybe on the lvm / journaled fs side?

[Jeff Schussele]

Hi,

lsof is HP-UX version-specific. I suspect you have an lsof compiled for another OS version. Search the forum for "lsof for 11i" I think you'll find a link to someplace where you can pull it down.

Try
fuser -u /path/to/file
until you get the correct lsof version. Problem with fuser is that it won't show you the process name nor PID like lsof does - just the user name.
I doubt it's journal cached as the buffers are flushed every so many seconds.

Rgds,
Jeff

[ppmnt]

Strangely enough the following command:

$ find . -type f |xargs sudo fuser -u

executed in the scripts directory on one of the 4 IM servers doesn't list ANY files as being open.

???

[Jeff Schussele]

Hi Eva,

Just try

fuser -u /path/to/scripts_dir

Just point fuser to the dir that contains the scripts.

I was incorrect earlier in fuser output it will look like

2427o(root) 7645c(user1)

or such. The PID & usernames are obvious, the letters aren't as much - here's the denotation

o = has file open
c = user has this as current dir
r = file is it's root dir - as in the chroot command
m = file is memory mapped
t = file is its text file

Then you can take the PID & do a

ps -ef | grep 2427

to see just *what* the process is.

Good Hunting,
Jeff