Who is cluttering /var/tmp with char dev files rdskAAA* ?

[Ralph Grothe]

Hello,

it seems I have a berserk process that is creating character special files by the minute.
Since the devices are only one block in size we don't run into space limits that fast.
However, I am not sure about the inode count,
but assume, since it is an vxfs, this isn't hit either that soon.
Nevertheless, there had been accumulated over 100,000 device files.
They have all the major 203

# ll -rt /var/tmp/rdskAAA*|tail -3
crw------- 1 root root 203 0x001000 Aug 8 13:50 /var/tmp/rdskAAAa24366
crw------- 1 root root 203 0x001000 Aug 8 13:50 /var/tmp/rdskAAAa24379
crw------- 1 root root 203 0x001000 Aug 8 13:50 /var/tmp/rdskAAAa24389


# lsdev 203
Character Block Driver Class
203 -1 sctl ctl



First I suspected an OmniBack process to be the culprit,
and we killed that and removed all rdskAAA* files.

But when I now find for them again that many in quite a short period

# find /var/tmp -xdev -type c -name rdsk\*|wc -w
361

and since OmniBack has been killed there's only some stm proc accessing /var/tmp now

# lsof +d /var/tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sysstat_e 5771 root 8w REG 64,0x8 0 29107 /var/tmp/sysstat_em.fmt

# ps -fp 5771
UID PID PPID C STIME TTY TIME COMMAND
root 5771 1 0 Jul 5 ? 1:04 /usr/sbin/stm/uut/bin/tools/monitor/sysstat_em


I couldn't see fitting entries in syslog.log,
and the last event.log entry from EMS dates back to July 5th.

Can anyone tell me if it really is STM that creates them, and for what purpose?

On our other servers I cannot observe these strange character devices.

Regards

Ralph

[Ivan Krastev]

What is your OS version ?
For 11.00 here is recomendations:

"The Sep 2002/A.34.00 version of OnlineDiag/STM contains further changes
to support the exclusion of removable media devices from polling.
Therefore, a recommendation was made to install the Sep 2002/A.34.00
version of OnlineDiag/STM, along with the (required) EMS core patch,
PHSS_26871 or later. Doing so resolved the problem. "

regards,
ivan

[Ralph Grothe]

Hi Ivan,

the OS is B.11.11
For STM on it I only found PHSS_34085 (which isn't installed) when querying HPs' patch database for sysstat_em.
But in its Readme I couldn't see mention of the symptom that I observe.

http://www4.itrc.hp.com/service/patch/patchDetail.do?patchid=PHSS_34085&sel={hpux:11.11,}&BC=main|search|

I have applied the latest Support+ bundle on July 5th

# swlist -l bundle -a install_date -a title GOLD\* HW\* OnlineDiag
# Initializing...
# Contacting target "gomera"...
#
# Target: gomera:/
#

GOLDAPPS11i 200707050948.26 Applications Patches for HP-UX 11i v1, June 2007
GOLDBASE11i 200707050948.26 Base Patches for HP-UX 11i v1, June 2007
HWEnable11i 200707051008.59 Hardware Enablement Patches for HP-UX 11i v1, December 2006
OnlineDiag 200707051021.01 HPUX 11.11 Support Tools Bundle, June 2007


Could I have introduced a new bug with it?

[Ivan Krastev]

Use tusc - http://h21007.www2.hp.com/portal/site/dspp/PAGE.template/page.document?ciid=61086d6e1de021106d6e1de02110275d6e10RCRD

to see what is the exact process generating these files.

If this is the same (EMS) it's possible bug or misconfiguration due to removed hardware.

regards,
ivan

[TwoProc]

I've seen them on my 11.11 systems as well. It would be nice to know what they are...

[Todd McDaniel_1]

Ralph,

I would do 3 things first.

Stop and start STM a few times...and see if that clears it up.

# /sbin/init.d/diagnostic stop|start

If that doesnt help, then reload STM to the latest version or fall back to the last good version you had before the current one.

Upgrade OnlineDiag to the latest version.


If that doesn't do it, then I think you need to call HP.

[Steven E. Protter]

Shalom,

I think I recall NFS or Samba leaving files like this behind. I'd be leaning toward NFS.

Do they correspond with disk devices that are part of NFS shares?

SEP

[TwoProc]

Re: NFS shares.

Those devices are not involved in NFS shares on my servers, especially the worst one, which isn't an NFS server at all.

[Sameer_Nirmal]

The possiblity of a bug or re-introducation of previous bug couldn't be ruled out here. However I am curious to know what if you move /var/tmp/sysstat_em.fmt to somewhere else, would you get those rdskAAA* files created?

There was a known issue (JAGae59420)with sysstat_em earlier where sysstat_em should delete the /var/tmp/sysstat_em.fmt file after event generation and it was fixed in the Dec'04 release.

[Ralph Grothe]

Ivan,

I am kind of despairing.
Whenever I issue an lsof on the directory /var/tmp (belongs to /var FS) I always get sysstat_em popping up as the only proc that has files open thereon.

# lsof +d /var/tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sysstat_e 5771 root 8w REG 64,0x8 0 29107 /var/tmp/sysstat_em.fmt

So I have been tusc-ing only open() and mknod() syscalls by this PID (also for possible sysstat_em forked children), because I don't know of any other syscall that could create a new char device.
But all I can see is that this proc seems to multiplex for pending input streams by a select() (probably on its listening socket),
then rereads its config file, then opens what looks like some lock file to me, and finaly opens its /dev/diag device, only to reenter the multiplex cycle.

# tusc -f -s open,mknod 5771
( Attached to process 5771 ("/usr/sbin/stm/uut/bin/tools/monitor/sysstat_em") [32-bit] )
select(2048, 0x77ff0cf4, 0x77ff0978, 0x77ff0a78, 0x77ff0f04) ............. [sleeping]
open("/var/stm/config/tools/monitor/sysstat_em.cfg", O_RDONLY, 0666) ..... = 7
open("/var/stm/data/.slicdata", O_RDWR, 0666) ............................ ERR#2 ENOENT
open("/dev/diag/diag2", O_RDONLY, 060100) ................................ = 9
select(2048, 0x77ff0cf4, 0x77ff0978, 0x77ff0a78, 0x77ff0f04) ............. [sleeping]


Please, could you tell how I can find out which process is creating these char devices?

Since yesterday over 3000 files have been created.
And they get created in rather short intervals as can be seen by this tail


# ll -rt /var/tmp/rdskAAA*|tail
crw------- 1 root root 203 0x001000 Aug 9 08:48 /var/tmp/rdskAAAa28188
crw------- 1 root root 203 0x001000 Aug 9 08:48 /var/tmp/rdskAAAa28197
crw------- 1 root root 203 0x001000 Aug 9 08:48 /var/tmp/rdskAAAa28210
crw------- 1 root root 203 0x001000 Aug 9 08:48 /var/tmp/rdskAAAa28220
crw------- 1 root root 203 0x001000 Aug 9 08:49 /var/tmp/rdskAAAa28233
crw------- 1 root root 203 0x001000 Aug 9 08:49 /var/tmp/rdskAAAa28242
crw------- 1 root root 203 0x001000 Aug 9 08:49 /var/tmp/rdskAAAa28260
crw------- 1 root root 203 0x001000 Aug 9 08:50 /var/tmp/rdskAAAa28269
crw------- 1 root root 203 0x001000 Aug 9 08:50 /var/tmp/rdskAAAa28282
crw------- 1 root root 203 0x001000 Aug 9 08:50 /var/tmp/rdskAAAa28292


SEP and John,

as for NFS (you both mentioned it),
this box neither exports nor mounts NFS shares.

Todd,

in /etc/opt/resmon/lbin/monconfig
(gosh, who thought up this arcane path for a utility?)
I shutdown EMS, and I also stopped diagmond by its init script

# /sbin/init.d/diagnostic stop;sleep 20;UNIX95= ps -C diagmond
PID TTY TIME CMD

Well, at least for the last half an hour since I executed these commands,
no new device files have been created.
But as soon as I restarted the diagnostic init script 7 new files were created.

I also attached tusc for open and mknod calls to the pid of diagmond and children,
but couldn't see any opens on /var/tmp.

So it really looks as if some frenzied STM proc is causing them.

Because I hadn't observed this before I applied the DIAGNOSTICS patch of last Support+ CD release last month,
I now suspect this to have introduced this behavior.