Impossible ip address in wtmp

[Gareth Tunstall]

I have a problem with wtmp entries.

The machine in question (HP-UX 11.11) is accessed from 2 different sites. I had a query from the manager from Site A as to why are his users ids being used at Site B (120 miles away). He has spoken to the owners of the ids in question who have not shared their passwords etc. I am at a third site and the actual server at a fourth.

Well a bit of last -Rf, awk, etc later I have a script to cross check sites/logins/ip addresses and find several instances of ids used at the 'wrong' site, both ways - hmmm

Today looking at wtmp files from last week I find an impossible entry.

A normal user, I can see other logins for them at Site B in wtmp, logged in from a machine kept in a locked room on my site, and (this is the best bit) disconnected from the network since February!!

This is not an old wtmp file, they are cycled daily, so it's not an entry from last year etc.

Anyone got any ideas or heard of any bugs of this sort ?

DETAILS

Numbers/names changed to protect etc etc ...

host04# last -Rf /var/adm/wtmp-5 | grep userl
userl pts/tE myhost Thu May 6 14:23 - 16:00 (01:37)
userl pts/ttb 155.223.160.44 Thu May 6 10:49 - 10:56 (00:07)

host04# /usr/sbin/acct/fwtmp userl ttb pts/ttb 23334 7 0000 0003 1083836948 May 6 10:49:08 2004 155.223.160.44 155.223.160.44
userl ttb pts/ttb 23334 8 0000 0000 1083837401 May 6 10:56:41 2004
userl td pts/tE 20750 7 0000 0003 1083849780 May 6 14:23:00 2004 99.9.194.5 myhost
userl td pts/tE 16825 8 0000 0000 1083857988 May 6 16:39:48 2004

Connection is via telnet.

The customer network is 155... and now has no routing to the 99.9.194. network anymore - the link is physically dead. I have even checked the 'myhost' machine ('coz I'm paranoid) and there has been no login since February either.

[Steven E. Protter]

Very interesting.

Trusting that the link is dead and the IP address is impossible, there are still a few possibilities:

1) Someone has manually altered the wtmp file in an attempt to cover their tracks. This attempt was clumsy but is typical of hackers, or inexperienced systems administrators. In the case of an admin, perhaps a mistake was made and the admin doesn't want to take responsibility.

Tripwire is a good tool to at least warn you that these important audits have been altered.

2) Someone in the networking department fired up a router for testing.

3) Someone is logging in from a server that has an old DNS database running on it containing support for the dead link.

I can't say which of the possibilities is true, but perhaps the ideas will spark you towards the solution.

Systems Administrators are supposed to be paranoid.

SEP

[MarkSyder]

Just because you're paranoid don't mean they're not out to get you.

Mark Syder (like the drink but spelt different)

[Paula J Frazer-Campbell]

Hi

What does trece route tell you about these users?

Paula

[Gareth Tunstall]

Further info - curiouser and curiouser

Sorry for all the traces - they'll need copying to a text editor to make sense.

last -R userl output
userl pts/tY 155.223.160.44 Mon May 10 12:39 - 17:37 (04:58)
userl pts/tcb 155.223.160.44 Mon May 10 08:22 - 11:58 (03:35)
userl pts/ty 155.223.160.44 Fri May 7 11:08 - 12:57 (01:49)
userl pts/tE myhost Thu May 6 14:23 - 16:00 (01:37)
userl pts/ttb 155.223.160.44 Thu May 6 10:49 - 10:56 (00:07)
userl pts/tg 155.223.160.44 Wed May 5 14:12 - 17:48 (03:36)
userl pts/tz 155.223.160.44 Wed May 5 12:18 - 12:20 (00:02)
userl pts/t2 155.223.160.44 Wed May 5 09:56 - 12:16 (02:20)
userl pts/tx 155.223.160.44 Tue May 4 15:17 - 17:31 (02:13)

Login on Thu May 6 looks like standard pm login from times

All processes on that line (tE) for the day in question
host04# /usr/sbin/acct/fwtmp < /var/adm/wtmp-5 | grep tE
LOGIN tE pts/tE 23794 6 0000 0000 1083825724 May 6 07:42:04 2004 155.223.160.60 155.223.160.60
usersh6 tE pts/tE 23794 7 0000 0003 1083825735 May 6 07:42:15 2004 155.223.160.60 155.223.160.60
usersh6 tE pts/tE 23794 8 0000 0000 1083837876 May 6 11:04:36 2004
LOGIN tE pts/tE 23783 6 0000 0000 1083837881 May 6 11:04:41 2004 155.223.80.82 155.223.80.82
userrjh tE pts/tE 23783 7 0000 0003 1083837894 May 6 11:04:54 2004 155.223.80.82 155.223.80.82
userrjh tE pts/tE 23783 8 0000 0000 1083849739 May 6 14:22:19 2004
LOGIN tE pts/tE 20750 6 0000 0000 1083849755 May 6 14:22:35 2004 155.223.160.44 155.223.160.44
userl td pts/tE 20750 7 0000 0003 1083849780 May 6 14:23:00 2004 99.9.194.5 myhost
LOGIN tE pts/tE 20750 8 0000 0000 1083855603 May 6 16:00:03 2004
LOGIN tE pts/tE 16291 6 0000 0000 1083855766 May 6 16:02:46 2004 155.223.160.153 155.223.160.153
usercr tE pts/tE 16291 7 0000 0003 1083855841 May 6 16:04:01 2004 155.223.160.153 155.223.160.153
userl td pts/tE 16825 8 0000 0000 1083857988 May 6 16:39:48 2004
usercr tE pts/tE 16291 8 0000 0000 1083858292 May 6 16:44:52 2004

Points to note
14:23 entry - ut_id (td) is different to ut_line (tE)
16:39 entry - PID is 16825, userl login PID was 20750

Trace the PID for the tE login @ 14:22
host04# /usr/sbin/acct/fwtmp < /var/adm/wtmp-5 | grep 20750
remshd 20750 8 0000 0000 1083835715 May 6 10:28:35 2004
LOGIN tE pts/tE 20750 6 0000 0000 1083849755 May 6 14:22:35 2004 155.223.160.44 155.223.160.44
userl td pts/tE 20750 7 0000 0003 1083849780 May 6 14:23:00 2004 99.9.194.5 myhost
LOGIN tE pts/tE 20750 8 0000 0000 1083855603 May 6 16:00:03 2004

Notes : Killed (DEAD_PROCESS) at 16:00

Trace the PID for the 16:39 entry
host04# /usr/sbin/acct/fwtmp < /var/adm/wtmp-5 | grep 16825
LOGIN td pts/td 16825 6 0000 0000 1083833157 May 6 09:45:57 2004 155.223.161.5 155.223.161.5
userlw2 td pts/td 16825 7 0000 0003 1083833168 May 6 09:46:08 2004 155.223.161.5 155.223.161.5
userl td pts/tE 16825 8 0000 0000 1083857988 May 6 16:39:48 2004

Notes : login starts as id userlw2 @ 09:46 on line td PID 16825

last follows PID 20750 for 14:23-16:00 login time
host04# last -f /var/adm/wtmp-5 userl
userl pts/tE Thu May 6 14:23 - 16:00 (01:37)
userl pts/ttb Thu May 6 10:49 - 10:56 (00:07)

last follows PID 16825 for 09:46(userlw2)-16:39(userl) login time
host04# last -f /var/adm/wtmp-5 userlw2
userlw2 pts/td Thu May 6 09:46 - 16:39 (06:53)
userlw2 pts/td Thu May 6 07:48 - 09:45 (01:57)

Steven
1) I don't think it's been changed, the accounts are captive to the application and the admins wouldn't be bothered to change it.

2) There is no longer any physical connection between the networks on the link that had been used, was pulled due to cost.

3) DNS is not used within this network, it's either /etc/hosts or nothing.


Paula
host04# traceroute myhost
traceroute to myhost (99.9.194.5), 30 hops max, 40 byte packets
1 155.223.58.114 (155.223.58.114) 0.693 ms 0.467 ms 0.482 ms
2 155.223.58.114 (155.223.58.114) 0.563 ms !H * 0.631 ms !H

From man :- !H Host is unreachable.

host04# traceroute 155.223.160.44
traceroute to 155.223.160.44 (155.223.160.44), 30 hops max, 40 byte packets
1 155.223.58.114 (155.223.58.114) 0.669 ms 0.452 ms 0.429 ms
2 155.223.199.254 (155.223.199.254) 0.670 ms 0.578 ms 0.466 ms
3 185.22.2.1 (185.22.2.1) 16.012 ms 15.472 ms 15.511 ms
4 185.22.0.5 (185.22.0.5) 15.990 ms 15.603 ms 15.701 ms
5 185.22.0.66 (185.22.0.66) 15.825 ms 15.787 ms 15.820 ms
6 185.22.0.37 (185.22.0.37) 16.426 ms 16.560 ms 17.537 ms
7 185.22.0.41 (185.22.0.41) 16.478 ms 16.460 ms 16.376 ms
8 185.22.1.18 (185.22.1.18) 23.998 ms 23.806 ms 24.019 ms
9 155.223.160.44 (155.223.160.44) 33.459 ms 23.468 ms 23.715 ms



Mark
What do you mean 'not out to get me'? Syder, like the drink, or InSyder - insider Arrrgh! the spooks are after me :o)

[Brian Hackley]

Hi,

Check to see if you have this telnetd patch and have followed the Special Instructions:

Special Installation Instructions:
PHNE_24829 contains a fix for the telnetd code defect
described in SR: 8606220839 (JAGad89975) - telnetd writes
to the wrong entry in /etc/utmpx on logout.

Although the SR: 8606220839 (JAGad89975) fix will prevent
any further corruption of /etc/utmpx(4), installing
PHNE_24829 will not correct any existing corruption in the
/etc/utmp(4) or /etc/utmpx(4) files.

Therefore if you are installing PHNE_24829 to fix the SR:
8606220839 (JAGad89975) defect, to completely resolve the
problem you must also ensure that the /etc/utmp and
/etc/utmpx files are cleared of any previous corruption
caused by this defect.

The /etc/utmp and /etc/utmpx files may be cleared using the
following procedure:

Before installing PHNE_24829 insert two lines into the
/etc/inittab(4) file as follows, then save /etc/inittab and
continue the PHNE_24829 patch installation.

init:3:initdefault:
utm1::sysinit:> /etc/utmp # clear current logon \
accounting files
utm2::sysinit:> /etc/utmpx # clear current login \
accounting files

After PHNE_24829 is installed and the system rebooted, you
may delete the above two entries from /etc/inittab or retain
them. In the latter case, /etc/utmp and /etc/utmpx will be
cleared every time the system is rebooted.

NOTE: The above steps are only required if the problem
described in SR: 8606220839 (JAGad89975) exists on
the system where PHNE_24829 is being installed.

HTH,
-> Brian Hackley

[Gareth Tunstall]

@Brian
Thanks for the reply, yes patch PHNE_24829 is installed and /etc/utmp cleared as per special instructions. It looks like the error SR: 8606220839 (JAGad89975) was telnetd writing duplicate lines, rather that 'wrong' lines.


Further investigation has shown this to be a more widespread error than just with the 'old' machine as reported above. I have found this on a second 11i machine too but not looked too hard yet.

It appears that LOGIN will start on a line (pts/t? record type '6' in col 5 is LOGIN_PROCESS) with a recorded ip address, the correct one for the subsequent logged in user. Once the user has logged in (record type '7' USER_PROCESS) a different ip address gets recorded. At logout it is the LOGIN process that gets recorded (record type '8' DEAD_PROCESS) rather than the user. See bottom 2 examples with normal logouts for the other listed user

Users starting aaa are on site/network 155.223.80.
Users starting bbb are on site/networks 155.223.16n.

155.223.162.126 aaapaw Fri May 7 06:34 - 07:28 (00:53)

LOGIN tm pts/tm 10047 6 0000 0000 1083908074 May 7 06:34:34 2004 155.223.80.130 155.223.80.130
aaapaw t0 pts/tm 10047 7 0000 0003 1083908091 May 7 06:34:51 2004 155.223.162.126 155.223.162.126
LOGIN tm pts/tm 10047 8 0000 0000 1083911290 May 7 07:28:10 2004

155.223.80.62 bbbprm5 Fri May 7 15:19 - 15:22 (00:03)

LOGIN tk pts/tk 21173 6 0000 0000 1083939545 May 7 15:19:05 2004 155.223.161.45 155.223.161.45
bbbprm5 tf pts/tk 21173 7 0000 0003 1083939560 May 7 15:19:20 2004 155.223.80.62 155.223.80.62
LOGIN tk pts/tk 21173 8 0000 0000 1083939741 May 7 15:22:21 2004

There seems to be no correlation between the last recorded use of the ip address or terminal and where the errors occur - shortened extracts below.

Last use of .62 address was for legit site aaa user who logged off cleanly.

host04# fwtmp < wtmp-5 | egrep -v ftp | egrep "\.62|161\.45|21173|8252"
LOGIN tA pts/tA 8252 6 0000 0000 1083912966 May 7 07:56:06 2004 155.223.80.62 155.223.80.62
aaapmk1 tA pts/tA 8252 7 0000 0003 1083912998 May 7 07:56:38 2004 155.223.80.62 155.223.80.62
aaapmk1 tA pts/tA 8252 8 0000 0000 1083927204 May 7 11:53:24 2004
LOGIN tk pts/tk 21173 6 0000 0000 1083939545 May 7 15:19:05 2004 155.223.161.45 155.223.161.45
bbbprm5 tf pts/tk 21173 7 0000 0003 1083939560 May 7 15:19:20 2004 155.223.80.62 155.223.80.62
LOGIN tk pts/tk 21173 8 0000 0000 1083939741 May 7 15:22:21 2004

Last use of tk terminal was for legit site aaa user who logged off cleanly before term id was reused.

host04# fwtmp < wtmp-5 | egrep -v ftp | egrep "tk"
LOGIN tk pts/tk 21986 6 0000 0000 1083935542 May 7 14:12:22 2004 195.36.75.3 195.36.75.3
aaacmpp tk pts/tk 21986 7 0000 0003 1083937493 May 7 14:44:53 2004 195.36.75.3 195.36.75.3
aaacmpp tk pts/tk 21986 8 0000 0000 1083938569 May 7 15:02:49 2004
LOGIN tk pts/tk 21173 6 0000 0000 1083939545 May 7 15:19:05 2004 155.223.161.45 155.223.161.45
bbbprm5 tf pts/tk 21173 7 0000 0003 1083939560 May 7 15:19:20 2004 155.223.80.62 155.223.80.62
LOGIN tk pts/tk 21173 8 0000 0000 1083939741 May 7 15:22:21 2004

I also have instances of a LOGIN being initialised by 1 user and terminated by another

LOGIN tc pts/tc 2393 6 0000 0000 1084251067 May 11 05:51:07 2004 155.223.162.9 155.223.162.9
bbbpri tc pts/tc 2393 7 0000 0003 1084251091 May 11 05:51:31 2004 155.223.162.9 155.223.162.9
bbbpkh tc pts/tP 2393 8 0000 0000 1084278523 May 11 13:28:43 2004

LOGIN t0 pts/t0 14503 6 0000 0000 1083914186 May 7 08:16:26 2004 155.223.161.128 155.223.161.128
bbbpks t0 pts/t0 14503 7 0000 0003 1083914201 May 7 08:16:41 2004 155.223.161.128 155.223.161.128
bbbpcr t0 pts/tY 14503 8 0000 0000 1083927182 May 7 11:53:02 2004

I've had enough of looking at this for today - I'm going cross eyed/home (delete as required :o) )