trying to do forensic on a hack server

[K.C. Chan]

All,
I am investigating how this server was compromise, doing an ls under /dev listed this weird output:

"actual blank line here"
/159
/26204
/26258
adbmouse
admmidi0
admmidi1


an ls -aF:
ls -aF | more
ls: /159: No such file or directory
|
./
../
/26204|
/26258|
adbmouse
admmidi0
admmidi1
admmidi2
admmidi3
adsp@

Any other type of ls options pukes on it. I am interested in getting at thise files
"|", "/26204|", and "/26258|". Any idea? Thanks.

[Alexander Chuzhoy]

I'm not sure I understood.
Try ls |grep ...

[K.C. Chan]

ls and grep seems to work:
ls | grep "/26204"
/26204

But I want to know if it's a dir or a file, so far I can not cd into it or view with less or more:
more "/26204"
/26204: No such file or directory
less "/26204"
/26204: No such file or directory
file "/26204"
/26204: can't stat `/26204' (No such file or directory).
Any idea?

[Alexander Chuzhoy]

actually you should try less //name of file

i.e. you should have preceding slash before the slash in the name and this how it should work...

[Alexander Chuzhoy]

Sorry , it suppose to be a preceding backslash
touch \\filename
to create
\filename

[K.C. Chan]

the backslash escape character is the first I've tried, didn't work; that's why I din't mention it here.

[K.C. Chan]

another thing, the no. of process in proc and ps -ax did not match, it's 88 proceses in proc table and 44 in ps -ax output.

[Michael Schulte zur Sur]

Hi,

do a ls -lb to see any unprintable characters.

Michael

[Martin P.J. Zinser]

ls -lQ might help too (quotes the actual name, so you can see e.g. spaces etc.

All the best, Martin

P.S. I suppose you disconnected the system from the network already.

[Michael Schulte zur Sur]

Hi again,

another thing, you could try is:
ls -l | od -x
gives you a hexdump.

Michael

[Jerome Henry]

Of course you're not using system ls I suppose, but an external, it looks like your command doesn't work properly, as if a rootkit was installed.
Tried chkrootkit also ?
http://www.chkrootkit.org/
file /26204 should tell you what kind of stuff it's supposed to be...

hth

J

[Bruce Copeland]

Please let us know what you find. These days we could all benefit from seeing examples of hacked linux systems--regardless how mundane the exploit.

Bruce

[K.C. Chan]

Michael,
ls -lb > /tmp/dev2.txt
ls: /159: No such file or directory
ls: : No such file or directory
ls: /26258: No such file or directory
ls: /26204: No such file or directory

didn't like it either.

[K.C. Chan]

Michael,
don't know how the hex dump can help me, but here it is:
/root/ls /dev | od -x | more
0000000 2f0a 3531 0a39 322f 3236 3430 2f0a 3632
0000020 3532 0a38 6461 6d62 756f 6573 610a 6d64
0000040 696d 6964 0a30 6461 6d6d 6469 3169 610a
0000060 6d64 696d 6964 0a32 6461 6d6d 6469 3369
0000100 610a 7364 0a70 6461 7073 0a30 6461 7073
0000120 0a31 6461 7073 0a32 6461 7073 0a33 6761
0000140 6770 7261 0a74 6c61 616f 4364 0a30 6c61
0000160 616f 4364 0a31 6c61 616f 4364 0a32 6c61
0000200 616f 4364 0a33 6c61 616f 5364 5145 610a
0000220 696d 6964 610a 696d 6964 0a30 6d61 6469
0000240 3169 610a 696d 6964 0a32 6d61 6469 3369
0000260 610a 696d 6167 6f6d 7375 0a65 6d61 6769
0000300 6d61 756f 6573 0a31 6d61 7869 7265 0a30
0000320 6d61 7869 7265 0a31 6d61 7869 7265 0a32
0000340 6d61 7869 7265 0a33 7061 5f6d 6962 736f
0000360 610a 6174 6172 6469 610a 6174 6972 6f6d
0000400 7375 0a65 7461 6269 0a6d 7461 6d69 756f
0000420 6573 610a 6475 6f69 610a 6475 6f69 0a30
0000440 7561 6964 316f 610a 6475 6f69 0a32 7561
0000460 6964 336f 610a 6475 6f69 7463 0a6c 7a61
0000500 6374 0a64 6562 7065 620a 6370 0a64 6163
0000520 6970 3032 630a 7061 3269 2e30 3030 630a
0000540 7061 3269 2e30 3130 630a 7061 3269 2e30
0000560 3230 630a 7061 3269 2e30 3330 630a 7061
0000600 3269 2e30 3430 630a 7061 3269 2e30 3530
0000620 630a 7061 3269 2e30 3630 630a 7061 3269
0000640 2e30 3730 630a 7061 3269 2e30 3830 630a
0000660 7061 3269 2e30 3930 630a 7061 3269 2e30
0000700 3031 630a 7061 3269 2e30 3131 630a 7061
0000720 3269 2e30 3231 630a 7061 3269 2e30 3331
0000740 630a 7061 3269 2e30 3431 630a 7061 3269
0000760 2e30 3531 630a 7061 3269 2e30 3631 630a
0001000 7061 3269 2e30 3731 630a 7061 3269 2e30
0001020 3831 630a 7061 3269 2e30 3931 630a 6963
0001040 7373 630a 7264 6d6f 630a 7564 3133 0a61

The first 4 line should be the file of interest.

[K.C. Chan]

All,
I believe the evidence is in the these files/dir in /dev directory. I can't even recreate create such file/dir with "/" as a character. As you noticed, the only options of ls which could see the file is ls without an options or options which does not require accessing attribute of the files. I've even tried "ls -lQ /dev/\/159" but it turns out as "ls: "/dev//159": No such file or directory". I am begining to think, this was created by some other utilities. Any idea?

[Martin P.J. Zinser]

Hi,

not sure it helps, but maybe file * works? Just letting it figure out the right quoting by itself.

Greetings, Martin

[Olivier Drouin]

you can try to see the hidden char by:

# cd ..
# ls dev/ [tab] [tab]

bash autocompletion will list the files in /dev, you may be able to see the characters inserted before the "/".

[Jerome Henry]

What architecture do you use ?
Assuming you're on an x86, the dump does provide a few things (Michael had a good idea) :
0x30 comes several times (317), which is the CMOS reg that hold the low byte of the mem count, this is wether a subfunction or an interrupt call.
0x33 comes 181 times, usually designing or getting mouse move (0x33 is mouse interrupt), so for some others 0x36 ioctl, 0x31 geteuid, 0x20 getchar.
This seems to be a stuff waiting for instruction from sopme media and deduce some read from it.. 've seen this kind of scheme in stuff like adore, but not exactly with the same nomenclature.
Getting the others would be great (but time consuming !)

J

[K.C. Chan]

Jerome,
How did you decode those hex characters? Any pointers on how it was done?

To answ. Your question, it's running redhat 7.2 on an IBM X intellistation.

According to ls -lF, the file type is of "|" and doing an ls -l on another system with similar category has this perm: "prw-r--r--", which is similar to gpmdata and initctl. I am assuming "p" means pipe? Thanks.

[K.C. Chan]

Olivier, here is what I got from your suggestion:
ls -l /dev/
Display all 5117 possibilities? (y or n)
irlpt3 sdab1 sdca14 sdg5 ttyP13
159 irlpt4 sdab10 sdca15 sdg6 ttyP14
26204 irlpt5 sdab11 sdca2 sdg7 ttyP15
26258 irlpt6 sdab12 sdca3 sdg8 ttyP2

[Jerome Henry]

Hi,
On inputting your string to an hexedit, and organising datas to find recurring strings.
Then on remembering my assembly lessons ! But they shouldn't be valid on intellistation.
Yes p means pipe (d==directory, l==symlink, p==pipe, s==socket).
Good thing would be to get intellistation guru to recompose these files purpose. Sad you don't have tcpdump of the transaction (but could have been done locally).
I still would consider having a diff for your ls and several others commands (more, useradd, del, dir, du, find, lsof, netstat, pstree, slocate, top, vdir), seeing part or all of them altered could help us determine if it's a rootkit or an exploit, implemented locally or through network connection.

Good luck

J

[K.C. Chan]

I am afraid I can not give it to you since it was overwritten by someone else in my group. Thanks for the info. It just bothers me that some file name created within /dev is call "/259" and noway of getting at it. Thanks for all your help.

[Michael Schulte zur Sur]

Hi,

can you do one more thing?
cd /dev
ls -lbi
and post it?

Could it be a faulty disk of file system?

Michael

[K.C. Chan]

Micheal,
this what I got:
ll -lbi /dev | more
ls: /159: No such file or directory
ls: /26258: No such file or directory
ls: /26204: No such file or directory
total 272
229377 drwxr-xr-x 17 root root 81920 Jan 26 09:24
229378 crw------- 1 root root 10, 10 Aug 30 2001 adbmouse
229379 crw-rw-rw- 1 root root 14, 14 Nov 25 2002 admmidi0
229380 crw-rw-rw- 1 root root 14, 30 Nov 25 2002 admmidi1
229381 crw-rw-rw- 1 root root 14, 46 Nov 25 2002 admmidi2
229382 crw-rw-rw- 1 root root 14, 62 Nov 25 2002 admmidi3
229383 lrwxrwxrwx 1 root root 10 Feb 25 2003 adsp -> /dev/ads
p0
229384 crw-rw-rw- 1 root root 14, 12 Nov 25 2002 adsp0
229385 crw-rw-rw- 1 root root 14, 28 Nov 25 2002 adsp1
229386 crw-rw-rw- 1 root root 14, 44 Nov 25 2002 adsp2
229387 crw-rw-rw- 1 root root 14, 60 Nov 25 2002 adsp3
229388 crw-r--r-- 1 root root 10, 175 Aug 30 2001 agpgart
229389 crw-rw-rw- 1 root root 116, 0 Nov 25 2002 aloadC0
229390 crw-rw-rw- 1 root root 116, 32 Nov 25 2002 aloadC1
229391 crw-rw-rw- 1 root root 116, 64 Nov 25 2002 aloadC2
229392 crw-rw-rw- 1 root root 116, 96 Nov 25 2002 aloadC3
229393 crw-rw-rw- 1 root root 116, 1 Nov 25 2002 aloadSEQ
229394 lrwxrwxrwx 1 root root 11 Feb 25 2003 amidi -> /dev/am
idi0
229395 crw-rw-rw- 1 root root 14, 13 Nov 25 2002 amidi0
229396 crw-rw-rw- 1 root root 14, 29 Nov 25 2002 amidi1
229397 crw-rw-rw- 1 root root 14, 45 Nov 25 2002 amidi2
229398 crw-rw-rw- 1 root root 14, 61 Nov 25 2002 amidi3
229399 crw------- 1 root root 10, 4 Aug 30 2001 amigamouse
229400 crw------- 1 root root 10, 7 Aug 30 2001 amigamouse1
229401 crw-rw-rw- 1 root root 14, 11 Nov 25 2002 amixer0

[U.SivaKumar_2]

Hi,

This output alone is not suggestive of any suspicious files as they refer to normal device files.

Have you taken tripwire checksum snapshot of the fresh system after complete installation ?. If you have it , run it the tripwire checker again to compare it with the current state of the filesystem.

you can run rootkit hunter to detect common rootkits in linux. (http://www.rootkit.nl/)

Always do forensic analysis of the compromised system by mounting that system's harddisk to a clean server and using forensic tools eg. coroner's , sleuth autopsy etc. Exceptions of analysis in the real running compromised host is to get process information and memory/registry traces.

Beware of elusive kernel rootkits , to find them use combination of different tools , knowledge of properties of rootkits and common sense. Never forget to analyze the network traffic originating from the compromised host and the open ports . This will enable a successful correlation . This activity can be done by connecting the compromised host to a clean ( secure ) host by means of cross cable.

regards,

U.SivaKumar