Answers by Dion Mendel

Back at his desk, Skodo continued staring at the screen. The
dot-dot-space directory unnerved him. He suspected that something was
not quite right, but he couldn't work it out. Skodo decided to call on
his old friend Tom Bombadil for help.

[root@MidEarthFileServer .. ]# ls -al
total 2020
drwxrwxr-x 5 smeagol smeagol 4096 Dec 28 20:35 .
drwxrwxrwt 7 root root 4096 Dec 28 19:47 ..
[root@MidEarthFileServer .. ]#

Pointing to the last command on the screen, Tom laughed and said "You've
been rooted, my fine fellow".

"That evil Smeagol", began Skodo. "Hold on now, Skodo", interrupted
Tom. "The attacker got root. It may have been Smeagol, or someone
trying to implicate him. Deagol is not the only one to be at odds with
Smeagol".

"How can you tell I've been rooted?", Skodo asked. Tom pointed to the
screen. "Notice the inconsistancies. That dot-dot-space directory
should have a number of files totaling 2020kb in size, plus it should
contain three subdirectories. If it weren't for these two mistakes,
you'd probably never know that you'd been rooted".

Skodo stared harder at the screen. With a flash, he understood. "Of
course! The ls output line "total 2020" shows the total amount of data
stored in the files of a directory. If there are no files, the total
should be zero". "And the second inconsistency?", prompted Tom.
Without hesitation Skodo responded, "The link count of 5. An empty
directory has a link count of two (one from the parent directory and one
for the "." entry). The link count increases by one for each
subdirectory (caused by the ".." entry of the subdirectory)." "What do
these inconsistencies imply?", questioned Tom. Skodo swallowed
nervously, "That there are hidden files. Only root can hide files, and
as I didn't hide these files, it means that someone else has root".

"What do I do now?", asked Skodo.

"The attacker is still active. Did you notice the time stamp change on
the dot-dot-space directory - 20:12 to 20:35? This shows that directory
entries are being modified. /bin/sync the disks (to minimise data
corruption) and forcibly power down. Don't shutdown nicely using
/sbin/halt or /sbin/shutdown as the attacker may have inserted hooks
into the shutdown scripts.

Next boot the system with your linux rescue bookdisk. mount the
filesystems readonly and noexec so you don't accidentally delete data,
or execute strange programs".

"But what am I looking for?", asked Skodo.

"Firstly you'll want to examine that dot-dot-space directory. As you're
using a clean kernel and clean tools from your rescue bootdisk, the
contents of that directory will no longer be hidden. You'll also want
to look for any backdoors installed on the system. Run your intrusion
detection software (aide or tripwire) to find any modifications to
system files. Also examine your logs to see if you can determine how
the attacker got root. If you've got lots of spare time, you can run
The Sleuth Kit, to attempt a forensic analysis".

"Gandalf won't let me reboot the file server!", Skodo wailed.

"You may still be able to get a file system image without rebooting the
file server. Use the netcat program to send a file-system image across
the network to your analysis machine. Do the following for each
partition of the fileserver. Then you can analyse the file systems
using the methods I mentioned before".

[root@AnalysisMachine]# nc -l -p 80 | dd of=/data/hda1.img
[root@MidEarthFileServer]# dd if=/dev/hda1 | nc AnalysisMachine 80

"But I don't have any machines to spare for analysis", said Skodo almost
in tears.

"Well, then you'll just have to do your analysis on the live machine",
retorted Tom.

Skodo brightened and he considered this. "I bet the attacker has
modified /bin/ls so that it hides certain files. I know, I'll use the
"echo .* *" trick to list files without the use of ls".

Skodo tried this but it too resulted in no files. Tom smiled, "That was
a nice try Skodo, but I bet that the attacker has modified the kernel
the hide the files. You'll need to access the file system directly.
Try using the debugfs command".

[root@MidEarthFileServer]# echo '
> cd "tmp"
> cd ".. "
> ls
> quit
> ' | debugfs /dev/hda1

"It worked!", yelled Skodo. "Now I can start tracking down just what
has happed here. Thank you very much for your help, Tom. Before you
go, please tell me how I can protect my kernel from being seized in the
future".

"You need to harden the file server to prevent this sort of thing from
happening again. Firstly, make sure that you are up to date with all
security patches. You may wish to consider using a hardened kernel,
such as the ones distributed by grsecurity.net. These things will help
you prevent users from obtaining root.

To prevent root from modifying the kernel (either by adding kernel
modules or directly modifying /dev/kmem, /dev/mem or /proc/kcore), you
will need to remove CAP_SYS_MODULE and CAP_SYS_RAWIO from the kernel
capability bounding set. The lcap program can do this if you add the
appropriate command to your init scripts. If you do this, you will not
have any more problems with hidden files. Finally, you may wish to look
at se-linux, as this has the ability to protect the system from root.

I need to go feed Fatty Lumpkin now, so I'll leave you to your work.
Hey dol! merry dol! ring a dong dillo! ... "