|
Lord of the Ring-Zero Challenge (March 2004) - Ed Skoudis
Author: Raul Siles (raul.siles@hp.com)
****
This challenge is completely dedicated to all the innocent people living in
Madrid (Spain), my city!... the city of my heart!, and affected by the 11-M terrorist attack. Peace and life for everyone!!
****
NOTE: A non-requested final extra section (number 5) has been included in order to analyze the evidence presented.
1. What two mistakes had Gollum/Smeagol and/or his tool made in this attack?
----
It seems Smeagol is not an 31733 hacker yet and some actions could be improved in order to hide his attack. The two main mistakes were:
- He tried to camouflage his toolkit under a directory using one of the
common names used by the script-kiddie community, ".. ". This mistake even helped a security novice administrator like Skodo to find him, in this case using the useful recommendations from the SANS's
"SysAdmin Intrusion Discovery Cheat Sheets".
Even when an attacker doesn't have a rootkit available, starting a file or directory name with a dot in Unix hides it from the typical "ls"
command. Common names used for covering the tracks are ".. ", ". ", "..." and " ".
Additionally, the files/directories to hid are commonly placed in locations where they
won't be noticed, such as "/tmp" (as in this case), "/dev", font directories, OS source code repositories or "/etc", due to the fact that these directories are over populated with several
files and directories and it is very difficult to identify new add-ons.
- It seems he is not very familiar with the rootkit used, therefore instead of using it to hide all his activities in the file system, he
only hid some files and directories (see the extra section 5) inside the "/tmp/.. " directory but forgot to hide the "/tmp/.. " directory itself.
Thus, Skodo was able to find it due to the
fact that it was visible for everyone (although it starts with the dot character this only "protects" it against admins unable to use the "ls -a" option ;-)).
In this case, both mistakes
helped Skodo to discover him. If only one of them wouldn't have taken place, Skodo hadn't found the compromise.
Besides, Smeagol didn't change the ownership of his newly created files and directories, so Skodo
was able to identify "smeagol" as the Linux user related to them and associate this strange objects to the "real" (Is Gollum a real being? ;-)) owner.
Other traces left by Smeagol helped Skodo
to notice strange system situations, such as the permissions on the "ballad_of_bilbo_baggins.mov" file (world readable).
2. Suppose Skodo is allowed to reboot the box. How can Skodo determine what really happened? What tools should he use?
----
TIP: In this case the analysis is made over a dead system, thus all the evidence is
only available in the static file system residing in the system hard drives.
Before suggesting the actions the Skodo should take, it is important to
remember the main two goals in an incident investigation (from the Sleuth Kit informer, http://www.sleuthkit.org/informer/):
- Minimize the amount of trust that you place in the system so that more accurate information is collected.
- Minimize the amount of data that you change on the system so that evidence is preserved.
From the
evidence extracted Skodo could confirm that a rootkit was installed in the system, so he cannot trust the information provided by the built-in software included in the system.
Up to this point (with the
available information) it is not possible to know if it is a user-mode or kernel-mode rootkit. However, based in the Gollum words, "... the precioussssssss kernel!", it seems Smeagol used a kernel rootkit:
"Who do you trust, Smeagol (White Hat, http://www.whitehat.org) or Gollum (Black Hat, http://www.blackhat.com)?" ;-)
It is recommended to reboot the box in order to run a clean kernel able to provide
real system information during the investigation. To do so, a Linux bootable (or Live) CD should by used, like Knoppix (http://www.knopper.net/knoppix/) (or its security variant, Knoppix-STD
(http://www.knoppix-std.org/)), FIRE (http://biatchux.dmzs.com/) or Penguin Sleuth Boot CD (http://www.linux-forensics.com/). Once booted Skodo just needs to get and analyze the real system status, starting by
inspecting the "/tmp/.. " suspicious directory and its real hidden contents.
To start up with Skodo should power off the system (unplugging the power cable in order to keep all the evidence) and the
first step should be getting a raw binary copy of the disk, using the "dd" command. The copy can be saved in a removable USB disk or it can be transferred through the network to another system, using the
netcat tool ("nc"). Skodo shouldn't forget to generate cryptographic hashes of the original and copied data through "md5sum" and verify them.
Knoppix, as many other Live CD distros
(http://www.frozentech.com/content/livecd.php), provides some common Linux administration tools very helpful in the incident handling processes and forensic investigations. Once the copy has been obtained, in case it
would be needed later for an in depth forensic analysis, Skodo can use the standard file system tools to analyze the disk contents: ls, echo, du, df, diff, strings, grep, egrep, fdisk, tar, vim, awk, sed, cat, less,
file, find, md5sum... (question 3 will show other tools not directly related with file system analysis).
To operate over the disk he should mount the copied file system in read-only mode, not to modify it; to do
so, Skodo should at least use the "mount -n -o noatime,ro" command. Other options are recommended, such as "-o nosuid,nodev,noexec" to block specific actions. It is recommended to only analyze the
disk copies, not the original disk, in case this incident will finish in the High Court of Gondor ;-)
Apart from the discovered suspicious directory, Skodo should investigate what was the vulnerability exploited
by Gollum to compromise the system and get root access. To do so, it is recommended to develop an in-depth analysis, reviewing:
- All the system log files for suspicious events, such as large number of
authentication failures, strange character strings, like the ones associated to the RPC exploitations (char: ^PM) or the assembler NOP codes used in buffer overflows (like 0x90), network cards entering in promiscuous
mode, zeroed entries trying to erase log events, empty log periods (without events)... There are lots of Linux log files to inspect: messages, secure, sulog, wtmp, utmp, btmp, maillog...
- All the user's history files searching for potentially executed commands, like ".bash_history".
- Recent events: recently modified binaries and newly created files (using the "find" command).
- The "find" command can also be used to look for files in an specific time frame, using the "-newer" option. Additionally, anomalous file system situations should be inspected, such as SUID or SGID
files, large files, strange named files (both, hidden and visible) or regular files in "/dev".
- Check the integrity of the system binaries through RPM (if possible) or Tripwire/AIDE (see question 4).
-
It is also recommended to look for unusual/unknown user accounts. Check the "/etc/passwd", "/etc/shadow", "/etc/group" and "/home" files and directories.
- Finally, check for
unusual initial processes (/etc/inittab, init.d and rc*.d) and scheduled tasks, through "crontab" or "at". Review their job definition files.
Almost all this information can be extracted using the Linux standard commands mentioned above.
Besides, the unknown binaries found and its purpose could be analyzed in detail through additional tools like: ldd,
objdump, strace, nm, ltrace, gdb, ptrace... (some of the ones mentioned previously are also very useful for this task). If required, Skodo could use advanced malware analysis tools, such as "Fenris"
(http://razor.bindview.com/tools/fenris/).
With the goal of obtaining as much information as possible from the compromised disks, a forensic analysis should be performed. One of the most useful tools available
for this purpose is The Sleuth Kit (http://www.sleuthkit.org/), THE file system digital forensics tool. It includes a graphical HTML browser, Autopsy.
This tools would allow Skodo identify what has changed in the
system and when (creating a timeline), how the incident occurred, commands used and even recover deleted files. Timelines are based on the 3 times associated to a Unix file: access time (atime, "ls -alu"),
modification (contents) time (mtime, "ls -al") and status change (inode) time (ctime, "ls -alc").
The Sleuth Kit is also capable of analyzing the file system bad block structures, sometimes used
by antiforensic tools (such as Runefs, http://www.phrack.org/show.php?p=59&a=6) to hide information.
The new Autopsy version, 2.00, that will be released at the end of March provides new live analysis
features, so it could be also used for question 3. It shows files that are hidden by rootkits without updating the access times on files when they are read.
At the beginning of this Middle Earth age, , the Sleuth
kit predecessor, The Coroner's Toolkit was the most used tool (http://www.porcupine.org/forensics/tct.html) ;-). Gandalf was famous for his knowledge of TCT and his friendship with Dan and Wietse ;-)
There are
also individual standard Linux tools like "debugfs" (inside the "e2fsprogs" package, http://e2fsprogs.sourceforge.net), a file system debugger for ext2/ext3 partitions that could help in getting data
from the system partitions.
For additional advanced forensic analysis tools it is recommended to check the list included in some of the mentioned distros, such as FIRE or Penguing Sleuth:
- Forensic tools: http://biatchux.dmzs.com/?section=tools&subsection=F
- IH tools: http://biatchux.dmzs.com/?section=tools&subsection=I
- Penguin tools: http://www.linux-forensics.com/forensics/pensleuth.html
Some of these recommendations were extracted from the Bilbo's system administration handbook; he was the first sysadmin this
Linux file server had and one of the famous security gurus of recognized prestige in the Middle Earth ;-)
Skodo should document all the steps taken in his uncle's (Bilbo) incident handling notebook.
3. Now suppose Skodo is unable to reboot the Middle Earth file server. What tools should he use to determine what is really happening without shutting the box down or rebooting it?
----
TIP: In this case
the analysis is made over a live system, thus the evidence is available in the dynamic memory, the network traffic and in the file system (similar considerations as the ones mentioned in question 2 apply, taking into
account the fact pointed out latterly about how trustable every binary is).
The option taken in question 2, although it is the recommended one from the incident handling perspective, is not always possible,
and in fact it provokes a DoS over the system service (probably this is the only attacker's goal).
Based on the type of rootkit used the method used by Skodo should be different:
- User-mode rootkit:
considering the kernel remains untouched, he should be worried about using clean trustable system commands, instead of the evil ones substituted by the attacker.
- Kernel-mode rootkit: the kernel (the system's heart)
has been compromised, it is very difficult to trust in any command if the system cannot be restarted. To analyze it see the comments bellow about anomalies searching.
Analyzing the first case, the easiest, Skodo should start by inserting an IH CD with all the analysis utilities.
It is recommended to statically link all the binaries to be used in order not to be disturbed by
the potentially non-trustable system libraries. To do so, Skodo should use a home-made CD, previously prepared, where all the binaries have been compiled using the "gcc -static" option. If the CD doesn't
exist, it could be generated in another similar Linux system.
Sometimes it is not possible to statically compile a program, for example because its source code is not available. In this case it is recommended to
ensure that the libraries used by it are trustable. To do so, he should start from a clean shell and set the search variables, for the execution (PATH) and the dynamic libraries (LD_LIBRARY_PATH):
# mount -n /dev/hdb /mnt/cdrom (the most difficult point, ensure "mount" has not been trojaned ;-))
# /mnt/cdrom/sbin/bash (get a clean shell)
# export PATH=/mnt/cdrom/sbin:...
# export LD_LIBRARY_PATH=/mnt/cdrom/lib:...
The "mount -n" option doesn't write information to the mount table in order to preserve the evidence.
The most critical step in this process is
ensuring that the "mount" command, needed to make the CD containing the tools available, has not been compromised. If this is the case, Skodo could not be sure if the program executed is its binary copy or the
attacker's one (as an exception he could change the program source code displaying some kind of initial string in order to verify that his own version is the one being executed, but this is only valid for user-mode
rootkits because kernel ones can manipulate just the system calls).
Unlike in question 2, Skodo now have all the volatile information available, so he must focus on gathering memory and process data, as well as
network connections. To analyze all these elements using the traditional way several tools could be used:
- Process analysis: ps (-aux, -auxeww), lsof (-p), top (-b -n1), ltrace, strace, ptrace.
- Enabled services: chkconfig (--list), /etc/inittab, rc*.d, init.d.
- Unlinked files: lsof (+L1).
- Unusual network usage: netstat (-nap), lsof (-i), arp (-van), ip (link | grep PROMISC), route (-vn), ifconfig.
- Users: who (-Hi), w, last (-aidx), lastb, finger.
- System: uptime, uname, date, dmesg.
- Disk and file system: (all the tools mentioned in question 2).
NOTE: Some interesting options are shown between brackets.
It would be recommended that Skodo will have a home-made script to run all this tools at a time, collecting all the required information. The output
could be send again to another box using "netcat", trying not to modify the system disk.
The "script" command is recommended in order to log all the manually steps taken by Skodo; the output could
be written to a clean floppy (previously mounted), again, not to write to the disks to be analyzed. Therefore, Skodo should put a special effort into trying not to write to the disk, not to kill processes, keep
timestamps untouched ... just to keep all the evidence intact.
It is recommended to double-check every item trying to find anomalies, like a dejavu or "a glitch in the Matrix" :-)
The basic idea
around this is based on comparing the output of different tools, like "ls" and "echo *". If different, something wrong happens. Another useful example in Linux, based on the way these commands get
their information, is the difference between "netstat -nap" and "lsof -i".
This technique is mainly useful against user-mode rootkits and could help against some kernel-mode rootkits too.
In Linux systems the "/proc" pseudo file system could be very useful for this double-checking process as well as for extracting lot of information about the system current state.
There is a tool, called
"procget" (http://www.rndsoftware.com/products.shtml) that allow saving the contents of the "/proc" file system to another system through the network. Using the complementary "procsave"
tool, the file system can be recreated for inspection.
Skodo could try some external actions to gather more information about the system status given the fact he cannot trust its own system, such as port scanning
the system trying to find unknown listening ports (using "nmap") or perform network monitoring to collect all the packets to and from the compromised box, through "snort" or "tcpdump".
All these suggestions represent detection and response methods in comparison with the protection or prevention methods described in question 4.
If the "mount" binary is not trustable, the
"anomaly search" is the only reliable option. An advanced method typically based in this idea is developed by some anti-rootkit tools, which help in the automation of the process:
- Chkrootik (http://www.chkrootkit.com/): this tool checks for about 50 different user and kernel-mode rootkits.
The output from this tool probably cannot be trusted in a compromised system, because it is based on
Linux standard commands, like awk, cut, echo, egrep, find, head...; thus, it provides special switches to specify the path to the trustable binaries (-p, path) and to work in a box you trust (-r, rootdir).
-
Rkscan: a kernel based rootkit scanner (http://www.hsc.fr/ressources/outils/rkscan/index.html.en, http://www.hsc.fr/ressources/breves/LKMrootkits.html.en). It detects specific kernel-mode rootkits such as Adore (v0.14,
0.2b and 0.24) and Knark (v0.59).
- Kstat (http://www.s0ftpj.org/en/tools.html, http://www.s0ftpj.org/docs/lkm.htm): a kernel analysis tool that checks "/dev/kmem" for information about LKMs and the
status of the system call table. It can slo be used to extract other information directly from the memory image, like the running processes or the network interface state.
- Rootkit Hunter, rkhunter (http://www.rootkit.nl/projects/rootkit_hunter.html): another user and kernel-mode rootkit scanner.
Besides, some specific rootkits have well-known behaviours against certain actions,
that can be manually (or automatically) tested in order to detect the rootkit existence looking for system discrepancies (like the ones in the Skodo file system): some features introduced by specific rootkits could be
executed to verify if they are present, such as concrete commands, response to certain process signals, the way they manage the promiscuous flag...
Nowadays there are other options apart from the traditional live
analysis method described (which most probably change the system state) based on saving the system memory for a later analysis. To perform this task the crash dump facility available to troubleshoot and debug kernel
bugs could be used. There are different toolsets available:
- LKCD, Linux Kernel Crash Dumps (http://lkcd.sourceforge.net/) from SGI: this package introduces a patch into the Linux kernel in order to generate a
crash dump. To save the dump Skodo just will need to request the system "SysReq" mode. This action requires a reboot and the location where the memory image will be written has some implications from a
forensic perspective. It is recommended to make an image of the swap partition and restore the image in a different system using the "lcrash" tool.
This time Skodo cannot used this feature because his
fileserver has not been prepared for it, but I'm sure he will think about it for future "Ring Zero" battles against Gollum ;-)
- The memory dump extracted with LKCD can be analyzed through
"crash" (http://freshmeat.net/projects/crash). This tool uses the dump image and the kernel, compiled with debugging symbols, to provide general information about the system state and processes running, active
network connections... and the contents of all the kernel structures.
This tool can also be used to investigate a live system, using "/dev/mem" as the dump file.
- There is also another tool,
"mcore" (http://oss.missioncriticallinux.com/projects/mcore/) to save the crash information in system memory instead of in the file system (swap partition). The image is saved in a compressed format and
requires a reboot to be recovered.
- Supposing the Skodo's system has not been prepared to dump the memory contents, he should use the "memget" tool (http://www.rndsoftware.com/products.shtml) to
extract the sparse "/dev/kmem" file and transfer it to another system using the network. Using the complementary "memseek" tool, the memory can be analyzed inspecting its contents by address.
Finally, the most trustworthy method would be the usage of "A hardware-based memory acquisition procedure for digital investigations", a PCI hardware device to acquire the volatile memory of a compromised
computer; it doesn't rely in the operating system or applications executing in the box (Journal of Digital Investigations, Volume 1, Issue 1: Item 10, pages 50-60; http://www.sciencedirect.com/science/journal/17422876).
All this memory analysis methods are really useful against non-LKM kernel rootkits, based on modifying "/dev/kmem".
If I were Skodo I would also analyze Smeagol's laptop (and cubicle ;-)) trying to find more evidences about this case ;-)
4. Short of storing it on a chain around his neck, how can Skodo protect his kernel from being seized by a user on the machine?
----
The first method in order to avoid the kernel being modified is based on
hardening the system as much as possible to avoid a root-level compromise. To do so Skodo will need to apply the common security countermeasures, such as keeping the system patched, disabling all unneeded services, good
account and password management procedures, having IDS (host and network) tools... This last option is really helpful to catch the attackers early in the compromisation process. Skodo should start by reviewing the ten
Unix vulnerabilities from the "SANS Top 20" list (http://www.sans.org/top20/).
There are some hardening tools that could help to improve and speed-up the process. This tools are more suitable for
dedicated servers, like the Skodo's fileserver, than a general purpose system:
- Bastille: http://www.bastille-linux.org
- CIS Linux benchmark and scoring tools: http://www.cisecurity.org/bench_linux.html
There are also several Linux hardening guides in the Web (http://www.linux-sec.net/Harden/howto.gwif.html). Use Google too!!
Skodo could also make use of the Linux Capabilities
(http://www.securityfocus.com/infocus/1400, http://www.linuxjournal.com/print.php?sid=5737), a kernel feature set to allow/deny certain privileges reserved for root. The LCAP tool (http://home.netcom.com/~spoon/lcap/)
provides an interface for managing these capabilities.
There is another Linux project related with the file system ACLs, a feature that increase the degree of access control over files (http://acl.bestbits.at).
A Hobbiton attacker can use two methods to manipulate the kernel: he needs to have root privileges in the system or own The One Ring ;-)
Another option for sensitive system is based on building a monolithic
kernel (http://www.kernel.org) in order not to allow Loadable Kernel Modules, LKMs. This proposal can be applied over non-changing production servers (from a hardware/features point of view), where the dynamic kernel
modification capabilities are not needed; this is probably the case with the Skodo's fileserver.
However, this method won't protect the system against kernel rootkits based on directly modifying the kernel memory
(/dev/kmem), such as SuckIT (http://www.phrack.org/show.php?p=58&a=7).
For a brief recipe about how to build a monolithic kernel in Red Hat 9.0, check out:
http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/custom-guide/s1-custom-kernel-monolithic.html
The system hardening process could be improved using a kernel version that doesn't export the system call table,
like RH 8.0 (2.4.18) or the official kernel version 2.5.41(http://www.linuxdevcenter.com/lpt/a/2996). This will limit some LKM kernel rootkits in manipulating the system calls, although there are ways of reexporting
back the table (http://xenion.antifork.org/files/addsym.c), from the same author of "kstat".
The Linux LSM, Loadable Security Module, project (http://freshmeat.net/projects/lsm/) is an intrusion
prevention LKM; once it has been loaded it ensures no other modules can be loaded and limits what other LKM can do. It also protects file attributes on "ext2" file systems and access to raw devices.
This project shouldn't be confused with the LSM kernel patch (http://lsm.immunix.org/)developed to create a security framework around the Linux kernel (through interface hooks) that will facilitate the inclusion of
modular security mechanisms (http://lwn.net/Articles/3467/).
The starting message of the LSM patch mailing list (http://mail.wirex.com/pipermail/linux-security-module/) is here:
http://www.linuxsecurity.com/articles/forums_article-2854.html. One of the top projects based on the LSM patch is implementing the security-Enhanced Linux (SELinux, http://www.nsa.gov/selinux/) as a LSM module:
http://www.nsa.gov/selinux/papers/module/t1.html.
Another famous Linux kernel protector is St. Michael, by Tim Lawless (http://sourceforge.net/projects/stjude). It alters the kernel to detect strange activity
such as the remapping of system calls when a module is loaded (init_module) or removed (delete_module), so it can prevent and detect some malicious module behaviors.
There is also a Linux Intrusion Detection
System, LIDS (http://www.lids.org), a kernel patch designed to improve the overall system security (a kind of HIDS): it adds a kernel monitor and MAC-kernel controls, reducing the powerful nature of root.
To
complete the protection suggestions, user-level rootkits replace critical components of the operating system (system binary applications) with the attacker's own malicious code. The usage of a file integrity checking
tool like Tripwire (http://www.tripwire.com) or AIDE (http://www.cs.tut.fi/~rammer/aide.html, sourceforge.net/projects/aide) allows detecting the system changes as soon as possible. On a Linux based RPM machine, the RPM
tool can also be used for this purpose: "rpm -Va"(http://www.sans.org/y2k/RPM.htm).
In order to compare the actual MD5 values with a trustworthy ones, Skodo will need the original OS media or can use an
external trusted source, like "http://www.knowngoods.org", a public database of crypto hashes (MD5 and SHA1) for different Unix flavors.
As was mentioned in question 3, it is recommended to regularly
run a rootkit detection tool, such as "chkrootkit" or "rkscan". They can detect different multi-platform user and kernel-level rootkits, as well as look for symptoms of rootkit infestation. These
tools can be run after the compromisation, while others like Tripwire/AIDE or "rkdet" (see bellow) need to be run first.
Due to the fact that most kernel rootkits compromise the system call table, there
are some tools to monitor the usage and modification of this kernel table:
- Syscall_sentry (http://www.usenix.org/publications/login/2001-11/pdfs/jones2.pdf). It can even restore the original table status when a new
change is detected.
- Systrace (http://niels.xtdnet.nl/systrace/, http://www.citi.umich.edu/u/provos/systrace/linux.html). It allows limiting the usage of the table, creating application system calls profiles.
A rootkit prevention tool is "rkdet" (http://vancouver-webpages.com/rkdet/). It is a daemon intended to catch someone installing a user-mode rootkit or running a packet sniffer. It must be run continually in
order to detect a compromise and alert about it.
As can be deducted from all the information presented, when a rootkit is detected, like Skodo did, it is very hard to be sure the system has been returned to a
trust state just removing or modifying its components. It is recommended to reinstall the OS and applications, patch them and harden the whole result, specially ... "the preciousssss kernel ;-)".
Skodo should monitor the system carefully after rebuilding it because it is probable Gollum will return to own the Ring-Zero ;-)
A year ago Bilbo recommended Skodo to keep current with the kernel
(http://www.linuxhq.com) and have a look at some rootkit repositories, but he didn't find time to do so:
- http://packetstormsecurity.org/UNIX/penetration/rootkits/
- http://www.rootkit.com (mostly Windows based)
- http://www.l0t3k.org/tools/Rootkit/
Finally, the most secure option would be to store the kernel in the Destiny Mountain, in the Mordor, controlled by Sauron. I don't think too many attackers will go there
to manipulate it ;-)
5. Extra section: Evidence analysis
----
- General concepts around the "ls" commands executed by Skodo:
The Linux "ls" command executed by Skodo used the following two options:
-a: do not hide entries starting with .
-l: use a long listing format
The long listing format shows a number in front of the file/directory owner: it shows the number of hard links associated to the object.
The timestamp showed by the "ls" command by default references
the modification time, that is the time of last modification of the file contents (mtime in the inode).
The first line in the "ls" output shows the total disk allocation for all files/directories in
that directory, expressed by number of blocks. The default "ls" block size is 1024 bytes.
NOTE: The total blocks value computed counts each hard link separately; this is arguably a deficiency (as denoted in the "ls" man page).
- Evidence analysis based on all the previous information: (see the Skodo's terminal session image)
The "t" character in the "/tmp" directory permission is correct and denotes that the sticky bit and the other-executable bit are both set.
An interesting fact about the output of "ls
-al" in the "/tmp" directory is that by default, the "ls" output is sorted by name (alphabetically) and this is not the case here.
Some of the "/tmp" directory contents could be
considered as normal, such as the ".fam_socket, .font-unix, .ICE-unix, .sawfish-root and .X11-unix" directories.
It seems that the umask value used by "smeagol" was 002, therefore the
permissions associated to the "/tmp/.. " directory are "drwxrwxr-x", although he could have changed them manually ("chmod").
Two methods have been identified to check the integrity of the evidence obtained through the "ls" command:
1)- Total block count
2)- Hard link count
1)- Total block count analysis:
A
standard Linux directory listing with the look&feel of the "/tmp/.. " directory should have a total count of 8 blocks: 4 blocks belonging to the "." entry and an additional 4 blocks for the
".." entry (both are 4096 bytes in size).
Example.-
# ls -al
total 8
drwxr-xr-x 2 root root 4096 Mar 11 12:56 .
drwxrwxrwt 8 root root 4096 Mar 11 12:56 ..
The Linux "ls" total block number is allocated based on the file
system block size, the basic file system allocation unit. The standard Linux file systems, ext2 and ext3, has a block size of 4096 bytes. This information can be checked using the "dumpe2fs" tool and getting
the corresponding parameter:
# dumpe2fs /dev/hda1 | grep "Block size"
dumpe2fs 1.27 (8-Mar-2002)
Block size: 4096
So, as a conclusion, the total block count displayed by the "ls" command is a multiple of 4.
So, when a file or directory is created, if its associated size is less than or equal to 4096 bytes,
at least one file system block will be allocated, 4Kbytes, and the total block count will be increased by 4. A file of 4097 bytes length will increase the total block count by 8 (two file system 4Kbytes blocks). The
zero bytes files doesn't increase the total block count at all.
However, in this case the total number of blocks is 2020.
If we ignore the block count associated to the visible directories (8 blocks),
"." and "..", we obtain a count of 2012 "missing" blocks. Generally speaking this confirms that the rootkit is hiding about 8 Mbytes of data (2012*4096 = 8241152 / 1Mb = 7,86 Mb).
CONCLUSION: It seems the rootkit is hiding some files !!
The total number of blocks is not the exact sum of the number of bytes for every file and directory round up to the nearest 4K multiple. Typically some
additional blocks are added due to the internal file system structures and the way larger files are allocated in the disk. The "ext2" structures support up to 12 direct blocks in a single inode, so the number
of blocks matches the file size if the file is less than 49152 bytes (12 x 4096). If it is greater than this value, indirect blocks must be used; they are not storing the file contents but auxiliar data (pointer to
other blocks).
The OS block count for each object can be obtained through the "-s" option of the "ls" command (it shows the block size for the file/dir):
Example.-
# ls -sl
total 3736
4 drwxrwxrwx 10 root root 4096 Mar 11 2002 fileutils-4.1.9
1668 -rw-r--r-- 1 root root 1703053 Mar 11 16:35 fileutils-4.1.9-11.src.rpm
1632 -rw-r--r-- 1 root root 1663297 Mar 11 2002 fileutils-4.1.9.tar.bz2
4 drwxrwsr-x 5 root root 4096 Mar 9 12:37 hping2-rc2
416 -rw-r--r-- 1 root root 419840 Mar 9 12:34 hping2.0.0-rc2.tar.gz.tar
8 drwx------ 2 root root 8192 Sep 11 16:28 orbit-root
4 drwx------ 2 rsiles rsiles 4096 Sep 11 2002 orbit-rsiles
As can be seen, the theoretical value "1664" for the
second file listed (1703053/4096 is 416; 416 * 4 = 1664), is different from the 1668 block count displayed by "ls".
The accurate information about the value shown by "ls" can be obtained from its
source code. The "ls" binary in Linux, for example Red Hat ("rpm -q -f /bin/ls") is included in the "fileutils-4.1.9-11" source package. The "ls.c" source file contains the
"gobble_file()" C function responsible of calculating the total block count.
2)- Hard link count analysis:
Another evidence extracted from the displayed output is the hard link count.
A standard Linux file has associated a count of 1 hard link (corresponding to itself). One Linux file having an additional hard link (for example, created with the "ln" command), goes to a count of 2 links
(itself and the hard link). The number of hard links is increased by one for every new hard link added.
The hard link count of the directory where the file resides (".") or the parent directory ("..") is not influence at all.
Besides, a standard Linux directory by default has 2 hard links,
corresponding to itself, the "." entry and to its entry belonging to its parent directory: the parent directory references this directory in its own listing.
However, this situation varies when the
directory has subdirectories because all them reference the parent directory through the ".." entry. So, a directory with 3 subdirectories will have a hard link count of 5, 2 associated to the default
behaviour (itself and the parent dir reference) and 3 associated to the ".." references of the 3 subdirectories.
This is exactly the situation of the "/tmp/.. " directory analyzed by Skodo;
although the rootkit hides the subdirectories we may guess that there are 3 hidden objects (at least, because this method cannot be applied to files).
Exercise yourself with the hard link count of the
"/tmp" directory. As can be seen, 7 is the correct hard link count, 2 (default) plus 5 subdirectories.
CONCLUSION: It seems the rootkit is hiding some directories too !!
Finally, the hard link
count of the ".." entry of the "/tmp" dir seems to be the usual one, 19 hard links associated to the "/" dir: "., .., bin, boot, dev, etc, home, initrd, lib, lost+found, misc, mnt,
opt, proc, root, sbin, tmp, usr and var".
To end up with the evidence analysis, the timestamps of the different files and directories should be analyzed. As explained before, the default "ls"
timestamp reflects the modification time of the object contents:
- File: it reflects the last time the file contents were modified.
In the example, all the relevant files inside the "/tmp/ directory
(.mov, .doc, .mp3, .mpeg and .rtf) were modified between "Dec 28 19:38" and "Dec 28 19:47".
- Directory: it reflects when the directory listing (its contents) has been modified, adding or removing entries (files or directories).
When a new file or directory is created (or removed from) inside a directory,
for example the "/tmp" directory, its contents change so its timestamp is refreshed in:
- Its entry in the parent directory: the "/tmp" entry of the "/" directory.
- Its own reference: the "." entry of the "/tmp" directory.
- The parent references of all its subdirectories: the ".." entries in all the "/tmp" subdirectories.
These
timestamps will have the same value as the timestamp of the new created object or the moment when the object was removed. These updates are not applied recursively, that is, the parent directory (of a given directory)
doesn't update its timestamp when the child directory does.
When a file changes its timestamp, the directory containing it doesn't have to change its timestamp; in a directory the timestamp is only changed to
reflect the variations in the components of its list.
NOTE: Don't be confused by the behaviour presented when a file is edited with, for example, "vim". Due to the fact that the editor creates a temporary
file, ".file.swp", the directory always change its timestamp.
Therefore, in the Skodo fileserver, the last time the "/tmp/ contents were modified was "Dec 28 19:47", confirmed by the
timestamp of the newest file, "star_wars_kid_lotr.mpeg". Thus, no files/directories were added or removed since then.
The timestamp of the "/tmp/.. " directory is greater than this value
("Dec 28 20:12"), which could be correct based on the previous comment related with the recursion.
Thus, the last time the "/tmp/.. " directory contents were modified is not clear based on the
evidence available, because its entry in the "/tmp" directory reflects a timestamp value of "Dec 28 20:12" while the timestamp of its own reference (".") reflects a value of "Dec 28
20:35". Both values should always be the same.
So, this situation is only possible (in a non-rootkitted system) if:
- The last time the "/tmp/.. " directory was modified, BEFORE Skodo typed the last "ls -al" command, was "Dec 28 20:12".
- Skodo was working December 28th, around 20:35 in this incident (the terminal session took place in this moment).
- During the short period between the two "ls -al" commands (of "/tmp" and
"/tmp/.. "), the contents of the "/tmp/.. " directory were modified. This could mean Smeagol was modifying the directory contents while Skodo was typing the "cd ".. "" command.
----
Raśl Siles (raul.siles@hp.com)
# # # #
## ## ## ##
# # # # # # # # ARCH
# # ##### # # #
# # # # ADRID
# # # #
##### ##### # #
|
|
