Monday (0800-1230)

The lab is split in two parts, 4 sessions this week and another week later this spring. We got files for examination on USB sticks and the files were also uploaded to Fronter. The main scenario is about a company "WidgetCo" where suspicious network flow to a web server has been discovered and the chief security officer think an old employee, the senior system administrator who was recently laid off, might have something to do with it. We are going to have a kind of "role play", court simulation after the 2nd lab week where we present our finding on what happen.

We got access to two images of Linux servers "hacked-rh62server.dd" and "widgetco.dd" and spent most of the time with a "command refresher"

On virtual machine (SIFT workstation):

"cd /" # change working directory to root
"ls | wc -l" # list folders and files, and (word) count number of lines
"df -h" # free space
"md5sum <file>" # find md5 hash of a file
"for file in /sbin/*; do md5sum $file; done" # loop trough all files in folder and print md5 sum for file
"whoami" # show current logged in user
"ps aux" # list running processes
"/etc/passwd" # a file containing user names and permissions, world readable
"/var/log/dmesg" # log file with device error messages
"du -h /var/log" # list disk usage for each folder
"su" # switch user for whole session
"sudo" # switch user for this action only

And on the hacked-rh62server.dd image (used for repetition)

"tar jxf hckdrh62.tar.bz2" # unzip file
"parted hacked-rh62server.dd" # find partition table and guess where the root is (it's #8)
"unit b" # byte offsets
"print" # get the offset of the partition for mounting
"sudo losetup -r /dev/loop0 <image-file>" # create the loopback device, read only from image file
"sudo mount -o ro,loop,offset=3742534656B /dev/loop0 <mount-point>" # mount loopback device at offset, read only in mounting point
"/etc/fstab" # contains the mount point for the remaining partitions
"sudo mount -o ro,loop,offset=1735566336B /dev/loop0 ./home/" # mounting /home inside current working directory. Inside it we find the bash history file with reference to folder with name " " (space).
"/home/drosen/.bash_history"
"find <path> -name ' '" # locate the folder with name <space>
"find <path> -newer ./home/drosen/.bash_history" # to find latest files

Starting on the analysis of log files, switching to the widgetco.dd image

"syslog" and "message" # System
"auth.log" and "secure" # Authentication
"wtmp" and "btmp" # Login
"access.log" and "error.log" # Apache
"grep '[0-1\{0,1\}[0-9]\{1,2\}.[0-1\{0,1\}[0-9]\{1,2\}.[0-1\{0,1\}[0-9]\{1,2\}.[0-1\{0,1\}[0-9]\{1,2\}'" # find IPs in plain text files
"cat <file> | cut -d " " -f 1 | sort | uniq -c" # cat/grep, then cut each line by space separator, select field 1, sort every line and get unique lines with count.

Wednesday (1300-1630)
We looked at the apache2 logs, mostly the access logs containing all GET (page) requests. The first thing to trigger suspicion is the size of some of the files. It's recommended to copy the files to local area because of the read only mounting and the current ownership of the files. (It's repeatable, so that's fine). We use command

gunzip *

to uncompress all the compressed old log files. We know that the address 10.10.10.200 and 10.10.10.199 is simulating external addresses, and we see a brute force attack from .200 using "nikto" vulnerability scanner. It's searching for known files to determine known vulnerabilities.

Next we use grep in order to remove all the noise with the -v flag:

cat access.log | grep -v "nikto"

We then are able to spot something strange. Actions against the Joomla installation with eval in the options. We also see the words UNION and SELECT which is a bad sign. The code given in the eval argument is encoded in base64 and we are therefore easily able to decode it using online translates.

We also learned how to parse stuff using sed to replace symbols (like encoded ASCII symbols):

echo "+ %3E %22" | sed 's\+\-\g' | sed 's\%3E\>\g' | sed 's\%22\"\g'

Less is more and we checked out 3 executable with file, strings and calculating/searching for hash (md5sum). It's amazing how much you can find just by using these high level tools.

Thursday (1330-1700)
We continue to figure out:

• Earliest evidence of attack (who, when, how)
  
• Any back doors still present?
  
• What was stolen (if anything)?
  

We then talked about the sleuth kit in more detail, and used it to create timelines of all the files. mmstat, mmls, fsstat, blkls, blkcalc, blkcat, istat, icat, ifind, fls to mention a few of them.

fls -r -m "/" -o 63 -z UTC <image>
mactime <bodyfile (output from fls)> <password_file> <group_file> -z UTC -d
istat -o 63 <image> <inode>
blkcat (-h) -o 63 <image> <block_id> | less
log2timeline -r -f linux -z UTC -o mactime -w widgetco.l2t /evicence/ # using mounted image at /evidence/

We had a discussion on disk sectors, advanced format drives and generally how the forensics community for a long time have convinced the courts the only sound way to do forensics is to unplug the computer, disconnect the drive, write block it and extract images. Now we see that the live system often contains too important information (RAM) and the disk could be encrypted. The forensics community is based on anecdotal knowledge.

The last part of today was a walk through of the report we are going to fill inn. Presentation of one of the students on Friday at around 1100.