Tag Archives: log2timeline

Intro to Linux Forensics

This article is a quick exercise and a small introduction to the world of Linux forensics.  Below, I perform a series of steps in order to analyze a disk that was obtained from a compromised system that was running a Red Hat operating system. I start by recognizing the file system, mounting the different partitions, creating a super timeline and a file system timeline. I also take a quick look at the artifacts and then unmount the different partitions. The process of how to obtain the disk will be skipped but here are some old but good notes on how to obtain a disk image from a VMware ESX host.

When obtaining the different disk files from the ESX host, you will need the VMDK files. Then you move them to your Lab which could be simple as your laptop running a VM with SIFT workstation. To analyze the VMDK files you could use the “libvmdk-utils” package that contain tools to access data store in VMDK files. However, another approach would be to convert the VMDK file format into RAW format. In this way, it will be easier to run the different tools such as the tools from The Sleuth Kit – which will be heavily used – against the image. To perform the conversion, you could use the QEMU disk image utility. The picture below shows this step.

Following that, you could list the partition table from the disk image and obtain information about where each partition starts (sectors) using the “mmls” utility. Then, use the starting sector and query the details associated with the file system using the “fsstat” utility. As you could see in the image, the “mmls” and “fsstat” utilities are able to identify the first partition “/boot” which is of type 0x83 (ext4). However, “fsstat” does not recognize the second partition that starts on sector 1050624.

This is due to the fact that this partition is of type 0x8e (Logical Volume Manager). Nowadays, many Linux distributions use LVM (Logical Volume Manager) scheme as default. The LVM uses an abstraction layer that allows a hard drive or a set of hard drives to be allocated to a physical volume. The physical volumes are combined into logical volume groups which by its turn can be divided into logical volumes which have mount points and have a file system type like ext4.

With the “dd” utility you could easily see that you are in the presence of LVM2 volumes. To make them usable for our different forensic tools we will need to create device maps from the LVM partition table. To perform this operation, we start with “kpartx” which will automate the creation of the partition devices by creating loopback devices and mapping them. Then, we use the different utilities that manage LVM volumes such as “pvs”, “vgscan” abd “vgchange”. The figure below illustrates the necessary steps to perform this operation.

After activating the LVM volume group, we have six devices that map to six mount points that make the file system structure for this disk. Next step, mount the different volumes as read-only as we would mount a normal device for forensic analysis. For this is important to create a folder structure that will match the partition scheme.

After mounting the disk, you normally start your forensics analysis and investigation by creating a timeline. This is a crucial step and very useful because it includes information about files that were modified, accessed, changed and created in a human readable format, known as MAC time evidence (Modified, Accessed, Changed). This activity helps finding the particular time an event took place and in which order.

Before we create our timeline, noteworthy, that on Linux file systems like ext2 and ext3 there is no timestamp about the creation/birth time of a file. There are only 3 timestamps. The creation timestamp was introduced on ext4. The book “The Forensic Discovery 1st Edition”from Dan Farmer and Wietse Venema outlines the different timestamps:

  • Last Modification time. For directories, the last time an entry was added, renamed or removed. For other file types, the last time the file was written to.
  • Last Access (read) time. For directories, the last time it was searched. For other file types, the last time the file was read.
  • Last status Change. Examples of status change are: change of owner, change of access permission, change of hard link count, or an explicit change of any of the MACtimes.
  • Deletion time. Ext2fs and Ext3fs record the time a file was deleted in the dtime stamp.the filesystem layer but not all tools support it.
  • Creation time: Ext4fs records the time the file was created in crtime stamp but not all tools support it.

The different timestamps are stored in the metadata contained in the inodes. Inodes are the equivalent of MFT entry number on a Windows world. One way to read the file metadata on a Linux system is to first get the inode number using, for example, the command “ls -i file” and then you use “istat” against the partition device and specify the inode number. This will show you the different metadata attributes which include the time stamps, the file size, owners group and user id, permissions and the blocks that contains the actual data.

Ok, so, let’s start by creating a super timeline. We will use Plaso to create it. For contextualization  Plaso is a Python-based rewrite of the Perl-based log2timeline initially created by Kristinn Gudjonsson and enhanced by others. The creation of a super timeline is an easy process and it applies to different operating systems. However, the interpretation is hard. The last version of Plaso engine is able to parse the EXT version 4 and also parse different type of artifacts such as syslog messages, audit, utmp and others. To create the Super timeline we will launch log2timeline against the mounted disk folder and use the Linux parsers. This process will take some time and when its finished you will have a timeline with the different artifacts in plaso database format. Then you can convert them to CSV format using “psort.py” utility. The figure below outlines the steps necessary to perform this operation.

Before you start looking at the super timeline which combines different artifacts, you can also create a traditional timeline for the ext file system layer containing data about allocated and deleted files and unallocated inodes. This is done is two steps. First you generate a body file using “fls” tool from TSK. Then you use “mactime” to sort its contents and present the results in human readable format. You can perform this operation against each one of the device maps that were created with “kpartx”. For sake of brevity the image below only shows this step for the “/” partition. You will need to do it for each one of the other mapped devices.

Before we start the analysis, is important to mention that on Linux systems there is a wide range of files and logs that would be relevant for an investigation. The amount of data available to collect and investigate might vary depending on the configured settings and also on the function/role performed by the system. Also, the different flavors of Linux operating systems follow a filesystem structure that arranges the different files and directories in a common standard. This is known as the Filesystem Hierarchy Standart (FHS) and is maintained here. Its beneficial to be familiar with this structure in order to spot anomalies. There would be too much things to cover in terms of things to look but one thing you might want to run is the “chkrootkit” tool against the mounted disk. Chrootkit is a collection of scripts created by Nelson Murilo  and Klaus Steding-Jessen that allow you to check the disk for presence of any known  kernel-mode and user-mode rootkits. The last version is 0.52 and contains an extensive list of known bad files.

Now, with the supertimeline and timeline produced we can start the analysis. In this case, we go directly to timeline analysis and we have a hint that something might have happened in the first days of April.

During the analysis, it helps to be meticulous, patience and it facilitates if you have comprehensive file systems and operating system artifacts knowledge. One thing that helps the analysis of a (super)timeline is to have some kind of lead about when the event did happen. In this case, we got a hint that something might have happened in the beginning of April. With this information, we start to reduce the time frame of the (super)timeline and narrowing it down. Essentially, we will be looking for artifacts of interest that have a temporal proximity with the date.  The goal is to be able to recreate what happen based on the different artifacts.

After back and forth with the timelines, we found some suspicious activity. The figure below illustrates the timeline output that was produced using “fls” and “mactime”. Someone deleted a folder named “/tmp/k” and renamed common binaries such as “ping” and “ls” and files with the same name were placed in the “/usr/bin” folder.

This needs to be looked further. Looking at the timeline we can see that the output of “fls” shows that the entry has been deleted. Because the inode wasn’t reallocated we can try to see if a backup of the file still resides in the journaling. The journaling concept was introduced on ext3 file system. On ext4, by default, journaling is enabled and uses the mode “data=ordered”. You can see the different modes here. In this case, we could also check the options used to mount the file system. To do this just look at “/etc/fstab”. In this case we could see that the defaults were used. This means we might have a chance of recovering data from deleted files in case the gap between the time when the directory was deleted and the image was obtained is short. File system journaling is a complex topic but well explained in books like “File system forensics” from Brian Carrier. This SANS GCFA paper from Gregorio Narváez also covers it well.  One way you could  attempt to recover deleted data is using the tool “extundelete”.  The image below shows this step.

The recovered files would be very useful to understand more about what happened and further help the investigation. We can compute the files MD5’s, verify its contents and if they are known to the NSLR database or Virustotal. If it’s a binary we can do strings against the binary and deduce functionality with tools like “objdump” and “readelf”. Moving on, we also obtain and look at the different files that were created on “/usr/sbin” as seen in the timeline. Checking its MD5, we found that they are legitimate operating system files distributed with Red Hat. However, the files in “/bin” folder such as “ping” and “ls” are not, and they match the MD5 of the files recovered from “/tmp/k”.  Because some of the files are ELF binaries, we copy them to an isolated system in order to perform quick analysis. The topic of Linux ELF binary analysis would be for other time but we can easily launch the binary using “ltrace -i” and “strace -i” which will intercept and record the different function/system calls. Looking at the output we can easily spot that something is wrong. This binary doesn’t look the normal “ping” command, It calls the fopen() function to read the file “/usr/include/a.h” and writes to a file in /tmp folder where the name is generated with tmpnam(). Finally, it generates a segmentation fault. The figure below shows this behavior.

Provided with this information, we go back and see that this file “/usr/include/a.h” was modified moments before the file “ping” was moved/deleted. So, we can check when this “a.h” file was created – new timestamp of ext4 file system – using the “stat” command. By default, the “stat” doesn’t show the crtime timestamp but you can use it in conjunction with “debugfs” to get it. We also checked that the contents of this strange file are gibberish.

So, now we know that someone created this “a.h” file on April 8, 2017 at 16:34 and we were able to recover several other files that were deleted. In addition we found that some system binaries seem to be misplaced and at least the “ping” command expects to read from something from this “a.h” file. With this information we go back and look at the super timeline in order to find other events that might have happened around this time.  As I did mention, super timeline is able to parse different artifacts from Linux operating system. In this case, after some cleanup we could see that we have artifacts from audit.log and WTMP at the time of interest. The Linux audit.log tracks security-relevant information on Red Hat systems. Based on pre-configured rules, Audit daemon generates log entries to record as much information about the events that are happening on your system as possible. The WTMP records information about the logins and logouts to the system.

The logs shows that someone logged into the system from the IP 213.30.114.42 (fake IP) using root credentials moments before the file “a.h” was created and the “ping” and “ls” binaries misplaced.

And now we have a network indicator. Next step we would be to start looking at our proxy and firewall logs for traces about that IP address. In parallel, we could continue our timeline analysis to find additional artifacts of interest and also perform in-depth binary analysis of the files found, create IOC’s and, for example, Yara signatures which will help find more compromised systems in the environment.

That’s it for today. After you finish the analysis and forensic work you can umount the partitions, deactivate the volume group and delete the device mappers. The below picture shows this steps.

Linux forensics is a different and fascinating world compared with Microsoft Windows forensics. The interesting part (investigation) is to get familiar with Linux system artifacts. Install a pristine Linux system, obtain the disk and look at the different artifacts. Then compromise the machine using some tool/exploit and obtain the disk and analyze it again. This allows you to get practice. Practice these kind of skills, share your experiences, get feedback, repeat the practice, and improve until you are satisfied with your performance. If you want to look further into this topic, you can read “The Law Enforcement and Forensic Examiner’s Introduction to Linux” written by Barry J. Grundy. This is not being updated anymore but is a good overview. In addition, Hal Pomeranz has several presentations here and a series of articles written in the SANS blog, specially the 5 articles written about EXT4.

 

References:
Carrier, Brian (2005) File System Forensic Analysis
Nikkel, Bruce (2016) Practical Forensic Imaging

Tagged , , , ,

Digital Forensics – SuperTimeline & Event Logs – Part I

In these series of articles about performing file system forensics on a Windows system we covered the evidence acquisition in the first article. The second article was about processing the evidence and creating a timeline of the NTFS metadata. The one below will be split in two parts and will cover the analysis of a Super Timeline and the different artifacts. The main focus will be on parsing and analyzing Windows Event Logs.

In this article we will not cover the creation of the super timeline because has been covered already in this article using Plaso engine. We will move on directly to the analysis and artifacts.

For contextualization  Plaso is a Python-based rewrite of the Perl-based log2timeline initially created by Kristinn Gudjonsson and enhanced by others. The creation of a super timeline is an easy process and it applies to different Microsoft Windows operating systems. However, the interpretation is hard. The Plaso engine is able to parse different type of artifacts such as Windows Event Logs, Windows Registry, Windows Prefetch files and many others. During the analysis it helps to be meticulous, patience and it facilitates if you have comprehensive file systems, operating system artifacts and registry knowledge.

One thing that facilitates the analysis of a Super Timeline is to have some kind of lead about when the event did happen. Then you start to reduce the time frame of Super Timeline and narrowing it down. Essentially we will be looking for artifacts of interest that have a temporal proximity with the event.  The goal is to be able to recreate what happen based on the different artifacts.

A good starting point is looking at the different Windows Event Logs that were recorded in the Super Timeline. But before we dig into our Super Timeline and the WinETX artifacts let’s review some concepts about the Windows Event Logs.

The Windows Event Logs are a good source of information when looking for potential attacks or misuse of the system. The Windows Event Logs include all the records that are a result of the enforced auditing policy i.e., the amount of information is dependent on the Audit Policy settings. For example on a standalone Windows 7 default installation the Audit Policies state that there is no auditing. This can been seen in the local group policy settings as you could see in the below image.

windowsauditpolicy

However, there is some auditing enable and you could verify it using the command line auditpol.exe tool. The below picture shows the auditpol.exe output in the same Windows installation from where the group policy stated that there was no auditing enable. This inconsistency is describe by Microsoft here.

windowsauditpolicy2

From Windows Vista onwards the Windows Event Logs are stored in the  %SystemRoot%\System32\Winevt\Logs folder and contain a .EVTX extension. This new format called Windows XML Event Log (EVTX) format super seeds the EVT format used in Windows XP. Apart of Event Logs from Vista onward there are Application and Service logs that record events about a particular component or application rather then system. On a Windows 7/2008 System many event log files can be found depending on the roles performed by the system. In here you can even find application event logs. For example if the system has Symantec Endpoint you will have a “Symantec Endpoint Protection Client.evtx” file.  The different categories are described here. Nonetheless, there are three important files. The description was taken from Microsoft Technet website.

Application.evtx – “The Application log contains events logged by applications or programs. For example, a database program might record a file error in the application log. Program developers decide which events to log.”

System.evtx – “The Security log contains events such as valid and invalid logon attempts, as well as events related to resource use, such as creating, opening, or deleting files or other objects. Administrators can specify what events are recorded in the security log. For example, if you have enabled logon auditing, attempts to log on to the system are recorded in the security log.”

Security.evtx – “The System log contains events logged by Windows system components. For example, the failure of a driver or other system component to load during startup is recorded in the system log. The event types logged by system components are predetermined by Windows.”

To be able to parse and read these files there are good open source tools available. The work pioneered by Andreas Schuster in 2007  and afterwards by Joachim Metz on Libevtx were key to develop this capability.

One tool is the Plaso engine but that we already covered in another post. Another is the Python-evtx created by Willi Ballenthin. This tool written in Python has the ability to read the EVTX format. The tool is available on Github and contains different modules. One is the evtxdump.py that allows us to dump the contests of the Event Log files. The following picture shows the installation and a simple usage of this tool in the SIFT workstation.

python-evtx

Other than using open source tools you can also use Windows Event Log Parser from TZworks. Or you can use the native Microsoft tools. For example using the Windows GUI and use Event Viewer or you could use Windows CLI and use wevtutil.exe. Another option is the versatile  Microsoft Log Parser.

The below pictures illustrates an example where we map a network drive to point to our evidence that is mounted on the SIFT workstation. Then we use wevutil.exe to query the Event Logs. First example with no filter and second example with a XML filter that was created using the Windows Event Viewer.

wevtutil

Please note that in this exercise the Security.evtx contains logon events because the system is not part of a Active Directory Domain. If that was the case the logon events will be recorded on the Domain Controllers.

Now, that we briefly covered the Windows Event Logs let’s review Audit Policy Settings. This is important because will allow us to to know what to expect on the Event Logs. The Audit Policy Settings control which events will be recorded on the Event Logs.

To be able to read the Audit Policy settings we need to read a special Registry key.  At the time of this writing I could not find an open source tool that allowed us to retrieve these settings from a forensic image. The Audit Policy settings are stored in the registry under the HKEY_LOCAL_MACHINE\Security\Policy\PolAdtEv key. RegRipper is able to parse and output the contents of this key on Windows XP but seems to have issues on Windows 7 as you could on the below image. Nonetheless, it can retrieve the contents of the key. By having the contents of the registry key we can use the paper created by Teru Yamazaki that describes the structure of this key on newer Microsoft operating systems. With the information we can understand which audit settings are defined. The below image illustrates a manual mapping between the different events and the possible settings.

windowsauditpolicy3

Teru Yamazaki also has an online tool on his website that allows you to parse this information automatically. You can get the audit policy from the SECURITY registry file using RegRipper and paste the contents into his tool. The output will be in Japanese but you can use Google Translator. In this exercise as seen by the Audit Policy settings we have Success and Failure turned on for all the event types.

Now that we know which kind of auditing settings we can expect and we reviewed some of the tools we could use to read the Event Logs we can start analyzing our Super Timeline and WinEVTX artifacts. This will be done on part II.

Tagged , , , , , , , ,

Forensics Evidence Processing – Super Timeline

After evidence acquisition, you normally start your forensics analysis and investigation by doing a timeline analysis. This is a crucial step and very useful because it includes information on when files were modified, accessed, changed and created in a human readable format, known as MAC time evidence. This activity helps finding the particular time an event took place and in which order. The traditional timeline analysis is done to the file system and has been used for several years and folks like Rob Lee and others have been championing it for years. The data is gathered using a variety of tools and is extracted from the metadata layer of the file system (inode on Linux or MFT records on Windows) and then parsed and sorted in order to be analyzed. The end goal is to generate a snapshot of the activity done in the system including its date, the artifact involved, action and source. The creation is an easy process but the interpretation is hard. During the interpretation it helps to be meticulous and patience and it facilitates if you have comprehensive file systems and operating system artifacts knowledge. To accomplish this step several commercial or open source tools exists such as the SANS Investigate Forensic Toolkit (SIFT) that is freely available and frequently updated.

In June 2010, the SANS reading room published a paper from Kristinn Gudjonsson as part of his GCFA gold certification introducing a new method called super timeline and a tool called log2timeline. The super timeline goes beyond the traditional file system timeline creation based on metadata extracted from acquired images by extending it with more sources of data in an automated fashion. As stated on Kristinn paper the super timeline includes more artifacts that provide vital information to the investigation. Basically the super timeline is a timeline of timelines that are gathered from the file system, registry keys and windows artifacts producing a single correlated timeline. To create the super timeline, Kristinn originally developed a framework that is materialized in the log2timeline tool written in Perl. The tool uses a modular approach that makes easier for someone to contribute to the project by developing a new module. A new module could be for example a new parser for a specific windows artifact. The first release  introduced several parsers for windows artifacts such as the ability to create time stamped data of Chrome, Firefox, Opera and Internet Explorer browser history, Mcafee Antivirus log file, Open XML used in the Microsoft Office documents metadata, PDF metadata, pcap network captures, UserAssist registry key and others. The log2timeline has been actively developed by Kristinn and others and its backend is currently being replaced by a new framework named Plaso that still contains the log2timeline tool as a frontend and other many other tools written in Python.

In order to practice the production of a super timeline I created a realistic scenario that consists of a Windows XP system that has been compromised by w32/Morto worm just by having remote desktop service exposed to the internet with poor administrator password. Moments after being compromised the machine was taken offline and a forensic image bit-by-bit was created.  The image was then moved to the SIFT workstation for analysis.

5 steps are needed to create the super timeline using the SIFT workstation and the logt2timeline.py front-end tool from the plaso suite.

supertimeline

 

Now with the evidence sorted and reduced I can start doing my analysis, investigation and looks for signs of Evil using for example Excel. An piece of the super timeline is showed in the following picture.

supertimeline-csv

 

The creation of a super timeline is an easy process and it applies to different windows operating systems but the interpretation is hard.  During the interpretation it helps to be meticulous and patience and it facilitates if you have comprehensive file systems, operating system artifacts and registry knowledge. With these type of methodology one can practice and improve their ability to determine past actions which have taken place and understand all kinds of artifacts which occur within the super timeline. On the SANS website there is another great article on how to create a super timeline with log2timeline. Here you can also view a 2011 webcast that looks at what is and how to do super timeline analysis. These and other steps to create a super timeline are well detailed in the log2timeline cheat sheet created by David Nides. The analysis and investigation will be part of other post.

Tagged , , , ,