Tag Archives: NTFS Metadata

May 25 2017
Leave a comment
Digital Forensics and Incident Response

Digital Forensics – NTFS Change Journal

Last year, I wrote a series of articles about digital forensics. I covered different artifacts that can be useful when conducting incident response. In the last article of those series, I covered The NTFS INDX attribute which is used to store metadata about files inside directories and the $LogFile metadata file, which keeps record of all operations that occurred in the NTFS volume. Both these artifacts can give the investigator a great amount of information about attacker activity. However, there is another special file inside NTFS that also contains a wealth of historical information about operations that occurred on the NTFS volume, the Update Sequence Number (USN) journal file named $UsnJrnl.

While the different file operations occur on disk, in a NTFS volume, the change journal keeps record of the reason behind the operation such as file creation, deletion, encryption, directory creation, deletion, etc. There is a USN change journal per volume, its turned on by default since Windows Vista, and used by applications such as the Indexing Service, File Replication Service (FRS), Remote Installation Services (RIS), and Remote Storage. Nonetheless, applications and Administrators can create, delete, and re-create change journals. The change journal file is stored in the hidden system file $Extend\$UsnJrnl. The $UsnJrnl file contains two alternate data streams (ADS). The $Max and the $J. The $Max data streams contains information about the change journal such as the maximum size. The $J data stream contains the contents of the change journal and includes information such as the date and time of the change, the reason for the change, the MFT entry, the MFT parent entry and others. This information can useful for an investigation, for example, in a scenario where the attacker is deleting files and directories while he moves inside an organization in order to hide his tracks. To obtain the change journal file you need raw access to the file system.

So, on a live system, you could check the size and status of the change journal by running the command “fsutil usn queryjournal C:” on a Windows command prompt with administrator privileges. The “fsutil” command can also be used to change the size of the journal. Fom a live system, you could also obtain the change journal file using a tool like RawCopy or ExtractUsnJrnl from Joakim Schicht. In this particular system the maximum size of the change journal is 0x2000000 bytes.

Now, let’s perform a quick exercise about obtaining the change journal file from a disk image. First, we use the “mmls” utility to see the partition table from the disk image. Then, we use “fls” from The Sleuth Kit to obtain a file and directory listing and grep for the UsnJrnl string. As you could see in the picture below the output of “fls” shows that the filesystem contains the $UsnJrnl:$Max and $UsnJrnl:$J files. We are interested in the MFT entry number which is 84621.

Next, let’s review MFT record properties for the entry number 84621 with the command “istat” from The Sleuth Kit. This MFT entry stores the NTFS metadata about the $UsnJrnl. We are interested in the attributes section, more specifically, we are looking for the identifier 128 which points to the $DATA attribute. The identifier 128-37 points to the $Max data stream which is of size 32 bytes and is resident. The identifier 128-38 points to the $J data stream which is of size 40-GBytes and sparse. Then we use the “icat” command to view the contents of the $Max data stream which can gives the maximum size of the change journal and then we also use “icat” to export the $J data stream into a file. Noteworthy, that the change journal is sparse. This means parts of the data is just zeros. However, icat from The Sleuth Kit will extract the full size of the data stream. A more efficient and faster tool would be ExtractUsnJrnl because it only extracts the actual data. The picture below illustrates the steps necessary to extract the change journal file.


Now that we exported the change journal into a file we will use the UsnJrnl2Csv utility. Once again another brilliant tool from Joakim Schicht. The tool supports USN_RECORD_V2 and USN_RECORD_V3, and makes it very easy to parse and extract information from the change journal. The output will be a CSV file. The picture below shows the tool in action. You just need to browse the change journal file you obtained and start parsing it.

This process might take some time, when finished, you will have a CSV file containing the journal records. This file be can easily imported into Excel. Then, filter based on the  reason and timestamp fields. Normally when you do such analysis you already have some sort of a lead and you have a starting point that will help uncover more leads and findings. After analyzing the change journal records we can start building a timeline of events about attacker activity.  Below picture shows a timeline of events from the change journal about malicious files that were created and deleted. These findings can then be used as indicators of compromise in order to find more compromised systems in the environment. In addition, for each file you have the MFT entry number that could be used to attempt to recover deleted files. You might have a chance of recovering data from deleted files in case the gap between the time when the file was deleted and the image was obtained is short.

The change journal contains a wealth of information that shouldn’t be overlooked. Another interesting aspect of the change journal is that allocates space and deallocates as it grows and records are not overwritten unlike the $LogFile. This means we can find old journal records in unallocated space on a NTFS volume. How to obtain those? Luckily, the tool USN Record Carver written by PoorBillionaire can carve journal records from binary data and thus recover these records .
That’s it! In this article we reviewed some introductory concepts about the NTFS change journal and how to obtain it, parse it and create a timeline of events. The techniques and tools are not new. However, they are relevant and used in today’s digital forensic analysis. Have fun!

References:

Windows Internals, Sixth Edition, Part 2 By: Mark E. Russinovich, David A. Solomon, and Alex Ionescu
File System Forensic Analysis By: Brian Carrier

Tagged , , ,
May 30 2016
2 Comments
Digital Forensics and Incident Response, Incident Handling and Hacker Techniques

Digital Forensics – NTFS INDX and Journaling

indxcrimesceneLet’s continue our digital forensics journey and start where we left off.  To contextualize the reader, the posts below are the previous articles on this series:

We now want to find traces of the DLL – CLB.DLL – that was invoked from the wrong directory and understand what happened e.g., was it deleted by the attacker?  To support our quest we need to look deep into the NTFS file system.

One of the first articles I wrote on this series was a brief explanation about the NTFS metadata and the Master File Table – which might give you more background on the topic below. To summarize, the Master File Table (MFT) is a special system file that resides on the root of every NTFS partition. The file is named $MFT and is not accessible via user mode API’s but can been seen when you have raw access to the disk e.g, forensic image. This special file is a hierarchical database and inside you have records that contains a series of attributes about a file, directory and indicates where it resides on the physical disk and if is active or inactive. The size of each MFT record is usually 1024-bytes. In the last post we reviewed three relevant attributes of the MFT records ; the $STANDART_INFORMATION, the $FILENAME and $DATA. The first two are rather important because among other things they contain the file time stamps..

In this article we will look at the MFT records for directories (further down we will also look at the NTFS Journaling).

Why? Because the MFT records for directories contain a special attribute called $I30. This attribute contains information about file names and directories that are stored inside a particular directory. This special attribute is also known as $INDX and consists of three attributes, the $INDEX_ROOT, $INDEX_ALLOCATION and $BITMAP.

So, What? Well, this attribute stores information in a B-tree data structure in order to keep data sorted so the operating system can perform fast searches in order to determine if a file is present. In addition, this attribute grows in size in order to keep track of file names inside the directory. However, when you delete a file from a directory the B-tree re-balances itself but the tree node with metadata about the deleted file remains in a form of slack space until it gets reused. This means we can view the $I30 attribute contents and we might find evidence of files that once existed in a particular directory but are no longer there. The contents of the $I30 attribute contain the file name, the MACE time stamps from the $FILE_NAME, the file size and the MFT entry number for the file and parent directory.

How can we see the $I30 contents for a particular directory? We will cover two different methods. First method, is a manual approach using the tools available in The Sleuth Kit to extract it and then we use the INDXParser.py tool from Willi Ballenthin to parse its contents. The second method is faster and we will use the tool LogFileParser from Joakim Schicht.

Let’s cover the first approach using the SIFT workstation. You can view the $INDX contents in four steps:

  1. Use fls against a disk image or ls against a mounted imaged to get the MFT entry number for the directory we want to analyze. In this case the Windows directory.
  2. Use istat to get the identifier number for the $I30 attribute. You can also verify if it contains resident data.
  3. Use icat to dump the $INDX contents into a file.
  4. Use INDXParser.py to parse its contents and view file names, size and MACE timestamps.

The image below illustrates the first step where we use fls to read the EWF disk image and get the its contents.

ntfsindx-fls

The image below illustrates the second step where we can view the MFT record properties for the entry number 506.

ntfsindx-tsk

In our case the Windows folder is stored in the MFT entry number 506 and the identifier we are looking is the 160 which points to the contents of the $INDX. The identifier 144 points to the root node.  With this information, we can dump the data using icat. Then we execute INDXParser.py with the suffix –d in order to search in slack space.

ntfsindx-parser

Unfortunately, in this case we could not find traces of our DLL. This might occur due to the B-tree nodes of the $I30 data structure being reused. This happens due to the normal file system activity. Normally, there is good chance to find evidence about deleted files in the $I30 attributes using the first method. But if for some reason we couldn’t find any traces we could try the second method.

The second method relies on LogFileParser.exe to parse a different NTFS metadata file which is called $LOGFILE. Before we try this method let’s review what is the $LOGFILE. NTFS has been developed over years with many features in mind, one being data recovery. One of the features used by NTFS to perform data recovery is the Journaling. The NTFS Journal is kept inside NTFS Metadata in a file called $LOGFILE. This file is stored in the MFT entry number 2 and every time there is a change in the NTFS Metadata, there is a transaction recorded in the $LOGFILE. These transactions are recorded in order to be possible to redo or undo file system operations. After the transaction has been logged then the file system can perform the change. When the change is done, another transaction is logged in the form of a commit. The $LOGFILE allows the file system to recover from metadata inconsistencies such as transactions that don’t have a commit. The size of the $LOGFILE can be consulted and changed using chkdsk /l and per default is 65536 KB.

Why would $LOGFILE be important for our investigation? Because the $LOGFILE keeps record of all operations that occurred in the NTFS volume such as file creation, deletion, renaming, copy, etc.  Therefore, we might find traces of our file here.

Now, let’s use our second method. We had to give you context about the $LOGFILE because the LogFileParser.exe is able to parse $INDX attributes based on information available in the transaction logs. We will now use the powerful, granular tool developed by Joakim Schicht called LogFileParser.  For those who are not familiar with Joakim Schicht, he is a brilliant engineer and researcher who has enormously contributed to the Forensics community with many powerful tools. This tool -LogFileParser – is a fantastic tool for low level analysis of the NTFS.  It runs on Windows and we can point it to our mounted evidence in the SIFT workstation.

ntfsindx-logfileparser

The output is quite verbose and you can read about its capabilities and meanings in the README file. The parsing won’t take long and then we can analyze the output by just grepping for our file.

ntfsindx-transactionlogs

When searching for our missing and malicious DLL across the LogFileParser output files we found evidence of it on INDX attributes and transaction logs.
Based on the transaction operations recorded on the $LOGFILE we can see evidence that the CLB.DLL file was created on the WINDOWS folder at 22:43:58 and seconds after the file was renamed to CLB.DLLBAK and finally deleted. This time stamp matches the Prefetch and Shimcache artifacts.. We can see it was created on the Windows folder because of the parent MFT entry number 506 which matches the Windows MFT entry we were analyzing in the beginning. Quite fascinating facts that we observed by looking at the $INDX attribute and the $LOGFILE from NTFS metadata.

That’s it! In this article we reviewed NTFS metadata and different tools and techniques that we can leverage to view artifacts about evidence of deleted files . The INDX attribute which is used to store metadata about files inside directories and the $LOGFILE which keeps record of all operations that occurred in the NTFS volume. Both this artifacts can give us a great amount of information and detail about attacker activity and should be part of your forensic toolkit. We now have another piece of our investigation that we can put into our SuperTimeline.

If you want to read more about $INDX attributes there is a great article written in 2011 by Chad Tilbury here. In addition William Ballenthin & Jeff Hamm wrote a four part blog post series with great detail about the $INDX attributes.

 

References:

Windows Internals, Sixth Edition, Part 2 By: Mark E. Russinovich, David A. Solomon, and Alex Ionescu
File System Forensic Analysis By: Brian Carrier
https://github.com/jschicht/LogFileParser

Tagged , , , , ,
Nov 10 2015
1 Comment
Digital Forensics and Incident Response

Digital Forensics – NTFS Metadata Timeline Creation

[This is my second post on a series of articles that I would like to cover different tools and techniques to perform file system forensics of a Windows system. The first article was about acquiring a disk image in Expert Witness Format and then mount it using the SIFT workstation. The below one will be about processing the disk image and creating a timeline from the NTFS metadata. LR]

After evidence acquisition, you normally start your forensics analysis and investigation by doing a timeline analysis. This is a crucial step and very useful because it includes information on when files were modified, accessed, changed and created in a human readable format, known as MAC time evidence. This activity helps finding the particular time an event took place and in which order. Different techniques and tools exist to create timelines. In recent years an approach known as super timeline is very popular due to ability to bring together different sources of data. However, in this article we will focus on creating a timeline from a single source. The Master File Table file.

Before we move to the hand-on exercise let’s review some concepts behind the Master File Table. The Master File Table is a special system file that resides on the root of every NTFS partition. This file contains a wealth of forensic evidence. The file is named $MFT and is not accessible via user mode API’s but can been seen when you have raw access to the disk e.g, forensic image. This special file contains entries for every file and directory including itself. As written by Brian Carrier the MFT is the heart of NTFS. Each entry of the $MFT contains a series of attributes about a file, directory  and indicates where it resides on the physical disk and if is active or inactive. The active/inactive attribute is the flag that tracks deleted files. If a file gets deleted, its MFT record becomes inactive and is ready for reuse. The size of these entries are usually 1Kb. Because each record doesn’t fill 1Kb each entry contains an attribute stating if  contains resident data or not. Due to file system optimization, NTFS might store files directly on MFT records. A good example of this are Internet cookie files.  Microsoft reserves the first 16 MFT entries for special metadata files. These entries point to a special file that begins with $. The $Bitmap and $LogFile are examples of such files. A list of the first MFT entries are shown in the below picture. As well, it shows how to read the MFT record of a disk image on SIFT workstation using istat. The 0 at the end of the command is the record number you want to read for this partition that starts at offset 206848. The record 0 is the $MFT file itself.

ntfsmetadata1

Each record contains a set of attributes. Some of the most important attributes in a MFT entry are the $STANDART_INFORMATION, $FILENAME and $DATA. The first two are rather important because among other things they contain the file time stamps. Each MFT entry for a given file or directory will contain 8 time stamps. 4 in the $STANDARD_INFORMATION and another 4 in the $FILENAME. These time stamps are known as MACE.

  • M – Modified : When the contents of a file were last changed.
  • A – Accessed : When the contents of a file were accessed/read.
  • C – Created : When the file was created.
  • E – Entry Modified : When the MFT record associated with the file changed.

For our exercise, this small introduction will suffice. Please see the references for great books on NTFS.

Now that we have reviewed some initial concepts on MFT let’s move to our hands-on exercise.  For this exercise we will need the SIFT workstation with our evidence mounted – this was done on previous article. Then we need a Windows machine where we will access the mounted evidence on the SIFT workstation using a network drive. Finally, we will need the Mft2Csv tool from Joakim Schicht on the Windows machine to read, parse and produce the MFT timeline.

To start we share the mounted evidence on our SIFT workstation. In this case its /mnt/windows1 and was mounted on previous article. To perform this we edit the smb.conf and we add the lines as shown in the below figure. Then we restart the SMB deamon.

ntfssharesmb

Next, from your windows machine, which needs to be in the same network segment as your SIFT workstation. you can view the shares by using the net view command. Then using the net use command you can map a drive letter. timeline-mftwindows With this step on our Windows machine we will have access to our mounted evidence over the Z: drive. Next step is to run Mft2Csv tool. Mft2Csv is a powerful and granular tool developed by Joakim Schicht. For those who are not familiar with Joakim Schicht, he is a brilliant engineer who has enormously contributed to the Forensics community with many powerful tools.The tool has the ability to read $MFT from a variety of sources, including live system acquisition. It runs on Windows and has GUI and CLI capabilities and needs admin rights. The tool can be downloaded from here. As we speak the last version is v2.0.0.33. In this case, we will launch it from our Windows machine. The command line parameters define from where you are reading the $MFT file and the Time zone. The output by default will be saved in a CSV format but could be saved in a log2timeline or bodyfile. If you are familiar with the log2timeline format than you could use /OutputFormat:l2t. Below picture illustrate this step. The command executed is Mft2Csv.exe /MftFile:Z:\$MFT /TimeZone:0.00 /OutputFormat:l2t 
timeline-mftwindows3   When the command is finished you can open the timeline in Excel or copy it to SIFT workstation and use grep, awk and sed to review the entries. Another approach to create a timeline of the MFT metadata is using an old version of log2timeline which is still available on the SIFT workstation.  This old version has a MFT parser. You can use log2timeline directly on the mounted evidence. First we capture the Time Zone information from the mounted evidence using Registry Ripper – which we will cover on another post. Then we run log2timeline with -f MFT suffix to read and parse the $MFT file. The -z defines the time zone and the -m is a marker that will show prepended to the output of the filenames.
ntfsmetadata3

Or if you don’t have the evidence mounted you can export the $MFT using icat from TheSleuthKit.

ntfsmetadata2

Below picture illustrates the output of both tools using the l2t format.  In this case the cache.txt is an executable file part of a system that has been compromised with w32.morto worm.

ntfsmetdata4

That’s it! In this article we reviewed some introductory concepts about the Master File Table and we used Mft2Csv and Log2timeline to read, parse and create a timeline of it. The techniques and tools are not new. However, they are relevant and used in today’s digital forensic analysis. Next step, review more NTFS metadata.

References:

https://github.com/jschicht/Mft2Csv/wiki/Mft2Csv
Windows Internals, Sixth Edition, Part 2 By: Mark E. Russinovich, David A. Solomon, and Alex Ionescu
File System Forensic Analysis By: Brian Carrier
SANS 508 – Advanced Computer Forensics and Incident Response

Tagged , , , , , , ,