Despite the fact that my capstone thesis was complete over
three months ago, it’s been a struggle to make this post. That being said, hopefully this blog post doesn't appear too much like a paper, but it was pretty hard to avoid doing. The full paper, as of right now, is not publically
available, but hopefully will be some time in the near future. Please reach out with any questions relating
to the research, the subject, my process, or anything.
Although it was previously highlighted in my prior blog
posts, I feel that it is necessary to outline the importance of this research
again with a little more detail.
Note: Clicking the pictures will enlarge them. I formatted most easy enough to read, but some need enlarging.
Note: Clicking the pictures will enlarge them. I formatted most easy enough to read, but some need enlarging.
Why is this important
to us?
Technology is an ever-evolving
creature. In the world of digital
forensics, attempting to keep up to speed with the constant changes is an
absolute must. Doing so will make not
only for a better investigator, but a greater impact on the work that needs to
be done. With the recent boom in the terms
“cloud computing” and “virtualization,” digital forensic examiners find
themselves needing to immerse into a new era of the investigation age. According to the State of SMB IT Report,
written by Spiceworks, the trends in the adoption of virtualization and cloud
computing have been on a constant rise.
Moving back to the first half of 2010, 44% of small to midsized
businesses (SMB’s) were using virtualized products. Progressing forward, it is estimated that nearly
79% of SMB’s will be using virtualized platforms, (Sweeney).
Figure 1: State of SMB IT Report, November 2012,
Spiceworks (Source: Sweeney)
With such a dramatic increase in
the implementation of virtualized software, it is extremely pertinent that
digital forensic investigators begin to understand the logic of the systems and
what evidence can be found on them.
Two of the most common companies
that investigators will see in practice are VMware and its patented VMware
Workstation, as well as Citrix and its XenDesktop and VDI-in-a-Box
platforms. The low long-term cost of
these products allow for small and large companies alike to implement and
maintain these technologies. According
to Phil Hochmuth of Forbes, “Providers of VDI technology say their systems can
save between 30% and 50% of the current cost for IT administrators to manage,
patch, upgrade and support employees' PCs in a client/server environment. By
some estimates, this could cut the annual per-desktop support cost from around
$530 to $376--a savings which can add up for the larger the organization,” (Hochmuth). With an increased return on investment,
companies that deploy VDI’s will quickly advertise and transition partner
companies to similar technologies.
Unfortunately with all good comes
some bad and implementation of cloud computing is no different. As virtualization becomes more prevalent in
business practice, malicious code and malware will quickly take a turn towards
attacking cloud computing. “Attackers
will go where users go, so it should come as no surprise that mobile platforms
and cloud services will be high-risk targets for attacks and breaches in 2013.
The rapid rise in malware on Android in 2012 confirms this,” (Powledge). As attackers move towards cloud services and
online platforms, it becomes more necessary to not only secure them better, but
to be able to analyze and examine what happened after an attack and where
vulnerabilities lie. According to
Symantec, “the median cost of downtime for an SMB is $12,500 per day,” (Powledge). If a small sized business can lose this much
money while being down for such a short period of time, imagine the
ramifications if a large company like Amazon was hit for even two hours. It simply cannot be overstated enough;
digital forensic examiners must be prepared for this to happen as it is
inevitable.
So, what are virtual
desktop environments?
A Virtual Desktop Infrastructure is
simply cloud computing. Virtualization changes the information
technology work place. Desktops and
workstations can be set up, configured, and dispersed in merely minutes instead
of hours or days. Costs are reduced
after initial investment while security and machine integrity are
increased. The user experience becomes
easier than before, allowing for employees to remotely connect and work from
anywhere. Centralized storage provides
administrators and incident responders with a plethora of information at their
fingertips in the event of an emergency or breach.
An extremely popular company in the
VDI business is Citrix. Through their
products of XenDesktop, XenApp, and VDI-in-a-Box, Citrix has emerged as a
forerunner, and major companies all over are to use their products. According to Eric Savitz of Forbes in his
article entitled Citrix Shares Rally As Q4 Results Crush Street Estimates, “the
enterprise software company posted revenue of $740 million, up 19% from a year
ago, and ahead of the Street consensus at $705.7 million,” (Savitz). It’s rather clear by these numbers that the
investment into cloud computing is booming and that all sizes of companies are
transitioning. Citrix is among the top
in VDI providers, and as such, is the focal point of this research.
Now we understand why
this is important and what they are, so what am I doing?
VDI-in-a-Box is a unique toolset
that provides a network administrator with all of the tools necessary to deploy
a VDI to a business. Through the use of
only a few physical machines, VDI-in-a-Box can be successfully setup and
running in a very short time. Considering
small to midsized businesses are the target audience for this product and that
so many are making the transition already, VDI-in-a-Box version 5.2.0 is the
software that is being analyzed.
Figure 2: Citrix VDI-in-a-Box Version 5.2
How did I go about
doing this then?
Acquisition of a Citrix Virtual
Desktop Infrastructure required a great deal of trial and error. Due to being unable to use some of the
enterprise level tools because of licensing limitations, or the software not
supporting remote imaging of a server, Windows Secure Copy (WinSCP) over port
22 was used to capture a live image.
The XenServer itself, being the
physical hypervisor, was the primary target to acquire in hopes of finding all
the virtual machines stored on it.
Figure 3: XenCenter VM Pool
WinSCP is a program that gives a
user a graphical user interface to the secure copy protocol and will allow a
choice of what information to copy. By
entering a server address and administrative credentials, WinSCP will create a
secure connection to a remote location allowing information to be viewed or
duplicated.
Figure 4: WinSCP Login Prompt
Once logged into the server, WinSCP
will provide the user with a side-by-side view of both the remote machine and
the local machine. At this point, the
entirety of the server folder structure was copied over from the server in an
attempt to create the closest to a logical image as possible.
Figure 5: WinSCP File Transfer Session
It is important to note that
multiple errors occurred while copying files and folders from the server to the
examination machine. These errors would cause
the file being copied to fail and not copy successfully. The errors would typically read either “Can’t
create file…” or “…not a regular file.”
Figure 6: "Not a regular file" error
message
Figure 7: "Can't create file" error
message
Although there were errors on the
transfer, it did complete successfully and allowed for examination of the
drive. The most important item for
review were the virtual hard disks (VHD’s) that contained information relevant
to the golden images as well as virtual machines that had been created.
Figure 8: Virtual Hard Disk Storage Location
These virtual hard disks are stored
in a location determined during the initial setup, and in this project they
were located at /var/run/sr-mount. Each
VHD is listed by a universally unique identifier (UUID). These UUID’s can be related back to specific
machines in different ways. If access to
the Citrix vdiManager is available, each virtual machines general properties
will list its UUID. This will be
extremely helpful when working with personal desktops as the VHD’s will not be
deleted, therefore finding the UUID through vdiManager would be a simple task. When working with pooled desktops, however, the
VHD files are destroyed after the machine is shut down.
Pooled virtual hard disks being
destroyed was an obstacle that was easily overcome by the use of
snapshots, thus the virtual hard disks are now not destroyed. Much like VMware, Citrix allows
for the use of snapshots.
Figure 9: XenCenter Snapshot View
Each of these snapshots will create
two VHDs. The user can determine which
VHD’s are snapshots by a command via the XenServer command line. From the root directory of the drive, the
command xe vdi-list is-a-snapshot=true will prompt a list of all virtual hard
disks that are snapshots.
Figure 10: XenServer Snapshot List by Command Line
Instruction
The list of snapshots that is
returned will provide the UUID of each snapshot, a label of the golden image
that it was generated from, the storage repository UUID that is holding the
snapshot, and the virtual size of the snapshot.
To verify that this command was accurately creating a list all snapshots,
a new snapshot was taken and the command again was run again. This appended a new entry to the end of the
list, and also created two new VHD’s on the server.
Figure 11: New Virtual Hard Disk Creation
Figure 12: New Snapshot Verification
When using Citrix VM Protection and
Recovery, two types of snapshots can be taken with three different scheduling
options. Disk-only snapshots can be
taken that will “store the VM's disks (storage) and metadata. They are
crash-consistent and can be performed on all VM types, including Linux VMs,” (Citrix
EDocs). Disk and memory snapshots are
also available which will “save the VM's disks (storage), metadata, and its
current memory state (RAM),” (Citrix EDocs).
Once the decision is made to either perform disk-only or disk and memory
snapshots, the administrator must choose whether to do hourly, daily, or weekly
snapshots. In any event, the snapshot
retention policy is limited to a maximum of 10 scheduled snapshots or until
archived, automatically deleting the oldest one when this number is exceeded.
Figure 13: Citrix VM Protection and Recovery
A daily or weekly archive schedule
can then be configured, independent from the scheduled snapshots, which will
archive all scheduled snapshots to a remote location for storage. Alternatively, all snapshots can be
automatically archived if the administrator wishes. When examining the archived virtual hard
disks, each archive folder is given a specific name. This naming convention will always be the VM
name followed by the first sixteen characters of the UUID. For example, if the VM name is Win7 and the
UUID is 6a88edee-5b42-453a-a60d-4605c689f338, the archive folder will be Win7_6a88edee-5b42-45. “This folder contains archived VM files, in
the YYYYMMDD-HHMM.xva form,” (Citrix RSS).
Depending on how this is done, the
investigator will need to link a user to a specific snapshot to alleviate the
need to sift through what could be hundreds of snapshots. One method would be to examine each virtual
hard disk file individually. Although
this will take quite some time, it will still provide accurate results. One benefit to this method is that no
information will be overlooked in the event that the suspect was using another
user account or username.
On the domain controller with
XenCenter installed, running the command “xe vm-list –s <server> -u
<user> -pw <password>” provides a list of all virtual machines currently
being run. The virtual machine
vdiManager_Xen is the interface that the administrator interacts with to
initially configure the server, the machine that keeps activity logs, and the
web portal that users must access in order to spawn a virtual desktop.
Figure 14: XenCenter Command Line VM List
Still being SSH connected into the
XenServer, under the directory /var/run/nonpersistent/xenops/VM, the
investigator can locate and view the configuration files for the vdiManager. This will point to the virtual hard disk that
retains these records.
Figure 15: vdiManager Configuration File
This configuration file will first
list the same UUID found when the XenCenter command was used. In the middle of the log is the VDI UUID,
with the storage repository UUID being listed, followed by \/, and then the
vdiManager’s UUID. Within the
/var/run/nonpersistent/xenops/VM directory are two files, vdb.xvda and
vdb.xvdb. The first of these (.xvda)
will correspond this information again, linking the two UUID’s.
Figure 16: vdiManager Virtual Hard Disk Verification
Once this is achieved, examination
of the virtual hard disks can begin. The
investigator should now be able to extract and examine the vdiManager machine
as well as all of the snapshots that were taken. Unfortunately, when examining a VHD in EnCase,
it is not recognized as a normal operating drive and therefore does not have
any form of folder structure. Rather, it
is all listed as unallocated and must be manually parsed.
What can I find from this?
There is a great deal of
information that is recoverable from the XenServer. Some of these files are relevant to the
server itself and will depict information about the XenServer, while others
reflect the vdiManager. A plethora of
virtual machine information is also recoverable as long as a snapshot is present.
The first file of interest from the
XenServer is xensource-inventory, located in /etc. This file contains information related to the
actual server itself and would be useful in an investigation. The installation date of the server is
recoverable, but more importantly, the primary disk and backup partition
locations are located within this file as well.
Figure 17: XenServer Disk and Installation Details
The next file of interest is
located again in /etc and is the timezone file.
Considering time zones are extremely important to any forensic
investigation, examining this artifact will be very helpful as it lists in
plaintext the time zone that the server is set for.
Figure 18: XenServer Timezone Details
The last file pertaining to
information from the server is the bash history. Bash is a unix command-line interpreter, or
shell, that will allow for a user to input commands and information into a
system. It is comparable to the Windows
command line. The bash history can be
located at /root and is named .bash_history.
Figure 19: Bash History
When
examining the vdiManager virtual hard disk, the investigator is capable of
pulling all users that were associated to the drive, what templates the user
was assigned to, the groups the user was a part of, the user’s ID in Citrix,
and more.
Figure 20: vdiManager User
Entries
This
VHD also contained logs in reference to the spawning of virtual machines,
including the time and date that it was created. The log will list the user that connected,
the machine IP address that the connection originated from, the template the
user logged in with, and the MAC address that the created virtual machine was
given.
Figure 21: vdiManager User Connection
Further
investigation of these logs provides more information on user interaction with
virtual machines, specifying when a user logged both in and out of a desktop,
what IP address it was given, what template it was provisioned from, and what
vdiServer it accessed.
Figure 22: vdiManager Connection Log
This
log file is also able to be obtained via the vdiManager web portal. Within the admin menu is a View Audit Log
function that provides an excel spreadsheet containing user activity.
Virtual Machine
Files
The virtual hard disk files are the
focal point of investigation when attempting to uncover information from
non-persistent machines. The
non-persistent VHD’s will be destroyed when the reset policy is set, either at
logout or on a schedule. Due to this,
snapshots of these hard disks are necessary to uncover potential evidence.
Each virtual hard disk contains a
header that points to its parent drive.
Although EnCase was listing these pointers at the beginning and making
it relatively easy to navigate between, server-side verification of this was
completed. Running the command xe
vdi-list params will provide a list of metadata about each virtual hard
disk.
Figure 23: VDI-List Parameters
Command
This command can populate a list
that will tell the user the UUID of the virtual disk, whether it is a snapshot
or not, the time of the snapshot if one was taken, the number of snapshots
taken, and the virtual hard disks parent.
Figure 24: Parent UUID Verification
By proving that the virtual disk
with the UUID of b54cb669-f4e6-44a9-8b97-2038aa7ea7a2 has the parent with the
UUID of 8edd0321-2f59-4fcb-9a1e-634fd5d68d37, the results being displayed by
EnCase were verified and thus trustworthy.
Figure 25: EnCase Parent UUID
Correlation to XenServer Verification
As previously mentioned, there are
two virtual hard disk files associated with each snapshot. The first of these disks is the identity
disk, which will typically range up to sixteen megabytes and simply provides
each VM with a unique identity. The
second disk, referred to as the difference disk, is “used to store any writes
made to the VM. The disk is thin
provisioned (if supported by the storage) and will increase to the maximum size
of the base VM if required,” (Feller).
Figure 26: Disk Creation (Source: Feller)
Review of the identity disk yields just
a pointer UUID to the difference disk associated with it. The difference disk contains content from the
virtual machine as well as a pointer to the golden image it was created from. Although this is all shown as unallocated
space and needs manual parsing, some very helpful artifacts remain and can be
found.
A virtual machine was created that
had limited activity on it but with enough different items to look for. The following table depicts the activity on
the machine.
Time
|
Activity
|
5:09pm
|
Visited Msn.com using Internet Explorer (IE)
|
5:09pm
|
Visited Amazon.com using IE
|
5:10pm
|
Visited Google.com using IE
|
5:10pm
|
Ran MSPaint.exe
|
5:11pm
|
Saved a paint file – “findthis.png” to the desktop
|
5:12pm
|
Visited www.champlain.edu
|
6:10pm
|
Google search for “april fools jokes”
|
6:11pm
|
Visited dig4n6.blogspot.com
|
6:11pm
|
Saved file from website – xzibit_meme.png
|
6:11pm
|
Changed the file type of xzibit_meme.png to xzibit_meme.bmp
|
6:13pm
|
Created a file named Deleted Item and deleted it, removed it from the
recycling bin
|
6:18pm
|
Changed file type of xzibit_meme.bmp to xzibit_meme.jpeg
|
Due to the disk being all
unallocated space, the quickest way to determine if any information was still
recoverable on the drive was by doing simple keyword searches. Keyword
searches were executed for the following phrases:
“msn.com”, “amazon.com”, “google.com”,
“paint”, “paint.exe”, “findthis.png”, “findthis”, “efleisher”, “www.champlain.edu”,
“april fools jokes”, “dig4n6.blogspot.com”, “deleted item”, “xzibit_meme”.
One of the most intriguing and
important artifacts that was recovered were multiple entries from the Master
file table (MFT). These MFT entries
contain accurate time stamps and accurate information in regards to the files
that were created on the virtual machine.
Figure 27: MFT Entry for "findthis.png"
with accurate creation date
A search was done for the phrase
“FILE0” as this is a typical indicator of a MFT record. There were a total of 1457 hits in response
to this keyword. This is at least
indicative of 1457 MFT entries being present on this virtual disk. There were MFT entries present for each file
and folder that was created on the machine which will at least provide
information about some pertinent items on the machine.
Figure 28: Keyword results for MFT
Entry Header "FILE0"
The keyword search for the phrase
“april fools jokes” successfully pulled the Google search in plaintext from the
unallocated space as well.
Figure 29: Retrieved Google Search
A keyword search was done with a
username of “efleisher” to search for any data that may be relevant to what the
user was doing on the computer. Nearly
two thousand hits responded containing information primarily relating to
browsing history. The data was in a
plaintext readable format, with results being displayed in similar format to “efleisher@amazon.com”.
Figure 30: User Activity via
Username Search
This data provides the investigator
with insight into the users browsing history.
By manually parsing these entries, the investigator is able to ascertain
when the username was accessing certain websites. These time stamps can be found prior to the
website address. It will be stored in little
endian hex format and will be eight
bytes long.
Figure 31: Internet History
Timestamp Verification
Through keyword searches of
websites that were visited, an entry was found that revealed each website that
was browsed to. The order that these
websites were presented was in the reverse order that they were visited, with
the last website browsed to being the first website listed in the entry. It is potentially indicative of a file that
contained all web browsing history, similar to an index.dat file. Unlike most internet history entries,
however, there were no records of potential timestamp information.
Figure 32: Complete Browsing History
With the previous findings
presenting a plethora of internet history information, the next step was to try
an automated tool. Internet history
tools will attempt to perform an automated acquisition of the internet history
on a hard disk and present the findings to an investigator in an easy to read
format. Internet Evidence Finder v5 was
able to successfully parse and extract internet history, including cookies,
with accurate time stamps and user correlation.
Figure 33: Successful Internet
Evidence Finder Parsing of VHD
Further research into web browsing artifacts
provided HTML code for specific websites that were visited. This code provides a plaintext view of some
of the information that was on the webpage when it was visited by the user.
Figure 34: Plaintext HTML Code of
Webpage
An important and common artifact
that is looked at in a forensic investigation is deleted items. A deleted item can mean many things to an
investigator, such as an attempt to hide information, an attempt at removing
evidence, or perhaps merely an attempt at removing clutter. Both data recovery and forensic
investigations frequently deal with deleted items, thus making the ability to
recover them a desirable skill.
During this research, a text file
was created named deleted item. This item
was created on April 1st, 2013 at 6:13:07pm and immediately deleted afterwards
at 6:13:20pm. The keyword search for
this particular artifact provided multiple hits, including accurate timestamps
for the file creation time.
Figure 35: Accurate Timestamp of
Deleted Item Creation Time
When a file is deleted on a Windows
machine, it creates two files in the recycling bin that are associated with
it. These files are known as “$I” and
“$R” files. The $I file will contain the
original name and path of the file, along with its deletion date. The $R file will retain the actual data of
the file. When examining the evidence
near the location to the original file, the investigator will find references
to $I and $R files associated with the deleted item.
Figure 36: Discovery of Deleted Item
as well as $I and $R files
In order to recover the deletion
time of the file, it is necessary to obtain the creation time of the $I and $R
files. This timestamp can be recovered
by moving twenty bytes prior to the $ symbol and then obtaining the previous
eight bytes prior to that. If done
correctly, these eight bytes can be decoded in little-endian format to produce
the created time of both $I and $R files, therefore the deletion time of the
original file.
Figure 37: Accurate Timestamp of
Deleted Item Deletion Time
Perhaps the most interesting piece
of evidence left behind on the virtual hard disk was the resemblance to a
timeline of system activity. An entry
within the virtual hard disk depicted a chain of events that occurred on the
hard drive, including website visits, usernames, applications opening, files
being created, and a list of some of the values that would have been updated on
a system during the activity that occurred.
Figure 38: Entry reflecting Timeline
of Events
What would you
recommend my company to do?
Analysis of a Citrix Virtual
Desktop Environment requires a fair amount of setup behind the scenes in order
for it to be accomplished successfully.
Without the presence of snapshots, an investigator would be hard pressed
in finding any information that occurred on a pooled virtual machine. Although network monitoring tools could be
put in place that would monitor everything that occurs, these options are often
very costly and can still miss important artifacts. For a SMB or large business that is
implementing a VDI, it would be recommended that snapshots be scheduled of at
least the hard disk. Depending on
storage available, if both disk and memory snapshots are capable, more information
would be recoverable. This type of
snapshot would require high quantities of storage.
Through implementation of snapshots
and with appropriate administrative access, an investigator is able to manually
parse through virtual hard disk files that the Citrix VDI-in-a-Box
maintains. User information such as web
history, created files and folders, opened programs, Google searches, passwords,
deleted items, and master file table entries are all recoverable. This information may not be everything that
is needed to crack the case, but it is surely a huge stepping stone into
uncovering more information and perhaps even the “smoking gun.”
When looking at the vdiManager
virtual hard disk, the investigator is able to obtain information about when a
user is logging into a virtual desktop.
Using this and information from the domain controller, further research
needs to be done to be able to successfully link a specific user to a given
UUID. Doing this will allow for the
investigator to not need to sift through hundreds of hard drives for information,
and instead have immediate access to
which snapshots are linked to which users.
Doing so will provide for a streamlined investigative process in both
acquisition and analysis. Although this
is just the tip of the iceberg, there is without a doubt much more information
that can be uncovered. While access to
these servers and machines are no longer available, I hope to someday expand
this research, or at least see another professional take what has already been
done and expand upon it to allow for even more thorough analysis of Citrix
VDI’s in the future.
References
"Citrix XenServer
® 5.6 Feature Pack 1 Administrator's Guide." Citrix RSS. Citrix, 24
Mar. 2011.
Web.
“Create the First
Windows Image." Citrix EDocs. Citrix, 12 Feb. 2013. Web.
"Create a VM
Protection Policy." Citrix EDocs. Citrix, 26 Nov. 2012. Web.
Feller, Daniel.
"Machine Creation Services Primer – Part 1." Citrix Blogs.
Citrix, 28 June 2011.
Web.
Hochmuth, Phil.
"The Benefits Of Virtual Desktops." Forbes. Forbes Magazine, 12
Jan. 2010.
Web.
"Internet Evidence
Finder." Magnet Forensics. N.p., n.d. Web.
"Master Boot
Record." Master Boot Record. Microsoft TechNet, n.d. Web.
"NTFS Master File
Table (MFT)." NTFS Master File Table (MFT). N.p., n.d. Web
Powledge, Tom.
"Top 7 SMB Information Protection Predictions for 2013." Endpoint,
Cloud,
Mobile & Virtual
Security Solutions. Symantec, 4 Dec. 2012. Web.
Savitz, Eric.
"Citrix Shares Rally As Q4 Results Crush Street Estimates." Forbes.
Forbes Magazine,
30 Jan. 2013. Web.
Sweeney, Brandon.
"State of SMB IT Report – Equip Yourself with These Stats to Help Make
Vital IT
Decisions." VMware SMB Blog. VMware, 20 Feb. 2013. Web.
