3.4.3. Gotcha: Use breakpoints moderately in a VM-exit handler |
This document describes how to use HyperPlatform to develop your own hypervisor-based tools and general knowledge on hypervisor development. This document is available in multiple formats:
•HTML (http://tandasat.github.io/HyperPlatform/userdocument/)
•PDF (http://tandasat.github.io/HyperPlatform/userdocument/UserDocument.pdf)
•ODT (/Documents/UserDocument.odt)
For more high level information on HyperPlatform, see the project page.
•https://github.com/tandasat/HyperPlatform
This chapter describes steps to create a new Visual Studio project derived from HyperPlatform and briefly explains where to modify to implement your own logic on the top of HyperPlatform.
You need the following packages to compile HyperPlatform:
•Visual Studio Community 2015 Update 3
◦https://beta.visualstudio.com/downloads/
•Windows Software Development Kit (SDK) for Windows 10 (10.0.10586.0 or later)
◦https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
•Windows Driver Kit (WDK) 10 (10.0.10586.0 or later)
◦https://developer.microsoft.com/en-us/windows/hardware/windows-driver-kit
1. Create a new project for 'Kernel Mode Driver, Empty (KMDF)'. In this document, a project is named 'EopMon'.
2. Place an entire HyperPlatform folder under the project folder. It can be done by copying files or using HyperPlatform as a git submodule. After placing files, a result of tree command should look like this:
> tree .
...
C:\USERS\USER\DESKTOP\EOPMON
├───EopMon
└───HyperPlatform
├───Documents
└───HyperPlatform
└───Arch
├───x64
└───x86
3. Add all cpp and h files under the copied HyperPlatform folder to the project.
4. Add x86.asm and x64.asm files under the HyperPlatform\Arch folder to the project.
5. Open properties of the x64.asm and switch 'Configuration' to 'All Configuration' and 'Platform' to 'Win32', then set 'Yes' to 'Excluded From Build'.
6. Similarly, open properties of the x86.asm and switch 'Configuration' to 'All Configuration' and 'Platform' to 'x64', then set 'Yes' to 'Excluded From Build'.
7. For x86.asm, set 'Yes (/safeseh)' to 'Use Safe Exception Handlers'.
8. Open properties of the project, switch 'Configuration' to 'All Configuration' and 'Platform' to 'All Platforms', and then set 'Windows 7' to 'Target OS Version', and 'Desktop' to 'Target Platform'.
9. Build the project for 'x86' or 'x64' (HyperPlatform does not support ARM architecture).
The rests are left to your ideas, but followings is a list of locations where you may want to take a look at and modify for your own purposes:
•DriverEntry and an unload handler => driver.cpp
•Configurations of when VM-exit occurs => VmpSetupVmcs() in vm.cpp
•An address of a sysenter handler => VmpSetupVmcs() in vm.cpp
•VM-exit handlers => VmmpHandleVmExit() in vmm.cpp
•Any assembler code => x64.asm, x86.asm and asm.h
Also, there are some side projects useful to developers.
•MemoryMon
◦A hypervisor-based tool detecting execution of kernel memory where is not backed by any image files using extended page table (EPT). It is implemented on the top of HyperPlatform.
•GuardMon
◦A hypervisor-based tool monitoring some of PatchGuard activities. It is implemented on the top of HyperPlatform.
•hyperplatform_log_parser
◦A user-mode program parsing logs created by HyperPlatform. Most useful with MemoryMon currently.
•ping_vmm
◦A user-mode program knocking at HyperPlatform's “backdoor”.
Enjoy the ring -1 programming!
This chapter explains basic technical know-how of developing and debugging hypervisors.
VMware Workstation supports nested hardware virtualization and allows your hypervisor to run inside a VM. In order for running a hypervisor in a VM, you need to change configurations of the VM as followings.
1. Check 'Virtualize Intel VT-x/EPT or AMD-V/RVI'
2. Open a corresponding VMX file and add the following lines.
hypervisor.cpuid.v0 = "FALSE"
mce.enable = "TRUE"
Although VMware works perfect for development majority of times, it can be difficult to chase some bugs through only a kernel debugger. For example, VMware provides no capabilities to trace why a VMX instruction failed or why a guest suddenly received a triple fault VM-exit. By using Bochs, you are able to trace exactly why it happened with Bochs' emulation source code.
Following sections describe steps to run a hypervisor in Bochs with the Visual Studio Debugger.
This section explains how to compile and run Bochs with Visual Studio.
1. Download and install Cygwin. At least, gcc and zip commands are required.
https://www.cygwin.com/
2. Download source code from SVN and extract it.
3. Open .conf.win32-vcpp and delete below three lines (as they failed to link when I tested).
--enable-usb \
--enable-usb-ohci \
--enable-usb-xhci \
4. Open the Cygwin console, navigate to the extracted directory, and run the following commands.
$ sh .conf.win32-vcpp
$ make win32_snap
5. Extract created bochs-20160107-msvc-src.zip, and open bochs-20160107\vs2013\bochs.sln
6. In the Solution Explorer, navigate to the win32res.rc file under the "bochs" project.
7. Right click > View Code, and comment out the below line as it causes a link error.
// Manifest for both 32-bit and 64-bit Windows
1 24 build/win32/bochs.manifest
8. Build the bochs project. You will see some errors in other projects but can ignore them. It should create boches.exe under bochs-20160107\obj-release or bochs-20160107\obj-debug according with build configuration.
This section explains how to install a VM on Bochs.
1. Download a pre-compiled Bochs (one with an exe extension).
2. Through the start menu, run Programs\Bochs 2.6.8\Disk Image Creation Tool (bximage.exe)
3. Create an enough size of a hard disk image. 2048MB should suffice for installing most of modern Windows versions.
1. Create new floppy or hard disk image
2. Convert hard disk image to other format (mode)
3. Resize hard disk image
4. Commit 'undoable' redolog to base image
5. Disk image info
0. Quit
Please choose one [0] 1<Enter>
Create image
Do you want to create a floppy disk image or a hard disk image?
Please type hd or fd. [hd] <Enter>
What kind of image should I create?
Please type flat, sparse, growing, vpc or vmware4. [flat] <Enter>
Enter the hard disk size in megabytes, between 10 and 8257535
[10] 2048<Enter>
What should be the name of the image?
[c.img] <Enter>
Creating hard disk image 'c.img' with CHS=4161/16/63
The following line should appear in your bochsrc:
ata0-master: type=disk, path="c.img", mode=flat
(The line is stored in your windows clipboard, use CTRL-V to paste)
This should create an image file under the below folder.
%USERPROFILE%\AppData\Local\VirtualStore\Program Files (x86)\Bochs-2.6.8
4. In the same folder, create a bxrc (eg. bochsrc.bxrc) with the below contents.
# configuration file generated by Bochs
plugin_ctrl: unmapped=1, biosdev=1, speaker=1, extfpuirq=1, parallel=1, serial=1, gameport=1, ne2k=1, e1000=1
config_interface: win32config
display_library: win32
memory: host=1024, guest=1024
romimage: file="C:\Program Files (x86)\Bochs-2.6.8/BIOS-bochs-latest"
vgaromimage: file="C:\Program Files (x86)\Bochs-2.6.8/VGABIOS-lgpl-latest"
boot: cdrom, disk
floppy_bootsig_check: disabled=0
# no floppya
# no floppyb
ata0: enabled=1, ioaddr1=0x1f0, ioaddr2=0x3f0, irq=14
ata0-master: type=disk, path="c.img", mode=flat, cylinders=20805, heads=16, spt=63, model="Generic 1234", biosdetect=auto, translation=auto
ata0-slave: type=cdrom, path="Win7x86.iso", status=inserted, model="Generic 1234", biosdetect=auto
ata1: enabled=1, ioaddr1=0x170, ioaddr2=0x370, irq=15
ata1-master: type=cdrom, path="d.iso", status=inserted, model="Generic 1234", biosdetect=auto
ata1-slave: type=none
ata2: enabled=0
ata3: enabled=0
pci: enabled=1, chipset=i440fx
vga: extension=vbe, update_freq=5, realtime=1
cpu: count=1, ips=1200000000, model=corei7_sandy_bridge_2600k, reset_on_triple_fault=1, cpuid_limit_winnt=0, ignore_bad_msrs=1, mwait_is_nop=0
print_timestamps: enabled=0
debugger_log: bochs_debugger.log
magic_break: enabled=0
port_e9_hack: enabled=0
private_colormap: enabled=0
clock: sync=none, time0=local, rtc_sync=0
# no cmosimage
# no loader
log: bochs.log
logprefix: %t%e%d
debug: action=ignore
info: action=report
error: action=report
panic: action=ask
keyboard: type=mf, serial_delay=250, paste_delay=100000, user_shortcut=none
mouse: type=ps2, enabled=0, toggle=ctrl+alt
sound: waveoutdrv=win, waveout=none, waveindrv=win, wavein=none, midioutdrv=win, midiout=none
speaker: enabled=1, mode=sound
parport1: enabled=1, file=none
parport2: enabled=0
com1: enabled=1, mode=null
com2: enabled=0
com3: enabled=0
com4: enabled=0
ne2k: enabled=0
e1000: enabled=0
Here is a list of some notable points of this configuration:
•References to a bios file under C:\Program Files (x86)\Bochs-2.6.8 (You may need to change the path)
•A VM can see contents of Win7x86.iso as a CD in the same folder.
•A Processor is 1.2 GHz, which is acceptably fast to install and run Windows 7
•Logs are stored in to bochs.log in the same folder
•No network connectivity
•Uni-processor system without the Bochs debugger (you can change those if needed, although you have to re-compile Bochs accordingly. See Bochs' document for details).
5. In the same folder, create a shortcut file to the compiled bochs.exe, and change 'Start in' to the current folder.