Why Isn't There a Chapter on .NET Debuggers?
You might be wondering why there isn't a corresponding chapter in this book on how Microsoft .NET debuggers work. Originally, I had intended to write that chapter, but as I was researching the .NET Debugging API, I realized that unlike Win32 debuggers, which are nearly undocumented, the .NET run-time team did a tremendous job documenting the .NET debugging interface. Additionally, the debugger sample provided shows how to do everything a .NET debugger is supposed to do. The sample is about 98 percent of the source code console debugger, CORDBG. (The only part that's missing is the native code disassembler commands.) I spent a couple of weeks working on a .NET debugger, and I quickly realized that I was going to be doing nothing more than rehashing the excellent .NET documentation and that I wouldn't be showing anything that wasn't already shown in the CORDBG sample. The Microsoft Word files, Debug.doc and DebugRef.doc, that describe the .NET Debugging API are already installed on your computer as part of the Visual Studio .NET installation and are in the <Visual Studio .NET Installation Directory>\SDK\v1.1\Tool Developers Guide\Docs directory.
Common Debugging Question: How can I protect my Win32 program from being debugged?
One of the most common requests I get from developers working on vertical market applications with proprietary algorithms is how can they protect those algorithms and keep their competitors from taking a peek at them with a debugger. Although you can call IsDebuggerPresent, which tells you whether a user-mode debugger is running, if someone has half a brain, the first thing she'll do when reverse engineering is patch IsDebuggerPresent to return 0 so that it appears as though no debugger is running.
Although there's no perfect way to protect against a very determined hacker who has physical access to your binaries, you can at least make life a little more different for him at run time. Interestingly enough, the check that IsDebuggerPresent does to see whether a debugger is running on the process has been the same check in all Microsoft operating systems up to the time of this writing. There's no guarantee that it won't change, but the odds are good that it will stay the same for the future.
The next bit of code is a function you can add to your code that does the same thing as IsDebuggerPresent. Of course, just adding that function won't make it impossible to debug your application. To make debugging tougher, you might want to look at interspersing innocuous instructions between the main instructions so that hackers can't simply search for the byte pattern of the IsDebuggerPresent code. A whole book can be written about anti-hacking techniques. However, if you can pass the "two-hour test," which is that it should take longer than two hours for an average developer to try to hack your application, your application is probably safe from all but the most determined and talented hackers.
BOOL AntiHackIsDebuggerPresent ( void ) { BOOL bRet = TRUE ; __asm { // Get the Thread Information block (TIB). MOV EAX , FS:[00000018H] // 0x30 bytes into the TIB is a pointer field that // points to a structure related to debugging. MOV EAX , DWORD PTR [EAX+030H] // The second WORD in that debugging structure indicates // the process is being debugged. MOVZX EAX , BYTE PTR [EAX+002H] // Return the result. MOV bRet , EAX } return ( bRet ) ; }