Hard disk hacking - Intro

(http://spritesmods.com/?art=hddhack&page=1)

Intro

Apart from this article, I also gave a talk at OHM2013 about this subject. As soonas the video recordings made at that time are available, I'll link there.

Hard disks: if you read this, it's pretty much certain you use one or more of the things. They're pretty simple: they basically presenta bunch of 512-byte sectors, numbered by an increasing address, also known as the LBA or Logical Block Address. The PC the HD is connected to can read or write data to and from these sectors. Usually, a file system is used that abstracts all thosesectors to files and folders.

If you look at an HD from that naive standpoint, you would thinkthe hardware should be pretty simple: all you need is something that connects to a SATA-port which can then position the read/write-headand read or write data from or to the platters. But maybe more is involved: don't hard disks also handle bad block management and SMART attributes, and don't they usually have some cache they mustsomehow manage?

All that implies there's some intelligence in an hard disk, and intelligence usually implies hackability. I'm always interestedin hackability, so I decided I wanted to look into how hard diskswork on the non-mechanical level. Research like this has been donebefore for various bits of hardware: from PCI extension cards toembedded controllers in laptops to even Apple keyboards. Usuallythe research has been done in order to prove the hackability of these devices can lead to compromised software, so I decided to take the same approach: for this hack, I wanted to make a hard disk that could bypasssoftware security.
 

Parts on the PCB

To figure out if hard disks are hackable, I first had to get to knowthem better. Luckily, like most of you, I had a whole stack of old and/or broken hard disks to look at:

Of course, we all know how the mechanical parts of a hard disk are supposed to work, and I wasn't really interested in those parts. My interest was in the little PCB that's on the back of most HDs andwhere the SATA and power connectors were located. This is what such aPCB looks like:

You can see that there are about four chips on the PCB. This is what I found out about them:

This is a bit of DRAM. It's a jellybean part, with easy-to-find datasheets. The capacity of these chips range from 8MB to 64MB, and these sizes correspond to the cache size the hard disk is supposedto have.

This is the spindle motor controller. It's not a standard part,so datasheets are hard to find, but some of the controllers seem to have brothers and sisters that are a bit easier to find. ST Smoothcontrollers seem to be the most used ones; apart from driving the spindle motor, they also do power regulation and have some A/D channels.

This is a bit of serial flash. It's also a jellybean part, withsizes ranging from 64KB to 256KB. It seems to be used to store the program the hard disk controller boots up from. Some hard disks don't have this chip but have the flash internal to the HD controller chipinstead.

These little devices aren't chips, but piezo-electric shock sensors.They can be used to move the heads somewhere safe when the HD experiences a mechanical shock, but more likely just set a bit somewhere to indicate your warranty is void because you dropped your HD.

And this is the bit where all the fun stuff happens: the hard diskcontroller. They are made by Marvell, ST and some other LSI companies.Some hard disk companies also make their own controllers: I've seen both Samsung and Western Digital do this. With almost everything else being a jellybean part, this is the device I was interested in.

Unfortunately, these parts are somewhat underdocumented. Saying the companies making the controllers aren't too kind on revealing information about them is an understatement: they don't evenmention the existence of the part numbers on their sites! Unfortunately,the rest of the Internet isn't too helpful either: looking for datasheetsonly reveal datasheet-sites not having the actual PDFs and obscure Chinese sellers claiming to have the ICs.

So, no datasheets of the most important IC, that means we're stranded,right?

Hooking up JTAG

Luckily, there are other ways to find out information about these ICsthan datasheets. One of my web searches actually resulted in something useful.

what I found was a thread from a guy called Dejan on the HDDGuru forums.Dejan had managed to corrupt the internal flash of his hard diskin some way and wanted to know if there's a way to either boot the controller from external flash, or a method to re-write the flash.For five days, he doesn't get a reponse, but the guy is inventive: thenext thing he posts is the message that he has found the pinout of theJTAG-port. That's a major find: the JTAG-port can be used to controla controller like a puppet. You can stop it, restart it, modify memory,set breakpoints etc with it. Dejan then figures out how to dump theboot ROM of the controller, figures out there's a serial port on oneof the hard disk headers and manages to restore his flash ROM. Hethen dumps a few more bits and pointers about the flash update processbefore finally disappearing into the mists of the Internet again.

All this was pretty useful information: it told me at least theWestern Digital controllers seem to have an ARM-core that's accessibleover the JTAG-port. It also told me these hard disks usually have a serial port, which is usually unused but could be useful for debuggingmy hack. With this, I should have enough information to start hacking.

So, this is my setup:

The red thing is an FT2232H-board, a cheap board you can get for about EUR30 which can do JTAG and serial, as well as SPI-communications. It'sconnected to the JTAG-interface of the hard disk, as well as the headerwhere the hard disk has its serial port. The HD is directly connectedto the SATA-port on my computers mainboard, as well as to an external ATX power supply. I use OpenOCD as the software to drive the JTAG-port.

Now, the question is: would it actually work? Dejan did this with a 2.5" 250G HD with an 88i6745-controller, and he detected an arm9-core.I grabbed a 3.5" 2TB HD with an 88i9146-controller instead, which hada different form factor and is a bit newer. Luckily, OpenOCD has a wayto detect what's on the JTAG chain by itself. This is what it found:

This confused me for a bit... I expected a single tap, for the single ARMcore that's inthere... but instead, I found three taps... doesthat mean this chip has three ARM-cores?

After some research, I found out that yes, the chip indeed seems to havethree cores. There's two Feroceons, which are quite powerful arm9-like cores, and a Cortex-M3 core, which is a bit smaller, more microcontroller-ish core. Some more playing around (and later research)indicating the controllers all had different functions:

• Feroceon 1 handles the physical reading and writing from/to the hard disk platters
• Feroceon 2 handles the SATA-interface
• Feroceon 2 also handles the cache and LBA to CHS translation
• The Cortex-M3 handles... nothing? I could stop it and still haveall hard disk functions.

Now, what core to start hacking at? My target was to try and compromisethe security of a system by using hard disk firmware mods. The easiestand probably hardest-to-detect way to do this was to modify data on thefly. That way, the data on the disk wouldn't need to be changed and thefirmware could just make itself invisible. To do this, I would needto find a suitable core to that kind of interception: I needed tohave a core that would have access to the data when it's in-transit from the disk to the SATA-cable, and also could be rigged to modifythe data while it was in between those two points.

Now, how would that data get from the HD platters to the SATA interface?Here's where I used a bit of intuition. My reasoning went something like this:
If the processors would use a standard memory copy, with them runningat 150MHz, they would only be able to reach 150*23/2=2.4Gbps, and inpractice most likely much less. The hard disk is specced at 6Gbps, sothere's probably some hardware acceleration involved. The most likelyhardware acceleration would be to use DMA. That would mean the datais copied directly from the head reading logic to memory, withoutactive involvment of the processor. The same goes for the SATA-port:the processor would have to only indicate where the data is, and theDMA logic would take care of reading the data directly from memory.

If this was the case, where would the memory that the DMA-engine wouldbe pointed at, be located? The cache of the hard disk would be a goodlocation: data read from the disk would need to be in cache anyway, soit would make sense to copy it there immediately when reading from the disk. I figured out earlier that Feroceon 2 was responsible for thecache handling; that'd make it a prime target for a hacking attempt.

So, I deduced that the data was read and written through DMA, withoutany CPU action involved. Now the question was: Even if the CPUs won't touch the data in normal operation, can theyactually access it? To answer this question, I first used the JTAG-connection and a bit of disassembly to figure out the memory mapof the 2nd Feroceon:

As you can see, the memory map is a bit fragmented. There are smallbits of RAM sprinkled around, there's some IO and IRQ space, and abit of internal boot ROM. There also is a big, 64MB segment of whatI suspected was the DRAM-chip with the cache in it. Let's find outif this is actually true. First, I mounted the disk on my machine andwrote 'Hello world!' to a file on it. Now, could I find the string in the 64MB mem region?

Yep, there it is. Seems the cache is accessible by the Feroceons andmapped to the 64MB DRAM region.
 

Injecting code

Ofcourse, if I wanted to change something in the cache, I couldn'tscan the complete 64MB of RAM every time: I needed to know how thecache works. For that, I would need to dump, disassemble and understandthe hard disk firmware at least enough to make sense of the cachingfunctions.

Disassembling this firmware is not a trivial task. First of all, thecode mixes ARM and thumb-style instructions, which is irritating ifyou don't have a disassembler which can automatically switch betweenthe two. Furthermore, something that usually makes disassemblingsoftware a lot easier is absent: Usually, routines are coded to spitout messages like "Couldn't open logfile! when something goes wrong.These messages are a huge help in figuring out what a routine does.This firmware, however, has none of these strings: you need to figureout what a routine does purely by the code. The codebase seems to bea bit old, though, and sometimes the disassembly feels like some featureshave been 'bolted on' to the code later, making everything a bit morecomplicated.

There also are a few things that make life easier, though. First of all,it seems Western Digital hasn't been intentionally obfuscating the code:no tricks like jumping in the middle of an instruction have been used.Also, because the JTAG-interface is available, you can meddle with thecode, set breakpoints or change it on-the-fly, making figuring out whatroutine gets run when immensely easier.

After a long time of staring at the code, trying to make sense of thingsand sometimes jumping into the debugger to see if a guess was correct,I managed to get to the core of the caching system: a table in RAMI call the 'cache descriptor table':

Every entry in the cache descriptor table describes a block in thecache. It contains the start LBA of the disk sectors that are or should be cached, how much of the cache actually is filled with disk data, some flags indicating the state of the cache entry and a number indicating where in memory the cached data resides.

Now, with the secrets of the cache descriptor table unraveled, couldI intercept a disk read before it'd go out the SATA-port to the PC?To do that, I'd need to be able to execute my own code on the harddisk controller. Moreover, I would have to make sure the codewould get run on the correct time: if it modified the cache too soon,the data wouldn't be in there yet; if it modified the cache too late,the data would've already gone to the PC.

The way I did this was by hooking an existing routine. My hack would be in Feroceon 2, and that CPU did all the SATA transfers, so there mustbe some routine that's responsible for setting up the SATA hardwareto pick up the data from cache. If I could find this routine, I couldperhaps run my own code before it.

After a lot of browsing, setting breakpoints, failing and trying again,I finally found some routine that fit the bill. I modified it to runmy code before it by hooking it. Here's the original code:

000167BE ; r0 - slot in sata_req000167BE sub_0_167BE:000167BE PUSH {R4-R7,LR}000167C0 MOVS R7, R0000167C2 LSLS R1, R0, #4000167C4 LDR R0, =sata_req000167C6 SUB SP, SP, #0x14000167C8 ADDS R6, R1, R0000167CA LDRB R1, [R6,#0xD]000167CC LDR R2, =stru_0_40028DC000167CE STR R1, [SP,#0x28+var_1C]000167D0 LDRB R0, [R6,#(off_0_FFE3F108+2 - 0xFFE3F0FC)]000167D2 LDRB R5, [R6,#(off_0_FFE3F108 - 0xFFE3F0FC)]000167D4 LSLS R0, R0, #4And here's what happens when the code is hooked to call my code:000167BE ; r0 - slot in sata_req000167BE sub_0_167BE:000167BE PUSH {R4-R7,LR}000167C0 MOVS R7, R0000167C2 LD R6, =hookedAddr000167C4 BX R6000167C6 .dw checksumFix000167C8 .dd hookedAddr000167CC LDR R2, =stru_0_40028DC000167CE STR R1, [SP,#0x28+var_1C]000167D0 LDRB R0, [R6,#(off_0_FFE3F108+2 - 0xFFE3F0FC)]000167D2 LDRB R5, [R6,#(off_0_FFE3F108 - 0xFFE3F0FC)]000167D4 LSLS R0, R0, #4...FFE3F000 PUSH {R0-R12, LR}FFE3F004 BX changeThingsInCacheFFE3F008 POP {R0-R12, LR}FFE3F00C LSLS R1, R0, #4FFE3F010 LDR R0, =sata_reqFFE3F014 SUB SP, SP, #0x14FFE3F018 ADDS R6, R1, R0FFE3F01C LDRB R1, [R6,#0xD]FFE3F020 BX 0x167CC

As you can see, some original instructions are replaced with a jump tonew code in an otherwise unused bit of ram at address 0xFFE3F000 andan extra word to make sure the checksum of the code region still is valid. If this isn't done, the HD will try to load a backup from its platters, which isn't what we want. The code that's jumped to executesa routine called changeThingsInCache and then does what the replacedcode would've done. It then continues execution in the original routinelike nothing has happened.

Now all I need to write was a routine to modify the cached data. For afirst test, I decided on a routine that in pseudocode went somethinglike this:

void hook() { foreach (cache_struct in cache_struct_table) { if (is_valid(cache_struct)) { foreach (sector in cache_struct.sectors) { sector[0]=0x12345678; } } }}

This little bit of code would replace the first 4 bytes of everysector in cache with 0x12345678 every time it's called, so if I uploaded all this to the hard disk, I should see that number on the start ofevery sector I read. I uploaded the bits of code over JTAG...

And lo and behold:

 

Persistence

Ofcourse, I could make this into a full hack, but needing to useJTAG to poke it in RAM every time the hard disk boots would make itpretty useless. I needed to make it persistant, that is, I needed tostore my modifications somewhere where it would be picked up again every time the hard disk powers on.

My location of choice was the flash rom. I could probably also have putit somewhere in the reserved sectors on the disk itself, but if I messedsomething up, I would have no way to recover my disk. The flash chipis just an eight-pin standard part, so I could easily take it out, flashit and put it in again. For that purpose, I desoldered it and put iton a bit of veroboard, so I could easily switch it between a programmerand the hard disk:

Now, what to put in the flash? Luckily, the format of what's storedin the chip already has been figured out: it consists of multipleblocks of data, with a table describing them at the very start. Thattable describes the location of the block in flash, how it's compressed(if it is compressed), the location where the block should be putin RAM and, for the final address, an execution point where the loaderwould jump to to start executing the program.

Unfortunately, I couldn't modify the code that was in the flash; the bits that contained the parts where I wanted to put my hooks was compressed withan unknown compression algorithm, so I couldn't modify that. What I howevercould do was add an extra block, and modify the execution address so that block would get executed before the rest. That made things a bit easier:when 'my' block got executed, I could just code it to insert the hooks in thenow decompressed bits of code.

Ofcourse, I had to dis- and re-assemble the flash binary for this. I created atool for that, unimaginatively called 'fwtool'. This tool can dump out the various blocks in the flash, plus translate the header into a text file for easymodification. You can then modify, delete or add a block and re-assemble everything into a single firmware file, ready to be re-flashed. I used that toadd my custom bit of code to the image, flashed everything back to the chip,put the chip back into the HD, booted everything back up and this was theresult:

The result isn't that shocking: it's exactly the same as I had before. The trick isthat I didn't need the JTAG-rig to get it.
 

Software flashing

While the flash mod was a good step forward, I still couldn't play out my imaginary hacker scenario: I don't think any server company accepts 'donations'of hard disks with de- and re-soldered flash chips. I needed to find a way tore-flash the chip while it was still soldered to the hard disk, preferably fromthe PC the hard disk was connected to.

The Western Digital firmware upgrade tools proves this is possible: it's basically a tool you run under DOS to put new firmware to both the flash andthe service area aka the reserved sectors of the hard disk. According to theInternet, the tools use so-called Vendor Specific Commands to There are also someother tools that can meddle with the firmware: for example, there is a bit ofproof-of-concept codethat can use unused reserved sectors to hide away data. Finally, there's a set oftools called idle3-tools thatcan be used to modify a byte in the firmware to modift the idle behaviour of thehard disk. This code also uses VSCs, and does this using the 'official' wayusing Linux scsi passthrough ioctls. I decided to 'borrow' this code, modify ita bit and integrate it in fwtool. After some messing around and guessing VSCparameters, fwtool could all of a sudden also read and write the flashof a HD attached to the PC it's run on.

With this, my attack was complete. If a blackhat hacker had somehowobtained root access to a server with this drive, he could use fwtool to remotely dump the flash of the disk, modify it and flash it back. Eventually, the owner ofthe box will find out I am using his box for nefarious purposes and will probablyre-install the system, securing the way the hacker orginally entered the machine.

With the firmware hack in place, however, the attacker could tell the hard disk to do something nefarious with the new install. He'd need to trigger that behaviour first, though, and that could be done by writing a certain magic string the firmware hack would look for to the disk. The magic string can be in any file; the attacker could for example upload a .jpeg-file with the string in it to the server. He could also request a file from the webserver with the magic string appended to the URL. That would eventually end up in the logs of the machines, triggering the exploit.

The hard disk firmware hack would then do something nefarious. For example, it could wait for the machine to read out the file /etc/shadow, where all the passwordsare stored on an Unix/Linux system, and modify the contents on-the-fly to somethingthe attacker hardcoded earlier. When the attacker would then try to log into thesystem with his own password, the machine would check this password against thenow-modified /etc/shadow and the attacker would be free to login again.

Here's the demonstration I did at the presentation. You can see me try to log into the root account of the machine unsuccessfully. I then enable the hack and give ita replacement password hash, namely for the password 'test123'. Because Linux caches the shadow file (like all files recently accessed), I have to generate a lotof disk activity for the file to be 'pushed out' of the cache; that way, when I try to login again, Linux will be forced to fetch the shadow file from disk again. Finally, with the cache cleared, I can just log into the root account with thefaked test123 password.
 http://spritesmods.com/hddhack/demo-pass.ogv

Other uses

Ofcourse, restoring access to servers which had their clandestine entry methods removed isn't the only useful way my reverse engineering efforts can be used for.It can also be used for defensive purposes.

For example, you could make an un-clonable hard disk: the hard disk would act normal if the access pattern for the sectors was somewhat random, like a normal OSwould access a filesystem. If the disk was accessed only sequentially, like adisk cloning utility would do, the hard disk could mangle the data, making the clonedifferent from the original.

The disk controller is also interesting as a generic controller board. You havethree fairly capable CPU cores, with a pretty big amount of RAM connected to it.There's also an uart, for the serial port, and at least two SPI interfaces; oneto the flash rom and one to the spindle controllers. You can load the code for the processor by updating an external flash chip, or even by using the serial portin the bootloader. To demonstrate the power of the chip, I ported a fairlyubiquitous bit of software to my HD. The demo is a proof-of-concept only, theserial port is the only peripherial that works, and no userspace is available yet.Nevertheless, I am still a bit proud to say I have installed Linux on my hard disk.On top, a standard command line (the HD is mounted under /mnt), on the bottom theoutput of my work on the serial port of the hard disk:
 http://spritesmods.com/hddhack/demo-linux.ogv

A bit more explanation about what happens here: the kernel and initare both packed in pieces with the size of exactly one sector, witha magic string and order number prepended. By reading the file fromthe disk, it will end up in the cache of the disk. The write of themagic string 'HD, lnx!' finally triggers the modified firmware tosearch the cache for all the sectors, re-assemble the kernel imageand boots it. The kernel is built for a MMU-less CPU (the disk controller doesn't have one) and only has a driver for the serialport. A MMU-less kernel unfortunately needs a specially formattedbit of userspace too. I couldn't get this to compile, so the kernelfinally panics because it can't find an init it can execute.
 

Conclusion

So, there you have it. While the hard disk controller is a beast without much data known about it, it's still perfectly well possible to reverse engineer it and towrite custom code for it. The unknown-ness of the controller does make it harderto write generic hacks, which makes me doubtfull that a thing like the evil firmware patch will ever be seen in the wild: it's much easier to just get anotherzero-day software exploit than reverse engineer the firmware of every single harddisk every server you stumble upon has.

I also hope to have proven that a broken hard disk is something you can still use. While the mechanics of a broken HD probably are shot, the PCB still contains an usable embedded system, which actually is pretty powerful considering you can usuallyget broken hard disks for free.

Releasing the source-code for a security project always is a nasty subject. I want torelease code, but I do not want to be responsible for a lot of permanently hackedservers... I decided to compromise: you can download the code I used here, but I removed the shadow-replacement code.Make note: I'm not going to support the process to get all this running in any way;it's a hack, you figure it out.