How to create an ESXi v4 Whitebox ISO with Windows Tools

How to create an ESXi v4 Whitebox ISO with Windows Tools

Monday, July 27th, 2009 | VMWare

This is a short tutorial on how to build a custom ESXi v4.0 Whitebox ISO with Windows Tools (and an Ubuntu-VMImage for VMWare Player) by integrating a new driver (RTL8169 network card) into the Community Driver Pack and integrating this pack into the ISO, too.

See “Create an ESXi v3.5 Whitebox ISO” for creating an ESXi v3.5 Whitebox.

1. Warning:
You need a 64bit CPU in order to run ESXi v4! Download Processor Check for 64-Bit Compatibility on the VMware site and determine whether your CPU is supported!
Btw: the Intel T2300 CPU is NOT a 64bit CPU…

2. Warning:
There are reports that the RTL driver may not work on all boards because of the new 64-bit portion of the ESXi driver model. Try ESXi 3.5 then. Or wait until a new full supported 64bit driver is made. Check vm-help forum regularly.


ESX Hardware Requirements
VMware ESX 4.0 will only install and run on servers with 64-bit x86 CPUs. Known 64-bit processors:

• All AMD Opterons support 64 bit.
• All Intel Xeon 3000/3200, 3100/3300, 5100/5300, 5200/5400, 7100/7300, and 7200/7400 support 64 bit.
• All Intel Nehalem (no Xeon brand number assigned yet) support 64 bit.

Use this tools
- UltraIso $29.95 USD / Free Trial
- Ubuntu Server 8.10 for VMWare Player Free
- Bitvise Tunnelier Free for individual use (or Putty, but Tunnelier is preferred)
- WinImage US$ 30.00 / Free Trial

Let’s prepare by downloading the VMWare ISO, the Community Driver Pack and the new driver.

Download VMware ESXi 4 Installable
Download Community Unified oem.tgz v1.1.0
Download Realtek RLT8168 Driver oem.tgz package
Download Realtek RTL8169 Driver (if needed, try above first!)

1. Extract p_w_picpath.tgz from the VMware ISO with UltraIso

2. Use Tunnelier to log into Ubunto and create a new directory called “esxi” (either with SFTP or by command console with mkdir esxi)

3. Move p_w_picpath.tgz to this esxi directory (use Tunneliers SFTP)
Extract the contents by changing to this directory
cd esxi
tar zxvf p_w_picpath.tgz

and remove the tgz file
rm p_w_picpath.tgz

4. Decompress the VMWare diskp_w_picpath
cd usr/lib/vmware/installer/
bzip2 -d VMware-VMvisor-big-171294-x86_64.dd.bz2

Skip step 5-12 if you’re using the Realtek RLT8168 Driver oem.tgz package

5. Make a new directory for the Community Driver Pack
cd
mkdir oem
cd oem


6. Upload the Driver Pack into this directory and rename it to oem.tgz then extract the contents and remove the file with
tar zxvf oem.tgz
rm oem.tgz


7. Prepare a new directory for the realtek driver
cd
mkdir rtl8169


8. Upload the mymods-0.1.tar into this directory and extract the content with
cd rtl8169
tar zxvf mymods.tgz


9. Then add this driver to the pack
cp mod/r8169.o ../oem/mod
cd ../oem


10. Add the PCIID to the simple.map by using the texteditor nano (no, we are not using the “thing called vi”. Thanks).
cd etc/vmware
nano simple.map


11. Insert the line 10ec:8169 0000:0000 network r8169 at the appropriate place:
...
10df:fd00 0000:0000 storage lpfc_740
10df:fe00 0000:0000 storage lpfc_740
10ec:8169 0000:0000 network r8169
1166:0211 0000:0000 storage ide
1166:0212 0000:0000 storage ide
...

12. Now create the updated oem.tgz file and download it to windows
cd
cd oem
tar cfvz oem.tgz *

13. Now download the usr/lib/vmware/installer/VMware-VMvisor-big-171294-x86_64.dd file to windows and open it with WinImage (open partition0).

14. Use WinImage to replace the oem.tgz on partition 0 and save.

15. Transfer the dd file to ubuntu /esxi/usr/lib/vmware/installer/ and compress it
cd
cd esxi
cd usr/lib/vmware/installer/
bzip2 VMware-VMvisor-big-171294-x86_64.dd

16. Now recreate the p_w_picpath.tgz file
cd
cd esxi
tar cfvz p_w_picpath.tgz *

17. Download p_w_picpath.tgz to Windows

18. Use Ultraiso to replace p_w_picpath.tgz and add oem.tgz (yes, as of v4 the file isn’t here by default as it was with v3.5)

19. Burn ISO and start installing ESXi.

ESXi Tips and Tricks
http://www.vm-help.com/