Entries in esxi (5)

Tuesday
Feb252014

Mac Mini Hosting (ZFS loopback)

ZFS

I’ve been a fan of ZFS storage for a while now and I find it particularly useful in this type of standalone environment. One of my constant preoccupations (personally and professionally) is data protection from both a backup and disaster recovery perspective. So in this case, I’m going to create a VM locally that is going to play the role of a storage server that will publish NFS shares to the ESXi server internally.

From here, I gain all of the advantages of the ZFS built-in features like snapshots, compression, and the ability to efficiently replicate data back to the ZFS servers at home.

There are basically two main streams of ZFS based systems: Solaris kernel based solutions and BSD ones. ZFS on Linux is making some headway, but I haven’t yet had the time to poke at the current release to check out its stability.

If you want a simple setup with a really nice web interface for management, FreeNAS is probably the best bet. I personally prefer OmnisOS for this kind of thing since I drive mostly from the command line and it’s a very lean distribution with almost no extra overhead. Generally speaking the big advantage of FreeNAS is the extensive hardware support because of its BSD roots. But in my case, the VM “hardware” is supported so this doesn’t matter as much.

I’ll give a walkthrough on the process for using OmniOS here.

Networking

You could simply deploy the ZFS VM directly on the virtual LAN network you’ve created, but I like to keep storage traffic segmented from the regular network traffic. So I create a new virtual switch with a port for connecting VMs called Storage (or NFS or whatever). I also create a new VMkernel interface on this vSwitch using a completely different subnet that will be used to connect the NFS server to the ESXi server.

If this server were completely isolated and there was no need for routing traffic, I would just leave it this way, but there are a couple of things to keep in mind: how to you manage the server and how does it replicate data? You could manage the server by using the Remote Console in the VMware client, but ssh is more convenient. For replication the server will also need to be able to connect to machines on the remote network so it will need to be attached to the routed LAN subnet.

So the preferred configuration for the VM is with two network cards, one for the LAN for management and replication, and one for NFS traffic published to the ESXi server.

Installation & Configuration

The install process is just a matter of downloading the ISO, attaching it to a new VM and following the instructions. I configured mine with 3Gb of RAM (ZFS like memory), an 8 Gb virtual hard disk from the SSD for the OS and two 400Gb virtual disks, one from the SSD and one from the HD. When I build custom appliances like this, I usually use the vmDirectPath feature in order to pass a disk controller directly to the VM, but it’s a little hard to bootstrap remotely.

Then from the console, there’s some configuration to do. The first stages are best done from the console since you’ll be resetting the networking from DHCP to a fixed address.

Network configuration

Here’s a set of commands that will configure the networking to use a fixed IP address, setup DNS and so on. You’ll need to replace the values with ones appropriate to your environment. Most of this requires that you run this as root, either via sudo or by activating the root account by setting a password.

echo "192.168.3.1" > /etc/defaultrouter # sets the default router to the LAN router
echo "nameserver 192.168.3.201" > /etc/resolv.conf # create resolv.conf and add a name server
echo "nameserver 192.168.2.205" >> /etc/resolv.conf # append a second DNS server
echo "domain infrageeks.com" >> /etc/resolv.conf # and your search domain
echo "infrageeks.com" > /etc/defaultdomain
cp /etc/nsswitch.dns /etc/nsswitch.conf # a configuration file for name services that uses DNS
svcadm enable svc:/network/dns/client:default # enable the DNS client service
echo "192.168.3.2 nfsserver" >> /etc/hosts # the LAN IP and the name of your server
echo "192.168.3.2/24" > /etc/hostname.e1000g0 # the IP for the LAN card - a virtual Intel E1000 card
echo "192.168.101.2/24" > /etc/hostname.e1000g1 # the IP for the NFS card
svcadm disable physical:nwam # disable the DHCP service
svcadm enable physical:default # enable fixed IP configuration
svcadm enable svc:/network/dns/multicast:default # enable mDNS

Reboot and verify that all of the network settings work properly.

Setting up zpools

The first thing you’ll need to do is to find the disks and their current identifiers. You can do this with the format command to list the disks and Ctrl-C to cancel the format application.

# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
       0. c2t0d0 <VMware-Virtualdisk-1.0 cyl 4093 alt 2 hd 128 sec 32>
          /pci@0,0/pci15ad,1976@10/sd@0,0
       1. c2t1d0 <VMware-Virtual disk-1.0-400.00GB>
          /pci@0,0/pci15ad,1976@10/sd@1,0
       2. c2t2d0 <VMware-Virtual disk-1.0-400.00GB>
          /pci@0,0/pci15ad,1976@10/sd@2,0
Specify disk (enter its number): ^C

In my case, I have my two 400 Gb disks available, and are available at the identifiers c2t1d0 and c2t2d0, which corresponds to Controller 2, Targets 1 & 2, Disk 0. The first entry is for the OS disk.

I’m going to create my first non-redundant zpool on the SSD disk which is the first one with:

zpool create ssd c2t1d0

Note that these names are cases sensitive. The result is a new zpool called ssd which is automatically mounted on the root file system so it’s contents are available at /ssd.

Before continuing there are a few settings that I like to configure since any filesystems I create will inherit these settings so I only need to do it once.

zfs set atime=off ssd
zfs set compression=on ssd

Disabling atime means that it won’t update the last accessed time metadata for files which incurs an unnecessary write overhead for our use case. On a regular file server this value may be more useful. Setting compression is a practically free way to get more usable space and the overhead is negligeable.

Now we have a pool, we can create filesystems on it. I’m going to keep it simple with top level filesystems only. So for my setup, I’m creating one filesystem for my mail server all by itself and a second one for general purpose VMs on the LAN.

zfs create ssd/mail
zfs create ssd/lan

To make these accessible to ESXi over NFS, we need to share the filesystems:

zfs set sharenfs=rw=@192.168.101.0/24,root=@192.168.101.0/24 ssd/mail

Here I’m publishing the volume over NFS to all IP addresses in the 192.168.101.0/24 subnet. Currently the only other IP in this zone is the VMkernel interface on the storage vSwitch, but you could imagine having others here. This means that the NFS share is not available to the LAN, which is the desired configuration since the only thing on this volume should be the mail server VM. You can also set this to use specific IP addresses by ommiting the @ prefix which designates a subnet.

Now I’ve had some permissions issues from time to time with NFS and VMware, so to keep things simple, I open up all rights on this filesystem. Not really a security issue since the only client that can see it is the ESXi server.

chmod -R a+rwx

VMware

Now on the ESXi Configuration we need to mount the NFS share so that we can start using it to store VMs. This is done in Configuration > Storage > Add Storage… and select NFS.

The server address is it’s IP address on the storage subnet and the path to the share is the complete path: /ssd/mail in this case.

Snapshots and backups

Now that we’ve got a VMware datastore where we can put VMs, we can profit from the advanced features of the ZFS filesystem underlying the NFS share.

The first cool feature is snapshots where you can make a note of the state of the filesystem that you can then use to roll back to previous points in time, or mount it on a new filesystem so you can go pick out files you want to recover. Now it’s worth remembering that snapshots are not backups since if you lose the underlying storage device, you lose the data. But it is a reasonable tool for doing data recovery.

Creating snapshots is as simple as:

zfs snapshot ssd/mail@snapshot_name

Any new writes are now written to storage in such as way as to not touch the blocks referred to the filesystem at the time the snapshot was taken. The side effect is that you can consume more disk space since all writes have to go to fresh storage so you also need to delete (or commit) snapshots so that they don’t hang around forever.

zfs destroy ssd/mail@snapshot_name

Will delete the snapshot and free up any associated blocks that were uniquely referred to by the snapshot. Obviously, this is the sort of thing that you want to automate and there are a number of tools out there. I’ve created a couple of little scripts that you can pop into cron jobs to help manage these tasks. simple-snap just creates a snapshot with a date/time stamp as the name. The following line in crontab will create a new snapshot every hour.

0 * 0 0 0 /root/scripts/simple-snap.ksh ssd/mail

For cleaning them up, I have auto-snap-cleanup which takes a filesystem and the number of snaps to keep as arguments, so the following command will delete all but the last 24 snapshots:

1 * 0 0 0 /root/scripts/auto-snap-cleanup.ksh ssd/mail 24

So you’ll have a day’s worth of “backups” to go back to.

For the remote replication go check out the scripts under the projects tab. If you have two physical machines, each with an SSD and an HD a reasonable approach would be to replicate the SSD filesystems to the HD of the other machine (assuming you have an IPsec tunnel between the two). In this case, if one machine goes offline, you simply set the replicated filesystems on the HD to read/write, and mount them to the local ESXi host, register the VMs and you can start the machines. Obviously there will be a performance reduction since you’ll be running from the hard drive, but the machines can be made available practically immediately with at most an hour’s worth of lost data.

Time Machine

You can also use a ZFS server as a Time Machine destination with Netatalk which adds the AFP protocol to the server. Compiling netatalk can be a picky process so on OmniOS I tend to use the napp-it package which automates all of this and provides a simple web interface for configuration, making it a similar system to FreeNAS (but not as pretty).

Tuesday
Feb252014

Mac Mini Hosting (step by step)

Basic ESXi configuration

For this you’ll need a windows machine to run the VI-Client (although you can do most of it from the command line, this will be much easier via the client if you’re just starting with ESXi). As as alternate method, you could download the vCenter Linux Appliance in evaluation mode and run it as a local virtual machine using Fusion or something else locally. Then you would manage the ESXi server with this machine and you can use the web interface from your browser to do the configuration. I would recommend doing this at some point anyway in order to get access to some of the features reserved for ESXi 5.5 like the latest VM versions for running OS X VMs.

At this stage, you’ll be talking directly to your ESXi server across the internet, so the first thing to do will be to set the root password to something horribly complicated (which you will then note in your password manager or some other secured storage).

The basic steps of getting the client and changing the root password can be found here.

The networking configuration is a little specific to this kind of all in one configuration. We start with the default vSwitch where we have the ESXi management port setup with its internet facing IP address.

First off, we need a new vSwitch to plug our VMs into. For this, we go to Configuration > Networking > Add Networking… Select “Create a vSphere standard switch” with no physical adaptors selected (it’s already allocated to the default switch anyway) and in the next screen, call it LAN with the default VLAN ID.

Now we have two virtual switches, one connected to the ethernet card, connected to the internet and one connected to nothing at all for the moment.

For the beginning storage, we’ll just format the internal drives as individual datastores. This is done under Configuration > Storage > Add Storage…, selecting Disk/LUN as the storage type.

Here you may run into problems if the disks were previously formatted by another OS. Refer to this note if this happens. Accept the defaults and name the volumes however you like. I went with the exceedingly unoriginal SSD and HDD.

Bootstrap your first virtual machines

To get started we have a bit of a chicken and egg situation. To install your first virtual machines, you’ll need an ISO image. Now if this machine were on the local LAN, it would be fairly easy to just map an ISO image from the machine you’re using to run the VI-Client. However, if you’re like me, your DSL upstream speed is pretty awful (~100KB/s) so interactively reading from a local disk is going to be really really slow.

So to get around this, we’ll go back to the ESXi command line via SSH. You’ll need to activate ssh and the shell under Configuration > Security Profile. On the Services section, click the Properties button to get the list of services. From there, select the ESXi Shell, Options and Start. Do the same for SSH.

Now we can open a terminal session and use

ssh root@<your_ip>

From here we can navigate to our VMFS volume :

cd /vmfs/volumes/SSD
mkdir sources
cd sources

and download an ISO image using wget. I’m going to start with pfSense:

wget http://files.nyi.pfsense.org/mirror/downloads/pfSense-LiveCD-2.1-RELEASE-amd64.iso.gz

This will download the image to the sources directory on the VMFS volume. Now we can create the pfSense VM and install the firewall.

Installing the pfSense VM

To create pfSense VM, we’ll start in the VM client or web interface and create a custom VM. There’s a great instruction page on the pfSense website, so I’m not going to repeat all of that. The key things that I’ve done in my configuration is named the interfaces for the virtual machines as follows :

  • WAN
  • LAN

To correspond with the pfSense naming conventions. Actually, I went a little further and have added another subnet called DMZ to separate my internal machines and services from internet facing stuff. Again, in this case, there is a separate vSwitch with a separate named connection associated. Unlike the configuration explained on the pfSense site, these vSwitches have no physical adapters associated with them.

Basic pfSense configuration

pfSense is a complete firewall solution and as such can be configured in many different ways. You can either work from the principle that we let everything through, and then block off places where we don’t want to go, or we can do the opposite and block everything by default and open up ports selectively.

Getting the VPN running

Setting up an IPsec VPN can be a bit of a pain, but on most modern systems it’s not too bad as long as you pay attention to making sure you have selected exactly the same settings on both sides. In fact the biggest problem with getting IPsec working is that there is no industry standard UI implementation that shows the same form structure to fill out so it’s easy to get various settings mixed up.

I found the Cisco RV180 to be a perfect fit for my use here. It can handle site to site VPN, some basic internal VLAN stuff and is quite inexpensive for the feature set. In France, it sells on Amazon for 109 € and in the US it’s priced at $108. There is also a version that includes wireless capability but since I already have my wifi setup using an Airport Extreme and various Expresses for extension and Airplay I went with the simple model.

Side note: If you’re currently replacing something like an Airport Extreme for your local wireless networking, you can just plug it into the back of the RV180 and set it to bridge mode so that it’s no longer routing, just offering wireless services.

So the architecture looks like this:

And here’s the configuration that’s running:

pfSense:

Cisco RV180:

Once you have this up and running, any machines on your local network should be able to route to the LAN subnet on the hosted machine and vice-versa. This makes it a transparent extension of your current network.

Routing changes

Now that we have a firewall and a solid VPN connection running, it’s a good idea to make a few more little changes to ensure that your ESXi server isn’t hanging out exposed directly to the internet.

First off, we can shutdown the SSH and ESXi shell in the Configuration > Security Profile > Services section.

Now the only exposed ports are the standar ESXi management ports. You can manually adjust the exposed ports firewall rules so that they only accept connections from your network in the ESXi configuration. But this means that if you need to connect to your server while you’re one the road, you’ll be blocked. So my approach is to add another vKernel Management interface on the internal LAN interface, and then use the LAN gateway as the default route. The net result of this configuration is that the server can only be accessed from machines either on the LAN subnet behind the pfSense or your home/office network that is connected via the VPN. Then I add a VPN service that I can connect to from my other machines while on the road.

I leave the original vKernel management port installed just in case something catastrophic goes wrong and I can get the hosting provider to reset the default gateway at the console to get remote access working via the internet.

Automatic boot

Now that we have set things up this way, there’s no way back in from the internet if the host ESXi machine reboots. So to get around this problem we need to make sure that the pfSense VM automatically starts up when the host starts.

This is configured in Configuration > Virtual Machine Startup/Shutdown. It’s just a matter of editing the list so that the pfSense VM is the first machine in the list to boot automatically when the ESXi server starts up. This is mostly an additional protection since ESXi is a remarkably stable environment so the only reason this should come into play is when you want to upgrade the ESXi itself.

Next Up : Loopback ZFS storage

Tuesday
Feb252014

Mac Mini Hosting

Backstory

Up until now, I’ve been handling my own hosting via my home DSL service for a number of years now, with a few VPS instances and Squarespace service for some other web sites. One key component is my mail server, currently hosted on OS X Server in a VM running on ESXi at home. I had an unfortunate incident where the local telco cut off my DSL line by accident and as a result I was without internet connectivity for almost three weeks starting around Christmas.

This meant that all mail destined to my infrageeks.com domain ended up bouncing after a certain amount of time which was the impetus to finally go out and get a Mac Mini hosted in a datacenter. I’ve been thinking about doing this for a long time, but put off by the cost. But now I find that some of the Mac Mini colocation services will install ESXi as a standard install, which makes it considerably more attractive since I am no longer limited to one OS instance and can run multiple OS environments. This also means that I’ll be able to consolidate some of my other VPS and web hosting services onto this machine.

Yes, I could have done the same thing with a base install and VMware Fusion, Parallels or VirtualBox, but the result is not as efficient nor flexible enough for what I have in mind.

For the notes and instructions, I’m assuming you have a basic understanding of ESXi so I won’t be filling in a ton of screen shots, although if I get some feedback, I’ll go back and try and add some more in. Or (shameless plug) ping me for consulting assistance.

Configuration

I went with an upgraded Mac Mini with a 500 Gb SSD + a 500 Gb internal HD, running ESXi from an SD Card which leaves the full disk space available for ESXi.

The basics

Networking

One nice feature of running ESXi is that you can easily create a set of virtual networks with a router instead of just putting your virtual machines directly on the internet. This also allows you to do some interesting stuff like create a private network on your ESXi instance where your machines live, but are connected to your own network via VPN so it becomes just another subnet that behaves like an extension of your network.

This is important since my mail server is a member of an Open Directory domain that is hosted on another machine. By setting things up as an extended private network, the servers can talk to each other through normal channels without opening up everything to the internet or doing firewall rules on a machine by machine basis.

I’m using pfSense in a virtual machine linked with a VPN to a Cisco RV180.

Storage

You can use the internal disks formatted as VMFS-5 volumes natively to store your virtual machines, but I wanted to ensure that I had an effective backup plan that fit into my current system. I could have used VMware’s built-in replication feature or a third party tool like Veeam or Zerto which are designed for this kind of replication, but they all require vCenter and I wanted to see what could be done without additional investment.

So in this case I’m using OmniOS to create a virtual NFS server backed by VMDK files. It adds some extra work on the server since it has to go through NFS to OmniOS virtual machine to the disks, but I gain snapshots and free replication back to my ZFS servers at home.

I’m in the middle of an existential debate concerning the most efficient configuration for the zpool in this situation. One option is to have two zpools, one for the SSD and one for the HD and to replicate from the SSD to the HD on a regular basis, and then off to the storage server at home. If the SSD fails, I can easily switch over to the HD. But in this case, I don’t have any real automated protection against bit-rot since each block is stored on non-redundant (from the ZFS point of view) storage. The other obvious option is to create a single zpool with a mirror of the SSD and the HD which would ensure that there are two copies of each block and if there is a problem with the data, ZFS will read both copies and use the one that matches the checksum. But the flip side is that performance will become less predictable since some reads will come from a slow 5400 RPM disk and others from the SSD, while all writes will be constrained by the speed of the spinning disk. I could also just set the SSD as a very large cache to a zpool backed by the hard drive, but this seems a little silly with no additional data protection.

The other side effect of slow disk IO is that I will end up in many more situations where the CPU is waiting on disk IO which will slow down the whole system, so I’m going with the two pool setup for now. Budget permitting, a mirrored zpool with two SSDs is the ideal solution.

Next up: Step by step

Thursday
Jun272013

Managing Thin Provisioning

This question has come to me via a number of different channels over the last few days. Thin provisioning is a really nice feature to give yourself some additional flexibility in managing storage usage. But things have gotten more than a little confusing lately since you can implement it on different levels with different issues.

The biggest fear is what I refer to as a margin call. You have declared more storage to your servers that you really have, and at some point, you exceed your physical capacity and everything grinds to a halt. We’ve already see similar issues with using Hypervisor snapshots where the delta file grows unattended and the LUN fills up and can no longer accept any new writes.

In practical terms, I have a couple of different approaches, mostly depending on the environment you’re working in.

Production

You don’t want to take any chances in production. But this still doesn’t mean that thin provisioning is a bad idea. I strongly recommend that VMware’s thin provisioning not be used in production since there is a noticeable performance impact. However there are still good reasons to use it on the storage system:

  • Modern storage systems tend to use some kind of pooling technology to stripe IO across many disks. If you use fixed provisioning we have a higher probability of running into hot and cold spots and you might be limiting your performance.
  • Unexpected demands can arrive and if you have fixed provisioning your time to reaction may involve waiting on new hardware purchases.

So my general policy on production systems is to use thin provisioning, but never to overprovision. If I have an unexpected project that arrives and needs space, I can allocate it quickly, and start the purchasing process to ensure that my overprovisioned state is temporary. The key is to ensure that the demand is dependent on getting that purchase order approved, so the risk exposure is minimized.

Test, Dev, Integration, Qualification, …

In these environments the lifecycle is very very different from production. Much of the choices here depend on how you use these types of environments.

Much of the time, the work can be exploratory with unexpected demands for additional machines and storage as problems are identified and new test cases appear. In these environments, I tend more towards fixed allocation for a given project, but let the developers and testers the autonomy of deploying into these environments. Thus, the logical choice is to lean more towards thin provisioning at the VM level.

However to maintain maximum flexibility it can be useful to continue to use thin provisioning on the storage system. But in this case, we have a different issue: how to reclaim disk efficiently in an environment where machines can be created and deleted frequently? The problem is that a deletion only writes to the allocation table, but the actual blocks that represent the deleted VM have been written to and thus are still allocated on storage.

Reclaiming thin provisioned storage today remains a PITA. Basically, we need to send some kind of command to clear the contents of unallocated blocks (zeroing out) and then instruct the storage system to reclaim these blocks, which generally involves a pretty brute force approach of reading everything to see what can be reclaimed.

To get around this issue I have adopted a rolling approach where long lived test and development environments are renewed quarterly (or monthly depending on the volatility). This involves scripting the following actions :

  • Create a new LUN
  • Format the LUN as a datastore
  • svMotion the VMs on the source datastore to the new datastore
  • unmap the old LUN
  • delete the old LUN
  • possibly rename the new LUN (or use a minimal date stamp in the name)

This results in a freshly thin provisioned datastore with only the current VMs storage allocated. Any thin provisioned blocks on the original source LUN have been freed up with the deletion of the LUN.

Of course, you could always just use NFS backed by ZFS and let the system do its thing.

Other issue can come into play depending on your internal operating procedures, such as do you do internal billing for allocated storage? In these cases, the question of how to bill for thin provisioned storage is an ongoing debate.

Monday
Oct082012

ESXi, OS X and a Mac Mini

I’ve been lusting after my very own ESX install at home for a while now, especially after following the news that you get ESXi 5 running on a Mac Mini. A key factor for me is that I want to run multiple OS X instances so the Mini is a requirement without going down the hackintosh route. The other is purely practical as I don’t really have any available space so whatever I build has to fit on my desk with the rest of the stuff already there. Noise and power consumption are key factors.

Starting with a Mac Mini Server (5.3) as the base component. I debated going with one of the regular models, but the fact that it has two 500Gb internal drives plus a quad core i7 gave it the edge. RAM is currently pretty cheap so I pushed it to 16 Gb with aftermarket memory.

I intend to run a mix of production machines and pure test bed stuff. So the idea is to have a VM running Solaris have a mirrored zpool created from two VMDK files, presented locally via NFS for any production machines which will be replicated to the main NAS (an HP N40L running Solaris) via my auto replicate scripts.

The left over space on the two drives will be used for all of the miscellaneous test machines that I don’t care about so losing a drive is not going to bother me. I debated hosting the production machine directly on the NAS, but it’s getting a little full and I’d prefer to do that with a proper isolated storage network and I need to buy an additional Thunderbolt GbE adaptor for the Mini and a network card for the HP so that can wait.

Step by step

I started with the ready to go ESXi 5.0 image available from Angry Cisco Guy. Simply burn it to a CD, and using the external Apple DVD player boot it up (hold down C on startup). Then it’s just a standard ESXi install process.

Note: there are some excellent instructions for building your own custom installer over at Paraguin and Virtually Ghetto that you can use to go straight to 5.1 with the latest driver, but I ran across the Angry Cisco Guy link first and had already downloaded the image so I went the lazy route.

Now that I have the baseline running, I opened up the SSH shell and copied over the 5.1 offline update bundle and installed it using:

esxcli software install vib -d ....

Now this setup ran OK for a couple of days until I had a network freeze. ESXi was still running, but it was completely off the grid. Activating the SSH shell access opened it up again, but signaled that for production machines, this was not acceptable.

After a little touring around, I found the latest tg3 Broadcom driver (which would be required for the Thunderbolt adaptor) and installed it, again using esxcli.

Note that the bundle you download from VMware is not directly installable. You need to unzip it and use the offline-bundle file inside.

VMs

Now that the machine is running nicely, I installed the Solaris VM, assigned it two VMDK files, one from each internal drive.

This allows me to create a mirrored zpool for the additional read performance and reliability.

On the ESX side, I mount the new NFS share and I’m ready to deploy my production machines.

The first project is a replacement server for my aging second generation Mini that is an OS X Server instance that has been running since 10.1 and getting upgraded to every version up until 10.7 so it’s just full of cruft.

I had started building the replacement server in Fusion on my MacBook, so I figured that I should just convert it over. This however is not something sufficiently mainstream to be supported by the regular P2V or V2V tools out there.

I tried a number of different approaches including VMware Converter Standalone, copying the Fusion VMDK file to the NFS share and mapping it to the VM without any luck.

So the best approach to virtualization OS X computers seems to be to create a new VM on ESXi and then use the Apple Migration Assistant to bring over the old machine’s files and settings.

At the moment, ESXi has support for OS X machines up to 10.7 so I started there, mapping the installer dmg file to the new VM and installing from there. VMware does have VMware Tools for OS X so these got installed too. Then I copied the 10.8 installer image into the VM and ran it with no problems.

A quick tour with Software Update and I now have a fully up to date VM.

But then I ran into a limitation that the Migration Assistant will not do a machine to machine migration for a machine using OS X Server. I tried restoring from a Time Machine backup, but for some reason the Migration Assistant didn’t like the backup in question (running on Netatalk on the Solaris servers). So I had to fall back to the tried and true SuperDuper clone of the disk to a network share and play musical disks mapping the disk to a temporary VM to do the restore, disconnecting and booting the destination machine.

OS X is sufficiently hardware agnostic that cloning the drive from a physical Mac should work equally well.

Gotchas

A few things that may come back to bite you with the default settings applied by OS X:

  • The screen saver is on by default which chews up CPU, and makes it almost impossible to VNC in some of the time.
  • The power saving features are on by default and the machine will go to sleep so you need to disable these options
  • Be very careful with international keyboard mappings when assigning the initial password. I was going from a Mac via RDP to a Windows machine, to the VI-Client console of the new OS X VM and my first attempt was impossible to recreate. Keep it simple until you get to the native Screen Sharing connection.

Side notes

The Mac Mini’s only real fault that makes it ineligible to be a supported ESXi platform is the fact that it doesn’t support ECC RAM. So if you have a Radar account, filling out a request might be help Apple decide to add that into future versions.

Something else to keep in mind about the current generation of Minis is that Thunderbolt is nothing more than a transparent extension of the PCI bus. So if you have the budget, you can imagine some silly/amazing/freaky configurations using the Sonnet External PCIe boxes with a 10GbE or FC card.

They also have a really interesting rack mount kit that includes PCIe slots. But it’s horribly expensive compared to a standard off the shelf 1U Intel based Server.