When you need another brain

Cloning Ubuntu 18.04 LTS cloud image on VMware, using cloud-init

leave a comment »

This entry is inspired by Myles Gray’s posts “Using cloud-init for VM templating on vSphere” and  “Creating an Ubuntu 18.04 LTS cloud image for cloning on VMware”. I’ve experienced the same issues as the author while trying to combine cloud-init and VMware customizations using DataSourceOVF and had to use workarounds from https://kb.vmware.com/s/article/54986.

Most annoyingly, I’ve hit https://bugs.launchpad.net/cloud-init/+bug/1835205 on clone reboots, which increased the clone boot times by 90 seconds. It looks like Myles finally gave up on cloud-init since he’s suggesting to just purge it from the template and go with the VMware legacy perl customization engine. I believe another reason for this is his usage of govc to manipulate the template, and govc does not have facilities to manipulate vApp properties. vApp properties customization is needed when using cloud-init with OVF data source.

VMware does have a project – https://github.com/vmware/cloud-init-vmware-guestinfo – that will allow to use guestinfo properties to inject user_data into the vApp, but it is not yet upstreamed into the cloud-init. So I’ve decided to just embrace cloud-init’s DataSourceOVF for now and keep VMware customization disabled. This approach also allows to automatically customize VMs that are not supported by VMware customization engine (Debian 10, for example).

Deploy Ubuntu 18.04 cloud image OVA

$src = "https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.ova"
$ova = "ubuntu-18.04-server-cloudimg-amd64.ova"
Start-BitsTransfer -Source $src -Destination $ova

Import OVA into vSphere:

$name = "ubuntu1804template"
$vmhost = get-vmhost | select -First 1
$network = "<portgroup_name>"
$ds = get-datastore -Name "<datastore_name>"
$dsf = "thin"
$ovfConfig = Get-OvfConfiguration $ova
$ovfConfig.NetworkMapping.VM_Network.Value = $network
Import-VApp -Source $ova -VMHost $vmhost -OvfConfiguration $ovfConfig -datastore $ds -DiskStorageFormat $dsf -name $name

Ubuntu cloud image is now deployed and powered off. We can use vSphere Web Client to examine vApp properties that will be used by cloud-init on the first boot to customize the VM (VM->Configure->vApp Options). Notice that OVF environment  transport is set to ISO Image. Once VM is powered on, vSphere will create a small ISO and will attach it to VM for cloud-init to read it’s values.


In general, the workflow would look like this:

  1. Deploy the unconfigured Ubuntu cloud image OVA (done above)
  2. Clone the cloud image into a working template VM using “template_user_data” cloud-init file, which will update the packages, inject ssh keys, and perform other customizations such as setting an NTP server.
  3. Use the working template as a clone source to provision new VMs with another “user_data” cloud-init file with will further customize the VM (for example, you can use Myles Gray’s “user_data” for Kubernetes cluster setup).

I wrote a small Powershell script that can be used to clone the VM, attach the user-data file, and change the instance-id to force cloud-init to run the customization. We would also need to get an additional script from William Lam, which provides a couple of functions – Get-VMOvfProperty and Set-VMOvfProperty – to manipulate vApp properties.

Start-BitsTransfer https://raw.githubusercontent.com/vmsysadm/scripts/master/vmclone-init.ps1
Start-BitsTransfer https://raw.githubusercontent.com/lamw/vghetto-scripts/master/powershell/VMOvfProperty.ps1

Source the scripts:

. .\VMOvfProperty.ps1
. .\vmclone-init.ps1

I use a simple user-data file to update the packages and inject the ssh keys into the template at first boot, but cloud-init provides a very extensive configuration capabilities.

Start-BitsTransfer https://gist.githubusercontent.com/vmsysadm/727fb3a855f904664a54f0d3002f0e7e/raw/1f6232475634dc9e69d630db54bc63ea1fba961c/template_user_data

Now you can use all this to prepare the template for the initial setup.

Cloning from base template

vmclone-init $name .\template_user_data base_image_01

This creates a new base_image_01 template which is an updated version of the original ubuntu1804template with ssh keys and other modifications from template_user_data cloud-init file. The same vmclone-init command now can be used to clone additional VMs off the base_image_01 with further modifications from a different user_data file. For example, you can use Myles Gray’s “user_data” for Kubernetes cluster setup.


/var/log/cloud-init-output.log is available with cloud-init execution steps, or your can run “cloud-init analyze show -i /var/log/cloud-init.log” to check the execution times. “cloud-init analyze blame -i /var/log/cloud-init.log” is also helpful. Note that some modules are set up with frequency “always”, so cloud-init does run on each reboot but since the instance-id does not change and the cache is present in /var/lib/cloud, it does not perform the customization. You can check which modules are set to always run with “cloud-init analyze dump -i /var/log/cloud-init.log | grep -i always”. To force cloud-init to perform re-customization, you can either change the instance-id in vApp properties or do “rm -rf /var/lib/cloud”, per https://git.launchpad.net/cloud-init/tree/doc/sources/ovf/README.


Written by vmsysadmin

September 20, 2019 at 1:47 pm

Posted in vSphere

Tagged with

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: