When you need another brain

Cloning Ubuntu 18.04 LTS cloud image on VMware, using cloud-init

with 6 comments

This entry is inspired by Myles Gray’s posts “Using cloud-init for VM templating on vSphere” and  “Creating an Ubuntu 18.04 LTS cloud image for cloning on VMware”. I’ve experienced the same issues as the author while trying to combine cloud-init and VMware customizations using DataSourceOVF and had to use workarounds from https://kb.vmware.com/s/article/54986.

Most annoyingly, I’ve hit https://bugs.launchpad.net/cloud-init/+bug/1835205 on clone reboots, which increased the clone boot times by 90 seconds. It looks like Myles finally gave up on cloud-init since he’s suggesting to just purge it from the template and go with the VMware legacy perl customization engine. I believe another reason for this is his usage of govc to manipulate the template, and govc does not have facilities to manipulate vApp properties. vApp properties customization is needed when using cloud-init with OVF data source.

VMware does have a project – https://github.com/vmware/cloud-init-vmware-guestinfo – that will allow to use guestinfo properties to inject user_data into the vApp, but it is not yet upstreamed into the cloud-init. So I’ve decided to just embrace cloud-init’s DataSourceOVF for now and keep VMware customization disabled. This approach also allows to automatically customize VMs that are not supported by VMware customization engine (Debian 10, for example).

Deploy Ubuntu 18.04 cloud image OVA

$src = "https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.ova"
$ova = "ubuntu-18.04-server-cloudimg-amd64.ova"
Start-BitsTransfer -Source $src -Destination $ova

Import OVA into vSphere:

$name = "ubuntu1804template"
$vmhost = get-vmhost | select -First 1
$network = "<portgroup_name>"
$ds = get-datastore -Name "<datastore_name>"
$dsf = "thin"
$ovfConfig = Get-OvfConfiguration $ova
$ovfConfig.NetworkMapping.VM_Network.Value = $network
Import-VApp -Source $ova -VMHost $vmhost -OvfConfiguration $ovfConfig -datastore $ds -DiskStorageFormat $dsf -name $name

Ubuntu cloud image is now deployed and powered off. We can use vSphere Web Client to examine vApp properties that will be used by cloud-init on the first boot to customize the VM (VM->Configure->vApp Options). Notice that OVF environment  transport is set to ISO Image. Once VM is powered on, vSphere will create a small ISO and will attach it to VM for cloud-init to read it’s values.


In general, the workflow would look like this:

  1. Deploy the unconfigured Ubuntu cloud image OVA (done above)
  2. Clone the cloud image into a working template VM using “template_user_data” cloud-init file, which will update the packages, inject ssh keys, and perform other customizations such as setting an NTP server.
  3. Use the working template as a clone source to provision new VMs with another “user_data” cloud-init file with will further customize the VM (for example, you can use Myles Gray’s “user_data” for Kubernetes cluster setup).

I wrote a small Powershell script that can be used to clone the VM, attach the user-data file, and change the instance-id to force cloud-init to run the customization. We would also need to get an additional script from William Lam, which provides a couple of functions – Get-VMOvfProperty and Set-VMOvfProperty – to manipulate vApp properties.

Start-BitsTransfer https://raw.githubusercontent.com/vmsysadm/scripts/master/vmclone-init.ps1
Start-BitsTransfer https://raw.githubusercontent.com/lamw/vghetto-scripts/master/powershell/VMOvfProperty.ps1

Source the scripts:

. .\VMOvfProperty.ps1
. .\vmclone-init.ps1

I use a simple user-data file to update the packages and inject the ssh keys into the template at first boot, but cloud-init provides a very extensive configuration capabilities.

Start-BitsTransfer https://gist.githubusercontent.com/vmsysadm/727fb3a855f904664a54f0d3002f0e7e/raw/1f6232475634dc9e69d630db54bc63ea1fba961c/template_user_data

Now you can use all this to prepare the template for the initial setup.

Cloning from base template

vmclone-init $name .\template_user_data base_image_01

This creates a new base_image_01 template which is an updated version of the original ubuntu1804template with ssh keys and other modifications from template_user_data cloud-init file. The same vmclone-init command now can be used to clone additional VMs off the base_image_01 with further modifications from a different user_data file. For example, you can use Myles Gray’s “user_data” for Kubernetes cluster setup.


/var/log/cloud-init-output.log is available with cloud-init execution steps, or your can run “cloud-init analyze show -i /var/log/cloud-init.log” to check the execution times. “cloud-init analyze blame -i /var/log/cloud-init.log” is also helpful. Note that some modules are set up with frequency “always”, so cloud-init does run on each reboot but since the instance-id does not change and the cache is present in /var/lib/cloud, it does not perform the customization. You can check which modules are set to always run with “cloud-init analyze dump -i /var/log/cloud-init.log | grep -i always”. To force cloud-init to perform re-customization, you can either change the instance-id in vApp properties or do “rm -rf /var/lib/cloud”, per https://git.launchpad.net/cloud-init/tree/doc/sources/ovf/README.

Written by vmsysadmin

September 20, 2019 at 1:47 pm

Posted in vSphere

Tagged with

6 Responses

Subscribe to comments with RSS.

  1. Can you maybe elaborate more on the change the instance-id task. for example i can see that instance ID for the 16.04 image is set to id-ovf .. what do i need to change it to or do i even have to change it ? thank you

    Maher AlAsfar

    February 7, 2020 at 5:09 pm

    • You can change the instance-id to any random value. The reason to change it is if you want to force cloud-init to re-run the customization which is helpful if you want to customize your template later.

      You don’t need to change it if it’s the first boot or if you don’t need cloud-init to run at the VM boot.


      February 7, 2020 at 5:18 pm

      • Thank you, The reason i m asking is i m testing these cloud images with vRealize Automation 8.. when you create cloud config code as part of your overall machine blueprints to deploy packages or run commands that cloud-init would execute on boot. we map our own iso ( user data ) to the provisioned machine that includes the predefined cloud config code in the blueprint.

        so in may case i am just wondering how in this case would i configure the vApp options ? the first thing that I noticed is uncheck the ISO image it attaches by default since we attach our own via vRA 8. so i m currently trying to figure out what options i need to set.

        Maher AlAsfar

        February 7, 2020 at 5:27 pm

  2. Have you tried to do this with Packer?


    May 19, 2020 at 6:51 pm

    • I have looked at Packer briefly. Specifically I was interested in building Debian images on ESXi host since the Debian cloud team does not provide the OVA or VMware-compatible images, but in the end I just started to use my own Debian template built from netinst.

      Take a look at https://nickcharlton.net/posts/using-packer-esxi-6.html, maybe it will provide some direction?


      May 19, 2020 at 7:12 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: