WilliamLam.com

You are here: Home / ESXi-Arm / How to run Raspberry Pi OS as a VM on ESXi-Arm

How to run Raspberry Pi OS as a VM on ESXi-Arm

by // 80 Comments

It has only been a week since the ESXi-Arm Fling was released, but the amount of experimentation and frankly cool s*** that people have been able to do in such a short period of time has been pretty mind blowing. For example, did you know you could run ESXi-Arm on Nintendo Switch!?🤯

Andrei recently published a blog post on the official ESXi-Arm blog showcasing some of the really cool stuff the community has shared on Twitter, definitely worth checking digest post #1 it out if you have not seen it.

Something that really caught my eye which I did not see mentioned in Andre's blog post was from Twitter user Joakim Korhonen who shared that he was able to run Raspberry Pi's (rPI) OS (formally Raspbian OS) as a Virtual Machine running on top of the ESXi-Arm Fling!

This is pretty interesting because rPI OS was designed to run on a physical rPI and there are no installers other than the image file which you download and copy onto the SD Card to boot. What is really exciting about this news is that you can now run any of the popular rPI applications such as RetroPi or Pi-hole which traditionally may have required several rPI to host.

In addition, this can also benefit the rPI OS development community by making it easier to build and test applications on top of rPI OS as you can now spin these up as VMs and get all the benefits of vSphere and ESXi such as snapshots, cloning, etc. The possibilities are endless and wanted to give a huge thanks to Joakim for sharing his hack on getting this to work on ESXi-Arm. For those interested, I have documented the detailed instructions below.

UPDATE (08/27/23) - See this post HERE on instructions for updating to the latest Linux kernel for rPI OS

UPDATE (11/106/20) - For those familiar with VMware Virtual Appliances (OVA) and using custom OVF properties for guest/application customization, be sure to check out this complimentary blog post on how to build your own rPI OS OVA that can allow you to easily deploy additional rPI OS VMs with ease, especially useful for testing and development purposes.

Step 1 - You need access to a system that has the qemu-img utility installed. In my setup, I am using a Photon OS 3.0 (x86) VM that I have lying around. This can also be an Ubuntu VM if you have one of those. You will also need about ~6.5GB of free storage for the converted file format.

The following packages should be installed if you are using Photon OS:

tdnf -y install qemu-img unzip wget

Step 2 - Download the latest rPI OS Buster image to your system

wget https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2020-05-28/2020-05-27-raspios-buster-arm64.zip
unzip 2020-05-27-raspios-buster-arm64.zip
rm 2020-05-27-raspios-buster-arm64.zip

Step 3 - Use qemu-img to convert the image file to a VMDK

qemu-img convert -f raw 2020-05-27-raspios-buster-arm64.img -O vmdk 2020-05-27-raspios-buster-arm64.vmdk

Step 4 - SCP the VMDK file to the datastore of your ESXi-Arm host

Step 5 - We need to actually convert the VMDK one more time so ESXi understands it, as the qemu-img utility only converts it to a hosted format which is normally used by Workstation/Fusion. To do so, we just use our good ol' friend, vmkfstools. You need to SSH (SSH is disabled by default) to the ESXi-Arm host after SCP'ing the VMDK and then run this command from ESXi-Arm host:

vmkfstools -i 2020-05-27-raspios-buster-arm64.vmdk -d thin raspios-buster-arm64.vmdk

After the conversion completes, you can delete the source VMDK.

Step 6 - Download the latest Debian 10.x Arm Network Installer ISO and also upload that to the datastore of your ESXi-Arm host

Step 7 - Create a New VM called rPI-OS with the following configuration:

  • Compatibility: ESXi 7.0 and later
  • Guest OS Family: Linux
  • Guest OS Version: Debian GNU/Linux 10 (64-Bit)
  • CPU: 2 (or desired value)
  • MEM: 2GB (or desired value)
  • Hard Disk 1: Remove default since we are going to attach the one we just converted. To so by clicking on "Add New Device" and select Existing Hard Disk and locate that on the ESXi-Arm datastore
  • CD/DVD dive 1: Select Datastore ISO File and locate the Debian Network Installer ISO from ESXi-Arm Datastore and ensure the device has checked box for "Connected"


Step 8 - Power on the VM and in then select Advanced options from the GRUB menu


Step 9 - Next, select Rescue Mode


Step 10 - Complete the basic setup screen (language, country, keyboard, hostname, domain and timezone) before we can boot into rescue mode.

Step 11 - Select /dev/sda2 to boot from


Step 12 - Go ahead select Yes to mount /boot as a separate partition.


Step 13 - Select the Execute a shell in /dev/sda2 and you will be prompted once more, hit continue.


Step 14 - You should now be in a shell, go ahead and run the following command to pull latest updates:

apt update


Step 15 - Run the following command to install Grub package for Arm:

apt install -y linux-image-arm64 grub-efi-arm64


Step 16 - Run the following command to install Grub onto our rPI image:

grub-install --efi-directory=/boot


Step 17 - Finally, run the following command to update the Grub configuration:

update-grub


Step 18 - Type "exit" to exit from the shell and then select Reboot the system.


At this point, you should disconnect the Debian ISO from the vSphere UI or ESXi Embedded Host Client. You probably will see the following warning stating the GuestOS has locked the CD-ROM drive, go ahead and click on Yes to disconnect and that should allow the VM to continue booting from updated VMDK now.


If all changes were applied successfully, you should see the the Debian Grub menu startup and it should boot into our rPI image.


In a few seconds, you should be taken to the familiar rPI OS desktop screen!


One thing you will notice is that there is not much space left on the filesystem to do anything interesting. We can easily resize our disk by performing these two steps below.

Step 1 - In the vSphere UI or ESXi Embedded Host Client, change the disk from 3GB to whatever size you wish to use (this can be done while the VM is already booted up). In my example, I selected 10GB and the next step works for any disk capacity.

Step 2 - Open terminal in rPI OS and run the following command which will automatically expand filesystem and then reboot for changes to go into effect.

sudo raspi-config --expand-rootfs


If you do not want to go through this manual process again if you make a mistake, I highly recommend you create a VM Template before performing further customization and this way you can deploy additional copies.

Lastly, I wanted to also share that I was able to compile VMware Tools for rPI OS itself and have it successfully running. For those instructions, please refer to this blog post for more details.

More from my site

Comments

  1. Hello,

    for the command "vmkfstools -i vmdk 2020-05-27-raspios-buster-arm64.vmdk raspios-buster-arm64.vmdk" is not working on the ESXi. The message is :

    vmkfstools -i vmdk 2020-08-20-raspios-buster-arm64.vmdk raspios-buster-arm64.vmdk -d
    thin
    Extra arguments at the end of the command line.

    Reply

      • Well, there's so many possibilities if put your self into multiple perspectives. rPI OS was design to run on physical rPI, to do any development, you'd need to have a physical device or for breaking things or just messing around, you'd potentially need multiple rPIs. If you mess up, you reinstall, etc. As a VM, you get benefits of snapshot, clone, etc. So this can certainly augment and enhance developer experience for rPI OS to build even cooler applications like pi-hole, retroPi, etc.

        For consumers, I know a number of admins who have multiple rPI running single-based apps 🙂 They don't always use all the resources and already know a number of folks consolidating and planning to consolidate all their different apps. Once a VM, you can apply all sorts of policies and get additional monitoring that you may not have for a physical device. In any case, lots of folks were quite excited for this as they saw the potential as I did

        Reply

  3. Thanks for this! I found running this too before booting the new image helps.
    vmkfstools -X 8G raspios-buster-arm64.vmdk

    Reply

  4. This method worked perfectly to get the 64Bit Raspberry Pi OS working.

    However I don't suppose there is way to get this method to work with the 32Bit version of Raspberry Pi OS?

    I have a project in mind that this could be very useful for.
    Specifically digital radio hotspots using Pi-Star which is Raspbian based.
    Normally 1 hotspot = 1 Raspberry Pi.

    If I can get it to boot, using USB passthrough on ESXi I could consolidate multiple Raspberry Pi's into a single Raspberry Pi running multiple Pi-Star/Raspbian VM's.

    I have tried these instructions on the 32 Bit Raspberry Pi OS but so far no luck.

    Or is it simply not possible to get a 32Bit OS booting?

    Reply

  5. Is there way to get arm based distributions like manjaro/arch linux who only ships img file to burn on sd card as well? The approach documented here is only applicable for debian based distribution?

    Reply

  6. Thank you very much for this guide William. It worked great!
    I am missing how to create this VM template. It is not so easy to figure that out because there is not a vmdk file for this. Is there any guides for making this template for this special setup with Debian and rPi?
    Thanks

    Reply

    • Once you finish creating the VM. Just right click and under “Template”, you’ll see option to convert the VM to VM template. From there, you can store it and simply clone.

      I don’t know if rPI is OSS and whether it can be redistributed, an OVA would be nice

      Reply

  7. Thanks you for the detailed and extremely helpful instructions. Lots of steps but they all work in the end. You saved me hours.

    Reply

  8. thank you for this guide.

    i have just a question, are you thinking of having vCenter working on arm ?

    Reply

    • It is because you download the 32bit ARMHF version of Raspberry and not the 64bit. I made the same mistake.

      Reply

  10. Any way to enable SSH ? I tried every way I know of
    - put ssh -file in /boot - nothing
    - systemctl enable/start ssh - nothing

    When I try to reach the ssh from a ubuntu machine in my local network I get
    ssh connection to host 192.168.2.xxx port 22: Connection timed out

    Now I'm pretty sure the address is correct since its the address I get via ifconfig -a in the raspiOS

    So what did I miss ?

    Reply

  11. Is there a way to do this for the 32bit version or raspian? There's more software support for that. I will want to run servers on this eventually.

    Reply

  12. Hello and thanks for this great guide!

    After booting from the ISO and running apt update I noticed that in your screenshot apt also gets from both the deb.debian.org and the raspberry repository. In my case it only uses archive.raspberry.org and raspbian.raspberry.org. So linux-image-arm64 and grub-efi-arm64 do not exist

    Any ideas? I did use the 2020-08-20-raspios-buster-armhf-lite image

    Reply

    • I didn't try with the other image, so its possible it won't work. I recommend following the guide as that's what I've been able to verify among others who have also successfully built rPI OS VM

      Reply

      • It is working fine now with the latest version too. I would suppose I had done something wrong but I did try it many times and some other users had the same issues, so something must have changed in the apt update that fixed it. Anyway, everything fine!

        Reply

  13. Hey this is an awesome guide! Now, quick question, would something like this help to convert my current RasPi setup into a VM? I'm thinking about creating an image, then exporting it to a vhdx, but can't find a noob-friendly way of doing it =/

    Reply

  14. Hey very cool How to Run, but the Problem is that lsb_release -a now says thats a debian and not a raspberry pi os ! :/ Thats not fine for some softwares

    Reply

  15. Hey, first of all, great guide

    I'm following the guide step by step but ending up with the same result, after removing the ISO and booting the VM the Debian boot hangs at "Loading initial ramdisk"
    Any idea on what I'm doing wrong?

    Reply

  16. Hey, I'm unable to convert the Raspberry Pi image from .img to .vmdk

    qemu-img convert -f raw 2020-05-27-raspios-buster-arm64.img -O vmdk 2020-05-27-raspios-buster-arm64.vmdk

    getting error like qemu-img: Invalid parameter 'vmdk'.

    Please help me to fix it

    Reply

    • Your syntax looks correct. Are you running qemu-img on a Photon VM or some other Linux VM? qemu-img is run on Linux and vmkfstools is run on ESXi

      Reply

  18. I tried this with Kali but keep getting an error when I try to install grub, cannot find efi partition. Not sure what to do at this point. Is it because the boot partition is formatted as fat32 or the efi directory and files are missing?

    Reply

  20. Thank you for this guide. It worked straight with the 2020-08-20-raspios-buster-arm64.iso image. But I wanted to have a smaller system without X11 so I tried the 2020-08-20-raspios-buster-arm64-lite.iso image. But it fails with Kernel panic while booting.
    Right after the Loading Linux message I get a "Kernel Panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)" message.

    Reply

  21. Hi,
    After following this guide and getting it running, I noticed this:

    pi@rpi-03:~ $ systemctl --failed
    UNIT LOAD ACTIVE SUB DESCRIPTION
    rng-tools.service loaded failed failed rng-tools.service
    rpi-eeprom-update.service loaded failed failed Check for Raspberry Pi EEPROM updates

    pi@rpi-03:~ $ systemctl status rpi-eeprom-update.service
    rpi-eeprom-update.service - Check for Raspberry Pi EEPROM updates
    Loaded: loaded (/lib/systemd/system/rpi-eeprom-update.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Sat 2020-12-26 12:53:06 CET; 1 day 9h ago
    Process: 1238 ExecStart=/usr/bin/rpi-eeprom-update -a (code=exited, status=2)
    Main PID: 1238 (code=exited, status=2)

    Dec 26 12:53:06 rpi-03 systemd[1]: Starting Check for Raspberry Pi EEPROM updates...
    Dec 26 12:53:06 rpi-03 rpi-eeprom-update[1238]: VCHI initialization failed
    Dec 26 12:53:06 rpi-03 rpi-eeprom-update[1238]: /usr/bin/rpi-eeprom-update: 311: /usr/bin/rpi-eeprom-update: arithmetic expression: expecting ')': "(0x >> 23) & 1"
    Dec 26 12:53:06 rpi-03 systemd[1]: rpi-eeprom-update.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
    Dec 26 12:53:06 rpi-03 systemd[1]: rpi-eeprom-update.service: Failed with result 'exit-code'.
    Dec 26 12:53:06 rpi-03 systemd[1]: Failed to start Check for Raspberry Pi EEPROM updates.

    It seems that due to this kernel is not getting updated. Is it just me or do all of you have this issue?

    Reply

  22. Thanks for this great post!
    I am struggling at the point when I boot the recovery debian.... As soon as I see the "UI" where I have to select the language, the system freezes... I cannot do/select anything...
    Has anybody had this problem as well?
    Or can anybody share a "final" OVA file with me? That would be really great and much appreciated!

    Reply

  23. I went throught the steps here and now I have uploaded the image to the host, i used scp to do so but now all of my imges on the datastore are greyed out? Any ideas?

    Thanks,
    Michael

    Reply

    • Hello Michael,
      you have uploaded the vmdk file, right? As you are talking about img - files ...

      Reply

      • Hello Chris, I got this done and I have it on the ESXi and yes I did both conversions. Thank you for answering so quick. Now i have the issue below.

        Thank you sir, I do appreciate it.
        P.S. This is so fun trying to figure it out ROFL!

        Reply

  24. Converting the image worked for me as well, however when i chose the debian netinst iso I got a Detect disks and then asking me to choose a disk driver. I chose the ones I thought would work but they didn't work. I do have a screenshot of the choice it's forcing me into if anyone could help me I would greatly appreciate it.

    Thanks Guys,
    Michael

    Reply

  25. Alright, I figured it out it was of course the noob who was making a mistake, I now have this running flawlessly. it is a really good feeling. If any one needs help please let me know I will be glad to help as best I can.

    Thanks everyone.

    Reply

    • Hello Michael,
      can you share the final "untouched" OVA with me please? I get stuck at the screen where I have to select the language (during recovery)... I can offer you an URL to upload the OVA file 🙂
      Thanks!

      Reply

      • So what I mean (as my previous post could be understood wrongly): Can you export the final VM (without any personal settings/data from you) as an OVA? 😉

        Reply

    • @Peter, ESXi supports x86 CPUs. Raspberry Pi uses an ARM processor so ESXi-ARM is required.

      Reply

      • Thanks Dennis, that explains the trouble I had booting. Wasted a couple hours before I realised that probably that was the issue. Should have read the full text :p

        Reply

  27. Would there be an instructions on how after we have successfully built up a image that we have been work on for months to convert it back to a Bootable directly off a raspberry pi? While I can run this project for myself, the fruits of the labor using packer, ansible, ansible to create the VM I want to distribute to a few of my fellow employees. Problem is, I am not sending them the Pi with everything installed and getting ESXI on their Pi;s will be to difficult. So I would like to make this image I made into a bootable SD Card that I could just export to a IMG and have them burn an SD card they have.

    Reply

  28. Hi,

    I am tryinnt ot get it work but I jut stauck at:

    apt install -y linux-image-arm64 grub-efi-arm64

    command I guess they are both obsolete. I tried to install grub2-common instead beau se both packages are unavailable. But in this case it does not install shim-signed package.

    As a result when I run

    grub-ınstall --efı-dırectory=/boot

    ıt says:

    grub-install: error: /ur/lib/grub/arm-boot/modınfo.sh doesn't exist. Please specıfy --target or --directory.

    Reply

    • Opps sorry I noticed I was using 32 bit platform. but still package structure changed in buster. No more linux image. you need to isntall grub2-common and grub-efı. I don't know if it is the same with 64 bit version though.

      Reply

  29. Hey there,
    thanks for this write-up, but somehow i am not able to follow these instructions.
    After booting from the dbian-iso i am trying to do these installations:
    apt install -y linux-image-arm64 grub-efi-arm64

    The linux-image-arm64 can not be located, and the grub-efi-arm64 isn´t installable as well.
    I tried the grub2-common, but this seams not to work:

    error: /usr/lib/grub/arm-uboot/modinfo.sh doesn´t exist.

    Do you have any recommandations? because i am stuck for a while now, and most of the help from the internet just addresses x86 arch.

    Thank you very much in advance.

    Reply

  30. Hi!

    Has someone figured out meanwhile how to export and re-import a working machine in an easy way? I have the problem, that the exported and re-imported machine will not boot from it's harddisk but from PXE only. I got it once working when modifying the ddb.adapterType = "lsilogic" in the vmdk but now it's not working anymore either. Any help is appreciated.

    BR
    Christian

    Reply

    • Same issue and after updated to the latest ESXi version (18175197) and after I spend whole weekend to setup everything from the scratch (ESXI, iSCSI, raspios template, OVA...) to realize, issue is persistent... found finally a workaround, think would be good to share

      deploy your ova template as usual, check for the guest OS "boot into bios-setup" (in VMware host settings).
      start the EFI internal shell, let the script pass

      type in the shell following:
      FS0:\EFI\debian\grubaa64.efi
      to start GRUB and successful boot up the system.

      After this, you need to adjust / repair GRUB / Boot as described here in Step 16 and 17.

      grub-install --efi-directory=/boot
      update-grub

      your guest should boot without any issues after this change.

      I don't know, what caused the issue, I think it is related to:
      https://flings.vmware.com/esxi-arm-edition/comments/19616#comment_19616
      Cyprien described a similar (or nearly the same) workaround, with the same result.

      Reply

  31. Hi guys, debian arm isos are not working on my ESXi on RPi4 environment.
    I stopped at step 8 because when I try to boot from the Debian ISO (I tried debian-10.10.0-arm64-netinst and debian-10.10.0-arm64-DVD-1) the VM doesn't boot and stucks on VMWARE logo if I let the VM find the DVD or it stucks on a pink blinking "_" if I force the start from the DVD on bios.
    I checked that the DVD is connected.
    It happens for every VM I try to create from those Debian ISOs, but not for others OSs like Ubuntu. So it's not dependent from the vmdk I've attached.

    I tried to redownload the ISOs from another mirror, you never know, but same problem.

    Any idea?

    Reply

    • OK I tried 10.09 and it works. Mistery!
      Actually I'm stuck at Step 12 because apparently none of the partition I find on the disk have seem to be good to be mounted and start a recovery shell.
      I'm using the Home Assistant OS based on Rasbian: haos_rpi4-64-6.1.img and it shows me 8 partitions, from SDA1 to SDA8.

      Any idea?

      Reply

  32. when i try to install the bootloader via grub-install --efi-directory=/boot
    iam getting a Warning that EFI Variables are not supported on this system

    Any Ideas?

    Reply

  34. These instructions don't seem to work anymore. There are errors of running out of space and EFI variables not supported. I'm not knowledgeable enough to figure it out.

    Reply

    • As far as I can tell the EFI variables not supported error shows up when you're trying to use rpi os based on debian 10 with net-install cd for debian 11.
      I am now trying to use the latest rpi os based on debian 11 with debian-11.1.0-arm64-netinst.iso but apparently there is no linux-image-arm64 or grub-efi-arm64 package when trying to apt install these after booting into rescue mode...investigation continues

      Reply

  37. I did all the steps but after reboot VM I land in bios screen.
    What change since 2022 that does not allow to boot properly?

    Reply

  38. I noticed in the latest 2022-04-04 that the disk image is too small and EFI vars are missing, so before you boot into the rescue mode, you need to expand the size of the drive, then when you're in rescue mode you need to do the following before you begin to do apt update:
    raspi-conf --expand-rootfs
    resize2fs /dev/sda2
    mount -t efivarfs efivarfs /sys/firmware/efi/efivars

    Reply

  39. Has anyone been able to get display resolutions working? I'm stuck with just a default 1024x768. Raspi-config doesn't show any resolutions available. Running scrreenfresh says No X Server found.

    Reply

  40. I have been running for over 2 years. It has just worked great. Now my VM is not booting, and crashes during boot with a kernel panic.
    So I am trying to make a new VM with a new 2023 image. I get stuck on step 15 apt install -y linux-image-arm64 grub-efi-arm64
    Progress gets to 20%
    It states cpio: write error: No space left on device

    I notice also that /dev/sda2 is used 100%
    I do not get why we are deleting the hard drive and adding the image as the only disk.
    I am using 2023-02-21-raspios-buster-arm64.vmdk
    Any help is appreciated.
    Thanks

    Reply

    • Hi, i had the same problem (used the raspios-arm64-lite) - could be the topic because i used the lite version.

      But anyway.
      This worked for me:

      1) Removed the vmdk from the virtual machine.

      2) Extended the vmdk via the ssh on the ESXi host:
      # vmkfstools -X 16G -d eagerzeroedthick vm.vmdk
      Used this description: https://kb.vmware.com/s/article/1004047

      2) Added the vmdk again and started in rescue mode as described and run cfdisk
      # cfdisk

      3) Now you should see the free disk space and you can expand the /dev/sda2
      (do not forget to write the changes)

      4) Run resizefs
      # resize2fs /dev/sda2

      5) Now you can check if it works with
      # df -h

      6) Run # apt install -y linux-image-arm64 grub-efi-arm64 again.
      Could be that you have to type in # dpkg --configure -a
      first.

      Best Regards from Austria,
      Timo

      Reply

  41. I've been running several RPi OS as VM's on multiple RPi ESX host. Today I noticed my VM's are running v5.10 kernel but my physical RPi are running v6.1. Is there a way to upgrade the VM to v6.1 kernel?

    Reply

Leave a Reply to BrunoCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.