WilliamLam.com

  • About
    • About
    • Privacy
  • VMware Cloud Foundation
  • VKS
  • Homelab
    • Resources
    • Nested Virtualization
  • VMware Nostalgia
  • Apple

How to Persist Configuration Changes in ESXi 4.x/5.x Part 1

08.08.2011 by William Lam // 12 Comments

The question of persisting configuration changes in ESXi and the expected behavior using the ESXi Shell is a question that comes up quite frequently. I think this is primarily due to user's experience with the classic ESX Service Console and the expectation that ESXi will behave exactly the same.

First off, any changes made to ESXi 4.x or 5.x using the vSphere Client, any of the vSphere SDK toolkits (vCLI, PowerCLI, etc) or going through vCenter will preserve the changes across system reboots. The caveat is when going directly into the ESXi Shell (formally TSM) and manually editing or adding files to the default ESXi installation, where the persistence of the files will vary depending on what was updated.

So why does editing files in ESXi filesystem not persisted across system reboots? The reason for this is ESXi uses an in-memory filesystem. For more details, check out The Architecture of VMware ESXi whitepaper by VMware. As I mentioned earlier, the persistence of files will depend on what was updated as there are exceptions to the rules.

There are certain configuration files (e.g. /etc/inetd.conf) that are automatically backed up through a cronjob which looks for particular files under /etc that have been marked with the stickybit. A user can not manually mark a file with the stickybit and have it automatically backed up, it requires one additional file which is implemented by the VisorFS. ESXi creates a copy of these stickybit files and renames the original as .#filename. The backup process will then look for any .#* files and back those up. Due to this special permission mechanism, you can not manually create/touch files with this format as explained by a VMware employee on this VMTN thread.

UPDATE (08/14/2014) - Take a look at this article for more details regarding the various times an automatic backup is performed by ESXi

The "automatic" backup process occurs in the following scenarios:

1) /sbin/auto-backup.sh runs via cron the first minute of every hour and performs a "find" operation under /etc looking for files that being with ".#":

~ # cat /var/spool/cron/crontabs/root
#syntax : minute hour day month dayofweek command
01 01 * * * /sbin/tmpwatch.sh
01 * * * * /sbin/auto-backup.sh #first minute of every hour (run every hour)

Here is an example of the files backed up in ESXi 4.x:

~ # find etc -follow -type f -name ".#*"
etc/dropbear/.#dropbear_dss_host_key
etc/dropbear/.#dropbear_rsa_host_key
etc/security/.#access.conf
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.crt
etc/vmware/ssl/.#rui.key
etc/vmware/.#dvsdata.db
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/.#hosts
etc/.#inetd.conf
etc/.#rc.local
etc/.#chkconfig.db
etc/.#random-seed
etc/.#resolv.conf
etc/.#syslog.conf
etc/.#shadow
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx

Here is an example of the files backed up in ESXi 5.x:

~ # find etc -follow -type f -name ".#*"
etc/vmware/vpxa/.#vpxa.cfg
etc/vmware/hostd/.#hostsvc.xml
etc/vmware/hostd/.#pools.xml
etc/vmware/hostd/.#vmAutoStart.xml
etc/vmware/ssl/.#rui.key
etc/vmware/ssl/.#rui.crt
etc/vmware/.#config
etc/vmware/.#esx.conf
etc/vmware/.#license.cfg
etc/vmware/.#locker.conf
etc/vmware/.#snmp.xml
etc/vmware/.#dvsdata.db
etc/sfcb/repository/root/interop/.#cim_indicationsubscription.idx
etc/sfcb/repository/root/interop/.#cim_listenerdestinationcimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationhandlercimxml.idx
etc/sfcb/repository/root/interop/.#cim_indicationfilter.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile.idx
etc/sfcb/repository/root/interop/.#sfcb_registeredprofile
etc/sfcb/repository/root/config/.#omc_namespaceconfig.idx
etc/sfcb/repository/root/config/.#omc_namespaceconfig
etc/sfcb/repository/root/config/.#omc_config.idx
etc/sfcb/repository/root/config/.#omc_config
etc/sfcb/.#uuid
etc/sfcb/.#sfcb.cfg
etc/ssh/.#ssh_host_rsa_key
etc/ssh/.#ssh_host_rsa_key.pub
etc/ssh/keys-root/.#authorized_keys
etc/ssh/.#ssh_host_dsa_key
etc/ssh/.#ssh_host_dsa_key.pub
etc/vmsyslog.conf.d/.#vmkernel.conf
etc/vmsyslog.conf.d/.#hostd.conf
etc/vmsyslog.conf.d/.#fdm.conf
etc/vmsyslog.conf.d/.#vpxa.conf
etc/security/.#access.conf
etc/.#chkconfig.db
etc/.#hosts
etc/.#ntp.conf
etc/.#random-seed
etc/.#resolv.conf
etc/.#vmsyslog.conf
etc/.#shadow

2) There is also another "internal" backup process that runs every 10 minutes which is not configurable that backups the ESXi filesystem. In the worse case, you may lose changes made in the last 10 minutes if you had a system crash and it reboots.

With the latest release of ESXi 5, ESXi Shell (previous known as unsupported mode and Tech Support Mode) is officially supported for troubleshooting purposes, modifying or adding files is still not officially supported. Though if you need to make changes or add custom files to either ESXi 4.x or 5.x, there are a few options to persist the changes.

Option #1 - If you edit/modify any of the files listed above, these are automatically backed up by ESXi, it's generally recommend you manually run /sbin/auto-backup.sh to ensure they are backed up after your changes.

Option #2 - Edit/Modify files using /etc/rc.local file which executes at the very end of the bootup process. You can make your file changes in this script and they will automatically execute once the system has ran through all the default start up scripts. There two different methods depending on the complexity of your modifications, you may choose to use something like a here-document to create new/modified files or copy configuration files which have been persisted through a local or remote VMFS/NFS datastore.

Here is an example of setting up SSH keys for ESXi and manually creating the .ssh directory and SSH public keys:

Here is an example of copying a script (ghettoVCB.sh) from persistent local datastore to /bin directory:

Once you have made your changes, you will need to manually run /sbin/auto-backup.sh which ensures the changes made to /etc/rc.local will be persisted upon the next system reboot.

Option #2 provides for the most flexibility and allows you to make various configuration changes or addition of files and can easily be integrated into kickstart installation. The entries would be part of your %firstboot stanza to generate the entries in /etc/rc.local file.

In part 2 of this article, I will go over another method that can be leveraged in both ESXi 4.x and 5.x where custom files can be added and persisted as part of the default installation process using kickstart or through a one time manual configuration. I will provide some examples including persisting custom firewall rules in ESXi 5, so stay tuned!

Categories // Uncategorized Tags // ESXi 4.1, ESXi 5.0, vSphere 4.1, vSphere 5.0

How to Disable a vmnic in ESX(i)

06.24.2011 by William Lam // 8 Comments

There was another interesting thread today on the VMTN community forums about disabling an unused vmnic from an ESXi host due to false alarms being generated from HP SIM. Due to the specific hardware version, disabling the vmnic through the BIOS was not an option and there were no alternatives from the discussion.

I decided to dig a bit to see what I could find and stumbled upon a neat little utility called vmkchdev (VMkernel Change Device?) and from what I can tell provides a method passing a particular device to be controlled by either the VMkernel or as a passthrough device to a virtual machine (think VMDirect Path).

Disclaimer: Please note, this is using an undocumented utility. You should test this out in a development/lab environment before using and you may want to also contact VMware support to get their blessings***

~ # vmkchdev
Usage:
-s (scan device)
-v (give device to vmkernel)
-p (give device to passthru/VM)
-l (list device state)
-L (list device state with details)
[0x][seg:[bus[:slot[.func]]]]

What I found while testing the utility is by passing it over as a passthrough device, the vmnic is actually unrepresented to the VMkernel and does not show up under network adapters or even the unused/unlinked adapter list in the vSwitch configurations. It seems that it is just masks the device away from the VMkernel as you can still see the active configuration in esx.conf and you can see the device listed using vmkchdev.

Here is an example of an unused physical nic vmnic1 that we would like to disable and unpresent to an ESXi host.

Here is the output from esxcfg-nics:

First we need to identify the vmnic's PCI slot, we do by running the "-l" or list operation and searching for the particular vmnice device.

~ # vmkchdev -l | grep vmnic1
000:002:01.0 8086:100f 15ad:0750 vmkernel vmnic1

Next we will pass the device from the VMkernel to passthrough/VM using the "-p" flag and specifying the PCI slot, which in this case it is 000:002:01.0. We will also need to refresh the network section so the changes are reflected in the vSphere Client by using vim-cmd.

~ # vmkchdev -p 000:002:01.0
~ # vim-cmd hostsvc/net/refresh

If we list our vmnic again using vmkchdev, you will notice the device is now owned by passthru versus the VMkernel.

~ # vmkchdev -l | grep vmnic1
000:002:01.0 8086:100f 15ad:0750 passthru vmnic1

Now if we check the output of esxcfg-nics and the vSphere Client, you will notice that vmnic1 is no where to be found

If you would like to enable or re-present the disabled vmnic, you just need to pass the device back over to VMkernel by using the "-v" flag.

~ # vmkchdev -v 000:002:01.0
~ # vim-cmd hostsvc/net/refresh

Your vmnic should now re-appear on all your screens and any existing NIC teams that may have exists is automatically restored. This trick actually works on both a used and unused vmnic

If you are trying to do this on ESX, vmkchdev actually has an additional option called "console" for the Service Console.

[root@esx41-1 ~]# vmkchdev
Usage:
-s (scan device)
-c (give device to console)
-v (give device to vmkernel)
-p (give device to passthru/VM)
-l (list device state)
-L (list device state with details)
[0x][seg:[bus[:slot[.func]]]]

I found that you need to pass the vmnic from VMkernel to Console, passing it to passthru/VM will not work and an error is thrown if you do. Again, you can easily re-enable by passing it back to the VMkernel

vmkchdev -c 000:002:01.0

If you would like to automatically persist this change across reboots, specifically for ESXi as changes are not saved. You will need to add the following lines to /etc/rc.local which will execute the disabling of the vmnic's after bootup.

/sbin/vmkchdev -p 000:002:01.0
vim-cmd hostsvc/net/refresh

You will also need to run /sbin/auto-backup.sh to ensure the changes to /etc/rc.local are saved and reloaded upon the next reboot. For ESX, you can place it in /etc/rc.local without having to do anything extra as the changes persists across reboots for classic ESX

Now you can play hide and seek with your vmnic's without resorting to a system reboot or touching the BIOS. Though ideally, if you do have unused devices, you should definitely disable them in the BIOS if you have the option. On a side note, this might be a fun trick to play on one of your co-workers by hiding all vmnics 😉

Categories // ESXi, Not Supported Tags // ESX 4.0, ESXi 4.1, vmkchdev, vmkdevmgr, vSphere 4.1

How to Create Custom Roles on Standalone ESX(i) Host

06.22.2011 by William Lam // 9 Comments

There was an interesting question on the VMTN forums that caught my eye this week in which a user posted a question about controlling access to a limited set of virtual machines for users on a standalone ESX(i) host. For those of you who are familiar with vCenter Server know that this can easily be accomplished by using/creating roles which allows for granular access to various resources within a vSphere environment such as Datacenter, Resource Pool, Host, Virtual Machine, etc. This was the suggestion from several community members and it ended up being the simple solution as the user actually had a vCenter Server.

So what if the user did not have a vCenter Server and actually had standalone ESX(i) host, would he/she be out of luck? The answer is no, you can actually utilize roles on a standalone host but the default roles that are available are pretty limited in scope. By default, there are three roles: No access, Read-only and Administrator and these roles can be applied to both local and/or Active Directory users if your ESX(i) host is joined to an AD domain.

By leveraging our good friend vimsh/vim-cmd from either the classic ESX Service Console or Tech Support Mode of ESXi, we can actually create custom roles just like we would on vCenter Server with fine grain permissions to a virtual machine(s). You can not create custom roles using the vSphere Client like you would when connecting to a vCenter Server.

We will be using the commands found under vim-cmd vimsvc/auth:

[root@vesx41-1 ~]# vim-cmd vimsvc/auth
Commands available under vimsvc/auth/:
entity_permission_add lockdown_mode_enter role_permissions
entity_permission_remove lockdown_mode_exit role_remove
entity_permissions permissions roles
lockdown_is_enabled privileges
lockdown_is_possible role_add

Before we get started, it is good to understand what is a permission and how that relates to a particular entity such as a virtual machine. Here are some definitions from VMware:

Privilege - The ability to perform a specific action or read a specific property
Role - A collection of privileges. Roles provide a way to aggregate all the individual privileges that are required to perform a higher-level task, such as administer a virtual machine
Object - An entity upon which actions are performed. VirtualCenter objects are datacenters, folders, resource pools, clusters, hosts, and virtual machines

Permission = User/Group + Role + Object 

Basically we will be creating a custom role which contains a specific set of privileges to administrator a virtual machine and assign this role to a set of users on a standalone ESX(i) host.

To view all available privileges, you can run the following command:

~ # vim-cmd vimsvc/auth/privileges | grep privId | sed 's/^[ \t]*//;s/[ \t]*$//;s/,//;s/"//g' | awk '{print $3}' | sort
Alarm.Acknowledge
Alarm.Create
Alarm.Delete
Alarm.DisableActions
Alarm.Edit
Alarm.SetStatus
Authorization.ModifyPermissions
Authorization.ModifyRoles
Authorization.ReassignRolePermissions
DVPortgroup.Create
DVPortgroup.Delete
DVPortgroup.Modify
DVPortgroup.PolicyOp
DVPortgroup.ScopeOp
DVSwitch.Create
DVSwitch.Delete
DVSwitch.HostOp
.....

To view existing roles and what privileges are associated with a given role, you can run the following command:

vim-cmd vimsvc/auth/roles | less

From the above privilege list, you will need to select the privileges that you would like in your new role, for now you can just copy the entries into a temporarily file (/tmp/vm-privileges) as that will be used later.

In this example, I've selected the basic VM privileges that is associated with the "Virtual Machine User" found in vCenter Server but adding on additional privileges to support snapshots.

[root@vesx41-1 ~]# cat /tmp/vm-privileges
Global.CancelTask
VirtualMachine.Interact.AnswerQuestion
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.SetCDMedia
VirtualMachine.Interact.SetFloppyMedia
VirtualMachine.Interact.ConsoleInteract
VirtualMachine.Interact.DeviceConnection
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Interact.Reset
VirtualMachine.Interact.Suspend
VirtualMachine.Interact.ToolsInstall
VirtualMachine.State.CreateSnapshot
VirtualMachine.State.RemoveSnapshot
VirtualMachine.State.RevertToSnapshot

Next we will need to know the virtual machine's VmId also known as the MoRefID (Managed Object ID) that uniquely identifies a given entity. To display the VmId of all virtual machines residing on your ESX(i) host, you can run the following command:
[root@vesx41-1 ~]# vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
32 SteveJablonsky [datastore1] SteveJablonsky/SteveJablonsky.vmx freebsdGuest vmx-07
48 HansZimmer [datastore1] HansZimmer/HansZimmer.vmx dosGuest vmx-07
64 JamesNewton [datastore1] JamesNewton/JamesNewton.vmx dosGuest vmx-07
80 BrianTyler [datastore1] BrianTyler/BrianTyler.vmx dosGuest vmx-07

Next we need to identify the users in which we will create the new permissions for. In this example, I will use both a local (william) and Active Directory user (PRIMP-IND\primp):

[root@vesx41-1 ~]# id william
uid=501(william) gid=100(users) groups=100(users)

[root@vesx41-1 ~]# id PRIMP-IND\\primp
uid=37225554(PRIMP-IND\primp) gid=37224960(PRIMP-IND\domain^admins) groups=37224960(PRIMP-IND\domain^admins),37225020(PRIMP-IND\denied^rodc^password^replication^group)

In this example, I have a single ESX host that contains 4 virtual machines and the first two will only be visible to a user called "william" and the second two will only be visible to a user called "PRIMP-IND\primp":

  •  Owner: william
    • HansZimmer
    • BrianTyler
  •  Owner: PRIMP-IND\primp
    • SteveJablonsky
    • JamesNewton

We will create a virtual machine administrator role called "VMAdmin" which will then be associated with their respective virtual machines and system owners.

First we need to create the custom "VMAdmin" role by using the file that contains all the necessary privileges, we will run the following command:

[root@vesx41-1 ~]# vim-cmd vimsvc/auth/role_add VMAdmin $(cat /tmp/vm-privileges | awk '$1=$1' ORS=' ')

Note: The "role_add" operation takes in a roleName and then a list of privileges which are separated out by a space, this list can be as many as you need and by specifying it in a file, it makes it easier to edit/adjust

You can verify that the new role was created running the following command:

[root@vesx41-1 ~]# vim-cmd vimsvc/auth/roles
......
(vim.AuthorizationManager.Role) {
dynamicType = ,
roleId = 12,
system = false,
name = "VMAdmin",
info = (vim.Description) {
dynamicType = ,
label = "VMAdmin",
summary = "VMAdmin",
},
privilege = (string) [
"Global.CancelTask",
"System.Anonymous",
"System.Read",
"System.View",
"VirtualMachine.Interact.AnswerQuestion",
"VirtualMachine.Interact.ConsoleInteract",
"VirtualMachine.Interact.DeviceConnection",
"VirtualMachine.Interact.PowerOff",
"VirtualMachine.Interact.PowerOn",
"VirtualMachine.Interact.Reset",
"VirtualMachine.Interact.SetCDMedia",
"VirtualMachine.Interact.SetFloppyMedia",
"VirtualMachine.Interact.Suspend",
"VirtualMachine.Interact.ToolsInstall",
"VirtualMachine.State.CreateSnapshot",
"VirtualMachine.State.RemoveSnapshot",
"VirtualMachine.State.RevertToSnapshot"
],
}
]

Note: If you created a role that did not contain all the privileges and you need to update the role, you actually have to delete the role first. I have not found a way to "append" privileges, but removing a role is very straight forward and you can use "vim-cmd vimsvc/auth/role_remove" to do so.

Next we will associate this new role with the users and the virtual machines to create the permission mappings. If you recall earlier, we retrieved a list of virtual machines and the first column contains the VM's VmId which will be needed in this next section. We will be using the "vim-cmd vimsvc/auth/entity_permission_add" operation and if you would like to know what arguments it accept, you can just run it by itself without any arguments.To associate the first two virtual machines to the user "william" we will be running the following commands:

[root@vesx41-1 ~]# vim-cmd vimsvc/auth/entity_permission_add vim.VirtualMachine:80 william false VMAdmin true
[root@vesx41-1 ~]# vim-cmd vimsvc/auth/entity_permission_add vim.VirtualMachine:48 william false VMAdmin true

To associate the new two virtual machines with the user "PRIMP-IND\primp" we will be running the following commands:

[root@vesx41-1 ~]# vim-cmd vimsvc/auth/entity_permission_add vim.VirtualMachine:64 PRIMP-IND\\primp false VMAdmin true
[root@vesx41-1 ~]# vim-cmd vimsvc/auth/entity_permission_add vim.VirtualMachine:32 PRIMP-IND\\primp false VMAdmin true

Note: The VM entity syntax is in the form of vim.VirtualMachine:XX where XX is the virtual machine's VmId extracted earlier which will need to be substituted in with your own VmId

We can verify the permissions by running a few commands, we can either check all permissions created with this particular role name, in our example, it is "VMAdmin" and the command to run would be the following:

[root@vesx41-1 ~]# vim-cmd vimsvc/auth/role_permissions VMAdmin
(vim.AuthorizationManager.Permission) [
(vim.AuthorizationManager.Permission) {
dynamicType = ,
entity = 'vim.VirtualMachine:32',
principal = "PRIMP-IND\primp",
group = false,
roleId = 12,
propagate = true,
},
(vim.AuthorizationManager.Permission) {
dynamicType = ,
entity = 'vim.VirtualMachine:48',
principal = "william",
group = false,
roleId = 12,
propagate = true,
},
(vim.AuthorizationManager.Permission) {
dynamicType = ,
entity = 'vim.VirtualMachine:64',
principal = "PRIMP-IND\primp",
group = false,
roleId = 12,
propagate = true,
},
(vim.AuthorizationManager.Permission) {
dynamicType = ,
entity = 'vim.VirtualMachine:80',
principal = "william",
group = false,
roleId = 12,
propagate = true,
}
]

You can also check a specific entity such as a virtual machine by running the following command:

[root@vesx41-1 ~]# vim-cmd vimsvc/auth/entity_permissions vim.VirtualMachine:64
(vim.AuthorizationManager.Permission) [
(vim.AuthorizationManager.Permission) {
dynamicType = ,
entity = 'vim.VirtualMachine:64',
principal = "PRIMP-IND\primp",
group = false,
roleId = 12,
propagate = true,
}
]

Now that we have confirmed the new permissions, we will want to verify that these users can only see the virtual machines we have specified both in the vSphere Client and webAccess (ESX only).

Here is a screenshot of user "william" connecting to the vSphere Client:

Here is a screenshot of user "PRIMP-IND\primp"connecting to the ESX webAccess:

Contrary to the default permissions on a stand alone host, you actually can reate custom roles just like on a vCenter Server to have fine grain access controls for your various users without having to give them full administrative privledges to an ESX(i) host. If you have vCenter Server, then of course that is the recommended approach but if you do not, you still have a way 🙂

Categories // Uncategorized Tags // ESX 4.0, ESXi 4.1, privilege, role, vim-cmd, vimsh

  • « Previous Page
  • 1
  • …
  • 3
  • 4
  • 5
  • 6
  • 7
  • …
  • 12
  • Next Page »

Search

Thank Author

Author

William is Distinguished Platform Engineering Architect in the VMware Cloud Foundation (VCF) Division at Broadcom. His primary focus is helping customers and partners build, run and operate a modern Private Cloud using the VMware Cloud Foundation (VCF) platform.

Connect

  • Bluesky
  • Email
  • GitHub
  • LinkedIn
  • Mastodon
  • Reddit
  • RSS
  • Twitter
  • Vimeo

Recent

  • Programmatically accessing the Broadcom Compatibility Guide (BCG) 05/06/2025
  • Quick Tip - Validating Broadcom Download Token  05/01/2025
  • Supported chipsets for the USB Network Native Driver for ESXi Fling 04/23/2025
  • vCenter Identity Federation with Authelia 04/16/2025
  • vCenter Server Identity Federation with Kanidm 04/10/2025

Advertisment

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy

Copyright WilliamLam.com © 2025