CBRC (Content-Based Read Cache) is another new/hidden feature of vSphere 5, not to be confused with the new Host Cache feature (swap to host cache). I initially thought CBRC was related to host cache and that it might have been an internal name for the feature. In a recent discussion on the VMTN community forums, a reader pointed out that CBRC is different and there is not whole lot of information on how it works. I decided to perform a quick Google search to see if anyone has written about this feature and one site had an interesting quote from a VMware sales rep on how CBRC works:
Content-Based Read Cache. A content-based read cache (CBRC) has been delivered for specific use with View (VDI) workloads. With this option configured in ESX, a read cache is constructed in memory optimized for recognizing, handling, and deduplicating VDI client images. The cache is managed from within the View Composer and delivers a significant reduction, as high as 90% by early estimates, in IOPS from each ESX host to the storage platform holding client images. This reduction in IOPS enables large scaling of the number of clients in case multiple I/O storms, typical in large VDI deployments, occur.
It looks like CBRC is implemented within hypervisor but it will be leveraged by VMware View 5 around provisioning Linked Clones? One question I had was whether or not this constructed cache in memory could be used without VMware View?
There are two new sections under the Advanced Settings of an ESXi 5 host, CBRC and Digest.
There are 4 configurable options to enable CBRC, to configure the cache size, to configure the cache size reservation and interval for digest journal.
I believe the Digest section is related with respect to the algorithms used for CBRC
CBRC looks to only support a maximum of 2GB of memory and default of 400MB reservation. In this example, I have a brand new ESXi 5 host with 8GB configured and without any running virtual machines, here is how much memory it is using.
Now let's go ahead and change the CBRC reserved memory from 400MB to 2048GB (max) and enable CBRC.
If we go back to ESXi summary page, we'll see that an additional 2GB is now being reserved by the ESXi host.
You can also enable and configure CBRC using the CLI, but the ability to enable is only available when using vim-cmd interface in ESXi Shell, the other three options can be configured using legacy esxcfg-advcfg, esxcli or vsish.
Here's an example of enabling CBRC and changing the CBRC memory reservation to 1GB:
vim-cmd hostsvc/advopt/update CBRC.Enable bool true
vim-cmd hostsvc/advopt/update CBRC.DCacheMemReserved long 1024
Ifyou are interested in other advanced settings for CBRC that are not publicly exposed, be sure to check out this post here.
Note: Something I noticed about CBRC is there is an admission control check when you initially enable the feature. If you do not have sufficient memory to pass the admission control check, you will get a very generic error. To see if it is related to admission control by looking in vmkernel.log, you may see a message such as the following: WARNING: cbrc_filter: CBRC_MemSetMemAllocation:1420:Failed to set memory resource parameters for CBRC (Admission check failed for memory resource)
So now that CBRC is enabled, how do we actually use the feature? Well, as I mentioned earlier in the post, this is both a new and hidden feature. Hidden in the sense, that it's not meant to be used directly on an ESXi or vCenter Server but by actually by View Composer. Though you can still access the hidden APIs using the vSphere MOB connected to either vCenter Server or ESXi host.
To get to the new CBRC managed object manager, you can point your browser over to following URL:
You will notice a new managed object manager "cbrcManager", go ahead and click on it.
There are 4 methods associated with configuring CBRC:
Note: A shortcut to getting to the CBRC managed object manager is using the following URL:
ESX(i) - https://[hostname]/mob/?moid=ha-cbrc-manager
vCenter - https://[hostname]/mob/?moid=CbrcManager
The ConfigureDigest_Task is what is needed to configure CBRC for a given virtual machine and specifically a virtual disk. The parameters that are needed is both the managed object reference ID of the virtual machine and the virtual disk device ID.
If we browse over to https://[hostname]/mob/?moid=ha-host, we'll be able to identify the virtual machine's MoRef ID and in this example, it's 1
Next we'll need to identify the virtual disk device ID, by traversing the virtual hardware array until we find the virtual disk we're interested in.
The ID for this virtual disk is 3000 and you will also notice a new property called digestEnabled, currently it is disabled.
Now we have everything we need to construct ConfigureDigest_Task method, click on the method name and it will open a new page for you to specify the virtual machine ID and the deviceKey for the virtual disk. You will see the next option is to enable CBRC, I'm not exactly sure what the latter two options but I'll go ahead and enable them anyways.
Once the task has completed, you can view the results and you will see that CBRC has successfully been configured for the virtual machine:
You can also see that the task has been kicked off within the vSphere Client depending if you're connecting to ESXi 5 directly or to vCenter Server:
If we go back to the virtual disk, we will see that digestEnable property is now enabled:
One interesting observation I found was when you enable CBRC, if you browse the datastore of the virtual machine, you will see a new VMDK created with name "digest", this is probably what keeps track of the blocks from the virtual machine and what is loaded into memory.
We can also use the QueryDigestRuntimeStatus to check whether a virtual machine is enabled with CBRC, again specifying the virtual machine's MoRef ID and virtual disk device ID:
We can use QueryDigestInfo on an offline virtual machine to get the details of the digest information for a virtual machine with CBRC enabled:
The last method RecomputeDigest_Task should be self explanatory and it allows you to compute partial or full digest.
By enabling CBRC, the CBRC kernel module is loaded and with that, there are some interesting statistics that can be viewed for a given virtual machine. We'll be leveraging the VMware vsish interface to access the CBRC statistics. To get started, just type vsish and then you will need to change into vmkModules/cbrc_filter path.
One interesting property is the dcacheStats, you can just cat this entry and it provides an enormous amount of statistics about the cache including the number of virtual machines that are using the cache and various IO counters.
You will see that all counters are currently zero, once we start to spin up some Linked Clone virtual machines, you will want to pay close attention to some of these counters.
To answer the question on whether or not CBRC would work outside of VMware View 5, I decided to perform a functional test of the feature. I generated my own VMware Linked Clones using the following vSphere SDK for Perl script vGhettoLinkedClone.pl. You will need a vCenter Server, but you will NOT need VMware View.
Step 1 - Create an offline snapshot for the virtual machine that will be used as the base/golden image to create Linked Clones, in this example, the snapshot will be called "base"
Step 2 - Create several Linked Clones based off of this base/golden virtual machine, in this example, I will be creating three Linked Clone virtual machine named: ALinkedCloneVM1,ALinkedCloneVM2,ALinkedCloneVM3
If we go over to your vCenter Server, you should see the following inventory:
Step 3 - Let's power on the first two virtual machine "ALinkedCloneVM1" and "ALinkedCloneVM2" and let's check out the dcacheStats from vsish and see what has changed.
As you can see we now have counters incrementing on the number of VMs using the cache that was created with CBRC and also counters regarding the backend IO. It looks like the new Linked Clones that was generated can in fact leverage CBRC without VMware View 5. Now this was purely a functional test, these VMs were basically dummy shells, I would be very interested to see if someone is able to get this really working and actually leveraging CBRC with real base images and seeing the reduction of IOPS during a VDI boot storm.
Derek Seaman, CISSP, MCITP:EA, VCP4 says
Wow I hope that Xendesktop can utilize this feature and that it really does cut down on IOs by 90%..
Very nice William!
Did you tried with truly cloned vm ?
when i do this on one of my esx host
it finishes the configure the virtual disk digest task, and when it then triese to reconfigure the virtual machine i get an access to resource settings on the host is restricted to the server (vcenter ip) which is managing it
hpw did you get past that since i can see from you screen dumps that you are doing it in a cluster as well.
I'm testing this and posting results on twitter, search for CBRC or just look me up I'm @gunnarwb
Thanks for posting this info William, its very interesting.
Has anyone been able to find where in vCenter one can ConfigureDigest_Task? I'm using vCenter 5 update 1 and can't locate it.
How does one configure the digest task "ConfigureDigest_task" thru the vCenter MOB? I can't seem to find it.
As I've noted in the article, you can connect to either an ESXi or vCenter host and discover the CBRC manager by calling the internalServiceContent() method. I've gone ahead and updated the article with the vCenter path, so you can just plugin the hostname of your vCenter Server.
@William, great article. I connected to vCenter for the ConfigureDigest_task and connected to ESXi host (hosting the VM) to discover the MOID and device id. Is that the right way? I obviously did something wrong because at the end of the steps, the digestEnabled parameter remains 'false'. Any clarification will be helpful. Thanks!
Any chance someone has written a PowerCli script to enable this on multiple VMs? I gave it a shot myself, but I'm no scripting guru and could not get it to work.
Not without public API. The method is not available in the object you get in PowerCLI, that's too bad but i guess you can post the url with the good arguments to enable digest on vmdk. I'll try to.
@Brendan, NiTRo is correct, there is no API, hence you won't be able to access it through PowerCLI or other CLIs. You could of course post to the URL as an alternative
sumedh.. truly terpsichorean says
Great post William, can you kindly give your comments to following questions too?
Like @curditem, I followed the steps mentioned, but digestEnabled field remains false.
Does it work only with linked clones?
What about the VM and vDisk I ConfigureDigest_task for?
Like @Glenn, I hit same issue with reconfigure_digest task being called.
You need to perform the entire task either in vCenter OR directly on ESXi host. It looks like @curditem perform part of it in vCenter and other directly on ESXi host which will not work
same question here, after configureDigest_task has run, it still shows as DigestEnabled = false.
replied too soon, it seems it will only work on a VM that doesn't have snapshot.
I am seeing the same result. The task seems to finish w/o issues, but the disk still shows disabled and I do not see the digest vmdk. I verified no snaphots. Does the VM need to be off? Does the type of virtual disk controller matter?
When I was testing this I could only get it to work under very specific circumstances.
1) The VM needed to be powered off
2) The ESXi host which was used to generate the digest needed to be completely stand alone and not managed by vCentre (I ended up disconnecting one of my hosts from vCentre and was surprised when that worked - I was able to repeat the process and get it to work again using that workaround so I suspect it wasn't just a once off oddity)
I can confirm that. I've got this working on a 5.1 host, but only after removing it from vCenter entirely, and readding it after the digest tasks were performed successfully.
If I attepted while connected to vCenter, it would run through the configure steps, but would always result in DigestEnabled = false
I've been seeing VDI style boot storms when using Site Recovery Manager, I would love to see VMware support the use of CBRC with SRM as well as View.
I am going to have to setup a lab environment to test this and see if it does anything helpful for this use case....
I followed on my original question about SRM - turns out that using array based replication the digestEnabled status is kept on the disk and the destination host (if configured for CBRC) appears to respect the setting.
Unfortunately it also appears that vSphere based replication does not copy the digest and therefore the recovery site doesn't have any data to use with CBRC.
Im unable to browse to the Ha-Host address on Vcenter but i can do this direct to ESX.
but the ha-host does work when going direct to ESX.
I grabbed my Device and Disk ID from there and ran the command still on a connect to the ESX Host.
It fails with unable to configure host after creating the hash and digest file (like others here have mentioned)
I can run the command connecting to the Vcenter as well , but this does not seem to do much i see no command running in the Vcenter task list windows.
Can anyone confirm they have completed this process with Vcenter.