In previous releases of vSphere, scaling up resources such as storage or memory for vCenter Server was a huge pain-point for our customers. Before the various vCenter Server services can consume the new resources, some additional manual steps were required. Though this type of an operation is usually infrequent, there is still an operational overhead which can potentially lead to increased downtime of your vCenter Server.
For example, increasing storage capacity for the VCSA was an offline operation that required adding an additional disk and then copying the existing content to the new disk which can be quite error prone and lead to a significant amount of downtime. In vSphere 6.0, the VCSA now uses LVM which provides the ability for online storage capacity increase without any downtime to vCenter Server. Increasing memory was also challenging because you had to manually adjust several configuration files that manages the JVM heap settings for various vCenter Server services as described in this VMware KB. Having complex workflows to perform basic resource expansion can increase risk of errors, especially when the process is foreign to those performing it for the very first time.
To help solve this problem, in vSphere 6.0 vCenter Server (Windows & VCSA) now includes a built-in dynamic memory reconfiguration process that automatically runs at boot up. This process includes a dynamic algorithm that inspects the current amount of CPU, Memory and Storage that is available to determine the appropriate size to configure the vCenter Server. This means that if you no longer have to tweak individual JVM settings for the various services within vCenter Server, this will happen automatically by analyzing the resources that are available and then calculating the configuration based on the supported maximums for vCenter Server.
Note: In vSphere 6.0, there are additional services going beyond just the core vCenter Server, vSphere Web Client, vCenter SSO and Inventory Services.
The dynamic memory algorithm is configured to understand the minimal amount of resources for running a vCenter Server and is bounded between a "Tiny" configuration which is 2vCPU and 8GB memory and a "Large" configuration which is 16vCPU and 32GB memory. This is important to note because if you try to configure the vCenter Server with less memory than the minimal supported, though the algorithim will dynamically distribute the available memory to the various resources, it could lead to performance degradation as the different services may not be receiving the amount of memory they require to run. YMMV if you decide to reduce the supported amount of memory but the algorithm will distribute what's available.
The process which does all the magic is a utility called cloudvm-ram-size and there are several useful options to be aware of. To view the current memory assignment for the various vCenter Server services including the OS, you can run the following command on the VCSA as an example:
cloudvm-ram-size -l
From the screenshot above, we can see a very simple break down of the current memory assignment for a "Tiny" deployment which has 8GB of memory.
To show that the dynamic memory algorithm is in fact running when more memory is added, the example below is of a VCSA that was initially configured with 8GB of memory. I then capture the running configuration and then shut down the vCenter Server and increased its memory to 10GB. I then power on the VCSA and capture the running state and you can see differences in the screenshot below.
Another useful command to be aware is being able to see the current memory usage for all services. You can do this by running the following command:
cloudvm-ram-size -S
As you can see the dynamic memory algorithm is a very much welcome feature for vCenter Server and will greatly simplify the operational tasks when needing to scale up or down resources such as CPU and Memory. I know this is definitely one of the enhancements I have been waiting for and I am glad to see it here in the new vSphere 6.0 release! As of right now, a system reboot is required but who knows maybe in the future we can increase memory while the VCSA is still running and simply reloading the services ...
Is configuring the JVM settings the same as configuring Tomcat server settings as described in kb.vmware.com/kb/2021302?
Sorry this may sound dumb but I am just confused.
If they are same, Tomcat server settings are meant for number objects a vCenter server is going to handle. How does changing the resource helps in managing the inventory.
Again, thanks for sharing all the tips and tricks. Always get to learn something when I come to your blog.
Yes, it is the same. You would increase CPU/Memory based on the size of the expected inventory which in vSphere 6 is broken down into Tiny, Small, Medium and Large which has a known supported maximum. Depending on where you land on the inventory size, you will want to configure your vCenter Server appropriately. In the past this was manually adjusting each of the services as you've shown in the KB but there's a lot more services and now this is all automatic for you.
I dont understand something. When i was installing vcenter on Windows OS i have had missconfigured memory for that vm. VM has 3TB of memory (what is not truth) After installing vcenter everything was working but when i decrease memory to 12 GB then vcenter vsphere client stop working. If this services are self managed then why i could not decrease memory size of VM ? Can you tell me how to decrease this memory ?
hmm....ok, now I understand. Thanks for the explanation.
Now the most interesting question for me is: How to bring down the vCSA 6.0 memory limit to 4G (for the small lab) as it was possible with 5.5. Because if I understand you correctly, even when I bring down the total memory of the VM to 4GB, it will default to the "tiny" preset, which still uses 8GB and will cause the vCSA to swap a lot. Correct?
You can actually reduce the memory to under the recommended 8GB, I recently found out the algorithm will just distribute the available configured memory so that swapping doesn't occur. What will happen is that you'll impact the services with potential unexpected behavior as 8GB is what has been tested. There's a lot that's changed between VCSA 5.5/6.0, so you definitely won't be able to get it to 4GB (I've tried ;)). The best I've been able to do is ~6-6.5GB and you will notice start time will take longer. Again, there's no guarantee even this amount will work as I only quickly tested it with several Nested ESXi VMs.
Thanks for the quick reply! That is bad news ;). I really only need the vCenter on this particular test server for convenience. It solely runs some nested VM for my small test-lab and I dont want to start the Windows VM on my Mac to use the vSphere Client directly on the ESXi. Also most of the VMs are version 10 or 11, which means administration over the fat client is not possible anymore. And "wasting" 8 GB of RAM is really hard if you dont have that much of it. That will change in the future when I have my own local test lab, but for now that server must suffice (as money does unfortunately not grow on trees ;).
I'm looking at moving to 6 shortly on my home lab, but would rather not commit that much RAM.
Do you know if its possible to force the VSCA VM to swap out to a local SSD datastore, so 4GB 'real' memory could be used with extra available on SSD swap?
sorry, not afaik
If dynamic memory resizing is working in vCenter Server 6.0 therefore no need to manual adjustment with Java heap, how to check the adjusted settings? In kb.vmware.com/kb/2021302, all is explaining file locations in 5.x not 6.0. When I searched I found file locations have changed in 6.0. Could you please explain on how to check adjusted java heap size and how to assure dynamic resizing is working correctly?
Changho,
You can find the details in the blogpost above đŸ™‚ Take a look at cloudvm-ram-size
Thanks, I thought it is just for VCSA not Windows based. But I found it in Windows based too.
Hello, I believe my vCSA was deployed with a "tiny" deployment but the RAM for the VM was later changed to 24GB. when i run the command above it shows total available is 24GB but the memory limit on the right is still 8. How can i increase that limit and the deployment size to medium? Thanks
What do you mean "limit"? Are you referring to the far right hand side, if so, that's in MB so that's not in 8GB. If you've rebooted and the -l command shows 24GB, then it's already using the new amount
Hello William,
do you have any clue why "cloudvm-ram-size.bat -S" is returning zero values for all the coutnters (from MaxMB up tp Memory Limit) only one column which is populated is AllocatedMB. It happens on vanilla installation of vCenter server 6.0U2 for Windows, all services are up and running...
Many thanks
Petr
I have this too...vCenter 6.0.2 is returning all 0 for the -S command.
Any update on this?
I am experiencing the issue described in KB 2144950...I get intermittent alerts on vmware-dataservice-sca and vsphere-client - the alert appear and disappear every few hours. We are using vCenter on Windows here.
So the KB suggest increasing the heap size...but to what value? I am using 8vCPU/16GB RAM with external PSC.
1. What value of MB should I increase the vsphere-client? (current value is 1141)
2. To what value of MB should I increase the vmware-sca? (current value is 128)
3. Should I also increase vmware-vws? (vSphere Web Services?)
Great post! Along the lines of increasing CPU and Memory resources, are there absolute maximums?
Say for example the current Windows vCenter (v5.5, external DB, external SSO) has more CPU than the maximums of the large implementation of vCSA 6.0 - is it possible to exceed the 16 vCPU(large) size and have it automatically recognized and supported?
(for the curious: nightly backup processing is the primary driver of the CPU resource utilization)
Tim, I have the same question. We do not want multiple vCenters and have an environment that also has a lot of backup activity and needs more than 16vCPU and 32 GB or ram. William, is it possible or advisable to oversize a VCSA? what are the maximums?
Does your deployment of VCSA use an embedded or separate database server? Thanks.
How do you do this on Windows?
Thanks
It's the same as the VCSA. You shutdown the VC, increase CPU/Memory and the system will automatically adjust the settings. Its transparent to the user
If I start off with the "Tiny" deployment and now want to go beyond 10 hosts.....How do I change it to "small" deployment? Could I simply add more memory and more CPU and reboot?
Steve,
That's correct. You would size your CPU/Memory based the various "sizes" and their supported VMs. This is exactly what's done when you deploy, so re-sizing is simply increasing your computes
Hi William,
I am new to the Vmware.
I am currently running 2 physical servers, 5 Virtual machines and a embedded vcenter in my lab.
The vcenter host memory always reaches the maximum after a few days and then I cannot login anymore until restart the Vcenter.
Even though I increased the memory to 32G, the problem persists.
Would you suggest what may be wrong, please?
Thanks,
Hi,
If u need to buid a lab and deploy a vcenter in version 6.0x u can still deploy it on a Windows VM hosted on a SSD.
Deploy the VM with 8GB, and proceed with the vCenter installation, after the installation is successfully completed, increase the pagefile to 8GB and reduce the VM memory to 3GB.
vCenter services will take arround 10 min to start but it's pretty stable, using since a year now without any problem, also tested on last Windows 2016 TP and seem fine so far.
What about storage of the vcsa, if you start with tiny and increase to small is the storage dynamically increased from 120 to 150 as well?
Beware of https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146507
The magic ends once vCenter VM RAM gets changed (increased or decreased) and manual intervention is required to adjust the limits of vpxd...
Hi Michael, I'd tested it with VCSA 6.0 U2, and the value of the config.vpxd.memChecker.memStopInMB was changing automatically after server boots.
Hi Willian,
I would like to thank you for the detailed review of the dynamic memory algorithm.
I'd recently run across the issue with the vsphere-client service and used 'cloudvm-ram-size -C XXX vsphere-client | service vsphere-client restart' to adjust the allocated memory.
The good news was the manual change remains after reboots.
My observations can be found in this blog post - https://virtualnomadblog.com/2016/10/17/vcenter-server-6-0-vmware-dataservice-sca-and-vsphere-client-status-change-from-green-to-yellow/.
Good read thanks for sharing. I have a question though what about scaling up the PCS? Are there any configuration tweaks needed if you scale the resources on a PCS? Or are you only able to scale PCS out by adding another instance to share the load?
Great post!! Is there a KB that mentions what the headers of cloudvm-ram-size -S output mean? We're seeing a vCenter show what it looks to me to be oversubscription for invsvc, vpxd and vpostgres. We're seeing vCenter crashes from vpxd not able to talk to the embedded database. I've got a feeling that this oversubscription might be playing a role
Hello everyone!!
I have an ESXi 6.5 with 20GB in RAM and 5 VMs, which VCSA consumes me 10GB
Will they have some type of update to reduce memory usage to VCSA 6.5?
Greetings and thanks in advance!
Richard