For customers who are running Mac OS X 10.9 (Mavericks) or newer in a Virtual Machine, you may have noticed that you can no longer set a custom display resolution beyond the default 1024x768 in either VMware Fusion and vSphere, regardless of the amount of video memory that has been allocated. The reason for this behavior is that Apple has changed the way in which it remembers previously used modes and would automatically fall back to this versus retaining the custom mode using the Display Preferences. Given this is a non-trivial fix, VMware Engineering has been working hard on a providing a workaround which would still allow users to set a custom resolution from within the GuestOS.

The workaround that has been developed is a tiny standalone command-line utility called vmware-resolutionSet which runs within the Mac OS X Guest and allows you to configure a custom display resolution. You will need to ensure you have VMware Tools installed and running before you can use this utility. As of right now, customers can get a hold of this utility by filing an SR with VMware Support and referencing PR 1385761. Although this tool has not been officially released and must go through the standard release process, the plan is to include it in a future update of VMware Tools and will available for use with both VMware Fusion and vSphere.

UPDATE (12/11/15) - Thanks to reader @elvisizer, it looks like the latest VMware Fusion 8.1 release now includes an updated version of VMware Tools (10.0.5) which includes the vmware-resolutionSet utility. You can find it under '/Library/Application Support/VMware Tools'. One thing to note is that there is a known issue right now for VMware Fusion 8.1 related to NAT and port forwarding, you may want to hold off on upgrading if you rely on this feature.

The syntax for the vmware-resolutionSet utility is pretty straight forward, it accepts a width and height argument. Make sure to use "sudo" if you want the display resolution to persist through a system reboot. For example, to set a 1920x1080 resolution, you would run the following command:

./vmware-resolutionSet 1920 1080

Note: Ensure you have sufficient video memory configured for your VM for larger display resolutions. In the example above, I have 16MB configured for my Mac OS X VM which would give you a max resolution of 2560x1600. 

If everything was successful, you should see that both the "Requested resolution" and the "Effective resolution" match in the output. If output does not match, it most likely means you need to increase the video memory and you can refer to this VMware KB 1003 for more details. If we take a look at our Mac OS X VM, we should now see that our new custom display has taken effect. Below is a screenshot of a Mac OS X 10.11 (El Capitan) running on vSphere 6.0 Update 1 configured with a 1920x1080 resolution.

One other thing to note is if you plan on using higher display resolution than 2560x1600, you may need to configure some additional VM Advanced Settings due to use of framebuffers that are larger than 16MB. In this case, you would need to also add the following two advanced settings to the VM which can be done using the vSphere Web/C# Client or the vSphere API. For example, if I want a 2880x1800 display resolution, I would add the following:

svga.maxWidth = "2880"
svga.maxHeight = "1800"

Lastly, I would like to give a big thanks to Michael Udaltsov, the Engineer who is responsible for creating the workaround and providing me with some additional context to this change in behavior. I know our customers will greatly appreciate this workaround!