A couple of weeks back I was asked by a customer who was interested in subscribing to my 3rd Party Content Library which hosted several of my Nested ESXi and VSAN OVF Templates. The problem was that in his environment, like many others, he did not have direct access internet access from within vCenter Server for the Content Library subscription to be created. The customer was wondering if the Content Library feature supported a proxy server which is a very common method for Enterprise customers to provide access to external sites requiring internet access. The Content Library Service does provide a way to configure a proxy server and below are the instructions for configuring both the VCSA and vCenter Server for Windows.
UPDATE (09/27/17): As of vSphere 6.5 Update 1, the Proxy Configurations for the Content Library has been pulled directly into the service itself and you no longer have to manually edit the Java wrapper.conf files. You can now access the proxy configurations by using the vSphere Web Client going to Administration->System Configuration->Services->Content Library Service->Transfer Service as shown in the screenshot below. For 6.0 and 6.5, you will need to continue to follow the instructions below on editing the wrapper.conf file.
vCenter Server Appliance (VCSA)
The configuration file that you will need to edit is /usr/lib/vmware-vdcs/wrapper/conf/wrapper.conf and below are the three lines to add:
wrapper.java.additional.20=-Dhttps.proxySet=true
wrapper.java.additional.21=-Dhttps.proxyHost=proxy.server.com
wrapper.java.additional.22=-Dhttps.proxyPort=8080
Once you have saved your changes, you will need to restart the Content Library service for the changes to go into effect by running the following command:
/etc/init.d/vmware-vdcs restart
The proxy server will now be used and assuming the proper ACL's have been added on the proxy server itself to allow traffic from your vCenter Server to the appropriate destination site, you should now be able to use the Content Library to subscribe to my 3rd Party Content Library.
vCenter Server for Windows
The configuration file that you will need to edit is C:\Program Files\VMware\vCenter Server\vdcs\wrapper\conf\wrapper.conf and below are the three lines to add:
wrapper.java.additional.20=-Dhttps.proxySet=true
wrapper.java.additional.21=-Dhttps.proxyHost=proxy.server.com
wrapper.java.additional.22=-Dhttps.proxyPort=8080
Once you have saved your changes, you will need to restart the Content Library service for the changes to go into effect by going to the Windows services panel.
So interestingly, this is getting me partway there.. I setup the /etc/sysconfig/proxy file, and I am able to use wget from the shell to download the json file, but the Content Library GUI still doesn't work (HTTP request error: connect timed out)
Are you specifying the URL to the JSON file when creating the Content Library using the vSphere Web Client?
You can also tail the following logs to see what error is being thrown: /var/log/vmware/vdcs/cls.log
Yep, definitely using the Web Client. It works great from a vCenter test machine that doesn't require a proxy..
Here's the snipped from the cls.log
2015-05-28T02:08:50.795Z | INFO | unset-opId | diagnostic-json-timer | JsonDumper | JSON diagnostics logger is not enabled
2015-05-28T02:08:55.778Z | DEBUG | unset-opId | content-library-Scheduler-1 | AutoSyncTask | refreshing automatic sync settings.
2015-05-28T02:08:58.271Z | DEBUG | unset-opId | tomcat-http--30 | HttpStreamingServlet | Received request from agent 'vAPI http client' with content-length 10048, content-type 'application/json' and accept header 'application/vnd.vmware.vapi.framed,application/json'
2015-05-28T02:08:58.272Z | DEBUG | unset-opId | tomcat-http--30 | JsonSignatureVerificationProcessor | Signature timestamp validated
2015-05-28T02:08:58.295Z | DEBUG | unset-opId | tomcat-http--30 | JsonSignatureVerificationProcessor | Signature validated
2015-05-28T02:08:58.307Z | DEBUG | unset-opId | tomcat-http--30 | OperationMetadataParser | Param privileges for operation com.vmware.cis.session.create: {}
2015-05-28T02:08:58.307Z | DEBUG | unset-opId | tomcat-http--30 | PrivilegeProviderImpl | Applying privileges for following structures on the actual operation input: []
2015-05-28T02:08:58.307Z | DEBUG | unset-opId | tomcat-http--30 | PrivilegeProviderImpl | Processing following ID fields for 'operation-input' structure: []
2015-05-28T02:08:58.308Z | DEBUG | unset-opId | tomcat-http--30 | PrivilegeProviderImpl | Operation privileges for com.vmware.cis.session.create: [System.Anonymous]
2015-05-28T02:08:58.308Z | DEBUG | unset-opId | tomcat-http--30 | AuthorizationFilter | Validating permissions for 1 objects, in invocation of com.vmware.cis.session.create
2015-05-28T02:08:58.308Z | DEBUG | unset-opId | tomcat-http--30 | AuthorizationServiceClientimpl | Operation: hasPrivileges. Invoking server API.
2015-05-28T02:08:58.358Z | DEBUG | bdc32c90-3369-4aee-a5dd-d107d9dfd036 | tomcat-http--30 | LocalProvider | call to invoke() for service 'com.vmware.cis.session', operation 'create'
2015-05-28T02:08:58.358Z | DEBUG | bdc32c90-3369-4aee-a5dd-d107d9dfd036 | tomcat-http--30 | InMemorySessionStoreImpl | Created a new session with id e579902f-431a-43f1-b548-931e5cb33727 for principal Name: 'srm', domain: 'VSPHERE.LOCAL'.
2015-05-28T02:08:58.361Z | DEBUG | unset-opId | tomcat-http--21 | HttpStreamingServlet | Received request from agent 'vAPI http client' with content-length 799, content-type 'application/json' and accept header 'application/vnd.vmware.vapi.framed,application/json'
2015-05-28T02:08:58.362Z | DEBUG | unset-opId | tomcat-http--21 | OperationMetadataParser | Param privileges for operation com.vmware.content.subscribed_library.probe: {}
2015-05-28T02:08:58.362Z | DEBUG | unset-opId | tomcat-http--21 | PrivilegeProviderImpl | Applying privileges for following structures on the actual operation input: []
2015-05-28T02:08:58.362Z | DEBUG | unset-opId | tomcat-http--21 | PrivilegeProviderImpl | Processing following ID fields for 'operation-input' structure: []
2015-05-28T02:08:58.362Z | DEBUG | unset-opId | tomcat-http--21 | PrivilegeProviderImpl | Processing following ID fields for 'com.vmware.content.library.subscription_info' structure: []
2015-05-28T02:08:58.363Z | DEBUG | unset-opId | tomcat-http--21 | PrivilegeProviderImpl | Operation privileges for com.vmware.content.subscribed_library.probe: [ContentLibrary.ProbeSubscription]
2015-05-28T02:08:58.363Z | DEBUG | unset-opId | tomcat-http--21 | AuthorizationFilter | Validating permissions for 1 objects, in invocation of com.vmware.content.subscribed_library.probe
2015-05-28T02:08:58.363Z | DEBUG | unset-opId | tomcat-http--21 | AuthorizationServiceClientimpl | Operation: hasPrivileges. Invoking server API.
2015-05-28T02:08:58.368Z | DEBUG | 84c62bc3-3e4f-4041-a03d-d1bd440b9a72-135-ngc | tomcat-http--21 | LocalProvider | call to invoke() for service 'com.vmware.content.subscribed_library', operation 'probe'
2015-05-28T02:08:58.387Z | DEBUG | 84c62bc3-3e4f-4041-a03d-d1bd440b9a72-135-ngc | tomcat-http--21 | VcspClientImpl | vcsp request 'GET https://s3-us-west-1.amazonaws.com/vghetto-content-library/lib.json HTTP/1.1'
2015-05-28T02:08:58.387Z | DEBUG | 84c62bc3-3e4f-4041-a03d-d1bd440b9a72-135-ngc | tomcat-http--21 | VcspClientImpl | header Vcsp-Op-Id:b8061e5b-7008-42e7-b5e3-57379a992fd1
2015-05-28T02:09:08.781Z | DEBUG | unset-opId | tomcat-http--9 | ServletHelper | Handling HTTP request; method:GET url:http://localhost:16666/cls/cmhealthstatus
2015-05-28T02:09:08.792Z | DEBUG | unset-opId | tomcat-http--9 | ServletHelper | Response body:GREEN
2015-05-28T02:09:08.792Z | DEBUG | unset-opId | tomcat-http--9 | ServletHelper | Completed HTTP request; method:GET url:http://localhost:16666/cls/cmhealthstatus
2015-05-28T02:09:24.293Z | DEBUG | opId-9a5b6138-f1c6-428a-88e8-7fa2bfe404c2 | cls-background-executor-2 | GarbageCollectTask | refreshing garbage collection settings.
2015-05-28T02:09:32.106Z | DEBUG | opId-4aee456d-278e-463c-ba2b-8c1b376b25f8 | ScmCacheManagerImpl-executor-1 | ScmHandle | wsdlName=ScaServiceInstance class = ServiceInstance
2015-05-28T02:09:32.250Z | DEBUG | opId-4aee456d-278e-463c-ba2b-8c1b376b25f8 | ScmCacheManagerImpl-executor-1 | ScmCacheManagerImpl | Populate cache ScmCacheManagerImpl<ScmClient> completed: 16 value(s) retrieved
2015-05-28T02:09:32.250Z | DEBUG | opId-4aee456d-278e-463c-ba2b-8c1b376b25f8 | ScmCacheManagerImpl-executor-1 | ScmCacheManagerImpl | Number of elements in the cache: 16
2015-05-28T02:09:40.876Z | DEBUG | unset-opId | tomcat-http--42 | ServletHelper | Handling HTTP request; method:GET url:http://localhost:16666/cls/cmhealthstatus
2015-05-28T02:09:40.880Z | DEBUG | unset-opId | tomcat-http--42 | ServletHelper | Response body:GREEN
2015-05-28T02:09:40.880Z | DEBUG | unset-opId | tomcat-http--42 | ServletHelper | Completed HTTP request; method:GET url:http://localhost:16666/cls/cmhealthstatus
2015-05-28T02:09:48.457Z | ERROR | 84c62bc3-3e4f-4041-a03d-d1bd440b9a72-135-ngc | tomcat-http--21 | VcspClientImpl | exception while getting vcsp endpoint https://s3-us-west-1.amazonaws.com/vghetto-content-library/lib.json
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.cl.vcsp.clients.impl.VcspClientImpl.postProcessAndExecuteInt(VcspClientImpl.java:211)
at com.vmware.cl.vcsp.clients.impl.VcspClientImpl.postProcessAndExecute(VcspClientImpl.java:237)
at com.vmware.cl.vcsp.clients.impl.VcspClientImpl.getLibrary(VcspClientImpl.java:301)
at com.vmware.cl.vapi.SubscribedLibraryImpl.probe(SubscribedLibraryImpl.java:164)
at com.vmware.content.SubscribedLibraryApiInterface$ProbeApiMethod.doInvoke(SubscribedLibraryApiInterface.java:203)
at com.vmware.vapi.internal.bindings.ApiMethodSkeleton.invoke(ApiMethodSkeleton.java:169)
at com.vmware.vapi.provider.ApiMethodBasedApiInterface.invoke(ApiMethodBasedApiInterface.java:82)
at com.vmware.vapi.provider.local.LocalProvider.invokeMethodInt(LocalProvider.java:471)
at com.vmware.vapi.provider.local.LocalProvider.invoke(LocalProvider.java:290)
at com.vmware.vapi.admin.interposer.impl.Invoker.execute(Invoker.java:46)
at com.vmware.vapi.admin.interposer.impl.PreInterposerHandler.execute(PreInterposerHandler.java:57)
at com.vmware.vapi.admin.interposer.impl.VetoInterposerHandler.execute(VetoInterposerHandler.java:51)
at com.vmware.vapi.admin.impl.InterposerImpl.invoke(InterposerImpl.java:277)
at com.vmware.vdcs.activation.ActivationFilter.invoke(ActivationFilter.java:123)
at com.vmware.vapi.core.DecoratorApiProvider.invoke(DecoratorApiProvider.java:37)
at com.vmware.vsphere.common.impl.SecurityContextInterceptorProvider.invoke(SecurityContextInterceptorProvider.java:72)
at com.vmware.vapi.cis.authz.impl.AuthorizationFilter.invoke(AuthorizationFilter.java:219)
at com.vmware.vapi.provider.introspection.ErrorAugmentingFilter.invoke(ErrorAugmentingFilter.java:74)
at com.vmware.vapi.security.AuthenticationFilter$1.setResult(AuthenticationFilter.java:180)
at com.vmware.vapi.security.AuthenticationFilter$1.setResult(AuthenticationFilter.java:166)
at com.vmware.vsphere.common.sessions.impl.SessionAuthnHandlerImpl.authenticate(SessionAuthnHandlerImpl.java:42)
at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:165)
at com.vmware.vapi.core.DecoratorApiProvider.invoke(DecoratorApiProvider.java:37)
at com.vmware.vsphere.vcde.diagnostics.DiagnosticsInterceptorProvider.invoke(DiagnosticsInterceptorProvider.java:46)
at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.processApiRequest(JsonServerConnection.java:281)
at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:206)
at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:124)
at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:92)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at com.vmware.vcde.common.services.cm.servlet.DispatcherServlet.service(DispatcherServlet.java:53)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
^C
Can you try restarting the Content Library service by running the following command: /etc/init.d/vmware-vdcs restart
I wonder if the service needs to be restarted for it to pickup the proxy configurations
I had completely rebooted the vCenter server which I assume would do the same.. I tried this as well to no avail...
FYI, I just tried on the windows based vCenter and the proxy configs you mention work fine!
I just spoke with Engineering, try the following and see if it works:
Add "wrapper.java.additional.19=-Djava.net.useSystemProxies=true" to /usr/lib/vmware-vdcs/wrapper/conf/wrapper.conf and restart vmware-vdcs service and see if that works
if that still doesn't work, can you try explicitly setting the proxy for the CL service by adding the following to /usr/lib/vmware-vdcs/wrapper/conf/wrapper.conf and restart vmware-vdcs service:
wrapper.java.additional.19=-Djava.net.useSystemProxies=true
wrapper.java.additional.21=-Dhttp.proxyPort=
wrapper.java.additional.20=-Dhttp.proxyHost=
OK! This got me on the right track! A few small tweaks to what you mentioned. There was already a wrapper.java.additional.19 in my config, so I needed to start at .20.. and I needed to use https (not http)
Here is the final 3 lines that did the trick!!! (in our case the proxy port for https is 8080)
wrapper.java.additional.20=-Dhttps.proxySet=true
wrapper.java.additional.21=-Dhttps.proxyHost=proxy.server.com
wrapper.java.additional.22=-Dhttps.proxyPort=8080
This is definitely something that would be good to document for customers!
Thanks again!! Love all your posts!!!
AK
Thanks for confirming the steps and I'll update the blog post with this new information
I've tried using the Service setting in vCenter 6.5 Update 2 (on an appliance), but they don't seem to work as expected when trying to import an item from a URL.
Watching what happens at the web server, the vCenter makes a "HEAD /filename.iso" request, followed 2 seconds later by a "GET /filename.iso" coming from the proxy server.
If I try a URL which isn't accessible from the vCenter directly, I just get the "Unable to connect to source. Connection timed out" error and the vCenter never tries the "GET".
I assume the "HEAD" request is failing because the vCenter can't reach the site, so never bothers with the "GET". The "HEAD" request should be going via the proxy as well, shouldn't it?
Did you ever resolve this?
I have the same issue in vCenter 6.7u3. I had to use the deprecated Flash client to configure the service as the HTML5 client doesn't have the options (although it looks like you could probably modify "/etc/vmware-content-library/config/ts-config.properties" by hand), but everytime I try and subscribe to "https://download3.vmware.com/software/vmw-tools/lib.json" I get a "HTTP request error: connect timed out.".
/var/log/vmware/content-library/cls.log shows a "java.net.SocketTimeoutException: connect timed out" is thrown.
I've tried including (and not) "http://" on the proxy urls, service restarts, etc. all without any joy.
I got timeout in 6.7.
https://kb.vmware.com/s/article/81210 say this does not work. for 6.7/
Is it a regression from earlier release ? Or there is some hack/manual config to get it working
Any pointers?