Monday, October 15, 2012

Fix vCO's SSH Plugin 'InternalError: Auth fail' Issue

For those of you using the VMware Orchestrator SSH plugin, you may have encountered the following error when invoking the Run SSH command library workflow:
[2012-10-15 16:40:08.400] [I] Connecting with password
[2012-10-15 16:40:08.439] [I] Unable to execute command InternalError: Auth fail (Workflow:Run SSH command / Execute SSH Command (item6)#14) (Workflow:Run SSH command / Execute SSH Command (item6)#31) 
Of course, when using a standard SSH client such as putty or ssh, there are no issues connecting with the same username and password credential set.

The authentication failure is the result of SSH configurations that disable password authentication in favor of keyboard-interactive or public key authentication options.  Unfortunately, vCO's current version of the SSH plugin only supports password and public key authentication.

Many current system configurations disable password authentication by default to reduce the effectiveness of brute force attacks on the SSH daemon.  Most SSH clients will properly negotiate the available authentication options with the SSH server, and will quietly switch to keyboard-interactive.  

You can validate your SSH configuration by looking for the PasswordAuthentication parameter in /etc/ssh/sshd_config on your target server system.  If this value is set to no, then the vCO SSH plugin commands will fail when using password authentication.

Here are a few workarounds to consider:
  1. Enable PasswordAuthentication on the SSH server host.  While this is generally considered less secure, it is the easiest fix on a small number of SSH hosts.
  2. Use Orchestrator's Command object to run an external program, such as plink.exe or ssh, that supports keyboard-interactive SSH authentication.
  3. Generate SSH key pairs and run the vCO workflow with key file authentication.  SSH keys are almost universally enabled by default on most SSH hosts.
Generating an SSH key pair is easily done by calling the vCO SSH plugin workflow Generate key pair.  Once the certificate is generated, you then run the workflow Register vCO public key on host to distribute your new SSH key.  Unfortunately, with password authentication disabled on the SSH host, the workflow will fail as well with the same internal authentication error!

Distributing the new SSH public key will require downloading the file from the VMware Orchestrator server and uploading the key to the ~/.ssh/authorized_keys file on the target SSH host.

The location of the vCO SSH public key will differ depending on which version of vCO you installed.

vCO Appliance:

vCO Windows Installation:

Once you have the vCO server's generated SSH public key, existing tools can be leveraged to distribute the key in bulk to any intended SSH target hosts.  If you already have an automation service account with an existing SSH key pair, that key can be imported into the vCO server to enable public key authentication by replacing the vco_key and files with your private and public keys, respectively.

Now that your SSH keys are configured and distributed, just be sure to call the Run SSH command workflow by specifying no in the Authentication input option.

Set key file authentication


  1. Any chance that a future version of the plugin might do keyboard-interactive authentication?

    1. VMware Engineering is aware of the issue and is planning to update the SSH plugin to support keyboard-interactive authentication. No official word on when that will happen, though.

  2. Enable PasswordAuthentication temporarily (set to yes), restart the SSHD service and use the above 2 workflows to generate and copy the key pair file to the host and then disable the PasswordAuthentication (set to no) and then restart the service again. Now you should be able able to use the key pair file for authentication. You can also notice the ~/.ssh/authorized_key file is updated with new information.