Pages

Wednesday, November 21, 2012

Best Practices for Faster vSphere SDK Scripts

The VMware vSphere API is one of the more powerful vendor SDKs available in the Virtualization Ecosystem.  As adoption of VMware vSphere has grown over the years, so has the size of Virtual Infrastructure environments.  In many larger enterprises, the increasing number of VirtualMachines and HostSystems is driving the architectural requirement to deploy multiple vCenter Servers.

In response, the necessity for automation tooling has grown just as quickly.  Automation to create daily reports, perform bulk operations, and aggregate data from large, distributed Virtual Infrastructure environments is a common requirement for managing the increasing virtual sprawl.

In a Virtual Infrastructure comprised of thousands of objects, even a simple script to list all VirtualMachines and their associated HostSystem and Datastores can result in very slow runtime execution.  Developing automation with the following, simple best practices can take orders of magnitude off your vSphere API tool's runtime.

Note:  The examples provided are written using the VI SDK for Perl, but the best practices can be incorporated into automation tools using other VMware SDK toolkits.
#!/usr/bin/perl
## VmHostDatastoreReport-Slow.pl
## 
## Sample script to illustrate the poor performance of a non-optimized 
## VMware SDK entity and property collection.
## Created by Reuben Stump (rstump@vmware.com | http://www.virtuin.com)

use strict;
use warnings;

use VMware::VIRuntime;

Opts::parse();
Opts::validate();

Util::connect();

# Fetch all VirtualMachines from SDK
my $vm_views = Vim::find_entity_views(view_type => "VirtualMachine") || 
  die "Failed to get VirtualMachines: $!";

# Enumerate VirtualMachines
foreach my $vm ( @{$vm_views || []} ) {
 # Fetch HostSystem properties from SDK
 my $host = Vim::get_view(mo_ref => $vm->get_property('runtime.host'));
 # Fetch Datastore properties from SDK; could have multiple datastores
 my $datastores = Vim::get_views(mo_ref_array => $vm->get_property('datastore'));
 
 printf("VirtualMachine '%s':\n   HostSystem: %s\n   Datastores: %s\n", 
   $vm->get_property('name'), 
   $host->get_property('name'), 
   join(',', map($_->get_property('name'), @{$datastores}) )
 );
}

# Disable SSL hostname verification for vCenter self-signed certificate
BEGIN {
 $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
}

Even in a small lab environment (two HostSystems, two VirtualMachines, two Datastores), the runtime of the script is pushing 4 seconds.

VmHostDatastoreReport-Slow.pl

In environments with thousands of VirtualMachines, hundreds of HostSystems and dozens of Datastores the same script logic will balloon the total execution time dramatically -- sometimes even measured in hours.

The following best practices can provide orders of magnitude faster vSphere API runtime execution.


Minimize the ManagedEntity's Property Set

The vSphere API is an extremely powerful SDK interface.  The volume of data exposed through the vSphere inventory is quite large.  But the robust, rich property values exposed by the vSphere API come with a price -- expensive network data transfer.

As an example, let's look at one of the larger managed entity objects in the vSphere API model, the HostSystem.  The following script will show the amount of data and the time elapsed to query a single HostSystem from the vSphere API.  In the first query, all HostSystem properties are retrieved from the API inventory (the VI Perl SDK default).  In the second query, only the name property is specified.
#!/usr/bin/perl

use strict;
use warnings;

use VMware::VIRuntime;
use Devel::Size qw(total_size);
use Time::HiRes qw(time);

my %opts = (
 host => {
 type => "=s",
 variable => "host",
 help => "HostSystem Name",
 required => 1,
 }
);

Opts::add_options(%opts);
Opts::parse();
Opts::validate();

my ($host_name, $host_view, $start, $elapsed);
$host_name = Opts::get_option("host");

Util::connect();

# Get HostSystem - All properties
$start = time();
$host_view = Vim::find_entity_view(view_type => "HostSystem",
         filter => { 'name' => $host_name }) || 
  die "Failed to get HostSystem '$host_name': $!";
$elapsed = time() - $start;
printf("Total size of HostSystem '%s' (Properties: all ): %.2f KB in %.2fs\n", $host_view->{'name'}, total_size($host_view)/1024, $elapsed);

# Get HostSystem - Name property only
$host_view = undef;
$start = time();
$host_view = Vim::find_entity_view(view_type => "HostSystem",
         filter => { 'name' => $host_name },
         properties => [ 'name' ] ) || 
  die "Failed to get HostSystem '$host_name': $!";
$elapsed = time() - $start;
printf("Total size of HostSystem '%s' (Properties: name): %.2f KB in %.2fs\n", $host_view->{'name'}, total_size($host_view)/1024, $elapsed);

BEGIN {
 $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
}

HostSystemSize.pl

When querying the HostSystem for all property values, the total size of the Perl SDK HostSystem object was 4.56 MB and took over 1.6 seconds to process the entire managed entity.  The same HostSystem with just a single property resulted in a Perl SDK HostSystem object size of 53.51 KB and was retrieved in under 0.07 seconds.

Note: Devel::Size calculates the size by following references, so the sizes reported here may not be indicative of actual memory usage.

In the VI SDK for Perl, the list of properties are specified by using the named parameter properties with an anonymous array reference to the various SDK view methods.  For example, properties => [ 'name', 'runtime.host', 'runtime.powerState' ].

Note: In most cases, using a parent property will result in the vSphere API returning all child properties.  For example, specifying the VirtualMachine property runtime will return runtime.hostruntime.powerStateruntime.connectionState, etc.  For best performance, use full and specific property paths as required by your vSphere SDK script logic.

Of course, every property will vary in size and performance.  Specifying individual properties will also increase the complexity of the code logic.  However, requesting the minimum required property set will provide the best results towards reducing the overall vSphere SDK automation tool runtime execution.



Pre-Fetch and Store ManagedEntities in Hash Tables


Reviewing the logic of the VmHostDatastoreReport-Slow.pl script example listed above, there is another performance issue - invoking the Vim::get_view() and Vim::get_views() methods in a loop structure. 

 There are two problems with this approach.  First, calling Vim::get_view() on each HostSystem individually incurs overhead for each SOAP request and response.  Second, since multiple VirtualMachines are sharing HostSystems and Datastores, the script is retrieving the same set of data each time a particular HostSystem or Datastore entity property set is requested.  In an environment with 30 VirtualMachines per HostSystem, the non-optimized script will end up fetching the same data set for each HostSystem 30 times.

I recommend not placing any property collections from the vSphere SDK in a loop structure.  There are exceptions, of course, but understanding when to make an exception will avoid non-optimized script runtime performance.

Let's combine both best practices into a new script, VmHostDatastoreReport-Fast.pl, and compare the runtime execution to the non-optimized script, VmHostDatastoreReport-Slow.pl.
#!/usr/bin/perl
## VmHostDatastoreReport-Fast.pl
## 
## Sample script to illustrate the performance of an optimized 
## VMware SDK entity and property collection.
## Created by Reuben Stump (rstump@vmware.com | http://www.virtuin.com)

use strict;
use warnings;

use VMware::VIRuntime;

Opts::parse();
Opts::validate();

Util::connect();

# Fetch all VirtualMachines from SDK, limiting the property set
my $vm_views = Vim::find_entity_views(view_type => "VirtualMachine",
          properties => ['name', 'runtime.host', 'datastore']) || 
  die "Failed to get VirtualMachines: $!";

# Fetch all HostSystems from SDK, limiting the property set
my $host_views = Vim::find_entity_views(view_type => "HostSystem",
          properties => ['name']) ||
  die "Failed to get HostSystems: $!";

# Fetch all Datastores from SDK, limiting the property set
my $datastore_views = Vim::find_entity_views(view_type => "Datastore",
            properties => ['name']) ||
  die "Failed to get Datastores: $!";
        
# Create hash tables with key = entity.mo_ref.value            
my %host_map = map { $_->get_property('mo_ref.value') => $_ } @{ $host_views || [] };
my %ds_map = map { $_->get_property('mo_ref.value') => $_ } @{ $datastore_views || [] };

# Enumerate VirtualMachines
foreach my $vm ( @{$vm_views || []} ) {
 # Get HostSystem from the host map
 my $host_ref = $vm->get_property('runtime.host')->{'value'};
 my $host = $host_map{$host_ref};

 # Get array of datastore moref values
 my @ds_refs = map($_->{'value'}, @{$vm->get_property('datastore') || []}); 
 # Get array of datastore entities from the datastore map by slicing %ds_map
 my @datastores = @ds_map{@ds_refs};
 
 printf("VirtualMachine '%s':\n   HostSystem: %s\n   Datastores: %s\n", 
   $vm->get_property('name'), 
   $host->get_property('name'), 
   join(',', map($_->get_property('name'), @datastores) )
 );
}

# Disable SSL hostname verification for vCenter self-signed certificate
BEGIN {
 $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
}

VmHostDatastoreReport-Fast.pl

In the optimized version, using both best practices for performance, the reporting script completed in under 0.6 seconds compared to the non-optimized version at 4 seconds.  Even in a small lab environment, the performance improvement is dramatic.  In larger vSphere environments, the performance gains can improve script execution runtimes from hours to minutes and minutes to seconds.

The vSphere API actually provides the capability to retrieve multiple entity types and property sets in a single RetrievePropertiesEx() method call.  Incorporating these best practices into a single API request, it's possible to generate a hash table containing every vSphere entity efficiently and with minimal runtime overhead.

I've successfully used these best practices to create custom data collection tools that executed in minutes across thousands of VirtualMachines while maintaining the relationships between HostSystems, Datastores, Networks, Events and Task entities across multiple, remote vCenter API inventories.


14 comments:

  1. Thanks a lot! This is really good advice!

    With Lab Manager Light (https://github.com/ImmobilienScout24/lab-manager-light) I also have a lot of performance issues that I hope to cure with this advice.

    ReplyDelete
  2. Glad it was helpful! I poked around your project on github, looks interesting. Definitely a lot of environments where lab manager is missed.

    ReplyDelete
  3. extremely useful, thanks!

    ReplyDelete
    Replies
    1. Thanks for the feedback! Good to hear it was useful.

      Delete
  4. I had no idea you could request only the properties you wanted returned from get_view(s) calls. This is exactly what I was looking for! I have a script that has to contact several VMs all across the globe and process huge amounts of data from each. It currently takes 15 hours to get all the info from our VC in India. I can't wait to start using the "properties" param and see how much time it saves. Thanks for this!

    ReplyDelete
    Replies
    1. By the way. Wanted to post my results. I just tested the updated code I mentioned in my first post - the script which previously took approximately 15 hours took just over 3 minutes once I updated it to use the "properties" param and only pull in the data I was actually looking for. Unbelievable. Thanks again.

      Delete
    2. That's awesome. I've seen similar results, but every environment is different. Huge speed increase though, you could probably run it hourly now as well!

      Delete
    3. I am trying to pull guest performance data from vCenter. Has anybody done something like this with perl and like to share the code? Powershell power-cli is taking a very long time?

      Delete
  5. Excellent write-up.

    It helped me to improve performance of my script from 25 seconds to just 1 second. And the impact is even better for bigger vSphere environment. My old version of script was almost useless and this simple improvement help me so much.

    Thanks Reuben for sharing this information.

    ReplyDelete
  6. Good techniques , Learnt a lot Could anyone help me to check on all properties available 'name', 'runtime.host', 'datastore' and more ....

    ReplyDelete
  7. Very good info, thanks!!! This helped me tremendously.

    ReplyDelete
  8. Very impressive; from over 5 minutes to less than a second..
    Thanks!

    ReplyDelete
  9. This helps! the techniques here improved the performance of my script from 10 mins to 11 sec. Thank you very much!

    ReplyDelete
  10. Great article - helped a lot with my project. One question though - any idea how to specify VirtualDisk property paths?

    Config.Hardware.Device is currently as far as I can go down the rabbit hole without getting argument exceptions thrown at me. I'd like to specify that I only want CapacityInKB, for example, on VirtualDisk objects. Hoping someone can point me in the right direction.

    Thanks,
    Stefan.

    ReplyDelete