Automate Vester with Jenkins and Docker

Summary

A code driven approach to automate VMware vSphere configuration management, by running Vester with Jenkins and Docker.  Based entirely on open source projects.

In this article, I’ll summarize configuration of each component and provide example code to get you started.

The main idea:

  1. You, the administrator, describe your vSphere desired state in a couple Vester config files (text files in JSON format)
  2. You check those config files into source control
  3. Jenkins periodically runs Vester against your vSphere environments, using your config files, and collects the results
  4. You can view the test results in a handy report format, as shown here:
Vester Test Results – Shown in Jenkins Blue Ocean UI

Project links

Code

Components

Also, if you want to run Vester in a container, for now you’ll need my branch of Vester (PR pending): https://github.com/jeffgreenca/Vester/tree/fix/docker

If you are curious about my jeffgreenca/mspowercli Docker image, you can review the Dockerfile and assets here: https://github.com/jeffgreenca/mspowercli

Design overview

Here’s an architecture diagram of the solution:

Diagram walk-thru

  1. The administrator commits Vester config files to internal source control.
  2. Jenkins pipeline is triggered on source control commit and time-based triggers, to run periodically
  3. When the Jenkins pipeline runs, Jenkins starts a Docker container with all the necessary components to run Vester, including the custom config files.
  4. Invoke-Vester is executed against each vCenter, in parallel for speed.  Read-only vCenter credentials stored in Jenkins are used to establish the connection.
  5. Jenkins saves the Vester test results using the NUnit plugin
  6. The administrator reviews the test results via the Jenkins Blue Ocean UI

About dockerizing Vester

If you’re just trying to replicate my results, you can skip reading this section, as I’ve already done this part of the work for you.

Initially, the simplest approach was to try and load Vester into the VMware PowerCLI core docker image.  This didn’t work, for a couple of reasons:

  • VMware’s PowerCLI core docker image didn’t really load the Pester module, the PowerShell testing framework that Vester is built on.
  • Vester code base is pretty free with assuming\ to be the path delimiter.  Enter Linux: what are these/‘s doing in my paths?!
  • One module, VMware.VumAutomation is not (yet?) ported to PowerCLI core.

I solved each problem as follows:

Generate Vester configs

Use New-VesterConfig per the normal Vester usage to generate configuration files for each vCenter environment.  You’ll maintain these files in a specific path in the git repository.

Set up the git repository

Hopefully, you’ll already have some kind of git-based source control system.  This could certainly use a cloud-based solution, but keep in mind you’ll be storing your Vester config files so be sure it is sufficiently secured.  For self-hosted, I am currently using https://gogs.io and enjoying it.

Create a new repository, “auto-vester” or whatever you’d like to call it.  To save time, you can start by forking my example repository here: https://github.com/jeffgreenca/auto-vester-example

The repository will have this structure:

The way I setup my environment, each vCenter has a Vester JSON config file under configs/<environment>/ where <environment> is for dev, prod, staging, etc.

The naming is important, as the JSON config files are referenced in the Jenkinsfile pipeline definition.

Set up Jenkins Blue Ocean

Deploy Jenkins

If you don’t already have a Jenkins instance, it’s easy to stand up a simple system using the provided Docker image.

For this project, I used Docker Compose to stand up a Jenkins Blue Ocean instance.  For example, this docker-compose.yml file should work on any Linux server with Docker CE engine installed and Docker Compose installed.  For production, you’d want to protect your system behind HTTPS but for a proof of concept this should be fine:

To start up the system:

$ docker-compose up -d

Attach to the container’s log to get the setup key required by the Jenkins web-based setup wizard.  Browse to http://<host>:8080/ and then to load the Blue Ocean interface, the URL will be http://<host>:8080/blue

Install a plugin

Install the NUnit plugin as it is used to read the Vester test results.

Create credentials

Vester needs credentials to connect to vSphere.  Lucky for us, Jenkins can store these credentials encrypted and make them available for a pipeline.  Follow the instructions here: https://jenkins.io/doc/book/using/using-credentials/

You’ll need to create a username/password credential type, with the ID vcenter-vester-prod (or whatever you edit the Jenkins pipeline definition to reference).

Configure the Jenkins pipeline

I created a Jenkins Declarative Pipeline, which you can modify to your needs.  This is stored in the Jenkinsfile in the repository and tells Jenkins what to do when the pipeline runs.  It also includes a time-based trigger to run the pipeline periodically.

A couple of key notes:

  • Edit the environment { vServer = ” } assignment to your vCenter(s)!
  • All steps are run inside a Docker container using my image jeffgreenca/mspowercli downloaded automatically from the Docker hub
  • In the “Download Vester” stage, I check out my dockerized version of the Vester project which supports running on PowerCLI core.
  • The parallel {} block speeds up execution by running multiple Invoke-Vester tests at once
  • Environment variables are used to define the vCenter to connect to for each stage, which enables using the appropriate config file and saving the test results to a unique XML file
  • For readability, each vCenter is just a copy and paste of the “stage” block.
  • All of the test results are collected in a single step at the end of the pipeline.  Because Invoke-Vester exits cleanly even when tests fail, this doesn’t need to be in a post { always { } } block.

Add the pipeline to Jenkins Blue Ocean

In Jenkins Blue Ocean (https://<host>:8080/blue) click New Pipeline and enter the information to connect to your git repository.  You’ll need to configure at least read access to your repository for Jenkins.  See the Jenkins documentation for more information: https://jenkins.io/doc/tutorials/build-a-multibranch-pipeline-project/

You already have a Jenkinsfile in your repository, so the pipeline will be created automatically once you point to the repository.

From here on out, you can manually trigger the master branch pipeline to execute, or wait for it to happen automatically based on the cron trigger.

Review test results

At this point, you should have a more or less working setup.  Here’s some screenshots of what you should be seeing in Jenkins when you run your pipeline.

In-progress pipeline executing

Here you can see a pipeline running, with one Invoke-Vester stage still executing and before test results have been collected.

Jenkins Pipeline Execution In Progress

Examine test results

Once the pipeline completes, the Test tab will be populated with the test results.  One of the really helpful things about this system is the reporting on “new tests failing,” so you can determine if something recently changed in your environment vs. the last execution of the pipeline, even when some tests are failing.

After the pipeline runs, viewing the test results

View details about a specific  failed test

Each failed test can be clicked on to get the stack trace which will show you exactly what is misconfigured for this object.  I’ve blurred out the names of my syslog servers and the hostname of this ESXi host, but you can see that the syslog server I configured in my Vester config desired state file does not match the actual configuration on this particular host:

Viewing details about a particular failed test. “Desired” and “Actual” show the specific misconfigured values.

What’s next?

It’s up to you!  One option would be to add a pipeline stage that applies Vester with the -Remediate flag turned on, to automatically remediate your environment.  This might be a good place to consider using the Jenkins input command, which allows the pipeline to pause and wait for user input before continuing with the actual environment-changing remediation.

Thanks for reading, and I hope this helps you out!

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Posts

Categories

Archives

GiottoPress by Enrique Chavez