vRealize Log Insight Forward to External SEIM with syslog-ng relay as intermediary, using Ansible

Bottom Line Technical Nugget: vRealize Log Insight Version 4.5.0-5654101 Event Forwarding with the syslog protocol will format messages with different formats if there are Tags defined vs. no tags defined in the Event Forwarding Destination entry.  Configuring syslog-ng to expect the newer IETF syslog protocol will only work when vRLI has a custom Tag assigned on the Destination.  Here’s an example of the difference:

Only the second message follows IETF syslog protocol, as understood by syslog-ng.  The first message, where I did not define a custom Tag in the Forwarding Destination, seems to be closer to the legacy BSD format.  For example, notice the subsystem (vmkernel and netcpa) is formatted as vmkernel:  in the first example, and structured data by field position in the second example.  Legacy BSD format uses the subsystemname:  syntax as in the first example.

If that’s all you came here for, great, leave a comment if you want and go back to work.  If you were looking for more of a use case documentation with config examples, read on!

Use Case

Using vRealize Log Insight to aggregate logs in a VMware-centric data center makes a lot of sense.  Intelligent parsing of logs from VMware products and many 3rd party technologies via Content Packs, it is simple, great interface, integrated load balancer, and so on.

If required to forward selected logs to a third party, for example a Managed SEIM scenario, the vRealize Log Insight Event Forwarding provides the ability to do so.  It also provides some filtering capabilities to control which logs are forwarded.  However, its filtering capabilities appear to be fairly basic (at least, what is available through the GUI).  For example, I cannot always take a query created from the Interactive Analytics tool and port it directly into a filter, because not all of the parsed fields appear to be available.  So, if more granular control is required before shipping logs to the External SEIM, or if you simply want another layer of abstraction and control, one possible solution is to deploy syslog-ng as a relay and configure filtering rules on the relay.  This is the solution I’ll be discussing here.

Ingesting

Setup your log sources (devices, nodes, hosts, whatnot) to send syslog traffic in either of the major supported formats to your vRLI integrated load balanced IP using one of the following ports and protocols:

  • 514 udp
  • 514 tcp
  • 1514 encrypted with TLS

You can also get logs into vRLI by installing vRLI Agents on various systems, and also via the Ingest API.  One of my favorite things about vRLI is its simplicity.

Log Insight Event Forwarding

Like ingesting, configuring Event Forwarding is straightforward.  Follow the the vRLI documentation.

The important caveat is that you MUST assign at least one Tag if you want “new IETF” formatted syslog messages emitted from vRLI, as I mentioned at the start of this guide.

In the example below, my custom tag is named “mytag” and the value is “something”.

( Please ignore the LastPass icon that is mistakenly floating in the middle of my screenshot.  It isn’t there if we all pretend it isn’t there. )

Syslog-ng Configuration

Using my go-to configuration management tool, Ansible, I drafted a quick and dirty “playbook” to setup a CentOS system as syslog-ng forwarder.

01-setup-syslogng-relay.yml

There are two supporting files.  Note – I should put variables into variables, so they can be included in the YML playbook, but I haven’t done that.  So in this case, just edit the template files directly.

syslog-ng.conf (removed all the good stuff from the default version)

relay.conf (the main configuration)

With all three files placed in a common directory, and Ansible installed via  yum install ansible, applying the configuration is as simple as one command:

If all is going according to plan, your upstream SEIM should start receiving syslog traffic.  I like to use ss  to do a basic check:

Enjoy!  As always, questions, comments, and feedback are welcome.

vROPS 6.6.1 Custom Certificates Time Zone Bug

EDIT – 10/3/17 Received a response from VMware GSS confirming this as a bug.  Expect a solution in a future release, for now you can use my workaround below.

vRealize Operations Manager 6.6.1 custom certificate import script has a bug related to checking validity period of the certificate.  It will read the certificate time in GMT, and compare to the local system time in local time zone.  It should translate GMT to the local time zone first, but it does not.  This will only affect you if you are close to the extremes of the validity period for the certificate you are trying to install, and the time zone vs. GMT delta is not in your favor.  I illustrate a workaround below, specifically I change the vROPs system local time zone temporarily to GMT.

Steps to reproduce
Generate a custom certificate using the VVD 4.1 CertGen tool on a properly time synchronized Windows Enterprise CA. Note the valid period for the certificate in my case:

Deploy vROPs OVA with time zone = PDT and proper NTP server, verify with:

Import a PEM file using the vROPs cluster installation wizard, and check the /var/log/casa_log/casa.log when it fails:

It appears the vropsCertificateTool.py script is ignoring the time zone information with the certificate and checking the validity date as if the certificate valid not before date was in the local time zone of the vROPS server, rather than converting from GMT to local time before verifying the validity of the cert.

Workaround

As a workaround, I changed the time zone on the vROPs system:

And then upload the correct PEM file in the vROPs cluster installation wizard, and it accepts the cert as valid.  After, you can change your time zone back by updating the symlink.

Note – I have a case open with GSS to report this bug, but probably most people just use the CertGen VVD tool early in the process, and thus enough time has passed between generating the certs and deploying the solution that this is a non-issue.  Hope this helps.

EDIT – See note at top.

 

VMware Validated Designs, Software Defined Data Center Version 4.1 (vRA 7.3, NSX 6.3.3)

I use these.  Generally speaking, they are really good.  It helps to have a team with mastery of the products, this is not quite a SDDC for dummies.

Glad to see the update including latest versions of the vRealize suite, NSX, and other core products.

VCAP7-CMA Design SME Certification Development Experience

A few months ago, I was invited to participate in VMware’s VCAP7-CMA Design exam development process.  What a great experience!

I’ll keep my comments brief to avoid any material that may be covered by NDA, but I would like to touch on one topic that seems to be a hot subject.  In the latest advanced design exams, VMware has announced they are removing the visual drag and drop style whiteboard questions.  Some folks (I’m looking at you reddit thread) seem to be under the impression that VMware doesn’t want to bother with a complex testing tool.  While I am not privy to the business drivers for this decision, I can certainly relay that the messaging around the change during the workshop I participated in was 100% focused on issues with test taker feedback and grading challenges.

In my technical circles here in Sacramento, this is a fairly common topic of discussion and I imagine elsewhere also.  The common complaint is the gap between real world applied skill, and certifications, and closing that gap so certifications carry weight and “mean something.”

The consensus seems to be for exams that test the skill of operating, troubleshooting, and configuring a system, the best test is an extended lab-based exam.  In this day of automation and cloud computing, I see no reason labs need to be expensive to administer or grade. Provided the scenarios are complex enough, test dumps become a much smaller issue.  You also cut down on nonsense questions and close the gap between what I’ve heard called “book exams” vs. “real world exams,” the former being memorizing every word of documentation to pass an exam, the later being a skilled administrator capable of working with the system in question.  At the end of the test, either you have configured the system to operate correctly, or you have not.  Simple!

Applying this idea to design exams becomes perhaps more tricky.  If I could craft an ideal scenario, it would require every candidate to present a live whiteboard session to a small panel of experts.  I suppose in some ways, this describes the later part of the VCDX process.  But it comes with high cost, high degree of uniqueness per session, difficult to scale, and so on.  Perhaps that is where the drag and drop canvas design idea came from – automate the whiteboard session.  Unfortunately, there are clearly problems in executing this approach.  One of those problems is the whiteboard session allows the panel to ask interactive questions and get clarification.  During a technical interview, this is a great way to see how the candidate thinks as well as how well they grasp the technology.

This is a long way to say the canvas design questions are a great aspiration but have proved challenging in practice.  So, how do you write a design exam without canvas questions?  There are lots of options.  For example, you could present the test-taker with a drawing of a design, perhaps with a flaw, and prompt for clarification regarding the design.  Or, you could use the old tried and true scenario based questions.  There are other options too.  These approaches in my estimation do not demand as much from the test taker as blank whiteboard and marker, but they seem highly repeatable and fair, do not require the test taker to explain their logic in order to see if their drawing is actually valid.  They should be sufficient for the degree of knowledge tested in an advanced level exam.  Again, we still have the expert level certification to denote true mastery of the subject and products.

All in all, I was very impressed by the test development process, with an enormous amount of metrics, analysis, and process behind the scenes that frankly, I had not really had occasion to think about as a test taker where the exam is a magical black box.  I have a high degree of confidence in the folks VMware has running the certification exam development program, and I expect to continue to see improvements in the program based on real-world data collected in the field.

 

 

vRealize Log Insight and vRA Embedded vIDM – Password Complexity

When configuring vRealize Log Insight 4.5 to use vRealize Automation 7.3 embedded VMware Identity Manager (per this blog post), I ran into an issue with password complexity.

I specified a tenant name and provided valid tenant administrator credentials to register with vIDM.  However, the web interface indicated an error when clicking Test Connection, related either to bad username/password or unknown response.  I resolved this by using what I’d characterize as less “special” complex characters in the the local tenant administrator user account password, then running Test Connection again.  Success!

Investigating the log files on the vRealize Log Insight system, I found a useful entry in one of the log files:

  • /var/log/loginsight/ui_runtime.log

For posterity, the steps to change a vRA 7.3 tenant local user account password are:

  • Log in to vRA as the default administrator
  • Navigate to Tenants -> Your Tenant Name -> Local Users
  • Click the local user account to manage
  • Click Edit
  • Change the password

So, this is the same issue we’ve seen in a few different places and products.  My recommendation is to always use passwords with a high degree of entropy, but in some cases you need to be careful of special characters that can be misinterpreted by some of the product line.  Fun times.  Hope this helps.