Understanding vRealize Log Insight Event Forwarding Filters Behavior

vRealize Log Insight enables generic event forwarding via the syslog protocol.  The Event Forwarding filters, at least to me, were initially not the most intuitive.  So, here’s what I’ve learned using vRLI 4.5.

Understanding Implicit Conditional Operators

First, notice the gap between *Key1* and *Key2*.  That indicates they are two separate criteria (values) that will satisfy the filter rule.  Filter rules evaluate true when ANY value for a specific filter rule meets the criteria.  So, there is an implicit OR between all the values for a particular rule.

Second, filter rules are additive.  There is an implicit AND between all the filter rules assigned to an Event Forwarding Destination.

“Not Match” Adds Complexity

Consider these two equivalent scenarios:

  • Scenario 1
    • text does not match *Key1*
    • text does not match *Key4*

There is an implicit AND between the two filters.  So messages will not be forwarded if they contain either Key1 or Key4.  Think of it as !(match *Key1*) AND !(match *Key4*).

  • Scenario 2
    • text does not match *Key1* *Key4*

Think of this as not (match *Key1* OR match *Key4*).  If the message contains Key1 or Key4, it will match the parenthesized condition, then the negation is applied.  Net result, a message containing any of the “not match” values.

Testing

I recorded actual test results, by crafting sample syslog messages using a generator pointed to Log Insight, and configuring an Event Forwarder using the following filters.  I observed the results with a generic syslog-ng instance configured to receive the forwarded events from Log Insight.

Because of its importance, I denoted where multiple values to a filter are separated by pressing [Enter] after typing each value.  If you just separate the values by a space, you will end up with a single value like “Key1 Key2” instead of “Key1”, “Key2”.  This is easy (for me) to mistype, so here’s one last screenshot to remind you:

Showing multiple separate values that have an implicit OR (*Key1* OR *Key2*).

Finally, here are my test results for reference:

Filters Results Analysis
hostname matches host1
hostname matches host2
no msgs forwarded filter1 AND filter2
hostname matches host1[Enter] host2[Enter] all msgs forwarded value1 OR value2
text matches Key1 no msgs forwarded entire text must match "Key1"!
text matches *Key1* msgs with Key1 forwarded
text matches *Key1*[Enter] *Key4*[Enter] all messages forwarded value1 OR value2
text matches *Key1*
text does not match *Key4*
msgs with Key1 and not Key4 (match value1) AND !(match value2)

Hope this helps.

vRealize Operations Manager 6.x UI issues with Touch Screen

If you have a touch-screen system and are using vROPs 6.x, you’re in for a bad time.  There is a KB about this:

Resolution is disable the touch screen device.

Go to Device Manager -> Human Interface Devices -> Disable device

Some of the symptoms I experienced:

  • Clicking the User icon and About did not open the About screen
  • Setting a filter type on Reports screen would reload the Home screen
  • Could not create a new vROPs View – UI crashes after configuring Subjects
  • Observed the following error in the Chrome dev console

I am simply leaving this here in case someone else, like me, searches for the browser client side error message.

vRealize Network Insight 3.5 – Installing a Custom Certificate, using CertGenVVD tool

Use Case

Building an SDDC following VMware’s Validated Designs, when adding vRealize Network Insight to the mix, it is possible to use the CertGenVVD tool to generate custom SSL certificates and install them to vRealize Network Insight.

Caveat – I assume you are familiar with the CertGenVVD tool and process!

Process to Generate Custom SSL Certificate and Install 

  1. Copy the existing vrops certificate template text file to vrni.txt
  2. Update vrni.txt CN to the vRNI platform VM FQDN
  3. Add the vRNI platform VM FQDN, short name, and IP to the SAN section
  4. Generate the certificate using the CertGenVVD PowerShell script
  5. Copy the public key full chain “vrni.crt” and place on a secure Linux system accessible to vRNI platform VM
    • When using 2-tier PKI, make sure to append the root CA public key to the end of the …chain.pem file generated by CertGenVVD
  6. Copy the private key to “vrni.key” on the same system (make sure to delete these files securely after completing all the steps here)
  7. Log in to vRNI platform using the CLI user (consoleuser, see default password)
  8. Follow the directions in this KB: https://kb.vmware.com/kb/2148128
    1. In the custom-cert command, the “host” is the Linux system where you copied the generated key files
  9. Verify by navigating to the platform VM URL

Note: Leaving out the fqdn in the SAN section caused Google Chrome to fail validation, although Internet Explorer 10 validated the cert with no issue.

Hope this helps.

vRealize Log Insight Forward to External SEIM with syslog-ng relay as intermediary, using Ansible

Bottom Line Technical Nugget: vRealize Log Insight Version 4.5.0-5654101 Event Forwarding with the syslog protocol will format messages with different formats if there are Tags defined vs. no tags defined in the Event Forwarding Destination entry.  Configuring syslog-ng to expect the newer IETF syslog protocol will only work when vRLI has a custom Tag assigned on the Destination.  Here’s an example of the difference:

Only the second message follows IETF syslog protocol, as understood by syslog-ng.  The first message, where I did not define a custom Tag in the Forwarding Destination, seems to be closer to the legacy BSD format.  For example, notice the subsystem (vmkernel and netcpa) is formatted as vmkernel:  in the first example, and structured data by field position in the second example.  Legacy BSD format uses the subsystemname:  syntax as in the first example.

If that’s all you came here for, great, leave a comment if you want and go back to work.  If you were looking for more of a use case documentation with config examples, read on!

Use Case

Using vRealize Log Insight to aggregate logs in a VMware-centric data center makes a lot of sense.  Intelligent parsing of logs from VMware products and many 3rd party technologies via Content Packs, it is simple, great interface, integrated load balancer, and so on.

If required to forward selected logs to a third party, for example a Managed SEIM scenario, the vRealize Log Insight Event Forwarding provides the ability to do so.  It also provides some filtering capabilities to control which logs are forwarded.  However, its filtering capabilities appear to be fairly basic (at least, what is available through the GUI).  For example, I cannot always take a query created from the Interactive Analytics tool and port it directly into a filter, because not all of the parsed fields appear to be available.  So, if more granular control is required before shipping logs to the External SEIM, or if you simply want another layer of abstraction and control, one possible solution is to deploy syslog-ng as a relay and configure filtering rules on the relay.  This is the solution I’ll be discussing here.

Ingesting

Setup your log sources (devices, nodes, hosts, whatnot) to send syslog traffic in either of the major supported formats to your vRLI integrated load balanced IP using one of the following ports and protocols:

  • 514 udp
  • 514 tcp
  • 1514 encrypted with TLS

You can also get logs into vRLI by installing vRLI Agents on various systems, and also via the Ingest API.  One of my favorite things about vRLI is its simplicity.

Log Insight Event Forwarding

Like ingesting, configuring Event Forwarding is straightforward.  Follow the the vRLI documentation.

The important caveat is that you MUST assign at least one Tag if you want “new IETF” formatted syslog messages emitted from vRLI, as I mentioned at the start of this guide.

In the example below, my custom tag is named “mytag” and the value is “something”.

( Please ignore the LastPass icon that is mistakenly floating in the middle of my screenshot.  It isn’t there if we all pretend it isn’t there. )

Syslog-ng Configuration

Using my go-to configuration management tool, Ansible, I drafted a quick and dirty “playbook” to setup a CentOS system as syslog-ng forwarder.

01-setup-syslogng-relay.yml

There are two supporting files.  Note – I should put variables into variables, so they can be included in the YML playbook, but I haven’t done that.  So in this case, just edit the template files directly.

syslog-ng.conf (removed all the good stuff from the default version)

relay.conf (the main configuration)

With all three files placed in a common directory, and Ansible installed via  yum install ansible, applying the configuration is as simple as one command:

If all is going according to plan, your upstream SEIM should start receiving syslog traffic.  I like to use ss  to do a basic check:

Enjoy!  As always, questions, comments, and feedback are welcome.

vROPS 6.6.1 Custom Certificates Time Zone Bug

EDIT – 10/3/17 Received a response from VMware GSS confirming this as a bug.  Expect a solution in a future release, for now you can use my workaround below.

vRealize Operations Manager 6.6.1 custom certificate import script has a bug related to checking validity period of the certificate.  It will read the certificate time in GMT, and compare to the local system time in local time zone.  It should translate GMT to the local time zone first, but it does not.  This will only affect you if you are close to the extremes of the validity period for the certificate you are trying to install, and the time zone vs. GMT delta is not in your favor.  I illustrate a workaround below, specifically I change the vROPs system local time zone temporarily to GMT.

Steps to reproduce
Generate a custom certificate using the VVD 4.1 CertGen tool on a properly time synchronized Windows Enterprise CA. Note the valid period for the certificate in my case:

Deploy vROPs OVA with time zone = PDT and proper NTP server, verify with:

Import a PEM file using the vROPs cluster installation wizard, and check the /var/log/casa_log/casa.log when it fails:

It appears the vropsCertificateTool.py script is ignoring the time zone information with the certificate and checking the validity date as if the certificate valid not before date was in the local time zone of the vROPS server, rather than converting from GMT to local time before verifying the validity of the cert.

Workaround

As a workaround, I changed the time zone on the vROPs system:

And then upload the correct PEM file in the vROPs cluster installation wizard, and it accepts the cert as valid.  After, you can change your time zone back by updating the symlink.

Note – I have a case open with GSS to report this bug, but probably most people just use the CertGen VVD tool early in the process, and thus enough time has passed between generating the certs and deploying the solution that this is a non-issue.  Hope this helps.

EDIT – See note at top.