vRealize Log Insight Forward to External SEIM with syslog-ng relay as intermediary, using Ansible

Bottom Line Technical Nugget: vRealize Log Insight Version 4.5.0-5654101 Event Forwarding with the syslog protocol will format messages with different formats if there are Tags defined vs. no tags defined in the Event Forwarding Destination entry.  Configuring syslog-ng to expect the newer IETF syslog protocol will only work when vRLI has a custom Tag assigned on the Destination.  Here’s an example of the difference:

Only the second message follows IETF syslog protocol, as understood by syslog-ng.  The first message, where I did not define a custom Tag in the Forwarding Destination, seems to be closer to the legacy BSD format.  For example, notice the subsystem (vmkernel and netcpa) is formatted as vmkernel:  in the first example, and structured data by field position in the second example.  Legacy BSD format uses the subsystemname:  syntax as in the first example.

If that’s all you came here for, great, leave a comment if you want and go back to work.  If you were looking for more of a use case documentation with config examples, read on!

Use Case

Using vRealize Log Insight to aggregate logs in a VMware-centric data center makes a lot of sense.  Intelligent parsing of logs from VMware products and many 3rd party technologies via Content Packs, it is simple, great interface, integrated load balancer, and so on.

If required to forward selected logs to a third party, for example a Managed SEIM scenario, the vRealize Log Insight Event Forwarding provides the ability to do so.  It also provides some filtering capabilities to control which logs are forwarded.  However, its filtering capabilities appear to be fairly basic (at least, what is available through the GUI).  For example, I cannot always take a query created from the Interactive Analytics tool and port it directly into a filter, because not all of the parsed fields appear to be available.  So, if more granular control is required before shipping logs to the External SEIM, or if you simply want another layer of abstraction and control, one possible solution is to deploy syslog-ng as a relay and configure filtering rules on the relay.  This is the solution I’ll be discussing here.

Ingesting

Setup your log sources (devices, nodes, hosts, whatnot) to send syslog traffic in either of the major supported formats to your vRLI integrated load balanced IP using one of the following ports and protocols:

  • 514 udp
  • 514 tcp
  • 1514 encrypted with TLS

You can also get logs into vRLI by installing vRLI Agents on various systems, and also via the Ingest API.  One of my favorite things about vRLI is its simplicity.

Log Insight Event Forwarding

Like ingesting, configuring Event Forwarding is straightforward.  Follow the the vRLI documentation.

The important caveat is that you MUST assign at least one Tag if you want “new IETF” formatted syslog messages emitted from vRLI, as I mentioned at the start of this guide.

In the example below, my custom tag is named “mytag” and the value is “something”.

( Please ignore the LastPass icon that is mistakenly floating in the middle of my screenshot.  It isn’t there if we all pretend it isn’t there. )

Syslog-ng Configuration

Using my go-to configuration management tool, Ansible, I drafted a quick and dirty “playbook” to setup a CentOS system as syslog-ng forwarder.

01-setup-syslogng-relay.yml

There are two supporting files.  Note – I should put variables into variables, so they can be included in the YML playbook, but I haven’t done that.  So in this case, just edit the template files directly.

syslog-ng.conf (removed all the good stuff from the default version)

relay.conf (the main configuration)

With all three files placed in a common directory, and Ansible installed via  yum install ansible, applying the configuration is as simple as one command:

If all is going according to plan, your upstream SEIM should start receiving syslog traffic.  I like to use ss  to do a basic check:

Enjoy!  As always, questions, comments, and feedback are welcome.

Leave a Reply

Your email address will not be published. Required fields are marked *