Forwarding Centrify Event Logs to VMware Log Insight using REST API

If you use both Centrify and VMware Log Insight, and need to pull event logs from Centrify into Log Insight (in near real-time), this tool is for you!

This tool queries the Centrify API for your tenant, pulling the most recent entries in the Event table (logins, application launches, configuration changes, etc.).  Then, it structures the event data into a syntax that can be imported to Log Insight, keeping event time accuracy and offering a series of structured fields to Log Insight to enable easy creation of dashboards, analytics queries, and so on.  All the good stuff that Log Insight offers.

I wrote this to be modular and and it should be relatively easy to adapt for other data sources and destinations.  So even if you are using only one of these two solutions, but trying to solve a similar problem, my code might be a starting point for you too.

Install this as a cron job, by scheduling run.sh on a 5 minute interval on some server.  You place some configuration in a config.py file, and that’s it.  Very simple.

Recurrent Neural Networks, Python, and the TV sitcom Frasier

Frasier, without the (recorded) laughs…

At a recent dinner party, some friends and I wondered: What would the TV show Frasier be without the audience laugh track? And further, could machine learning be used to automatically mute out the undesired audience audio?

Well actually, Frasier… with audio feature extraction visualized in the lower corner

The Code

So, I built a tool to automatically mute the sections of audience laughter.

The repository includes a Jupyter notebook with detailed explanation, example video clip with processed audio, a trained model, and the CLI tool to apply this to your own audio files.

I wrote this in Python and used several open source libraries, notably Keras for describing a multi-layer recurrent neural network, librosa for audio feature extraction, and (of course) numpy to manipulate the data.  It’s amazing what you can do in under 300 lines of code, thanks to contributions of the open source community.

More detail

What makes this an interesting problem, in my opinion, is that the “laugh track” audio is highly variable, real studio audience recordings, so properly classifying the audio required a more complex machine learning approach. Using a model that considered time-series data, rather than simple point-in-time consideration.

I used a number of fantastic tools and resources along the way – starting with Jupyter to rapidly prototype different approaches to the problem, and visualize the data as I went along. After I had built a solution I was happy with, I worked to convert it from a notebook, which is good for reference, exploration, and demonstration, to a standalone Python tool with a command line interface, which can be run on-demand to train or apply the model.

What’s next?

In theory, this tool could be trained on any two classes of audio, and then used to mute out one of those two classes. I don’t believe there is anything particularly special about identifying “laughs” vs other kinds of distinct audio!  All my code is released under MIT license, so you are welcome to do whatever you’d like with it.

Enjoy.

Deleting custom adapter kinds from vRealize Operations Manager 6.6.x

Summary

If you need to delete custom adapter kinds and custom resource kinds from vROPs 6.6.x, VMware support can create a .pak file tailored to your resource kind names.  By installing the .pak file as a solution to vROPs, and then uninstalling it, it will remove the custom adapter kinds entirely.

There is no way to do this thru the GUI or API.  I’d like to see VMware add a REST DELETE method to support this in a future release of the product.

Context – Adding Custom Metrics in vRealize Operations

Let’s say you want to use vRealize Operations Manager to track metrics for some kind of resource that isn’t already available in the solution adapter ecosystem.  For example, performance and capacity information for Cloudian Hyperstore such as S3 operations per second and bytes used by group.

This is totally possible using the vROPS REST API, and you might even use Python to do it, possibly with the vROPS Python client “nagini”.  This has been documented several times already.

What happens when you add custom metrics via the REST API?

When you push custom metrics, you define a new adapter kind and resource kind(s).  vRealize Operations Manager creates a new adapter of type “OPENAPI” to track your objects.  They look like this in the Environment tree:

Unfortunately, it is not possible through the REST API nor the web user interface to DELETE these adapter kinds and resource kinds after they have been created.  You can delete the instances of the adapters and resources, but not remove the kind from the schema.

Why do we need to remove custom adapter kinds anyway?

This can be a problem for several reasons.  For example, when developing against vROPs, certain characteristics of resources are defined the first time the resource is created and cannot be modified later.

In particular, using custom resourceIdentifiers, used to identify objects by a set of IDs beyond simple object name matching, the custom resource identifier fields must be defined the first time the resource is created, in my experience anyway.

From the Administration -> Inventory view, here’s an example of custom resource identifiers, all 4 items listed under “Basic Settings” in the screenshot:

The only way to modify the schema here, as far as I am aware, is to entirely remove the resource kind and recreate it ensuring all desired custom resource identifier fields are defined on the first push to vROPs.

Hope this helps.

VMware vROPs custom resourceIdentifiers API Error 1501 with Nagini.py

When pushing custom resource kinds using the vROPs Python nagini client’s find_create_resource_push_data() method, and you include custom resourceIdentifiers, you might get an error:

You can fix this by creating your own version of the find_create_resource_push_data() and find_create_resource_with_adapter_key() methods.  You need to handle the API exception, and treat the error condition as if your call to find the resource succeeded but returned no results.

This code is just an incomplete example.

vROPs Python Client and Unexpected “Resource not found”

Using the vRealize Operations Manager 6.6 Python nagini client (version 1.0), I encountered an interesting unexpected result when using the client to push custom metrics to vROPs.  For context, see this blog post on creating a custom metric adapter for vROPs.

This error occurred when pushing resource metrics to vROPS:

In particular, last line indicates the vROPS API returned an error “resource… already exists”, which means the nagini client is attempting to create the resource:

The method find_create_resource_push_data() is supposed to find existing resources and not attempt to recreate them. So what is going wrong?

To figure this out, I followed my normal debugging procedure of breaking this down into smaller parts. Reading through the nagini client Python code, the nagini client makes a call to get_resources_with_adapter_and_resource_kind() and parses the results to determine if the resource already exists.

So, I ran just that method with my specific parameters, and to my unfortunate surprise received two results back – two resources were found. However the method find_create_resource_with_adapter_key() expects only a single result, otherwise it goes into its “create resource” mode. A more helpful error message here would be great and easy to add, something like “if len(results) > 1 throw an error that you have too many resource hits.” Oh well.

From the API documentation page for the underlying method, getResourcesWithAdapterandResourceKind, I identified the root cause:

Query for Resources within a particular Adapter Kind and Resource Kind.
Optionally filter these resources based on resource name. The resource name (specified as a query parameter) will be used for doing a partial match.

In my case, the specific resource name is also a subset of another resource name.  Hence the bug.

Without any way to control partial vs. exact match in the API currently, my two options are to re-implement this portion of the nagini client logic and do the exact name match filtering in Python, or to use the richer resource identifier method which allows specifying multiple keys for a resource to help identify it with more than simple name matching.

Hope this helps.

 

Recent Posts

Categories

Archives

GiottoPress by Enrique Chavez