To Splunk or not to Splunk - Either Way Listen to Your Machine Data

Listen to your Machine Data. Yes, Do.

Machine data and log analytics is all the rage these days, but should it be, and why should you invest to gather, centralise and analyse something as boring and mundane as machine data and logs?

Structured, Semi-Structured and Unstructured

We are used to structured data, and storing such in relational databases and more recently in file and blob based data stores. This kind of data has always been interesting, and used in a business and application context. However, today we are starting to latch on to the power inherent in semi-structured and unstructured data. There are a number of innovative things we can do if we index, store, correlate and analyse all kinds of machine generated data. This will only get more interesting with the proliferation of IoT devices gathering telemetry data.

So what can you do with Machine Data and Logs?

Well, there is no definitive list, and really it's quite open to imagination. Data rules the world. Data driven businesses with new business models are popping up everywhere. In short, data has currency.

For the purpose of this post, let's focus on a common and popular use case in Digital/IT making use of machine and log data in an IT Operations context. Leveraging and mining data to improve IT service delivery, availability and performance makes sense, and adds to an IT department's capability and service offering.

Drivers for IT Ops Analytics

Through correlation and analysis of all our machine data we may be able to

• proactively identify issues
• predict time and point of potential future failures
• pinpoint root cause and reduce mean time to restore
• reduce cost through smarter delivery of services
• gain insights into environments in new ways to drive digital innovation

to name just a few key points.

So this sounds good, right? We want a piece of that for sure. But how do we go about it, and what kind of tools would we need to deliver such new capabilities for IT Operations Management?

Tools that can help us meet the IT Operations Analytics challenge

The good news is that there are a number of mature products and solutions out there. There are both commercial and open source options readily available. The following table lists a few popular options that are worth looking into in my opinion. It is in no particular order nor an exhaustive list.

Products	Commercial / Open Source	On Premise / Cloud
Splunk	Commercial	Both
Sumo Logic	Commercial	Cloud
Azure Log Analytics	Commercial	Cloud
Logstash / ElasticSearch / Kibana	Open Source	On Premise
IBM Log Analytics	Commercial	Both
Loggly / Solarwinds	Commercial	On Premise

On Premise vs Cloud

Depending on what is important to you and/or your organisation, there is no definite answer on what is the best delivery model for a log analytics solution - on premise or cloud.

Some of the reasons as to why you would go on premise are:

• Retain full control of your data
• Flexibility of customisation
• Data sovereignty, data security and backup concerns
• Unreliable or low bandwidth links to cloud providers
• Frequent need to bring back data on premise, egress costs

On the other hand, most of the above cloud log analytics providers are pretty mature, highly available and secure by design these days. In other words, they are Enterprise ready. And of course there is the promise of infinite capacity, so you can ingest data to your heart's content and not have to worry about investing into costly, capital intensive on premise infrastructure.

So unless you are facing major regulatory or compliance hurdles, I'd suggest to give the cloud a go. But do your homework on your projected data volumes and associated costs to avoid bill shock and make sure going into the cloud is indeed the most cost effective path for your business. Major organisations may be able to build their own infrastructure and run at lesser cost than cloud providers.

The Wrap

Hopefully the above thoughts have provided some hints and pointers to get you started on your log analytics journey. Personally, I think the potential is significant, and investing in this space is the right thing to do.

Make sure you have people who are interested in using the technology creatively. Define your use cases, then actively get answers to your burning questions by driving value through analysis and visualisation of your existing log and machine data.

Ah yes, to Splunk or not to Splunk…

Cheers

MB

AdSense Account Stats UWP App Released

Introducing AdSenseAccountStats App

For all those AdSense account owners with Windows mobiles or tablets out there, I recently released a new UWP app that may come in handy. This simple app will allow you to quickly and easily check up on your AdSense performance.

Watch your monetization efforts pay off and your earnings grow

After logging in with your AdSense account and authorising the app for read-only access, you can easily see metrics such as:

• Earnings Clicks, Cost Per Click
• Page Views, Click Through Rate and RPM
• Coverage

Code components that make it work

The app simply leverages Google APIs, specifically the AdSense Management API v1.4, and Google OAuth v2 for authentication of the user.  Basically, all calls to the AdSense APIs have to be authenticated or else they won't work.

So the first problem was to tackle Google OAuth v2 for a UWP app.  There are complexities around the Google provided libraries and support for UWP, and I have touched on these in another blog post.

OAuth v2 Authentication

The following code snippet deals with authentication successfully:

            UserCredential credential;
            try
            {
                using (var stream = new FileStream("client_secret.json", FileMode.Open, FileAccess.Read))
                {
                    credential = await GoogleWebAuthorizationBroker.AuthorizeAsync(
                        GoogleClientSecrets.Load(stream).Secrets,
                        new[] { AdSenseService.Scope.AdsenseReadonly },
                        "user",
                        CancellationToken.None,
                        new PasswordVaultDataStore()
                        );
                }
            }
            catch (UwpCodeReceiver.AuthenticateException ex)
            {
                // Login failed or authorisation not granted
                credential = null;
                await LoginFailed(ex);
            }
            catch (Exception ex)
            {
                credential = null;
                await SomethingWrong(ex);
            }

Creating a client service for your API calls

Once the user has logged on and authenticated their AdSense account, and granted permissions to the app so it can access the user's AdSense details, you then create a client service to use for your API calls.  Pass the credential object created by the code above to the http initializer.

            var adSenseService = new AdSenseService(new BaseClientService.Initializer()
            {
                HttpClientInitializer = credential,
                ApplicationName = "AdSense Accounts Stats",
            });

Setting up and executing an API request

After the user has logged on and allowed access for the app, the next piece is to construct a valid API request, execute it and handle the response in a meaningful way.  This can be achieved along the following lines:

            var earningsRequest = adSenseService.Reports.Generate(startString, endString);
            earningsRequest.Dimension = dimensionString;
            earningsRequest.UseTimezoneReporting = true;
            earningsRequest.Metric = new List<string>{ "earnings","page_views","individual_ad_impressions","clicks",
                                                "page_views_ctr","individual_ad_impressions_ctr",
                                                "page_views_rpm","individual_ad_impressions_rpm","cost_per_click",
                                                "ad_requests_coverage" };

            var earningsResponse = await earningsRequest.ExecuteAsync();

Once you have that response object, you can use it in any way you see fit, for example use its member items as a list source etc.

Hope you enjoyed this post and if you like go grab the app from the Microsoft Store.

Cheers

MB