Listen to your Machine Data. Yes, Do.
Machine data and log analytics is all the rage these days, but should it be, and why should you invest to gather, centralise and analyse something as boring and mundane as machine data and logs?
Structured, Semi-Structured and Unstructured
We are used to structured data, and storing such in relational databases and more recently in file and blob based data stores. This kind of data has always been interesting, and used in a business and application context. However, today we are starting to latch on to the power inherent in semi-structured and unstructured data. There are a number of innovative things we can do if we index, store, correlate and analyse all kinds of machine generated data. This will only get more interesting with the proliferation of IoT devices gathering telemetry data.
So what can you do with Machine Data and Logs?
Well, there is no definitive list, and really it's quite open to imagination. Data rules the world. Data driven businesses with new business models are popping up everywhere. In short, data has currency.
For the purpose of this post, let's focus on a common and popular use case in Digital/IT making use of machine and log data in an IT Operations context. Leveraging and mining data to improve IT service delivery, availability and performance makes sense, and adds to an IT department's capability and service offering.
Drivers for IT Ops Analytics
Through correlation and analysis of all our machine data we may be able to
- proactively identify issues
- predict time and point of potential future failures
- pinpoint root cause and reduce mean time to restore
- reduce cost through smarter delivery of services
- gain insights into environments in new ways to drive digital innovation
to name just a few key points.
So this sounds good, right? We want a piece of that for sure. But how do we go about it, and what kind of tools would we need to deliver such new capabilities for IT Operations Management?
Tools that can help us meet the IT Operations Analytics challenge
The good news is that there are a number of mature products and solutions out there. There are both commercial and open source options readily available. The following table lists a few popular options that are worth looking into in my opinion. It is in no particular order nor an exhaustive list.
Products
|
Commercial / Open Source
|
On Premise / Cloud
|
Commercial
|
Both
|
|
Commercial
|
Cloud
|
|
Commercial
|
Cloud
|
|
Open Source
|
On Premise
|
|
Commercial
|
Both
|
|
Commercial
|
On Premise
|
On Premise vs Cloud
Depending on what is important to you and/or your organisation, there is no definite answer on what is the best delivery model for a log analytics solution - on premise or cloud.
Some of the reasons as to why you would go on premise are:
- Retain full control of your data
- Flexibility of customisation
- Data sovereignty, data security and backup concerns
- Unreliable or low bandwidth links to cloud providers
- Frequent need to bring back data on premise, egress costs
On the other hand, most of the above cloud log analytics providers are pretty mature, highly available and secure by design these days. In other words, they are Enterprise ready. And of course there is the promise of infinite capacity, so you can ingest data to your heart's content and not have to worry about investing into costly, capital intensive on premise infrastructure.
So unless you are facing major regulatory or compliance hurdles, I'd suggest to give the cloud a go. But do your homework on your projected data volumes and associated costs to avoid bill shock and make sure going into the cloud is indeed the most cost effective path for your business. Major organisations may be able to build their own infrastructure and run at lesser cost than cloud providers.
The Wrap
Hopefully the above thoughts have provided some hints and pointers to get you started on your log analytics journey. Personally, I think the potential is significant, and investing in this space is the right thing to do.
Make sure you have people who are interested in using the technology creatively. Define your use cases, then actively get answers to your burning questions by driving value through analysis and visualisation of your existing log and machine data.
Ah yes, to Splunk or not to Splunk…
Cheers
MB
No comments:
Post a Comment