Are You Just Digging in Your Data to Make Decisions?

It is very tempting to want to dig in your data to find opportunities to find business gold, because it really appeals to get busy to mine patterns. It's a bit like wanting to run dirt into a sluice to find gold. The problem is that not all dirt has pay in it and hoping to accidentally find some seems a bad bet. There are a number of factors that make this a low probability approach.

The Amount of Data to Consider is Ever Increasing

Finding gold is easier than finding great decisions, because gold is usually found near water or in gold bearing land. There are limited areas known for gold. Data however is ever increasing by the second and most of what we do is store it for future refinement. While we can point to past successful sources, there is more data created than we can possibly refine. As Quantum computing and  other forms of speed processing come on board, the chances will get greater for finding nuggets, but it seems like a solution looking for a problem. Its just pouring through more data.

The Speed of the Data is Ever Increasing

We are finding that fast data is complicating the probability of finding finding and acting out great decisions because the window of opportunity is quite short. Fast data is moving quite rapidly and it usually contains, signals, events and patterns that lose their importance or value quickly. This data works best when it is analyzed in real time or saved temporarily to be associated with other data. It can be mined in retrospect, but the value has usually waned. Speed is a compounding effect to big data. With IoT this is a huge challenge that maybe bots/agents at the edge can help with in some cases. 

Having the Right Algorithm is Also a Challenge

Not only is the data coming faster than we can deal with it and the amount is over whelming, matching the data up to the proper algorithms or combinations of algorithms seems a difficult approach. It seems like trying to A/B test these algorithms alone or in combination a near impossible task. Of course, new forms of technology will help, but again this seems backwards to me. It seems too inside out of an approach and trial and error in nature.

Net; Net:

The odds are stacked against coming up with brilliant business decisions by just digging into data even with faster computing. While we hear of examples from time to time, but finding great nuggets that way is a lot of work and does not always pan out. While I wouldn't abandon looking at data for opportunities, especially with machine learning, deep learning within the bounds of explainable AI, but starting with great decisions and working to the data and algorithms makes more sense to me. Successful miners narrow down where the gold is likely to be for the payoffs they are seeking.