Is Chasing Perfect Data a Reasonable Quest?

We have heard many quotes about the poor quality of data. In fact, there are those that want perfect data before they make a decision, Is that a realistic attitude towards data quality? While in some situations where data that is nearly perfect is an absolute must, there are other situations where you make the best decision under the circumstances. Let's Explore some of the issues a bit more.

Figure 1 Interaction of Data Quality and Decisions. 

What is Data Quality?

Gartner defines quality this way:

"The term "data quality" relates to the processes and technologies for identifying, understanding, and correcting data that support effective data and analytics governance across operational business processes and decision making. The packaged solutions available include a range of critical functions, such as profiling, parsing, standardization, cleansing, matching, enrichment, monitoring, and collaborating. 

I'd like to add my analysis to this solid base by referring to Figure 1 above that tries to show the interaction of time, decisions, and data. Looking at the X-axis we see the data quality increasing as efforts to make it better move it to a more clean, concise, and crisp state. The Y-axis represents a time continuum that goes from right this instant to all the time ever needed. Given all the time and all the money necessary, data can approach or even attain perfection until entropy enters into the equation. 

When data is new or first brought into an organization it is good for emergent and morphing sets of problems that are often under pressure to make decisions and even take actions on those decisions. Things are fuzzier at this point and a precise answer is often not possible, but progress can be made even with data of less quality. Often the decisions needed are not fully understood at this point in time. 

As the decisions become more known and even routine, the priority of the crucial decisions, goals, and outcomes tend to sort themselves out. This helps identify the critical data sources that need attention and efforts to get better over time. Some decisions become so important that operational excellence and customer interaction drive the need to make the decisions excellent too. This excellence demands more prefect data in most instances. 

When is Perfect Data Needed? 

Circumstances that Demand Perfect Data

When it comes to safety and the lives of people, it is hard to argue against perfect data, The problem occurs when the timing of getting that data perfect flies in the face of a need to make a decision to avoid downstream negative consequences. Sometimes decisions have to be made with less than perfect data. There are two strong forces pushing here that have to be balanced within the context of emergent or known scenarios.

Good Enough Data Sometimes Works

While it is easy to demand perfect data in order to make decisions and take action, sometimes real leadership finds good enough data to move forward. There is a danger to rely on just gut feel, so leveraging the data the best way possible considering its flaws is sometimes the best road taken. Sometimes additional scenario simulation is the answer. Sometimes some quick and dirty approaches to incremental data bolstering make sense. It may mean that you look at other similar decisions and data for insight.   

Net; Net: 

While we all chase better data, all data can't be perfect, so don't die trying. Certain decisions demand perfect data because of their importance, but the critical nature of decision timing fights against this desire. It is very easy to sit back and say that all data has to be perfect and not take any responsibility or action because the data isn't perfect. Don't get caught in that trap. On the other hand, don't stop investing in great data quality because it might cost some effort. There is a delicate and dynamic balance to be struck here.