Prioritizing Data Management in the Monster Data Era

As much as organizations would like perfect data under their complete governance/management rules and guidelines, those ideal outcomes disappeared already with the big data era's reality. In the "monster data" era, it will be much rarer to have complete data management, especially with the inclusion of more data types, more complexity, and less reliable data sources coming at organizations at an accelerated speed. This writing aims to give some advice on deciding which data sources are likely to get more intense data management. Prioritizing will allow data management to focus better on balancing its efforts. It will also enable the data usage to be separated from Ideal/excellent data management allowing new data resources to enter the organization and grow into data management over time. Prioritizing data sources for robust data management should consider goals, measures, and change channels.

Prioritize on Critical Goals

Organizations can't focus on all the data coming their way, and they certainly can't depend on AI supercharged data scientists to find interesting anomalies from the bottom up. The first cut at the data under management should directly relate to business/organizational outcomes necessary as a base. Many organizations set up strategic goals that represent the most desirable for the circumstances at hand. Savvy organizations plan alternative strategies/scenarios and practice the most likely emergent strategies. Most organizations, these days, tie both their functional and operational goals to the strategy du jour. Savvy organizations look at the end to end processes/journeys to link goals together to minimize organizational conflict and consider constituent outcomes as legitimate goals. These efforts would include customer journeys laced with highly desirable customer experiences. These goals will help narrow down the data sources that need proper data management, although much more than in the past.

Prioritize on Real World Measures

An excellent place to start is where most of the data activity occurs. Many organizations create heat-maps of highest use and apply the highest level of governance for these sources regardless of whether they are operational or decision focused data. There more enlightened organizations push the data timing to real-time, representing the Database of Now. These same organizations practice data mining to look at the reality of the current and past situations that may point to future operation optimizations. The aware organization also mine new data sources like voice and video for customer experience improvement potential. While much of the mining is descriptive, the more advanced efforts start prescribing potential for change and even predicting future behaviors. In this kind of environment, decision optimization is considered a major important improvement arena.

Prioritize on Change Channels

Like it or not, things change and almost always get more complicated. Rapid change may mean cobbling together creative responses in a quick fashion. To that end, organizations need to sniff many data sources to change potential impacts that could affect strategy or operations. Sometimes it might be subtle to look at signals and events in these data streams, but it could range to finding emerging patterns within or across data sources. The likely channels would often include competitors, markets, industries, demographics, vendors, suppliers, geographies, legal frameworks, and governmental pushes. Additionally, best practices involve finding new contexts and sources.

Figure 1 Optimal Data Governance Defined

 

Net; Net:

Applying excellent data management governance, depicted in Figure 1, is no longer possible to all data resources. Organizations will have to focus on the data sources that are key for business outcomes that enable organizations to survive, thrive, and even optimize their impact. While Data Management is necessary for specific resources at a high level, the time has come to identify those particular resources and set data governance levels across various data sources. 

 

 

 

 

 

Monster Data is Headed Our Way

If you thought Big Data was a challenge for us all, wait until the new wave of Monster Data hits us. We will have to manage it, make decisions, and build processes and applications leveraging this monster data. Just what is monster data, and how will it affect us. 

Monster Data represents data that is overwhelmingly large, unduly complex, can’t be trusted for accuracy. Typically it is composed of multiple kinds of data including structured, unstructured text, voice, image, or video. Some monster data may be unknown or emergent, making it scary to deal with for most individuals, technologies, or organizations.  

Even Larger Volumes        

We have long been concerned about “the IoT Awakening," exposing large amounts of critical data that would likely need immediate attention often at the edge. While managing all the moving parts of Industry 4.0 is a challenge, we see new value chains that employ GPS, tracing, and original digital identities adding data to the mix. As organizations want to leverage data for more refined business outcomes, more data will be needed.

Organizations leverage more powerful AI and computer-analysis techniques to gain insight into human behavior using personality, social, and organizational psychology data. This need will yield data sets that are much larger than what we have today and certainly too large for traditional processes and applications. Data will likely include recorded conversations that could process into usable information.

More and more data is piling up from digital footprints left in social media, cell phones, business transactions in various contexts, shopping, surfing, and other devices that record our every moment, freely given or not. Sometimes this new data is just taken from sites as people pass through, leaving crumbs behind.

Even More Complex  

To order to utilize technology to empower us, the complexity of the data will also become much more diverse. Because large data collections can be computationally analyzed to reveal new signals, patterns, and trends, the complexity of that data will have to be managed well.  Organizations want to deliver insights from human behavior and interactions collected everywhere, every second of the day.

The data will come from various contexts that imply context-sensitive meaning. Hopefully, this new and emergent data will be available on the cloud, but the cost and security issues will make this data more hybrid cloud in nature. The data will likely be a hybrid of structured and unstructured data and require new data management means with ownership and dynamism challenges.

This complex and dynamic set of data sources will become more challenging to manage, but it is on its way to becoming a precious asset that can be leveraged by machine and deep learning. While dynamic and emergent, its use will become more stunning over time.

Even More Inaccurate

Because of speed and size alone, the accuracy of monster data will be a constant challenge. When combining data in new ways understanding its source, context, and ultimate meaning at all levels of granularity, this becomes more of a critical problem for the data management professionals as well as the end-users.

There will be ownership issues and who will be held accountable for the accuracy of any data leveraged. All of this will have to be sorted and managed under the gun with the pressure of speedy results. Of course, internal data sets will have a better-understood pedigree than those data sources from outside an organization and in contexts not well understood.                    

Net; Net:

As we grow to zettabytes, the amount and variety of data being accessed, collected, stored in the cloud, stored on-premises, and analyzed will keep increasing in an exponential fashion. This seems like a near-impossible task until the promise of better analysis and prediction to correct problems takes over our desires. Business outcomes will likely drive this growth in this extreme competition and dynamic environment these days.

 

Decisions Need to Drive Data Science

There has been and will continue to be a significant shift in how data is leveraged in this continually changing world. Until recently, the data science process of collecting, cleaning, exploring, model building, and model deployment ruled the data management mindset. In a world that is "steady as she goes," this makes a great deal of sense. The amount of data to be curated is growing impressively, but the data science mindset is still on the scene dealing though pressed to its limits with big data. Two things will break this sole reliance on the data science process. 

Dynamic Business Scenarios

In a world with operational KPIs staying steady with minor adjustments over time, focusing on data makes sense. That world is virtually gone with the elephant in the room being a pandemics at the moment. Tomorrow it could be natural disasters and the down-stream effects of climate change impacting geopolitical behaviors. There are many business scenario possibilities and combinations headed our way. We can't afford just to explore data and have knee jerk responses.

Monster Data is Lurking

If you think big data is worthy of concern, just think about the monster data just around the corner, driven by higher volumes, more complexity, and even more inaccurate by nature. Organizations are bound and determined to take advantage of behavioral data that is further away from standard core operational data. Monster data includes all kinds of unstructured data that will contain digital footprints worthy of new types of decisions.

Either of these would require a major addition of new data processes, but combined data science processes alone just won't suffice. I am not saying that data science will dim, but it needs some new additional turbocharging and methods that are not just focused on exploring structured and clean data.

Dealing with Changing Scenarios

There are several ways of dealing with scenario planning and practicing responses, but here is what I would encourage organizations to do. Many decisions will drive the data that is leveraged during these efforts.

• ·        Plan probable scenarios by having executives brainstorm and list likely scenarios and their outcomes.
• ·        Simulate and practice these likely scenarios, so they become part of the muscle memory of an organization. It will involve leveraging key data sources cascading to tactics and operations. Build communications mechanisms ahead of time and communicate readiness.
• ·        Identify unlikely dangerous scenarios and simulate the effects and plan responses appropriately.
• ·        Identify critical decisions, events, and patterns to scour appropriate data resources (owned or not).
• ·        Identify key leverage points in processes, systems, applications, and the data that could be involved

Dealing with Changing Tactics

Middle management is always trying to optimize outcomes for their functional areas though savvy organizations try to link results to remove friction points for overall optimization. Optimization often leads to self-imposed changing goals that need to be operationalized or tweaked in operations. When executives want different outcomes based on a refined organizational charter, new governance rules, and critical trends delivered by business scenarios in place, a bigger picture is in play. Tactics are the essential glue to hold together operational outcomes guided by goals. As these goals shift in a dynamic set of business demands, managers would be wise to be ready for new guidance coming at faster speeds by following this list of practices.

• ·        Understand the impact of significant changes by modeling or simulating the effects of change.
• ·        Be aware of all executive expected sets of scenarios and search for critical events and patterns to detect new scenario emergence.
• ·        Implement various approaches to near real-time responses, including digital war rooms, dynamic process/application changes, and low-code methods.

Dealing with Operational Change:

Typically operational processes and systems are in place to deal with day to day operations. Changes in behaviors, markets, tactics, or scenarios will cascade down to operations. There may be additions and changes to procedures dictated by outside factors over the routine operational optimizations that occur on an ongoing basis. Processes tend to be more stable, but some changes could rock the house. To deal with functional change, I would encourage the following activities.

• ·        Model key decisions that affect KPIs and desired business outcomes.
• ·        Generate procedures from the models—manuals for human resources and code for processes and applications.
• ·        Perform a volatility analysis based on past changes to identify hot spots. Enable hot spots for change, particularly for code using late binding techniques or low-code.

Net; Net:

We are entering an era of significant change linked to constant change. It means that just shining and studying data alone does cut it as a sole strategy. While data affects decisions as they are made, deciding what is going to change is emerging as a dominant new organizational competency area. We need to add some new disciples/practices to thrive going forward and call it Decision Science.