Dealing with Emergent Data

The recent and ongoing battle with COVID-19 has raised a goodly number of issues in and around getting surprised, arguments around emergent data, and slow/appropriate responses. The lessons learned so far are pretty rich, but I think there is more to discover. Scenario planning seemed to be lacking, the early warning systems seemed to have broken down, responses seemed slow and unpracticed once the denial hurdle was overcome. This was not a "Black Swan" event, so why did this pandemic seem to throw a monkey wrench into humankind's systems and processes? It may a bit early as all the dust hasn’t settled yet, but there are some obvious conclusions even now. What can we say now?

Scenario Planning

While pandemics are a practiced and expected scenario, the level of detail in this kind of scenario was tested in new ways. There were new discoveries in how trends in supply chains were working against us. With longer supply chains for medical supplies and equipment, the stress put on existing supply chains because of panic buying of many items, including food, and how do deal with rescinding demand in finely tuned supply chains. The detailed scenario planning and modeling really seemed off the mark this time on a worldwide basis. There were also some emergent geopolitical effects not completely thought through for sure. Scenario planning needs to handle models with more emergence in a fine-tuned fashion. Businesses and individuals need to up their game in this arena as well.

Early Warning

It is not surprising that less than effective scenario planning would lead to missing emergent data that was not expected, but as the emergent data morphed/changed, there were shadow events, signals, and patterns that took longer to recognize. Early warning needs to be able to recognize event patterns that go beyond expected events. These events and patterns need to participate in more complexity theory and real-time recognition that understands emergence in complex and interconnected systems and supply/value chains. This starts out at endpoint detection and merges with associated and real-time unassociated events to create emergent patterns. Agents/bots sniffing at the edge and event responding in the case of known situations and contexts is minimal, but merging and emergent complexities need to be tested, and models/scenarios need to be updated in near real-time.

Appropriate Responses

It seemed pretty clear that many responses were unpracticed, emergency stores were over-whelmed and tactical responses were being invented on the fly. Since appropriate responses are dependent on in scenarios and early warning, the compounding effect on responses were evident. There were some pretty impressive examples of human creativity/inventiveness and sacrifice to make up for the deficiencies, but can we plan on this always happening for the good of all? We saw governments taking over supply chains, people quarantined late and long, and decisions walking the razor edge between mortality and economic suffocation.

  

Net; Net:

We can do better.  We have to do better as more negative scenarios are emerging as nature deals out an accelerating frequency of earthquakes, hurricanes, volcanoes, pandemics, regional famines, and shifting geopolitical events. I suggest we invest in Emergent Data Recognition (EDR) tied to improvements in scenario planning and practiced responses from a strategic and tactical perspective. There are lessons in emergent data that we have to be ready to leverage. The more prepared we are, the better it will be for us all. It's worth the investment in terms of life and livelihood.

Scenario Planning Is Essential to Survive & Thrive

I have heard many say that it takes bravery to complete a strategic plan and I would agree. I would also say that NOT having a strategic plan that stands up to many practiced scenarios is dangerous. We can see that in the case of COVID-19 in 2020 shows the importance of scenario planning. It's not just the fact that scenarios may have been planned for, they certainly seemed dusty and rusty.  I'm sure there has been some innovative thinking applied to our in-flight adaptation, but the rollout has had more than a few bumps in terms of recognition, speed, and coordination. The results have life-changing for all of us and sadly life-ending for too many.

Scenario Planning Must Stitch Together Strategy, Tactics and Operations

Scenario thinking and analysis generally uses simulation/gaming for policymakers to combine know facts with potential emerging situations in single or complex arenas of economic, geopolitical, demographic, military, resources, and industrial capabilities. This includes tapping models with data and combining them in a predictive fashion until a particular scenario seems to be imminent or happening with anticipated opportunities or threats. These can be detected by leveraging emergent data recognition by looking for signals, events, or patterns. For scenarios that are moving from possible to probable to active, links to emergent and real-time analysis and data sources. This will allow for adjustments to tactics and operations that flex to the best shape possible for the active scenario(s) with optimum operational actions.

Scenario Planning Must Consider Multiple and Ever-Changing Contexts

Often scenario planning can be blinded by a single laser-focused approach on a single probable scenario that leads to the desired future. There need to be exercises that leverage lateral thinking and hybrid sets of complex scenario combinations. The scenario planning effort is not a "one and done" effort, but needs to factor in emergence in the worst case and subtle changes in the typical case. There can be a quick shift from just a scenario to a real possible future and in rare instances probable scenarios. Best practices would expect organizations to identify all likely scenarios and even a few "black swan" scenarios.

Scenarios Must be Practiced to be Effective

Likely scenarios (all of the probable and some of the possible) should be practiced at a reasonable frequency. The practice should include surprise changes to test organizational skills, processes, and system agility/responsiveness. Even though these are necessary drills, all participants must be communicated to as if the situations are real. Many hospitals have evacuation plans that they practice
on a regular basis pretending there is a real event like a hurricane, chemical spill, or flood.

Net; Net:

There is a good lesson for organizations in watching the COVID-19 situation emerge. Organizations can no longer afford not to have up to date scenarios with practiced actions on the shelf that include both the tactics and operations. A communication program for practiced or active emergent scenarios is a must.

Should We Start with Models or Measurement?

There is has been a running battle in and around measuring vs. modeling. It's come to a head with the COVID-19 Pandemic. There has been criticism on the models and their assumptions. There has been a cry for more data before releasing the economy and what speed the restrictions should be removed. While COVID is the issue today, there is a big issue going forward on any number of situations that involve Model vs. Measure. I would like to layout some guidance as a rule of thumb

Models are Great for Emergent Situations

In the absence of sufficient data for a situation that has rarely happened, models are ideal. Models are great for scenario planning, where there are numerous unknowns and untried situations. Models are helpful in operational planning where new approaches are being tried out before a significant investment of funds is necessary to change operations. Models are also helpful for people to understand complex aggregations of data from known situations. Models have a great impact when used properly and improved with real data over time.

Data is the Great Equalizer or Diviner of Truth

Having real data for known operational problems is the sweet spot of improvement and automation. There is little substitute for real data leveraged with know and true scientific methods or measures. While models might help visualize the data, the data is the key to assured decisions and lower risks. Some would say that all models are flawed; let's stick with the data. In fact, some would go as far as saying that the money and time used for modeling should be used to clean and normalize data and data sources. They have a good point for known and stable operational situations.

Models and Data Need Each Other

The real truth is that models nor data are ever perfect. Knowing this, we need to exercise wisdom in how to leverage both together, if possible. Models need more data to become more accurate, and data need models/algorithms to make the data digestible and in a form where the best the decisions have a chance to be made. I think the real issue is which one to start within any given situation. This is especially true where there is disagreement. Models let one try things out, and data, if correct and up to date, represent current conditions very well.

Net; Net:

Start with models for fuzzy and emergent situations. Start with data for known problem domains, hoping to discover opportunities.  Work towards a balanced and dynamic relationship between models and data no matter where you start.

Thank You For Your Support!

It's been seven full years since I left Gartner to pursue helping others progress with various forms of digital technologies. My commitment has been to help push digital technology forward in supporting real business goals, competencies, and progress. I am delighted to still be adding value to our collective efforts. I really want to thank my readers for continuing to be interested in my writings.

There were those who:

Learned something new
Were encouraged to think differently
Corrected my grammar, spelling, etc.
Corrected my thinking
Encouraged and extended my points
Pointed me to parallel efforts
Pointed to areas of opportunities

I want to thank you all and hope to continue to add value. So far my blog has had nearly 650,000 hits over the seven years spread across 537 posts yielding an average of over 1200 hits per post. My reach is now on Forbes and Data Decisioning where I have a pretty new presence. Some of the details of the activity are posted below. While most of the activity has been in the US (67%), there is a wide interest internationally. If you have ideas for further writings or just want to discuss something, please reach out to me at jim.sinur@gmail.com.

Net; Net:

Thanks for your continued support!!!  Stay in contact :)