Sometimes organizations think that they can plan the typical business and process scenarios and even have planned responses, but sometimes the "The Black Swan" shows up. No organization is omniscient, even if they have some of the best minds and experience. Dark events are bound to happen even though these dark events are more likely than black swans. In the last post, I briefly discussed the concept of Dark Events as being something that belonged to the area of the unknown.
“There are known
knowns. These are things we know that we know. There are known unknowns. That
is to say, there are things that we know we don't know. But there are also
unknown unknowns. There are things we don't know we don't know.”
As described in the quote above
from former US Secretary of Defense, Donald Rumsfeld, this can be either a
known unknown, like a dark spot where you know something is going on and are
aware you don’t have the data or something that is totally unknown. In the latter case, it is an unknown, unknown
or something that is totally surprising.
In both cases, we need some way provide transparency into an area where
the activities are not readily observable.
For our purposes here, let’s focus on business events. Most business events are part of business
processes and the information is captured through business IT systems. Normally, business executives rely on business intelligence (BI)
solutions that focus on schema-driven reporting on transactions and documents
or from business activity monitoring (BAM) solutions that provide pre-defined
dashboards and metrics. Unfortunately, these and other silo-centric analysis
type solutions provide only a partial view of what is really occurring. What we are interested in exploring are the
spaces between the actions we currently “know” exist, the “known knowns’” and
determine if there are any events that are occurring but are not being observed
In the figure above, we can see the known steps (shaded in light blue) in a typical
transaction system. However, between
steps 4 and 5 (Transaction Paid) are
a number of other steps that are happening (shaded in gray) that detail a customer
service intervention process that corrects the data on the customer transaction
so it can process to conclusion. These
steps are part of a “Dark Process” that encapsulates the “Dark Events”.
Dark Events are
Temporal
Most
organizations have events logs where they track and store data such as a record
that a customer was on the self-service portal, an order placement or a call
record. However, as in the example above, these logs are usually information
poor. Dark Events, by their very nature, are time-based and found in the
spaces between “known events”. Each
event is an instance of task execution, comprises a definitive unit of work and
has a time-based start and stop tag associated with its existence. While they are additive to “known events”
they are do increase their overall time but rather provide descriptive detail
about how the duration of time is being used.
In this way, understanding the number and duration of Dark Events
provide the intelligence required for making decisions about their improvement.
Dark Event Analysis (DEA)
Dark Event Analysis focuses on the time-based reporting
of process execution information, differentiating it from BI and BAM. DEA provides this insight because it analyses are
at the ‘event’ level. Events are a new source of data for most organizations. DEA
provides analytics and drill down capabilities to the details of the process execution
in order to understand both time gaps that identify Dark Events as well as
their effect on the root cause of process variations. Dark Event Analysis focuses
at the level of the work steps as they occur, no matter where they occur.
Dark Event Analysis Techniques- Mining vs. Discovery
There
are a number of ways to approach both the discovery and analysis of Dark
Events. The two most commonly used
techniques are Automated Business Process
Discovery and Process Mining.
The
concept of Automated Process Discovery
was first discussed in 1996 by Cook and Wolf in an attempt to understand and improve
the software development process through the use of algorithmic techniques
(e.g., neural networks, Markov models, etc.). This differs from conventional process
analysis techniques (such as Six-sigma and Lean Management) that are based on a-priori biased models. While these
representations are good for detecting high level and readily observable issues,
the data they capture is insufficient to understand the essential time spent
between observable events as well as the nature of variations, exceptions and
deficiencies that are being promulgated during event, task and process
execution. Automated Process Discovery
often involves the use of platform specific data collectors that are designed
to capture the complete picture of what occurred during a specific time period. These data are then stored, structured and
analyzed to reveal the entirety of the event record as the basis for
improvement.
Process Mining is a statistical approach
to understanding the "general behavior” in processes, in the way they are
observed in the working environment. Process Mining looks to interrogate
business system event logs as the basis for understanding the behavior of
business systems. The statistical
techniques in Process Mining often concentrate on finding the “main” or
“average” process that is being executed. This information is used to build a model of
the way the system is actually operating as a basis for comparison against
expectations and a starting point for discussions about improvement. Further, the models produced through Process
Mining can be used for simulation and other predictive modeling techniques
providing the basis for an advanced analytical capability.
Both
of the approaches described above can be quite useful in both understanding business
events and analyzing how to improve them.
When it comes to Dark Events, Automated Process Discovery tries to go
further than Process Mining by capturing more detailed data than usually exists
in business system event logs. It is by
capturing and analyzing this data that we illuminate the “un-knowns” and increase our ability to make meaningful
improvements.
Dark
Events; next topic:
In the next post, Jim Sinur
and I will discuss Dark Events and their relationship to Dark Processes and
Dark Data.
