Monday, May 25, 2015

Dark Events; An Introduction to the Unexplored Area of Digitalized Darkness

Being unaware in many areas of life have risks for all in involved. The same is certainly true for organizations. It is imperative in a changing world to have better than 20-20 vision to anticipate appropriate decisions and actions to deal with activity that grows in the dark. Dr. Edward M.L. Peters defines dark events below:

The Digital Darkness Trifecta:

With the ever-expanding universe of digital information, concepts about what we do not know are also emerging.  Gartner has defined the term “Dark Data” to describe data that are collected, processed and stored by organizations only to be put to little or no use.  As an example, thy cite data that are captured and stored for compliance purposes as well as analytics data. Additionally, Jim Sinur, formally a Distinguished Analyst at Gartner, has defined the concept of “Dark Processes” (  According to Sinur, “…it is a process that is hidden or partially blind as it executes”.  In other words, it has become part of an undocumented process with rouge tasks executing outside of the official workflow.  Now, to this dynamic duo of digital darkness, I propose an addition, “Dark Events”.  Dark Events are typically part of a business process and are unknown and unreported due the nature of the data collection technology provided by most workflow and Enterprise Resource Planning Systems.

Dark Events: an example

Have you ever called a taxi service and been given an expected arrival time, only to find that after 30 minutes or more, the ride has yet to arrive? So, frustrated, you place another call to the dispatcher and, once again, you are told that the taxi should arrive shortly.  And after yet another 10 minutes; still no luck!  Well, the last time this happened to me, I was sitting at a restaurant in a residential district in N.W. Washington, D.C. and decided to download Uber, set up my account and request a car.  Immediately, I was aware of a number of things; first, my request was accepted, the driver was identified, a mobile number with a name a picture of the diver was provided, an estimated pick-up time was noted and maybe best of all, a map was presented that showed my location, the location of the driver and the car’s progress toward me with the time to arrival decremented as the driver approached.  I had all the information I needed to reduce the uncertainty of not being picked up.  In the dispatched taxi case, all of the information between the request for a ride and my current situation was dark; I had no way of knowing anything about the expected pick-up from when I placed the call to the taxi company to when the driver would, if ever, arrive.  In the case with Uber, the entire event from the call to the actual pick-up was visible to me, giving me a high level of confidence in the process as well as the expected outcome.  In both cases, what we are talking about is a process which consists of a series of events. Once again, the events are the same as in both cases. The difference is that in the dispatched taxi process, most of the events are dark, meaning that I had no visibility into what was happening, leaving me with simply the faith that the driver would appear: or not!  In data science terms, this illustrates the concept of “opacity” whereby, much of the phenomena are unable to be seen, hence are opaque. 

Dark Events in Organizations:

As with the case of the opaque taxi process, Dark Events are an everyday occurrence within most organizations.  Sometimes they called “dark spots”, places where it is known that the information about the event is incomplete but there few solutions that will help provide visibility into the situation (e.g., a known unknown).  Similar to dark matter in physics, Dark Events comprise a significant percentage of the universe of events within a typical business process. Research has shown that Dark Events can outnumber knowingly captured events, those captured by business process focused IT systems, by a factor of 15 leaving 60 % of the organizations events un-illuminated.  Imagine the effect on enabling desired outcomes through the use of digital information if actionable decisions are being taken based on understanding what actually occurred in less than half of all the events?  It does not matter how good your analytical models are if the data are grossly incomplete! 

Dark Events; next topic: 

The next post will discuss Dark Event discovery and analysis showing a path toward illuminating the “unknowns”.