In recent posts Edward M.L. Peters has been illuminating one of the newer areas of unknowns in describing dark events and their potential impact, negative and positive, on organizations. This post we will concentrate on integrating the three major areas of darkness and their interactions. The three areas are dark processes, dark data and dark events.
A dark process is set of actions that are not visible to the owners or managers of a process for mass adoption, change, improvement or dissolution. Dark processes steal opportunities for more benefits. There are many reasons that processes are dark, but many are a result of just dealing with the emergence in business which will accelerate in the digital age. Almost all processes are subject to losing their effectiveness over time due to exceptions, dealing with customer differences and using processes in new contexts such as a piece of an new end to end process or a in a new legal framework. Process variation is the key driver for dark processes and variations will become the norm rather than the exception as we transform to digital.
Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing). Technopedia adds that dark data is a type of unstructured, untagged and untapped data that is found in data repositories and has not been analyzed or processed. It is similar to big data but differs in how it is mostly neglected by business and IT administrators in terms of its value. They also add that dark data is also known as dusty data. Whether structured or unstructured, analyzed or un-analyzed, a few thighs are true; dark data are often collected and are not seen as providing business value. In terms of dark events, if the events are dark, then the data that describes their duration and existence is necessarily dark, whether it already exists in some form or is yet to be captured. In other words, if there is a knowledge gap in what one knows about a process (e.g., dark event(s), it can be closed only by either providing a new capture method for the data that describes the event (e.g., new collector) or by analyzing the data to actually “discover” the events from the data themselves. This latter approach often involves analyzing log files and databases as the starting points for understanding how data are clustered around events and how events are similarly clustered into process activities or tasks.
A dark event, as described previously, is an occurrence that happened inside or around a process that might make a business difference, if one knew about it happening. It could be an action outside a process, a trend, a shift, a change in measurement, a sensor trigger or a change in data. Knowing that something has occurred may or may not be of interest. Knowing that an event has occurred can be something of note or just unnecessary noise. Establishing rules for expected events helps, but mining for the unknowns is usually the cure for dark events. Once known, then decisions or actions can be made and rules established / updated.
Dark Data are encapsulated within Dark Events that may in turn be encapsulated within a Dark Process. While Dark Data are necessarily part of a Dark Event, the Process may or may not be “Dark”; it could simply be a case of determining missing information about a know process. However, it would be a know process with unknown events and data that when illuminated add to its understanding.
Additional Reading on Darkness: