Big data, unstructured or
structured, fast or slow, in multiple contexts or one is a beast to manage. Big
data is growing fast fueled by the democratization of data and the IoT
environment. Often organizations simply control what they know they get results
from and then store the rest for future leverage. In fact, most organizations
use less than 20% of their data, leaving the remaining 80%, and the insights it
contains, to be left outside to the operational and decision-making
Processes. Imagine if you used only 20%
of any service, you paid for every month and ignored the other 80%! This is exactly what we are doing with data. Fortunately, there is hope as this is where Big
Data can start to rely on AI and engage in a “cycle of leverage”. Presently, the
interaction between AI and Big Data is in the early stages, and organizations
are discovering helpful methods, techniques, and technologies to achieve
meaningful results. Typically these efforts are neither architected nor managed
holistically. Our work has shown there is an emerging “Cycle of Big Data” that we and would like to describe and share
with you where we see AI can help.
Big Data Cycle
The “Big Data Cycle” is the
typical set of functional activities that surround the capture, storage, and
consumption of big data. Big data is defined as a field that treats ways to
manage, analyze and systematically extract information from, or otherwise deal
with, data sets that are too large and complex to be managed with traditional
software. The “Cycle” is, in short, the process
of leveraging big data into desired outcomes. Typically the cycle flows in a
left to right fashion with iteration.
(Data-> Trigger->Pattern->Context->Decision->
Action-> Outcome->Feedback->Adjustments).
Data Management
Data management is a process
that includes acquiring, validating, storing, protecting, and processing the
required data to ensure the accessibility, reliability, and timeliness of the
data for various users. Today this is a more complicated process due to the
increase of speed of data (near real-time) and the increased complexity of the
data resources (text, voice, images, and videos). This situation has had the effect of
outstripping the processing capabilities of both humans and traditional
computing systems.
AI can assist here in several
ways, including assisting with hyper-personalization by leveraging machine
learning and profiles that can learn and adapt. AI can also help in the recognition
of knowledge from streams of data through NLP categorization and relationship
capture. AI can watch static or in motion images to find and manage like
knowledge. Not only can AI help recognize and learn by watching human system or
machine interactions, but it can also do it in less than an instant. This can
be performed either at the edge of the cloud or through an IoT Network. AI combined with other algorithms can help in
finding “black swan events” that can be used to update strategies.
Pattern Management
Organizations need to keep their
pulse on incoming signals and events to stay in tune with the current state of the
world, industries, markets, customers, and other constituents while sifting out
distracting noise events. While savvy organizations that employ strategy
planning to actively look for specific patterns of threat and opportunity, unfortunately,
most organizations are reactive suffering at the whims of events. Both types of
organizations should be continually looking for “patterns of interest” from
which to make decisions or to initiate actions that are already defined and
stored for execution.
AI can help by recognizing both
expected and unexpected signals, events, and patterns to recognize anomalies
that might warrant attention potentially. When combined with analytics, AI can learn and
expose the potential for additional responses. AI also recognizes and learns adaptations for
patterns, decision opportunities, and the need for further actions. In some
cases, automation opportunities can be identified to deliver faster and higher
quality results.
Context Management
The understanding of data can
often change with the context from which it is viewed and the outcome for which it
can be leveraged. The “subject” of data can mean something slightly or significantly
different in one context versus another.
Understanding the context is
as important as understanding the data itself. Information about the context
and the interaction of its contents (aka worlds) is essential to capture and
maintain. This allows for a
classification of data in context and especially in relation to other contexts
as big data sources may contain many contexts and relationships within it.
AI can assist the dynamic
computer processes that use “subjects” of data in one context (industry,
market, process or application) to point to data resident in a separate
(industry, market, process or application) that also contains the same
subject. AI can learn the subtle
differences and context-specific nuances to track the evolution of the data’s
meaning in multiple contexts, whether it is “interacting” or not. This is particularly
useful in understanding conversations and human interactions with NLP as
interpretation grids often differ.
Decision Management
Decision management (aka,
EDM) has all the aspects of designing, building and managing the automated
decision-making systems that an organization uses to manage its decision making
processes both internally as well as any interactions with outside parties such
as customers, suppliers, vendors, and communities. The impact of decision
management is felt in how organizations run their business for the goals of
efficiency and effectiveness. Organizations depend on descriptive,
prescriptive, and predictive analytics leveraging big data to provide the fuel
that drives this environment.
AI can play a crucial role in
supercharging knowledge and expertise utilization in a continually evolving and
changing world. AI can also help scale key resources by leveraging an ever-growing
base of big data at the speed of business that is ever-increasing while
supporting today's operational requirements and ensuring its application to the
ever-growing user expectations. Specifically, increasing the use of AI in human
interactions will be a significant contribution to improving customer
experiences and increasing the speed of resolution regarding customer issues. AI can also suggest where to look for
decision opportunities, model decisions, and their outcomes, and actively monitor
performance against key performance indicators.
Action Management
Action management involves planning
and organizing the desired proactive or reactive actions and work activities of
all humans, processes, bots applications, and devices employed by the
organization. It includes managing, coordinating, and orchestrating tasks,
developing project plans, monitoring performance, and achieving desired outcomes
represented by goals in accordance with approved principles and agreed
parameters. The logging of these actions also feeds the big data pools for
further analysis and potential optimizations or increased freedom levels
through goal adjustments.
AI can help by associating
proper actions in the direction of the previous decision steps. It may mean
selecting an inventoried action, changing some of the rules/parameters of an
inventoried action or suggest the creation of new actions not available in the
current inventory. AI can be embedded in any of the steps or detailed tasks
that are performed in the selected actions. AI can monitor the actions and
report the outcomes to management. AI,
along with algorithms, can pre-test and suggested changed action before
deployment, thus ensuring the desired outcome with be achieved.
Goal Management
Goal management is the
process of defining and tracking goals to provide guidance and direction, help
evaluate performance and give feedback to all resources (humans, processes,
applications, bots, and managers) for performance improvement. This also
includes the “people-pleasing” and optimization arenas. As organizations move to implement increased
employee empowerment, edge computing, and dynamic bots, the importance of self-directed
goal attainment increases. New freedom levels that ratchet-up up autonomy include
a heightened focus on goal attainment and monitoring..
AI can help guide autonomous
humans, bots, process snippets, apps, and flexible infrastructures through the
automatic adjustment of goals that take advantage of edge conditions or “just
in time learning” within the guardrails of constraints and rules. All of these
resources can receive new guidance from real-time learning AI capabilities either
built-in or “externally called” depending on the feedback loops and logs
contributing to the big data pools.
Risk Management
Risk management is the
identification, evaluation, and prioritization of risks mitigated by the
coordinated and intelligent application of resources to minimize, monitor, mitigate,
and control the impact of threats. This
will require tapping into the big data pool to continually monitor events and
identify emerging threats and opportunities.
AI can help organizations
recognize the emergence of situations that might require a response and enable
mitigation responses. Key patterns and anomalies
can be recognized in events, patterns, logs of systems, and human feedback
(including social networks) for potential or emerging risks. Additionally, any
attacks or issues that exist within the perimeter, such as, cultural behavior, can
be detected early and the development of necessary defenses enabled.
Net; Net:
Big Data development and
management is a core capability that an organization needs to master in order to
either become or remain competitive. It is clear to us that AI is the engine
that will create value from the ever-increasing Big Data resource. Big Data has a critical role to play over
time as we journey deeper into the new digital world. AI can handle speed, volume, and change much better
than any technology that we have worked with, and this is just what Big Data
needs!
For more information see:
This post is a collaboration with Dr. Edward Peters
Edward M.L. Peters, Ph.D. is an award-winning technology entrepreneur and executive. He is the founder and CEO of Data Discovery Sciences, an intelligent automation services firm located in Dallas, TX. As an author and media commentator, Dr. Peters is a frequent contributor on Fox Business Radio and has published articles in The Financial Times, Forbes, IDB, and The Hill. Contact- epeters@datadiscoverysciences.com
