AI & Big Data: a Lethal Combo

Big data, unstructured or structured, fast or slow, in multiple contexts or one is a beast to manage. Big data is growing fast fueled by the democratization of data and the IoT environment. Often organizations simply control what they know they get results from and then store the rest for future leverage. In fact, most organizations use less than 20% of their data, leaving the remaining 80%, and the insights it contains, to be left outside to the operational and decision-making Processes.  Imagine if you used only 20% of any service, you paid for every month and ignored the other 80%!  This is exactly what we are doing with data.  Fortunately, there is hope as this is where Big Data can start to rely on AI and engage in a “cycle of leverage”. Presently, the interaction between AI and Big Data is in the early stages, and organizations are discovering helpful methods, techniques, and technologies to achieve meaningful results. Typically these efforts are neither architected nor managed holistically. Our work has shown there is an emerging “Cycle of Big Data” that we and would like to describe and share with you where we see AI can help. 

Big Data Cycle

The “Big Data Cycle” is the typical set of functional activities that surround the capture, storage, and consumption of big data. Big data is defined as a field that treats ways to manage, analyze and systematically extract information from, or otherwise deal with, data sets that are too large and complex to be managed with traditional software.  The “Cycle” is, in short, the process of leveraging big data into desired outcomes. Typically the cycle flows in a left to right fashion with iteration.

(Data-> Trigger->Pattern->Context->Decision-> Action-> Outcome->Feedback->Adjustments).

Data Management

Data management is a process that includes acquiring, validating, storing, protecting, and processing the required data to ensure the accessibility, reliability, and timeliness of the data for various users. Today this is a more complicated process due to the increase of speed of data (near real-time) and the increased complexity of the data resources (text, voice, images, and videos).  This situation has had the effect of outstripping the processing capabilities of both humans and traditional computing systems.

AI can assist here in several ways, including assisting with hyper-personalization by leveraging machine learning and profiles that can learn and adapt. AI can also help in the recognition of knowledge from streams of data through NLP categorization and relationship capture. AI can watch static or in motion images to find and manage like knowledge. Not only can AI help recognize and learn by watching human system or machine interactions, but it can also do it in less than an instant. This can be performed either at the edge of the cloud or through an IoT Network.  AI combined with other algorithms can help in finding “black swan events” that can be used to update strategies.

Pattern Management

Organizations need to keep their pulse on incoming signals and events to stay in tune with the current state of the world, industries, markets, customers, and other constituents while sifting out distracting noise events. While savvy organizations that employ strategy planning to actively look for specific patterns of threat and opportunity, unfortunately, most organizations are reactive suffering at the whims of events. Both types of organizations should be continually looking for “patterns of interest” from which to make decisions or to initiate actions that are already defined and stored for execution.

AI can help by recognizing both expected and unexpected signals, events, and patterns to recognize anomalies that might warrant attention potentially.  When combined with analytics, AI can learn and expose the potential for additional responses.  AI also recognizes and learns adaptations for patterns, decision opportunities, and the need for further actions. In some cases, automation opportunities can be identified to deliver faster and higher quality results.  

Context Management

The understanding of data can often change with the context from which it is viewed and the outcome for which it can be leveraged. The “subject” of data can mean something slightly or significantly different in one context versus another.

Understanding the context is as important as understanding the data itself. Information about the context and the interaction of its contents (aka worlds) is essential to capture and maintain.  This allows for a classification of data in context and especially in relation to other contexts as big data sources may contain many contexts and relationships within it.

AI can assist the dynamic computer processes that use “subjects” of data in one context (industry, market, process or application) to point to data resident in a separate (industry, market, process or application) that also contains the same subject.  AI can learn the subtle differences and context-specific nuances to track the evolution of the data’s meaning in multiple contexts, whether it is “interacting” or not. This is particularly useful in understanding conversations and human interactions with NLP as interpretation grids often differ.

Decision Management

Decision management (aka, EDM) has all the aspects of designing, building and managing the automated decision-making systems that an organization uses to manage its decision making processes both internally as well as any interactions with outside parties such as customers, suppliers, vendors, and communities. The impact of decision management is felt in how organizations run their business for the goals of efficiency and effectiveness. Organizations depend on descriptive, prescriptive, and predictive analytics leveraging big data to provide the fuel that drives this environment.

AI can play a crucial role in supercharging knowledge and expertise utilization in a continually evolving and changing world. AI can also help scale key resources by leveraging an ever-growing base of big data at the speed of business that is ever-increasing while supporting today's operational requirements and ensuring its application to the ever-growing user expectations. Specifically, increasing the use of AI in human interactions will be a significant contribution to improving customer experiences and increasing the speed of resolution regarding customer issues.  AI can also suggest where to look for decision opportunities, model decisions, and their outcomes, and actively monitor performance against key performance indicators. 

Action Management

Action management involves planning and organizing the desired proactive or reactive actions and work activities of all humans, processes, bots applications, and devices employed by the organization. It includes managing, coordinating, and orchestrating tasks, developing project plans, monitoring performance, and achieving desired outcomes represented by goals in accordance with approved principles and agreed parameters. The logging of these actions also feeds the big data pools for further analysis and potential optimizations or increased freedom levels through goal adjustments.

AI can help by associating proper actions in the direction of the previous decision steps. It may mean selecting an inventoried action, changing some of the rules/parameters of an inventoried action or suggest the creation of new actions not available in the current inventory. AI can be embedded in any of the steps or detailed tasks that are performed in the selected actions. AI can monitor the actions and report the outcomes to management.  AI, along with algorithms, can pre-test and suggested changed action before deployment, thus ensuring the desired outcome with be achieved.

Goal Management

Goal management is the process of defining and tracking goals to provide guidance and direction, help evaluate performance and give feedback to all resources (humans, processes, applications, bots, and managers) for performance improvement. This also includes the “people-pleasing” and optimization arenas.  As organizations move to implement increased employee empowerment, edge computing, and dynamic bots, the importance of self-directed goal attainment increases. New freedom levels that ratchet-up up autonomy include a heightened focus on goal attainment and monitoring..

AI can help guide autonomous humans, bots, process snippets, apps, and flexible infrastructures through the automatic adjustment of goals that take advantage of edge conditions or “just in time learning” within the guardrails of constraints and rules. All of these resources can receive new guidance from real-time learning AI capabilities either built-in or “externally called” depending on the feedback loops and logs contributing to the big data pools.

Risk Management

Risk management is the identification, evaluation, and prioritization of risks mitigated by the coordinated and intelligent application of resources to minimize, monitor, mitigate, and control the impact of threats.  This will require tapping into the big data pool to continually monitor events and identify emerging threats and opportunities.

AI can help organizations recognize the emergence of situations that might require a response and enable mitigation responses.  Key patterns and anomalies can be recognized in events, patterns, logs of systems, and human feedback (including social networks) for potential or emerging risks. Additionally, any attacks or issues that exist within the perimeter, such as, cultural behavior, can be detected early and the development of necessary defenses enabled.

Net; Net:

Big Data development and management is a core capability that an organization needs to master in order to either become or remain competitive. It is clear to us that AI is the engine that will create value from the ever-increasing Big Data resource.   Big Data has a critical role to play over time as we journey deeper into the new digital world.  AI can handle speed, volume, and change much better than any technology that we have worked with, and this is just what Big Data needs!

For more information see:

www.datadiscoverysciences.com/blog

www.sinurblogspot.com

This post is a collaboration with Dr. Edward Peters 

Edward M.L. Peters, Ph.D. is an award-winning technology entrepreneur and executive. He is the founder and CEO of Data Discovery Sciences, an intelligent automation services firm located in Dallas, TX.   As an author and media commentator,  Dr. Peters is a frequent contributor on Fox Business Radio and has published articles in  The Financial Times, Forbes, IDB,  and  The Hill. Contact- epeters@datadiscoverysciences.com