Real-Time Data Mixes Well with Archival Data Now

Combining real-time situational data with archival trend data gives context for the understanding of emerging situations. This powerful combination of data sources could imply a need to take actions in the form of immediate response or longer-term change to policies or processes. However, there has always been a bridge troll preventing organizations from bridging real-time data with data archives. That troll was performance issues with volumes of locked data. Well, the troll has taken a permanent position at another bridge somewhere. What has changed is the modern data mesh that leverages the cloud while taking advantage of these emerging combinations of real-time, operational, and archived data sources. The data mesh leveraging the hybrid cloud releases many of these technological constraints.

What are the Challenges of Real-Time Data?

First, organizations have to leverage these new data mesh capabilities in their technical architectures. These technical architectures will reveal some new challenges surrounding creating schemas, selecting data aggregation or location approaches, deciding on data formats, tuning development cycles, and establishing situational testing approaches for new kinds of algorithms leveraging real-time data. Once technically enabled, then the real fun begins. People tend to resist new approaches because they must learn something new and become acceptable to stay even. Smart managers will incent folks to take the risk of taking on something new. Rewarding risk is essential because recognizing real-time patterns will significantly impact the organization operationally and tactically and even tip strategies in new directions. Once the benefits become evident, others will follow suit quickly. There will be new pressures to establish data quality in organizations' fabric, ergo a culture change.

What are the Challenges of Archival Trend Data?

The problem with archival data is that it is locked in the format in which it was created. The unlocking of the data may require a severe unpacking of the real meaning or even change the format or context. In other words, it might have to be recoded to answer new questions and situations. Even if the data was unpacked and readily usable, how well will it behave? Was the data quality tolerance set at the right level? Will the meaning of the data have to be normalized to use it in combination with other data sources? With more and more archival data going online and even in the cloud, this challenge grows daily. It may require a data archaeologist to understand and leverage archival data optimally truly.

What are the Opportunities of Combining Real-Time Data with Archives?

Despite the challenges, the opportunities loom large. The ability to mix emergence with historical data and trends over time offers new insights to the learning organization.  I think the best way to describe the upside is through use cases.

Healthcare:

Using real-time monitoring data has changed patient care for the better. The problem is that the real-time data does not consider the patient's history that is locked in various patient history sources. While doctors will always be needed, they aren’t always available with the right history record. Imagine a world where the patient's history can be leveraged in an instant with emerging situational monitoring data. Add some machine learning and advisory AI; patients can be assisted more responsively.

Investments:

Using real-time trading has been getting better over time and even influenced by overall market guardrails to slow runaway downtrends. The trading bot may not use individual investors' goals that depict risk tolerances or yield plans, or mixes. Imagine a human or bot-based trader making trades influenced in real-time by the wishes of individual investors' aggregation by risk and yield personas.

Supply Chains:

Imagine a dynamic supply chain that can shift shipments under changing conditions that can take supplies in-flight and change their destination based on need. While we have faster delivery times and real-time monitoring of shipping progress details, we don’t link it to customer history. Imagine shifting delivery to adjust to emerging conditions and history, delivering vaccines to traditional hot spots like nursing homes in an accelerated fashion when more supplies appear unexpectedly.

Net; Net:

New data mesh capabilities will enable new opportunities for organizations to combine real-time data with both archival data and real-time operational data. It will require some preparation and some changes in how we all behave, but the outcomes will not be possible before. Situational computing partners with trends and history now. There is real power in interpreting the emerging real-time situation in the context of historical conditions or behavior.