For those of us who write and think about the future of healthcare, the story of rapid and systemic change rocking the healthcare system is a recurrent theme. We usually point to the regulatory environment as the source of change. Laws like the Affordable Care Act and the HITECH Act are such glaring disruptive forces, but what empowers these regulations to succeed? Perhaps the deepest cause of change affecting healthcare, and the most disruptive force, is the digitalization of our clinical records. As we continue to switch to electronic charts, this force of  the vast data being collected becomes increasingly obvious. One-fifth of the world’s data is purported to be administrative and clinical medical records. Recording medical observations, lab results, diagnoses, and the orders that care professionals make in binary form is a game-changer.

Workflows are dramatically altered because caregivers spend so much of their time using the system to record clinical facts and must balance these record-keeping responsibilities with the more traditional bedside skills. They have access to more facts more easily than before, which allows them to make better judgments. The increasing ability of caregivers to see what their colleagues are doing, or have done, across institutional boundaries is allowing for better coordination of care. The use of clinical data for research into what works and what is efficient is becoming pervasive. This research is conducted by combining records from several institutions and having the quality committees of individual institutions look at the history of care within their institutions to enhance the ways in which they create the institutional standards of care. The data represents a vast resource of evidence that allows great innovation.

Another trend we see in healthcare that rises from the digitalization of clinical records is the evolution of new value-based payment models; for unlike efforts to control cost via methodologies to constrain what the system can pay for prior to the clinical digitalization, now significant evidence from the data is required to create the standards of care associated with payments for quality.

Now it is not simply the caregiver that must consider the clinical records, insurance companies must understand the outcomes for interventions, encounters, and tests that they pay for; and the pharmaceutical industry will be able to easily study the clinical record of off-label uses of its therapies and, via retrospective research, create new on-label uses for their therapies without necessarily setting up double-blind studies to prove the efficacy of these new drug indications. By “retrospective research,” I mean that the data may be algorithmically or interrogatively examined based on what has occurred and, by combining facts about disease state, interventions, and results as manifested in quantifiable facts, new on-label uses of drugs can be approved.

The data yields answers to the questions of what was  done and what the results were. However, there is so much data and so many possible ways that one can interrogate it, algorithmic pattern recognition from the data’s facts becomes an imperative. Healthcare has a bold future doing retrospective research by using the clinical data stored in the electronic medical records (EMRs) of hospitals and ambulatory practices. But utilizing the techniques of machine learning that are being invented across industries is a requirement in order to expedite the data mining that must take place.

 The use of complex and large sets of data for machine learning and cognitive engagement abound in healthcare, especially at the most prestigious institutions, or those emerging as prestigious because of an understanding of the  disruptive potential of digital clinical records. Projects like the digitization of best practices at Memorial Sloan-Kettering using tools like IBM’s Watson demonstrate that the vast corpus of medical research can be consumed by sophisticated software that will theoretically allow people interrogating the system to quickly understand what the recommended care paths in oncology are. Projects like the investigation of complex OMIC networks affecting Crohn’s disease at Mt. Sinai’s Institute of Genomic and Scalar Medicine using the tools of machine learning in hypothesis-free techniques for initial discovery are poised to change fundamental research capabilities in the complex reality of genetic medicine.

The ability to combine the complexity of medical data with recent innovations in machine learning will quicken and deepen the pace of medical innovation. New standards of care will be discovered. Software-assisted diagnoses will become commonplace. Discovering new uses for medicines will occur regularly. By understanding the cause of these changes, the contemporary healthcare organization — and in particular the groups that support the software tools that are required to produce, analyze, and act on the clinical record — can better understand what types of things to look for as the data requirements of these tools.

As we move forward, it’s important for EMR vendors to understand that, at its core, the not only represents a single person’s medical history, but also the collective medical history of the global population. By combining these data across the ecosystem, we get a longitudinal view of clinical history that will empower more efficient use of resources and better outcomes.

As with most significant opportunities, there is great risk. The balance between people’s right to privacy must be weighed against the society’s right to have access to important medical population data. The debate over this dialectical problem — our right to personal privacy and the public right to access detailed epidemiological facts — will continue through the next generation. As software assists in more and more medical decisions, and even in some cases makes decisions without immediate oversight of licensed professionals, the chance that care providers will overly rely on this software and miss opportunities at the edge of knowledge is great. The challenges of how we use the new digital records are immense, but so are the opportunities we get from this new data stream.