Lawrence Livermore National Laboratory

Calibrated approach to AI and deep learning models could more reliably diagnose and treat disease

29-May-2020 6:05 AM EDT, by Lawrence Livermore National Laboratory

Newswise — As artificial intelligence (AI) becomes increasingly used for critical applications such as diagnosing and treating diseases, predictions and results regarding medical care that practitioners and patients can trust will require more reliable deep learning models.

In a recent preprint (available through Cornell University’s open access website arXiv), a team led by a Lawrence Livermore National Laboratory computer scientist proposes a novel deep learning approach aimed at improving the reliability of classifier models designed for predicting disease types from diagnostic images, with an additional goal of enabling interpretability by a medical expert without sacrificing accuracy. The approach uses a concept called confidence calibration, which systematically adjusts the model’s predictions to match the human expert’s expectations in the real world.

“Reliability is an important yardstick as AI becomes more commonly used in high-risk applications, where there are real adverse consequences when something goes wrong,” explained lead author and LLNL computational scientist Jay Thiagarajan. “You need a systematic indication of how reliable the model can be in the real setting it will be applied in. If something as simple as changing the diversity of the population can break your system, you need to know that, rather than deploy it and then find out.”

In practice, quantifying the reliability of machine-learned models is challenging, so the researchers introduced the “reliability plot,” which includes experts in the inference loop to reveal the trade-off between model autonomy and accuracy. By allowing a model to defer from making predictions when its confidence is low, it enables a holistic evaluation of how reliable the model is, Thiagarajan explained.

In the paper, the researchers considered dermoscopy images of lesions used for skin cancer screening — each image associated with a specific disease state: melanoma, melanocytic nevus, basal cell carcinoma, actinic keratosis, benign keratosis, dermatofibroma and vascular lesions. Using conventional metrics and reliability plots, the researchers showed that calibration-driven learning produces more accurate and reliable detectors when compared to existing deep learning solutions. They achieved 80 percent accuracy on this challenging benchmark, in contrast to 74 percent by standard neural networks.

However, more important than increased accuracy, prediction calibration provides a completely new way to build interpretability tools in scientific problems, Thiagarajan said. The team developed an introspection approach, where the user inputs a hypothesis about the patient (such as the onset of a certain disease) and the model returns counterfactual evidence that maximally agrees with the hypothesis. Using this “what-if” analysis, they were able to identify complex relationships between disparate classes of data and shed light on strengths and weaknesses of the model that would not otherwise be apparent.

“We were exploring how to make a tool that can potentially support more sophisticated reasoning or inferencing,” Thiagarajan said. “These AI models systematically provide ways to gain new insights by placing your hypothesis in a prediction space. The question is, “How should the image look if person has been diagnosed with a condition A versus condition B?” Our method can provide the most plausible or meaningful evidence for that hypothesis. We can even obtain a continuous transition of a patient from State A to State B, where the expert or a doctor defines what those states are.”

Recently, Thiagarajan applied these methods to study chest X-ray images of patients diagnosed with COVID-19, arising due to the novel SARS-CoV-2 coronavirus. To understand the role of factors such as demography, smoking habits and medical intervention on health, Thiagarajan explained that AI models must analyze much more data than humans can handle, and the results need to be interpretable by medical professionals to be useful. Interpretability and introspection techniques will not only make models more powerful, he said, but they could provide an entirely novel way to create models for health care applications, enabling physicians to form new hypotheses about disease and aiding policy makers in decision-making that affects public health, such as with the ongoing COVID-19 pandemic.

“People want to integrate these AI models into scientific discovery,” Thiagarajan said. “When a new infection comes like COVID, doctors are looking for evidence to learn more about this novel virus. A systematic scientific study is always useful, but these data-driven approaches that we produce can significantly complement the analysis that experts can do to learn about these kinds of diseases. Machine learning can be applied far beyond just making predictions, and this tool enables that in a very clever way.”

The work, which Thiagarajan began in part to find new techniques for uncertainty quantification (UQ), was funded through the Department of Energy’s Advanced Scientific Computing Research (ASCR) program. Along with team members at LLNL, he has begun to utilize UQ-integrated AI models in several scientific applications and recently started a collaboration with the University of California, San Francisco School of Medicine on next-generation AI in clinical problems.

Co-authors on the paper included Prasanna Sattigeri and Deepta Rajan of IBM Research AI, and Bindya Venkatesh of Arizona State University.

Filters close

Showing results

110 of 2530
Released: 13-Jul-2020 11:15 AM EDT
UTHealth joins study of blood pressure medication’s effect on improving COVID-19 outcomes
University of Texas Health Science Center at Houston

An interventional therapy aimed at improving survival chances and reducing the need for critical care treatment due to COVID-19 is being investigated by physicians at The University of Texas Health Science Center at Houston (UTHealth). The clinical trial is underway at Memorial Hermann and Harris Health System’s Lyndon B. Johnson Hospital.

Newswise: Drug that calms ‘cytokine storm’ associated with 45% lower risk of dying among COVID-19 patients on ventilators
Released: 13-Jul-2020 7:25 AM EDT
Drug that calms ‘cytokine storm’ associated with 45% lower risk of dying among COVID-19 patients on ventilators
Michigan Medicine - University of Michigan

Critically ill COVID-19 patients who received a single dose of a drug that calms an overreacting immune system were 45% less likely to die overall, and more likely to be out of the hospital or off a ventilator one month after treatment, compared with those who didn’t receive the drug, according to a new observational study.

10-Jul-2020 9:00 AM EDT
Long-term strategies to control COVID-19 pandemic must treat health and economy as equally important, argue researchers
University of Cambridge

Strategies for the safe reopening of low and middle-income countries (LMICs) from months of strict social distancing in response to the ongoing COVID-19 pandemic must recognise that preserving people’s health is as important as reviving the economy, argue an international team of researchers.

Released: 10-Jul-2020 3:05 PM EDT
Simple blood test can predict severity of COVID-19 for some patients
University of Texas Health Science Center at Houston

An early prognosis factor that could be a key to determining who will suffer greater effects from COVID-19, and help clinicians better prepare for these patients, may have been uncovered by researchers at The University of Texas Health Science Center at Houston (UTHealth). Results of the findings were published today in the International Journal of Laboratory Hematology.

Released: 10-Jul-2020 12:50 PM EDT
Genetic ‘fingerprints’ of first COVID-19 cases help manage pandemic
University of Sydney

A new study published in the world-leading journal Nature Medicine, reveals how genomic sequencing and mathematical modelling gave important insights into the ‘parentage’ of cases and likely spread of the disease in New South Wales.

Released: 10-Jul-2020 12:35 PM EDT
Our itch to share helps spread COVID-19 misinformation
Massachusetts Institute of Technology (MIT)

To stay current about the Covid-19 pandemic, people need to process health information when they read the news. Inevitably, that means people will be exposed to health misinformation, too, in the form of false content, often found online, about the illness.

Newswise: Pandemic Inspires Framework for Enhanced Care in Nursing Homes
Released: 10-Jul-2020 12:25 PM EDT
Pandemic Inspires Framework for Enhanced Care in Nursing Homes
University of Pennsylvania School of Nursing

As of May 2020, nursing home residents account for a staggering one-third of the more than 80,000 deaths due to COVID-19 in the U.S. This pandemic has resulted in unprecedented threats—like reduced access to resources needed to contain and eliminate the spread of the virus—to achieving and sustaining care quality even in the best nursing homes. Active engagement of nursing home leaders in developing solutions responsive to the unprecedented threats to quality standards of care delivery is required.

Newswise: General Electric Healthcare Chooses UH to Clinically 
Evaluate First-of-its-kind Imaging System
Released: 10-Jul-2020 12:15 PM EDT
General Electric Healthcare Chooses UH to Clinically Evaluate First-of-its-kind Imaging System
University Hospitals Cleveland Medical Center

University Hospitals Cleveland Medical Center physicians completed evaluation for the GE Healthcare Critical Care Suite, and the technology is now in daily clinical practice – flagging between seven to 15 collapsed lungs per day within the hospital. No one on the team could have predicted the onset of the COVID-19 pandemic, but this technology and future research with GEHC may enhance the capability to improve care for COVID-19 patients in the ICU. Critical Care Suite is now assisting in COVID and non-COVID patient care as the AMX 240 travels to intensive care units within the hospital.

Released: 10-Jul-2020 11:50 AM EDT
COVID-19 Can Be Transmitted in the Womb, Reports Pediatric Infectious Disease Journal
Wolters Kluwer Health: Lippincott Williams and Wilkins

A baby girl in Texas – born prematurely to a mother with COVID-19 – is the strongest evidence to date that intrauterine (in the womb) transmission of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) can occur, reports The Pediatric Infectious Disease Journal, the official journal of The European Society for Paediatric Infectious Diseases. The journal is published in the Lippincott portfolio by Wolters Kluwer.

Showing results

110 of 2530