Lawrence Livermore National Laboratory

Calibrated approach to AI and deep learning models could more reliably diagnose and treat disease

Newswise — As artificial intelligence (AI) becomes increasingly used for critical applications such as diagnosing and treating diseases, predictions and results regarding medical care that practitioners and patients can trust will require more reliable deep learning models.

In a recent preprint (available through Cornell University’s open access website arXiv), a team led by a Lawrence Livermore National Laboratory computer scientist proposes a novel deep learning approach aimed at improving the reliability of classifier models designed for predicting disease types from diagnostic images, with an additional goal of enabling interpretability by a medical expert without sacrificing accuracy. The approach uses a concept called confidence calibration, which systematically adjusts the model’s predictions to match the human expert’s expectations in the real world.

“Reliability is an important yardstick as AI becomes more commonly used in high-risk applications, where there are real adverse consequences when something goes wrong,” explained lead author and LLNL computational scientist Jay Thiagarajan. “You need a systematic indication of how reliable the model can be in the real setting it will be applied in. If something as simple as changing the diversity of the population can break your system, you need to know that, rather than deploy it and then find out.”

In practice, quantifying the reliability of machine-learned models is challenging, so the researchers introduced the “reliability plot,” which includes experts in the inference loop to reveal the trade-off between model autonomy and accuracy. By allowing a model to defer from making predictions when its confidence is low, it enables a holistic evaluation of how reliable the model is, Thiagarajan explained.

In the paper, the researchers considered dermoscopy images of lesions used for skin cancer screening — each image associated with a specific disease state: melanoma, melanocytic nevus, basal cell carcinoma, actinic keratosis, benign keratosis, dermatofibroma and vascular lesions. Using conventional metrics and reliability plots, the researchers showed that calibration-driven learning produces more accurate and reliable detectors when compared to existing deep learning solutions. They achieved 80 percent accuracy on this challenging benchmark, in contrast to 74 percent by standard neural networks.

However, more important than increased accuracy, prediction calibration provides a completely new way to build interpretability tools in scientific problems, Thiagarajan said. The team developed an introspection approach, where the user inputs a hypothesis about the patient (such as the onset of a certain disease) and the model returns counterfactual evidence that maximally agrees with the hypothesis. Using this “what-if” analysis, they were able to identify complex relationships between disparate classes of data and shed light on strengths and weaknesses of the model that would not otherwise be apparent.

“We were exploring how to make a tool that can potentially support more sophisticated reasoning or inferencing,” Thiagarajan said. “These AI models systematically provide ways to gain new insights by placing your hypothesis in a prediction space. The question is, “How should the image look if person has been diagnosed with a condition A versus condition B?” Our method can provide the most plausible or meaningful evidence for that hypothesis. We can even obtain a continuous transition of a patient from State A to State B, where the expert or a doctor defines what those states are.”

Recently, Thiagarajan applied these methods to study chest X-ray images of patients diagnosed with COVID-19, arising due to the novel SARS-CoV-2 coronavirus. To understand the role of factors such as demography, smoking habits and medical intervention on health, Thiagarajan explained that AI models must analyze much more data than humans can handle, and the results need to be interpretable by medical professionals to be useful. Interpretability and introspection techniques will not only make models more powerful, he said, but they could provide an entirely novel way to create models for health care applications, enabling physicians to form new hypotheses about disease and aiding policy makers in decision-making that affects public health, such as with the ongoing COVID-19 pandemic.

“People want to integrate these AI models into scientific discovery,” Thiagarajan said. “When a new infection comes like COVID, doctors are looking for evidence to learn more about this novel virus. A systematic scientific study is always useful, but these data-driven approaches that we produce can significantly complement the analysis that experts can do to learn about these kinds of diseases. Machine learning can be applied far beyond just making predictions, and this tool enables that in a very clever way.”

The work, which Thiagarajan began in part to find new techniques for uncertainty quantification (UQ), was funded through the Department of Energy’s Advanced Scientific Computing Research (ASCR) program. Along with team members at LLNL, he has begun to utilize UQ-integrated AI models in several scientific applications and recently started a collaboration with the University of California, San Francisco School of Medicine on next-generation AI in clinical problems.

Co-authors on the paper included Prasanna Sattigeri and Deepta Rajan of IBM Research AI, and Bindya Venkatesh of Arizona State University.




Filters close

Showing results

110 of 4206
Newswise: Proteolytic Enzymes for Covid-19 Studied in 3D for the First Time in Thailand by Chula Biochemists
Released: 4-Dec-2020 8:45 AM EST
Proteolytic Enzymes for Covid-19 Studied in 3D for the First Time in Thailand by Chula Biochemists
Chulalongkorn University

A team of biochemists from Chulalongkorn University became the first researchers in Thailand to study proteolytic enzymes for the Covid–19 virus at a molecular level in 3D, possibly leading to the development of Covid–19 treatments.

Newswise: UC San Diego Bolsters Aggressive Return to Learn Plan to Prevent Outbreaks on Campus
Released: 4-Dec-2020 8:35 AM EST
UC San Diego Bolsters Aggressive Return to Learn Plan to Prevent Outbreaks on Campus
University of California San Diego

UC San Diego’s nationally recognized, evidence-based Return to Learn program employs a comprehensive suite of education, monitoring, testing, intervention and notification tools that no other university is using. And the program continues to expand—including a recent introduction of weekly self-administered student testing kits, growth of the campus’s wastewater viral monitoring program and widespread use of the cellphone-based CA COVID Notify exposure notification system.

Newswise: Pediatric ER Saw Steep Drop in Asthma Visits During Spring COVID-19 Lockdown
1-Dec-2020 8:00 AM EST
Pediatric ER Saw Steep Drop in Asthma Visits During Spring COVID-19 Lockdown
American Thoracic Society (ATS)

A new study published online in the Annals of the American Thoracic Society discusses a steep drop off from prior years in asthma-related emergency department (ED) visits at Boston Children’s Hospital during the spring 2020 COVID-19 surge and lockdown.

access_time Embargo lifts in 2 days
Embargo will expire: 9-Dec-2020 4:00 PM EST Released to reporters: 3-Dec-2020 4:50 PM EST

A reporter's PressPass is required to access this story until the embargo expires on 9-Dec-2020 4:00 PM EST The Newswise PressPass gives verified journalists access to embargoed stories. Please log in to complete a presspass application. If you have not yet registered, please Register. When you fill out the registration form, please identify yourself as a reporter in order to advance to the presspass application form.

access_time Embargo lifts in 2 days
Embargo will expire: 9-Dec-2020 4:00 PM EST Released to reporters: 3-Dec-2020 3:50 PM EST

A reporter's PressPass is required to access this story until the embargo expires on 9-Dec-2020 4:00 PM EST The Newswise PressPass gives verified journalists access to embargoed stories. Please log in to complete a presspass application. If you have not yet registered, please Register. When you fill out the registration form, please identify yourself as a reporter in order to advance to the presspass application form.

Newswise: 250384_web.jpg
Released: 3-Dec-2020 3:05 PM EST
Study finds COVID-19 hindering US academic productivity of faculty with young children
University of Tennessee Health Science Center

The academic productivity of higher education faculty In the United States in the science, technology, engineering, mathematics, and medicine (STEMM) fields with very young children suffered as a result of the stay-at-home orders during the early months of the coronavirus pandemic, according to a new study by researchers at the University of Tennessee Health Science Center, the University of Florida College of Medicine, and the University of Michigan School of Medicine.

Released: 3-Dec-2020 2:50 PM EST
Kidney disease leading risk factor for COVID-related hospitalization
Geisinger Health System

An analysis of Geisinger's electronic health records has revealed chronic kidney disease to be the leading risk factor for hospitalization from COVID-19.

Newswise: Identity Verification During the Age of COVID-19
Released: 3-Dec-2020 2:25 PM EST
Identity Verification During the Age of COVID-19
Homeland Security's Science And Technology Directorate

S&T's Biometric Technology Rally focused on the ability of acquisition systems and matching algorithms to recognize travelers without asking them to remove their masks, thereby reducing risk for frontline workers.

access_time Embargo lifts in 2 days
Embargo will expire: 10-Dec-2020 11:00 AM EST Released to reporters: 3-Dec-2020 2:20 PM EST

A reporter's PressPass is required to access this story until the embargo expires on 10-Dec-2020 11:00 AM EST The Newswise PressPass gives verified journalists access to embargoed stories. Please log in to complete a presspass application. If you have not yet registered, please Register. When you fill out the registration form, please identify yourself as a reporter in order to advance to the presspass application form.


Showing results

110 of 4206

close
3.07965