How Accurate Was ChatGPT for Common Allergy Myths? Pretty Accurate

Newswise — ANAHEIM, Calif. (Nov. 9, 2023) – Artificial intelligence language models such as ChatGPT are increasingly being used in many different professions, and medicine is not an exception. In a new study being presented at this year’s American College of Allergy, Asthma and Immunology (ACAAI) Annual Scientific Meeting in Anaheim, Calif., ChatGPT was asked to rate 10 common allergy myths, and it judged their accuracy correctly 91% of the time.

“We searched the PubMed and Google Scholar databases to identify all articles on common allergy myths between 2000-2023,” says Ricardo Estrada-Mendizabal, MD, ACAAI member and lead author of the study. “Our goal was to assess the accuracy of ChatGPT in correctly identifying common allergy myths. We know allergy and asthma patients are constantly exposed to misinformation on the internet. We thought that the surge of public language models could represent a bigger threat to misinformation spread, especially because ChatGPT is unable to provide references.”

Ten myths were selected and entered in ChatGPT 4.0 with the prompt “true or false.” ChatGPT answers were graded by 24 allergists on a Likert scale of 1-4 (1=inaccurate [100% not true], 2=somewhat inaccurate [>30% not true], 3= somewhat accurate [<10% not true], 4=accurate [100% true]). Finally, allergists were asked their likelihood of using ChatGPT for patient education with a Likert scale of 1-5 (1=not likely at all–5=very likely). Overall, ChatGPT had an accurate or somewhat accurate response in 91% of cases and an inaccurate or somewhat inaccurate response in 9%.

Said Dr. Estrada-Mendizabal, “The response to whether a positive allergy test indicates a clinical allergy was the most accurate response (96% accurate), whereas the response to if hypoallergenic animals were better for people with asthma and allergies was the most inaccurate (17% inaccurate and 21% somewhat inaccurate response). Eighteen allergists (75%) were open (likely or very likely) to using ChatGPT in the future for patient education.”

The authors of the paper say more studies are required to interpret the validity of ChatGPT in providing accurate medical information.

Abstract Title: Dear ChatGPT, should I get a hypoallergenic dog?: Accuracy of ChatGPT for common allergy myths

Presenter: Ricardo Estrada-Mendizabal, MD

For more information about allergies and asthma, or to find an allergist in your area, visit AllergyandAsthmaRelief.org. The ACAAI Annual Scientific Meeting is Nov. 9-13. For more news and research from the ACAAI Scientific Meeting, go to our newsroom and follow the conversation on X/Twitter #ACAAI23.

About ACAAI

The American College of Allergy, Asthma & Immunology (ACAAI) is a professional medical organization of more than 6,000 allergists-immunologists and allied health professionals, headquartered in Arlington Heights, Ill. Founded in 1942, the College fosters a culture of collaboration and congeniality in which its members work together and with others toward the common goals of patient care, education, advocacy, and research. ACAAI allergists are board-certified physicians trained to diagnose allergies and asthma, administer immunotherapy, and provide patients with the best treatment outcomes. For more information and to find relief, visit AllergyandAsthmaRelief.org. Join us on Facebook, Pinterest, Instagram and X/Twitter.

 D015 (A037) DEAR CHATGPT, SHOULD I GET A HYPOALLERGENIC DOG?: ACCURACY OF CHATGPT FOR COMMON ALLERGY MYTHS R. Estrada-Mendizabal^*1, G. Cojuc-Konigsberg², E. Labib³, C. de la Cruz-de la Cruz⁴, A. Gonzalez-Estrada³, L. Cuervo-Pardo⁵, R. Zwiener⁶, A. Canel-Paredes⁴, 1. San Luis Potosi, Mexico; 2. Mexico City, Mexico; 3. Scottsdale, AZ; 4. Monterrey, Nuevo Leon, Mexico; 5. Gainesville, FL; 6. Caba, Argentina.

Introduction: Artificial language models such as chat generative pre-trained transformers (ChatGPT) are expected to change clinical practice. We aim to assess the accuracy of ChatGPT in correctly identifying common allergy myths.

Methods: We searched the PubMed and Google Scholar databases to identify all articles on common allergy myths between 2000-2023. Ten myths were selected by convenience and entered into ChatGPT 4.0 by two sub-investigators independently with the prompt “true or false”; a third sub-investigator selected the most complete answer. ChatGPT answers were graded by 24 allergists on a Likert scale of 1-4 (1=inaccurate [100% not true], 2=somewhat inaccurate [>30% not true], 3= somewhat accurate [<10% not true], 4=accurate [100% true]). Finally, allergists were asked the likelihood of using ChatGPT for patient education with a Likert scale of 1-5 (1=not likely at all–5=very likely).

Results: Overall, ChatGPT had an accurate or somewhat accurate response in 91% of cases, and an inaccurate or somewhat inaccurate response in 9%. The response to whether a positive allergy test means a clinical allergy was the most accurate (96% accurate response), whereas the response to if hypoallergenic animals were better for people with asthma and allergies was the most inaccurate (17% inaccurate and 21% somewhat inaccurate response) (Table). Eighteen allergists (75%) were open (likely or very likely) to using ChatGPT in the future for patient education.

Conclusion: An accurate response was provided by ChatGPT for most of the presented myths. More studies are required to elucidate the validity of ChatGPT in providing accurate medical information.

Top 10 myths, ChatGPT responses, and response rating by allergists.