Exploring Pain Metaphor Comprehension Abilities in Large Language Models

Since the COVID-19 pandemic, the importance of digital health has grown significantly. One of the implementations of digital health is telemedicine to help patients remotely. This includes the implementation of Large Language Models (LLMs) to help reducing the workload of health workers and provide quicker support to patients. In the recent years, we have seen powerful Large Language Models that are able to pass many evaluation metrics such as Med-PaLM 2, which consists of the US Medical License Exam questions. However, the reliability of LLMs in the field of biomedicine remains questionable, and model interpretability is still a relatively developing field. This thesis explores the capabilities of Large Language Models in comprehending pain metaphors: metaphorical descriptions by patients of their experienced pain. As for the LLMs, GPT-3.5, GPT-4o, Gemini Pro, Llama-2 and BioMistral-7B are considered. We also explore if Large Language Models are able to explain the meaning behind...Show moreSince the COVID-19 pandemic, the importance of digital health has grown significantly. One of the implementations of digital health is telemedicine to help patients remotely. This includes the implementation of Large Language Models (LLMs) to help reducing the workload of health workers and provide quicker support to patients. In the recent years, we have seen powerful Large Language Models that are able to pass many evaluation metrics such as Med-PaLM 2, which consists of the US Medical License Exam questions. However, the reliability of LLMs in the field of biomedicine remains questionable, and model interpretability is still a relatively developing field. This thesis explores the capabilities of Large Language Models in comprehending pain metaphors: metaphorical descriptions by patients of their experienced pain. As for the LLMs, GPT-3.5, GPT-4o, Gemini Pro, Llama-2 and BioMistral-7B are considered. We also explore if Large Language Models are able to explain the meaning behind the pain metaphors used. Using so-called elenchus and maieutic prompting schemes, we found that the models are somewhat capable to understand that the pain metaphors are not to be taken literally, especially if more context is given to the pain that is being described. However, there are instances where the models do not associate such pain sensations as metaphorical. The less context is provided by the user, the easier LLMs are distracted and the more they are likely to answer incorrectly. We found that GPT-4o, GPT-3.5-Turbo-0125, Gemini-1.0-Pro, and Llama-2-7b-chat are able to understand pain metaphors relatively well, whereas BioMistral-7B fails most of our tests. Our experiment also suggests that models trained on more generalized knowledge such as GPT-4o, GPT-3.5-Turbo, Gemini-1.0-Pro, Llama-2-7b-chat seem to be aware of what entails as linguistic metaphors, however, these models are not capable of transferring this knowledge to biomedicine, which means that it can understand metaphors, but it will have a difficult time to decide if a lexical item is a metaphor. For example, these models will not assign “stabbing pain” as a metaphor, instead suggesting that it is a pain sensation.Show less

Leiden University Student Repository

Documents

In Collections

Exploring Pain Metaphor Comprehension Abilities in Large Language Models