RESPONSible Service: Evaluating Language Style in LLM-Generated Customer Service Interactions

Language style plays a crucial role in effective customer service, and the growing use of AI chat- bots in this field underscores the need to assess their ability to recognize and reproduce stylistic variation across different scenarios. However, there is a lack of publicly available datasets specifi- cally focused on language style in customer service. To address this gap, I created RESPONSible Service, a synthetic dataset of 4,000 GPT-4o-generated customer service interactions. The dataset is divided into four subsets, each targeting a specific communication style: clarity, friendliness, empathy, and politeness. In the absence of a human-authored ground truth, I evaluated the dataset using a range of NLP metrics and compared its stylistic features to linguistic patterns identified in prior research on human customer service communication. Results show that GPT-4o generally aligns with human communication trends, although it occasionally shows model-specific deviations and...Show moreLanguage style plays a crucial role in effective customer service, and the growing use of AI chat- bots in this field underscores the need to assess their ability to recognize and reproduce stylistic variation across different scenarios. However, there is a lack of publicly available datasets specifi- cally focused on language style in customer service. To address this gap, I created RESPONSible Service, a synthetic dataset of 4,000 GPT-4o-generated customer service interactions. The dataset is divided into four subsets, each targeting a specific communication style: clarity, friendliness, empathy, and politeness. In the absence of a human-authored ground truth, I evaluated the dataset using a range of NLP metrics and compared its stylistic features to linguistic patterns identified in prior research on human customer service communication. Results show that GPT-4o generally aligns with human communication trends, although it occasionally shows model-specific deviations and inconsistencies in context-sensitive generation. To assess the dataset’s usability for text generation, I fine-tuned a smaller model (LLaMA 3.2-1B-Instruct) using in-context learn- ing, supervised fine-tuning, and direct preference optimization. Outputs were evaluated using BERTScore, ROUGE-L, and manual annotation. In-context learning consistently outperformed the other methods, suggesting possible label noise or limited representativeness in the synthetic data. Furthermore, clarity emerged as the most linguistically unpredictable style. This study provides an initial framework for evaluating language style in LLMs for customer service, with the broader goal of improving alignment with real human communication.Show less

Leiden University Student Repository

Documents

In Collections

RESPONSible Service: Evaluating Language Style in LLM-Generated Customer Service Interactions