Pragmatic Analysis of AI-Generated Conversations in A Simulated Financial Services Setting
About this article
DOI:
https://doi.org/10.14419/y0048v42Keywords:
Pragmatic Analysis; Natural Language Processing; Chatbot; Financial Services; Grice's MaximsAbstract
Advances in natural language processing have led to the development of AI-based chatbots; however, their performance in specialized domains remains underexplored. Pragmatics involves inferring meaning from situational, cultural, and interpersonal cues. Currently, there is limited research on the pragmatic competence of chatbots in the financial sector. In this study, we investigated ChatGPT-generated conversations simulating bank account inquiries to evaluate pragmatic coherence, user-friendliness, and similarities to human communication. Thirty discrete utterances across nine conversational turns were qualitatively assessed for adherence to Grice's Maxims, pragmatic encoding, inference processes, and formality. Our analysis revealed that ChatGPT consistently decoded conversational intent and effectively delivered contextually appropriate content, although responses were occasionally verbose, repetitive, or overly formal. In addition, a user-opinion experiment was conducted to compare original and pragmatically modified responses with contextual and social features removed. Notably, the removal of pragmatic cues did not significantly affect participants' perceptions of naturalness or authenticity. Overall, current AI can implement certain pragmatic adaptations to interpret user intentions and contextual relevance. However, in specific contexts such as financial services, achieving a balance between communicative efficiency and naturalness remains an ongoing challenge. Our results suggest exploratory evidence that targeted pragmatic editing can nudge perceptions in specific contexts.
References
[1] Yang CH. How artificial intelligence technology affects productivity and employment: firm-level evidence from Taiwan. Res Policy. 2022;51(6):104536. https://doi.org/10.1016/j.respol.2022.104536.
[2] Enholm IM, Papagiannidis E, Mikalef P, Krogstie J. Artificial intelligence and business value: a literature review. Inf Syst Front. 2021;24:1709-1734. https://doi.org/10.1007/s10796-021-10186-w.
[3] Alharbi N, Ud Din F, Paul D, Sadgrove E. Driving AI chatbot adoption: a systematic review of factors, barriers, and future research directions. J Open Innov Technol Mark Complex. 2025;11(3):100590. https://doi.org/10.1016/j.joitmc.2025.100590.
[4] Adamopoulou E, Moussiades L. Chatbots: history, technology, and applications. Mach Learn Appl. 2020;2:100006. https://doi.org/10.1016/j.mlwa.2020.100006.
[5] Zhang G. AI linguistics. Nat Lang Process J. 2025;10:100137. https://doi.org/10.1016/j.nlp.2025.100137.
View more references (24)
[6] Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: an introduction. J Am Med Inform Assoc. 2011;18(5):544-551. https://doi.org/10.1136/amiajnl-2011-000464.
[7] Cambria E. Understanding natural language understanding. Switzerland: Springer Nature; 2025.
[8] Alomari EA. Unlocking the potential: a comprehensive systematic review of ChatGPT in natural language processing tasks. Comput Model Eng Sci. 2024;141(1):43-85. https://doi.org/10.32604/cmes.2024.052256.
[9] Newton PM, Summers CJ, Zaheer U, Xiromeriti M, Stokes JR, Bhangu JS, Roome EG, Roberts-Phillips A, Mazaheri-Asadi D, Jones CD, Hughes S, Gilbert D, Jones E, Essex K, Ellis EC, Davey R, Cox AA, Bassett JA. Can ChatGPT-4o really pass medical science exams? A pragmatic analysis us-ing novel questions. Med Sci Educ. 2025;35:721-729. https://doi.org/10.1007/s40670-025-02293-z.
[10] Bouteraa M, Chekima B, Thurasamy R, Bin-Nashwan SA, Al-Daihani M, Baddou A, Sadallah M, Ansar R. Open innovation in the financial sector: a mixed-methods approach to assess bankers' willingness to embrace Open-AI ChatGPT. J Open Innov Technol Mark Complex. 2024;10(1):100216. https://doi.org/10.1016/j.joitmc.2024.100216.
[11] Kshetri N. Generative artificial intelligence in the financial services industry. Computer. 2024;57(6):102-108. https://doi.org/10.1109/MC.2024.3382452.
[12] Graham G, Nisar TM, Prabhakar G, Meriton R, Malik S. Chatbots in customer service within banking and finance: do chatbots herald the start of an AI revolution in the corporate world? Comput Hum Behav. 2025;165:108570. https://doi.org/10.1016/j.chb.2025.108570.
[13] Sugumar M, Chandra S. Do I desire chatbots to be like humans? Exploring factors for the adoption of chatbots for financial services. J Int Technol Inf Manage. 2021;30(3):38-77. https://doi.org/10.58729/1941-6679.1501.
[14] Grice HP. Logic and conversation. In: Cole P, Morgan JL. Syntax and semantics, volume 3: speech acts. New York: Academic Press; 1975. pp. 41-58. https://doi.org/10.1163/9789004368811_003.
[15] Grice P. Studies in the way of words. Cambridge: Harvard University Press; 1989.
[16] Setlur V, Tory M. How do you converse with an analytical chatbot? Revisiting Gricean Maxims for designing analytical conversational behavior. Pro-ceedings of the 2022 CHI Conference on Human Factors in Computing Systems; 2022, 1-17. https://doi.org/10.1145/3491102.3501972.
[17] Predelli S. Hybrid indexicals and ellipsis. Erkenntnis. 2006;65:385-403. https://doi.org/10.1007/s10670-006-9010-5.
[18] Reboul A. Truthfully misleading: truth, informativity, and manipulation in linguistic communication. Front Commun. 2021;6:646820. https://doi.org/10.3389/fcomm.2021.646820.
[19] Ghafar ZN. The experimental pragmatics: a review of language constructing and interpreting. J Lang Pragmat Stud 2023;2(2):89-99. https://doi.org/10.58881/jlps.v2i2.30.
[20] Luo B, Lau RYK, Li C, Si YW. A critical review of state-of-the-art chatbot designs and applications. Wiley Interdiscip Rev Data Min Knowl Discov. 2022;12(1):e1434. https://doi.org/10.1002/widm.1434.
[21] Chen X, Li J, Ye Y. A feasibility study for the application of AI-generated conversations in pragmatic analysis. J Pragmat. 2024;223:14-30. https://doi.org/10.1016/j.pragma.2024.01.003.
[22] Miehling E, Nagireddy M, Sattigeri P, Daly EM, Piorkowski D, Richards JT. Language models in dialogue: conversational maxims for human-AI in-teractions. In: Al-Onaizan Y, Bansal M, Chen YN. Findings of the association for computational linguistics: EMNLP 2024. Stroudsburg, PA: Associa-tion for Computational Linguistics; 2024. pp. 14420-14437. https://doi.org/10.18653/v1/2024.findings-emnlp.843.
[23] Adegbite MA. The intersection of AI, banking and fintech in seamless financial services. World J Adv Res Rev. 2025;25(3):1516-1526. https://doi.org/10.30574/wjarr.2025.25.3.0613.
[24] Hultgren AK. Vocatives as rationalized politeness: theoretical insights from emerging norms in call centre service encounters. J Socioling. 2017;21(1):90-111. https://doi.org/10.1111/josl.12224.
[25] Sun Y, Wang G, Ren H. To entertain or to serve: Chinese and US banks' online identity based on a genre analysis of social media. IEEE Trans Prof Commun. 2021;64(2):121-136. https://doi.org/10.1109/TPC.2021.3064395.
[26] Andersson M, McIntyre D. Can ChatGPT recognize impoliteness? An exploratory study of the pragmatic awareness of a large language model. J Pragmat. 2025;239:16-36. https://doi.org/10.1016/j.pragma.2025.02.001.
[27] Fatahi S, Vassileva J, Roy CK. Comparing emotions in ChatGPT answers and human answers to the coding questions on Stack Overflow. Front Artif Intell. 2024;7:1393903. https://doi.org/10.3389/frai.2024.1393903.
[28] Kusal S, Patil S, Choudrie J, Kotecha K, Vora D, Pappas I. A systematic review of applications of natural language processing and future challenges with special emphasis in text-based emotion detection. Artif Intell Rev. 2023;56:15129-15215. https://doi.org/10.1007/s10462-023-10509-0.
[29] Piastra M, Catellani P. On the emergent capabilities of ChatGPT 4 to estimate personality traits. Front Artif Intell. 2025;8:1484260. https://doi.org/10.3389/frai.2025.1484260.