Citation:

Praneeth Vadlapati, 2023. "Investigating the Impact of Linguistic Errors of Prompts on LLM Accuracy ESP Journal of Engineering & Technology Advancements" 3(2): 144-147.

Abstract:

Large Language Models (LLMs), such as GPT-3.5, have demonstrated exceptional abilities in processing text by understanding and generating text. Language models are trained on diverse data, which is typically free of linguistic errors such as spelling and grammatical mistakes. However, in real-world scenarios, the queries from the users commonly include linguistic errors. This research systematically examines the robustness of LLM in handling uncommon linguistic errors. The study utilizes original error-free text, text with errors in spelling, and text with errors in grammar. The study analyzes accuracy across multiple types of tasks such as quantitative reasoning, text manipulation, and linguistic tasks to test across various scenarios and evaluate whether the resilience of the model to linguistic errors varies across multiple types of tasks. In addition, the study explores the vulnerabilities of generating harmful text using jailbreaking through adversarial prompts that include grammatical errors. The results underscore the necessity of handling linguistic errors and implementing advanced mechanisms to mitigate threats from adversarial inputs. This study contributes to the research on investigating the reliability and robustness of AI systems in real-world applications. The source code is available at github.com/Pro-GenAI/PromptSpell.

References:

[1] Vaswani et al., Attention is all you need, in Proceedings of the 31st International Conference on Neural Information Processing Systems, in NIPS17. Red Hook, NY, USA: Curran Associates Inc., Dec. 2017, pp. 6000–6010. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

[2] Chen and A. Wettig, Understanding Large Language Models, Princeton University. Accessed: Mar. 31, 2023. Available: https://www.cs.princeton.edu/courses/archive/fall22/cos597G/

[3] L. Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling, Dec. 2020, arXiv:2101.00027. Available: http://arxiv.org/abs/2101.00027

[4] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, Large Language Models are Zero-Shot Reasoners, Oct. 2022, arXiv:2205.11916. [Online]. Available: https://arxiv.org/abs/2205.11916

[5] S. Agrawal, Are LLMs the Master of All Trades?: Exploring Domain-Agnostic Reasoning Skills of LLMs, Mar. 2023, arXiv: arXiv:2303.12810. [Online]. Available: http://arxiv.org/abs/2303.12810

[6] Y. Zhou et al., Large Language Models are Human-Level Prompt Engineers, in The Eleventh International Conference on Learning Representations, Feb. 2023. [Online]. Available: https://openreview.net/forum?id=92gvk82DE-

[7] J. White et al., A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT, Feb. 2023, arXiv:2302.11382. [Online]. Available: http://arxiv.org/abs/2302.11382

[8] Bryant, Z. Yuan, M. R. Qorib, H. Cao, H. T. Ng, and T. Briscoe, Grammatical Error Correction: A Survey of the State of the Art, Nov. 2022, arXiv:2211.05166. [Online]. Available: http://arxiv.org/abs/2211.05166

[9] de Beer, Grammarly both helps, hinders students, The Standard. Accessed: Mar. 31, 2023. [Online]. Available: https://standard.asl.org/16178/opinions/does-grammarly-help-or-hinder-students/

[10] M. Tran and P. Burman, Rating the English Proficiency of Countries and Industries Around the World, Harvard Business Review. Accessed: Mar. 31, 2023. [Online]. Available: https://hbr.org/2016/11/research-companies-and-industries-lack-english-skills

[11] Hendrycks et al., Measuring Massive Multitask Language Understanding, in International Conference on Learning Representations, Jan. 2021. [Online]. Available: https://openreview.net/forum?id=d7KBjmI3GmQ

[12] S. Ruder, Challenges and Opportunities in NLP Benchmarking, Ruder.io. Accessed: Mar. 31, 2023. [Online]. Available: https://www.ruder.io/nlp-benchmarking/

[13] S. Qiu, Q. Liu, S. Zhou, and W. Huang, Adversarial attack and defense technologies in natural language processing: A survey, Neurocomputing, vol. 492, pp. 278–307, Jul. 2022, doi: 10.1016/j.neucom.2022.04.020.

[14] S. Goyal, S. Doddapaneni, M. M. Khapra, and B. Ravindran, A Survey of Adversarial Defences and Robustness in NLP, Mar. 2022, arXiv:2203.06414. [Online]. Available: http://arxiv.org/abs/2203.06414

[15] K.-W. Chang, H. He, R. Jia, and S. Singh, Robustness and Adversarial Examples in Natural Language Processing, in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, J. Jiang and I. Vulić, Eds., Punta Cana, Dominican Republic & Online: Association for Computational Linguistics, Nov. 2021, pp. 22–26. doi: 10.18653/v1/2021.emnlp-tutorials.5.

[16] Y. Yao et al., Adversarial Language Games for Advanced Natural Language Intelligence, AAAI, vol. 35, no. 16, pp. 14248–14256, May 2021, doi: 10.1609/aaai.v35i16.17676.

Keywords:

Large Language Models (LLMs), LLM Robustness, Natural Language Processing (NLP), Spelling Errors, Grammatical Errors, Adversarial Prompts.