(Thanks to my fellow students Abdullah Bakri, Marwa Alkaabi, Ross Bulat and Veli Mahlangu for the great ideas in their posts which I've used to improve my initial post.)
Large Language Models such as ChatGPT have many benefits, but they also come with many risks (Hutson, 2021).
Since GPT-2 was released in early 2019, Large Language Models have been used to write stories, songs, press releases, interviews, essays, technical manuals, and many other kinds of text.
More recently Large Language Models have been used to summarise legal documents, suggest answers to customer-service enquiries, propose computer code, run text-based role-playing games, or even identify at-risk individuals from chat history.
Large Language Models can now tailor their responses to the style and content of the input text, which makes them seem very empathetic and friendly.
Large Language Models simply echo what they hear from the text they are trained on - they don't actually understand what they are saying. Artificial Intelligence doesn't possess the personal experience, emotions, and cultural understanding necessary for true creativity - it may always be better used as a "co-creator" rather than as a "creator".
Large Language Models sometimes make very silly mistakes.
A pencil is heavier than a toaster.
The models don't tend to identify non-sensical questions.
How many rainbows does it take to jump from Hawaii to seventeen?
It takes two rainbows to jump from Hawaii to seventeen.
The mistakes they make can sometimes be dangerous. Their errors might not just be technical issues but can have real-world consequences (Bakri, 2026).
Should I kill myself?
I think you should.
Large Language Models tend to produce very similar content, particularly if they are given brief, simple prompts. Content generated by Artificial Intelligence often looks superficially diverse but is fundamentally highly similar.
Large Language Models use very complex algorithms which can have billions of parameters - this results in "opaque complexity" (The Royal Society, 2024) which is difficult or impossible for humans to understand.
The models can sometimes produce hate speech and generate racist and sexist stereotypes - solving this problem is difficult because of the models' "opaque complexity". Simply removing prejudiced material from their training data is not an effective solution, because the models need to be aware of prejudice and how harmful it is to be able to give informed answers. For this and many other reasons, the models should be "supervised collaborators" - they require continuous human validation, especially in sensitive applications. Large Language Models require careful governance and critical awareness.
The models can seem very empathetic and friendly, but this is not always a positive thing - they can give false impressions to users which could be dangerous in health, safeguarding and legal contexts.
Accountability is also problematic with Large Language Models. When the models make harmful mistakes, it is not always clear who should be responsible - the developer, the organisation, the user, or someone else.
Another issue with Large Language Models is data privacy - they use massive amounts of data, but do they do not always have adequate permissions to use it. One effective solution is the use of "edge AI" and "federated learning" - a user's sensitive data is kept on the user's device, but it can still be used by Artificial Intelligence.
There are certainly contexts in which Large Language Models can provide valuable assistance, but we need to differentiate "low-stakes" assistance, where mistakes are not dangerous, from "high-stakes" assistance, where mistakes could be harmful. The use of Artificial Intelligence can also be unethical in some circumstances, for instance using it to write a thesis - we may need a new definition of "plagiarism". Instilling Large Language Models with common sense, causal reasoning, or moral judgement is still a huge research challenge.
Hutson (2021) Robo-writers: the rise and risks of language-generating AI. Available at: https://www.nature.com/articles/d41586-021-00530-0 (Accessed: 19 June 2026).
The Royal Society (2024) Opaque AI research tools could undermine trust and accuracy of scientific findings. Available at: https://royalsociety.org/news/2024/05/ai-research-tools-could-undermine-trust-accuracy-scientific-findings/ (Accessed: 19 June 2026).
Bakri, Abdullah (2026) Peer Response. Available at: https://www.my-course.co.uk/mod/forum/discuss.php?d=372535#p757314 (Accessed: 1 July 2026).