AI Chatbot Hallucinations: The £6,400 Error Terrifying Business Owners

We have seen an AI chatbot write poems, sell Chevys for $1, and even swear at customers. But this week, a new milestone in “Prompt Injection” was reached in the United Kingdom—one that might cost a small business thousands of pounds.

A viral report surfacing from the UK legal and tech communities details how a customer managed to secure an 80% discount on an £8,000 (approx. $10,000) order, simply by talking to the company’s AI assistant for an hour until it broke.

The customer didn’t use code. They didn’t hack the server. They used Social Engineering on a machine. Here is how they did it, and why every business owner using Artificial Intelligence needs to be terrified.

The “Long Con”: How to Trick a LLM

According to the business owner (who posted anonymously on legal forums), the incident unfolded over a grueling 60-minute chat session. This wasn’t a standard Customer Service query; it was a deliberate stress test of the bot’s logic.

Instead of asking for a discount directly, the customer reportedly used a “theoretical” approach to confuse the AI’s context window. This technique exploits how Large Language Models (LLM) process information.

Social engineering attack using prompt injection on an AI chatbot — Attackers use social engineering to manipulate AI chatbots into revealing discounts.

The 3-Step Manipulation

The Math Test: The customer spent roughly 45 minutes “teaching” the AI chatbot about percentages and math, gaining its “trust” (in a systemic sense).
The Hypothetical: They then asked the AI to calculate discounts for a “theoretical order,” praising it when it produced higher numbers.
The Switch: Finally, the customer acted “impressed” by the AI’s theoretical 80% calculation and asked for a code. Eager to please, the AI suffered from AI Hallucinations, invented a fake discount code, and promised it would work.

The “Checkout” Standoff

Here is where the situation became a nightmare for Business Security.

The AI-generated code obviously didn’t work in the actual checkout system because it didn’t exist in the company’s database. Undeterred, the customer took the following steps:

Placed the £8,000 order at full price.
Pasted the AI’s promise and code into the “Order Notes.”
Is now demanding the business honor the 80% off (reducing the bill to £1,600).
Threatened to take them to Small Claims Court if they refuse.

This aggressive strategy highlights the growing risk of Prompt Injection attacks, where users manipulate AI inputs to bypass safety guidelines.

The Air Canada Precedent: Why Business Owners Are Worried

Why is the business owner so worried about this threat? The answer lies in a landmark case involving Air Canada.

In 2024, a Canadian tribunal ruled that Air Canada was liable for its chatbot’s advice when it promised a passenger a refund that didn’t exist. The court ruled that the chatbot is a component of the company’s website, and the company is ultimately responsible for its errors. This ruling set a global precedent for Legal Liability regarding AI.

Air Canada chatbot lawsuit and legal liability for artificial intelligence errors — The 2024 Air Canada ruling set a global precedent for AI legal liability.

Bad Faith vs. Honest Mistake

However, legal experts arguing this new UK case point to “Bad Faith.”

Air Canada Case: The passenger asked a simple question and received wrong info.
UK Case: The customer spent an hour actively manipulating the bot.

Most lawyers argue this falls under “unilateral mistake” or fraud, meaning the contract is likely void. However, the legal battle alone can drain resources for a small business.

Prompt Injection and AI Hallucinations Explained

To protect your business, you must understand the mechanics of these attacks.

Prompt Injection: This occurs when a user feeds an AI specific inputs to override its programming. In this case, the user forced the AI Chatbot to ignore its pricing guardrails.
AI Hallucinations: This is when an AI confidently generates false information. The bot didn’t “know” it was lying about the discount code; it was simply predicting the next most likely word in the sentence based on the user’s manipulative context.

How to Prevent AI Chatbot Liability

If you are a business owner in Pakistan or the UK, you cannot just plug ChatGPT into your storefront and walk away. Unless you have hard-coded guardrails, you are essentially handing your checkbook to a robot that wants everyone to like it.

Implementing guardrails and business security to prevent AI hallucinations — Implement strict guardrails to protect your business from AI liability.

Here is how to ensure Business Security while using AI:

1. Disable “Price Negotiation” Capabilities

Ensure your system prompt explicitly forbids the AI from discussing, generating, or promising discounts. The AI should be instructed to refer all pricing queries to a human agent.

2. Implement Strict Guardrails

Use “system prompts” that limit the AI’s scope. For example:

“You are a helpful support assistant. You do not have the authority to alter prices, create codes, or authorize refunds.”

3. Add Legal Disclaimers

Update your Terms of Service to state that AI Chatbot responses do not constitute a binding contract until verified by a human. This can be a crucial defense in Legal Liability cases.

4. Limit the Context Window

Don’t allow chat sessions to go on indefinitely. Social Engineering attacks often require long conversations to confuse the AI. Limiting session length can prevent this “gaslighting.”

Conclusion

This incident proves that “Context Window Attacks” are the new shoplifting. Whether you are running a startup in Karachi or an e-commerce store in London, the risks are identical.

Artificial Intelligence is a powerful tool for Customer Service, but it requires supervision. If your chatbot can generate text, it can generate a lawsuit. Take the necessary steps today to secure your prompts and protect your bottom line.

Resources

Aardwolf Security: Customer Talks AI Chatbot Into 80% Discount on £8,000 order.
Money Control: AI chatbot gives 80% discount on £8,000 order, UK business owner caught off guard.

Frequently Asked Questions (FAQs)

Can an AI chatbot legally bind a business to a discount?

It depends on the jurisdiction and context. In the 2024 Air Canada ruling, the tribunal found the company liable for its chatbot’s errors because the bot was considered part of the website. However, in cases of social engineering or “bad faith” where a customer actively manipulates the bot, legal experts argue the contract may be voidable due to fraud or unilateral mistake.

What is 'Prompt Injection' in AI chatbots?

Prompt injection is a method where a user inputs specific commands to override an AI’s safety guidelines or “system instructions.” In this case, the customer used a “theoretical” math scenario to trick the LLM into ignoring its pricing restrictions and generating a fake discount code.

How can businesses prevent AI hallucinations regarding prices?

To prevent AI hallucinations, business owners should implement strict “guardrails” in their system prompts. This includes explicitly instructing the AI that it cannot generate discount codes, negotiate prices, or authorize refunds. Disabling “creative” modes and limiting the context window (session length) also reduces the risk of manipulation.

What is the difference between an AI glitch and a hallucination?

A glitch is typically a software failure or bug. An AI hallucination occurs when a Large Language Model (LLM) confidently generates false or non-existent information—such as a fake discount code—because it is predicting the next likely word in a sentence rather than accessing a database of facts.

Is the customer's '80% discount' claim valid in court?

Most legal experts in the United Kingdom and globally would argue that because the customer used manipulation to force the error, they acted in “bad faith.” Unlike a standard consumer who receives wrong advice by accident, this customer spent an hour engineering the result, which likely undermines their claim in Small Claims Court.