You’ve Tried AI for Your Website Copy. Here’s Why It Didn’t Work.

Melissa: Hi, everyone. Thanks for joining us today. I’m Melissa, Client Success Manager at iOnline, and this is Signal Sessions episode three: You’ve tried AI for your website copy. Here’s why it didn’t work. I’m joined today by Duncan, our content strategist. Over to you, Duncan. Let’s start with what LLMs are and how they actually work.

Duncan: I’m sure everyone listening is familiar at a basic level with what a large language model is. That’s the type of program we’re talking about when we say AI — a computer program that can read and generate text.

Duncan: These models are trained on enormous amounts of data — trillions of words, taken from websites, books, speeches and more. That training data gets broken down into units called tokens, each roughly three-quarters of a word. The model then maps out associations between tokens, learning how different words and concepts relate to each other. In a rough sense, it’s learning which words are most likely to sit next to each other in a given string of text.

Duncan: This style of text generation is called probabilistic. Rather than being deterministic — like a traditional automation system, where you input X and always get Y out the other side — the result is generated based on probability. That means an LLM will produce a slightly different response every time.

Melissa: So if I asked the same question twice in ChatGPT or Claude, would I get two completely different answers?

Duncan: You’d probably get answers that are similar, but not identical — how different depends on the question. That probabilistic nature is also behind an issue you may have heard of: hallucination. That’s false information generated as a side effect of the way these models work — a probability-based guess that comes out wrong instead of right.

Duncan: Worth clarifying some terminology here too: LLMs are the underlying models. The chatbots consumers actually use — ChatGPT, Gemini and others — sit on top of an LLM.

Melissa: If a business has already started using an LLM for their website copy, and the copy reads fine — it sounds professional, it’s accurate to the business — why isn’t that enough?

Duncan: Sometimes it is enough. That’s actually a big part of what I want to cover today: there are good use cases for AI copy, and there are cases where you’re better off with human-written copy — it depends on the context.

Duncan: Rather than focusing on how the copy was generated, look at what level of quality it needs to hit to deliver a marketing outcome. That’s what this graph is about. I won’t go through it in full — it’s a little technical — but it shows the constituent components of copy, stacked from foundational to refined.

Duncan: At the bottom: the fundamentals — brand identity, customer research. One level up: offering, messaging, information architecture, and broader research into the subject matter. Above that: rhetorical elements, like actually making an argument for change. At the top: polish — still valuable as a differentiator, but the least important layer. Whether any of this matters depends entirely on the use case.

Duncan: It’s worth clarifying what copy actually is, since the term gets used loosely in marketing. Copy is words designed to drive a specific action from the reader. It makes an argument for change and gives the reader the tools to act on it — whether that’s information, a walkthrough, or something else. Website pages, billboards, digital ads and persuasive speeches are all examples.

Duncan: Whether copy works or not comes down to how it performs in the real world — once it’s published and in front of consumers, does it deliver the result you want? That’s the threshold we’re aiming for. How the copy was created doesn’t matter. Whether it works does.

Duncan: That brings in a useful marketing concept: threshold effects. Most things you do in marketing aren’t linear — effort doesn’t produce a gradually increasing outcome. Often, nothing happens until you cross a specific quality threshold. If you’re posting on social media consistently but the quality isn’t there, you won’t break through no matter how long you keep going. You need to hit a certain bar before you start connecting with an audience and generating revenue.

Duncan: The same applies to copy: you need to clear a quality threshold to get cut-through and persuade someone to act. Synthetic copy generated purely by an LLM often falls short of that threshold. “Something is better than nothing” isn’t always true here — if you’ve drafted a whole website using AI and it sits there for three months producing nothing, you’ve lost real money in opportunity cost. And there are other costs too: brand affinity, and straightforward lost revenue.

Duncan: The examples on this slide illustrate brand affinity and revenue impact. I won’t go through every one, but a couple of quick examples: brands that published AI-generated copy and saw a clear negative reaction from their audience. People can detect AI copy fairly reliably, and they don’t respond well when a brand they already felt some affinity toward starts publishing it. The reasons are complex and tied up with broader concerns — jobs, environmental impact and so on — but the reaction itself is consistent.

Duncan: The other charts here are SEO traffic graphs over time. The spikes show where a site started publishing large volumes of AI-generated copy — traffic jumps, which is exactly what the business hoped for. Then it crashes on the other side, once both consumers and Google recognise the content isn’t serving its purpose. In this example, traffic actually dropped below where it started — in this case, that initiative likely damaged the brand, possibly permanently. Not something to find out the hard way.

Melissa: Does that risk only apply to big names, or does it scale down to local businesses? Most people watching today are probably local or small to medium businesses.

Duncan: On the brand affinity side, those examples are large corporations, and some of that reaction is tied to their scale specifically — so I’d say it applies less directly. That said, people still don’t like it, so if you’re building a positive brand reputation, it’s not a risk worth taking regardless of size. On the SEO side, that’s about AI copy used at scale, and that applies to any website — it doesn’t differentiate between large and small. If you publish a large volume of AI-generated pages, you’re likely to see a similar pattern.

Melissa: We’ve touched on cases where AI copy can be helpful and time-efficient, and cases where human input really matters. Let’s look at the strengths and weaknesses.

Duncan: LLMs are fast — no human can write as quickly. That’s the main strength. They’re also currently cheap, though that’s worth flagging in brackets: most providers are on monthly subscriptions right now, but if that shifts to token-based billing, the cost of generating copy could increase significantly.

Duncan: LLMs are also grammatically clean — they rarely produce errors and write tidily, which is useful for things like emails. And they’re very good at quickly synthesising information from multiple sources — aggregating what several websites say, or summarising a set of documents you feed them.

Duncan: Those strengths suit certain applications well: documentation, basic product descriptions, high-volume ad variations, localised landing pages — if you’re running an SEO campaign across a number of suburbs, an LLM can turn out a solid first pass for each one — sales decks, instruction manuals, and notification emails. These are functional, high-volume use cases where an LLM will be more cost-effective than a human copywriter.

Duncan: Now, where LLMs fall short. Because of their probabilistic nature, they hallucinate. They also blend sources — taking two sources that cover the same topic from slightly different angles and merging them into something that’s technically coherent but doesn’t actually make sense. Both are real weaknesses.

Duncan: They also require prompting — someone has to sit behind the chatbot, feed it information, and manage the process. If you’re a business owner doing that yourself, it’s worth asking what your time is actually worth and how long the process is taking you. Often it’s more cost-effective to pay a professional.

Duncan: LLMs also struggle to hold a consistent style or brand voice. Even with a style guide or similar reference, they tend to revert to their base style — this is a byproduct of reverting to the mean of their training data. That distinctive default voice can work against you if you’re trying to achieve genuine cut-through.

Duncan: They also don’t have empathy for users. At the end of the day, these are computer programs — good at replicating patterns from data they’ve seen a lot of, but not equipped to handle something genuinely novel, which sits at the fringe of their training data. A human copywriter, by contrast, can design intuitively with other humans in mind. And we’ve already covered how AI copy can damage consumer sentiment.

Duncan: Most importantly: LLMs can’t breach the digital divide. They only ever return information that already exists online — they can’t go out into the physical world, talk to your customers, or experience your product. All of that has to be fed to them. You need a human in the loop to bridge that gap.

Duncan: So, to summarise: keep the human touch anywhere that’s carrying your brand differentiation. For low-stakes content — basic descriptions, documentation, anything where consistency matters more than distinctive voice — an LLM can help you build that content efficiently.

Melissa: Let’s look at how to work out whether your web copy is actually working. The goal of your website is always to convert visitors into enquiries, so we want to see that reflected in the data.

Duncan: As I mentioned earlier, it doesn’t really matter whether you used an LLM or a human — what matters is whether the copy works. So let’s say you’ve already got copy live on the site, however it was written, and you want to know if it’s working or needs reworking.

Duncan: Copy doesn’t exist in isolation — it sits within your website design, which makes it hard to isolate and test in isolation. Split testing is one option, but as a small business, you probably don’t get enough traffic to make that viable. A simpler litmus test: does two to five per cent of your website traffic convert? That’s a solid general range. If you’re not landing in that range, that’s your signal to start drilling down — is it the audience, the messaging, one of the other rhetorical elements we covered earlier, usability, or something else. At that point, you’re in problem-solving mode.

Melissa: How would I actually find that number?

Duncan: Through Analytics, assuming you’ve got it set up — that’s your website measurement tool. Divide your number of conversions by your number of active users and multiply by one hundred (worth double-checking that formula, but that’s the gist). The key events you’re measuring depend on your business, but for most small businesses that’s phone call clicks, form submissions and email clicks, divided against active users.

Melissa: If my conversion rate comes in below two to five per cent, what else can I do to assess the effectiveness of my website copy?

Duncan: Start by ticking off the low-hanging fruit — the basics. I won’t run through every one of these on screen, but they’re the fundamental questions most product and service pages should answer. If you’ve got a service page that isn’t converting the way you’d like, go through it and role-play your ideal customer. These are typically your highest-conversion pages, so they’re worth the attention.

Duncan: A customer should be able to land on that page and immediately understand what you offer — it might surprise you how many websites don’t actually make that clear. They tend to talk around it, or lean on jargon customers don’t fully understand. Go through the page and check it against the checklist questions — that’s a strong starting point for most businesses. You’ll get the full checklist, along with guidance on how to apply it to each of your pages, in the replay email, so there’s no need to go through every question individually now.

Duncan: Diagnosing underperforming copy can be difficult, since there’s a lot wrapped up in it — it could be the design, or any number of other factors. As a starting point, underperformance is more often caused by the lower-order elements in that pyramid we covered earlier — the likelihood shifts toward those higher, more polished elements the further up you go, though it’s still worth paying attention to them, including how the copy actually reads.

Duncan: Once you’ve run the litmus test and checked where you land against that two to five per cent range, you can start a root cause analysis to pin down what to improve. If that’s difficult to do on your own, you can always talk to us, or another provider, for help.

Duncan: Final takeaway: if you’re considering an AI chatbot to draft your copy, be cautious about relying on it outside the strong use cases we’ve covered. If you go ahead and build out a whole website on that basis, it can take a while to discover it isn’t working — and that delay costs real business and money. Worth keeping in mind whenever you’re weighing up that decision.

Duncan: That’s everything from me. Back to you, Mel.

Melissa: The main takeaway from today: run the litmus test on your own website. Check whether two to five per cent of the people who land on your site actually convert. If not, that’s where to start.

Melissa: If you come away from today with any questions we haven’t covered, or think of something afterwards, please reach out — we’re happy to talk. You’ll get the replay in your inbox within 48 hours, along with the free checklist Duncan mentioned: thirteen questions every product or service page should answer, plus some of the extra examples we touched on today.

Melissa: Thanks so much for joining us, everyone, and we’ll see you next month