Judith Lewis: AI reports look good, but break under pressure

You're smarter than an AI. Yes, I said it. You are smarter than an AI. Why? So, there's nobody in any agency or in-house business that has time to wait for Chad GBT to get around to things. And don't forget, it lies. It hasn't advanced anything. But it did it really, really, really fast. I mean, if I didn't know better, I would be like, "Oh my god, I'm out of a job." We can trust SE Rank. We cannot trust Chat GPT to give us the truth because it is not a tool. It it is an LLF. No matter what your imposttor syndrome says, you are better than an AI, believe it or not, because we need to use the right tools in the right way. Please welcome Judith Lewis, founder at Decabit Consultancy. And please tell a few words about yourself and uh where you occupied with right now. A few words about myself. I'm Canadian. Just uh before anybody asks the obvious question, you sound really American. No, I'm Canadian. Uh, what am I doing right now? Mostly panicking about the fact

that my slideshow is more than 50 megabytes, so could load. So, I'm going to be sharing my screen and I'm sort of hoping I've not left anything embarrassing open. Uh, although of course, briefly, people will see the tabs of death that I keep open all the time. That's my pride and joy. Over a hundred tabs open at any one time in any one folder. That's that's both my pride and my shame. Other than that, I do digital uh marketing consulting. So, I do more than just SEO, but of course, I am an SEO first. So, um uh SEO is my first love. And I did a course for Wix, which I put a a link in for everybody here on reporting, but my younger brother is a mathematician, so he and I talk about algorithms and AI and things like that quite frequently. So, that's um what I'm going to be talking to everybody about is AI and reporting. We already are too excited to keep waiting. So, please take the stage and let's dive into. Right. So, this is the part where

I share my screen and I'm going to share my entire screen in a risky move that everybody thinks will go wrong and it might very well go wrong. But hey, what can I say? I am very much hoping that everybody can see the lovely presentation that I have prepared for everybody. Hopefully, you're all seeing this and you're seeing the title slide, which is a little longer than probably something from AI could have come up with, but it's AI enhanced reporting for smarter strategic insights. I did put a little bit about me in here, but like you can use the search engine of your choice to look me up. I've got stuff all over the place. Uh, I could, you know, just wow you guys with tools. As you can see, lovely animated slide here because I love my animations and my gifts. Uh, I could show you how to, you know, use chat GPT to create a report in seconds. Um, you can see here I've done a, uh, screen capture of me asking ChatGpt to do just that. And it's kind of worrying how quickly it actually did that because

it was seconds of time and it came up with a deck. Uh, I can show you how to impress clients with graphs and data because data is the sexiest thing. Um, but, and there is a big butt, and I know I'm a I'm a bigger girl, so I've got a big butt, but there's a big butt, a big caveat to this. Um, and that is, how is this actually going to help your clients? How's this actually going to help you? Because if you think about it, you've been hired by um either an agency or by a client inhouse direct um to actually make a difference. event every month or every quarter or well I hope that uh if you're inhouse you're doing something to sort of demonstrate the awesomeness that you are monthly or quarterly uh but normally you've got to demonstrate that you're adding value besides just ranking keywords higher. You know it's it's not just about doing stuff. It's about helping people make other people look good, right? You look

good because you've ranked higher, but you're helping your manager look good because they are getting more traffic and more traffic hopefully means more sales and that is ultim or leads or whatever and that is ultimately what we're aiming for. But I got to thinking like this is about reporting um and using AI to to do some reporting and I thought well you know can AI really do a better job than us? I mean is this it? Is this the end of SEOs? And I always go to my handydandy image generators because I love them because they prove to me and to the world and my younger mathematician brother that it's not quite there yet. And uh while I work in machine learning uh we know you have to teach and teach and teach and it still gets it wrong. So um I I love using images like this because I'll tell you it's supposed to be pigs in blankets. Um and it just missed the blankets. It also missed the pigs and blankets. That is not gravy. That is a Christmas cracker. But it did get the roast potatoes and braised red cabbage, right? So, hey, you know, sort of quasi. Okay. Mostly 20%

right. It's good, right? It's fine. And then it gave me this presentation and it made some pictures and I was like, what? Why? It's a I've asked you for a technical audit and you've given me a woman with fruit and then another woman with no elbows and then another woman that's kind of having problems with how many arms and hands she has. And this is what the AI chose to give me. And why does it happen? Well, I just want to talk briefly about why it happens so that you know why you're smarter than an AI. Yes, I said it. You are smarter than an AI. Why? Well, AIS have to break stuff down into they call them tokens, but you can think about it like creating something in a language like translating it into a language that the computer understands. Now, normally, you know, when we were working in basic and Pascal because I am that old. I mean, you know, I started in 1985 and that's a long time ago now. You know, I don't want to say how many you can do the math, but hey, it was a long time ago that I started programming and we were taught about assembler and other fundamental core languages. And that's basically what's

happening here. But what do we know about things like translations, especially movies, lost in translation, uh it can not always go as well as we hope. anybody that has used Google to translate a sign as I did once. Uh I work in Fodenneal. Uh and I went in the in the the lab and I was trying to translate something and the things it came up with were hilarious. I have taken so many menu screenshots of what Google was trying to tell me the food was. To a certain extent, I'm not sure what canvas parachute is, but one day I will learn what that dish was. I wasn't brave enough to order it. But that's the hallucination of translate. And it's the hallucination that you'll find because if you think about it, it is a narrative explanation. And if it doesn't understand and it just fills in the blanks, uh that's when it goes wrong. Now, here you can see an example. I was asked I I figured this was safe, right? It's not a brand new Mac OS. It's been around for a while. So, the algorithm should have been trained on it. It should know. It should have learned. you

know, it should know that, you know, Seoia is a new Mac OS operating system from uh 2024, I believe, and it's 2025. So, I gave it time. It's not like it happened yesterday. And it still didn't know. And I kept beating it uh until I gave it a URL and I said, "This is what I'm talking about." And it was like, "Oh, yeah. Well, I don't know." Uh so, the thing is, we've been dealing with ambiguity as searchers for decades. It's not even years. It's decades. We have learned that Google doesn't give us or Bing or BU or Yandex or um Sesnam or neighbor don't give us the answers that we need out of the box all the time. Sometimes we have to use the most important tool in our toolbox, our brains to figure out what's going on. And as a result, we have dealt with ambiguity for decades and worse ambiguity back in the spammy days of your I remember them fondly. We used to rank on a few hundred pounds, but uh

we've learned to deal with it. Not so much for AI. Machine learning algorithms and deep learning and and now with what we're calling AI, which is large language models, they don't really handle ambiguity. They fill in with the best answer that they know. And that leads to what we call hallucinations, which isn't totally fair because it's not really hallucinations. What it is is a near match, which Google has been doing for years and Bing, etc. Um, but because it doesn't have the answer, it hallucinates. Which is why you have to stay involved in the reporting process because this is only partially correct. And if you didn't know what these things were, and let me tell you, those soup dumplings, best in the world. If you didn't know that it had misnamed that, you would have gone to Georgia and ordered them and been sorely disappointed. So I said to Chad GBT, "Let's do a technical audit." Yeah. What What's the worst that can happen? So many times it's worse than what you think. So I said, "Let's do a technical audit." Okay, if Hotel Chocolate never had them as a client, like to eat their

products. So I went to their website and I was like, "Oh, you know, this makesh it makes sense what you've done. So let's make some slides." And it went about as well as you can imagine. Uh here's its first slide. I I I think my niece, who is 10, could do a better job. And it gave me some lovely pictures there. SEO audit picture. Pulled in that picture from the internet. And then it gave me some sections and it went through the sections and I was like the random bridge picture. Sure, why not? And then, you know, another SEO tile picture, you know, they're all just onepoint letters and Scrabble, which is disappointing, but you know, it's given me some more stuff and it's given me more stuff. And at this point, I'm like, you know what? If the boss, if my manager at my first agency when I stopped being inhouse and I actually had to like prove my worth um saw these slides, I would have been shocked. Um so like really honestly I like the little typewriter with the Google Core update. Sometimes I feel like that's the um most

advanced technology some of the clients I have is. Um but uh you can see here it hasn't really helped. It hasn't advanced anything, but it did it really, really, really fast. I mean, if I didn't know better, I would be like, "Oh my god, I'm out of a job. This thing is doing an awesome job." And it said that there were orphan pages, and I was like, "So, so where is it? What is it?" Like, I don't even have to run Screaming Frog. Oh my god, you know, my laptop overheats half the time, so, you know, I don't have to do it. And then it was like, "This is what orphan pages are, and this is where I think they are." And I was like, "Did you really check?" Like, "Did you actually do the work?" It's like dealing with a like a new exact like somebody who's just come in out of university or something. You've just hired them from another another industry. Maybe I don't know, maybe they were a marine biologist before and they've come in to SEO and and I did hire a marine biologist once. That's not a random example. Um, but she was very good. Maybe you hire a new marine biologist that wants to switch careers

and wants to become an SEO. And you're like, well, you know, did you did you check anything? And hopefully a well-trained marine biologist would say, "Yes, yes, I did." Um, but chat GPD is like, well, you know, I kind of reviewed this and then I just guessed. I just guessed is basically what it's saying. Oh, I did these things, but then I didn't. If you ask ChatGpt for the details of what it did, it didn't do those things. It lied. It said it did things and it didn't. But it's a bit scary because it looks like it did. And if you don't know any better, like if you don't know to ask the questions, cuz don't forget people, we ask the questions. Computers, not so much. Also, CEOs, not so much. If we don't ask the questions, then it looks like Chat GBT did a better job. small business owners, the ones who are more likely to be like, "Well, try GBD." Um, are the ones that we're now going to have to demonstrate our value to. You know, it's it's not the first time. It's

not my first rodeo. Um, it won't be my last time saying, "Well, you know, if you can just go to Chad GPT and say, do a technical audit. Don't need SEOs anymore, right?" Uh-uh. Uh-uh. I've already shown you. I have already proven that we are better than AI. But it falls down here. See when you start asking the questions cat and you can see I start getting cross with Mr. AI I I try to be light, conversational, and almost Googlel like in my instructions. And then I said, "Can you just list the orphan URLs you were able to find?" And it was like, "Well, they're likely orphaned." Like, you know, I didn't quite do the work. I just One time when I was doing a report for a client, I was in Toronto visiting my family. The client needed a report. I just exported the data and I tried to make it look sexy and I didn't check it. And there was a link that we had gained um due to some weird placement of things from a gay Thai bride's uh website which was not aligned with the client's direction

which was a B2B solutions provider. Um so you know that website provided solutions but not the kind that the client did. So when we don't check our work we end up being Judith uh in her more jun junior days. So let's not be like junior Judith right. Um, so I asked Chachi Kim like, "Can you just tell me? Just give me the list because I want to go in and I I didn't say I want to go in. I want to check I want to check your work." Well, couldn't give me my work. Um, and it can't run a live crawl. So, Screaming Frog still absolutely has a place in our toolkit, as does SE ranking, of course, because of the lovely AI reporting help that they can give us. And then it's like, well, you can do it yourself. But you've just said that you did it. Oh, jeez, man. You missed a bunch. Seriously, it's a rubbish, rubbish, rubbish result, but it looked convincing. Despite the rather questionable styling of the uh slides, it looked convincing and the output

without the slides looked even better. It looked really convincing because it gives you five bullet points and they look reasonable, but they're not because it lies. So we can't trust the analysis. We can trust SE ranking. We can trust Samrush ARS etc. We cannot trust Chat GPT to give us the truth because it is not a tool. It it is an LLM. And then I went in and I was like can you create a monthly report? And it was like sure here's the traffic. Here's the this. And I was like I'm sorry the traffic. Do you see there on the bottom of the slide where it's like this is the traffic? Yeah. that that part sessions organic users bounce rate that how did it get that oh what the let's see so it made the numbers up it was like well it's a representative simulation I was not telling you the whole truth when I said well here's the data I was

just making it up as I went along thanks Chad GBT you just got me fired if you weren't checking your work but you know and I know that's why we are still important. So, we know that Chachi just like a small Santa sits on a throne of lies. Don't trust it. So, what else did it say? Oh, I don't have uh access to any of this. But you could if you have these tools. Sure, I have these tools. But that wasn't what I asked. I asked you to do it for me. So, it tried to simulate realistic traffic and it gave me lies and then I said, "Well, what data would you need?" And it was like, "Well, you know, I'd do this, this, and this." And, "Well, you could have just done it yourself." No matter what your imposttor syndrome says, you are better than an AI, believe it or not, because we need to use the right tools in the right way. And these are some tools, like, you know, I showed you a lot of tools in the

beginning. A lot of those are writing assists. These are the ones that are actually going to help you with reporting and you are going to help you with reporting. Um if you don't have access to an excellent tool like um SE ranking and I'm not saying that because this is their webinar. It's actually a really good tool. Um, otherwise I wouldn't say it was. I would say, you know, somebody else was, uh, Semrush or or AHS. But no, it it is a really good tool and I think that, you know, tools are important, but so is your brain. And I want to make sure that we're not dismissing the importance of people's brains and adding information and strategic insight to a report that is being produced because you know yes actually you know all these different tools could actually do a good job but they don't explain things. So that smart small CEO agent um not agency small company CEO that goes into chat GPT and says can you tell me what I need to improve about my website is wowed by all the tech speak isn't going to realize

that it's all lies and incorrect information based on assumptions that don't necessarily hold out for their business. Only you can do that. And also if you're inhouse it is a great way for you to demonstrate why it's important that you are in house. It's your brain that is the most important bit. It's not chat GPT. You can use chat GPT carefully, but maybe, you know, be very very cautious. I tried to force it to to create a template. It couldn't create a template. It said it would need about a day. In an agency, that's way too long. Usually, your manager has said, "Oh my god, the client wants a report now." Um, or you're overworked and underpaid. usually. I mean, I think that's not a generalization. I think that's broadly the truth in any agency. Um, we all feel overworked and underpaid in an agency. Um, or in house as well. I've been both sides of the coin. We feel the same. Um, and also the extra added bonus, no one

listens to us. They hire somebody in from outside to say what we just said only for more money. So, you'll see that you can ask it. It takes forever and then never gets back to you. kind of like that guy that you met or girl that you met at the bar and gave you their number and then they just blocked you immediately. There are a lot of things you can I'm married. I'm using these analogies. I'm married. Back in the day, we didn't have phones, you know, when we were dating. I had a pager though. I was way ahead of the curve. It was, you know, the 90s, but you know, I am that old. Uh you can use AI for a lot of stuff. you can use it to to actually, you know, crunch the numbers and help you visualize things, help you explain things a little better because don't forget CEOs of small firms need things a little bit more people friendly. And I especially, I've been doing this over 25 years now. I use jargon still with clients that they don't understand. And I have to keep catching myself and explaining things using analogies that they can actually understand. There are

lots of things that AI can do for you, but a lot of it is like a tool. It can do an analysis. It can check things. It can rewrite things. It can automate bits. But if you ask it to build a Looker Studio template for you, it will take 48 hours, but once you've got it, you know, you can play with it and add your commentary and it's great. Um, so the problem with it, you know, taking a long time, we're faster as people. We have the right tools, whether it's SE Ranking or Semrush or AHFS or anything. We have the tools always watching for us in the background and we know how to use them so we can put together a report. But the key is and I'm going to show you this in a second. I'm getting ahead of myself that we need to add the insight. So there's nobody in any agency or in-house business that has time to wait for Chad GBD to get around to things. And don't forget it lies. So you don't have to double check all of its work anyway. So it's only making more work for you. You can use it but use it sparingly. You know, because my younger brother is a mathematician and we talk

about this stuff a lot, I really don't trust them because I know the myriad of ways in which they don't quite work as we expect them to. But what we can do and what I'd like you to do hopefully the one thing that you'll take away if nothing else from this is that I would like you to create a template and I would like you to populate that template. Okay? So if you know if chat GPT is tools, what do we do? We add insight. Chat GPT has no idea what your client or what your boss wants. And so what I want you to do is create whether it's a set of slides or it's a spreadsheet. Please don't give the client a a spreadsheet or the boss. It's ugly. It's very difficult to see and read. I'm dyslexic. I have trouble even with spreadsheets for clients. I have to break them down, break out um speech boxes. So, I actually create boxes where I put text in just to help me, let alone the client. So, what I'd like you to do is I would like you to create a template if you don't already have one. Um, and a

lot of people don't have them. They just kind of export stuff out of the tool and then they put it together and then they have like a front page of like a really quick exact summary and it can be really baffling for clients. And if you want to keep the client and you want to grow the client, you need to demonstrate the value you're bringing. And it needs to be close to reflecting the bottom line. So maybe you know for yourself, even if it's just for yourself and you never share this. You you save a different version of the presentation every single month. Put a slide in there that says, you know, this brand, I don't know. I I for here I was like okay brand X whatever bring swag to a new level and will continue to increase their market share. That's what they want to do. They they want to be global swag providers for companies to brand stuff up. So that's your slide one that reminds you always what the north star that you're pointing the whole company is pointing towards. Then I want you to think about what are your KPIs. Is your KPI

financial? Is your KPI I don't know uh traffic based? I had a client once whose promotion and bonus was based on more subscribers to the newsletter one quarter. Yeah, that was that was fun. It was easy, but it was fun because that was what his needs were. And so I did what he needed. It didn't matter what was also needing doing. We had that later on in the deck. But the first thing I talked about was what he needed out of us and that's what we had to report on. And so here I've said you know like say SEO's KPIs are like 2 million in annual turnover and that is X proportion of the turnover probably 20% or something like that. And then I want you to think about how each part of the report relates back to demonstrating the value that you're bringing. It's as simple as that and as complicated as that. But if you relate everything back to that, each month that you report to that client, you demonstrate how you as the agency, the freelancer, the the the employee has brought more value to the

company as a result of you being there. And you're going to show why AI, while an interesting tool and somewhat slow, is actually really just a tool. And you are the one that is helping drive the SEO strategy. the digital marketing strategy. You're talking about the time spent in these different places. There are lots of things that AI can do, but there's a lot of things that it can't do. And it can't understand your client, your client's politics, your client's needs. It doesn't understand that structure, the internal structure of your client or or your own company. So, you really need to think about this. AI analysis can be flawed, but your report is checked by you. And if you This is a free course that I developed with Wix. I I I didn't want to just restate the obvious here that you could get in that course. But if you design your report to demonstrate your value, you're not only

going to help keep your job or help keep your agency employed by that client, you're going to help win them more work or yourself because you're going to demonstrate how you are focused not on just doing SEO for SEO's sake or or PPC for PPC sake. You're focused on delivering value. And that is what every client wants is for you to deliver value. So that's the one thing I want you to do is create that template that is going to help you demonstrate value because tools can help us but only if you are delivering proper strategic insight and only you can actually do these things. AI doesn't know what your boss wants. AI doesn't know what the direction clearly what the direction of the company is because hello it just said some lies to me about traffic. Those were lies. Outright lies. Please don't let AI drive your strategy or your reporting. You are the most intelligent tool in the toolbox. You're the sharpest tool in the toolbox. You're the brightest spark in the Tinder box. And

don't let AI make you feel less because it lies. And it makes you look it makes it look like it knows what it's talking about, but it's lying. So just be careful and check its work or use a tool like SC Ranking, Semrush, or AHS because those tools are designed for the job. AI can be your best friend. It can really help you, but it'll never replace you because you are the one that's smart enough to understand what needs to get done. So, don't just blindly trust Chat GPD. It looked really impressive. I know those slides, they were like amateur hour, but the slides and its output looked so impressive, but it was full of lies. So, don't trust it. Trust yourself. Add commentary to your slides. Add speech boxes. add whatever it takes, but you need to interpret the data because without your interpretation of the data, the client just can't understand cuz they are not the expert. You are. So that's it. Thank you very

much for listening to my rant. Thank you very much, Dudes. That was really inspiring, especially the last bit when you're calling us the heroes of this SEO landscape and not uh underestimating with the AI not letting them push us through. That's that's a great shift of confidence for us. Thanks very much for for this part. And uh moving on with the questions. Are you ready? Uh yeah, hopefully you know I I I did try to go quickly and leave some time, but I I definitely want to encourage everyone to have those commentary sections in their reports. So definitely definitely absolutely ready for questions and also to make sure we're fighting the good fight here. Awesome. Awesome. Thank you. Then uh the first one is if we let an AI tell the story of our SEO results, how do we doublech checkck its conclusions without doing all the work ourselves either way? And do you trust AI in understanding the contextual data? So I

I I think that we all know now because so it's the translation element, right? So hopefully I wasn't too I I'm not wanting to be dismissive of AI, but we need to be the ones that look at the data and interpret it ourselves because the tokenization element, it it's not reading the results like we would, integrating them into what we know and then spewing out an answer. It is creating something that really looks convincing. And my goodness, when I said, can you do a technical audit of hotel.com? And it was like, sure. and and it went down and did everything. I was like, "Oh, oh my god, that was fast. How did you do it that fast?" Like, I can't spider the site that fast. You must already have it in memory, etc., etc. And then I started to dig and it was like, "Oh, I just made that up." So, no, don't don't trust a conclusion. Trust data and data analysis. But the thing about it is that we are better placed

than chat GPT because don't forget what I was saying earlier about we've been dealing with ambiguity for decades. So we can look at the data as SEOs as professionals. Our our whole reason for being and the reason why SEO is not dead is because we know how things work. We know how a bit of how Google works and we can look at the results that we've gotten out of let's say J4 and suddenly something's tanking and we're like you know what it's not actually tanking somebody broke the uh the the tag Google tag manager needs fixing chat GPT can't do that for you only people can do that because we're our brains are much more attuned to what we need to be looking for whereas chat GPT is a general thing so you can train you can train an instance to a certain degree but even Even in law, my my dad was a lawyer, so I'm very interested in this. Even in law, they can't do this. They can't use a pre-trained pre like very selective, very like ring fenced guard rails everywhere um LLM for law because it

still makes stuff up and it still gets it wrong. So, please don't don't let Chatt or any other um AI give you the conclusions. You're the one giving conclusions. You're the one adding value. you're the one that is looking at the data and trust me it's really good at taking multiple sources of data and bringing them together and helping you understand them better like taking 8 10 12 competitors and helping you understand it better but like with any tool and I do this with Screaming Frog with Seamrush with every tool I use I still double check it because not even tools are accurate and you have to use more than one tool so I'm checking Majestic for links I'm checking Samrush for links I'm checking everyone for links I'm checking like my own searches for links. I'm checking Google Search Console. Don't let it do it. Sorry, that was a long rant. You are the most important tool in the toolbox. You are the professional. You do the summary. Beautiful, beautiful answer. Thanks a lot. I think we can move to another

question. What are the ethical and practical considerations when deploying AI for both content creation and technical optimization, especially in maintaining search engine compliance and user trust? And and I think this is a really important question because what we're saying to to an a any sort of a an algorithm I'll say when we're saying please do this to my website. I needed to be uh DDS disability discrim or DDA disability discrimination act compliant. That's in the UK. So what changes do I need to make? you're asking something of the uh of the algorithm that you're not necessarily sure about and it might spew back incorrect information. So from a I think from a tech from a technical optimization point of view, the code could be wrong. There is a really good example out there and I don't remember what it was but there is a really good example of even on GitHub code being uploaded where a JavaScript library was made a dependency and that JavaScript library was owned by someone and at any time it could either be removed or it

could be used to inject malware into a website that was using that code and hadn't checked what that JavaScript include that that file was. So, if you're using it and you're not double-checking things, even if you're using GitHub and you're not checking things, then you could uh land yourselves in in big trouble for anything. Also, don't forget that it from an ethical or moral point of view, I burned a small lake worth of water um making those images and having chat GPT do those chats. So, I burned more energy doing a less efficient analysis that came back with lies than if I had done it myself. But it's good for bringing data together and visualizing. It's also good for helping you crunch a lot of stuff, but so are tools. So I think it's really important that we look at this as a help, but that we're not fully reliant on it because it isn't giving us the answers. That's why we are the experts and not a tool. And that's why we'll at the moment with all the guardrails in the world, we don't have an AI lawyer or

an AI doctor because even even in a closed ecosystem, it still doesn't work quite right. Now granted, next year because these things are logarithmic and it's going up at the moment. Uh next year we could have a situation where we do have an AI lawyer, but we can't. And again, it would burn a small city's amount of electricity just to to do one court case. And the question is, should we be doing that or should we not be relying on the experts that we already have who don't burn a small city's worth of power to do one court case. So, thanks a lot for covering at least several questions. The presentation was absolutely great and uh the comments support this idea. So, thank you very much and thanks for joining us today. Thank you so much. And remember everybody, you're the smartest person in the room.