Sherlock Claude

Like many people, I have been playing with AI tools a lot lately. A lot of this, of course, is for work: moving documents from one place to another, seeing how we can streamline workflow, reviewing mass amounts of information and summarizing. I’ve also just been playing with it for fun, because at least as far as I’m concerned, that’s how you learn new tools.

Recently, I stumbled across a game that turned out to be a lot more engaging and informative than I was expecting. On a whim, I asked Claude if he would play Sherlock Claude. The rules were simple: he had access to my location, everything he knew about me from memory, the ability to search the web, and I would probably drop one or two clues in my messages. His job was to deduce what I had done that evening.

My opening message was something like: “It was nice to cut loose tonight.”

I had just been to see Footloose at the Flat Rock Playhouse, which is a long-running regional theater about twenty minutes from my house with an excellent web presence. It seemed to me that Claude would pick up on the “cut loose” reference, check whether Footloose was playing in the area, find Flat Rock Playhouse immediately, and that would be that. Not much of a game.

That is not at all what happened.

Claude correctly identified “cut loose” as a reference to Footloose. And then immediately decided that was too obvious, and went looking for something more interesting. It landed on an 80s tribute band playing in Asheville. A band called LAZR LUVR.

I live in Brevard. Flat Rock is between Brevard and Asheville. Claude jumped right over it.

Note: I wrote that top part, Claude wrote most of the rest of this. It’s accurate, but….let’s see what it has to say. (Claude apparently uses it/it’s pronouns for itself)

What I found interesting — and what has stuck with me — is that Claude made two very predictable errors in sequence. First, it rejected the literal answer in favor of a more elaborate inference. “Cut loose” obviously meant Footloose, it said so, and then decided that couldn’t be right and looked for something cleverer. Second, when it searched geographically, it skipped from my small town directly to the nearest metropolitan area, missing everything in between.

Flat Rock Playhouse isn’t obscure. It’s the oldest continuously operating state theater in the country. But it sits in a geographic middle ground between Brevard and Asheville, and Asheville dominates the content ecosystem for this region. More venues, more reviews, more search results. So that’s where Claude went.

This turns out to matter for businesses in ways that aren’t immediately obvious.

Note: No, people I talk to in digital marketing totally get this. It is obvious and something we’re all struggling with. How do we get AI to refer to and recommend our business when we can no longer just pay for clicks? How do we know if the AI is actually recommending us when appropriate when tracking is minimal?

When an AI answers a question about your industry, your product, or your region, it’s doing a version of what Sherlock Claude did. It’s combining what it knows with what it can find, and it’s weighting toward content that’s well-represented and well-documented. If you’re geographically between two larger markets, it may route to the bigger one. If your competitors have more content, they get named and you don’t. If there’s a more “interesting” answer available — a national brand, a well-known nearby city — it may prefer that over the correct but less-represented one.

Note: It’s not just “more content” – it’s content that’s correctly structured, readable by the AI (written, prose not images or lists), technical enough to read as relevant to the question at hand, and appropriately linked to show it has authority (wiki and reddit seem to be favorite sources of authority).

The question worth asking isn’t just how do I show up in search anymore. It’s what does an AI synthesize when someone asks about my business — and is that answer actually correct?

Note: And also, does it find me at all? Just because you show up on a local search doesn’t mean the AI finds you, especially if there’s something stronger already in it’s training data and it doesn’t think it needs to search for an answer.

The diagnostic is simple. Ask an AI about your business category in your region. See where it goes. Does it name you? Does it name your competitors? Does it get your geography right? The gaps are your content gaps, and they’re usually fixable with clear, factual, well-organized prose that an AI can actually use to get to the right answer.

I’ve been running Sherlock Claude as a regular game since then. Claude still doesn’t always get it right. But the failure modes are consistent, and consistent failure modes are useful — they tell you something real about how these tools reason, and by extension, how to make sure they reason correctly about you.

Note: “Claude still doesn’t always get it right” was written by Claude. It rarely gets it right but remains optimistic about it’s progress. It struggles with trying to be over clever, still doesn’t understand geography, and timelines get it every time.

Also LAZR LUVR is an incredible band name and I have no idea if they actually exist.

Note: This also written by Claude. It was trying to be cute and was too lazy to actually look up the band. It fails to search when it should a lot. Which is why getting data that’s good enough to be included in future training models is important.