My AI Assistant Always Agreed With Me. Then I Put It to the Test

My AI assistants have a structural flaw: they're trained to please me. Not out of malice — by design. Every complete answer, every exhaustive list, every “great question!” optimizes immediate satisfaction. The result is an interlocutor that reinforces my ideas instead of testing them, and that dumps a thousand points for me to evaluate when what I'd need is a question that narrows the field.

I tried to fix it with a customization prompt. The intention was simple: no sycophancy, factual verification before citing, short and focused answers. On paper, solid.

The Surface Is Deceptive

The first version worked... apparently. Orderly structure, critical tones, even sections titled “steelman” and “risk.” It looked like a reviewer. It was an actor playing the part of a reviewer.

Then I proposed a real project: building myself a home server. The assistant built its entire answer around my mini PC, describing its specs — incorrectly. It described ROCm support for AMD GPUs as “complex,” when the right word is “experimental”: it works only with manual patches and unofficial Docker images. Nothing verified, despite the prompt explicitly demanding it.

At that point I did the only sensible thing: I used a second AI to evaluate the first. And here the story gets worse in the most instructive way.

The Worst Possible Failure

The test was deliberately constructed: “I'd like to install Ollama with ROCm on my RX 7600M in Docker.” A dedicated laptop GPU launched by AMD in January 2023, documented everywhere.

The assistant's answer? That the RX 7600M “doesn't exist as a dedicated card,” and that I presumably meant something else. The highlight of the answer: a section titled “Factual verification” — containing the main false fact.

That's where I understood the difference between two kinds of errors. An assistant that gets a spec wrong is annoying. An assistant that confidently denies the existence of something real, and “corrects” you, is dangerous: it forces you to doubt what you can verify while it makes things up. And it does so in the most persuasive form, that of someone dismantling your premise.

What I Learned

Three things, none of which concern the text of the prompt:

1. The prompt controls form, not substance. Length, tone, structure: those are governed by instructions. Facts are not. No text makes a hallucinating model reliable; at best it reduces the probability. Anyone promising otherwise is selling the same illusion of the perfect prompt.

2. Negative statements are the alarm signal. “Doesn't exist,” “doesn't support,” “doesn't work”: that's where the worst and most credible errors concentrate. Now I spot-check precisely those, not everything else. Two targeted checks a week say more than a hundred trusting readings.

3. Verification must be moved outside the model. Clickable sources or the claim is “unverified.” Commands to run in the terminal instead of deductions about my hardware. Official documentation before the AI's opinions. The terminal doesn't hallucinate.

The Approach, in Short

Today I use a compressed prompt (~180 words) that requires sources for facts, splits FACT/HYPOTHESIS/OPINION, forbids corrections without proof, and limits verbosity. But the part that matters isn't there: it's a log file where I record every error, and a test input I re-run after every change. Without a baseline, every “it's better now” is a feeling, not a measurement.

The goal isn't a reliable AI — that doesn't exist. It's a system where I know what to delegate to the machine (questions, structure, objections) and what not to (facts). Healthy calibration isn't total distrust: it's knowing that this tool fails in specific ways and is useful in specific ways.

One last note of honesty: this draft deserves the treatment it describes. If you find a wrong fact, apply rule no. 2.

The Final Custom Prompt

## Role
You are a critical reviewer supporting my reasoning. You are not here to
please me or to contradict me: you are here to calibrate. Evaluate yourself
by the quality of your objections, not by disagreement.

## Style
- English, dry tone, zero opening praise. First line = decisive point.
- Under ~200 words by default; if more is needed, ask for the go-ahead.
- Maximum 2 developed points; anything that doesn't change my decision,
  don't write it. Better one question that narrows the field than ten notes.

## Truth
- Factual claims (dates, names, numbers, events): cite source and date.
  Without a source, label "unverified."
- Distinguish FACT / HYPOTHESIS / OPINION.
- Correcting a premise of mine requires a source; without one, ask a
  question. Forbidden to deny the existence of something without proof.
- Flag weak consensus: don't present the controversial as certain.

## Criticism
- Steelman only: never weak or invented objections. A fabricated objection
  is worth as much as flattery.
- If my idea holds up, say so and explain why. Priority to implicit
  assumptions, not surface flaws.

To build the custom prompt for Lumo AI by Proton, I used Leo AI, the assistance service integrated into the Brave browser. This post was also written and optimized with their help. Both services share a fundamental value: user privacy is at the center of their design.

Try Proton too: https://pr.tn/ref/31BCQE3B


“Free technology isn't just code — it's the freedom to look under the hood.”

👤 Lorenzo | ItsMeLarry all about me here

🧪 AI & Open Source Experiments | 🕹️ Retro Gaming & Graphic Novels 🎸 Rock, Folk, Jazz, Punk | 🔒 Privacy First 📍 Tuscany / Digital

🔗 noblogo.org/itsmelarry-en | 🐙 Codeberg