The Quiet Skill of Vetting Expertise: A Title Is Not the Truth

Sixty-eight market experts. Six thousand five hundred and eighty-four public predictions over eight years. Average accuracy: 47 percent — lower than a coin flip. That scorecard comes from CXO Advisory’s “Guru Grades” project (2005–2012), and it is exactly why vetting expertise — the quiet skill of checking whether a claimed authority actually holds up — belongs in your daily media diet.

This is part three of a five-part series on how not to be fooled. Part two asked whether the smooth, confident text in front of you was written by a human or a machine. This installment moves the suspicion one step back — from the words to the person saying them, and to the title stitched onto their name.

The penetrating question stays the same across the whole series: but is this real? A diploma, a broadcast slot, a “Doctor” or “Analyst” in front of a name — none of them are a truth guarantee. They are a smooth surface. What we are after is the rougher thing underneath: a record you can actually check.

To be clear up front, this is not “distrust every expert.” That would be its own kind of laziness. Real expertise is precious precisely because it is rare. The aim is narrower and far more useful — learning to tell the smooth fake from the rough real.

Where Vetting Expertise Begins: A Credential Is Not a Guarantee

Start with the uncomfortable research. Over roughly twenty years, the political scientist Philip Tetlock tracked about 28,000 predictions from 284 experts and reported that most did only slightly better than chance — and on three-to-five-year horizons, many did worse than a simple trend-following rule. (Tetlock, Expert Political Judgment.)

The CXO Advisory scorecard points the same way from a different field. Across 6,582 market calls, the average hit rate landed near 47 percent, the spread looked like random noise, and only five gurus out of sixty-eight cleared a 60 percent accuracy bar. (CXO Advisory, “Guru Grades.”)

Here is the finding that should reset your instincts. Tetlock reported an inverse relationship between fame and accuracy — the experts most in demand by the media were, on average, the ones whose forecasts held up worst. “Seen most often” is not “right most often.”

Why the Title Feels Like Truth

Two mental shortcuts do the damage. Authority bias is the habit of granting a claim more accuracy simply because an authority made it — trusting the source instead of the argument. The halo effect is when one good impression (a past achievement, a prestigious degree) bleeds into unrelated judgments. (Britannica; the concept traces back to psychologist Edward Thorndike.)

Put together, they explain how a title quietly turns into a feeling of truth. The credential is real; the leap from credential to “therefore correct about this” is the trick. This series also has a companion essay on how meritocracy launders a handed-down advantage into the clean language of earned merit — the same machinery, dressed for a different occasion.

None of this says the expert is a fraud. It says the diploma settled a different question than the one in front of you. A cardiologist’s board certification is strong evidence about hearts and close to zero evidence about your mortgage.

The Four Marks of Vetting Expertise

So what does a checkable expert actually look like? Research on the rare people who forecast well gives a surprisingly concrete answer. In the Good Judgment Project, the “superforecasters” stood out not for raw intelligence but for a disposition — active open-mindedness and belief updating, which the data found roughly three times more predictive of accuracy than IQ. (Tetlock & Gardner, Superforecasting.) In plain terms: they keep score, and they change their minds. Vetting expertise comes down to four checks that look for exactly that.

Mark One and Two: The Record and the Money

The first mark is track record. Not “does this person sound confident,” but “what did they predict before, and did it come true?” A forecast nobody wrote down can never be graded — which is precisely why the confident pundit rarely volunteers last year’s misses. A claim you cannot check is a claim you cannot trust.

The second mark is conflict of interest — the plain question of who is paying for this view. Industry-funded studies tend to produce conclusions that favor the sponsor, a pattern documented often enough that medicine made disclosure mandatory: the ICMJE standard asks authors to reveal financial relationships, and its disclosure form requests those going back 36 months. (ICMJE; PMC review.) “Who benefits if I believe this?” is not cynicism. It is a question a whole profession decided to institutionalize.

Mark Three and Four: The Other Side and the Territory

The third mark is the opposing expert. One specialist saying something is an anecdote; a field is a spectrum. If a source can only be defended by insisting they are the lone brave voice against everyone else, that is a warning light, not a badge.

The fourth mark separates consensus from personal opinion, and checks whether the claim even sits inside the speaker’s field. The cautionary case is what skeptics call the “Nobel disease” — the tendency of a few Nobel laureates to make unscientific claims well outside their specialty after winning. (Wikipedia; Skeptical Inquirer.) Expertise does not automatically transfer across domains; the laureate Paul Nurse has warned colleagues against hiding behind the prize to opine confidently on nearly everything.

Read the four marks together and a checkable expert has a shape: public predictions, disclosed funding, named critics, and claims that stay inside the field. That shape is the rough surface. It is what a smooth title is missing.

Who Put That Person on the Stage

There is a layer beneath the individual, and vetting expertise means also asking who decides which experts get a microphone in the first place. Fame, remember, tracked inversely with accuracy — so something other than being right is selecting who we see.

Often that something is the algorithm and the ratings. A broadcast panel rewards a confident, dramatic take over a careful “it depends.” A recommendation feed amplifies whatever holds attention. Korean regulators have flagged how the view-count economy pushes creators toward ever more sensational certainty. (KCIE; MS Today.) Fame, in other words, may be manufactured by clicks, not by accuracy — a point this series’ companion essay on the propaganda model takes further.

This is also where the friendliest-looking authority hides its risk. Wellness and finance influencers build parasocial trust — the one-sided intimacy of a familiar face who feels like a friend — while carrying none of a journal’s safeguards. In one survey, 35.61 percent of respondents said they encountered misleading health information on social media “a lot,” with another large share saying “some.” (PMC; BMJ Group.) A personal anecdote, delivered warmly enough, starts getting mistaken for evidence.

The Korean Landing: Three Questions, Made a Habit

Bring this home to a Korean weeknight. A real-estate YouTuber says buy now or you’ll never own a home. A “YouTube doctor” promises a supplement will fix your sleep. A TV panelist declares, with total composure, where the KOSPI lands next quarter. Each arrives wrapped in a title.

The defense is not to switch off and believe nothing. It is to install three small questions and run them automatically the moment a credential appears on screen.

The Three-Question Reflex

“Was this person’s last call right?” — the track-record question. If you cannot find a scored, public record, treat the confidence as decoration.

“Who benefits if I believe this?” — the conflict-of-interest question. Follow the money to the affiliate link, the paid course, the fund, the sponsor.

“What does the other side say?” — the triangulation question. If no credible expert disagrees, you probably have not looked; if the source insists nobody credible disagrees, look harder.

Here is the honest scoreboard behind all of this — the numbers, and what each one actually measures.

Recognition, Not Cynicism

It would be easy to walk away from all this having decided that every expert is a liar and every title is a scam. That is the wrong lesson, and a lazy one.

Expertise is not the enemy here — undressed authority is. The whole point of these four marks is that they do not throw the expert out; they help you find the real one faster. Someone who publishes their record, names their funding, and corrects their mistakes is showing you the rough surface on purpose. That is the person worth listening to.

So the takeaway is a reflex, not a verdict. Next time a title lands in front of you — Doctor, Analyst, Founder, Professor — do not salute it and do not sneer at it. Just run the three questions and let the answers do the sorting.

What Comes Next in the Series

Part four turns to a subtler trick, one that even honest experts use without meaning to: the way a chart or a statistic can be technically true and completely misleading. If this installment taught you to check the person, the next one teaches you to check the number.

Bottom Line. Vetting expertise is not cynicism; it is a habit — the small discipline of trading a smooth title for a rough, checkable record before you believe anything.

Everyday Takeaway. The next time an authority appears on your screen, run three questions before you nod: Was the last call right? Who benefits if I believe this? What does the other side say?

Frequently Asked Questions (FAQ)

Q. How do I start vetting expertise when I am not an expert myself? A. You do not need domain knowledge to run the four checks — you need a habit. Ask whether the person has a public, scored track record, who funds their view, what rival experts say, and whether the claim sits inside their actual field. Those questions test the surface of a claim, not its technical content, so anyone can run them.

Q. Doesn’t a strong credential still mean something? A. Yes, within its lane. A board certification is powerful evidence about the field it certifies and close to zero evidence outside it. The mistake, which psychologists call the halo effect, is letting one real credential vouch for unrelated claims — which is why domain and consensus are one of the four marks.

Q. Isn’t checking every expert exhausting and a little paranoid? A. It is far cheaper than it sounds once the three questions become a reflex, and it is the opposite of paranoia. The goal is not to distrust everyone but to spend your trust well — on the experts who keep score, disclose their funding, and admit when they were wrong.