Professional fact-checkers and fact-checking organizations provide a critical public service. Skeptics of modern media, however, often question the accuracy and objectivity of fact-checkers. The current study assessed agreement among two independent fact-checkers, The Washington Post and PolitiFact, regarding the false and misleading statements of then President Donald J. Trump. Differences in statement selection and deceptiveness scaling were investigated. The Washington Post checked PolitiFact fact-checks 77.4% of the time (22.6% selection disagreement). Moderate agreement was observed for deceptiveness scaling. Nearly complete agreement was observed for bottom-line attributed veracity. Additional cross-checking with other sources (Snopes, FactCheck.org), original sources, and with fact-checking for the first 100 days of President Joe Biden’s administration were inconsistent with potential ideology effects. Our evidence suggests fact-checking is a difficult enterprise, there is considerable variability between fact-checkers in the raw number of statements that are checked, and finally, selection and scaling account for apparent discrepancies among fact-checkers.

This paper focused on how much two prominent fact-checking sources, The Washington Post’s Fact Checker and PolitiFact, agreed on the statements to fact-check and the strength of their deceptiveness ratings. The study examined presidential fact-checks, and the data suggest some divergence in what to check. Deceptiveness ratings were moderately correlated for The Washington Post and PolitiFact, suggesting general agreement on the presence of falsity but indicating variability in severity assessments of false or misleading statements.