How to read a paper and judge whether to trust it
The goal is not to finish the paper but to decide whether its conclusion applies to your problem. Separate the claim from the evidence first, then decide how deep to go.
How to do it
Ask what it actually proved
Paste only the abstract, the last paragraph of the introduction, and the conclusion into “Reading a Paper” and have it answer three things: what the authors claim to have shown, what that claim covers, and which single sentence you would keep if you kept one. Hold back the method section — it pulls you into details before you know whether they matter.
Map the argument
With “Mapping an Argument,” have it turn the premises, inference, and conclusion into one chain, and fill in the step the authors left out. That missing step is where most papers leak: from “works on this dataset” to “the method works generally,” with no bridge in between.
Check the data behind each claim
Use “Does the Claim Have Data Behind It?” on every “studies show” and “significantly better,” asking one by one: how large was the sample, what was the control, how was the metric defined? The “significant” in an abstract and the “significant” in a results table are often different things; one is rhetoric, the other is statistics.
Check whether the metric definition moved
Use “The Trap in the Measurement” on three points: who the denominator is, whether the baseline was tuned by the authors, and whether only the best run was reported. The same data tells a different story under a different definition. It is not fabrication, but it makes the result look stronger than it is.
Decide between correlation and causation
With “Correlation or Causation”: if the paper says A causes B, check whether it used random assignment, a natural experiment, or only observational data. With observational data, “we controlled for variables” is not the same as ruling out confounders. You can still cite the conclusion, but cite it with the conditions attached.
Prompts used
FAQ
Can I judge a paper without understanding the math?
Most of the time, yes. Trustworthiness depends more on design than on formulas: was there a control, where did the sample come from, who defined the metric, does the conclusion exceed the data. The math decides whether the derivation is right; these questions decide whether the study is worth believing.
Can I cite a preprint?
You can read it; just label it as a preprint that has not been peer reviewed. When judging, check whether data and code are public, whether the authors have answered objections, and whether anyone has reproduced the result. Without those, treat it as a lead rather than a basis.
The paper says it works. Can I put it in production?
Check three things first: how far the experimental setting is from yours, whether the effect size is the size you need, and what happens when it fails. Papers report average effects and rarely the worst case. A small live test teaches you more than ten readings of the method section.