You hit “Submit” and then… silence. Months of work, hundreds of edits, one click—and the long wait for a verdict from those enigmatic figures: Reviewer 1, Reviewer 2… It’s one of the most stressful moments in a researcher’s life. But what actually happens on the other side? Peer review is the backbone of science—a quality-control system meant to ensure that published work is important, original, and rigorous (Sense about Science).
The Reviewer’s Mind Map: What gets checked, and how?
Across fields, reviewers look for answers to the same fundamental questions—but they use different tools to get there. The table below shows how the same principles of rigor are interpreted in the medical/social sciences versus in computer science and machine learning.
| Assessment Aspect / The Reviewer’s Question | Implementation (Medical / Social Sciences) | Implementation (Computer Science / Machine Learning) |
|---|---|---|
| 0. Conflicts of Interest and Expertise | Do I have ties to the authors/institution? Do I understand this clinical intervention and the statistical methodology? | Do I have ties to the authors/lab? Is this my subfield (algorithms, model architecture, theory)? |
| 1. Originality and Significance | Does the study address an important clinical/social problem? Does it add something genuinely new relative to existing studies and reviews? | Is the problem significant (theoretically or practically)? Is the method truly novel, or merely incremental? Empirically: does it improve the state of the art in a meaningful way? |
| 2. Methodological Rigor (Reproducibility) | Protocol and preregistration: is the study registered (e.g., OSF)? Is the protocol available and followed? | Artifacts and code: are code, data, and the environment available (GitHub/Zenodo)? Is there a README to run the experiments? Do we meet ACM Artifact Badging criteria (Available/Evaluated/Results Validated)? (Badges assess artifacts and reproducibility, not the scientific quality of the paper itself.) ACM policy · SIGIR note |
| 3. Methodological Rigor (Correctness) | Reporting standards: is the appropriate EQUATOR Network checklist used—CONSORT 2025 for RCTs, PRISMA 2020 for systematic reviews, STROBE for observational studies? | Technical craft: correctness of proofs and definitions; fair apples-to-apples comparisons; appropriate metrics; ablation to assess key components. |
| 4. Results and Analysis | Statistics: correct choice of tests; uncertainty reporting (CIs); multiple-testing corrections. | Experimental credibility: standard, public datasets; repeatability (different random seeds); valid significance tests for model comparisons. |
| 5. Conclusions and Interpretation | No overclaiming: conclusions match the data; no population-wide claims from small samples; a candid discussion of limitations. | Proportionality: no “we solve X” if the gain is +0.5%; discuss computational cost, scalability, and practical limits. |
| 6. Research Ethics | Participant protection (ICMJE/COPE): IRB approval, informed consent, data anonymization. ICMJE · COPE | Data and model ethics: licenses, privacy, bias, potential harms/dual-use; transparency about data/model sources. |
Simplification: in interdisciplinary work, reviewers combine both perspectives.
A journey through your manuscript: three reviewer passes
Pass 1: Quick triage (15–30 min) – is this worth reading further?
- Fit: is the work original and within the venue’s scope?
- Logic: do conclusions broadly follow from the data?
- Red flags:
- CS/ML: a “new” algorithm is a known method under a new name; non-standard, trivial dataset.
- Medicine: RCT without IRB approval or without a control group.
- Universal: conclusions contradict the authors’ own results.
Pass 2: Microscope (2–5 h) – can I trust these results?
- Med/Soc: point-by-point compliance with CONSORT 2025/PRISMA/STROBE.
- CS/ML: review repository and artifacts; fair baselines; correctness of claims.
Pass 3: Narrative (1–2 h) – does the story make sense?
- Do the promises in the introduction materialize later?
- Is there an honest Limitations section?
- Are figures/tables readable and properly described?
Thinking like a reviewer: pre-submission self-review checklist
Before you click “Submit,” do a tough, point-by-point self-review.
Big picture and contribution
- Problem: do I clearly define what I solve and why it matters?
- Originality: where in the text do I demonstrate novelty vs. prior work?
- Impact: will anyone want to cite this in 2–3 years?
Section by section
Title: does it accurately reflect the content (neither too broad nor too narrow)?
Abstract:
- WHAT? question/goal; WHY? significance; HOW? method; RESULTS: key numbers; NOVELTY: one-sentence summary of what’s new.
Introduction: context → gap → goal → novelety → solution; end with clear contributions.
Methods/Approach:
- Reproducibility: could a peer in the field replicate the experiment?
- Justification: why these methodological choices?
- Details: (CS/ML) hyperparameters, architecture, datasets, metrics; (Med/Soc) inclusion/exclusion criteria, randomization, sample size.
- Transparency: are code and data available? If not, provide a solid rationale.
Results & Discussion: clear figures; strong baselines; claims supported by data; a separate paragraph on limitations.
Conclusions: answer the research question + offer broader perspective and future directions.
Box: standards and transparency in peer review
Point readers to ANSI/NISO Z39.106-2023—a standard terminology for peer review models (single/double anonymized, openness of reports, etc.). NISO
AI in service of peer review: assistant or usurper?
Large language models (LLMs) have entered our toolbox. Surveys (e.g., Zhuang et al., 2025) outline both potential and risk. Below is a policy-safe minimal practice set.
What genuinely helps?
- Fast triage and summaries: initial mapping of content, scoping against the venue.
- Formal checklists: matching study type to the right standard (e.g., RCT → CONSORT 2025).
- Sanity-checking consistency: basic mismatches between text and tables/figures.
Where are the boundaries?
- No deep expertise: models can miss the true contribution or deep flaws.
- Hallucinations and phantom citations: risk of fabricated sources/claims.
- Bias and score inflation: studies report a tendency to over-recommend.
The ugly truth: ethics, confidentiality, and publisher policies
Manuscript confidentiality is sacred. Uploading an unpublished paper to a public LLM breaches review confidentiality.
- Elsevier: reviewers should not upload manuscripts to genAI tools or use genAI for substantive reviewing. Policy
- Springer Nature: asks reviewers to not upload manuscripts to genAI; confidentiality first. Policy
- IEEE: prohibits using AI to draft or generate reviews. Reviewer Guidelines
- ACM: only permits language assistance on your own review text, and only without breaching confidentiality (prefer privacy-preserving enterprise solutions). ACM Peer Review FAQ
- ACL/ARR: do not share confidential content with third-party tools (practically: do not use popular LLMs for review materials). ARR Reviewer Guidelines
Exception: publisher-provided tools. Some publishers develop closed internal tools to support integrity and completeness checks (e.g., Frontiers—AIRA). These handle pre-screening and technical checks (plagiarism, figures, language); the final judgment remains human. Always verify the current policy for your journal/conference.
Minimum safety rule: never upload someone else’s manuscript to a public LLM. If you need language help for your own review text, use privacy-preserving solutions or the publisher’s in-house tools.
The art of responding to reviews
- Professional tone: thank the reviewers and the editor.
- Point-by-point list: address every comment separately.
- Show the changes: cite exact locations in the manuscript (page/line/section).
- When you disagree: be polite, provide evidence-based arguments.
References and further reading
Standards and guidelines
- CONSORT 2025 (RCTs)—update with parallel publications in BMJ/JAMA/Lancet/Nat Med/PLOS Med.
- PRISMA 2020 (systematic reviews) · STROBE (observational studies)
- COPE Core Practices · ICMJE Recommendations
- ANSI/NISO Z39.106-2023—peer review terminology standard
Reproducibility and artifacts
AI policies (selected)
- Elsevier—AI and peer review · Springer Nature—confidentiality & AI · IEEE—prohibition on AI-written reviews
- ACM—Peer Review FAQ · ACL/ARR—third-party tool restrictions
- Frontiers AIRA—closed, integrity-supporting pre-screening
Introductions and guides
- Peer Review: The Nuts and Bolts (Sense about Science)
- Anatomy of the research process: a practical framework for rigorous science (your previous post)
LLMs in peer review—surveys and studies
- Zhuang, Z. et al. (2025). Large language models for automated scholarly paper review: A survey, Information Fusion.