Academic paper reviews are a necessary civic duty for researchers in all fields, humanities, science, engineering, or anything in between. Coming from the inter-disciplinary field of natural language processing (NLP) and/or computational linguistics (CL), we have to deal with the humanities aspects of language use (social, political, psychological, communication) and mechanical descriptions of computation often hidden behind algorithmic pseudo-codes and mathematical formulas.

This article is purely an opinion piece based on my experience reviewing NLP papers for conference and workshop papers that usually end up on the canonical repository for NLP/CL publications on ACL Anthology.

Singing to the Chorus

Before I proceed with my opinion, I must point out the numerous resources for good reviewing guides and empirical studies of reviews in the NLP field.

• How to write good reviews? - from Conference on Empirical Methods in Natural Language Processing (EMNLP) organizers
  • They used to be the conference that gives “Best Reviewer Awards”
• How to review for ACL Rolling Review? - from Association of Computational Linguistics (ACL) Rolling Review organizers
  • There are a lot of related resources available in their repository.
  • Primarily based on the study: What Can We Do to Improve Peer Review in NLP?

Pre-review Googling

Often, we are assigned a paper to review on a particular reviewing platform, OpenReview, SoftConf, EasyChair, or Conferency. For now, there is no good way to track the “provenance” of the paper, i.e. how many times have the papers been submitted, revised, and re-submitted and to which conference. This is especially difficult in a double-blind review where both reviewers and paper authors remain anonymous.

Before reviewing the paper, I would usually google the title and abstract or conclusion to make sure that the content I’m going to review is indeed “fresh” or has made sufficient effort to improve from a related study. Often, this pre-review Googling also led me to gisting related papers that give me context on the topics that the paper writes about.

And here’s a comeback line to a phenomenon that academic reviewers know too well, authors submitting to conference A, got me as a reviewer, getting a rejection, then re-submitting the paper to conference B, then got me again as a reviewer in conference B.

Great minds think alike, not-so-great ones re-submit a rejected paper to another conference without making any revision/extension.

Summary, Strength, Weakness

When I first learned to write a review for an academic paper as a secondary reviewer, I was told to always start with a summary of the paper and what I understand the paper to be doing. I strongly encourage any mentors to nudge students/mentees to also take baby steps as a secondary reviewer.

The summary not only helps you to better crystalize the main idea and contribution of the paper in a short paragraph, but it also acts as a communication to the author to say, “I get what you’re doing there”.

A clear summary in the review, gives the author an assurance that you understood the content of the paper.

Improvements and Suggestions

Always catch any typos and awkward writing expressions and help out a fellow researcher to improve the paper. Even as a sort of native English speaker, I make mistakes in any sort of writing.

With traditional commercial publications, I have had experiences where the article writing style looks like a 180-degree change from my writing style after an editor revision. But in an academic paper, the editorial effort is a luxury. My professor used to tell me to ask a friend, relative, spouse, kid, or neighbor to read the draft and pay them $0.10 for each error they catch =)

As an author, I appreciated all the help I get in writing better papers and as a reviewer, I strongly encourage you to pay it forward to help catch those pesky errors.

Not everyone is a native English speaker, even native English speaker makes writing mistakes.

Be Nice!

As a paper author, I am sure you hate it when a reviewer:

• obviously is clueless about your idea, esp. that reviewer #2
• gives you that 1-liner review that translates to “It’s good” or “It’s bad”.
• has an inherent bias in the methods or content

Stepping back as a paper reviewer, I try to always:

• do my best to understand what the idea is, even if it’s something too hard to understand due to the writing style or topic; worst case scenario, inform the area chair (the person co-organizing the reviewing duties) that the paper content is not-understandable and put a low confidence score for your review.
• never give that one-liner review; I try to give it at least a ~30 mins effort to review a paper and that’s already a lot more than what I spend on “paper blitzing”
• try my best to “remain neutral” as much as possible, and when I really can’t, I do what’s in the next section.

Don't do unto others what you [as a reviewer] would not have done unto you [as an author].

Be Truthful!

It’s important to “be nice” as a reviewer but if something is technically wrong, you have to state the facts on what’s wrong with as little emotion as possible.

I totally understand that sometimes the lack of clarity, technical absurdity, or things that go against your technical belief drive you mad during the review process. Keep those words on the review first but don’t submit them, chill out, take a walk come back later/tomorrow to revise your review.

… sometimes [paper] makes reviewer mad and [you] just wants to lash out in the review … chill out, take a walk come back later/tomorrow to revise your review.

Reviewing as Shepherds

Accepting/Rejecting a paper seems to be the primary utilization of reviewing conference papers, but what I find more important is feedback on the idea, experiments, results, and writing style.

Sometimes as an author, we should approach paper submission less on scoring brownie points on Google Scholars / Semantic Scholar, but more on validating if the idea and experiments prove an interesting and/or important hypothesis.

When it comes time to decide if a paper is an accept/reject in the review, I’ll encourage reviewers to go beyond your reviewing duties a little and try this.

• Strong Accept: Put in the good words if the paper is really good, and encourage the authors in the reviews.
• Accept: Point out places and give suggestions on how to improve the paper
• Marginal: Provide suggestions that can move the Marginal to an Accept
• Reject: Explain why it is a reject and how the work needs to improve, what can be added to make the work sufficient enough to make up content for the paper
• Strong Reject: Keep calm, explain as neutrally as possible why it is a reject and what are the fundamental and technical issues; advise the author with the baby step to “restart" the main idea of the paper. Be nice!

Incentives of being a Reviewer

Conference paper reviewing is a necessary chore. I’m not going to say “chores can be fun” but a chore isn’t necessarily boring or unrewarding. Often, paper reviewing gives your first-hand information on the latest technology/techniques. Sometimes it challenges your perceived limitations of a certain method, task, or problem. Occasionally, it gives you some satisfaction of “at last someone tried that and got it to work” or “phew, now I can skip that idea that I know didn’t work”. Altruism aside, if you want to submit a paper to a conference/journal, somebody’s got to review it.

Concluding with a quote, as a reviewer and authors of academic conference, we have to remind ourselves,

“I want to be necessary and do good works. I ain't here to waste nobody's time, because I don't want you to waste mine.” - Mos Def