Responding to Reviewers

“Revise and resubmit” is really the best outcome of academic peer review – acceptance for publication as submitted is so rare it may as well not exist, and most papers are genuinely improved through the peer review and revision processes. Generally speaking, an additional document detailing changes must accompany the revised submission, but the conventions for writing these “change logs” are a little opaque because they’re not typically part of the public discussion of the research.

San Antonio Botanical Gardens during CSCW 2013

There are a couple of great examples of change logs from accepted CSCW 2013 papers from Merrie Morris, and I’m offering my own example below as well. It’s no secret that my CSCW 2013 paper was tremendously improved by the revision process. I wrote the initial submission in the two weeks between submitting my final dissertation revisions and graduation. For a multitude of reasons, it wasn’t the ideal timing for such an endeavor, so I’m glad the reviewers saw a diamond in the rough.

My process for making revisions starts with not getting upset about criticism to which I willingly subjected myself – happily, a practice that becomes easier with time and exposure. (If needed, you can substitute “get upset/rant/cry in private, have a glass of wine, cool off, sleep on it, and then come back to it later,” which is a totally valid way to get started on paper revisions too.) Hokey as it sounds, I find it helpful to remind myself to be grateful for the feedback. And that I asked for it.

Then I print out the reviews, underline or highlight the items that need attention, and summarize them in a few words in the margin. Next, I annotate a copy of the paper to identify any passages that are specifically mentioned, and start to figure out where I need to make changes or could implement reviewers’ suggestions. I find these tasks much easier to do on paper, since being able to spread out all the pages around me sometimes helps when working on restructuring and identifying problem points.

During or after that step, I create a new word processing document with a table and fill it in with terse interpretations of the comments, as you’ll see in the example below. In the process, I sort and group the various points of critique so that I’m only responding to each point once. This also ensures that I’m responding at the right level, e.g., “structural problems” rather than a more specific indicator of structural problems.

The actual columns of the table can vary a little, depending on the context – for example, a table accompanying a 30-page journal manuscript revision in which passages are referenced by line number would naturally include a column with the affected line numbers to make it easier for the reviewer to find and evaluate the updated text. In the example below, I made such substantial changes to the paper’s structure that there was no sense in getting specific about section number, paragraph, and sentence.

As a reviewer, I’m all for process efficiency; I strongly prefer concise documentation of revisions. At that stage, my job is to evaluate whether my concerns have been addressed, and the documentation of changes should make that easier for me, rather than making me wade through unnecessary detail. Likewise, as an author, I consider it a problem with my writing if I need to include a lengthy explanation of why I’ve revised the text, as opposed to the text explaining itself. That heuristic holds under most circumstances, unless the change defies expectations in some fashion, or runs counter to a reviewer’s comment — which is fine when warranted, and the response to reviewers is the right place to make that argument.

Therefore, the response to reviewers is primarily about guiding the reviewer to the changes you’ve made in response to their feedback, as well as highlighting any other substantive changes and any points of polite disagreement. In a response to reviewers, the persuasive style of CHI rebuttals, the closest parallel practice with which many CSCW authors have experience, seems inappropriate to me because the authors are no longer in a position of persuading me that they can make appropriate revisions, but are instead demonstrating that they have done so. Ergo, I expect (their/my) revisions to stand up to scrutiny without additional argumentation.

Finally, once all my changes are made and my table is filled in, I provide a summary of the changes, which includes any other substantive changes that were not specifically requested by the reviewers, and note my appreciation for the AC/AE and reviewers’ efforts. A jaded soul might see that as an attempt at flattering the judges, but it’s not. I think that when the sentiment is genuine, expressing gratitude is good practice. In my note below, I really meant it when I said I was impressed by the reviewers’ depth of knowledge. No one but true experts could have given such incisive feedback and their insights really did make the paper much better.

——————————

Dear AC & Reviewers,

Thank you for your detailed reviews on this submission. The thoroughness and depth of understanding that is evident in these reviews is truly impressive.

To briefly summarize the revisions:

  • The paper was almost completely rewritten and the title changed accordingly.
  • The focus and research question for the paper are now clearly articulated in the motivations section.
  • The research question makes the thematic points raised by reviewers the central focus.
  • The analytical framework is discussed in more depth in the methods section, replacing less useful analysis process details, and is followed up at the close of the discussion section.
  • The case comparison goes into greater depth, starting with discussion of case selection.
  • The case descriptions and comparison have been completely restructured.
  • The discussion now includes an implications section that clarifies the findings and applicability to practice.

Below are detailed the responses to the primary points raised in the reviews; I hope these changes meet with your approval. Regardless of the final decision, the work has unquestionably benefited from your attention and suggestions, for which I am deeply appreciative.

Reviewer Issue Revisions
AC No clear research question/s A research question is stated toward the end of page 2.
AC, R1, R3 Findings are “obvious” The focus of the work is reframed as addressing obvious assumptions that only apply to a limited subset of citizen science projects, and the findings – while potentially still somewhat obvious – provide a more useful perspective.
AC, R2 Conclusions not strong/useful A section addressing implications was added to the discussion.
AC Improve comparisons between cases Substantial additional comparison was developed around a more focused set of topics suggested by the reviewers.
AC Structural problems The entire paper was restructured.
R1 Weak title The title was revised to more accurately describe the work.
R1 Does not make case for CSCW interest Several potential points of interest for CSCW are articulated at the end of page 1.
R1 Needs stronger analytic frame & extended analysis The analytic framework is described in further detail in the methods section, and followed up in the discussion. In addition, a section on case selection criteria sets up the relevance of these cases for the research question within this framework.
R1 Quotes do not add value Most of this content was removed; new quotes are included to support new content.
R1, R3 Answer the “so what?” question & clarify contributions to CSCW The value of the work and implications are more clearly articulated. While these implications could eminently be seen as common sense, in practice there is little evidence that they are given adequate consideration.
R1 Include case study names in abstract Rewritten abstract includes project names.
R1 Describe personally rewarding outputs in eBird These are described very briefly in passing, but with the revised focus are less important to the analysis.
R2 Compare organizational & institutional differences Including these highly relevant contrasts was a major point of revision. A new case selection criteria section helps demonstrate the importance of these factors, with a table clarifying these contrasts. The effects of organizational and institutional influences are discussed throughout the paper.
R2 Highlight how lessons learned can apply to practice The implications section translates findings into recommendations for strategically addressing key issues. Although these are not a bulleted list of prescriptive strategies, the reminder they provide is currently overlooked in practice.
R2 Comparison to FLOSS is weak This discussion was eliminated.
R2 Typos & grammatical errors These errors were corrected; hopefully new ones were not introduced in the revision process (apologies if so!)
R3 Motivation section does not cite related work Although the rewritten motivation section includes relatively few citations, they are more clearly relevant. For some topics, there is relatively little research (in this domain) to cite.
R3 Motivation section does not discuss debated issues The paper now focuses primarily on issues of participation and data quality.
R3 Consistency in case description structure The case descriptions are split into multiple topics, within which each case discussed. The structure of case descriptions and order of presentation is consistent throughout.
R3 Include key conclusions about each case with descriptions The final sentence of the initial descriptions for each case summarizes important characteristics. I believe the restructuring and refocusing of these revisions should address this concern.
R3 Does not tie back to theoretical framework used for analysis The Implications section specifically relates the findings back to the analytical framework, now discussed in greater detail in the methods section.
R3 No discussion of data quality issues This is now one of the primary topics of the paper and is discussed extensively. In addition, I humbly disagree that expert review is unusual in citizen science (although the way it was conducted in Mountain Watch is undoubtedly unique). Expert data review has been shown to be one of the most common data validation techniques in citizen science.
R3 No discussion of recruitment issues This topic is now one of the primary topics of the paper and is discussed extensively.
R3 Introduce sites before methods The case selection criteria section precedes the methods and includes overview descriptions of the cases. They are also given a very brief mention in the motivation section. More detailed description as relevant to the research focus follows the methods section.
R3 Do not assume familiarity with example projects References to projects other than the cases are greatly reduced and include a brief description of the project’s focus.
R3 Tie discussion to data and highlight new findings While relatively few quotes are included in the rewritten discussion section, the analysis hopefully demonstrates the depth of the empirical foundation for the analysis. The findings are clarified in the Implications section.
R3 Conclusions inconsistent with other research, not tied to case studies, or both To the best of my knowledge, the refocused analysis and resultant findings are no longer inconsistent with any prior work.

 

2 thoughts on “Responding to Reviewers

  1. Dear Dr. Wiggins,

    Thank you so much for this wonderful blog. Your articles are really interesting and helpful. You posted an article about working weekends when you were a Ph.D. student. I would really appreciate if you can share with us how you used to spend a typical weekday. Your daily routine as a Ph.D. candidate: when did you usually wake up; how many hours you spent daily working on your dissertation research, what advices would you give to people to stay focused and not to lose confidence in the process..I understood from your working weekends article that you spent more than 40 hours a week when you were a student and that even in the weekends, you worked between 4-6 hours. I admire you and I salute you for being this productive. I am inspired by you!

    Thank you so much for being a good model!

    • Hi Khaled – I’m glad you find the blog useful. To be honest, the typical working weekday as a PhD student has faded in my memory, and a lot of it depended on the stage I was at in the program – although I’ve always gotten up at 6 AM. I always tried to put in an 8 (or so) hour work day, but whether or not that time felt “productive” on any given day is another thing entirely!

      However, I’d caution against overworking – at times I worked more than was good for my health. As a postdoc, I’ve found a better balance for myself. I don’t often work evenings or weekends unless I’m against a deadline. Even then, I pretty much always take a day off from academic work every weekend because managing everyday life requires a little time and attention too. I found that taking at least one day off makes my work better on the whole.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>