Responding to Reviewers

“Revise and resubmit” is really the best outcome of academic peer review – acceptance for publication as submitted is so rare it may as well not exist, and most papers are genuinely improved through the peer review and revision processes. Generally speaking, an additional document detailing changes must accompany the revised submission, but the conventions for writing these “change logs” are a little opaque because they’re not typically part of the public discussion of the research.

San Antonio Botanical Gardens during CSCW 2013

There are a couple of great examples of change logs from accepted CSCW 2013 papers from Merrie Morris, and I’m offering my own example below as well. It’s no secret that my CSCW 2013 paper was tremendously improved by the revision process. I wrote the initial submission in the two weeks between submitting my final dissertation revisions and graduation. For a multitude of reasons, it wasn’t the ideal timing for such an endeavor, so I’m glad the reviewers saw a diamond in the rough.

My process for making revisions starts with not getting upset about criticism to which I willingly subjected myself – happily, a practice that becomes easier with time and exposure. (If needed, you can substitute “get upset/rant/cry in private, have a glass of wine, cool off, sleep on it, and then come back to it later,” which is a totally valid way to get started on paper revisions too.) Hokey as it sounds, I find it helpful to remind myself to be grateful for the feedback. And that I asked for it.

Then I print out the reviews, underline or highlight the items that need attention, and summarize them in a few words in the margin. Next, I annotate a copy of the paper to identify any passages that are specifically mentioned, and start to figure out where I need to make changes or could implement reviewers’ suggestions. I find these tasks much easier to do on paper, since being able to spread out all the pages around me sometimes helps when working on restructuring and identifying problem points.

During or after that step, I create a new word processing document with a table and fill it in with terse interpretations of the comments, as you’ll see in the example below. In the process, I sort and group the various points of critique so that I’m only responding to each point once. This also ensures that I’m responding at the right level, e.g., “structural problems” rather than a more specific indicator of structural problems.

The actual columns of the table can vary a little, depending on the context – for example, a table accompanying a 30-page journal manuscript revision in which passages are referenced by line number would naturally include a column with the affected line numbers to make it easier for the reviewer to find and evaluate the updated text. In the example below, I made such substantial changes to the paper’s structure that there was no sense in getting specific about section number, paragraph, and sentence.

As a reviewer, I’m all for process efficiency; I strongly prefer concise documentation of revisions. At that stage, my job is to evaluate whether my concerns have been addressed, and the documentation of changes should make that easier for me, rather than making me wade through unnecessary detail. Likewise, as an author, I consider it a problem with my writing if I need to include a lengthy explanation of why I’ve revised the text, as opposed to the text explaining itself. That heuristic holds under most circumstances, unless the change defies expectations in some fashion, or runs counter to a reviewer’s comment — which is fine when warranted, and the response to reviewers is the right place to make that argument.

Therefore, the response to reviewers is primarily about guiding the reviewer to the changes you’ve made in response to their feedback, as well as highlighting any other substantive changes and any points of polite disagreement. In a response to reviewers, the persuasive style of CHI rebuttals, the closest parallel practice with which many CSCW authors have experience, seems inappropriate to me because the authors are no longer in a position of persuading me that they can make appropriate revisions, but are instead demonstrating that they have done so. Ergo, I expect (their/my) revisions to stand up to scrutiny without additional argumentation.

Finally, once all my changes are made and my table is filled in, I provide a summary of the changes, which includes any other substantive changes that were not specifically requested by the reviewers, and note my appreciation for the AC/AE and reviewers’ efforts. A jaded soul might see that as an attempt at flattering the judges, but it’s not. I think that when the sentiment is genuine, expressing gratitude is good practice. In my note below, I really meant it when I said I was impressed by the reviewers’ depth of knowledge. No one but true experts could have given such incisive feedback and their insights really did make the paper much better.

——————————

Dear AC & Reviewers,

Thank you for your detailed reviews on this submission. The thoroughness and depth of understanding that is evident in these reviews is truly impressive.

To briefly summarize the revisions:

  • The paper was almost completely rewritten and the title changed accordingly.
  • The focus and research question for the paper are now clearly articulated in the motivations section.
  • The research question makes the thematic points raised by reviewers the central focus.
  • The analytical framework is discussed in more depth in the methods section, replacing less useful analysis process details, and is followed up at the close of the discussion section.
  • The case comparison goes into greater depth, starting with discussion of case selection.
  • The case descriptions and comparison have been completely restructured.
  • The discussion now includes an implications section that clarifies the findings and applicability to practice.

Below are detailed the responses to the primary points raised in the reviews; I hope these changes meet with your approval. Regardless of the final decision, the work has unquestionably benefited from your attention and suggestions, for which I am deeply appreciative.

Reviewer Issue Revisions
AC No clear research question/s A research question is stated toward the end of page 2.
AC, R1, R3 Findings are “obvious” The focus of the work is reframed as addressing obvious assumptions that only apply to a limited subset of citizen science projects, and the findings – while potentially still somewhat obvious – provide a more useful perspective.
AC, R2 Conclusions not strong/useful A section addressing implications was added to the discussion.
AC Improve comparisons between cases Substantial additional comparison was developed around a more focused set of topics suggested by the reviewers.
AC Structural problems The entire paper was restructured.
R1 Weak title The title was revised to more accurately describe the work.
R1 Does not make case for CSCW interest Several potential points of interest for CSCW are articulated at the end of page 1.
R1 Needs stronger analytic frame & extended analysis The analytic framework is described in further detail in the methods section, and followed up in the discussion. In addition, a section on case selection criteria sets up the relevance of these cases for the research question within this framework.
R1 Quotes do not add value Most of this content was removed; new quotes are included to support new content.
R1, R3 Answer the “so what?” question & clarify contributions to CSCW The value of the work and implications are more clearly articulated. While these implications could eminently be seen as common sense, in practice there is little evidence that they are given adequate consideration.
R1 Include case study names in abstract Rewritten abstract includes project names.
R1 Describe personally rewarding outputs in eBird These are described very briefly in passing, but with the revised focus are less important to the analysis.
R2 Compare organizational & institutional differences Including these highly relevant contrasts was a major point of revision. A new case selection criteria section helps demonstrate the importance of these factors, with a table clarifying these contrasts. The effects of organizational and institutional influences are discussed throughout the paper.
R2 Highlight how lessons learned can apply to practice The implications section translates findings into recommendations for strategically addressing key issues. Although these are not a bulleted list of prescriptive strategies, the reminder they provide is currently overlooked in practice.
R2 Comparison to FLOSS is weak This discussion was eliminated.
R2 Typos & grammatical errors These errors were corrected; hopefully new ones were not introduced in the revision process (apologies if so!)
R3 Motivation section does not cite related work Although the rewritten motivation section includes relatively few citations, they are more clearly relevant. For some topics, there is relatively little research (in this domain) to cite.
R3 Motivation section does not discuss debated issues The paper now focuses primarily on issues of participation and data quality.
R3 Consistency in case description structure The case descriptions are split into multiple topics, within which each case discussed. The structure of case descriptions and order of presentation is consistent throughout.
R3 Include key conclusions about each case with descriptions The final sentence of the initial descriptions for each case summarizes important characteristics. I believe the restructuring and refocusing of these revisions should address this concern.
R3 Does not tie back to theoretical framework used for analysis The Implications section specifically relates the findings back to the analytical framework, now discussed in greater detail in the methods section.
R3 No discussion of data quality issues This is now one of the primary topics of the paper and is discussed extensively. In addition, I humbly disagree that expert review is unusual in citizen science (although the way it was conducted in Mountain Watch is undoubtedly unique). Expert data review has been shown to be one of the most common data validation techniques in citizen science.
R3 No discussion of recruitment issues This topic is now one of the primary topics of the paper and is discussed extensively.
R3 Introduce sites before methods The case selection criteria section precedes the methods and includes overview descriptions of the cases. They are also given a very brief mention in the motivation section. More detailed description as relevant to the research focus follows the methods section.
R3 Do not assume familiarity with example projects References to projects other than the cases are greatly reduced and include a brief description of the project’s focus.
R3 Tie discussion to data and highlight new findings While relatively few quotes are included in the rewritten discussion section, the analysis hopefully demonstrates the depth of the empirical foundation for the analysis. The findings are clarified in the Implications section.
R3 Conclusions inconsistent with other research, not tied to case studies, or both To the best of my knowledge, the refocused analysis and resultant findings are no longer inconsistent with any prior work.

 

Getting It Done: Tools for Organizing and Writing

Some people believe that I never sleep, but that’s really not true. I do sleep, at least sometimes, and I’m also fairly productive.

Achieving a relatively high level of productivity depends in part upon having good tools to support your work, and tools that work well for your working style. So this is the first of two posts on the subject of software that supports academic work. “What software should I use for X?” is a perennial question posed by PhD students everywhere, and software is now pretty essential to academic productivity. This post focuses on tools for organizing, writing, and presenting (I covered poster design previously); the follow-up post will describe my favorite research tools.

The big disclaimer: I use a Mac. If you don’t use a Mac, your mileage may vary, but some of these programs do have versions for Windows and other operating systems. I generally avoid Microsoft software in favor of Apple software (much cheaper and generally good design) and open source software (generally awesome, and free!)

Organizing

Everyone has to stay organized somehow. Some of us make a lot of lists. I definitely make an excessive number of lists. To the point where I’ve made lists of lists. It eventually becomes unsupportable; at some points in time, I spent more effort on keeping lists updated than doing the stuff on the lists. But there’s a reason to make lists – it gets all that stuff out of your head, leaving your brain free to think about more important stuff!

My main tool for keeping all my to-do items in order is OmniFocus, which is wonderful Mac-specific software from OmniGroup. I’m a big fan of OmniGroup software; they make very well designed and thoughtful tools. There are versions of OmniFocus for desktop, iPhone, and iPad – and I use them all. This is one of those tools that can be really useful for supporting GTD, if that’s your thing. If you have your own way of doing things, you can still adapt your use of OmniFocus to do things your own way. So now I get to have my to-do list readily synced across my digital devices at all times. And the OmniGroup ninjas (that’s actually what they call their tech support) are responsive and have a sense of humor. How much better can it get?

Another aspect of keeping your stuff together is keeping files synced, if you use multiple machines. Keeping files synced becomes a problem the second you start using more than one machine. At this point, I work on my (dying) 15″ MacBook Pro, a beautiful zippy still-new 27″ iMac, plus my iPad. And sometimes my iPhone 4, when I have no other options. I use MobileMe to sync the Apple-specific stuff, like Contacts and Mail, but I use Dropbox (platform agnostic) for everything else.

Recent security hullaballoos aside, it’s a very usable solution, and that’s why so many people have adopted it. Although I already pay for MobileMe, it doesn’t behave the way I want, with the exception of the Apple-specific syncing, so now I pay for Dropbox storage as well. Without any additional effort or change to thoroughly-ingrained file management behaviors on my part, I can manage all my files locally in the same fashion that I always have, with the only change being that I use my Dropbox folder as my primary storage space instead of my Documents folder. And everything is then magically synced across machines.

Dropbox also gives me double-plus file backup: my files are now backed up three ways from Sunday, because they’re synced on every machine I use, they’re backed up on my Time Capsule (a simply brilliant piece of personal computing infrastructure), and they’re backed up to Dropbox in the cloud. That adds up to serious peace of mind when it comes to irreplaceable research data. Even better, there’s version control, so if I really screw something up and don’t have a Time Machine backup for whatever reason, there’s a Dropbox backup. On top of everything else, there’s nice file sharing with Dropbox, so that’s also been very handy for research collaboration, particularly when concurrent editing is not a concern.

Writing

Once you’ve got your stuff in order, you have to write about it; this is how we produce new knowledge (the part where I talk about producing the stuff to write about will be in the next post…) Everyone has their preferences for organizing ideas to write, and for word processing. Some of us even eschew word processing altogether, and go for the gold with typesetting.

OmniOutliner is my favorite tool for organizing ideas. It’s also from the OmniGroup (obviously, I think?) and is a really simple but highly functional program for making, what else, outlines! I find it more useful than most other tools when it’s time to start organizing ideas for writing. The interface is simple enough as to be non-distracting, and I like the ease of the drag-and-drop interface. It doesn’t export to other formats as easily as I want, but cut-and-paste will always save the day.

When doing collaborative writing where concurrent editing may occur (e.g., last minute papers with crazy late jam sessions) then Google Docs is a winner. It’s browser-based, so it doesn’t matter what kind of operating system you’re using. The interface has really improved, since there’s now an embedded chat functionality, commenting, and you can see the other people’s cursor positions. Google seems to have taken the best features from EtherPad and integrated them with the existing Google Docs functionality for a hands-down winner. Sadly, one of the only things it doesn’t do to my satisfaction is support LaTeX, but that’s only an inconvenience and easy enough to work around.

Some of you don’t know what LaTeX is. That’s OK, you probably don’t need to know. But I’m going to tell you anyway. It’s a free/open source software document preparation system with structural markup, much like HTML, but for making beautifully typeset documents, and it too is platform agnostic. Note that document preparation is not the same as “word processing.” LaTeX is what I prefer to use to write my papers, largely because Word has a tendency to crash on me, is inordinately slow, and is badly behaved in innumerable other ways. And I hate that stupid ribbon. I only use Word when my collaborators are unable to use anything else. I won’t lie – there’s a definite learning curve with LaTeX. It takes a little work, but I’ve found it completely worthwhile.

LaTeX is also nice because having structural markup means you can use style sheets, so you can change the appearance of multiple documents, and link documents, with relative ease. You can use any text editor to write a .tex file, so you can have a completely minimalist interface or something with lots of distracting buttons all over, whatever you prefer. Another benefit is the easy availability of many packages to do just the thing you want, and it is the only system that I have yet encountered that does any justice to mathematical equations. Math rendered in LaTeX looks like math ought to look. One of the few things I think it does really poorly, however, is tables. You can make great looking, tightly controlled tables in LaTeX, but it requires some patience. Even if you don’t want to get all control-freaky over your tables, you’re probably going to have to do that anyway.

Working with LaTeX becomes a little easier with the use of macros and a nice editing environment. You can edit your .tex files in emacs or vim (as I’ve done in the past) but I really like TeXShop for the easy, non-intrusive GUI. It comes with the MacTeX distro, so if you just download that nice big package, you’ll have all the pieces in one place. Another essential tool, for when an editor tells you to submit a final copy in Word after you’ve prepared the original submission in LaTeX, is latex2rtf. This tool lets you use your command-line (e.g., Terminal) interface to produce a Word-readable .rtf file out of your nice pretty LaTeX file. It won’t look as good as it once did, but all the stuff will be there in the right places, more or less. It’s the fastest way that I’ve yet found to convert a LaTeX file into Word, even if it does require a little post-hoc cleanup.

Reference Management

There are really only a few robust reference management software options out there. I’m sure Zotero has improved substantially since I last used it, but it was just plain inadequate when I last tried it, and I don’t have the time or energy to wait around for software to evolve. I am not about to spend a lot of time with Mendeley either, because it actually does way too much for what I want out of reference management software, and I prefer my tools as simple and reliable as possible.

I started off my academic career using EndNote, before I became a LaTeX convert. EndNote is nice enough, and has a bunch of good features, but I haven’t spent the additional $100 per update since EndNote X, largely because BibDesk is free (open source), works with LaTeX, and pretty much all of the reference managers are able to translate among one anothers’ formats. With greater or lesser ease, of course. A nice detail when using Google Scholar in a logged-in state is that you can set your preferences to provide a link for references in .bib format, suited for pasting into the bibtex files that go along with LaTeX documents.

Presenting

Everyone has to make a slide deck at some point, even if it makes Edward Tufte kill a kitten. I’m not a big fan of slides, but maybe I’m just being old-fashioned because I grew up on chalkboard dust.

Regardless, there are only a couple of options for presentation software. Most people use Powerpoint. I don’t really like it. It’s not as horrible as some other Microsoft Office products (I’m looking at you, Entourage) but it’s not great. I suspect the Google Docs version of ppt is much nicer, if only because it’s probably a bit more stripped-down – but I don’t use that either. And no, Open Office is just not adequate. I’m sorry, I really want to go with the open source option, but its interface (last time I looked at it) was stuck in the mid-1990’s and hurt my eyes.

Instead, I use Keynote, another Apple product. It’s slick, adequately functional, and pretty smart about a lot of little details. I get tired of the canned themes, but it’s easy enough to make your own. One of my favorite details about Keynote is that there’s an iPad (and iPhone!) version, so I can actually edit slides on my iPad, and present from it. That’s just lovely. Of course, the iOS version of Keynote is limited (this should be obvious, iOS is not OS X, just like an iPad is not a MacBook) but it’s quite functional for editing and presenting. I’ve never built an iOS Keynote presentation from scratch, but it can be done, and that’s nice flexibility to have.

Finally, when making your presentation, the last thing you want to have happen is your screen saver kicking in, or the power saving settings overtaking your display while you debate some point. I’ve seen this happen way too often, and it just doesn’t look very professional. Rather than edit your system preferences every time you get ready to make a presentation, I now rely on Caffeine. It’s a free Mac app (via the App Store, or just download and install as software the normal way) that, when activated, prevents your power saving settings from invoking and doesn’t permit your screen saver to take over your display. You turn it on and off from the menu bar by clicking on the little coffee cup icon. Simply brilliant!