Shortly you’ll get feedback on your first case study from your professor, a TA, and 3-4 peer reviewers. A critical part of life as a scientist, whether in academia, industry, the business world, government, an NGO, or elsewhere, is how to hear, process, and incorporate critical feedback on your work.
Two resources we’ll use are the “How to Revise a Statistical Manuscript” document on Sakai and the Noble paper on revising a manuscript (10 simple rules).
Read the feedback
Think about why you were given that feedback, and whether (and how) the feedback can improve your work
Acknowledge to those who provided the feedback that you’ve done (2) and make improvements
Stephen Witte at University of Texas describes a process he calls select-prioritize-reflect. After going through the process, in class (and in journal article review) you’ll create a comprehensive point by point response to all the reviews.
Read the feedback, and consider each point.
Selecting which feedback to address, and how comprehensively to address it, is a key skill in research and in the workplace.
Order the feedback from most to least important. Under time constraints, you may need to decide which comments are the most important to address. If you can’t address a good comment, you should mention it in the discussion section of your paper or report.
Reflect on the feedback, revising your analysis plan and report.
As described in Noble (2017), when a journal article is revised, a comprehensive response to reviews is also submitted.
Point by point: This is literally writing down every single thing the reviewers said, say in a font like italics, and then underneath every single thing describing how you addressed it, or if you didn’t, why not. You start with the comment from the most “powerful” or important person (e.g., the journal editor, the highest ranking person in your company, the professor) and work your way down. So for a journal the ordering would be comments from the editor, comments from the associate editor, comments from reviewer 1, reviewer 2, etc.
The overview mentions major criticisms, criticisms mentioned by multiple reviewers, and generally highlights the big issues (with a focus on what you addressed successfully of course). For example, it might be something like this.
We would like to take this opportunity to thank the Editor, Associate Editor, and Reviewers for their insightful comments that have greatly improved the quality of this work. All praised the clarity of presentation, innovative model, and clear interpretation of results. Concerns were raised about the model validation, which we have addressed comprehensively on pages 3 and 10-12 of the revision, as well as about readability of the figures, which we have also improved. While one reviewer was hopeful we could incorporate data from the second quarter of 2020, we are unable to do so due to its delayed release. However, because these data are publicly available, we have provided reproducible code and clear instructions on how readers could do so on their own after the release date (now anticipated to be December 2020).
Then, write down every single thing everyone said, labeled by their name, and include your response, e.g.
Editor:
Insufficient evidence is provided showing validation and appropriateness of the model in the data application considered.
We have now included comprehensive external validation and sensitivity analysis, including results from fitting our model to data obtained during the same time period in a similar market. We have checked sensitivity of the results to prior distributions as well as to the form of the linear predictor in the model. Residual diagnostics show no evidence of lack of fit or violation of the normality assumption for our outcome.
Equation (1) on page 3 appears to be missing indices.
Thank you for catching this mistake; we have now added \(i\) to index individuals and \(j\) to index time, where appropriate.
If you have to, write a nasty response first, tear it up, and then be polite. Even if the reviewer fails to understand your work, it likely means that you could have done a better job communicating. You want to be sure the work is accessible to as broad an audience as possible.
Sometimes you won’t understand a reviewer’s comment. You can state this in your revision, saying you did your best and if you misinterpreted you’re happy to revise again as needed.
Sometimes a reviewer clearly is not well-matched to your paper. Be polite to the reviewer, but you can include a separate note to the editor that you have concerns (do this sparingly).
If the reviewer misunderstood, apologize for being unclear. If the reviewer points out the 5th typo or grammatical error, apologize for not proofreading carefully.
Remember the figures I said I improved? They should all be inserted in the response to reviews to show you did the work. In fact, it’s nice to quote every change directly in the response. Don’t assume the reviewer will read your revision – in fact, with a paper, you are often hoping the reviewer skims your response and decides you’ve comprehensively addressed everything, recommending acceptance of your manuscript!
Even if you feel the reviewer will disagree with your response, you should not try to avoid a difficult point by ignoring it.
This will tend to be a long document. Change the typeface, color, or indentation (or all three) to help the reviewer discriminate between the review you’ve copied and pasted, your response to the review, and any changes you have made to your manuscript.
If possible, answer “yes” nor “no” directly, and then elaborate if needed.
In a journal review, the reviewer holds considerable power over whether your manuscript is accepted. Don’t give the impression you can’t be bothered to do additional analysis or simulations the reviewer requests. If you don’t think a request is useful, do it anyway, and put it in the response to reviews and then make a case for why you don’t think the results belong in the main body of your manuscript (you can often offer to put them in an online supplement).
Sometimes it can be hard for a reviewer to tell what changed, so refer explicitly to the previous and revised versions of the paper and make clear what the change is.
You can interpret this one multiple ways! Sometimes it’s really cleansing to write the profanity-laden, insulting response you feel burning in your belly. Get it down on paper and imagine the reviewer cares. Then come back to reality, throw it away, and write the real one.
Other times, your initial draft can be a working document for you and your coauthors as you consider the different ways to respond and the cost-benefit tradeoffs. Then as you make final decisions, you’ll clean up the document to become something geared more towards convincing the editorial team that you responded comprehensively.
Comments from the Editor
The editor often will summarize comments from all other reviewers, at times providing hints or keys to what the editor feels is most important. Ignore comments from the editor at your own peril. (Sometimes you should stick your neck out, but generally, it’s a good idea to try to make the editor happy – either by taking their advice or impressing them with your brilliance!)