Introduction
The Science Mission Directorate at NASA is committed to ensuring that proposals are reviewed in a fair and equitable manner. To this end, the Science Mission Directorate will evaluate proposals for the various elements of the ROSES program using a double anonymous peer review (DAPR). In this system, proposers do not know the identities of the review committee members, and reviewers are not informed of the proposers’ names even after all proposals have been evaluated and ranked (see below). The goal of double anonymous peer review is to reduce the impact of implicit or unconscious bias in evaluating the merit of proposals.
Initial Results from the Double Anonymous Peer Review Experiment
Results from the double anonymous peer review experiment in ROSES-2020 showed an improvement in the overall quality of the review process, as well as an improvement in the diversity of awardees. For example, in the ADAP program, prior to the double anonymous peer review, women made up 26% of the total applicants, but they only ranked in the top two in committee rankings 16% of the time. After the shift to double anonymous peer review, the percentage of women in the cohort increased to 31% and they ranked in the top two 32% of the time. Moreover, the success rate of early-career researchers even surpassed that of more experienced researchers, enriching the talent pool further.
General Guidelines for Proposers
Each program element text includes a section related to the preparation requirements for the double anonymous review for that specific program. Additionally, the NSPIRES page for any program element utilizing double anonymous peer review contains guidance on how to prepare proposals for double anonymous review in the “Other Documents” section. There are two separate guidance documents, one for most ROSES programs and another for first-stage proposals for the General Observer/General Investigator Opportunities in Astronomy.
Ideal Texts for Double Anonymous Proposals
Below are some ideal texts for double anonymous proposals:
Example Text 1:
Over the past five years, we have used 2MASS infrared photometry to conduct a census of nearby cold dwarf stars (Cruz et al., 2003; 2006). We have identified 87 cold dwarf stars of type L in 80 systems at a hypothetical distance of less than 20 parsecs from the sun. This is the first true census of cold dwarf stars in a large and volume-limited sample. Most distances rely on spectral parallax measurements, which have a 20% accuracy—sufficient for current purposes. We already have high-resolution images of 50 systems, including our rapid programs from years 9 and 13, #8581 and #10143. We propose to target the remaining sources through this current proposal.
Example Text 2:
In the study by Rogers et al. (2014), we concluded that the best explanation for the dynamics of the shock wave and the spectrum from the shocked interstellar medium and the shocked inverted reference material is that a Type Ia supernova exploded in a pre-existing cavity of winds. This object is the only known example of such a phenomenon, and therefore provides a unique opportunity to shed light on the nature of Type Ia supernovae and their progenitors. If our model from Rogers et al. (2014) is correct, a unified channel for producing SNe Ia must exist. We propose here to obtain a second stage of observations which we will compare with the first stage obtained in 2007 to measure the proper motion of the shock wave.
Example Text 3:
Before and after radiation exposure, we will test changes in the composition of ice using our established cryogenic mass spectrometry technique (2S-LAI-MS) [Henderson and Gudipati 2014; Henderson and Gudipati 2015]. Our technique uses an infrared laser tuned to the absorption wavelength of water to gently vaporize the sample into the gas phase, where it can be ionized by an ultraviolet laser and analyzed by time-of-flight mass spectrometry. The main advantage of our technique is that it can provide in-situ compositional information at temperatures relevant to Europa (e.g., 50, 100, 150 Kelvin) without the need to warm it to room temperature or any other sample preparation. We will also perform continuous mass spectrometric analysis (using a residual gas analyzer and a quadrupole mass spectrometer already installed) during the irradiation to measure the quantity of volatile materials and evolved gas products.
Guidelines
For Reviewers
The overall goal of double-blind peer review is to ensure that proposals are evaluated based on their scientific merit, without regard to the qualifications of the proposal team. Below are some specific points:
– Evaluate proposals solely based on the scientific merit of what is proposed. Do not waste time trying to identify the principal investigator or the proposal team. This applies even if you believe you know the identities of team members. Remember to discuss the science, not the people. In committee discussions, do not speculate on the identities of individuals, hint at their possible identities, or raise discussions about the team’s previous work. When writing evaluations, use neutral pronouns (such as “what they propose” or “the team previously evaluated similar data”).
– After completing the scientific evaluation of all proposals, the committee will be provided with “non-blinded expertise and resources” documents for a specific set of proposals (typically the top third, based on assigned scores and expected selection rates). The committee will assess the team’s qualifications to ensure their ability to carry out the proposed scientific research. This assessment will not affect the scientific score and will be used by the selection official to help determine if there are any risks in funding the proposal.
Guidelines for the Equalizer
An “Equalizer” will be appointed by NASA to attend the committee room during all discussions. The equalizer is not a reviewer or a committee member, but a trained individual to ensure that committee discussions focus on the strengths and weaknesses of the proposal and do not veer into discussion of the identity and qualifications of the principal investigator and their team.
Common Mistakes in Preparing Double-blind Proposals
Below is a non-exhaustive list of common mistakes when preparing double-blind proposals:
– Including metadata (such as PDF tags, document properties) that reveal the name of the principal investigator.
– Reusing proposals prepared before the double-blind peer review and not carefully testing the text preparation.
– Providing the names of investigators on the table of contents or in the header or footer of the page.
– Providing travel sources for professional travel (such as conferences).
– Mentioning the name of the institution in the financial context.
– Including the names of the principal investigator or participants in budget tables.
– Attempting to “black out” identifiable information by inserting a black rectangle over parts of the text instead of redacting the text using specialized programs.
– Including a “non-blinded expertise and resources” document in the main proposal file.
Questions and Answers about Double-blind Peer Review
Here are some questions and answers about double-blind peer review:
Q1: If I accidentally reveal my identity while preparing my proposal, will it be returned without review?
A1: NASA understands that double-blind peer review represents a significant shift in proposal evaluation, and thus some slips in writing double-blind proposals may occur. However, NASA reserves the right to return proposals that are particularly attention-grabbing regarding identifying the proposal team.
NASA also acknowledges that some proposed work may be so specialized that, despite efforts to prepare the proposals anonymously, the principal investigator’s identity and team members can be easily discerned. As long as the guidelines are followed, these proposals will not be returned without review.
Q2: How will you handle conflicts of interest and bias?
A2: NASA program scientists will screen the names of team members and participating institutions. During the discussion of the blinded proposal, if the identities of team members become clear to the reviewer, the reviewer should disclose to NASA if there are any strong biases that prevent the reviewer from providing an objective assessment.
Q3: If the identity of the “proposal teams and institutions” is blinded, how can proposal teams and institutions discuss their record, ongoing work, complementary efforts, and institutional assets? For example, if an institution has worked closely with NASA for over 40 years in a specific area (such as radar over ice), won’t all the programs and institutional qualifications and the main qualifications of the team be lost in the review process?
A3:
There is no prohibition in the double-blind proposal on discussing these aspects; rather, they should be discussed without specifically mentioning the principal investigator or the group in the main part of the proposal. In such cases, NASA recommends writing “previous work” instead of “our previous work”; or using phrases like “obtained in private communication” when referring to potential work that might identify investigators. Researchers must be able to substantiate their case by describing their proposed program of observations, analyses, and skills necessary for success. If there are specific skills required, the committee will point them out and will be able to verify that when reviewing the “unidentified experience and resources.” The committee will provide a comprehensive analysis of the “unidentified experience and resources” document and will vote using a triple scale.
Guidelines for the Equalizer
An “Equalizer” will be appointed by NASA to attend the committee room during all discussions. The equalizer is neither a reviewer nor a committee member; rather, they are a trained individual to ensure that the committee discussions focus on the strengths and weaknesses of the proposal and do not veer into discussions about the identity and qualifications of the principal investigator and their team.
Questions and Answers about Double-Blind Peer Review
Here are some questions and answers about double-blind peer review:
Q1: If I slip in preparing my proposal in a way that reveals my identity, will it be returned without review?
A1: NASA understands that double-blind peer review represents a significant shift in proposal evaluation; therefore, some slips in writing double-blind proposals may happen. However, NASA reserves the right to return proposals that are particularly apparent regarding identifying the proposal team.
NASA also acknowledges that some proposed work may be so specialized that, despite efforts to prepare the proposals anonymously, the identity of the principal investigator and team members can be easily discerned. As long as the guidelines are followed, these proposals will not be returned without review.
Q2: How will you handle conflicts of interest and bias?
A2: NASA program scientists will review the names of team members and participating institutions. During the anonymous proposal discussion, if team members’ identities become clear to the reviewer, the reviewer must disclose to NASA any strong biases that prevent them from providing an objective assessment.
Q3: If the identity of the “proposal teams and institutions” is unknown, how can proposal teams and institutions discuss their record, ongoing work, complementary efforts, and institutional assets? For example, if an institution has been closely working with NASA for over 40 years in a specific field (like airborne radar), won’t all of the program and institutional qualifications and the main qualifications of the team be lost from the review process?
A3: There is no prohibition in the double-blind proposal on discussing these aspects, but they should be discussed without specifically mentioning the principal investigator or the group in the main part of the proposal. In such instances, NASA recommends writing “previous work” instead of “our previous work”; or using phrases like “obtained in private communication” when referring to potential work that might identify the investigators. Researchers must be able to substantiate their case by describing their proposed program of observations, analyses, and skills necessary for success. If there are specific skills required, the committee will point them out and will be able to verify that when reviewing the “unidentified experience and resources.” The committee will provide a comprehensive analysis of the “unidentified experience and resources” document and will vote using a triple scale.
Virtual Meetings on Double-Blind Peer Review
Virtual meetings on double-blind peer review have been held across various programs and fields. Presentation slides from these meetings can be found in the links below:
–
Aura Science Team/ACMAP and Earth Science USPI
– Cryospheric Sciences
– Planetary and Exoplanets Research Programs
– Habitable Worlds
– Earth Sciences USPI
– Astrophysics Data Analysis Program (ADAP)
– SMD-wide town hall
– Astrophysics GO/GI
This is some guidance and information regarding the double anonymous peer review (DAPR) at NASA. This system aims to ensure that proposals are assessed fairly and equitably, and to achieve diversity and equity in the review process.
Source: https://science.nasa.gov/researchers/dual-anonymous-peer-review/
Leave a Reply