Introduction
The Science Mission Directorate at NASA is committed to ensuring that proposals are reviewed in a fair and equitable manner. To this end, the Science Directorate will evaluate proposals for the numerous elements of the ROSES program using a double-blind peer review (DAPR). In this system, proposers do not know the identities of the review committee members, and reviewers are not informed of the identities of the proposers even after all proposals have been evaluated and ranked (see below). The goal of double-blind peer review is to reduce the impact of implicit or unconscious bias in assessing the merit of a proposal.
Preliminary Results
Results from NASA’s double-blind peer review experiment in ROSES-2020 showed improvements both in terms of the overall quality of the review process as well as the demographic diversity of the beneficiaries. For instance, in the ADAP program, prior to the double-blind review, women made up 26% of the total applicants but only occupied the top two positions in committee rankings 16% of the time. After transitioning to double-blind review, the percentage of women rose to 31% of the total and they occupied the top two rankings 32% of the time. Furthermore, the success rate of early-career researchers even exceeded the success rate of more experienced researchers, enriching the talent pool further.
General Guidance for Proposers
Each program element text contains a section addressing the requirements for anonymity for those specific programs. In addition, the NSPIRES page for any program element utilizing double-blind review includes guidance on how to prepare proposals for double-blind review in the “Other Documents” section. There are two separate instructional documents, one for most ROSES programs and another for phase one proposals for the General Investigator/Observer opportunities in astrophysics.
Tips for Reviewers
The overarching goal of double-blind peer review is to ensure that proposals are evaluated based on their scientific merit without consideration of the qualifications of the proposing team. Below are some specific points to keep in mind.
– Evaluate proposals solely based on the scientific merit of what is proposed. Do not waste time trying to ascertain the identity of the principal investigator or proposing team. This applies even if you believe you know the identities of team members. Remember to discuss the science, not the people.
– In committee discussions, do not speculate about the identities of members, hint at their possible identities, or raise discussions about potential prior work of the team. When writing evaluations, use neutral pronouns (such as “what they propose” or “the team has previously evaluated similar data”).
– As part of the final review, and only after completing the scientific evaluation of all proposals, the committee will be provided with “non-anonymous expertise and resources” documents for a subset of proposals (typically the top third, according to assigned scoring distributions and expected selection rates). This aims to allow reviewers to assess the capabilities of the team necessary to carry out the proposed scientific investigation. This assessment will not affect the score of the science and will be utilized by the selecting official to assist in determining any risks associated with funding the proposal.
Examples of Double-Blind Texts
Below are some examples of double-blind texts for proposals:
– Over the past five years, 2MASS infrared imaging has been used to conduct a census of nearby cold brown dwarfs (Cruz et al., 2003; 2006). A total of 87 cold brown dwarfs have been identified in 80 systems at approximate distances of less than 20 parsecs from the Sun. This is the first true census of cold brown dwarfs in a large and volume-limited sample. Most distances rely on spectral parallax measurements, which are accurate to 20%, sufficient for current purposes. We already have high-resolution images for 50 systems, including rapid imaging programs for cycles 9 and 13, #8581 and #10143. We propose to target the remaining sources through the current proposal.
–
In a previous study [12], we reached the conclusion that the best explanation for the dynamics of the shock wave and the spectrum from the forward-driven super medium and the backward-driven reflective material is that a Type Ia supernova explosion occurred in a pre-existing cavity of winds. This object is the only known example of such a phenomenon, thus providing a unique opportunity to shed light on the nature of Type Ia supernovae and their potential progenitors. If our model from Rogers et al. (2014) is correct, a single degraded channel for producing SNe Ia should exist. Here we propose to obtain a second phase of observations that we will compare with the first phase obtained in 2007 to measure the proper motion of the shock wave.
– Before and after radiation irradiation, we will test changes in ice composition using our established cryogenic mass spectrometry technique (2S-LAI-MS) [Henderson and Gudipati 2014; Henderson and Gudipati 2015]. This technique uses an infrared laser tuned to the water absorption wavelength to gently push the sample into the gas phase, where it can be ionized by a UV laser and analyzed by time-of-flight mass spectrometry. The main advantage of this technique is that compositional information can be obtained directly in situ, at temperatures relevant to Europa (such as 50, 100, 150 Kelvin), without the need to heat it to room temperature or any other sample preparation. We will also conduct continuous mass spectral analyses (using a residual gas analyzer and a quadrupole mass spectrometer) during irradiation to measure the amount of volatile materials and evolved gas products.
Common Issues in Preparing Double-Blind Proposals
Here is a non-exhaustive list of common issues when preparing double-blind proposals:
– Including metadata (such as PDF tags, document properties) that reveals the name of the principal investigator.
– Reusing proposals that were prepared prior to double-blind peer review and not carefully experimenting with text preparation.
– Providing investigator names in the table of contents or in the header or footer of the page.
– Stating the source of travel for professional travel (e.g., conferences).
– Mentioning the institution name in the financial narrative.
– Including names of the principal investigator or participants in budget tables.
– Attempting to “beautify” specific information by inserting a black rectangle over parts of the text, instead of formally redacting text using specialized software.
– Including a “Unblinded Experience and Resources” document in the main PDF proposal.
Q&A Regarding Double-Blind Peer Review
Here are some questions and answers regarding double-blind peer review:
– Q1: If I slip in preparing my proposal anonymously, will it be returned without review?
– A1: NASA understands that double-blind peer review represents a significant shift in proposal evaluation, and therefore there may be some slips in writing double-blind proposals. However, NASA reserves the right to return proposals that are particularly egregious in revealing the identity of the proposing team.
– Q2: How will you handle conflicts of interest and bias?
– A2: NASA program scientists will review the names of team members and institutions involved. During the discussion of the anonymous proposal, if the identities of team members become clear to the reviewer, the reviewer must disclose any strong biases that prevent them from providing an objective assessment.
– Q3: If the identity of “proposing teams and institutions” is wrapped in secrecy, how can proposing teams and institutions discuss their record, ongoing work, supplemental efforts, and institutional assets?
– A3: There is no restriction in the anonymous proposal from discussing these aspects; rather, they should be discussed without referring to any specific investigator or group. In such cases, NASA recommends writing “previous work” instead of “our previous work”; or using phrases like “obtained in private communication.” Proposers should be able to demonstrate their case by describing their proposed program of observations and analyses that require the necessary skills for success; if specific skills are needed, the committee will note that and will be able to verify it when consulting the document “Unblinded Experience and Resources.” The committee will provide a comprehensive review of the “Unblinded Experience and Resources” document and will vote using a triadic scale.
–
Q4: Although it is not possible to avoid showing information in the proposals that might reveal the identity of the proposers, such as the context and motivation for the proposed research, unique methodologies, and cited references, why are reviewers kept guessing the identities of the proposers (leading to undesirable consequences)? Furthermore, the proposers’ records should be part of the merit of the proposals.
– A4: It is appropriate to address the context and motivation of the research, as well as the unique methodologies and references, and so forth. The main difference is that these aspects should be discussed without referring to a specific investigator or group in the main part of the proposal. Remember that the goal of double-blind peer review is not to make it completely impossible to guess the identities of the investigators but to shift the focus from individuals to the proposed science.
– Q5: The institution is also supposed to be anonymous; how do reviewers determine whether there are sufficient institutional resources to conduct the research?
– A5: The records of the proposing teams will be handled in the document “Unidentified Experience and Resources” and will be voted on using a three-point scale (uniquely qualified; qualified; unqualified).
– Q6: What are the expected unintended consequences of this procedure? Does it really serve the meritocratic system?
– A6: Experience gained from the anonymous proposal process for the Hubble Telescope indicates that there are few unintended consequences. However, NASA takes proactive measures to ensure: SMD programs are aligned in the experience with double-blind peer review. Proposers have sufficient information and guidance to adequately prepare proposals anonymously. Review committees are adequately informed about double-blind peer review. The duration of each committee is not significantly increased. High-risk/high-impact proposals are not disproportionately affected in funding.
– Q7: An additional page was allocated for the proposal summary. But I no longer see that; what happened?
– A7: This is no longer required as reviewers will now be able to see the summary from the NSPIRES page. So make sure your summary on NSPIRES is anonymous!
– Q8: Is a table of contents allowed? I can’t find any reference to it in the double-anonymous guidelines or in text B.4 HGIO.
– A8: The proposal guidelines and the summary of ROSES calls set forth the default rules unless replaced by B.1 The Heliophysics Research Program Overview or a program element. Since the proposer guidelines and ROSES calls summary allow a table of contents (up to one page), it is permitted unless explicitly prohibited. There is no mention of it in either B.1 or in B.4 Heliophysics Guest Investigators Open, so it is allowed and does not count against the limited page part of the program element.
– Q9: Even if the reviews are referenced by numbers [1, 2, etc.] and reviewed in the third person, there may be a large number, and possibly a predominant number, of references to my previous work. Therefore, it is very possible that the reviewer could guess who is submitting the proposal or at least from which group or institution the proposal comes.
– A9: Yes, this cannot be completely avoided. NASA chose square brackets in the text of the anonymous proposal so that reviewers are not constantly reminded of potential proposers while reading the proposal. However, in the absence of a bibliography, reviewers have no way to evaluate any claims made in the proposal. Thus, we have chosen a balanced way to handle the references. One note: there may be a predominant number of references to a specific author’s work, but the double-blind review process still does not make it 100% clear who the principal investigator is. For example, the principal investigator could be a different or former member of the group frequently cited in the proposal.
Q10: Can more information be provided about the budget content and justification in the anonymous proposal?
– A10: First of all, proposers must understand that the requirements for concealing all information related to salary levels, benefits, and general interest rates from the budget and budget justification included in the main proposal document have not changed. This requirement is the same as it has been in several previous proposal rounds. In addition to this requirement, under the dual anonymous peer review process, proposers must also ensure that the budget and financial justification in the main proposal document are anonymized by removing any language or logos that reveal the names and/or institutions of the proposing team members. Please note that these requirements – concealment and anonymous preparation – do not apply to the “overall budget” document, which is uploaded separately from the main proposal document and is not seen by the reviewers.
– Q11: Should I prepare identifying information in my proposal anonymously?
– A11: Yes. Please ensure that identifying information (such as PDF tags) that could provide information about the proposing team and/or institutions is removed.
More topics from NASA
– James Webb Space Telescope
– Perseverance Rover
– Parker Solar Probe
– Juno Mission
Source: https://science.nasa.gov/researchers/dual-anonymous-peer-review
Leave a Reply