IARPA - Solicitations - Office of Incisive Analysis, Sirius Program

Solicitations - Office of Incisive Analysis

Sirius Program - BAA Questions

IARPA-BAA-11-03
Proposers' Day Date: February 24, 2011
BAA Release Date: March 21, 2011
BAA Question Period: March 21, 2011 - May 2, 2011
Proposal Due Date: May 16, 2011

# Question Answer Date Posted
001 Does IARPA have a list of Frequently Asked Questions about IARPA BAAs? Yes. Please see the Frequently Asked Questions (FAQs) page on the IARPA website: http://www.iarpa.gov/faq.html 03/28/11
002 Does the 35% cap on indirect costs for 6.1 research apply to the Sirius program? No. The 35% cap on indirect costs does not apply to the type of funding available for Sirius. 03/28/11
003 In Volume 1, Section 4 of the proposal, the BAA says "This information does not contribute to the page count of Volume 1." Does that refer to the copies of the three relevant papers, the bibliography, or both? Does the bibliography count toward the 20 page limit? See BAA Section 4.B.1 (Volume 1, Technical and Management Proposal). “Except for the cover sheet, transmittal letter, table of contents (optional), signed Academic Institution Acknowledgement Letter(s) if required, OCI waiver/certification, draft human subjects protocol (see “Section 3: Detailed Proposal Information” below), game design information (see “Section 3 Detailed Proposal Information” below), bibliography, and relevant papers, Volume 1 shall not exceed 20 pages.” As noted here, the relevant papers and the bibliography do not count toward the 20 page limit. 04/01/11
004 Can a successful applicant narrowly focus on one field where cognitive bias is prevalent, and therefore test outcomes with relevant subjects within that field? See BAA Section 5.A.3 (Contribution and relevance to the IARPA Mission and Program Goals). The BAA does not specify a domain or topic area for the game(s), nor is there an occupation or educational requirement for the test subjects. However, the proposer should provide clear and compelling arguments that the selected field and subject pool will have relevance to the intelligence community. Intelligence analysts are drawn from a broad selection of technical backgrounds and as a population do not possess a homogeneous field of knowledge. Game play therefore should not require specialized knowledge. 04/01/11
005 What is the budget for this work? Are there any budget ratios or constraints related to when different monies need to be spent, for example for games production or research? See BAA Section 5.A.5 (Cost Realism). There is no predetermined award size for Sirius proposals. Proposers should submit proposals that will result in successful research efforts within the BAA criteria and timeline. Proposals should advance compelling research ideas based on sound science that respond to the stated challenges and identify how much they will cost. 04/01/11
006 Can the study design be changed in any way, or is it strictly constrained as prescribed in the BAA? In other words, will proposals that offer an alternative study design, with justification based on current research, be favorably considered? See BAA Section 1.A (Program Overview). The BAA specifies the required elements of the experimental design (pre-test, post-test, control and experimental conditions, and longitudinal follow-up), the experimental condition (a game), the control condition (a government-furnished training video), the independent variables (see Table 1), the dependent variables (see Table 2) and the moderating variables (see Table 3). A proposer may propose and justify additional research elements as long as the research design fits within these constraints. 04/11/11
007 Does the program require research teams to employ an expert in intelligence analysis? Should we locate our own, or would it be better to work with someone at IARPA, if we are selected? No. Subject matter expertise in intelligence analysis is not necessarily required to create a Serious Game for mitigating cognitive bias. Proposers should, however, staff their teams with all expertise they believe is required for successful completion of their project; see BAA Section 3.D.1 (Collaboration Efforts). IARPA will not provide subject matter experts. 04/11/11
008 Can performers facilitate discussions with subjects after game play? No. The Sirius research program is designed to test the effects of Serious Games and game variable manipulation, not games that include post-discussions. Please see BAA page 8 (Experimental Intervention). 04/11/11
009 Will the Sirius program offer opportunities to integrate an online game into the IC intranet? No. 04/11/11
010 What will the Government provide as GFI in Phase 1 (see page 16, Table 6)? The Government will provide a video lecture (see BAA page 6) as GFI in Phase 1. 04/11/11
011 What will the Government provide as GFI in Phase 2a (see page 17, Table 6)? The Government will provide a video lecture as GFI in Phase 2a. The Phase 2a video lecture will differ from the Phase 1 video lecture. The Phase 1 video lecture will cover only the 3 cognitive biases being studied in Phase 1. The Phase 2a video lecture will cover only the 3 cognitive biases being studied in Phase 2. 04/11/11
012 For the longitudinal studies over 8 or 12 weeks, is the treatment limited to the initial single 30 minute game playing training session, or can it include subsequent voluntary continued game play training that the participants access during their own time? See BAA, Page 8. ”Teams may use the longitudinal test period to continue to study the effects of game session duration and repetition, to determine the optimal or minimal amount of game play that may be required for a positive transfer of training. For ecological validity with respect to the notional use case, the control condition (training video) shall be administered only once.” Regardless of whether continued game play is “voluntary” or scheduled by the researcher, all game play time must be objectively measured (see BAA Table 1). 04/11/11
013 For the independent experimental replication by the IV&V team, the BAA specifies that the IV&V team will be “using analysts or analyst surrogates.” Do you expect that “analyst surrogates” will be non-specialists like college students? Proposer knowledge of the IV&V process is not required for the proposer to respond to the BAA. See also the response to Question 004. 04/22/11
014 Pre-test and post-test results, as well as the influence of experimental variables, may differ significantly with college students as opposed to analysts or analyst surrogates. Is the expectation that our results will generalize across both populations? Yes. See BAA Section 1.B (Program Milestones and Metrics, Phase 1 Summary, Phase 2 Summary). “IV&V Team replicates results using delivered Alpha games.” 04/22/11
015 We know from your response to question 008 that you are not interested in games that include post-discussions as part of the experimental intervention. However, are we allowed to engage in post-discussions as a means of collecting data about our variables? Yes. Researchers may engage with subjects after gameplay, for the purposes of additional data collection or for subject debriefing. 04/22/11
016 Can we examine subsets of required dependent variables with complete coverage achieved across a suite of experiments rather than examining all required dependent variables in every experiment? See BAA Section 1.A (Program Overview, Program Requirements, Table 2). All required dependent variables must be examined in each experiment. 04/22/11
017 Can we examine subsets of required moderating variables with complete coverage achieved across a suite of experiments rather than examining all required moderating variables in every experiment? See BAA Section 1.A (Program Overview, Program Requirements, Table 3). All required moderating variables must be examined in each experiment. 04/22/11
018 The BAA says: “"Multiple gameplay segments of approximately 30 minutes may be developed. A segment may address one, or multiple cognitive biases… Available game in-class play time will be about 30 minutes, p.11” If we do separate game segments for each bias, does that mean we have multiple 30-minute sessions to cover all three biases (i.e., up to 90 minutes in total), or three ten-minute sessions for each bias? In other words, are you trying to limit overall experiment game play to thirty minutes? See BAA Section 1.A (Program Overview, Notional Use Case and Design Constraints). We envision that gameplay time in a classroom may be approximately 30 minutes. A player does not necessarily have to complete all gameplay within that 30 minutes; this is a notional use case for an initial exposure to the game. We also expect that the games may be played outside of the classroom (“Games as Homework”). Researchers have the flexibility to design the gameplay in the fashion that they believe (and can justify) will result in the best outcome for mitigating the specified cognitive biases. Total game play time for a subject may exceed 30 minutes (see Question 019). 04/22/11
019 Are we unconstrained in the length of session duration (e.g., 10 minutes—or 90 minutes)? See BAA Section 1.A (Program Overview, Program Requirements). Research teams have the option of varying session duration and session repetition as they see fit “…to determine the optimal or minimal amount of game play that may be required for a positive transfer of training.” 04/22/11
020 What does “recognition of cognitive bias” refer to? Is it OK to measure cognitive bias inside the game? See BAA Section 1.A (Program Overview, Approach). This refers to the subjects’ ability to “…to recognize cognitive biases in themselves,” outside of the game. The BAA does not specify a requirement to measure the subjects’ ability to recognize bias in others, nor does the BAA does specify a requirement to measure the subjects’ ability to recognize bias while inside the game. 04/22/11
021 What does “discrimination among biases” refer to? See BAA Section 1.A (Program Overview, Program Requirements, Table 2). This refers to the subjects’ ability to correctly classify a particular bias as one of the three being studied in a program Phase. Subjects should be able to discriminate among their own biases. The BAA does not specify a requirement to measure the subjects’ ability to discriminate among biases being displayed by others. 04/22/11
022 How does IARPA want us to assess and measure bias mitigation? See BAA Section 1.A (Program Overview, Program Requirements). “Proposers shall describe the design of their own tests for recognition, discrimination among, and mitigation of the cognitive biases, crafted to fit the particular bias that they are studying and the bias elicitation protocol that they are using.” 04/22/11
023 Can pre- and post-tests for mitigation be administered inside the game? See BAA Section 1.A (Program Overview, Program Requirements). In order to test for transfer of training to conditions outside of the game, behavioral elicitation and pre-test and post-test measures must be done outside of the game. 04/22/11
024 What do you mean by the term “analyst”? See BAA Section 1.A (Program Overview, Study Population). The BAA does not specify a specific type of analyst. 04/22/11
025 If an offeror is looking at multiple aspects of a single game variable (like how provided, timing, quality), can the various aspects be considered different “optional” variables across Phase I and Phase 2 experiments? Yes. See BAA Section 1.A (Program Overview, Program Requirements, Table 1). “Proposers may suggest additional game independent variables for study; justification shall be provided. The 3 optional variables chosen for Phase 2 shall be different from the 3 optional variables chosen for Phase 1.” 04/22/11
026 Do session duration and repetition have to be manipulated and studied? Do they both have to be examined in both phases? Yes and yes. See BAA Section 1.A (Program Overview, Program Requirements). “…Each research team shall test the variable of session duration and repetition in both Phases.” 04/22/11
027 Is it a requirement that the 3 biases in each Phase be taught/experimented together at the same time (i.e., participants learn and are tested on 3 biases in the same experiment)? See BAA Section 1.A (Program Overview, Program Requirements). The BAA does not specify a requirement for teaching the 3 biases at the same time. 04/22/11
028 Is it a requirement that all 3 biases be tested in every 3-month experiment cycle? The BAA does not specify a requirement for testing all 3 biases in every Cycle. 04/22/11
029 If multiple games are created, is it acceptable to manipulate game mechanism Independent Variables in only a subset of the games (i.e., where they may be expected to have the most impact)? The BAA does not specify a requirement for how the game(s) should be designed. 04/22/11
030 To maximize the likelihood of identifying a successful game candidate, is “debiasing method” an acceptable game mechanism independent variable? See BAA Section 1.A (Program Overview, Program Requirements). “Proposers may suggest additional game independent variables for study; justification shall be provided.” Anything that is manipulated within the game can be considered a game independent variable. 04/22/11
031 During Phase 1, is the requirement to conduct post-tests and 8-week longitudinal follow-up applicable to both Phase 1a and Phase 1b? Yes. See BAA Section 1.B (Program Milestones and Metrics, Program Overview) “Testing for longitudinal effects at 8 weeks after each initial immediate post-test.” 04/22/11
032 Do we need to prepare 4 different proposals for each Phase 1a, 1b, 2a and 2b, or just one proposal that covers all phases? See BAA Section 2 (Award Information). “This BAA will result in awards for all phases of the program.” One proposal should be submitted that covers all phases of the program. 04/22/11
033 For each phase, will the GFE training video be designed so that it is possible to use separate portions addressing the 3 specified cognitive biases *separately* to avoid contaminating subjects who are being trained and tested with regard to one specific bias in the other Cycles of the study? See BAA Amendment 02 Section 1.A (Program Overview, Program Requirements). “One of these video segments will address the three cognitive biases to be studied in Phase 1 and the other will address the three cognitive biases to be studied Phase 2. For ecological validity and consistency across research teams, each of these video segments must be presented in their entirety and subjects shall view each video only once.” 05/05/11
034 Must all studies in all cycles be conducted using the same version of the game(s), or may we enhance and adapt the game(s) as the experiments proceed (e.g., different versions in different cycles)? We are concerned with what is required to be developed by month 9 as opposed to month 19. Is continued iterative refinement of the game(s) until month 19 permitted? See BAA Section 1.B (Program Milestones and Metrics, Program Overview). “The Sirius Program structure is based on an iterative development and experiment strategy.” Performers may use different versions of a game or different games in each Cycle. Final versions must be delivered in month 19 (Phase 1) and month 38 (Phase 2). See BAA Table 6. 05/05/11
035 Does the game produced at 9 months (or multiple games produced at month 9) need to cover training on all 3 biases, or is it acceptable to produce a game(s) that trains only some of the cognitive biases by month 12 and continue to enhance/add the game(s) for additional biases and game conditions until month 19. See BAA Section 1.A (Program Overview, Program Requirements). “Each research team will examine three (3) cognitive biases in Phase 1 and three (3) additional cognitive biases in Phase 2.” The BAA does not specify when or how each bias should be trained and studied within the Phase and the Cycles. Proposers are responsible for designing and justifying their experimental and development approach. 05/05/11
036 Do the targets in Table 4 ("reduction in cognitive bias") refer only to the 'Mitigation of Cognitive Biases' dependent variable/metric from Table 2, or do they also refer to 'Recognition of Cognitive Biases' and 'Discrimination among Cognitive Biases' variables/metrics? Table 4 refers to mitigation of cognitive bias. 05/05/11
037 On page 11 of the BAA it states that "Game segments shall be replayable." Can you clarify if this refers to 'play back, 'play again' or something else?

 

For example, does this mean one can play the same session again (i.e., same starting point/seed) and have the same experience? Or that you can play back a session (i.e., view, as in watch a recording of, a session) after the fact? Or that we must support multiple plays of the same segment so that it can be played again at a later time -- and if so, must it be novel on each play?		 See BAA Section 1.A (Program Overview, Notional Use Case and Design Constraints). “Replayable” does not refer to a record and play back capability, but means “to play again”. The BAA does not specify whether the game experience must be the same, or novel, on each replay. 05/05/11
038 The offeror assumes that the original signatures required in BAA Section 4.C.2 refer to the signed Organizational Conflicts of Interest (OCI) Waiver and the Academic Institution Acknowledgement Letter signatures. Are any other original signatures required for submittal? Yes. See BAA Section 4.A.1 (Proposal Information). ”Offerors must submit two hard copies and one soft copy of their proposals: one original hard copy with original signatures; one hard copy with original or copied signatures; and 1 electronic copy with Volume 1, Volume 2 and any permitted, additional information (.pdf format preferred) on a CD-ROM.” 05/05/11
039 What comprises the "final system documentation" that is part of the Months 24, 42 and 48 deliverables? The final system documentation should include both system administrator documentation and user documentation required to successfully install and run the game. 05/05/11
040 Are there any specific definitions/thresholds for key personnel, significant contributor, and contributor? See BAA Section 4.B.1 (Volume 1, Technical and Management Proposal), Section 2, Summary of Proposal, F. Project contributors: “Interested parties are encouraged to leverage personnel that are dedicated to BAA requirements no less than 10% of their time.” Other than this, the BAA does not provide specific definitions or thresholds for the various roles. 05/05/11
041 The Technical and Management Proposal requires "a detailed description of the roles that contributors (including Principal Investigator(s)) will play based on their qualifications and on their level of effort in each year of the Program.” This information is typically found in a Cost Volume, and accumulates significant page count. May we instead include this information in the Cost Volume, and just reference it in the Technical Volume? No. See BAA Section 4.B.1 (Volume 1, Technical and Management Proposal). Proposers should describe project contributors in Section 2, Summary of Proposal, F. Project contributors. Also note that “Volume 1 shall not exceed 20 pages. Any pages exceeding this limit will be removed and not considered during the evaluation process.” 05/05/11
042 Does the Government require the game development tools to be delivered along with the required developmental licenses? The BAA does not specify delivery of the development tools nor development licenses. See BAA Section 4.B.1 (Volume 1, Technical and Management Proposal). BAA Section 3, Detailed Proposal Information, Statement of Work (SOW) Section 4 provides that “offerors shall address the delivery of the following:

 

h. A listing of all game development tools and the version used, including but not limited to game engine, graphics rendering, music editing, art/graphics creation and editing, artificial intelligence design tools, system developer kits (SDKs) and any other development infrastructure necessary to integrate those tools”

 

The Government does require the right to use the game created using these development tools. See BAA Section 6.B.3.a.2 (Commercial Items (Technical Data and Computer Software))		 05/05/11
043 Must the research teams test all dependent variables across all biases? Yes. See Question 016. 05/05/11
044 The Phase 1 timeline specified in the BAA seems to require play-testing the game in advance of receiving IRB approval and government approval to move forward with human-subject tests. Is this correct? Some IRBs may view conducting play-testing in advance of receiving approval for a protocol as inappropriate. Proposers should check with their local IRBs to determine if play-testing is considered human subjects experimentation. Regardless, all IRB approvals should be completed by month 6, and performers will report on play-testing at month 9, allowing 3 months for play-testing after IRB approvals. See BAA Table 6. 05/05/11
045 The BAA provides a description of "Projection Bias" but I have seen other definitions in the literature. Which do you mean? Please use the definition provided in BAA Section 1.A. 05/05/11
046 Please clarify the timeline constraints related to longitudinal testing and the Cycles. See BAA Section 1.B (Program Milestones and Metrics, Program Overview). Experimental and test activities can continue into all of the following Cycles within Phase 1 and within Phase 2. For example, longitudinal testing with subjects from Cycle 1 may occur during the Cycle 2 timeframe. The experimental constraint is that there must be a time gap (of 8 weeks in Phase 1 and 12 weeks in Phase 2) from the last time a subject is exposed to the game, until the test for persistence of the effect. 05/09/11
047 Can we get a list of brief descriptions of representative kinds of analytical taskings that might inform scenario development? No. Proposers are responsible for providing all subject matter expertise for developing proposals and performing research & development, including scenario development (see also Question 007). However, the BAA does not specify a requirement for analytic scenarios (see also Question 004). 05/09/11
048 Considering the length and complexity of the project, as well as the large number of deliverables and milestones that must be described, would IARPA consider relaxing the technical volume page limit to 30? No. See BAA Section 4.B.1 (Volume 1, Technical and Management Proposal). In addition to the described 20-page limit for the Technical and Management Proposal in Volume 1, the proposer is allowed an additional 20 pages for the Draft Human Subjects Protocol (see Appendix E) and an additional 20 pages for the Game Design Information (see Appendix F), thus allowing the proposer a total of 60 pages in Volume 1, excluding the cover sheet, transmittal letter, table of contents (optional), signed Academic Institution Acknowledgement Letter(s) if required, OCI waiver/certification, bibliography, and relevant papers. 05/09/11
049 There is a requirement for the offeror to provide “cost or pricing” data if their proposed cost exceeds $650k. Does the offeror provide the cost data if the $650k exceeds Year 1 costs, Phase 1 costs, or the total for the entire 50 months of the proposal? See BAA Section 4.B.2 (Volume 2, Cost Proposal). “ ‘cost or pricing data’ shall be required if the offeror is seeking a procurement contract award of $650,000 or greater…”. This amount refers to all 50 months of the program. 05/09/11
050 If the offeror is required to provide “cost or pricing” data, is it required for just Year 1 or all years? See BAA Section 4.B.2 (Volume 2, Cost Proposal), “Section 2: Detailed Estimated Cost Breakdown”, items 1 through 7. The offeror shall provide cost or pricing data for all years. 05/09/11
051 With respect to page 12 of the BAA, Figure 1 (Sirius Program Structure), can you please clarify the beginning and ending months for Phase 2a independent play-tests performed by the Government? In Figure 1, it appears to be months 25 through 35. However, according to Table 6 (page 17 of the BAA), the Phase 2a initial game prototype is not expected to be complete until month 28. Could you please clarify the months for Government Independent play-tests to begin and end for Phase 2a? See BAA Section 4.B.1 (Volume 1, Technical and Management Proposal). “Play-testing shall be conducted and feedback incorporated by research teams prior to formal human subjects experimentation; the methods and schedule for play-testing shall be described in the proposal. In addition, independent play-testing will be conducted by up to 10 Government representatives, during the development/experiment cycles in both Phases. Proposers shall address how they will integrate Government representatives into play-testing.”

 

Play-testing may continue concurrently with development activities. Therefore, for Phase 2a, play-testing may begin as early as month 25 and continue through month 37. Awarded research teams will be expected to coordinate play-testing activities with the Government.

 

Also note that play-testing is a different activity from the experimental replication being conducted by the IV&V team. 		05/09/11
052 In Phase 1b, we are to submit our game(s) to IV&V at month 19. Is the version of the game(s) that we provide to IV&V at month 19 to be the exact version that we used for Cycle 3, or are we allowed to make updates to our Cycle 3 game(s) prior to providing it to the IV&V team at month 19? This question also applies to Phase 2a (Cycle 6, turning in to IV&V at month 38). The BAA does not specify a version of the game(s) to be submitted, only that a “final version” be submitted at month 19 (Phase 1b) and month 38 (Phase 2a). 05/09/11
053 If an offeror plans to cite an exemption from Federal regulations for human subject protection should they address the detailed defense of that claim in "Data Sources" section of the technical proposal volume (and thus counting against the 20 page total for that volume) or should it be addressed in Appendix E? The offeror may address this in Appendix E “Draft Human Subjects Protocol”. See BAA Section 4.B.1 (Volume 1, Technical and Management Proposal). BAA Section 3, Detailed Proposal Information, Data Sources states that “Offerors shall include the documentation required in Appendix E. Documentation must be well written and logical; claims for exemptions from Federal regulations for human subject protection must be accompanied by a strong defense of the claims.” 05/09/11
054 Will the government IV & V test be made available to researchers to use as a pre- and post-test before and after game play? No. See BAA Section 1.A (Program Overview, Program Requirements). “The Government will develop and test a standardized instrument to measure cognitive biases, which will be used by the Independent Validation & Verification (IV&V) team. The contents of this standardized instrument will NOT be shared with the research teams, but will be designed to test transfer of training to new contexts.” 05/09/11
055 Would you clarify what is meant by session "repetition" in Table 1? Does repetition refer to the number of repetitions of a specific scenario/bias or game session (repeating the same scenario again) or does it refer to the total number of different trials completed within a given game session? Is it intended to be manipulated separately from session duration? A “session” is a continuous period of time spent with a human subject in an experimental or control environment. Session repetition therefore involves inviting the subject back for an additional exposure to the experimental environment. Session duration should be measured as the continuous amount of time that the subject is exposed to the experimental or control condition in a single session. 05/09/11
056 Are all experimental participants expected to view the control video (within subjects design) or could a separate control group be used that only views the GFI video, and the game intervention group(s) would not view the video (between subjects design)? The BAA does not specify whether a within subjects or between subjects design should be used. See BAA Section 1.A (Program Overview, Program Requirements). ”Proposers shall describe how they will address possible experimental confounds associated with their experimental design.” 05/09/11
057 Section 6.B.3.a.2 says “Beta Pilot Game: For technical data and computer software not first produced in the performance of the contract (including all integrated commercial, proprietary, and/or third party data and software), the Government requires, for a minimum of one year after the conclusion of the Sirius program, the right to use and display such technical data and computer software within the Government.”

 

Please confirm that the Government expects no other rights in data other than “the right to use and display” any Commercial Items that may be delivered. Also, please explain “within the Government” – is this within ODNI, the IC, DoD, or the entire US Government? 		See the response to Question 042, and see also BAA Section 4.B.1 (Volume 1, Technical and Management Proposal). BAA Section 3, Detailed Proposal Information, Statement of Work (SOW), item 4.h.ii through item 4.h.iv.

 

See also BAA Section 6.B.3.a.1 (Noncommercial Items (Technical Data and Computer Software)). The Government will assume unlimited rights unless otherwise stated in the proposal.

 

“Government” refers to the entire US Government. 		05/09/11
058 Our team is made up of representatives of 3 organizations—two small companies and one university. Neither company has a standing IRB, but the university has an IRB with a FWA. Will using the university IRB be sufficient for the human subjects review requirement? Also, if two faculty members from the university are on the team, do the two companies need an “IRB Authorization Agreement” with the university as well? You should consult with the university IRB regarding their willingness to conduct reviews as part of an agreement with the two companies. Such an arrangement is acceptable to IARPA if it conforms to HHS requirements. See http://www.hhs.gov/ohrp/policy/engage08.html 05/10/11