3. Research for Service Analysis (12 pgs) 3-1 Research for Service Analysis Section 3 deals with the integration of HF prior to approval to initiate the formal Acquisition Management System (AMS) process and covers the topics indicated below. You can page through Chapter 3 to review these topics. P. 3-2: Triggers for developing proposed acquisition P. 3-3: HF guidance when proposing small acquisition projects P. 3-4: HF integration when proposing large acquisition projects P. 3-5: Completing Requirements of In Service Review (ISR) Checklist (FAA network user only) P. 3-6: Sample Requirements in ISR Checklist (FAA network user only) P. 3-7: Planning the HF Effort P. 3-8: Specifying HF Activities P. 3-9: Monitoring and Reviewing HF Activities and Deliverables P. 3-10: Specifying Contents of the Transfer Package P. 3-11: Sample Contents of a Transfer Package P. 3-12: Conclusion: Research for Service Analysis 3-2 Research for Service Analysis Triggers for Developing Proposed Acquisition. There are a number of different sources of triggers that need to be documented in order to produce a request for approval by the Joint Research Council (JRC) to start the formal Acquisition Management System (AMS) process for a project or to define needs and preliminary requirements for a project to develop an enhancement within an existing program. See Operational Needs Assessment (ONA) for an example. Such sources include but are not limited to the following: Internal to FAA Air Traffic Organization (ATO) Technology Development & Prototyping Division (ANG-C5 (FAA network user only)) External to FAA - through a transfer package or recommendation submitted to the Program Management Organization (AJM) and Mission Support (AJV) Collaborative Decision Making (CDM) Program MITRE NASA RTCA SWIFT (System Wide Information Management (SWIM) Industry-FAA Team) 3-3 Research for Service Analysis HF Guidance When Proposing Small Acquisition Projects. This can be provided by the FAA Human Factors Division (ANG-C1) or the FAA Planning and Analysis Team (AJM-131 (FAA network user only)). These offices can assist with identifying HF analyses relevant to defining a shortfall and in outlining possible solutions. And they can help ensure that these analyses are tailored to be consistent with the size, cost, and complexity of the solution being acquired (HF-STD-004A). 3-4 Research for Service Analysis HF Integration When Proposing Large Acquisition Project. For larger projects, significant Research for Service Analysis needs to be conducted in order to prepare an Operational Needs Assessment (ONA) or to define needs and preliminary requirements for a project that is an enhancement of an existing program. Additional Research for Service Analysis may also be completed after a preliminary ONA has been approved to move forward by the ATO Directors Forum or in response to the submission of a transfer package submitted to AJM-S by an external organization such as NASA or MITRE. Preliminary Concept Exploration, Development and Evaluation. For larger projects that merit significant Research for Service Analysis, concept exploration, development and evaluation is necessary. HF involvement in this research includes: Specifying HF considerations relevant to a preliminary shortfall analysis Defining users and their associated operational needs and constraints Defining use cases, scenarios and critical tasks Identifying important elements of the broader system environment Generating and evaluating design alternatives Inclusion of HF input during this initial stage is important because production pressures during Solution Implementation tend to limit the extent to which design alternatives can be adequately considered and evaluated. This initial stage prior to initiation of the formal Acquisition Management System (AMS) process provides an opportunity to more fully define user needs and shortfalls, as well as to consider the broader use environment and evaluate design alternatives. 3-5 Research for Service Analysis Completing Requirements of In Service Review (ISR) Checklist. Another example of important HF issues that are relevant for consideration during Research for Service Analysis is provided by the In Service Review (ISR) Checklist (FAA network user only). The ISR Checklist is an FAA management tool that is used to ensure that important criteria have been met prior to fielding of an acquisition. For large projects, early consideration of these specified HF criteria during Research for Service Analysis can help to ensure that critical HF issues are addressed, as production pressures during Solution Implementation may result in inadequate, cursory consideration of these factors.3-6 Research for Service Analysis Sample Requirements in ISR Checklist. There are a number of requirements in the ISR Checklist (FAA network user only) that can be addressed in the Research for Service Analysis. Examples are provided below. Item # 6.1.1  Have human factors requirements, plans, analyses, and verifications been determined and documented for the product in accordance with the FAA Human Factors Acquisition Job Aid, FAA Human Factors Design Standard (HF-STD-001B), Human Factors Engineering Requirements (HF-STD-004A), FED-STD-795, Uniform Federal Accessibility Standards (UFAS), and other appropriate standards and conventions? 6.1.2  Have usability issues and human factors risks for all user (e.g., operators, maintainers, supervisors, administrators) populations been addressed for all intended sites? Item # 6.1.3  Have the human-system performance studies and analyses (such as those related to workload, training, functional design, computer-human interface, staffing, safety and health, special skills and tools, work space, automation functions/roles, displays and controls, information requirements, display presentation, visual/aural alerts, input/output devices, communications, procedures, anthropometrics, documentation, and environment) been conducted to identify, document, and resolve/mitigate human factors issues? Item # 6.1.6  Have steps been taken to identify, eliminate, reduce, or mitigate the sources of human error with the highest risks (combination of likelihood, severity, detection, and recovery)? See additional HF criteria in Items 6.1.4, 6.1.5, 6.3.1, 8.4.2 and 8.6.10 of the ISR Checklist (FAA network user only).3-7 Research for Service Analysis Planning the HF Effort. For large projects, there are several steps that are important to ensure that the organization conducting the Research for Service Analysis integrates HF into its work. Step 1. Has the FAA Human Factors Division (ANG-C1) or the FAA Planning and Analysis Team (AJM-131) been consulted for guidance on how to effectively integrate HF into the Research for Service Analysis? Step 2. Has a qualified HF specialist been added as part of the FAA Systems Engineering Management Team managing the project? Step 3. Has this HF specialist on the FAA Systems Engineering Management Team defined the HF activities and deliverables that need to be included as requirements in the Statement of Work (SOW) for the organization conducting the Research for Service Analysis, tailored to be consistent with the size, cost, and complexity of the solution being acquired (HF-STD-004A)? Step 4. If the Research for Service Analysis is competitively bid, has the HF specialist been involved in evaluating the submitted bids for the HF components? Has the organization conducting the research included a qualified HF specialist on its team and included a user group to provide input?  Has this organization specified the appropriate HF activities and deliverables as part of its proposal? 3-8 Research for Service Analysis Specifying Human Factors Activities. A large project should include a focus on workflow, usability, procedures, equipment, workload/staffing, human error and operational suitability. This can include both material (e.g., equipment) and non-material (e.g., procedures) needs, shortfalls and concepts and inputs needed by stages of the Acquisition Management System (AMS) process. Activities also should include consideration of the requirements for the ONA and ISR Checklist (FAA network user only). Keeping in mind the need to tailor the effort to reflect the nature and magnitude of the project (HF-STD-004A), possible activities include: Ethnographic studies (on site observation/interviews of prospective users) Input from operational users and other stakeholders using questionnaires, focus groups, structured interviews and use case reviews Task, functional and user analyses, including development of use cases, scenarios and Critical Task Analysis Reports (see p. B-7), considering the broader hardware, software and distributed work context HF input to a preliminary CONOPS, with a separate CONOPS for the different user groups (including operations and maintenance) as needed Specification of the HF aspects of the ONA and associated set of preliminary functional requirements Design of storyboards, user interface mockups and functional prototypes Completion of formative evaluations of storyboards, user interface mockups and prototypes, including evaluations of alternative designs and design features using heuristic analyses, cognitive walkthroughs, usability studies and/or Human-In-The Loop studies (HITLs) to evaluate usefulness and usability 3-9 Research for Service Analysis Monitoring and Reviewing HF Activities and Deliverables. For large projects, there are several steps that are important to ensure that the organization conducting the Research for Service Analysis effectively integrates HF into its work. Step 1. Has the HF specialist on the FAA Systems Engineering Management Team reviewed the details of proposed activities by the organization conducting the research and observed the completion of these activities? Step 2. Has the HF specialist on the FAA Systems Engineering Management Team reviewed the Operational Needs Assessment (ONA) and associated set of preliminary requirements based on the Research for Service Analysis? Step 3. Has the HF specialist on the FAA Systems Engineering Management Team reviewed the HF deliverables and the contents of the transfer package prepared by the organization conducting the Research for Service Analysis? The following pages provide examples of HF components of sample transfer packages. 3-10 Research for Service Analysis Specifying Contents of the Transfer Package. For large projects, the findings from the Research for Service Analysis can provide valuable insights for the stages in the formal AMS process that are not adequately communicated in an Operational Needs Assessment (ONA), Concept of Operations CONOPS or requirements. As one FAA Program Manager indicated: “We found that the contractor [who was implementing the solution] was revisiting things we solved 2 years ago, so we decided to let the contractor see the prototype developed during concept development. We’d spent years doing the original concept exploration to develop the CONOPS and requirements. Why throw all that expertise away? There’s a lot that they have learned that isn’t captured in the CONOPS and requirements. Let’s give the contractor a head start. They will have new ideas too. I wish I had done this earlier. It would have saved a lot of money.” The transfer package should include both documents and any storyboards or prototypes that have been developed during the Research for Service Analysis. The next page provides examples of HF content for a range of transfer packages. 3-11 Research for Service Analysis Sample Contents of a Transfer Package. The table of contents of a sample transfer package developed as part of the Research for Service Analysis for an FAA acquisition is provided below. Note that many of the included artifacts have important HF content based on the results of the Research for Service Analysis. Note, however, that the contents of the transfer package should be tailored to be consistent with the size, cost, and complexity of the solution being acquired (HF-STD-004A). Tech Transfer Package Overview Discussion Outcomes (The information in the Discussions Outcomes section was gathered from operational users and other stakeholders using Questionnaires, Focus Groups, Use Case Reviews, Human Factors Working Group (HFWG) / Working Group (WG) sessions, and Data Administration (DA) Working Group sessions. It included 25-30 briefings based on early UI mockups and associated feedback.) Discussion Outcomes Intro and ConOps Overview briefing Discussion Outcomes briefing for each topic CSS-Wx Products CSS-Wx Products Overview briefing Textual Wx Products spreadsheet Graphical Wx Products spreadsheet Use Cases Use Cases Overview briefing Use Cases List and Details spreadsheet Use Cases [series of DOCs] (60 focusing on Air Traffic users; 32 on Maintenance users and 1 on data administration) Preferred Architecture Preferred Architecture briefing Technical Analysis Technical Analysis Summary briefing Supporting Items Enterprise Information Management (EIM) Package EIM Overview briefing Position and Information Management Utilizing EIM document Static Data List spreadsheet Supporting Items Computer Human Interaction (CHI) Guidelines CHI Guidelines Overview briefing CHI Guidelines document Concept of Operations (There were three very different user groups, so three CONOPS were developed. One for ATC, one to support national management for standardization across facilities while allowing some customization by facility, and one for enterprise-wide system management, which had both National and local components. They were described in separate CONOPS.) Solution CONOPS document Data Admin CONOPS document Maintenance CONOPS document Screening Information Request (SIR) to Contract Award Summary of CONOPS Changes briefing Maintenance CONOPS Overview briefing Support Personnel Synopsis Summary of Requirements Changes (SIR to Contract Award) SIR to Contract Award Summary of Requirements Changes briefing Tech Transfer Glossary 3-12 Research for Service Analysis Research for Service Analysis: Conclusion Triggers. Research for Service Analysis is triggered by identification of an operational need or opportunity to improve performance. In order to help ensure that an acquisition meeting such an operational need will be useful, usable and actually used, FAA Order 9550.8, Human Factors Policy requires that HF considerations be addressed in the earliest phases, including this initial stage. HF Guidance. The FAA Human Factors Division (ANG-C1) or the FAA Planning and Analysis Team (AJM-131) can be consulted to assist with identifying the HF analyses for defining a shortfall and outlining an operational concept for a solution. This includes advice on how to tailor the effort to be consistent with the size, cost, and complexity of the solution being acquired (HF-STD-004A). HF Management. HF guidance can be provided for small projects to produce an ONA, as well as for larger projects that require significant research. It also can help ensure that information is generated to support later stages in the formal AMS process and address requirements of the ISR Checklist (FAA network user only). For larger projects, the HF specialist appointed to be part of the FAA Systems Engineering Management Team can provide input on the HF requirements for inclusion in the SOW for the organization conducting the Research for Service Analysis, including HF activities and components of the transfer package, as well as providing overall management of the HF components of the research.