Avionics Test & Integration Center Air Force Flight Test Center Edwards Air Force Base, California

     The bulk of this material is taken from Air Force Manual 99-112, Electronic Warfare Test and Evaluation Process -- Direction and Methodology for EW Testing written by Howard W. Leaf, Lt General USAF (Retired)

Why do we need a disciplined EW T&E Process?
     First, the troubled histories of EW programs that came on line too late, were over budget, or unable to meet user needs. Past EW programs have displayed a pattern of latent deficiencies manifesting themselves late in their development, necessitating expensive fixes and retesting.
     Second, we need to use the limited and costly resources that exist to support EW T&E efficiently. This means using concepts that take advantage of current and emerging Modeling and Simulation (M&S) and ground test technologies to streamline Developmental Test and Evaluation (DT&E) and Operational Test and Evaluation (OT&E). Test concepts that promote a fly-fix-fly methodology or emphasize open air range testing as the primary method are not prudent approaches.
     Third, we need a process that will help us do a better job of assessing and managing risk. Risk meaning the probability that a product will have latent deficiencies that will not show up until later testing or when fielded. This risk will likely cause significant,

• disruption of schedule,
• increase in cost, and/or
• degradation of performance.

     Fourth, we need a quality product for the user. Quality means customer (user) satisfaction is our goal. Customer requirements will be the basis for our T&E efforts. We want the end result of Air Force acquisition and T&E efforts to be an effective and efficient EW product.

Predict, Test, and Compare.
     The EW T&E Process is built upon the three pillars of Predict, Test, and Compare. Its foundation is to do ground testing before flight testing. Tools are used with the process to plan, execute and record T&E efforts. This is illustrated in figure 1.
     The Predict pillar is supported by Modeling and Simulation (M&S) tools. These are used to help develop test concepts, and predict test results prior to testing. A model of the EW System Under Test (SUT) interacts with other models at various levels of performance to estimate and predict system performance. The EW system model will usually be a Digital System Model (DSM) that can be run on a digital computer with other digital models such as platform models, threat models, environmental models and scenario models.
     The Test pillar is supported by test facilities and reporting tools. These are used to produce and record data and information from the tests. Test facilities will be selected to be efficient and cost effective. The EW Single-Face-To-Customer (SFTC) office is available to assist with this significant task. Test reports must be concise, timely and contain information needed by decision makers. Risks areas identified in pre-test planning must be addressed along with risk mitigation measures evaluated.
     The Compare pillar is supported by the T&E record tools. These are T&E requirements documents, test data, plans, evaluations, test results and summaries that are explained in more detail in section C of Air Force Manual 99-112. The T&E record is implemented in the form of a Test Process Archive (TPA) that is used to track test progress as the EW system matures.

EW T&E Process
     Figure 2 illustrates the six step EW T&E Process, which follows from the AF T&E Process described in AFI 99-103. Using it involves taking actions and making decisions that answer the following questions:
     Step 1) Determine Test Objectives--What are the technical and operational issues that must be proved? What are the risk areas? What T&E information is needed by decision makers? Are the test objectives based upon mission, task and performance requirements? What are the underlying assumptions supporting these objectives, and are they likely to change?
     Step 2) Conduct Pre-Test Analysis--What is the test concept? What are the test points? What are the predicted outcomes? What analytical tools must be developed? What types and quantities of data are needed?
     Step 3) Test--Are the appropriate T&E resources being used to conduct the tests? Will they accomplish the major test objectives? Will the tests show if risk mitigation measures work? Is the required data being collected and analyzed? Are results being reported?
     Step 4) Evaluate--Have conclusions been reached and recommendations made? How do results compare with predictions? Has analysis identified the root cause of discrepancies? Have technical and operational judgments been applied to the results? Has the information been reported to decision makers?
     Step 5) Acceptable Risk--This is a decision step, a judgment call by a decision maker. Was the test outcome satisfactory? Have technical and operational risks been reduced to acceptable levels? Will user needs be met? If yes, proceed forward. If no, go to step six of the process.
     Step 6) Improve--What must be changed or refined? Who must take corrective action? These are actions to improve the EW system design, correct a flawed test method, find and fix errors in models and simulations, or improve the test process.

     An example that has EW implications is the development of a new aircraft. Assume a DoD review of the issues (Step 1) decides the aircraft must be twice as survivable as the aircraft it is replacing. From that perspective the T&E process includes whatever it takes to answer the survivability question for a later decision in Step 5.
     When viewed from the Test Engineer�s perspective, an important part of survivability is the lethality envelope of the threat which is often not identified until detailed test planning in discussions between the RTO, SPO and the user.
     From this narrow perspective the test conducted in Step 3 could be the determination of the detection range for the Radar Warning Receiver (RWR) for a specific threat. The associated action in Step 5 is a decision by the program manager as to the adequacy of the demonstrated detection range in meeting the overall survival requirement.

More Productive Flight Testing.
     The EW T&E Process replaces the fly-fix-fly test philosophy with the more scientific predict-test-compare philosophy. Rigorous ground testing is done before and during flight testing to permit a high confidence flight test. Simulations are used to predict ground and flight test results at specific points in the performance envelope. Ground tests are then conducted, differences analyzed, and if appropriate, deficiencies corrected.
     Once ground testing achieves positive results, flight testing starts using the predict-test-compare philosophy with deficiencies investigated on the ground. Verification of the ground test data at the proper envelope points means flight testing will not have to be done throughout the entire performance envelope. This approach amounts to flight testing smarter, not harder.

Use of Government Test Facilities.
     There has been a lack of discipline among programs as to which test facilities should be used. Large programs have a tendency to delegate EW developmental testing to a prime contractor, who formulates his own test process (with Government approval) and may develop program-unique test facilities to carry it out. This usually results in tests that do not address or adequately demonstrate key performance parameters, and data that cannot later be compared with data from Government facilities.
     The EW T&E Process requires that test criteria reflect operational performance requirements, and pre-test analysis provides a means to statistically correlate test results from multiple test facilities. If programs require their contractors to follow the EW T&E Process and use existing Government test facilities as much as possible, data will be more reliable, comprehensive, repeatable, and correlatable. This will enhance the integration of DT&E and OT&E, and yield the following additional benefits:

• Increased confidence in test results;
• Increased commonality in data products;
• More standardized data analysis and reporting;
• Reduced test schedules and cost; and
• Enhanced credibility in modeling and simulation.

Additional Benefits of the EW T&E Process:
Early and thorough evaluation of system concepts. Early feedback to the design process. The creation and evolution of test requirements through rigorous analysis and evaluation. The identification of performance parameters that are critical to operational effectiveness. Establishment of validated linkages between operational requirements and test criteria. Timely and credible test results to support milestone decision making. A closer tie between intelligence analysis, systems engineering, test facilities and testers. Early identification and acquisition of test assets.

Two Failed Programs
Two brief histories follow that show the need to use the right resources in the proper order.

Program A. A bomber EW system leapfrogged directly from contractor System Integration Laboratories (SIL) to flight testing, skipping Hardware-In-The-Loop (HITL) facilities and Installed System Test Facilities (ISTFs) altogether. The program considered and rejected HITL testing because it was not mandatory and would have elongated the schedule. Flight testing revealed serious problems, and the system was sent back to HITL facilities and ISTFs for diagnostic testing. By then, however, it was too late. The design was locked in, the hardware had been built and packaged, and the fixes were prohibitively expensive. At that point the only alternatives were a major modification program or scrapping the existing design and starting over. Using a DSM and ground testing against an updated threat would have easily highlighted a flawed system architecture.

Program B. A fighter EW system went from its SIL to an ISTF to an Open Air Range, then finally to a HITL facility. By using facilities in the improper order, the program failed to achieve the timely design evaluation and deficiency identification HITL facilities uniquely provide before the hardware is completely designed, fabricated, and packaged. As a result, after nominal completion of OT&E, the system was unable to obtain OSD/DOT&E certification. HITL facilities, initially ignored by the program, were then used to solve the problems. Changes made at this stage were very expensive.

Radar Warning Receiver Example.
     The EW T&E Process supports the EW Acquisition Cycle. For example, a typical radar warning receiver (RWR) development or upgrade could use the various EW T&E resources during the five phases of the acquisition cycle.
     For this RWR example, using the EW T&E Process, modeling and simulation can be used for concept validation and effectiveness predictions. When the system components are subjected to the SIL for system integration tests, high density threat presentations (thousands of emitters) and other environmental factors such as terrain can be simulated. This is the first opportunity to actually test the system. In the HITL facility, the system can be tested for closed-loop effectiveness and the results correlated with those from the earlier effectiveness assessments made with the digital models.
     The next step is the ISTF. Installed systems testing provides the first opportunity to test the system in its as-installed configuration. At this point, performance validation can be accomplished. High-density threat signals can be reintroduced and on-aircraft EMI/EMC testing can be completed. With all this ground testing accomplished, the system under test can next be evaluated in the OAR and dynamic performance with actual environmental effects assessed.
     In the open air range test, the system's effectiveness can be revalidated as well, this time in an environment of controlled tactics with multiple threats, aircraft, or systems in flight. Throughout the entire test process, M&S is used to predict test outcomes, and measurement facilities are sequentially used to measure signature characteristics, including low observable characteristics, to fully assess dynamic system performance. The objective is to integrate these EW evaluation results into an overall assessment of the weapon system's military worth.