TESTING : our job

   1. Test every requirement listed in the DSS. 
   2. Get 100% TCA (cover all if-then branches). 
   3. Sign off all FPTEST SCRs. 

====================================================================
The Initial Development Process we follow is:
--------------------------------------------------------------------
 1. Pick a Baseline Load and its corresponding DSS (Keep them safe).
 2. Plan the effort required to Test that Baseline (Make metrics). 
 3. Design Tests and run Tests on the Baseline Load and DSS.
 4. All SCRs in the Baseline Load are CLOSED as 'covered by Init.Dev.'
 5. All SCRs AFTER the Baseline are done in the SCR REWORK PHASE.
====================================================================

A REQUIREMENT IS DESIGNATED BY AN ANCHOR.
NO ANCHOR, NO REQUIREMENT.

==============================
Initial Development ESTIMATION
------------------------------
 The best way to determine the amount of change is to
 do an ILEAF DIF on the DSS with a previously-tested DSS.

  New or Changed Requirement (NEW or UPDATE):
   1/4 page New or Changed = 10 ManHours.
   1/2 page New or Changed = 20 ManHours.
    1  page New or Changed = 40 ManHours.
 
  Unchanged Requirement (RERUN or REUSE):
    A. RERUN Tests checked against code changes = 2 ManHours per Test.
    B. REUSE inspection overhead = 2 ManHours per Test.

  New or Changed DI TERM:
    Many times, DI TERMs don't have ANCHORS, therefore THEY ARE NOT REQUIREMENTS.
    They are implemented as FUNCTIONS, and often put into .INC files.
   1/4 page New or Changed = 1 ManHours.
   1/2 page New or Changed = 2 ManHours.
    1  page New or Changed = 4 ManHours.

 You should also examine the SCRs involved.

 A VAX DIF on the code with a previously-tested version
 can also point to possible test effort.
   
 If the changes are not yet complete, speak with the code
 or docs people to determine the extent of the coming changes.
 They sometimes have ESLOCS data available.

 The trace matrix will point you from the changed
 code or docs anchor to the appropriate testids.

========================
UPDATE, RERUN and REUSE 
------------------------
Update: When a DSS section or corresponding code has
        substantially changed, we will need to Update the 
        test that covers that section and code. The TDF,
        and any BD,CT,INC,driver files that changed must be
        put in ACM. All results go in ACM.

Rerun:  When a DSS section has NOT changed, but the
        corresponding code is different, The test must
        be Rerun to make sure everything is ok. 
        if NOT ok: it becomes an Update.
        if ok: the TDF is INSERTed into the new CONFIG.
               All results go in ACM.

Reuse:  When a DSS section has NOT changed, and the
        corresponding code is the same. 
        The test does NOT have to be Rerun.
        An INS file will be created to document all Reused TDFs.
        Put the INS file in ACM using the Reuse SCR.  

        If you identify a DSS section (using ILEAF DIF) that
        has little or no changes, then you have found a 
        Rerun or Reuse section. To find out if it should be
        Rerun or not, do a VAX DIF on the ada code that 
        implements that section. If there is NO difference,
        you don't have to Rerun the test: It is Reuse.

===========================
Initial Development METRICS
---------------------------
 Copy the Old Metrics to make the New Metrics. It should be close.

 We use EXCEL SPREADSHEET. 
 The testids are listed in the Spreadsheet View,
 and the Graph View charts the plan and progress.

 Group Tests by section into "Phases" of ~1000 hours.
 Each Phase has its own Spreadsheet and Graph.

 Schedule the Phases so that the more stable sections
 get completed first. Phases can run concurrently,
 depending on manpower.

 I like to have SCR Rework metrics and RFS metrics as well.

 ManDays  = total ManHours / 8.
 WeekDays = (Finish_Date - Start_Date) * 5/7
 People_Needed = ManDays / WeekDays

===================================
The Initial Development Baseline:
-----------------------------------
 Baseline: Each test area will pick a Baseline load.
 All Initial Development testing will be done on this Baseline Load.

 Tell the SRB to issue a Test Copy of any DSS/Code SCR that affects
 our Tests and is targeted for loads beyond the Baseline Load. 
 These will be completed during the SCR REWORK PHASE.

=============================
KEEPING UP WITH THE PLAN:
-----------------------------
 Have the BASELINE SUT conveniently available.
 Have the BASELINE DSS conveniently available.
 Develop to the BASELINE.

 Initial development is designed to be quick.
 The metrics progress rapidly and there is little tolerance 
 for falling below the plan line.
 It does not include SCR rework time. 
 It does not include 100% TCA coverage.
 An initial development Test is a first-pass rough-sketch of the final Test.
 Don't get hung up on the details. 
 Get the requirements tested and move on to the next.
 If the Test is designed properly (with multiple condition tables)
 the Test will naturally get 90% or more branch coverage.
 Coverage problems can be addressed during SCR rework.
 If you follow the process and test to a baseline,
 you will progress rapidly. 

===========================
FLEXIBILITY IN THE PLAN:
---------------------------
   Sometimes the code changes are delayed for one reason or another.
   All the code could be done in all the other sections,
   but they haven't really even started in your section!
   Other times, the build runs fine for every section but yours.
   Hopefully, the DSS will be done, and you can start redesigning
   your Tests, but you can't run them until you get a build
   with those code changes in it. Life is not fair. Be flexible.

   Move the baseline. Adjust the metrics. Go as far as you can
   without actually running the Test. Do what you can.

   When a testid is going to be created to test a DI term,
   and this term is not implemented yet, there is not much
   reason to attempt to write this Test. That is a good reason
   to search the code for key variables before you start
   writing a Test.

   High-change and unimplemented areas should be set aside for last.
   This can be reflected in the metrics:
     1. zero out the hours
     2. mark the TDF "to be completed during SCR rework"
       
===========================
Example:
---------------------------
  I spoke to the coder and he has not implemented some terms yet.
  They are due to be included in a later load, but still on the
  code initial development SCR.
  Rather than waiting for that load, (it might be even later..)
  if you are mostly done with the Test,
  comment out the troublesome testcases,
  put a note in the Test indicating that
  these testcases check code that will be implemented
  in load #X, finish the inspection,
  put the stuff in ACM, and move on.
  We can catch this as SCR rework or RFS changes.
  (put a keyword like "SCR REWORK" in the TDF)
  Don't let unimplemented or changing code slow us up.

  However, it IS circumnavigating the metrics.
  The metrics are meant for you and for me, 
  so we can understand how we are doing.
  It doesn't benefit you or me to get caught at the deadline.
  So, I would only like to do this when there is a clear advantage.
  If an SCR was written because there were test failures,
  it is clearly not an instance where we would zero out the hours.

===========================
WHEN TO CLAIM CREDIT:
---------------------------
 New or Changed Requirement (NEW or UPDATE):
   1. Starting to design or fix a Test (design START)
   2. Put TDF in ACM. Test Development Inspection (design END)
   3. Executing a Test (exe START)
   4. Put VER, TCR in ACM. Test Execution Inspection (exe END)

 Unchanged Requirement (RERUN or REUSE):
   1. Examine the Test and the code it tests (design START)
   2. DIF the code from the old CERT to the new (design END)
   3. RERUN test or Build REUSE .INS file(exe START)
   4. Put RERUN TDF,RPT in ACM or REUSE Inspection (exe END)

 Test Development Inspection (TDI) main points: 
   1. make sure the requirements are being tested.
   2. make sure that the trace matrix is accurate.

 Test Execution Inspection (TEI) main points:
   1. make sure the test outputs are the same as those
      designed in the multiple condition table (MCT).  
   2. identify how to fill in the missing coverage.
      fill in coverage holes until total coverage is
      at least 90%. the higher the coverage, the better.

 We use a Peer-Review Inspection Process.
 We invite SQA to every Inspection.
 The Inspection Checklist is in FPDPROC:TEST_CHECKLIST_DWS.TXT

===========================
SCR REWORK:
---------------------------
 When the initial development is done, begin the SCR REWORK. 
 complete and sign off ALL OPEN TEST SCRs.

===========================
ITRR:
---------------------------
 Schedule an ITRR a month before you plan to start RFS.
 
===========================
RUN-FOR-SCORE:
---------------------------
 When you are certain there are no more upcoming SCRs, and
 you have some testers freed up,
 it is time for the RUN-FOR-SCORE (RFS).

 SEND OUT THE RFS INVITATION TO ALL CONCERNED. (SEE "RFS" LINK)

 You can still be working on SCR REWORK when you
 start RFS, just be sure that when you finish RFS:
  1. You notified SQA and Boeing before you ran every Test.
  2. There is no upcoming SCR that will affect any Test.
  3. All results are configured in ACM. 

 Any new SCR affecting a set of Tests will force you
 to re-RFS those Tests, so make sure that there are
 no unknown incoming SCRs when you are ready to RFS.

===================
SAMPLE RFS ESTIMATE
-------------------
               QTY of TDF    ESTIMATED RFS hrs
 Phase 1 & 3     120         120 X 2.25 = 270
 Phase 2 & 4     125         125 X 2.25 = 280
                             Total      = 550 Hrs

=======
LESSONS
-------
 1. on a weekly basis, run a REPORT PFD_SUMM_ALL on ACM,
    sort the results, and send it to HISO.
 2. on a weekly basis, generate the trace matrix and attach it
    to HISO. 
 3. keep a clean ACM work directory with no extra files and
    logical, meaningful filenames.
 4. every once in a while, compare a list of TCR headers with
    a list of all source procedures in ACM and keep a list of
    uncovered procedures. 
 5. find out how to RFS efficiently in batch mode.
 6. forward a copy of the SRB packet to HISO
    to keep them informed of upcoming SCRs.
 7. forward a copy of the build notifications to HISO
    to keep them informed of new builds and their contents.
 8. get a copy of the MD-11 cockpit pilot's guide and
    a copy of the EIS operation description.
 9. don't get mad.