scr rework
[email protected]
==============================
SCR REWORK ESTIMATION
------------------------------
When we find a problem, and write an FPCODE or FPDOCS SCR,
we usually get a TEST COPY or a TEST SPLIT.
Sometimes an FPCODE SCR gets SPLIT off into many other SCRs.
These SCRs could be copied to others.
This "SCR TREE" can get large, and each SCR in the tree
needs to be looked at because ANY of them could COPY or
SPLIT into a code change that will affect our tests.
The COPIED TO and COPIED FROM is easy to see,
but the "No. Split SCRs:" is easy to miss.
If the FPDOCS portion of the SCR has been completed,
the best way to determine the amount of change is to
do an ILEAF DIF on the DSS with a previously-tested DSS.
New or Changed Requirement (NEW or UPDATE):
1/4 page New or Changed = 10 ManHours.
1/2 page New or Changed = 20 ManHours.
1 page New or Changed = 40 ManHours.
Unchanged Requirement (RERUN or REUSE):
A. RERUN Tests checked against code changes = 2 ManHours per Test.
B. REUSE inspection overhead = 2 ManHours per Test.
A VAX DIF on the code with a previously-tested version
can also point to possible test effort.
If the changes are not yet complete, speak with the code
or docs people to determine the extent of the coming changes.
They sometimes have ESLOCS data available.
The trace matrix will point you from the changed
code or docs anchor to the appropriate testids.
===========================
SCR REWORK:
---------------------------
When the initial development is done, begin the SCR REWORK.
complete and sign off ALL TEST SCRs in the REV state.
We should not be working any SCR that is not in the REV state.
If it is OPE or RFR, the SCR has not been approved by the SRB.
There should not be any effort to work on these SCRs.
At best, we should look at them to notify the SRB (or the author)
that the change has not yet been approved.
However, until approval is granted by the SRB there should not be
any effort spent on making changes identified by the SCR.
What Build? Look at the SCR: there will be a field called
TARGET CONFIGURATION. That's the Load it should be fixed in.
The Build will have a similar name.
B717_LOAD_5_0 == M955014_NP
The SW verification team (that's us!) has a responsibility to find all errors
of the type where the FPD software does not meet its requirements. (We can
also help point out illogical or untestable requirements, although these
should be caught and fixed in the requirements reviews before we start
creating tests). Sometimes these errors are due to incorrect requirements,
and sometimes they are due to incorrect code. In either case, the
discrepancy will typically be reported first as a PR. After investigation to
understand the problem (and be sure it's not a testing error or a driver
problem), an SCR is created. This SCR goes to a software review board (SRB)
consisting of systems, software and test engineers, which analyzes the impact
of the problem on the safety, functionality, and maintainability of the FPD.
Following this analysis, the SRB assigns each SCR a priority code. SCRs
with highest priorities (fixes required for safety or certification) must be
addressed first, so they are quickly targeted to be fixed in a specific load,
often the next deliverable load. Lower priority SCRs (those with less
serious effects on the flight deck or that cause aircraft maintenence
problems or software maintainablity difficulties) may or may not be fixed
depending upon the time and resources available. These SCRs are put into a
SRBTBD category, meaning that the SRB review board has not determined which
load or document revision (if any) will contain a fix for the problem.
While we would all prefer to fix every error in our software and
documentation, the economic reality of the situation is that we (Honeywell
and our customers) do not allocate sufficient time and money to fix every
problem. Some are judged to be not worth the cost to fix them, and others
are found too late in the process to get them fixed without adding too much
risk to the schedule. Sometimes a lower priority SCR will be targeted to be
fixed because the customer elevates its priority or because another higher
priority problem is being fixed in a related area and it is not too costly to
fix the lower priority SCR at the same time. Many will never be fixed.
It is not a certification requirement to fix all the lower priority SCRs.
The threshold for deciding which problems must be fixed is established
through discussions and negotiations with the customers and the certification
authorities. In every case, the verification team is doing its job when the
problem is found and the SCR has been created. Test failures caused by the
unfixed problems are referenced to the SCR to demonstrate that the failure
was noted and that its impact has been analyzed. When I write up the final
report on the test results, I list every test failure, refer to the open SCR
written for that failure and show the priority code assigned to the SCR. The
certification authorities have to right to review and question any of these
SCRs, if they feel that they might be certification issues. If the problems
result in missing structural coverage (e.g. there is a missing or incorrect
requirement documented by the SCR), there must be a systems/software analysis
of the uncovered code to verify that the code does what it should do and that
there are no safety issues in the untested code. This analysis is documented
on a structural coverage deficiency SCR.
Honeywell and the customer may choose to fix some of the SRBTBD SCRs in later
certifications, or they may remain in this state indefinitely. I'm trying to
get some action here in Phoenix on a process that identifies the SCRs which
describe incorrect or missing requirements and adds them to the trace
database so that we can actually use such SCR corrections/changes for writing
test cases. This is not as good as actually getting all the corrections put
into the DSS/SRD, but would be better than the current situation where we
have no alternative other than failing tests and referencing the
corresponding SCR.
It is understood that the test team needs as much advance notification as is
possible when SRBTBD SCRs are retargeted to a specific load. Such retargets
are only made through another meeting of the SRB. The test team is expected
to participate in the SRB by providing feedback on the impact of retargeting
the SCRs, so there should be no big surprises.
SCR REWORK
Below is a list of FPTEST SCRs which were written for PFD. Most,
if not all, of the work done for the SCRs will have been covered
during the initial development of our tests. However, we really need
to ensure that the work has been covered. In order to verify that we
have covered the SCRs in our tests, I would like you to determine
which HISO engineer is responsible for the section covered by the SCR,
then ask them to verify that they have covered the SCR. We will quite
probably be able to avoid most of the normal SCR rework, but we need
to double check now, so we don't get hit later. I see a couple of
possibilities for the verifications:
1) FPTEST created only from FPDOCS SCR
Please ask the responsible engineer to verify that they have
covered any DOCS logic changes in their tests, perhaps most easily by
verifying that they wrote their tests using the changed version of the
DSS or a more recent version.
2) FPTEST created from FPCODE SCR
Please ensure that the tests have been run on a build which
includes the changed software or a more recent version of the changed
software. (see acm_config_contents.txt in b3845np).
SCR REWORK
When doing SCR rework, there is one very important point that must be
done for all SCR rework. As a checklist item, you must:
1) Verify that testcases which supposedly verify that an SCR problem has
been corrected duplicate the problem in the original build (old)
build.
Another way of saying this is: Your testcases must be debugged on
the old build to verify that the testcase produces the problem.
On the -700, there were several examples of TDFs which were supposedly
changed to verify that a problem did not occur. When run on the new
build, the problem did indeed not occur. However, the TDFs were not run
on the old build. When the TDFs were run on the old build, it was found
out that the new testcases DID NOT CAUSE THE PROBLEM!!!
In short, if you do not debug tests to ensure that they cause the problem
on the old build, it is possible that new tests are doing ABSOLUTELY
NOTHING. This is an important point. Let me know if you have any
questions.
SCR REWORK
There were quite a few changes made to the code and to the docs for
this SCR. Whenever you analyze the test SCRs for SCR rework, it is
necessary to go back through the ENTIRE SCR tree to look for changes. It
is not sufficient just to look at the problem description to determine if
the problem pertains to PFD. Therefore, for FPTEST:1365, one would have
to go back to 1365.00 to find the parent SCR, FPBOEING:428. FPBOEING:428
then refers to FPDOCS:2490 and FPCODE:3265. Both the code and docs SCRs
have splits. If you look at FPCODE:3265.01 and FPDOCS:2490.02, you will
find MANY PFD changes. Please have someone verify that the initial
development of our tests have covered the changes on FPDOCS:2490.02 and
FPCODE:3265.01. Also, please try to help the PFD testers understand how
to look back through the entire SCR tree. This is very important when
doing SCR rework.
SCR REWORK
> what scr covers a change in
> CDS_V05_03_ANNUN.DSS/GEN=28 ???
lavcu s
rlogin els522
fpdocsacm
sho gen/scr CDS_V05_03_ANNUN.DSS/GEN=28
It tells me that gen 28 was created on
4-dec-1997 by wendorf on FPDOCS scr 3138.
Then look at the FPDOCS scr and trace it
to the FPTEST scr.
NOTES:
To do this, you must be ABSOLUTELY SURE
that gen=28 was the FIRST time this change
appeared. For example, a doc
change in gen=26 would also be
in gen=28. You may mistakenly think that
gen=28 was the gen of the change, and get
the WRONG SCR!
So, it will be necessary to DIF the gens.
Start with a really old gen that
you KNOW is too old to include the change.
sho his CDS_V05_03_ANNUN.DSS
gen=26 is likely the oldest we need to check.
gen=25 was created a long time ago.
Fetch the gens 25,26,27,28 into a directory
and DIF them, naming the outputs something
meaningful because you are going to be searching
the DIF outputs for keywords associated with
the change, looking for the earliest appearance
of the change.
DIF2526.txt
DIF2627.txt
DIF2728.txt
Search carefully through the DIF output files
until you have found the earliest appearance
of the change, then you will know FOR SURE
which gen it first appeared in.
SCR REWORK
b7376testacm
report pfd_summ_all
exit
sort/key=(pos:42,siz=17) pfd_summ_all.dat pfd_summ_all.dat
--edit and remove all garbage and stuff from other projects
sort/key=(pos:20,siz=12) pfd_summ_all.dat pfd_summ_all.dat
--edit and remove all non SW TEST stuff
sort pfd_summ_all.dat pfd_summ_all.dat
--now you got a clean output
SCR REWORK
quick SCR tutorial:
1. When you have decided to start work on an SCR:
a. MOD the SCR.
b. Put a brief plan of action in the analysis/solution field.
c. Date it and put your full name beside it.
d. Ctl-z to exit the field.
Comment: After this is done, we get credit for STARTING A TEST.
2. When you have completed an SCR & inspected it:
a. MOD the SCR.
b. Put your final summation of the problem and your final
solution in the analysis/solution field after your initial
analysis. Include relevant TESTIDs and TESTCASES.
c. Date it, and put your full name beside it.
d. Ctl z to exit the field.
e. Hit return at the closure catagory position and select
'fixed/added' or 'nonproblem'.
f. Hit return at the project status position and select 'done'.
g. Email the ASSIGNEE to sign it off.
3. The ASSIGNEE puts an X in the underlined space.
Comment: After this is done, we get credit for ASSIGNEE.
4. The VERIFIER, usually me, will then run INSPECTSCR
which will produce an output i will send you if there
is a problem. If there is no problem, i will sign.
Comment: This gets us credit for VERIFIER.
5. The ORIGINATOR will look at the SCR and sign sometime
in the dim future.
SCR REWORK VS INIT DEV
INITIAL DEV
1. notice a change in a DSS section.
2. fix the tests that test that section.
3. DIF the GENs in ACM until you locate the first
appearance of the change.
4. find the associated FPDOCS SCR that changed the GEN.
5. find the associated FPTEST SCR.
6. put the test changes on the FPTEST SCR.
7. fill in analysis/solution on FPTEST SCR.
SCR REWORK
1. pick an FPTEST SCR.
2. find the associated FPDOCS SCR.
3. DIF all GENs on the FPDOCS SCR to see all changes.
4. fill in proposed analysis/solution on FPTEST SCR.
5. fix the tests that test the changed sections.
6. put the test changes on the FPTEST SCR.
7. fill in final analysis/solution on FPTEST SCR.
SCR REWORK
DEFINITIONS:
-An update is a change in requirements or code causing a change to a tdf or
support file. An update will require the test to be re-executed and the results
configured.
-Rerun is change in code with no tdf or support file effect. A rerun will
generate a new set of results that are configured.
-Reuse is no change in a tdf, support files or results.
PROCESS:
The regression analysis is a key part of the reuse process. The regression
analysis will identify tests that are candidates for reuse. The process
outlined below begins from the point of tests identified as candidates for
reuse.
Test reuse can be done in different ways. Test definition files (tdfs) can be
re-executed to get new results. This is called rerun. Tdfs and results can be
reused from a previous certification configuration. This is called reuse.
If the reused (or rerun) tests can be identified at the time of creation of the
initial development SCR, a split should be created for reuse/rollover. If tests
are found to be reusable (or rerunable) after the initial development SCR has
been created, a split will be created when reusable (or rerunable) tests are
identified. All reusable (and rerunable) tests should be identified at the end
of initial development, before run for score begins. If a test has to be
modified after it has undergone a reuse inspection, the results will be
documented on the SCR where the rework is performed.
All reused tests (and tests to be rerun) will be inspected on a reuse/rollover
SCR. A reuse checklist exists for this inspection. The elements do not need to
be inserted into the new configurations, unless the test will be rerun. If the
test, including results, is reused the only place the information will appear
is on the inspection (.ins) file. The elements will not appear in the working
or certification configurations of the program where the tests are being
executed. If the test is to be rerun, it can be inspected using the
reuse/rollover SCR and inserted into the working configuration.
Some DOs and DON'Ts about what to put on SCRs.
DO put in ACM on the RFS SCR:
VER,RPT,EXC,TCR,TCE for all updated/rerun tests
DO put in ACM on RFS Fixes SCR:
TDF,INC,BD,CT,ADA,etc. for all updated/rerun tests
DON'T put in ACM on the RFS SCR:
TDF,VER,RPT,EXC,TCR,TCE for any reused tests
DO put in ACM on the RFS SCR:
INS for any reused tests
A test may be rerun only to obtain information to conduct structural
coverage analysis. In this case a test from a previous configuration is
executed but only the xinfo and pth files are kept to assist in the structural
coverage analysis. The test and results are reused and do appear on the list of
reused elements in the inspection (.ins) file but do not appear in the working
or certification configurations. The test results are not configured, but are
reused from a previous certification.
Structural coverage can be "reused" in the inspectSCT tool. The tool is being
modified to accept a list of configurations to search to determine coverage.
However, any MD90 test results that are reused in -800 will need to be inserted
into the -800 configuration.
SCR REWORK
Software Test Inspection Checklist (Reuse)
===============================================================================
DISPLAYS SOFTWARE TEST REUSE INSPECTION CHECKLIST
TEST_REUSE_CHECKLIST.TXT VER 001 DEC 13, 1997
===============================================================================
1. Test Analysis
A. Is each test case in the software test applicable and necessary to the
new application?
B. Is the software test, as written, compatible with the new application's
exeuction environment? (i.e., Does the new application have identical
operational characteristics such as inputs, data ranges, update rates,
and partition assignments?)
C. Does the software test contain any test inputs unused in the new
application? Will the behavior be predictable and acceptable if they
are not used?
D. Are any changes required to make the original software test function
in the new application? (This type of change will require a more
extensive inspection depending on the scope of changes.)
E. Are generations of code elements being tested the same for both
configurations (the baseline configuration and the new configuration)?
2. Convention Compliance
A. Does the software test filename comply with naming conventions?
B. Do anchor names comply with anchor naming conventions?
3. Traceability and Configuration Management
A. Has traceability to the requirements been verified?
B. Has traceability to the design information (SDD) been verified (if
applicable)?
C. Is the reused element in the correct ACM configuation?
D. If a variant has been created, does the variant genertion comply with
DDN 019?
E. Does the inspection file contain a reference to the most recent
inspection of the original element?
back
