4.
5
TESTING THROUGHOUT SYSTEM
DEVELOPMENT
Testing and Evaluation
• Testing and evaluation are not separate
functions from design but rather are inherent
parts of design.
• Design is a closed loop process in which T & E
constitutes the feedback that adjusts the
result to the requirements that it is intended
to meet.
Unknowns
• In any new system development project, there are a
great many unknowns that need to be resolved in the
course of producing a successful product.
• For each significant departure from established
practice, the result cannot be predicted with
assurance.
• The project cost depends on a host of factors, none of
them known precisely.
• The resolution of interface incompatibilities often
involves design adjustment on both sides of the
interface, which frequently leads to unexpected and
sometimes major technical difficulties.
Unknowns
• An essential task of systems engineering is to
guide the development of the system so that
the unknowns are turned into knowns as early
in the process as possible.
• Any surprises occurring late in the program
can prove to be many times more costly than
those encountered in its early phases.
Unknowns
• Many unknowns are evident at the beginning, and may be called
known unknowns.
• These are identified early as potential problem areas and are
therefore singled out for examination and resolution.
• Usually, this can be accomplished through a series of critical
experiments involving simulations and/or experimental hardware
and software.
• However, many other problem areas are only identified later when
they are discovered during system development.
• These unanticipated problems are often identified as unknown
unknowns to distinguish them from the group of known unknowns
that were recognized at the outset and dealt with before they
could seriously impact the overall development process.
Transforming the Unknown into the Known
• The existence of unknown unknowns makes the task far more
difficult.
• It requires searching for hidden traps in the favored places of
technical problems.
• It is the task of the systems engineer to lead this search based on
experience gathered during previous system developments and
supported by a high degree of technical insight.
• Since every unknown poses an uncertainty in the accomplishment
of the final objective, it represents a potential risk.
• In fact, unknowns present the principal risks in any development
program. Hence, the task of risk assessment and integration is one
and the same as that of identifying unknowns and resolving them.
Transforming the Unknown into the Known
• The tools for resolving unknowns are analysis,
simulation, and test, these being the means for
discovering and quantifying critical system
characteristics.
• In using an approach never attempted before under
same circumstances, there are a number of unknowns
regarding how the new design when implemented
will perform.
• In such cases, the process of testing will tell us about
unanticipated difficulties due to unknown factors.
Which may require significant design changes or even
abandonment of the approach.
Transforming the Unknown into the Known
• When a new design approach is undertaken, it is unwise to wait
until the design is fully implemented before determining whether
or not the approach is sound. Instead, testing should first be done
on a theoretical or experimental model of the design element,
which can be created quickly and at a minimum cost.
• In doing so, a judgment must be made as to the balance between
the potential benefit of a greater degree of realism of the model
and the time and cost of achieving it. This is very often a system
level rather than a component level decision, especially if the
performance of the element can have a system impact.
• If the unknowns are largely in the functional behavior of the
element, then a computational model or a simulation is indicated.
If, on the other hand, the unknowns are concerned with the
material aspects, an experimental model is required.
Systems Engineering Approach to Testing
• The systems engineering approach to testing can be
illustrated by comparing the respective views of testing by
the design engineer, the test engineer, and the systems
engineer.
• The design engineer wants to be sure that a component
passes the test, wanting to know, “ Is it OK? ” The test
engineer wants to know that the test is thorough so as to
be sure the component is stressed enough.
• The systems engineer wants to be sure to find and identify
all deficiencies present in the component.
• If the component fails a test, the systems engineer wants to
know why, so that there will be a basis for devising changes
that will eliminate the deficiency
Systems Engineering Approach to Testing
• the emphasis of systems engineering is not only
on the test conditions but also on the acquisition
of data showing exactly how the various parts of
the system did or did not perform.
• Furthermore, the acquisition of data itself is not
enough; it is necessary to have in hand
procedures for analyzing the data.
• These are often complicated and require
sophisticated analytical techniques, which must
be planned in advance.
Systems Engineering Approach to Testing
• It also follows that a systems engineer must be an
active participant in the formulation of the test
procedures and choice of instrumentation.
• In fact, the prime initiative for developing the test
plan should lie with systems engineering, working
in close cooperation with test engineering.
• To the systems engineer, a test is like an
experiment is to a scientist, namely, a means of
acquiring critical data on the behavior of the
system under controlled circumstances.
System T & E
• The most intensive use of testing in the
system life cycle takes place in the last phase
of system development, integration and
evaluation
