Testing
Overview and Black-Box Testing Techniques
Software testing is
an important technique for assessing the quality of a software product.
In this chapter, we
will explain the following:
• the
basics of software testing, a verification and validation practice, throughout
the entire software
development lifecycle
• the
two basic techniques of software testing, black-box testing and white-box
testing
• six
types of testing that involve both black- and white-box techniques.
• strategies
for writing fewer test cases and still finding as many faults as possible
• using
a template for writing repeatable, defined test cases
1
Introduction to Testing
Software
testing is the process of analyzing a software item to detect the
differences
between
existing and required conditions (that is, bugs) and to evaluate the features
of
the
software item. Software testing is an activity that should be done
throughout
the whole development
process .
Software testing is
one of the “verification and validation,” or V&V, software practices.
Some other V&V
practices, such as inspections and pair programming, will be discussed
throughout this book.
Verification (the first V) is the process of evaluating a system or
component
to determine whether the products of a given development phase satisfy the
conditions
imposed at the start of that phase. Verification
activities include testing
and reviews. For
example, in the software for the Monopoly game, we can verify that
two players cannot
own the same house. Validation is the process of evaluating a system
or
component during or at the end of the development process to determine whether
it
satisfies
specified requirements. At the end of development validation
(the second V)
activities are used
to evaluate whether the features that have been built into the software
satisfy the customer
requirements and are traceable to customer requirements. For
example, we validate
that when a player lands on “Free Parking,” they get all the money
that was collected.
Boehm has informally defined verification and validation as
follows:
Verification:
Are we building the product right?
Through verification,
we make sure the product behaves the way we want it to. For
example, on the left
in Figure 1, there was a problem because the specification said that
players should
collect $200 if they land on or pass Go. Apparently a programmer
implemented this
requirement as if the player had to pass Go to collect. A test case in
which the player
landed on Go revealed this error.
Validation: Are we building the right product?
Through validation,
we check to make sure that somewhere in the process a mistake
hasn’t been made such
that the product build is not what the customer asked for;
validation always
involves comparison against requirements. For example, on the right
Verification
Are we building the
product right?
Validation
Are we building the right product?
Both of Boehm’s
informal definitions use the term “right.” But what is “right”? In
software we need to
have some kind of standard or specification to measure against so
that we can identify
correct results from incorrect results. Let’s think about how the
incorrect results
might originate. The following terms with their associated definitions are helpful for
understanding these concepts:
o Mistake
– a human action that produces an incorrect result.
o Fault
[or Defect] – an incorrect step, process, or data definition in a program.
o Failure
– the inability of a system or component to perform its required
function
within the specified
performance requirement.
o Error
– the difference between a computed, observed, or measured value
or
condition and the true,
specified, or theoretically correct value or condition.
o Specification
– a document that specifies in a complete, precise, verifiable
manner, the
requirements, design, behavior, or other characteristic of a system or
component, and often
the procedures for determining whether these provisions
have been satisfied.
A mistake committed
by a person becomes a fault (or defect) in a software artifact, such
as the specification,
design, or code. This fault, unless caught, propagates as a defect in
the executable code.
When a defective piece of code is executed, the fault may become a
visible anomaly (a
variance from the specification or desired behavior) and a failure is
“I know this game has
money and
players and “Go” –
but this is not the
game I wanted.”
“I landed on “Go” but
didn’t get my
$200!”
observed. Otherwise,
the fault remains latent. Testing can reveal failures, but it is the
faults that must be
found and removed ; finding a fault (the cause of a failure) can be
time consuming and
unpredictable. Error is a measure of just how incorrect the results
are.
A purpose of testing is
to cause failures in
order to make faults visible so that the faults can be fixed and not
be delivered in the
code that goes to customers. Another purpose of testing is to assess
the overall quality
level of the code. For example, a test team may determine a project
with too many
high-severity defects should be sent back to development for additional
work to improve the
quality before the testing effort should continue. Or, the
management may have a
policy that no product can ship if testing is continuing to reveal
high-severity
defects.
Compared with
specification or
desired
use/functionality
A programmer makes a
mistake.
The mistake manifests
itself as a fault1 [or
defect] in the
program.
A failure is
observed if
the fault [or defect]
is
made visible. Other
faults remain latent
in
the code until they
are
observed (if ever).
1.1 The Economics of Software Testing
In software
development, there are costs associated with testing our programs. We need
to write out test
plan and our test cases, we need to set up the proper equipment, we need
to systematically
execute the test cases, we need to follow up on problems that are
identified, and we
need to remove most of the faults we find. Actually, sometimes we
can find low-priority
faults in our code and decide that it is too expensive to fix the fault
1 The
IEEE does not define defect however, the term defect is considered to be
synonymous with fault.
because of the need
to redesign, recode, or otherwise remove the fault. These faults can
remain latent in the
product through a follow-on release or perhaps forever.
For faults that are
not discovered and removed before the software has been shipped,
there are costs. Some
of these costs are monetary, and some could be significant in less
tangible ways.
Customers can lose faith in our business and can get very angry. They can
also lose a great
deal of money if their system goes down because of our defects. (Think
of the effect on a
grocery store that can’t check out the shoppers because of its “down”
point-of-sale system.)
And, software development organizations have to spend a great
deal of money to
obtain specific information about customer problems and to find and fix
the cause of their
failures. Sometimes, programmers have to travel to customer locations
to work directly on
the problem. These trips are costly to the development organization,
and the customers
might not be overly cheerful to work with when the programmer
arrives. When we
think about how expensive it is to test, we must also consider how
expensive it is to
not test – including these intangible costs as well as the more obvious
direct costs.
We also need to
consider the relative risk associated with a failure depending upon the
type of project we
work on. Quality is much more important for safety- or missioncritical
software, like
aviation software, than it is for video games. Therefore, when we
balance the cost of
testing versus the cost of software failures, we will test aviation
software more than we
will test video games. As a matter of fact, safety-critical software
can spend as much as
three to five times as much on testing as all other software
engineering steps
combined !
To minimize the costs
associated with testing and with software failures, a goal of testing
must be to uncover as
many defects as possible with as little testing as possible. In other
words, we want to
write test cases that have a high likelihood of uncovering the faults
that are the most
likely to be observed as a failure in normal use. It is simply impossible
to test every
possible input-output combination of the system; there are simply too many
permutations and
combinations. As testers, we need to consider the economics of testing
and strive to write
test cases that will uncover as many faults in as few test cases as
possible. In this
chapter, we provide you with disciplined strategies for creating efficient
sets of test cases –
those that will find more faults with less effort and time.
1.2 The
Basics of Software Testing
There are two basic
classes of software testing, black box testing and white box testing.
For now, you just
need to understand the very basic difference between the two classes,
clarified by the
definitions below :
o Black
box testing (also called functional testing) is testing that ignores the
internal
mechanism of a system or component and focuses solely on the outputs
generated
in response to selected inputs and execution conditions.
o White
box testing (also called structural testing and glass box testing) is testing
that
takes into account the internal mechanism of a system or component.
The classes of testing are denoted by colors to depict the opacity of the testers of the code.
With black box
testing, the software tester does not (or should not) have access to the
source code itself.
The code is considered to be a “big black box” to the tester who can’t
see inside the box.
The tester knows only that information can be input into to the black
box, and the black
box will send something back out. Based on the requirements
knowledge, the tester
knows what to expect the black box to send out and tests to make
sure the black box
sends out what it’s supposed to send out. Alternatively, white box
testing
focuses on the internal structure of the software code. The white
box tester (most
often the developer
of the code) knows what the code looks like and writes test cases by
executing methods
with certain parameters. In the language of V&V, black box testing is
often used for
validation (are we building the right software?) and white box testing is
often used for
verification (are we building the software right?). This chapter focuses on
black box testing.
All software testing
is done with executable code. To do so, it might be necessary to
create scaffolding
code. Scaffolding is defined as computer programs and data files
built
to
support software development and testing but not intended to be included in the
final
product
. Scaffolding code is code that simulates the functions of
components that
don’t exist yet and
allow the program to execute . Scaffolding code involves the
creation of stubs and
test drivers. Stubs are modules that simulate components that aren’t
written yet, formally
defined as a computer program statement substituting for the body
of a
software module that is or will be defined elsewhere .
For example, you might
write a skeleton of a
method with just the method signature and a hard-coded but valid
return value. Test
drivers are defined as a software module used to involve a module
under
test and often, provide test inputs, controls, and monitor execution and report
test
results
. Test drivers simulate the calling components (e.g.
hard-coded method calls)
and perhaps the
entire environment under which the component is to be tested .
Another concept is
mock objects. Mock objects are temporary substitutes for domain
code that emulates
the real code. For example, if the program is to interface with a
database, you might
not want to wait for the database to be fully designed and created
before you write and
test a partial program. You can create a mock object of the database
that the program can
use temporarily. The interface of the mock object and the real
object would be the
same. The implementation of the object would mature from a dummy implementation
to an actual database.
