Página |1

Diego Fernando Gómez Páez

Facultad de Ciencias Físico-Químicas

Escuela de Ingeniería de Petróleos

Mayo de 2010

Numerical Approximation

1. Introduction

Frequently in the use of mathematical modelling the answers are generated with too many
digits, and using all of these digits for more calculations results in complication and often
leads to errors, that’s the reason we must approximate the numbers we’re using until they
become manageable, but this procedure must be done with extremely precaution,
because a wrong approximation is worse than no one.

2. Significant Digits [1]

The number of significant digits in an answer to a calculation will depend on the number
of significant digits in the given data, as discussed in the rules below. Approximate
calculations (order-of-magnitude estimates) always result in answers with only one or two
significant digits.

2.1 When are Digits Significant?

Non-zero digits are always significant. Thus, 22 has two significant digits, and 22.3 has
three significant digits.

With zeroes, the situation is more complicated:

a. Zeroes placed before other digits are not significant; 0.046 has two significant
digits.

b. Zeroes placed between other digits are always significant; 4009 kg has four
significant digits.

c. Zeroes placed after other digits but behind a decimal point are significant; 7.90 has
three significant digits.

Universidad Industrial de Santander

 Página |2

d. Zeroes at the end of a number are significant only if they are behind a decimal
point as in (c). Otherwise, it is impossible to tell if they are significant. For
example, in the number 8200, it is not clear if the zeroes are significant or not. The
number of significant digits in 8200 is at least two, but could be three or four. To
avoid uncertainty, use scientific notation to place significant zeroes behind a
decimal point:

8.200 ´ 103 has four significant digits

8.20 ´ 103 has three significant digits
8.2´ 103 has two significant digits
2.2 Significant Digits in Multiplication, Division, Trig. functions, etc.

In a calculation involving multiplication, division, trigonometric functions, etc., the number

of significant digits in an answer should equal the least number of significant digits in any
one of the numbers being multiplied, divided etc.

Thus in evaluating sin(kx), where k = 0.097 m-1 (two significant digits) and x = 4.73 m
(three significant digits), the answer should have two significant digits.

Note that whole numbers have essentially an unlimited number of significant digits. As an
example, if a hair dryer uses 1.2 kW of power, then 2 identical hairdryers use 2.4 kW:

1.2 kW {2 sig. dig.} ´ 2 {unlimited sig. dig.} = 2.4 kW {2 sig. dig.}

2.3 Significant Digits in Addition and Subtraction

When quantities are being added or subtracted, the number of decimal places (not
significant digits) in the answer should be the same as the least number of decimal places
in any of the numbers being added or subtracted.

Example:

5.67 (two decimal places)

1.1 (one decimal place)

0.9378 + (four decimal places)

7.7 (one decimal place)

2.4 Keep One Extra Digit in Intermediate Answers

When doing multi-step calculations, keep at least one more significant digit in intermediate
results than needed in your final answer.

Universidad Industrial de Santander

 Página |3

For instance, if a final answer requires two significant digits, then carry at least three
significant digits in calculations. If you round-off all your intermediate answers to only two
digits, you are discarding the information contained in the third digit, and as a result the
second digit in your final answer might be incorrect. (This phenomenon is known as
"round-off error.")

2.5 The Two Greatest Sins Regarding Significant Digits

Writing more digits in an answer (intermediate or final) than justified by the

number of digits in the data.

Rounding-off, say, to two digits in an intermediate answer, and then writing three
digits in the final answer.

3. Rounding [2]

Rounding a numerical value means replacing it by another value that is approximately

equal but has a shorter, simpler, or more explicit representation; for example, replacing
US$ 23.4476 by US$ 23.45 or the fraction 312/937 by 1/3.

Rounding rules are not perfect, they have some problems, and they may fail. But being
careful they can be very helpful. In the next tables you can find rounding rules organized
by their accuracy.

3.1 Rounding Numbers[3]

Rule One. Determine what your rounding digit is and look to the right side of it. If
the digit is 0, 1, 2, 3, or 4 do not change the rounding digit. All digits that are on
the right hand side of the requested rounding digit will become 0.
Rule Two. Determine what your rounding digit is and look to the right of it. If the
digit is 5, 6, 7, 8, or 9, your rounding digit rounds up by one number. All digits that
are on the right hand side of the requested rounding digit will become 0.

3.2 Rounding with decimals[3]

When rounding numbers involving decimals, there are 2 rules to remember:

Rule One: Determine what your rounding digit is and look to the right side of it. If
that digit is 4, 3, 2, or 1, simply drop all digits to the right of it.
Rule Two: Determine what your rounding digit is and look to the right side of it. If
that digit is 5, 6, 7, 8, or 9 add one to the rounding digit and drop all digits to the
right of it.

Universidad Industrial de Santander

 Página |4

3.3 Standard Rounding Rules

Operation Rounding Rule Accuracy Failure Modes

Addition and The number of decimal 100% for None for a series of 2
Subtraction places in the result equals a series of to 9 numbers
the smallest number of up to 9 Can predict too many
decimal places in the input numbers digits for a series of 10
numbers. or more numbers

Multiplication The number of significant 31.0% Predicts 1 less digit

figures in the result equals than justified 68.8% of
the smallest number of the time
significant figures in the Predicts 1 more digit
input numbers. than justified 0.2% of
the time

Division (Same as Multiplication) 45.3% Predicts 1 less digit

than justified 54.4% of
the time
Predicts 1 more digit
than justified 0.3% of
the time

Common The number of significant 57.18% Predicts 1 more digit

Logarithm decimal places in the than justified 42.82%
result equals one plus the of the time
number of significant
digits in the input.
Natural The number of significant 97.00% Predicts 1 more digit
Logarithm decimal places in the than justified 3.00% of
result equals the number the time
of significant digits in the
input.
Common The number of significant 0.00% Predicts 2 fewer digits
Exponential digits in the result equals than justified 61.46%
the number of significant of the time
decimal places in the input Predicts 1 less digit
minus one. than justified 38.54%
of the time

Natural The number of significant 2.11% Predicts 1 less digit

Exponential digits in the result equals than justified 97.89%
the number of significant of the time
decimal places in the
input.

Universidad Industrial de Santander

 Página |5

Source: http://www.angelfire.com/oh/cmulliss/standard_rounding_rules_summary.htm

3.4 Recommended Rounding Rules

Operation Rounding Rule Accuracy Failure Modes

Addition and The number of decimal 100% for None for a series of 2
Subtraction places in the result equals a series of to 9 numbers
the smallest number of up to 9
decimal places in the input numbers Can predict too many
numbers. digits for a series of 10
or more numbers

Multiplication The number of significant 68.8% Predicts 1 or 2 more

figures in the result equals digits than justified
one plus the smallest 31.2% of the time
number of significant
figures in the input Very rarely (< 1%)
numbers. predicts 2 digits too
many

Division (Same as Multiplication) 54.7% Predicts 1 or 2 more

digits than justified
45.3% of the time

Very rarely (< 1%)

predicts 2 digits too
many

Common The number of significant 57.18% Predicts 1 more digit

Logarithm decimal places in the than justified 42.82%
result equals one plus the of the time
number of significant
digits in the input.

Natural The number of significant 97.00% Predicts 1 more digit

Logarithm decimal places in the than justified 3.00% of
result equals the number the time
of significant digits in the

Universidad Industrial de Santander

 Página |6

input.

Common The number of significant 61.46% Predicts 1 more digit

Exponential digits in the result equals than justified 38.54%
one plus the number of of the time
significant decimal places
in the input.

Natural (Same as Common 98.13% Predicts 1 more digit

Exponential Exponential) than justified 1.87% of
the time

Source: http://www.angelfire.com/oh/cmulliss/recommended_rounding_rules_summary.htm

3.5 High Accuracy Rounding Rules

Operation Rounding Rule Accuracy Failure Modes

Division The “Portuguese Rule”: 68.99% Predicts 1 less digit

than justified 16.95%
Count numbers whose of the time
leading digit is 5 or Predicts 1 more digit
greater as having an extra than justified 14.06%
significant digit and then of the time
apply the standard rule for
division.
Common The Chen-Lee-Mulliss Rule 90.21% Predicts 1 more digit
Logarithm for Common Logarithms: than justified 9.79% of
the time
If the leading digit of the
input is 4 or greater then
the number of significant
decimal places in the
result equals one plus
the number of significant
digits in the input.

Otherwise, the number of

significant decimal places
in the result equals the
number of significant

Universidad Industrial de Santander

 Página |7

digits in the input.

Common The Refined Chen-Lee- 90.16% Predicts 1 more digit

Exponential Mulliss Rule for Common than justified 9.84% of
Exponentials: the time

If the leading digit of the

result is 4 or smaller then
the number of significant
digits in the result equals
one plus the number of
significant decimal places
in the input.

Otherwise, the number of

significant digits in the
result equals the number
of significant decimal
places in the input.

Source: http://www.angelfire.com/oh/cmulliss/haccurate_rounding_rules_summary.htm

4. Accuracy and Precision [4]

Accuracy and precision both concern the quality of a measure. However, the terms have
different meanings and should not be used as substitutes for one another.

4.1 Precision

Precision is the reproducibility of multiple measurements and is usually described by the

standard deviation, standard error, or confidence interval.[5]

Precision depends on the unit used to obtain a measure. The smaller the unit, the more
precise the measure. Consider measures of time, such as 12 seconds and 12 days. A
measurement of 12 seconds implies a time between11.5 and 12.5 seconds. This
measurement is precise to the nearest second, with a maximum potential error of 0.5
seconds. A time of 12 days is far less precise. Twelve days suggests a time between 11.5
and 12.5 days, yielding a potential error of 0.5 days, or 43,200 seconds! Because the
potential error is greater, the measure is less precise. Thus, as the length of the unit
increases, the measure becomes less precise.

The number of decimal places in a measurement also affects precision. A time of 12.1
seconds is more precise than a time of 12 seconds; it implies a measure precise to the
nearest tenth of a second. The potential error in12.1 seconds is 0.05 seconds, compared
with a potential error of 0.5 seconds with a measure of 12 seconds.

Universidad Industrial de Santander

 Página |8

Although students learn that adding zeros after a decimal point is acceptable, doing so can
be misleading. The measures of 12 seconds and 12.0 seconds imply a difference in
precision. The first figure is measured to the nearest second—a potential error of 0.5
seconds. The second figure is measured to the nearest tenth—a potential error of 0.05
seconds. Therefore, a measure of 12.0 seconds is more precise than a measure of 12
seconds.

Differing levels of precision can cause problems with arithmetic operations. Suppose one
wishes to add 12.1 seconds and 14.47 seconds. The sum, 26.57 seconds, is misleading.
The first time is between 12.05 seconds and12.15 seconds, whereas the second is
between 14.465 and 14.475 seconds. Consequently, the sum is between 26.515 seconds
and 26.625 seconds. A report of 26.57 seconds suggests more precision than the actual
result possesses.

The generally accepted practice is to report a sum or difference to the same precision as
the least precise measure. Thus, the result in the preceding example should be reported
to the nearest tenth of a second; that is, rounding the sum to 26.6 seconds. Even so, the
result may not be as precise as is thought. If the total is actually closer to 26.515 seconds,
the sum to the nearest tenth is 26.5 seconds. Nevertheless, this practice usually provides
acceptable results.

Measurements in industrial settings such as a rubber manufacturing plant must be both

accurate and precise. Here a technician is measuring tire pressure.

Multiplying and dividing measures can create a different problem. Suppose one wishes to
calculate the area of a rectangle that measures 3.7 centimeters (cm) by 5.6 cm.
Multiplication yields an area of 20.72 square centimeters. However, because the first
measure is between 3.65 and 3.75 cm, and the second measure is between 5.55 and 5.65
cm, the area is somewhere between 20.2575 and 21.1875 square centimeters. Reporting
the result to the nearest hundredth of a square centimeter is misleading. The accepted
practice is to report the result using the fewest number of significant digits in the
original measures. Since both 3.7 and 5.6 have two significant digits, the result is rounded
to two significant digits and an area of 21 square centimeters is reported. Again, while the
result may not even be this precise, this practice normally produces acceptable results.

4.2 Accuracy

The accuracy of an analytical measurement is how close a result comes to the true value.
Determining the accuracy of a measurement usually requires calibration of the analytical
method with a known standard. [5]

Rather than the absolute error to which precision refers, accuracy refers to the relative
error in a measure. For example, if one makes a mistake by 5 centimeters in measuring
two objects that are actually 100 and 1,000 cm, respectively, the second measure is more

Universidad Industrial de Santander

 Página |9

accurate than the first. The first has an error of 5 percent (5 cm out of 100 cm), whereas
the second has an error of only 0.5 percent (5 cm out of 1,000 cm).

How Are Precision and Accuracy Different? To illustrate the difference between
precision and accuracy, suppose that a tape measure is used to measure the
circumference of two circles—one small and the other large. Suppose a result of 15 cm for
the first circle and 201 cm for the second circle are obtained. The two measures are
equally precise; both are measures to the nearest centimeter. However, their accuracy
may be quite different. Suppose the measurements are both about 0.3 cm too small. The
relative errors for these measures are 0.3 cm out of 15.3 cm (about 1.96 percent) and0.3
cm out of 201.3 cm (about 0.149 percent). The second measurement is more accurate
because the error is smaller when compared with the actual measurement. Consequently,
for any specific measuring tool, one can be equally precise with the measures. But
accuracy is often greater with larger objects than with smaller ones.

Confusion can arise when using these terms. The tools one uses affect both the precision
and accuracy of one's measurements. Measuring with a millimeter tape allows greater
precision than measuring with an inch tape. Because the error using the millimeter tape
should be less than the inch tape, accuracy also improves; the error compared with the
actual length is likely to be smaller. Despite this possible confusion and the similarity of
the ideas, it is important that the distinction between precision and accuracy be
understood.

Source: http://dynamo.uoregon.edu/phys290/pub/Material/ErrorPropagationGuide/accuracy-precision.png

Universidad Industrial de Santander

 P á g i n a | 10

References

[1] http://www.physics.uoguelph.ca/tutorials/sig_fig/SIG_dig.htm

[2] http://www.angelfire.com/oh/cmulliss/

[3] http://math.about.com/od/arithmetic/a/Rounding.htm

[4] http://www.bookrags.com/research/accuracy-and-precision-mmat-01/

[5] http://elchem.kaist.ac.kr/vt/chem-ed/data/acc-prec.htm

Universidad Industrial de Santander