ERROR ANALYSIS
(UNCERTAINTY ANALYSIS)
16.621 Experimental Projects Lab I
1
TOPICS TO BE COVERED
• Why do error analysis?
• If we don’t ever know the true value, how do we estimate the error
in the true value?
• Error propagation in the measurement chain
– How do errors combine? (How do they behave in general?)
– How do we do an end-to-end uncertainty analysis?
– What are ways to mitigate errors?
• A hypothetical dilemma (probably nothing to do with anyone in the
class)
– When should I throw out some data that I don’t like?
– Answer: NEVER, but there are reasons to throw out data
• Backup slides: an example of an immense amount of money and
effort directed at error analysis and mitigation - jet engine testing
2
ERROR AND UNCERTAINTY
• In engineering the word “error”, when used to describe an aspect of
measurement does not necessarily carry the connotation of mistake
or blunder (although it can!)
• Error in a measurement means the inevitable uncertainty that
attends all measurements
• We cannot avoid errors in this sense
• We can ensure that they are as small as reasonably possible and
that we have a reliable estimate of how small they are
[Adapted from Taylor, J. R, An Introduction to Error Analysis;
The Study of Uncertainties in Physical Measurements]
3
USES OF UNCERTAINTY ANALYSIS (I)
• Assess experimental procedure including identification of
potential difficulties
– Definition of necessary steps
– Gaps
• Advise what procedures need to be put in place for measurement
• Identify instruments and procedures that control accuracy and
precision
– Usually one, or at most a small number, out of the large set of
possibilities
• Inform us when experiment cannot meet desired accuracy
4
USES OF UNCERTAINTY ANALYSIS (II)
• Provide the only known basis for deciding whether:
– Data agrees with theory
– Tests from different facilities (jet engine performance) agree
– Hypothesis has been appropriately assessed (resolved)
– Phenomena measured are real
• Provide basis for defining whether a closure check has been
achieved
– Is continuity satisfied (does the same amount of mass go in
as goes out?)
– Is energy conserved?
• Provide an integrated grasp of how to conduct the experiment
5
[Adapted from Kline, S. J., 1985, “The Purposes of Uncertainty
Analysis”, ASME J. Fluids Engineering, pp. 153-160]
UNCERTAINTY ESTIMATES AND HYPOTHESIS ASSESSMENT
600
500
400
Distance [cm]
300
200
100
0
0 20 40 60 80 100 120
Mass [g]
600 600
500 500
400 400
Distance [cm]
Distance [cm]
300 300
200 200
100 100
0 0
0 20 40 60 80 100 120 0 20 40 60 80 100 120
Mass [g] Mass [g]
6
HOW DO WE DEAL WITH NOT KNOWING
THE TRUE VALUE?
• In “all” real situations we don’t know the true value we are
looking for
• We need to decide how to determine the best
representation of this from our measurements
• We need to decide what the uncertainty is in our best
representation
7
AN IMPLICATION OF NOT KNOWING THE TRUE VALUE
• We easily divided errors into precision (bias) errors and random
errors when we knew what the value was
• The target practice picture in the next slide is an example
• How about if we don’t know the true value? Can we, by looking at
the data in the slide after this, say that there are bias errors?
• How do we know if bias errors exist or not?
8
A TEAM EXERCISE
• List the variables you need to determine in order to carry out your
hypothesis assessment
• What uncertainties do you foresee? (Qualitative description)
• Are you more concerned about bias errors or random errors?
• What level of uncertainty in the final result do you need to assess
your hypothesis in a rigorous manner?
• Can you make an estimate of the level of the uncertainty in the final
result?
– If so, what is it?
– If not, what additional information do you need to do this? 11
HOW DO WE COMBINE ERRORS?
• Suppose we measure quantity X with an error of dx and quantity Y
with an error of dy
• What is the error in quantity Z if:
• Z = AX where A is a numerical constant such as π?
• Z = X + Y?
• Z = X - Y?
• Z = XY?
• Z = X/Y?
• Z is a general function of many quantities?
12
ERRORS IN THE FINAL QUANTITY
• Z=X+Y
• Linear combination
– Z + dz = X + dx + Y + dy
– Error in Z is dz = dx + dy BUT this is worst case
• For random errors we could have
– dz = dx − dy
or dy − dx
– These errors are much smaller
• In general if different errors are not correlated, are
independent, the way to combine them is
dz = dx2 + dy2
13
• This is true for random and bias errors
THE CASE OF Z = X - Y
• Suppose Z = X - Y is a number much smaller than X or Y
dz = dx2 + dy2
dx dy
• Say = =ε (say 2%)
X Y
dz 2 dx dx
• = may be much larger than
Z X−Y X
• MESSAGE ==> Avoid taking the difference of two numbers of
comparable size
14
ESTIMATES FOR THE TRUE VALUE AND THE ERROR
• Is there a “best” estimate of the true value of a quantity?
• How do I find it?
• How do I estimate the random error?
• How do I estimate the bias error?
15
SOME “RULES” FOR ESTIMATING
RANDOM ERRORS AND TRUE VALUE
• An internal estimate can be given by repeat measurements
• Random error is generally of same size as standard deviation (root
mean square deviation) of measurements
• Mean of repeat measurements is best estimate of true value
• Standard deviation of the mean (random error) is smaller than
standard deviation of a single measurement by
1 Number of measurements
• To increase precision by 10, you need 100 measurements
16
GENERAL RULE FOR COMBINATION OF ERRORS
• If Z = F (X1, X2, X3, X4) is quantity we want
• The error in Z, dz, is given by our rule from before
• So, if the error F due to X1 can be estimated as
Error in X1
∂F
dF1 = dx and so on
∂X1 1
Influence coeff.
2 2 2
∂F 2 ∂F ∂F
dz = dx1 + dx22 + dx 2
• ∂X1 ∂X 2 ∂X n
n
• The important consequence of this is that generally one or few of
these factors is the main player and others can be ignored
17
DISTRIBUTION OF RANDOM ERRORS
• A measurement subject to many small random errors will be
distributed “normally”
• Normal distribution is a Gaussian
• If x is a given measurement and X is the true value
1 −(x−X2 ) 2σ 2
Gaussian or normal distribution = e
σ 2π
• σ is the standard deviation
18
A REVELATION
• The universal gas constant is
accepted R = 8.31451 ±0.00007 J/mol K
• This is not a true value but can be “accepted” as one
20
ONE ADDITIONAL ASPECT OF COMBINING ERRORS
• We have identified two different types of errors, bias (systematic)
and random
– Random errors can be assessed by repetition of measurements
– Bias errors cannot; these need to be estimated using external
information (mfrs. specs., your knowledge)
• How should the two types of errors be combined?
– One practice is to treat each separately using our rule, and then
report the two separately at the end
– One other practice is to combine them as “errors”
• Either seems acceptable, as long as you show that you are going
to deal (have dealt) with both
21
REPORTING OF MEASUREMENTS
• Experimental uncertainties should almost always be
rounded to one significant figure
• The last significant figure in any stated answer should
usually be of the same order of magnitude (in the same
decimal position) as the uncertainty
22
[from Taylor, J., An Introduction to Error Analysis]
COMMENTS ON REJECTION OF DATA
• Should you reject (delete) data?
• Sometimes on measurement appears to disagree greatly with all
others. How do we decide:
– Is this significant?
– Is this a mistake?
• One criteria (Chauvenaut’s criteria) is as follows
– Suppose that errors are normally distributed
– If measurement is more than M standard deviations (say 3),
probability is < 0.003 that measurement should occur
– Is this improbable enough to throw out measurement?
• The decision of “ridiculous improbability” [Taylor, 1997] is up to
the investigator, but it allows the reader to understand the basis
for the decision
– If beyond this range, delete the data 23
A CAVEAT ON REJECTION OF DATA
• If more than one measurement is different, it may be that
something is really happening that has not been envisioned, e.g.,
discovery of radon
• You may not be controlling all the variables that you need to
• Bottom line: Rigorous uncertainty analysis can give rationale to
decide what data to pay attention to
26
SUMMARY
• Both the number and the fidelity of the number are important in a
measurement
• We considered two types of uncertainties, bias (or systematic
errors) and random errors
• Uncertainty analysis addresses fidelity and is used in different
phases of an experiment, from initial planning to final reporting
– Attention is needed to ensure uncertainties do not invalidate
your efforts
• In propagating uncorrelated errors from individual measurement to
final result, use the square root of the sums of the squares of the
errors
– There are generally only a few main contributors (sometimes
one) to the overall uncertainty which need to be addressed
• Uncertainty analysis is a critical part of “real world” engineering
projects
27
SOME REFERENCES I HAVE FOUND USEFUL
• Baird, D. C., 1962, Experimentation: An Introduction to Measurement
Theory and Experiment, Prentice-Hall, Englewood Cliffs, NJ
• Bevington, P. R, and Robinson, D. K., 1992, Data Reduction and
Error Analysis for the Physical Sciences, McGraw-Hill, New York, NY
• Lyons, L., 1991, A Practical Guide to Data Analysis for Physical
Science Students, Cambridge University Press, Cambridge, UK
• Rabinowicz, E, 1970, An Introduction to Experimentation, Addison-
Wesley, Reading, MA
• Taylor, J. R., 1997, An Introduction to Error Analysis, University
Science Books, Sauselito, CA
28
BACKUP EXAMPLE: MEASUREMENT OF JET ENGINE
PEFORMANCE
• We want to measure Thrust, Airflow, and Thrust Specific Fuel
Consumption (TSFC)
– Engine program can be $1B or more, take three years or more
– Engine companies give guarantees in terms of fuel burn
– Engine thrust needs to be correct or aircraft can’t take off in
the required length
– Airflow fundamental in diagnosing engine performance
– These are basic and essential measures
• How do we measure thrust?
• How do we measure airflow?
• How do we measure fuel flow?
29
THRUST STANDS
• In practice, thrust is measured with load cells
• The engines, however, are often part of a complex test facility
and are connected to upstream ducting
• There are thus certain systematic errors which need to be
accounted for
• The level of uncertainty in the answer is desired to be less than
one per cent
• There are a lot of corrections to be made to the raw data
(measured load) to give the thrust
32
TEST STAND-TO-TEST STAND DIFFERENCES
• Want to have a consistent view of engine performance no matter
who quotes the numbers
• This means that different test stands must be compared to see
the differences
• Again, this is a major exercise involving the running of a jet
engine in different locations under specified conditions
• The next slide shows the level of differences in the
measurements
35
