Mathematical

processing of
experimental data

Basic concepts and rules for carrying

a mathematical study
of data obtained under experimental
conditions
Basic terms
Performing laboratory operations is associated with any measurement
of physical quantities.
Measuring is a process of comparison of a measurand with
homogeneous it the quantity accepted for a unit of measure. Owing to
an irregularity of our sense organs and measuring devices are carried out
with a restricted degree of accuracy, i.e. value of a measurand differs from
the true.
The device degree of accuracy is understood as that least part of
a unit of measure to which with confidence in correctness of effect
measuring of any quantity (for example can be executed, the degree of
accuracy of a school ruler is equal 1 mm).
The result of the measuring of the investigated quantity is
the determination of its value using an instrument and mathematical
calculations. This value is called a measurement.

Measurements

Direct measurements Indirect measurements

- numerical values of the - the values of the required value,
required quantity obtained which depend on the measurements
by direct comparison with of other quantities associated with
the unit of measurement this value by mathematical functions.
(by using a measuring In other words, indirect
device) measurements are values obtained as
For example, length, time a result of mathematical calculations.
value, sphere For example,
diameter, etc. the volume of a solid body (depends
on the direct geometric parameters of
the body), the speed of movement
(depends on distance and time), the
viscosity of the liquid, etc.
 Under experimental conditions, it is almost impossible
to obtain a perfectly accurate, only correct result due
to the occurrence of so-called ERRORS

Systematic errors
- errors that preserve the magnitude and sign from measurement to
measurement. They depend on the inaccuracy of the device,
the unsuccessfully chosen measurement method, etc. Since systematic
errors are constant, they cannot be eliminated, but these errors can be
identified and taken into account when processing data.

Random errors
- errors, whose the magnitude and sign unpredictably change from
measurement to measurement. They are a consequence of the imperfection
of our senses, the action of factors whose influence cannot be taken into
account, etc.
• It is impossible to eliminate them, but they obey statistical laws, they
can be calculated using methods of mathematical statistics. The value of
the random error decreases significantly with an increase in the number of
measurements (n ) due to a special statistical parameter t ,n - the Student’s
coefficient (t-criterion, two-sample unpaired t-criterion).
• A fragment of the Student's table with the condition 𝛂 = 𝟎.𝟗𝟓 (95%
confidence of the obtained result) is given below:

n 2 3 4 5 6 7 8 9 10 20 

t ,n 12.7 4.3 3.2 2.8 2.6 2.4 2.4 2.3 2.3 2.1 2.0
CALCULATION OF ERRORS OF
DIRECT MEASUREMENT

Let it is made n measurements of some magnitude Х and a series of

values of this magnitude is received: X1, X2,…, Xn .
The most probable is the arithmetic average value of this
magnitude 𝑿:
𝒏
𝟏 𝟏
𝑿= 𝑿𝟏 + 𝑿𝟐 + ⋯ + 𝑿𝒏 = ෍ 𝑿𝒊 ,
𝒏 𝒏
𝒊=𝟏
where 𝑖 = 1,2,3, … , 𝑛.

The value ∆𝑿𝒊 = 𝑿 − 𝑿𝒊 - difference between average and unit

measurement- is called as absolute error of separate measurement.
Average arithmetical error ∆𝑿 is called an arithmetic average
value of absolute errors of separate measurements:

𝒏
𝟏
∆𝑿 = ෍ ∆𝑿𝒊 .
𝒏
𝒊=𝟏

The average arithmetic error ∆𝑋 is defined as the interval in which

the true value of the measured value X is located.

For a more accurate calculation of the absolute error usually the total
error (the summarized error or an aggregate error) is used. It includes
both systematic and random errors.

The summarized error ( an aggregate error) ∆𝑿 considers

 a random error ∆𝑋° ,
 a device error ∆𝑫,
 a rounding error ∆𝑹

and is defined by a relation

𝟐
∆𝑿 = ∆°𝒙 + ∆𝑫 𝟐 + ∆𝑹 𝟐 (1)
CALCULATION OF ERRORS OF
DIRECT MEASUREMENTS

° 𝟐 𝟐 𝟐
∆𝑿 = ∆𝒙 + ∆𝑫 + ∆𝑹

The random error is

defined under the
formula of Student:
these parameters are
σ𝑛𝑖=1 ∆𝑋𝑖 2
°
∆𝑥 = 𝑡𝛼,𝑛 ∙ taken for the device that
𝑛(𝑛 − 1) was used to measure
or a specific value
σ𝑛 2
𝑖=1 𝑋 − 𝑋𝑖
∆°𝑥 = 𝑡𝛼,𝑛 ∙
𝑛(𝑛 − 1)

The quality of the measurement results is

characterized by an average relative error.
The average relative error is the ratio of the arithmetic
mean error to the average value of the measured value.
This magnitude usually is represent in percentage:
∆𝑿
𝑬= ∙ 𝟏𝟎𝟎%.
𝑿
CALCULATION OF ERRORS OF
INDIRECT MEASUREMENTS

Let required magnitude Z is function of two variables

𝑍 = 𝑓 𝑥, 𝑦 .
It is established that the absolute error of function 𝑦 = 𝑓 𝑥 is
equal to product of derivative this function on an absolute error
of argument:
𝑑𝑦
Δ𝑦 = 𝑦 ′ ⋅ Δ𝑥 = 𝑑𝑥 ∆𝑥.
Therefore for definition of an absolute error of function 𝑍 =
𝑓 𝑥, 𝑦 discover a total differential of this function:

𝜕𝑧 𝜕𝑧
𝑑𝑧 = 𝜕𝑥 𝑑𝑥 + 𝜕𝑦 𝑑𝑦 (2)

𝜕𝑧 𝜕𝑧
where and - partial derivatives of function 𝑍 on
𝜕𝑥 𝜕𝑦

arguments 𝑥 and 𝑦.
Each partial derivative is as a simple derivative of function
𝑍 = 𝑓 𝑥, 𝑦 on corresponding argument. The remained argument
is considered as a constant factor.
At small values of differentials of arguments dx and 𝑑𝑦 (or
increments of arguments ∆𝑥 and Δ𝑦) an increment of a function
Δ𝑍 = 𝑑𝑍 .
CALCULATION OF ERRORS OF
INDIRECT MEASUREMENTS

In this case the formula (2) becomes

𝜕𝑧 𝜕𝑧
Δ𝑍 = ∆𝑥 + 𝜕𝑦 ∆𝑦.
𝜕𝑥

The mean square error Δ𝑍 is taken as the mean absolute

error and is defined by a relation

𝜕𝑧 2 𝜕𝑧 2
∆𝑍 = ∆𝑥 + ∆𝑦 (3)
𝜕𝑥 𝜕𝑦

where ∆𝑥 and ∆𝑦 - the total errors (summarized errors) of

measurements of magnitude 𝑥 and 𝑦, defined under the formula
(1).
The average relative error of magnitude 𝑍 pays off under
the formula
∆𝑍
𝐸= .
𝑍

Hence, having divided both parts of expression (3) on 𝑍, we

will get a relative error of function 𝑍:
2 2
∆𝑍 1 𝜕𝑧 1 𝜕𝑧
𝐸= = 2 ∆𝑥 + 2 ∆𝑦 .
𝑍 𝑍 𝜕𝑥 𝑍 𝜕𝑦
Knowing a relative error, discover an absolute error of
magnitude 𝑍:
𝜟𝒁 = 𝜠 ⋅ 𝒁 .
Final output of measurements is presented as:
𝒁 = 𝒁 ± 𝚫𝒁.
CALCULATION OF ERRORS OF
INDIRECT MEASUREMENTS

Let's consider the calculation of errors by the example of

determining the density of a solid with a regular geometric shape.
For a cylinder of a mass m, a height h and diameter D
the average density is defined by a relation
𝑚 𝑚 4𝑚
ρ= = = 2 .
𝑉 ℎ∙𝐴 ℎ∙𝜋∙𝐷
So, we have the function 𝜌 𝑚, ℎ, 𝐷 .
Based on the formula (3), we obtain
2 2 2
𝜕𝜌 𝜕𝜌 𝜕𝜌
∆ρ = ∆𝑚 + ∆ℎ + ∆𝐷 ,
𝜕𝑚 𝜕ℎ 𝜕𝐷
Substitute the values of the partial derivatives
2 2 2
4∆𝑚 4𝑚∆ℎ 4𝑚2∆𝐷
∆ρ = 2 + 2 + 2 .
ℎ∙𝜋∙𝐷 ℎ∙𝜋∙𝐷 ℎ∙𝜋∙𝐷
Then
2 2 2 2
∆ρ ∆𝑚 ∆ℎ 2∆𝐷
= + + ,
ρ 𝑚 ℎ 𝐷
from here
2 2 2
∆ρ ∆𝑚 ∆ℎ 2∆𝐷
= + + .
ρ 𝑚 ℎ 𝐷
Thus, the density relative error
2 2 2
∆ρ ∆𝑚 ∆ℎ 2∆𝐷
𝐸= = + + .
ρ 𝑚 ℎ 𝐷
Knowing a relative error, we discover the density absolute error
Δρ :
Δρ = Е ⋅ ρ .
Final output of measurements is presented as:
ρ = ρ ± Δρ.
 RULES FOR THE MATHEMATICAL PROCESSING OF THE
RESULTS OF EXPERIMENTAL MEASUREMENTS

The number of reliably fixed digits in the measurement result

record is called the number of significant digits.
The zeros at the end of the number are significant digits.
Zeros at the beginning of a number are not significant digits:
0.002540960 - this number contains 7 significant digits;
10.02340060 - all the digits are significant (10 significant digits).

The decimal digits of a number are all the digits in the number
after the decimal point. The number of decimal places in a numeric
value determines its accuracy:
25.125987- this number contains 6 decimal places;
0.287-this number has an accuracy of up to 3 digits

When calculating the average values of measured values, as well as

for indirect measurements, the following rules are used:
 «+, −» when adding and subtracting numbers of different
precision, the numerical answer after rounding retains as many
decimal places as there are in the number with the smallest
number of decimal places
0.2365 + 12.302-8.12045 + 0.1 = 4.51805 ≈ 4.5

 « ×,÷ » when multiplying and dividing several numbers, only

as many significant digits remain in the numerical answer after
rounding as in the value of direct measurements with a
minimum number of significant digits
0.235 · 2.3 = 0.5405≈0.54

 when performing the actions 𝑋 𝑛 , 𝑋 , log 𝑋, it is necessary to

leave after rounding as many significant digits as in the number
X itself
0.25=0.50
2 2
30 =900=9.0∙10
 RULES FOR THE MATHEMATICAL PROCESSING OF
THE RESULTS OF EXPERIMENTAL MEASUREMENTS

The significant digits rule does not apply to error

calculation!
The absolute error is recorded with the same
accuracy as the obtained average value calculated
during mathematical processing

Correctly Wrong
𝒁 = 𝟐𝟖𝟒 ± 𝟏 𝒁 = 𝟐𝟖𝟒. 𝟓 ± 𝟏

𝒁 = 𝟓𝟐. 𝟕 ± 𝟎. 𝟑 𝒁 = 𝟓𝟐. 𝟕𝟒 ± 𝟎. 𝟑

𝒁 = 𝟒. 𝟕𝟓𝟎 ± 𝟎. 𝟎𝟎𝟔 𝒁 = 𝟒. 𝟕𝟓 ± 𝟎. 𝟎𝟎𝟔

3
𝒁 = (𝟏. 𝟔𝟐 ± 𝟎. 𝟎𝟒) ∙ 𝟏𝟎𝟑 𝒁 = 𝟏. 𝟔𝟐 ∙ 𝟏𝟎𝟑 ± 𝟎. 𝟎𝟒 ∙ 𝟏𝟎𝟑
The practical part

In ordinary mathematical calculations, the answer can often contain

either a lot of digits (both in the whole part and after the decimal point), or
the answer can be irrational, i.e. infinite decimal fraction. However, as
applied to natural science disciplines, obtaining an irrational number as
an answer in an experiment is not correct (for example, the values of body
mass, movement speed, solution concentration, etc. should have a final,
strictly defined value). In such cases, it is convenient to use real numbers
written in a special standard form
𝑴 ∙ 𝟏𝟎𝒏
where M is called the mantissa of a number,
n is the order of a number.
Moreover, strict conditions are imposed on the mantissa (always) and order
(when solving applied problems):
𝟏 ≤ 𝑴 < 𝟏𝟎
𝑴 ∙ 𝟏𝟎𝒏 ⇔ ቊ
𝒏 ≠ 𝟎; 𝟏 .

Algorithm for representing a number in a standard form

Step 1. Put a new decimal point in the given number after the first nonzero
digit.

Step 2. Calculate the number of positions by which the decimal point has
moved. This quantity determines the order of number n.
If the point has moved to the left, then the value of n is
increased;
moving to the right decreases the value of n.

Step 3. Write down the number in standard form: 𝑋 = 𝑀 ∙ 10𝑛

For example,
𝟏𝟎𝟎𝟓𝟎𝟎 = 𝟏. 𝟎𝟎𝟓 ⋅ 𝟏𝟎𝟓 ;
𝟎. 𝟎𝟔 = 𝟔 ⋅ 𝟏𝟎−𝟐 ;
−𝟎. 𝟎𝟕𝟕𝟕 ⋅ 𝟏𝟎−𝟐 = −𝟕. 𝟕𝟕 ⋅ 𝟏𝟎−𝟒 .

This method of writing numbers is especially convenient in

calculations and when writing units of measurement.
The standard form helps you perform calculations with numbers of
varying accuracy. The skills of working with standard numbers allow you to
evaluate the numerical result even without using a calculator.
The practical part

Let's consider the following solution algorithm:

 Write down all numbers in standard form
𝟎. 𝟎𝟏𝟓 − 𝟎. 𝟎𝟐𝟐 + 𝟒 ∙ 𝟏𝟎−𝟐 𝟐 =
𝟏𝟓 ∙ 𝟏𝟎−𝟑 − 𝟐 ∙ 𝟏𝟎−𝟐 𝟐 + 𝟒 ∙ 𝟏𝟎−𝟐 𝟐 =⋯
 Represent all values with the same order (at this step, for convenience
calculations, the mantissa format can be neglected)
… = 𝟏𝟓𝟎 ∙ 𝟏𝟎−𝟒 − 𝟒 ∙ 𝟏𝟎−𝟒 + 𝟏𝟔 ∙ 𝟏𝟎−𝟒 = ⋯
 Perform all arithmetic operations with mantissas, considering same
order as common factor
…= 𝟏𝟓𝟎 ∙ 𝟏𝟎−𝟒 + 𝟏𝟐 ∙ 𝟏𝟎−𝟒 = 𝟏𝟔𝟐 ∙ 𝟏𝟎−𝟒 ≈ ⋯
 Write down the final answer in the correct standard form (with
the correct mantissa)
… ≈ 𝟏𝟑 ∙ 𝟏𝟎−𝟐 .

Tasks for self-control

I. Represent these numbers in standard form:

1) 9280= 2) 125,05= 3) 0,0081=

4) 17 000 000= 5) 40503= 6) 0,0023=

II. Calculate the values of the expressions, presenting the answers in

standard form with 2 significant digits:

1) 0.0052= 7) 0.0052 +0.007=2 =

2) 0.00072= 8) 0.0022 +0.0032 =
3) 0.005= 9) 0.0025+0.052 − 7∙10−2 2

4) 0.0007= 10) 0.0011−10−3 =

5) 0.005+0.0007= 11) 0.0112 +0.3∙10−5 +0.0027=
6) 0.005+0.0007= 12) 0.0004+0.22 −5 ∙ 10−5 =
Prefixes for the International System of Units (SI)
Prefix English word
Base 10 Decimal
Name Symbol Short scale Long scale
yotta- Y- 1024 1000000000000000000000000 septillion quadrillion
zetta- Z- 1021 1000000000000000000000 sextillion trilliard
exa- E- 1018 1000000000000000000 quintillion trillion
peta- P- 1015 1000000000000000 quadrillion billiard
tera- T- 1012 1000000000000 trillion billion
giga- G- 109 1000000000 billion milliard
mega- M- 106 1000000 million
kilo- k- 103 1000 thousand
hecto- h- 102 100 hundred
deca- da- 101 10 ten
100 1 one
deci- d- 10−1 0.1 tenth
centi- c- 10−2 0.01 hundredth
milli- m- 10−3 0.001 thousandth
micro- μ- 10−6 0.000001 millionth
nano- n- 10−9 0.000000001 billionth milliardth
pico- p- 10−12 0.000000000001 trillionth billionth
femto- f- 10−15 0.000000000000001 quadrillionth billiardth
atto- a- 10−18 0.000000000000000001 quintillionth trillionth
zepto- z- 10−21 0.000000000000000000001 sextillionth trilliardth
yocto- y- 10−24 0.000000000000000000000001 septillionth quadrillionth

Examples
−2
• 5 cm = 5∙10 m = 5 ∙ 0.01 m = 0.05 m
−3 −1
• 300g = 300∙10 kg = 3.00∙10 kg
−3 −3 −6 −6
• 5 mV ∙ 5 mA = 5∙10 V ∙ 5∙10 A = 25∙10 V⋅A =25∙10 W=25 μW
−3 −6 −6 −6 −6
• 5.00 mV + 10 μV = 5.00∙10 + 10∙10 V = 5000∙10 V +10∙10 V =5010∙10 V=
=5.01 mV.
When powers of units occur, for example, squared or cubed,
the multiplicative prefix must be considered part of the unit, and thus included
in the exponentiation:
2 3 2 3 2 2 6 2
• 9 km = 9 ∙ 10 m = 9 ∙ 10 m = 9∙10 m
2 −2 2 −4 2 −3 2
• 15cm =15∙ 10 m =15∙10 m =1.5∙10 m .
Prefixes for the International System of Units (SI)

Tasks for self-control

III. Write down the presented data in the required units with
an accuracy of 2 significant digits:

For example,
85.3 cm= ? m
−2 −1
85.3cm = 85.3 ∙ 10 m = 8.53 ∙ 10 m…
…with the accuracy of 2 significant digits…
−1
≈ 8.5 ∙ 10 m

1) 23.5 cm=⋯ m 7) 25 cm=⋯ m

2) 14 mm=⋯ m 8) 2.3 kg=⋯ g

2 2
3) 35.7 m=⋯ km 9) 2.0 cm =⋯ m
3 3
4) 348 g=⋯ kg 10) 30.5 mm =⋯m
2 2
5) 50 mg=⋯ kg 11) 9.51 mm =⋯ m

6) 123 mg=⋯kg
 QUESTIONS FOR SELF-CONTROL

1. What is the measurement?

2. What types of measurements are there?
3. What kind of errors occur in the measurement process?
4. How to calculate an arithmetic mean (an average value) of
a test quantity? With what precision is it necessary to
determine an average value?
5. The absolute error of a single measurement: how to calculate
it?
6. What are the components of an aggregate error?
7. What is tα, n in the random error formula? What is the value
of t,n for 5, 6 and 9 experiments?
8. How to assess the quality of experimental work?
9. What parameters does the relative error of the investigated in
the experiment magnitude depend on?
10. How to record the final result of math processing of
an experiment? What can you say about the precision of
the absolute error in this recording?
 Check your answers to practical tasks

Task I

1) 92809.3·103 2) 125.051.3·102 3) 0,00818.1·10-3

4) 17 000 0001.7·107 5) 405034.1·104 6) 0,00232.3·10-3

Task II
1) 0.0052 2.5·10−5 7) 0.052 +0.0072 5.0·10-2

2) 0.00072 4.9·10-7 8) 0.0022 +0.0032 3.6·10−3

3) 0.005 7.1·10-2 9) 0.0025+0.052 − 7∙10−2 2 1.0·10−2

4) 0.0007 2.6·10-2 10) 0.0011−10−3 1.0·10−2

5) 0.005+0.00075.7·10-3 11) 0.0112 +0.3∙10−5 +0.0027 5.3·10−2

6) 0.005+0.0007 7.5·10-2 12) 0.0004+0.22 −5 ∙ 10−5 2.0·10−1

Task III

1) 23.5 cm=2.4·10-1 m 7) 25 cm=2.5·10-1 m

2) 14 mm=1.4·10-2 m 8) 2.3 kg=2.3·103 g

2 2
3) 35,7 m=3.6·10-2 km 9) 2.0 cm =2.0·10−4m
3 3
4) 348 g=3.5·10-2 kg 10) 30.5 mm =3.1·10−8m
2 2
5) 50 mg=5.0·10-5 kg 11) 9.51 mm =9.5·10−6m

6) 123 mg=1.2·10-4kg