Computational Materials Science 158 (2019) 1–13

Contents lists available at ScienceDirect

Computational Materials Science

journal homepage: [Link]/locate/commatsci

A fast algorithm for material image sequential stitching T

a,b,c a,b,c,⁎ a,d,e,⁎ a,d a,b,c f
Boyuan Ma , Xiaojuan Ban , Haiyou Huang , Wanbo Liu , Chuni Liu , Di Wu ,
Yonghong Zhig
a
Beijing Advanced Innovation Center for Materials Genome Engineering, Xueyuan Road 30, Haidian District, Beijing 100083, China
b
School of Computer and Communication Engineering, University of Science and Technology Beijing, Xueyuan Road 30, Haidian District, Beijing 100083, China
c
Beijing Key Laboratory of Knowledge Engineering for Materials Science, Xueyuan Road 30, Haidian District, Beijing 100083, China
d
Institute for Advanced Materials and Technology, University of Science and Technology Beijing, Xueyuan Road 30, Haidian District, Beijing 100083
e
Key Laboratory of Environmental Fracture (Ministry of Education), University of Science and Technology Beijing, Xueyuan Road 30, Haidian District, Beijing 100083,
China
f
Department of ICT and Natural Sciences, Norwegian University of Science and Technology, Aalesund, Norway
g
Mechanical and Electrical Design and Research Institute of Shanxi Province, Shengli Street 228, Xinghualing District, Taiyuan City, Shanxi Province 030009, China

ARTICLE INFO ABSTRACT

Keywords: In material research, it is often highly desirable to observe images of whole microscopic sections with high
Micrograph stitching resolution. So that micrograph stitching is an important technology to produce a panorama or larger image by
Feature matching combining multiple images with overlapping areas, while retaining microscopic resolution. However, due to
GPU acceleration high complexity and variety of microstructure, most traditional methods could not balance speed and accuracy
of stitching strategy. To overcome this problem, we develop a method named very fast sequential micrograph
stitching (VFSMS), which employ incremental searching strategy and GPU acceleration to guarantee the accu-
racy and the speed of stitching results. Experimental results demonstrate that the VFSMS achieve state-of-art
performance on three types’ microscopic datasets on both accuracy and speed aspects. Besides, it significantly
outperforms the most famous and commonly used software, such as ImageJ, Photoshop and Autostitch. The
software is available at [Link]

1. Introduction based superalloy turbine blade with a high resolution (10 μm) obtained
by Prof. Feng’s group to assist the optimization of service performance.
Microstructure is the classic term used in metallography to describe However, due to the limitation of imaging principle, it is usually
small scale structure features in the scale less than 1000 μm [1]. The difficult to get a single image with high resolution to view a whole sample
microstructure of a material (such as metals, ceramics, polymers or with large size [4]. A usual solution is to stitch several local images with
composites) can strongly influence almost all properties such as me- overlapping areas to form a global composite one, which refers to as
chanical properties, physical properties, corrosion resistance, etc., Stitching [5,6]. Some examples of optical micrograph stitching in material
which govern the application of the material in industrial practice. The researches are shown in Fig. 2. We demonstrate three types’ microscopic
microstructure is often observed and analyzed using various micro- images with different shooting paths, which represented by red dashed
scopy methods such as optical microscopy, electron microscopy, etc. line. Besides, the red translucent regions refer to a shot of microscope. As
according to the necessary of examining and determining magnifica- shown in the figure, there are some challenges in the stitching processes:
tions [2,3]. With the development of material design, integrated design First, the micrographs have high complexity and compose of diverse tiny
from micro- or nano-structure of materials to large-scale structure of microstructures, which increases demand for robustness of stitching
components has aroused more and more attention from material sci- strategy. Second, microscopic images usually have high resolution which
entists and engineers. Multi-scale material design often need illustrate increases demand for efficiencies of stitching strategy. Third, there is no
macro- and micro-structure simultaneous. A large-scale panorama of static shooting path in actual practice, so the stitching strategy should
material or component with micro-level resolution may help multi-scale self-adaptive to any situation.
and integrated design of material and component. Fig. 1 shows a sample Currently, many stitching methods and software can be categorized
of a large-scale panorama (35 mm × 17 mm) of a cross section for Ni- into two classes: Feature based and Phase correlation based [7].

⁎
Corresponding authors at: Beijing Advanced Innovation Center for Materials Genome Engineering, Xueyuan Road 30, Haidian District, Beijing 100083, China.
E-mail addresses: banxj@[Link] (X. Ban), huanghy@[Link] (H. Huang).

[Link]
Received 17 September 2018; Received in revised form 31 October 2018; Accepted 31 October 2018
0927-0256/ © 2018 Elsevier B.V. All rights reserved.
B. Ma et al. Computational Materials Science 158 (2019) 1–13

Fig. 1. a large-scale panorama (35 mm × 17 mm) of a cross section for Ni-based superalloy turbine blade with a high resolution (10 μm) for multi-scale and
integrated design of material and component.

(e)

(a) (b) (f)

(g)

(h)

(c) (d) (i)

Fig. 2. Three types’ local and global micrographs and their shooting path. The red translucent regions represent one shot from microscope. The red dashed lines refer
to the shooting path. (a) Optical microscopy (OM) image of Iron polycrystal with its detail imaging. (b) Pairwise shooting path of (a) with 2 local images. (c) OM
image of dendritic crystal with its detail imaging. (d) Grid shooting path of (c) with 90 local images. (e) Scanning electron micrographs of zircon in reflected mode
(SEM-RE), (f) in transmission mode (SEM-TR), (g) in black scattered mode (SEM-BS), and (h) cathodoluminescence mode (SEM-CL) respectively with its detail
imaging. (i) Shooting path for (e)–(h), the number of local images depends on the length of sample. (For interpretation of the references to color in this figure legend,
the reader is referred to the web version of this article.)

Feature based stitching methods can achieve high accuracy with homography [16,17] to match two sequential micrographs, which are
invariant features, such as SIFT [8,9], SURF [10,11] and ORB [12]. suited for nature scene and will cause some deformations for local
These methods establish correspondences between points, lines, edges, micrographs in horizontal translation model.
corners, or other geometric entities. Characteristics of robust detectors Phase correlation based methods can achieve real time stitching by
include invariance to image noise, scale invariance, translation in- using implementation of fast fourier transformation (FFT) [19,20].
variance, and rotation transformations. However, since the micro- Most software build upon those methods by considering the speed de-
structures are always highly complicated, most of these methods can mand of users [21,24,25]. However those may yield ambiguous results
extract thousands of features to match, which may cause time con- when micrographs contain much of similar microstructures [26], which
sumption and a heavy computational burden so that couldn’t be applied will lead low accuracy when stitching.
in real time [13,14]. Some works try to sample the image in order to In this work, we describe a developed stitching method, named very
decrease the time consumption of feature searching and matching fast sequential micrograph stitching (VFSMS), to balance the accuracy
[6,18]. However, these methods will unavoidably lose the information and speed of stitching strategy. Our algorithm presents three con-
in micrograph. Besides, most of them apply ransac [15] and tributions:

2
B. Ma et al. Computational Materials Science 158 (2019) 1–13

(1) Incremental searching strategy and corresponding matching and offset (between original point of next image and that of last image) and
fusing method. Those make the algorithm achieve improvement stitch next micrograph. Otherwise the algorithm will interrupt the
both on accuracy and speed aspects. stitch and restart from next micrograph if not correct. Therefore,
(2) GPU acceleration. We introduce GPU acceleration in feature search VFSMS will save stitch previous results and restart the stitch on the next
process which make stitching could be applied in real time, even for images when it faces mismatch, which is designed for multi section
the complicated micrograph with huge resolution. It is the first stitch. Third, it uses trigonometric function to fuse images and elim-
demonstration of GPU acceleration in this context that in particular inate obvious seamline. The detailed of fusing algorithm can be found
has substantial gains in the aspects of stitching speed. in the Section 2.4. Finally, it gets stitching output result.
(3) Sequential stitching. The algorithm developed in this paper is
completely automated for fast stitching of micrographs. There is no
need to design the shooting path. 2.2. Incremental searching strategy

Experimental results demonstrate that VFSMS achieves state-of-art We demonstrate the schematic diagram and flowchart of incre-
performance on several microscopic datasets on both accuracy and mental searching strategy in Figs. 4 and 5. Let f1(x, y) and f2(x, y) re-
speed aspects. Besides, it significantly outperforms the most famous and present the two sequential local micrographs. There are three para-
commonly used software, such as ImageJ [21,27], Photoshop [24] and meters influence the algorithm’s performance: local direction,
Autostitch [28,29]. The average time consumption of VFSMS is 60% of incremental parameter and feature search length. Local direction means
ImageJ, which is most popular software in material image processing. stitching direction of two sequential micrographs. As shown in the
In addition, VFSMS achieves 100% accuracy in experimental datasets, Fig. 4, when it equals 0, it means the first image locates at the above of
which is more robust than other software. the second image. 1 means the first image locates at the left of the
The remainder of this paper is organized as follows: In Section 2, we second image. 2 means the first image locates at the bottom of the
provide a description of the VFSMS. In Section 3, we present experi- second image. And 3 means the first image locates at the right of the
ments on different stitching methods to show the effectiveness and ef- second image. Incremental parameter is constrained to be any of −1, 0
ficiencies of the proposed approach. Besides, we discuss the results of or 1, which control stitching direction in the next stage. For example,
different methods in Section 4. Finally, we conclude the paper in let local directionprevious represents the previous local stitching direction
Section 5. of the first and the second micrographs and local directionnext represents
the next local stitching direction of the second and the third micro-
2. Method graphs. If incremental parameter equals 0, the local directionnext is
consistent with local directionprevious . If incremental parameter equals 1,
2.1. Framework of the VFSMS the algorithm will turn local directionprevious 90 degrees clockwise to
form local directionnext . If incremental parameter equals −1, the algo-
VFSMS is a feature based image stitching method. For traditional rithm will turn local directionprevious counterclockwise 90 degrees to
feature based method, it is used to extract thousands of features in one form local directionnext . The algorithm uses following Eq. (1) to update
local image, which may cause time consumption and a heavy compu- the local direction when the algorithm cannot find correct result in
tational burden so that cannot be applied in real time [13,14]. In order certain local direction, where “mod” denotes remainder calculation.
to handle this problem, we proposed to use incremental searching The third parameter is feature search length. We believe that stitching
strategy and GPU acceleration to increase stitching speed and robust- task will only require the partial region of image to compare the si-
ness. milarity and it is no need to calculate all information in two images.
The flowchart of VFSMS is shown in Fig. 3. First, we input micro- Therefore, feature search length is the key parameter to control feature
graph sequences, the algorithm will carry on if there is any micrograph search region. It will be initialized as relative small value (for example,
never been calculated. Second, it uses incremental searching strategy to 20% of the corresponding edge) and expanded if the algorithm cannot
find pairs of feature points and match them. The detailed flowchart and find correct result. By the way, if the feature search length beyond the
diagram of incremental search can be found in the Section 2.2. The 50% of the corresponding edge and the algorithm does not find correct
detailed match method could be found in the Section 2.3. If the last two stitch, we will assume that these two sequential images cannot be
sequential micrographs match correct, the algorithm will record the stitched.

Fig. 3. The flowchart of the VFSMS.

3
B. Ma et al. Computational Materials Science 158 (2019) 1–13

Fig. 4. Diagram of incremental stitching strategy. (a)–(e) shows the updating process of stitching direction by using local direction and incremental parameter. In this
example, initial local direction equals 0 and incremental parameter equals 1.

Fig. 5. Flowchart of incremental stitching strategy.

local directionnext = (local directionprevious + Incremental parameter ) mod 4 initializes feature search length (20% of the corresponding edge in this
(1) paper’s experiments). Third it stitches images if feature search length is
below the certain threshold (50% of the corresponding edge). Fourth, it
We will show the procedure of incremental strategy step by step uses two search regions in corresponding areas of two sequential
using the Figs. 4 and the 5. First, the VFSMS sets local direction and images, which represented by red translucent regions in the Fig. 4, to
incremental parameter or inherits it from last stitch (In the Fig. 4, the find feature points and match them. If match correct, it records the
local direction equals 0 and incremental parameter equals 1). Second, it offset and goes back to other processes in the Fig. 3. Otherwise, it

4
B. Ma et al. Computational Materials Science 158 (2019) 1–13

Fig. 6. Custom stitching parameters according to shooting path.

Fig. 7. Matching strategy for two sequential micrographs. M, N denote the rows and columns of the micrograph.

modifies the local direction by using above Eq. (1), for example, from to left, the initial local direction can be set to 3, and the incremental
the Fig. 4(a)–(d). We can use default value of local direction and in- parameter can be set to 0. Therefore, the algorithm can accelerate the
cremental parameter to let algorithm to find correct stitching result stitching speed by customizing key parameters to decrease the number
(local direction equals 0, and incremental parameter equals 1). In ad- of searching, according to different shooting paths. Fifth, we double the
dition, we can customize these parameters to let the algorithm change search length in order to access more micrographs information and
the direction of feature search length increment, according to different carry the loop if it returns to the initial direction, as shown in the
shooting paths (see Fig. 6). For example, if the image is taken from right Fig. 4(e). Finally, it will record the offset if match correct. Otherwise, it

5
B. Ma et al. Computational Materials Science 158 (2019) 1–13

Fig. 8. Fusing demonstration. (a) left-right pairwise image fusion. (b) upper-bottom pairwise image fusion.

Fig. 9. ‘Corner’ stitching fusing demonstration. (a) left-bottom corner. (b) right-upper corner. The red translucent region denotes corner region. (For interpretation of
the references to color in this figure legend, the reader is referred to the web version of this article.)

will report to the algorithm that these two sequential images cannot be In the Fig. 7, the first row represents horizontal stitching. The
stitched. second row represents vertical stitching. The green1 line denotes the
By using incremental searching strategy, the algorithm needs only location of border of the first micrograph actually correspond to the
partial of images to calculate the offset. Therefore, it will save the location in the second micrograph. The blue line denotes a pair of re-
consumption of resources and increase the speed of the stitching task. gistered surf feature points. SURF is a feature search algorithm, and it
To realize the stitching of microscopic images with high speed and can extract many pairs of feature in the two sequential micrographs. In
high accuracy, SURF algorithm [10] was chosen to extract image fea- addition, we use ratio of closest to second-closest neighbors [8] of each
tures in this paper. It has been proved to have a good performance than key-point to register pair of key-points. For our stitching experiment,
SIFT and ORB. Besides, the speed of searching feature points would we reject all matches in which the distance ratio is greater than 0.75.
have notable enhancement because of GPU acceleration. For each pair, we can calculate the offset (‘dy’ and ‘dx’) according to the
location of two points. Finally, we could get the mode of offsets, which
is the value that appears most often in data set. In addition, if the
2.3. Matching
number of mode value beyond a threshold (in our experiment, we set
3), we can convince that the offset is correct.
Incremental searching strategy needs algorithm to detect the mis-
match in order to transform to the next direction. However, traditional
2.4. Image fusion
matching strategies of feature based methods such as homography and
ransac cannot detect the mismatch effectively. Besides, phase correla-
There are overlapping areas in the stitching process, as described in
tion based methods just only pick the first peak value in correlation
the Figs. 8 and 9. Due to imaging principle of microscope, the border of
plane of image, which also cannot detect the mismatch.
micrograph will have lower luminance than the center region, which
Therefore, we build a matching method that is suited to incremental
will cause obvious seamline in stitching result, as shown in Fig. 10.
searching strategy. We assume all transformations between overlapping
We use trigonometric functions [30] to fuse two overlapping regions
images to be translation only which is reasonable for most commonly
used microscopy techniques where the static, fixed sample rests on a
level microscope slide moved by a motorized stage. The matching 1
For interpretation of color in Fig. 7, the reader is referred to the web version
strategies are shown in the Fig. 7. of this article.

6
B. Ma et al. Computational Materials Science 158 (2019) 1–13

No fusion Trigonometric fusion

Dendritic
Crystal
OM

Zircon
SEM-RE

Zircon
SEM-TR

Fig. 10. Fusing results. The first column represents stitching without fusing operation. The second column represents stitching with trigonometric fusing operation.
The red arrows denote obvious seamlines. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 1
Images datasets.
Types Dendritic Crystal OM Iron Polycrystal OM Zircon SEM-BS Zircon SEM-CL Zircon SEM-RE Zircon SEM-TR
Dimensions (pixel × pixel) (1936 × 2584) (1936 × 2584) (1024 × 1280) (1024 × 1280) (1728 × 2592) (1728 × 2592)

Group 1 90 2 4 24 11 9
2 90 2 6 20 12 12
3 90 2 27 20 13 13
4 90 2 25 23 9 13
5 91 2 22 24 14 8
6 90 2 28 23 10 13
7 91 2 30 24 8 9
8 90 2 22 21 14 12
9 90 2 24 16 12 18
10 90 2 24 18 9 18

in order to eliminate seamline. Let f1(x, y) and f2(x, y) represent the two However, this method is designed to fuse pairwise micrographs. We
sequential local micrographs, f(x, y) denotes the fusing result, each make an improvement according to sequential stitching strategy espe-
pixel value can be given by the following equation: cially for the ‘corner’ stitching (see the Fig. 9). The equation of corner
region stitching is:
f1 (x , y ), x , y f1
f (x , y ) = w1 f1 (x , y ) + w2 f2 (x , y ), x , y (f1 f2 ) f1 (x , y ), x , y f1
f2 (x , y ), x , y f2 (2) w1 f1 (x , y ) + w2 f2 (x , y ), x , y l r region
where f (x , y ) = w3 f1 (x , y ) + w4 f2 (x , y ), x , y u b region
w1 w3 f1 (x , y ) + (1 w1 w3 ) f2 (x , y ), x , y c region
(x xleft ) (x right x) f2 (x , y ), x , y f2
w1 = cos 2 , w2 = sin2 or (3)
2 (x right xleft ) 2 (x right xleft )
where w3 and w4 are calculated as same as pairwise stitching. l means
(Y Yupper ) (Ybottom Y ) left, r means right, u means upper, b means bottom, c means corner.
w1 = cos 2 , w2 = sin2
2 (Ybottom Yupper ) 2 (Ybottom Yupper ) Some examples of fusing results are showing in the Fig. 10. As
shown in the figure, trigonometric image fusion can eliminate obvious
where Yupper , Ybottom , Xright , Xleft denote the upper edge, lower edge, right
seamline perfectly.
edge and left edge of overlapping region of the two images respectively
(see the Fig. 8). w1 and w2 are weighting parameters which control
fusion result of two pixel value.

7
B. Ma et al. Computational Materials Science 158 (2019) 1–13

Fig. 11. Time consumption with different methods.

Table 2
Time per image of VFSMS and other software (* is stitching mode in Image-J, the unit is second).
Types Dendritic crystal OM Iron polycrystal OM Zircon SEM-BS Zircon SEM-CL Zircon SEM-RE Zircon SEM-TR Average

VFSMS 1.55 0.75 0.36 0.35 0.63 1.00 0.77

Grid Stitch* 2.10 1.99 0.30 0.27 1.28 1.30 1.21
Sequential Stitch* 2.78 3.15 0.65 0.41 2.30 2.13 1.90
Pairwise Stitch* / 2.57 / / / / /
Photoshop photomerge 3.43 3.56 3.92 5.63 2.46 3.75 3.79
AutoStitch 0.65 2.08 0.55 0.35 0.83 3.32 1.30

Table 3
Accuracy of VFSMS and other software (*is stitching mode in Image-J).
Types Dendritic crystal OM Iron polycrystal OM Zircon SEM-BS Zircon SEM-CL Zircon SEM-RE Zircon SEM-TR Average

VFSMS 100% 100% 100% 100% 100% 100% 100%

Grid Stitch* 50% 100% 60% 100% 100% 100% 73%
Sequential Stitch* 0% 100% 100% 0% 30% 100% 55%
Pairwise Stitch* / 100% / / / / /
Photoshop photomerge 40% 100% 0% 0% 0% 20% 27%
AutoStitch 0% 0% 60% 0% 20% 70% 25%

3. Experimental results of method. Scientists need image stitching method to observe and
analysis large scale of microstructures with high resolution. Type 3
3.1. Datasets micrographs have a little zircon block in one local image, however, the
texture of zircon is very similar which increase demand for robustness
As shown in the Fig. 2, we chose three types’ micrographs of dif- of method. Inspectors need large scale of sample to measure geological
ferent materials, different structure and different imaging modes as age. Each type has different dimensions (image sizes, the unit is pixel).
datasets. Type 1 micrograph is optical microscopy (OM) image of pure Each type has ten groups of sub-image datasets for parallel tests to
iron polycrystalline structure (Iron Polycrystal OM), which as a sample ensure the reliability of experimental results. Each group has different
for polycrystalline grain structure. Type 2 micrograph is OM image of numbers of micrographs according to the shooting situation as shown in
dendritic crystalline structure of Al-La alloy (Dendritic Crystal OM), Table 1. All the local micrographs in each group can be stitched to form
which as a sample for fractal structure. Type 3 micrograph is scanning a global composite one. In total, we have 1584 micrographs for
electron micrographs of zircon block embedded in resin matrix in re- stitching performance testing.
flected mode (SEM-RE), transmission mode (SEM-TR), black scattered
mode (SEM-BS), and cathodoluminescence mode (SEM-CL) as the
samples for two phase/composition structure. Type 1 and type 2 mi- 3.2. Implementation and evaluation details
crographs are provided by Beijing advanced innovation center for
materials genome engineering, and type 3 are provided by hebei bureau Our implementation of VFSMS was derived from the publicly
of geo-information. Type 1 and type 2 micrographs have thousands available python [31], numpy [32] and opencv toolbox [33]. In addi-
detailed highly complex and diverse tiny microstructures with different tion, all the experiments are conducted on an E5-2687W Intel core of a
sizes and shapes which increase demand for effectiveness and efficiency 2.60 GHz Windows workstation with 16 GB of memory and Nvidia
Quadro K1200 with a graphics memory of 4 GB [34].

8
B. Ma et al. Computational Materials Science 158 (2019) 1–13

VFSMS ImageJ-Grid stitch ImageJ-Sequential stitch

Dendritic
Crystal OM - 2

VFSMS

Zircon
ImageJ Grid Stitch
SEM-BS - 5

VFSMS

Zircon
SEM-CL - 1

ImageJ Sequential Stitch

VFSMS

Zircon ImageJ Sequential Stitch

SEM-RE - 9

Fig. 12. Some examples of VFSMS and ImageJ.

In this work, we use the accuracy and time consumption to evaluate micrographs. For micrographs of dendritic crystal and iron polycrystal
performance of stitching strategy and software. The accuracy in each in particular, those microstructures are always highly complicated.
type was calculated as following formula and it was evaluated by ma- That will cause much more time consumption shown in Fig. 11. As
terial scientists. In experiments, they judged whether the stitching was shown in the Fig. 11(a), we demonstrate feature numbers and matching
correct by observing whether there was a misplacement which can tell numbers per image in the three types’ datasets. Dendritic crystal and
by human eyes on both sides of the stitching line. iron polycrystal images have nearly same dimensions compared to
other datasets, however, the feature numbers of those are eight times
Nright
AccuracyT = × 100% larger than the other’s. Therefore, the time consumption of those is
Ntotal (4) thirty-six times larger than the other’s in CPU mode as shown in the
where Nright means the number of groups which stitching result have no Fig. 11(b).
observably misplacement, Ntotal means total groups in type T. Incremental searching strategy is a highly effectively method which
suites for feature based and sequential stitching. In the Fig. 11(b), in-
cremental strategy is much faster than the method without it, even in
3.3. Time consumption for feature search in VFSMS the CPU mode. Besides, GPU acceleration can further improve the speed
of stitching especially for the dendritic and iron polycrystal which have
Feature based stitching would have high accuracy, but it will cost a thousands of features. However, GPU acceleration would have little
lot of time consumption if there are thousands of feature points in local

9
B. Ma et al. Computational Materials Science 158 (2019) 1–13

VFSMS Photoshop-photomerge

Dendritic
Crystal
OM- 8

VFSMS

Zircon
SEM-BS - 3
Photoshop-photomerge

VFSMS

Zircon
SEM-CL - 4
Photoshop-photomerge

VFSMS

Zircon
Photoshop-photomerge
SEM-RE - 2

VFSMS

Zircon
SEM-TR - 10
Photoshop-photomerge

Fig. 13. Some examples of VFSMS and Photoshop.

influence when applied to the certain datasets which have lower feature ImageJ (1.51 s) [21,22], Photoshop-photomerge [23,24] and Autostitch
points. That is because that there is some inevitable time consumption [17,28]. There are three types of stitching strategies in ImageJ (1.51 s):
spends on data migration from CPU to GPU. grid stitch, sequential stitch and pairwise stitch. Grid stitch needs user
to specify the size of grid and shooting path in order to increase accu-
racy of stitching and decrease time consumption by using multi-
3.4. Performance evaluation
threaded CPU programing [27]. Sequential stitch do not need to design
the shooting path as same as VFSMS. Pairwise stitch can only stitch two
We compare the performance of VFSMS with other software, such as

10
B. Ma et al. Computational Materials Science 158 (2019) 1–13

Dendritic
Crystal
OM - 1

V FSMS A ut oStit ch

Iron
Polycrystal
OM- 2

VFSMS

Zircon
SEM-BS - 4 AutoStitch

VFSMS

Zircon
SEM-CL - 1 AutoStitch

VFSMS

Zircon
SEM-RE - 1 AutoStitch

VFSMS

Zircon
SEM-TR - 6 AutoStitch

Fig. 14. Some examples of VFSMS and Autostitch.

sequential micrographs which are not suited to large numbers of local some micrographs that it cannot match, so that the duration of calcu-
micrographs. For Photoshop, we use collage mode in photomerge lation time is very small, however, the accuracy of stitching result is
plugin. In addition, ImageJ and Photoshop use phase correlation based zero. For zircon SEM-CL and zircon SEM-BS datasets, the VFSMS would
stitching strategy. While Autostitch use feature based stitching strategy. be a little slower than the ImageJ with grid mode, that is because there
All experiments are implemented on the machine that has described in is some duration time wasted in the data transformation between CPU
the Section 3.2. and GPU. However, the VFSMS is more robust than ImageJ in the ac-
We compare the performance of VFSMS and other software in curacy aspect. Some examples are shown in the Fig. 12.
Tables 2 and 3. We calculate the average time consumption per image There are three types of micrographs with 1584 micrographs in
and statistic the accuracy of the images for each type in different total. Each type of micrograph has ten groups of sub image data and
methods. For dendritic crystal micrographs, Autostitch would omit each group have a series of images which could be stitched to a global

11
B. Ma et al. Computational Materials Science 158 (2019) 1–13

one. Therefore, we refer one group of images stitching task as one test lose color information in micrographs. Besides, there is little tiny dis-
and there are 60 tests in total. However, the assessment is a very stick tortion in the dendritic crystal’s stitching result which might be caused
standard. It means the result would be correct only if the stitching result by imaging principle of microscope. Our team will work on this pro-
of all micrographs in one group has no observably misplacement. blem in subsequent work.
According to the accuracy assessment method described in Section 3.2,
the denominator Ntotal is always 10. And the numerator Nright is range CRediT authorship contribution statement
from 0 to 10.
Experimental results demonstrate that VFSMS achieves state-of-art Boyuan Ma: Methodology, Writing - original draft. Xiaojuan Ban:
performance on several microscopic datasets on both accuracy and Conceptualization. Haiyou Huang: Conceptualization. Wanbo Liu:
speed aspects. Besides, it significantly outperforms the most famous and Data curation. Chuni Liu: Validation. Di Wu: Writing - review &
commonly used software, such as ImageJ, Photoshop and Autostitch. editing. Yonghong Zhi: Writing - review & editing.
The average time consumption of VFSMS is 60% of ImageJ, which is
most popular software in material image processing. In addition, Acknowledgements
VFSMS achieves 100% accuracy in experimental datasets, which is
more robust than other software. The authors acknowledge the financial support from the National
Key Research and Development Program of China (No.
4. Analysis and comparison 2016YFB0700500), and the National Science Foundation of China (No.
61572075, No. 61702036, No. 51574027), and Fundamental Research
We demonstrate some examples of stitching results of VFSMS and Funds for the Central Universities (No. FRF-TP-17-012A1), and China
other software in this section. Postdoctoral Science Foundation (No. 2017M620619). Besides, we
In the Fig. 12, we show some stitching results of VFSMS and ImageJ gratefully thanks to Dr. Siliang He and Prof. Qiang Feng for providing
(1.51 s). Each row refers to one sample. ‘Dendritic crystal OM–2’ re- Fig. 1 for illustration.
presents the second group in dendritic crystal OM dataset. As described
in the Section 3.4, there are grid, sequential and pairwise mode in Appendix A. Supplementary material
ImageJ. Grid stitch needs user to specify the shooting path and grid size.
It can eliminate accumulative error so that error in one line would not Supplementary data to this article can be found online at https://
influence the other line, as shown in the second image of the first row. [Link]/10.1016/[Link].2018.10.044.
Moreover, ImageJ is a phase correlation based software. It may yield
ambiguous results when micrographs contain much of similar micro- References
structures [26]. Besides, ImageJ doesn’t split the mismatch result which
causes troubles for users, shown in the dendritic crystal OM-2, Zircon [1] ASM handbook: metallography and microstructures, ASM International, 1985.
SEM-BS-5, Zircon SEM-CL-1 and Zircon SEM-RE-9 samples. [2] L. Duval, M. Moreaud, C. Couprie, et al., Image processing for materials char-
acterization: issues, challenges and opportunities, 2014 IEEE International
In Fig. 13, we show some stitching results of VFSMS and Photoshop. Conference on, Image Processing (ICIP), IEEE, 2014, pp. 4862–4866.
For fairly comparison, we use collage mode in photomerge plugin. As [3] J. Hu, Y.N. Shi, X. Sauvage, et al., Grain boundary stability governs hardening and
same as ImageJ, Photoshop is a phase correlation based software. In softening in extremely fine nanograined metals, Science 355 (6331) (2017)
1292–1296.
addition, it picks up an improved version which can detect rotation of [4] Q. Wu, F. Merchant, K. Castleman, Microscope Image Processing, Academic press,
image. However, for most microscopic research, we commonly use 2010.
microscope equipped with motorized stage, which will not cause rota- [5] M. Sonka, V. Hlavac, R. Boyle, Image Processing, Analysis, and Machine Vision,
Cengage Learning, 2014.
tion in sequential micrographs. Unfortunately, it also yields ambiguous [6] F. Yang, Z.S. Deng, Q.H. Fan, A method for fast automated microscope image
results when micrographs contain much of similar microstructures. stitching, Micron 48 (2013) 17–25.
In Fig. 14, we show some stitching results of VFSMS and AutoStitch. [7] E. Adel, M. Elmogy, H. Elbakry, Image stitching based on feature extraction tech-
niques: a survey, Int. J. Comput. Appl 0975–8887 (2014).
AutoStitch is a feature based software. It is particularly suited to stitch
[8] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J.
images in nature scene and doesn’t have good results in micrograph Comput. Vision 60 (2) (2004) 91–110.
stitching. It employ SIFT, ransac and homography strategy which can [9] A.A. Fathima, R. Karthik, V. Vaidehi, Image stitching with combined moment in-
detect projective transformation in sequential images. However, those variants and sift features, Procedia Computer Science 19 (2013) 420–427.
[10] H. Bay, T. Tuytelaars, L. Van Gool, Surf: speeded up robust features, European
methods can cause large deformation when applied to micrograph Conference on Computer Vision, Springer, Berlin, Heidelberg, 2006, pp. 404–417.
stitching, as shown in dendritic crystal OM-1, iron polycrystal OM-2 [11] L. Juan, G. Oubong, SURF applied in panorama image stitching, 2010 2nd
and zircon SEM-TR-6 samples. Besides, Autostitch will abort stitch International Conference on Image Processing Theory Tools and Applications
(IPTA), IEEE, 2010, pp. 495–499.
when it detect mismatch, which is not suited to scientific research. By [12] E. Rublee, V. Rabaud, K. Konolige, et al., ORB: An efficient alternative to SIFT or
contrast, VFSMS will save previous stitching result and restart the stitch SURF, 2011 IEEE International Conference on Computer Vision (ICCV), IEEE, 2011,
on the next images when it faces mismatch. pp. 2564–2571.
[13] Y. Pang, W. Li, Y. Yuan, et al., Fully affine invariant SURF for image matching,
Neurocomputing 85 (2012) 6–10.
5. Conclusion [14] Y. Ruiyang, Research of Medical Microscopic Image Mosaic Based on SURF
Algorithm, Lanzhou University, 2014.
[15] M.A. Fischler, R.C. Bolles, A paradigm for model fitting with applications to image
In this work, in order to handle the task of micrograph stitching, we
analysis and automated cartography, Comm. ACM 24 (6) (1981) 381–395.
develop a very fast sequential micrograph stitching method, named [16] M. Brown, D.G. Lowe, Recognising panoramas, in: IEEE International Conference on
VFSMS, which employ incremental searching strategy and GPU accel- Computer Vision, 2003, 3, pp. 1218.
[17] M. Brown, D.G. Lowe, Automatic panoramic image stitching using invariant fea-
eration to guarantee the accuracy and speed of the stitching results. By
tures, Int. J. Comput. Vision 74 (1) (2007) 59–73.
using those strategies, we can stitch several local images with over- [18] F. Yang, Y. He, Z.S. Deng, et al., Improvement of automated image stitching system
lapping areas to form a final global composite one with huge resolution. for DR X-ray images, Comput. Biol. Med. 71 (2016) 108–114.
Experimental results demonstrate that VFSMS achieves state-of-art [19] B.S. Reddy, B.N. Chatterji, An FFT-based technique for translation, rotation, and
scale-invariant image registration, IEEE Trans. Image Process. 5 (8) (1996)
performance on three types’ microscopic datasets on both accuracy and 1266–1271.
speed aspects. Besides, it significantly outperforms the most famous and [20] H. Dogan, M. Ekinci, Automatic panorama with auto-focusing based on image fu-
commonly used software, such as ImageJ, Photoshop and Autostitch. sion for microscopic imaging system, Signal, Image Video Process. 8 (1) (2014)
5–20.
The limitation of VFSMS is that it only fuses local micrographs in [21] J. Schindelin, I. Arganda-Carreras, E. Frise, et al., Fiji: an open-source platform for
gray mode and doesn’t fuse micrographs in RGB channels, which will

12
B. Ma et al. Computational Materials Science 158 (2019) 1–13

biological-image analysis, Nature methods 9 (7) (2012) 676. [28] AutoStitch, Available online: < [Link]
[22] ImageJ: Available online: < [Link] > . [Link] > .
[23] David Biedny, The Official Adobe Photoshop Handbook, Random House Inc., 1991. [29] B. Ma, T. Zimmermann, M. Rohde, et al., Use of Autostitch for automatic stitching of
[24] Adobe PhotoShop CC 2018. Available online: < [Link] microscope images, Micron 38 (5) (2007) 492–499.
products/[Link] > . [30] W. Dan, L. Hui, L. Ke, An image fusion algorithm based on trigonometric functions,
[25] L. Carozza, A. Bevilacqua, F. Piccinini, An incremental method for mosaicing of Infrared Technol. 39 (1) (2017) 53–57.
optical microscope imagery, 2011 IEEE Symposium on Computational Intelligence [31] Python Software Foundation: Python language reference. [Link]
in Bioinformatics and Computational Biology (CIBCB), IEEE, 2011, pp. 1–6. [32] Numpy. Available online: [Link]
[26] J.N. Sarvaiya, S. Patnaik, K. Kothari, Image registration using log polar transform [33] R. Laganière, OpenCV 3 Computer Vision Application Programming Cookbook [M].
and phase correlation to recover higher scale, J. Pattern Recognition Res. 7 (1) Packt, Publishing Ltd, 2017.
(2012) 90–105. [34] Quadro K1200. Available online: < [Link]
[27] S. Preibisch, S. Saalfeld, P. Tomancak, Globally optimal stitching of tiled 3D mi- product-literature/data-sheets/11306_DS_NV_Quadro_K1200_FEB15_NV_US_HR.
croscopic image acquisitions, Bioinformatics 25 (11) (2009) 1463–1465. pdf > .

13