JOURNAL OF LATEX CLASS FILES, VOL. 14, NO.

8, AUGUST 2015 1

A Bio-Inspired Multi-Exposure Fusion Framework

for Low-light Image Enhancement
Zhenqiang Ying, Student Member, IEEE, Ge Li, Member, IEEE, and Wen Gao, Fellow, IEEE

Abstract—Low-light images are not conducive to human ob-

servation and computer vision algorithms due to their low
visibility. Although many image enhancement techniques have
been proposed to solve this problem, existing methods inevitably
arXiv:1711.00591v1 [cs.CV] 2 Nov 2017

introduce contrast under- and over-enhancement. Inspired by hu-

man visual system, we design a multi-exposure fusion framework
for low-light image enhancement. Based on the framework, we
propose a dual-exposure fusion algorithm to provide an accurate
contrast and lightness enhancement. Specifically, we first design
the weight matrix for image fusion using illumination estimation
techniques. Then we introduce our camera response model to (a) (b) (c)
synthesize multi-exposure images. Next, we find the best exposure
Fig. 1. Our mental image is a fused version of different eye exposure.
ratio so that the synthetic image is well-exposed in the regions (dynamically adjusting). (a) Eye focuses on background. (b) Eye focuses on
where the original image is under-exposed. Finally, the enhanced foreground. (c) Our mental image. [28]
result is obtained by fusing the input image and the synthetic
image according to the weight matrix. Experiments show that
our method can obtain results with less contrast and lightness
distortion compared to that of several state-of-the-art methods. reflect the true lightness and contrast of the scene. So, accurate
light and contrast enhancement based on a single image is still
Index Terms—Image enhancement, contrast enhancement, ex-
a challenging problem.
posure compensation, exposure fusion.
With a set of different exposure images in the same scene,
High Dynamic Range (HDR) techniques can synthesize im-
I. I NTRODUCTION ages that are close to the perceived scene. We know that the
camera and the human eye have a lot of similarities, then,
W ITH the development of photographic techniques, the
image quality is greatly improved in both resolution
and bit-depth. However, images captured by standard imag-
why we can perceive an image that is well-exposed every-
where while the camera cannot? The reason lies in the post-
ing devices often suffer from low visibility in non-uniform processing in our brains that have an image fusion mechanism
illuminated environments such as back lighting, nighttime and similar to HDR technique [28]. Human eye exposure changes
low-light indoor scene. Those images may lose information with its focus point, resulting in multi-exposure image set
in under-exposed regions, making the image content invisible which is then sent to the brain. Although each one of those
to human eyes. Since the camera dynamic range is limited, images suffers from under-exposed or over-exposed problem
if we increase camera exposure to reveal the information of in some regions, our brain can fuse these images into an image
under-exposed regions, the well-exposed regions will be over- that is free from under- and over-exposed problems, as shown
exposed or even saturated. To address the problem, many in Fig. 1.
image enhancement techniques have been proposed including Can we introduce this fusion mechanism of human visual
histogram-based methods [1], [2], [3], [4], [5], [6], [7], [8], system (HVS) to help build an accurate image enhancement
[9], [10], [11], [12], [13] , Retinex-based methods [14], [15], algorithm? Although many exposure fusion techniques have
[16], [17], [18], [19], [20], Logarithmic Image Processing been proposed in the domain of HDR, the additional images
methods [21], [22], and filtering-based methods [23], [24], taken with different exposures are often not available for the
[25], [26], [27]. Although some methods can obtain results low-light enhancement problem. Fortunately, those images are
with good subjective quality, those results may not accurately highly correlated. The mapping function between two images
that only differ in exposure is called Brightness Transform
This work was supported by the grant of National Science Foundation of Function (BTF). Therefore, we can first use BTF to generate
China (No.U1611461), Shenzhen Peacock Plan (20130408-183003656), and
Science and Technology Planning Project of Guangdong Province, China (No. a series of multi-exposure images and then fuse those images
2014B090910001). This paper was recommended by Associate Editor X. XX. to obtain the enhanced result.
Z. Ying, G. Li, and W. Gao are with the School of Electronic In this paper, we propose a multi-exposure fusion frame-
and Computer Engineering, Shenzhen Graduate School, Peking University,
518055 Shenzhen, China (e-mail: [email protected]; [email protected]; work inspired by the HVS. There are two stages in our
[email protected]). framework: Eye Exposure Adjustment and Eye Exposure
Color versions of one or more of the figures in this paper are available Adjustment. The first stage simulates the human eye to adjust
online at http://ieeexplore.ieee.org.
Digital Object Identifier XX.XXXX/TCYB.20XX.XXXXXXX the exposure, generating an multi-exposure image set. The
Manuscript received XXX XX, 20XX; revised XXX XX, 20XX. second stage simulates the human brain to fuse the generated
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 2

…
Generator

Evaluator Combiner

Input Image Sampler Output Image

Eye Exposure Adjustment Brain Exposure Fusion

Fig. 2. Our framework.

images into the final enhanced result. Based on our frame- because a global processing can not ensure all local areas be
work, we propose a dual-exposure fusion method. Specifically, well enhanced.
we first employ the illumination estimation techniques to
build the weight matrix for image fusion. Then we derives B. Local Enhancement Algorithms
our camera response model based on observation. Next, we
find the optimal exposure for our camera response model By making use of spatial information directly, local en-
to generate the synthetic image that is well-exposed in the hancement can achieve better results and become the main-
regions where the original image is under-exposed. Finally, we stream of recent techniques. Local histogram equalization
obtain the enhanced results by fusing the input image with the [36], [37] adopts the sliding window strategy to perform
synthetic image using the weight matrix. Experiments on five HE locally. Based on the observation that the inverted low-
challenging datasets are conducted to reveal the advantages of light images are closed to hazy images, dehazing techniques
our method in comparison with other state-of-the-art methods. are borrowed to solve low-light image enhancement in some
methods [38], [39]. However, the basic models of above
methods are lacking in physical explanation [14]. To provide
II. R ELATED W ORK a physical meaningful model for image enhancement, Retinex
In general, image enhancement techniques can improve the theory assumes that the amount of light reaching observers
subjective visual quality of input images and support the can be decomposed into two parts: illumination and scene
extraction of valuable information for some computer vision reflection. Most Retinex-based methods get enhanced results
techniques [29], [16]. Low-light image enhancement, as one by removing the illumination part [18], [20], [15] while others
of enhancement techniques, can reveal the information of the [40], [16], [14] keep a portion of the illumination to preserve
under-exposed regions in an image. Broadly speaking, existing naturalness. Fu et al. [41] adjust the illumination components
low-light image enhancement techniques can be divided into by fusing it with two enhanced illumination. As far as we
two major categories: global enhancement and local enhance- know, there is no multi-exposure fusion method for this task
ment. since lowlight enhancement problem usually takes a single
image as input.

A. Global Enhancement Algorithms

III. M ULTI -E XPOSURE F USION F RAMEWORK
Global enhancement performs same processing on all image Our framework mainly consists of four main components:
pixels regardless of their spatial distribution. Linear ampli- the first component, named Multi-Exposure Sampler, deter-
fying is a simple and straightforward global enhancement mines how many images are required and the exposure ratio of
method. However, bright regions might be saturated after each image to be fused; the second component, named Multi-
linear amplifying, causing some detail loss to the enhanced Exposure Generator, use a camera response model and the
results. To avoid the problem, some image enhancement specified exposure ratio to synthetic multi-exposure images;
methods adopt non-linear monotonic functions (e.g. power- the third component, named Multi-Exposure Evaluator, deter-
law [30], logarithm [31] and gamma function [32]) to perform mines the weight map of each image when fusing; the last
enhancements. As another way to avoid saturation, histogram component, named Multi-Exposure Combiner, is to fuse the
equalization (HE) [33] can improve the contrast effectively generated images to the final enhanced result based on the
and became a widely-used technique. Many extensions of HE weight maps. In this section, we introduce them one by one.
are proposed to take some restrictions into account such as
brightness preservation [29], [34], [10] and contrast limita-
tion [35]. By extending the notion of the histogram, some A. Multi-Exposure Sampler
algorithms take spatial image features into consideration to Before we generate multi-exposure images, we need to
further improve the performance [3], [6], [7]. However, global determine how many images are required and their exposure
enhancement may suffer from detail loss in some local areas ratios. Since some images in the multi-exposure set cannot
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 3

provide additional information, taking these images into con-

sideration is a waste of computation resources and may even
deteriorate the fused result. A good sampler can use as few
images as possible to reveal all the information in a scene
by choosing appropriate exposure ratios. The output of the (a) Input image (b) µ = 0 (c) µ = 0.5 (d) µ = 1
sampler is a set of exposure ratios {k1 , k2 , ...kN } where N is
the number of the images to be generated.

B. Multi-Exposure Generator
As aforementioned, images taken with different exposures (e) µ = 1.25 (f) µ = 1.75 (g) µ = 2.25
are correlated. Multi-Exposure Generator maps the input im-
Fig. 3. The enhanced results using different µ.
age into multi-exposure images according to the specified ex-
posure ratio set. The key part of the Multi-Exposure Generator
is the camera response model used to find an appropriate BTF A. Dual-Exposure Evaluator
for mapping. Given an exposure ratio ki and a BTF g, we can
The design of W is key to obtaining an enhancement
map the input image P to the i-th image in the exposure set
algorithm that can enhance the low contrast of under-exposed
as
regions while the contrast in well-exposed regions preserved.
Pi = g(P, ki ). (1)
We need to assign big weight values to well-exposed pixels
C. Multi-Exposure Evaluator and small weight values to under-exposed pixels. Intuitively,
the weight matrix is positively correlated with the scene illu-
To estimate the wellness of each pixel in the generated
mination. Since highly illuminated regions have big possibility
images. The Multi-Exposure Evaluator takes in an image and
of being well-exposed, they should be assign with big weight
outputs a weight matrix that indicates the wellness of each
values to preserve their contrast. We calculate the weight
pixel. The weight matrix is nonuniform for all pixels: the well-
matrix as
exposed pixels are given a big weight while the poor-exposed
Ŵ = Tµ (5)
pixels are given a small weight. After all images are evaluated,
the output matrices are pixel-wise normalized to ensure their where T is the scene illumination map and µ is a parameter
summation equals one for each pixel as controlling the enhance degree. When µ = 0, the resulting
N
X R is equal to P, i.e. , no enhancement is performed. When
f i = Wi 
W Wi , (2) µ = 1, both the under-exposed pixels and well-exposed pixels
i=1 are enhanced. When µ > 1, pixels may get saturated and the
where  is the element-wise division, and Wi and W f i are resulting R suffers from detail loss. As shown in Fig. 3. In
the i-th matrix before and after normalization, respectively. order to perform enhancement while preserve the well-exposed
regions, we set µ to 0.5. The scene illumination map T is
D. Multi-Exposure Combiner estimated by solving an optimization problem.
To obtain an image with all pixel well-exposed, we can 1) Optimization Problem: The lightness component can be
simply fuse these images based on the weight matrix as used as an estimation of scene illumination. We adopt the
N
lightness component as the initial estimation of illumination:
X
Rc = f i ◦ Pc ,
W i (3) L(x) = max Pc (x) (6)
i=1 c∈{R,G,B}

where c is the index of three color channels and R is for each individual pixel x. Ideal illumination should has local
the enhanced result. Other fusion techniques like multi-scale consistency for the regions with similar structures. In other
fusion [42] and Boosting Laplacian Pyramid fusion [43] can words, T should keep the meaningful structures of the image
also be used to obtain a better fusion results. and remove the textural edges. As in [14], we refine T by
IV. D UAL -E XPOSURE F USION A LGORITHM solving the following optimization equation:
In this section, we use the proposed framework to design a min ||T − L||22 + λ||M ◦ ∇T||1 , (7)
T
low-light image enhancement algorithm. To reduce complex-
ity, we only generate one image with appropriate exposure and where || ∗ ||2 and || ∗ ||1 are the `2 and `1 norm, respectively.
obtain the enhanced result by fusing the input image and the The first order derivative filter ∇ contains ∇h T (horizontal)
generated one. Based on our framework, the fused image is and ∇v T (vertical). M is the weight matrix and λ is the
defined as coefficient. The first term of this equation is to minimize the
difference between the initial map L and the refined map T,
Rc = Ŵ ◦ Pc + (1 − Ŵ) ◦ g(Pc , k̂). (4) while the second term maintains the smoothness of T.
The enhancement problem can be divided into three parts: The design of M is important for the illumination map
the determination of multi-exposure Evaluator (Ŵ), multi- refinement. A major edge in a local window contributes
exposure generator (g), and multi-exposure sampler (k̂). In more similar-direction gradients than textures with complex
the following subsections, we solve them one by one. patterns [44]. Therefore, The weight in a window that contains
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 4

first propose our BTF model based on the observation of two

different exposure images. Then we derive the corresponding
CRF model by solving the comparametric equation. Finally,
we discuss how to determine the model parameters and present
the final form of g.
(a) (b) (c) 1) BTF Estimation: To estimate the BTF g, we select a pair
of images P0 and P1 that differ only in exposure. Then we
plot their histograms of each color channel, as shown in Fig. 5.
Noticing that the histograms of the under-exposed image
mainly concentrate in low-brightness area, if we perform
linear amplification of pixel values before traditional gamma
(d) (e) correction, then the resulting image will be very close to
Fig. 4. (a) Input image. (b) Estimated illumination map by [14] (0.21s). (c) the real well-exposed image. Therefore, we can use a two-
Our illumination map (0.15s). (d) Enhanced result using (b). (e) Enhanced parameter function to describe the BTF model as
result using (c).
P1 = g(P0 , k) = βPγ0 , (11)
meaningful edges should be smaller than that in a window only where β and γ are parameters in our BTF model related to
containing textures. As a result, we design the weight matrix exposure ratio k. The observation also shows that different
as color channels have approximately same model parameters.
1 The underlying reason is that the response curves of different
Md (x) = P , d ∈ {h, v}, (8)
| y∈ω(x) ∇d L(y)| +  color channels are approximately identical for general cam-
eras.
where | ∗ | is the absolute value operator, ω(x) is the local 2) CRF Estimation: In our BTF model, β and γ are
window centered at the pixel x and  is a very small constant determined by the camera parameters and exposure ratio k.
to avoid the zero denominator. To find their relationship, we need to obtain the corresponding
2) Closed-Form Solution: To reduce the complexity, we CRF model. The CRF model can be derived by solving
approximate Eq. 7 as in [14]: the following comparametric equation (plug g = βf γ to
X Md (x) ∇d T(x) 2  f (kE) = g(f (E)):


X
2
min T(x) − L(x) + λ .
T
x
|∇d L(x)| +  f (kE) = βf (E)γ . (12)
d∈{h,v}
(9)
As can be seen, the problem now only involves quadratic The closed-form solution of f is provided in as follows (see
terms. Let md , l, t and ∇d l denote the vectorized version Appendix for detail):
of Md , L, T and ∇d L respectively. Then the solution can be
( a
eb(1−E ) , if γ 6= 1,
directly obtained by solving the following linear function. f (E) = (13)
X Ec, if γ = 1.
(I + λ (D|d Diag(md  (|∇d l| + ))Dd )t = l (10)
where a and b are model parameters in the case of γ 6= 1:
d∈{h,v}

where  is the element-wise division, I is the unit matrix, ln β

a = logk γ, b= ; (14)
the operator Diag(v) is to construct a diagonal matrix using 1−γ
vector v, and Dd are the Toeplitz matrices from the discrete And c is a model parameter in the case of γ = 1:
gradient operators with forward difference.
The main difference between our illumination map estima- c = logk β. (15)
tion method and that in [14] is the design of weight matrix M.
We adopt a simplified strategy which can yield similar results
as in [14]. As shown in Fig. 4. although the illumination map
in [14] is sharper than ours, our method is more time-saving
while the two enhanced results show no significant visual
difference.

B. Dual-Exposure Generator
In this section, we present a camera response model to
implement the Multi-Exposure Generator. A camera response
model consists of two parts: Camera Response Function (CRF)
Fig. 5. Observation. From left to right: An under-exposure image P0 , apply
model and BTF model. The parameters of CRF model is multiplication αP0 , apply gamma function (αP0 )γ and the well-exposure
determined only by camera while that of BTF model is deter- image P1 under the same scene. The histograms of red, green and blue color
mined by camera and exposure ratio. In this subsection, we channels are plotted above the corresponding image respectively.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 5

Fig. 6. Examples of multi-exposure image sets and their entropy values. The red bar on the left of each image shows the exponential of its entropy value.

Two CRF models can be derived from two cases of Eq. 13. arithmetic mean) since it has the same BTF model parameters
When γ = 1, the CRF model is a power function and the (β and γ) with all three color channels, as shown in Eq. 19.
BTF model is a simple linear function. As some camera q
manufacturers design f to be a gamma curve, it can fit B0 := 3 Q0r ◦ Q0g ◦ Q0b
these cameras perfectly. When γ 6= 1, the CRF model is a q
= 3 (βQγr ) ◦ (βQγg ) ◦ (βQγb ) = β( 3 Qr ◦ Qg ◦ Qb )γ
p
two-parameter function and the BTF model is a non-linear
function. Since the BTF is non-linear for most cameras, we = βBγ .
mainly consider the case of γ 6= 1. Our BTF g is solved as (19)
The visibility of a well-exposed image is higher than that
a a
g(P, k) = eb(1−k ) P(k ) . (16) of an under/over-exposed image and it can provide a richer
information for human. Thus, the optimal k should provide
where β and γ are two model parameters that can be calculated the largest amount of information. To measure the amount of
from camera parameters a, b and exposure ratio k. We assume information, we employ the image entropy which is defined
that no information about the camera is provided and use a as
N
fixed camera parameters (a = −0.3293, b = 1.1258) that can
X
H(B) = − pi · log2 pi , (20)
fit most cameras. i=1

where pi is the i-th bin of the histogram of B which counts

the number of data valued in [ Ni , i+1
N ) and N is the number
C. Dual-Exposure Sampler of bins (N is often set to be 256). As shown in Fig. 6, the
image entropy of a well-exposed image is higher than that
As aforementioned, our algorithm only generate one image.
of an under/over-exposed image. Therefore, it is reasonable
So, in this subsection, we only need to determinate the optimal
to use the entropy to find the optimal exposure ratio. The
exposure ratio of the generated image. In order to represent
optimal exposure ratio k̂ is calculated by maximizing the
as many information as possible using only the input image
image entropy of the enhancement brightness as
and the generated one, we find the best exposure ratio so that
the synthesis image is well-exposed in the regions where the k̂ = argmax H(g(B, k)). (21)
original image under-exposed. k

First, we exclude the well-exposed pixels and obtain an Since the image entropy increases first and then decreases with
image that is globally under-exposed. We simply extract the the increase of the exposure ratio, k̂ can be solved by one-
low illuminated pixels as dimensional minimizer. To improve the calculation efficiency,
we resize the input image to 50 × 50 when optimizing k.
Q = {P(x)|T(x) < 0.5}, (17)
V. E XPERIMENTS
where Q contains only the under-exposed pixels.
To evaluate the performance of our method, we com-
The brightness of the images under different exposures pare it with several state-of-the-art methods, including Multi
changes significantly while the color is basically the same. Scale Retinex with Color Restoration (MSRCR) [45], Natural-
Therefore, we only consider the brightness component when ness Preserved Enhancement algorithm (NPE) [16], dehazing
estimating k. The brightness component B is defined as the based method (Dong) [38], Multi-deviation Fusion method
geometric mean of three channel: (MF) [41], Illumination Estimation based method (LIME)
p [14] and Simultaneous Reflection and Illumination Estimation
B := 3
Qr ◦ Qg ◦ Qb , (18) (SRIE) [46]. We test those methods on hundreds of low-light
images from five public datasets: VV [47], LIME-data [14],
where Qr , Qg and Qb are the red, green and blue channel of NPE-data and its extension (NPE, NPE-ex1, NPE-ex2 and
the input image Q respectively. We use the geometric mean NPE-ex3) [16], DICM [3], and MEF [48]. The datasets are
instead of other definitions (e.g. arithmetic mean and weighted briefly introduced as follows:
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 6

Input MSRCR Dong NPE LIME MF SRIE Ours

Fig. 7. Comparison of lightness distortion. The odd rows show the original image and the results of various enhancement methods, and the even rows show
the visualization of each method’s lightness distortion (RD).

TABLE I
Q UANTITATIVE MEASUREMENT RESULTS OF LIGHTNESS DISTORTION (LOE).

VV LIME-data NPE-data NPE-ex1 NPE-ex2 NPE-ex3 DICM MEF

MSRCR 2727.7 1835.5 1889.7 1870.3 1944.7 1776.3 1795.3 1686.2
Dong 853.35 1244 1012 1426.1 1096.3 1466.2 1180 1065.4
NPE 820.93 1471.3 646.34 840.83 775.82 1130 662.29 1158.2
LIME 1274.7 1323.8 1119.6 1321.9 1215.4 1319.1 1260.8 1079.4
MF 470.93 629.82 488.07 851.87 541.85 749.72 667.45 525.95
SRIE 551.39 823.61 533.24 653.05 564.49 760.76 623.32 754.2
Ours 287.22 478.57 308.12 319.93 323.72 378.65 351.82 325.86

VV1 . This dataset is collected by Vassilios Vonikakis in his In order to maintain the fairness of the comparison, all the
daily life to provide the most challenging cases for enhance- codes are in Matlab and all the experiments are conducted on
ment. Each image in the dataset has a part that is correctly a PC running Windows 10 OS with 64G RAM and 3.4GHz
exposed and a part that is severely under/over-exposed. A CPU (GPU acceleration is not used). The parameters of our
good enhancement algorithm should enhance the under/over- enhancement algorithm are fixed in all experiments: λ = 1,
exposed regions while not affect the correctly exposed one.  = 0.001, µ = 1/2, and the size of local window ω(x) is 5.
LIME-data2 . This dataset contains 10 low-light images The most time-consuming part of our algorithm is illumination
used in [14]. map optimization. We employ the multi-resolution precondi-
NPE3 . This dataset contains 85 low-light images down- tioned conjugate gradient solver (O(N )) to solve it efficiently.
loaded from Internet. NPE-data contains 8 outdoor nature
scene images which are used in [16]. NPE-ex1, NPE-ex2 and A. Lightness Distortion
NPE-ex3 are three supplementary datasets including cloudy We use lightness order error (LOE) to objectively measure
daytime, daybreak, nightfall and nighttime scenes. the lightness distortion of enhanced results. LOE is defined as
DICM4 . It contains 69 captured images from commercial m
digital cameras collected by [3]. 1 X
LOE = RD(x) (22)
MEF5 . This dataset was provided by [48]. It contains m x=1
17 high-quality image sequences including natural sceneries, where RD(x) is the relative order difference of the lightness
indoor and outdoor views and man-made architectures. Each between the original image P and its enhanced version P 0 for
image sequence has several multi-exposure images, we select pixel x, which is defined as follows:
one of poor-exposed images as input to perform evaluation. m
X    
RD(x) = U L(x), L(y) ⊕ U L0 (x), L0 (y) , (23)
1 https://sites.google.com/site/vonikakis/datasets y=1
2 http://cs.tju.edu.cn/orgs/vision/∼xguo/LIME.htm
3 http://blog.sina.com.cn/s/blog a0a06f190101cvon.html where m is the pixel number, ⊕ stands for the exclusive-
4 http://mcl.korea.ac.kr/projects/LDR/LDR TEST IMAGES DICM.zip or operator, L(x) and L0 (x) are the lightness component at
5 https://ece.uwaterloo.ca/∼k29ma/ location x of the input images and the enhanced images,
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 7

Input MSRCR Dong NPE LIME MF SRIE Ours

Fig. 8. Comparison of contrast distortion. The loss of visible contrast is marked in green, the amplification of invisible contrast is marked in blue, and the
reversal of visible contrast is marked in red. Different shades of color represent different degrees of distortion.

TABLE II
Q UANTITATIVE MEASUREMENT RESULTS OF VISUAL INFORMATION FIDELITY (VIF).

VV LIME-data NPE-data NPE-ex1 NPE-ex2 NPE-ex3 DICM MEF

MSRCR 0.42134 0.24045 0.41425 0.29822 0.38625 0.65951 0.44966 0.27995
Dong 0.50477 0.32519 0.43440 0.38049 0.41687 0.50236 0.52637 0.35322
NPE 0.69006 0.50885 0.71471 0.58572 0.67769 0.74368 0.72497 0.52376
LIME 0.34932 0.20500 0.33934 0.28473 0.29498 0.38248 0.41498 0.22764
MF 0.72414 0.44752 0.63859 0.57687 0.61976 0.71747 0.70703 0.51293
SRIE 0.65968 0.52139 0.66528 0.58634 0.63547 0.69676 0.67866 0.55311
Ours 0.76098 0.74205 1.04930 0.70719 0.69787 0.76016 0.74524 0.60063

respectively. The function U (p, q) returns 1 if p >= q, 0 proposed method obtains the most realistic results with the
otherwise. least distortion.
As suggested in [14], [16], down-sampling is used to reduce
the complexity of computing LOE. We notice that LOE C. Visual Information Distortion
may change significantly when an image is down-sampled to
different sizes since RD will increase as the pixel number m To measure the distortion of visual information, we employ
increases. Therefore, we down-sample all images to a fixed Visual Information Fidelity (VIF) [50] in reverse mode. As a
size. Specifically, we collect 100 rows and columns evenly to full reference image quality assessment index, VIF models the
form a 100 × 100 down-sampled image. quality assessment problem as a information fidelity criterion
As shown in Table I, our algorithm outperforms the others in that quantifies the mutual information between the reference
all datasets. This means that our algorithm can maintain the image C and the distorted image F relative to the information
naturalness of images well. We also provide a visualization of C extracted by the HVS. VIF measure is given by
of lightness distortion on two cases in Fig. 7, from which, I(C; F )
we can find our results have the smallest lightness distortion. V IF = , (24)
I(C; E)
The results of MSRCR lose the global lightness order and
suffer from severe lightness distortion. Although the results of where E is the image that the HVS perceives. The mutual
LIME is visually pleasant, they are full of lightness distortion. information I(C; F ) and I(C; E) represent the information
The results of Dong, NPE, MF and SRIE can only retain the that could be extracted by the brain in the reference and the
lightness order in the well-exposed regions. test images respectively.
Like most of full reference image quality assessment meth-
ods, VIF were designed for and tested on degraded images.
B. Contrast Distortion The normal version of VIF treats the original image as the
As aforementioned, the image that only differ in exposures reference image and the image outputted by algorithm as the
can be used as a reference for evaluating the accuracy of degraded image. For image enhancement problem, however,
enhanced results. DRIM (Dynamic Range Independent Metric) the original image is the degraded one. Therefore, we employ
[49] can measure the distortion of image contrast without VIF in reverse mode by specifying the enhanced version of the
the interference of change in image brightness. We use it image as the reference and the original image as the degraded
to visualize the contrast difference between the enhanced image. VIF provides consistently high value of correlation be-
result and the reference image. As shown in Fig. 8, the tween subjective MOS (Mean Opinion Score) and its scores, as
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 8

Input MSRCR Dong NPE LIME MF SRIE Ours

Fig. 9. Visual comparison among the competitors on different scenes.

shown in [51]. Besides, it is suitable for automating evaluation images), it may make the results look whitish. Besides, in
process of nonlinear image enhancement algorithms [52]. some dark show halo artifacts around sharp edges. The results
As shown in Table II, our algorithm outperforms the others of MSRCR shows severe halo artifacts around sharp edges and
in all datasets. This means that our algorithm can maintain the obvious noise in very dark areas (see the two astronauts) and
visual information of images well. suffer from detail loss in some bright areas (see the Christmas
hat). The results of Dong is noisy and full of bold edges
D. Time Cost making it look like exaggerated art paintings. The results
of LIME is so bright that many bright areas are saturated.
Fig. 10 gives the comparison among different methods Also, the noise in dark areas are amplified and de-noising
in terms of time cost. Although SRIE and NPE produce method are therefore required to obtain better results. MF may
small distortion, they are quite time-consuming. Our method introduce color over-enhancement (see the ground beneath the
achieves the smallest distortion than others with an acceptable astronauts’ feet) and SRIE may produce slight halo effects in
time cost. some edges (see the Christmas hat).

E. Subjective Evaluation VI. L IMITATION A ND F UTURE W ORK

Fig. 9 shows more examples for visual comparison. Al- Fig. 11 shows a failure case of our technique that the hair of
though the color correction post-processing in MSRCR can the man turns to be grey because of over-enhancement. This is
handle the color cast in some cases (e.g. underwater and hazy due to the dark area behind his head blending with his black
hair. As shown in Fig. 11 (c), the hair is mistaken as the dark
background in the estimated illumination map and therefore
104
3.5
SRIE
is enhanced along with the background. Such mistake is a
3
NPE
MSRCR
result of the existing illumination map estimation techniques.
Dong

2.5
LIME This highlights a direction for future work. To avoid the
MF
Ours over-enhancement due to the ignorance of the scene content,
Run time (ms)

2
semantic understanding is required. With further refinement,
1.5
we might employ the deep learning techniques to estimate the
1 illumination map.
0.5
Besides, we only use two images to obtain the enhanced
0
result. The over-exposure problem is remain unsolved. Im-
0 0.2 0.4 0.6 0.8 1 1.2 1.4
Image Size (width × height)
1.6 1.8 2
106
ages with smaller exposures than the input image should be
considered in our framework to obtain a better result. We will
Fig. 10. Time comparison among different methods with varying image sizes address this problem as future work.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 9

where a and b are model parameters in the case of γ 6= 1:

ln β
a = logk γ, b= ; (30)
1−γ
And c is model parameter in the case of γ = 1:

(a) (b) (c) c = logk β. (31)

Fig. 11. An example of a failure case. (a) Raw image. (b) Enhanced result.
(c) Estimated illumination T. R EFERENCES
[1] H. Xu, G. Zhai, X. Wu, and X. Yang, “Generalized equalization model
for image enhancement,” IEEE Transactions on Multimedia, vol. 16,
VII. C ONCLUSION no. 1, pp. 68–82, 2014.
In this paper, we propose an exposure fusion framework [2] T. Celik, “Spatial entropy-based global and local image contrast en-
hancement,” IEEE Transactions on Image Processing, vol. 23, no. 12,
and an enhancement algorithm to provide an accurate contrast pp. 5298–5308, 2014.
enhancement. Based on our framework, we solve three prob- [3] C. Lee, C. Lee, and C.-S. Kim, “Contrast enhancement based on layered
lems: 1) We borrow the illumination estimation techniques to difference representation,” in Image Processing (ICIP), 2012 19th IEEE
International Conference on. IEEE, 2012, pp. 965–968.
obtain the weight matrix for image fusion. 2) We introduce our [4] K. Arya, M. Pattanaik et al., “Histogram statistics based variance
camera response model to synthesize multi-exposure images. controlled adaptive threshold in anisotropic diffusion for low contrast
3) We find the best exposure ratio so that the synthetic image image enhancement,” Signal Processing, vol. 93, no. 6, pp. 1684–1693,
2013.
is well-exposed in the regions where the original image under- [5] C. Lee, C. Lee, Y.-Y. Lee, and C.-S. Kim, “Power-constrained contrast
exposed. The final enhanced result is obtained by fusing the enhancement for emissive displays based on histogram equalization,”
input image and the synthetic image according to the weight IEEE transactions on image processing, vol. 21, no. 1, pp. 80–93, 2012.
[6] T. Celik and T. Tjahjadi, “Contextual and variational contrast enhance-
matrix. The experimental results have revealed the advance of ment,” IEEE Transactions on Image Processing, vol. 20, no. 12, pp.
our method compared with several state-of-the-art alternatives. 3431–3441, 2011.
To encourage future works and allow more experimental [7] T. Arici, S. Dikbas, and Y. Altunbasak, “A histogram modification
framework and its application for image contrast enhancement,” IEEE
verification and comparisons, we make the source code open Transactions on image processing, vol. 18, no. 9, pp. 1921–1935, 2009.
on our project website6 , as well as the relevant test code used [8] D. Coltuc, P. Bolon, and J.-M. Chassery, “Exact histogram specification,”
to reproduce the experiment in the paper. We also provide IEEE Transactions on Image Processing, vol. 15, no. 5, pp. 1143–1152,
2006.
the results of other competitors to facilitate the validation of [9] S.-D. Chen and A. R. Ramli, “Contrast enhancement using recursive
quality measures. mean-separate histogram equalization for scalable brightness preserva-
tion,” IEEE Transactions on Consumer Electronics, vol. 49, no. 4, pp.
1301–1309, 2003.
A PPENDIX [10] ——, “Minimum mean brightness error bi-histogram equalization in
D ERIVATION OF C AMERA R ESPONSE M ODEL contrast enhancement,” IEEE transactions on Consumer Electronics,
vol. 49, no. 4, pp. 1310–1319, 2003.
The CRF model can be derived by solving the follow- [11] S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz,
ing comparametric equation (plug g = βf γ to f (kE) = T. Greer, B. ter Haar Romeny, J. B. Zimmerman, and K. Zuiderveld,
g(f (E), k): “Adaptive histogram equalization and its variations,” Computer vision,
graphics, and image processing, vol. 39, no. 3, pp. 355–368, 1987.
f (kE) = βf (E)γ . (25) [12] E. D. Pisano, S. Zong, B. M. Hemminger, M. DeLuca, R. E. Johnston,
K. Muller, M. P. Braeuning, and S. M. Pizer, “Contrast limited adaptive
Since β is positive, we can take the logarithm of both sides histogram equalization image processing to improve the detection of
of Eq. 12: simulated spiculations in dense mammograms,” Journal of Digital
ln f (kE) = ln β + γ ln f (E). (26) imaging, vol. 11, no. 4, pp. 193–200, 1998.
[13] Y.-T. Kim, “Contrast enhancement using brightness preserving bi-
Differentiate with respect to E, we get histogram equalization,” IEEE transactions on Consumer Electronics,
vol. 43, no. 1, pp. 1–8, 1997.
f 0 (kE) f 0 (E) [14] X. Guo, Y. Li, and H. Ling, “Lime: Low-light image enhancement via
k =γ . (27) illumination map estimation,” IEEE Transactions on Image Processing,
f (kE) f (E) vol. 26, no. 2, pp. 982–993, 2017.
0 [15] L. Wang, L. Xiao, H. Liu, and Z. Wei, “Variational bayesian method
The power function ff (kE)
(kE)
= C1 E a−1 where a = logk γ can for retinex,” IEEE Transactions on Image Processing, vol. 23, no. 8, pp.
be a solution of Eq. 27. Then f can be solved by integration 3381–3396, 2014.
as [16] S. Wang, J. Zheng, H.-M. Hu, and B. Li, “Naturalness preserved
Ea
(
enhancement algorithm for non-uniform illumination images,” IEEE
eC1 a +C2 , if γ 6= 1, Transactions on Image Processing, vol. 22, no. 9, pp. 3538–3548, 2013.
f (E) = (28)
C3 E C1 , if γ = 1. [17] R. Kimmel, M. Elad, D. Shaked, R. Keshet, and I. Sobel, “A varia-
tional framework for retinex,” International Journal of computer vision,
The constants C1 , C2 and C3 can be determined by the vol. 52, no. 1, pp. 7–23, 2003.
[18] D. J. Jobson, Z.-u. Rahman, and G. A. Woodell, “Properties and
restriction of f (1) = 1 and f (k) = β (plug f (1) = 1 to performance of a center/surround retinex,” IEEE transactions on image
Eq. 12) and the resulting f is obtained as follows: processing, vol. 6, no. 3, pp. 451–462, 1997.
( a [19] M. K. Ng and W. Wang, “A total variation model for retinex,” SIAM
eb(1−E ) , if γ 6= 1, Journal on Imaging Sciences, vol. 4, no. 1, pp. 345–365, 2011.
f (E) = (29) [20] D. J. Jobson, Z.-u. Rahman, and G. A. Woodell, “A multiscale retinex
Ec, if γ = 1, for bridging the gap between color images and the human observation
of scenes,” IEEE Transactions on Image processing, vol. 6, no. 7, pp.
6 https://baidut.github.io/BIMEF. 965–976, 1997.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 10

[21] K. Panetta, S. Agaian, Y. Zhou, and E. J. Wharton, “Parameterized [46] X. Fu, D. Zeng, Y. Huang, X.-P. Zhang, and X. Ding, “A weighted vari-
logarithmic framework for image enhancement,” IEEE Transactions on ational model for simultaneous reflectance and illumination estimation,”
Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 41, no. 2, pp. in Proceedings of the IEEE Conference on Computer Vision and Pattern
460–473, 2011. Recognition, 2016, pp. 2782–2790.
[22] K. A. Panetta, E. J. Wharton, and S. S. Agaian, “Human visual system- [47] V. Vonikakis, R. Kouskouridas, and A. Gasteratos, “On the evaluation
based image enhancement and logarithmic contrast measure,” IEEE of illumination compensation algorithms,” Multimedia Tools and Appli-
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), cations, pp. 1–21, 2017.
vol. 38, no. 1, pp. 174–188, 2008. [48] K. Ma, K. Zeng, and Z. Wang, “Perceptual quality assessment for
[23] Y. Wang, S. Zhuo, D. Tao, J. Bu, and N. Li, “Automatic local exposure multi-exposure image fusion,” IEEE Transactions on Image Processing,
correction using bright channel prior for under-exposed images,” Signal vol. 24, no. 11, pp. 3345–3356, 2015.
Processing, vol. 93, no. 11, pp. 3227–3238, 2013. [49] T. O. Aydin, R. Mantiuk, K. Myszkowski, and H.-P. Seidel, “Dynamic
[24] L. Yuan and J. Sun, “Automatic exposure correction of consumer range independent image quality assessment,” ACM Transactions on
photographs,” Computer Vision–ECCV 2012, pp. 771–785, 2012. Graphics (TOG), vol. 27, no. 3, p. 69, 2008.
[25] G. Deng, “A generalized unsharp masking algorithm,” IEEE transactions [50] H. R. Sheikh and A. C. Bovik, “Image information and visual quality,”
on Image Processing, vol. 20, no. 5, pp. 1249–1261, 2011. IEEE Transactions on image processing, vol. 15, no. 2, pp. 430–444,
[26] Q. Shan, J. Jia, and M. S. Brown, “Globally optimized linear windowed 2006.
tone mapping,” IEEE transactions on visualization and computer graph- [51] K. Gu, G. Zhai, W. Lin, and M. Liu, “The analysis of image contrast:
ics, vol. 16, no. 4, pp. 663–675, 2010. From quality assessment to automatic enhancement,” IEEE transactions
[27] J.-H. Hong, S.-B. Cho, and U.-K. Cho, “A novel evolutionary approach on cybernetics, vol. 46, no. 1, pp. 284–297, 2016.
to image enhancement filter design: method and applications,” IEEE [52] P. A. Kumar and P. Sankaran, “Visual information fidelity in evaluating
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), retinex enhancement algorithms,” in Communications and Signal Pro-
vol. 39, no. 6, pp. 1446–1457, 2009. cessing (ICCSP), 2014 International Conference on. IEEE, 2014, pp.
[28] S. T. McHugh, Understanding Photography. Cambridge in Colour, 167–171.
2016.
[29] H. Ibrahim and N. S. P. Kong, “Brightness preserving dynamic histogram
equalization for image contrast enhancement,” IEEE Transactions on
Consumer Electronics, vol. 53, no. 4, pp. 1752–1758, 2007.
[30] A. Beghdadi and A. Le Negrate, “Contrast enhancement technique based
on local detection of edges,” Computer Vision, Graphics, and Image
Processing, vol. 46, no. 2, pp. 162–174, 1989.
[31] E. Peli, “Contrast in complex images,” JOSA A, vol. 7, no. 10, pp.
2032–2040, 1990.
[32] R. Gonzalez and R. Woods, “Digital image processing: Pearson prentice
hall,” Upper Saddle River, NJ, 2008.
[33] A. K. Jain, Fundamentals of digital image processing. Prentice-Hall,
Inc., 1989.
[34] C. Wang and Z. Ye, “Brightness preserving histogram equalization with
maximum entropy: a variational perspective,” IEEE Transactions on
Consumer Electronics, vol. 51, no. 4, pp. 1326–1334, 2005.
[35] A. M. Reza, “Realization of the contrast limited adaptive histogram
equalization (clahe) for real-time image enhancement,” Journal of VLSI
signal processing systems for signal, image and video technology,
vol. 38, no. 1, pp. 35–44, 2004.
[36] J. A. Stark, “Adaptive image contrast enhancement using generalizations
of histogram equalization,” IEEE Transactions on image processing,
vol. 9, no. 5, pp. 889–896, 2000.
[37] M. Abdullah-Al-Wadud, M. H. Kabir, M. A. A. Dewan, and O. Chae,
“A dynamic histogram equalization for image contrast enhancement,”
IEEE Transactions on Consumer Electronics, vol. 53, no. 2, 2007.
[38] X. Dong, G. Wang, Y. Pang, W. Li, J. Wen, W. Meng, and Y. Lu, “Fast
efficient algorithm for enhancement of low lighting video,” in 2011 IEEE
International Conference on Multimedia and Expo. IEEE, 2011, pp.
1–6.
[39] L. Li, R. Wang, W. Wang, and W. Gao, “A low-light image enhancement
method for both denoising and contrast enlarging,” in Image Processing
(ICIP), 2015 IEEE International Conference on. IEEE, 2015, pp. 3730–
3734.
[40] X. Fu, D. Zeng, Y. Huang, X.-P. Zhang, and X. Ding, “A weighted vari-
ational model for simultaneous reflectance and illumination estimation,”
in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2016, pp. 2782–2790.
[41] X. Fu, D. Zeng, Y. Huang, Y. Liao, X. Ding, and J. Paisley, “A
fusion-based enhancing method for weakly illuminated images,” Signal
Processing, vol. 129, pp. 82–96, 2016.
[42] T. Mertens, J. Kautz, and F. Van Reeth, “Exposure fusion: A simple and
practical alternative to high dynamic range photography,” in Computer
Graphics Forum, vol. 28, no. 1. Wiley Online Library, 2009, pp. 161–
171.
[43] J. Shen, Y. Zhao, S. Yan, X. Li et al., “Exposure fusion using boosting
laplacian pyramid.” IEEE transactions on cybernetics, vol. 44, no. 9,
pp. 1579–1590, 2014.
[44] L. Xu, Q. Yan, Y. Xia, and J. Jia, “Structure extraction from texture via
relative total variation,” ACM Transactions on Graphics (TOG), vol. 31,
no. 6, p. 139, 2012.
[45] A. B. Petro, C. Sbert, and J.-M. Morel, “Multiscale retinex,” Image
Processing On Line, pp. 71–88, 2014.