2012 International Conference on Systems and Informatics (ICSAI 2012)

CMOS Image Sensor Interface

Controller Design for Video Surveillance

Qian Tian Zhixiao Jin

National ASIC System and Engineering Center Integration Circuits College
Southeast University Southeast University
Nanjing, China Nanjing, China

Abstract—In this paper, a design of CMOS image sensor

interface controller based on FPGA is presented. The controller II. STRUCTURES OF THE VIDEO SURVEILLANCE SYSTEM
works as an interface between CMOS image sensors and other The purpose of the video surveillance system is to acquire
units for video surveillance. The top-down design method is image information from CMOS image sensors, store and
adopted to partition and design each sub-module of the controller. transmit the real-time image information in distortion. Finally,
This controller can perform the functions of acquiring,
this system provides the real-time and distorted image sources
processing and storing information from image sensors. The
for receivers or users. The main part of the system is made
asynchronous FIFOs, the three-way division unit and the
embedded DMA are designed to improve the speed of data using system on chip (SoC) design method. It includes ACORE
transmission and avoid the conflicts of data transmission on AHB. designed by the ASIC team in southeast university, ESRAM,
It is successfully implemented on the video surveillance system DMA, CMOS interface controller, SD card controller and other
based on a FPGA board. The simulating and verification results units. Fig.1 shows the system structures.
show that the image can be captured and transmitted without
distortion.

Keywords-CMOS image sensor; interface controller;

Asynchronous FIFO;

I. INTRODUCTION
The advances of low power and low cost hardware (i.e.,
micro-electro-mechanical systems (MEMS)) and wireless
communications result in the growing focus on wireless sensor
networks (WSNs) during the decade years [1-3]. WSNs have
been applied to many fields including human healthcare [4],
intelligent transportation system (ITS) [5], security monitoring
[2] and other industrial scenarios. Recently, the availability of Figure 1. The main structures of the video surveillance system.
complementary metal-oxide semiconductor (CMOS) camera
and small microphones make it possible WSNs capable of In Fig.1, ACORE is a kind of processor core. The units of
gathering video and audio information from the surrounding AHB and APB support AMBA high speed bus and AMBA low
environment [1], which is called wireless multimedia sensor speed bus respectively. INTC is an interrupt unit, which
networks (WMSNs). The advantage of WMSNs helps to manages the interrupt request from the CMOS image interface
capture more details of the scenarios such as the congest controller (CMOS ISI) and other units. I2C is the unit that
statuses in ITS, invasion features in insurance monitoring supports the transmission protocol named I2C. When the
systems and so on. system is powered on, the units of INTC and I2C are initialized
In WMSNs, how to acquire raw data from camera sensors in turn. Then, CMOS sensors with I2C interfaces can be
without time delay and distortion is a key point. In this paper, a configured by I2C controller. As the system is working, the
new design of an interface controller for CMOS image sensors image data from sensors outside are transmitted to CMOS ISI
(CMOS ISI) is proposed. This controller is a basic module as in which those data are processed to be suitable for storage in
the whole system of a video surveillance system based on the buffer memories. Finally, the image data are transmitted to SD
FPGA board. But it provides the accurate and real-time data. cards by the SD controller in the system.
Therefore, the controller design plays a significant role for the
whole system performance. In the design, CMOS ISI acquires III. CMOS ISI DESIGN
raw data from CMOS image sensors, synchronously transmits This CMOS ISI unit supports both of ITU-R BT.65 and
the data format to the standard format, and de-interleaves the ITU-R BT.601 digital video synchronous formats, and also
image data for storage in specific memories. supports QVGA and QCIF graphic formats. The processing
speed is above 15 frames per second and can be adjusted. It

2165 978-1-4673-0199-2/12/$31.00 ©2012 IEEE

 also supports the transformation of YCrCb422 and YCrCb420.
Its storage format is the planner mode of YCbCr420. These
function requirements are satisfied by sub modules such as
asynchronous FIFOs, conversions and so on. As a design task
of the whole system, this unit also has to include AHB
interface. The structure diagram of CMOS ISI is shown in Fig.
2.

Figure 3. ITU-R BY.601 timing diagram

Figure 4. ITU-R 656 timing diagram

Figure 2. The CMOS ISI structure diagram. B. Asynchronous FIFO

Transferring data between different clock domains will
A. Format 656 to 601 generate meta-stability phenomenon. This is because trigger
Both of ITU-R BT.656 and ITU-R BT. 601 are the signals and the clock signal can’t satisfy the requirement of
recommended standards for digital video in ITU. ITU-R BT. setup/hold time and the output of triggers are in unstable status.
601 specifies digital coding methods of video signals, using the When the trigger is in meta-stability status, its output can’t be
color space of YCbCr to make best use of channel bandwidth. decided. During this period, the trigger is oscillating, which
The format of 4:2:2 YCbCr is the first choice. Besides, several causes cascade propagation of the useless output. Therefore,
synchronous signals are provided including horizontal when a signal is from one clock domain to another clock
synchronous signal, vertical synchronous signal and so forth to domain, the receiving circuit has to synchronize the signal to
define the boundaries in active video area. Fig.3 shows the avoid the propagation of meta-stability in clock domains.
typical timing relationship among synchronous signals, timing In CMOS ISI unit, the image data in the clock domain of
clock signal and other signals. ITU-R BT. 601 also specifies image sensors and the horizontal and vertical synchronous
the value of Y within its ratings from 16(totally black) to signals have to be synchronized to the clock domain in the
235(totally white), the values of Cb and Cr within their ratings system. Because both clock domains have no time relation, two
from 16 to 240. On the other hand, ITU-R BT.656 defines the methods of handshake and asynchronous FIFO can be used to
physical interfaces and data stream format to implement ITU-R avoid meta-stability. Considering the quantities and real time of
BT. 601, and defines bit parallel and bit serial modes. Bit the image data in this system, several asynchronous FIFOs are
parallel mode requires clock signal and eight or ten bits data designed to store and buffer the transferring data between
signals depending on pixel resolution. All the synchronous different clock domains as real-time data interfaces. The
signals are in data stream without extra hardware wires. Its asynchronous FIFO consists of three parts: dual- port RAM,
timing diagram is shown in Fig.4. read/write address generation, read/write pointer
The unit of format 656 to 601 is to transfer the image data synchronization [7]. Its structure diagram is shown in Fig.5.
from sensors from ITU-R BT.656 to ITU-R BT.601. If the In Fig.5, the read pointer Rptr and the write pointer Wptr
input data is in the format of ITU-R BT.656 standard, the four are coded in GRAY code format. Because only one bit changes
synchronous flags in data stream including frame start flag, each time in GRAY code, the probability of meta-stability
frame end flag, horizontal start flag and horizontal end flag are greatly decreases. The signal of Sync_R2W synchronizes the
used to implement transformation. Furthermore, the read pointer Rptr to the write clock domain, and then compares
polarizations of the four synchronous signals are configurable. it with the write pointer Wptr to generate the full flag signal of
If the input data is in the format of ITU-R BT.601, they are by- the FIFO. The signal of Sync_W2R synchronizes the write
pass through this unit [6]. pointer Wptr to the read clock domain, and then compares it

2166
with the read pointer Rptr to generate the empty flag signal of FIFO and the data of Cr in even lines are stored in FIFO. The
the FIFO. first method will increase extra buffers to store data. In contrast,
the second method will lose some data. But considering the
system applications, the effects of the losing data can be
ignored. The second method can also lower the system power
which is important for the wireless surveillance terminals.
Therefore, this unit uses the second method to divide the raw
data into three ways of Y, Cr, and Cb.

D. AHB interfaces
Quantities of image data have to be stored into ESRAM so that
the interface unit has to be connected to high speed bus with
high performance. The master interface unit implements the
storage, transmitting data through AHB including address
signals and relative control signals to the specified space in
ESRAM. This method implements the DMA transmission. The
storage mode is Y/Cr/ Cb planer mode, so it is equal to three
Figure 5. Asynchronous FIFO structure channel DMA transmission. In this case, this unit work as a
master device connected to AHB, making a write data
operation to AHB according to the configuration of the slave
C. Three-way division
device.
This unit is to de-interleave the data stream
CbYCrYCbYCrY……from image sensors, separating them in The slave interface unit completes the configuration of
Y, Cr, and Cb respectively and storing them into three FIFOs: registers and status output. When ACORE sets parameters to
FIFO_Y, FIFO_Cb and FIFO_Cr to implement the function of the interface controller, ACORE write data to the slave
“ping-pong” operation. Furthermore, it can transfer YCbCr 422 interface device through AHB to configure the working mode
to YCbCr 420 by respectively generating write-FIFO-enable of each sub unit. Besides, ACORE obtains the current status of
signals of Cb and Cr in parity lines. the interface device. In this case, the interface controller works
as a slave device connected to AHB.
Because human visual system is more sensitive to
illumination than chroma, illumination data can be separated E. Others
from chroma data to obtain more image definition and more
shadow bandwidth [8]. In this unit, the storage bandwidth of There are also the global synchronization sub-unit and
YCbCr422 is twice of data, while the storage bandwidth of clock-frequency division sub-unit in the interface control unit.
YCbCr420 is one and a half of data. Therefore, using YCbCr The global synchronous unit is designed to synchronize the
420 mode can shorten 25% system storage bandwidth. There cross-clock-domain control and status signals using the two-
are two methods to transfer data from YCbCr422 mode to level registers and knotting method. On the other hand, the
YCbCr420 mode. The first method uses mean shown in Eq. (1). clock frequency division clock is designed to output the
fractional frequency clock signals to image sensors as their
Cb’(2i-1)＝0 clock input.
Cb’(2i) = (Cb(2i-1)＋Cb(2i))/2
Cr’(2i-1)＝0 (1) IV. SIMULATION AND VERIFICATION
Cr’(2i) = (Cr(2i-1)＋Cr(2i))/2 A. Module-level Simulation
According to the data stream, each sub unit is simulated
Here, Cb’ and Cr’ are the converted values, and i is the line in sequence. Fig.6 shows that FORMAT 656 TO 601 unit
number. In the circuit, the chroma data from the first line is separates data and control signals, and extracts the
locked in the inner buffers, when the data from the second line horizontal/vertical signals of cam_hsync_o/cam_vsync_o, and
are received, the first line data from the buffers and the second active image data of cam_data_o using the input data of
line data are computed according to Eq.(1). The second method cam_data_i in ITU-R BT.656 format.
is to sample interlace chroma data to make computation
according to Eq. (2).
Cb’(2i-1) = Cb(2i-1)
Cb’(2i)=0
Cr’(2i-1)=0 (2)
Cr’(2i) = Cr(2i)

Figure 6. FORMAT 656 TO 601 unit simulation results.

Here, Cb’ and Cr’ are the converted values, and i is the line
number. In the circuit, the data of Cb in odd lines are stored in

2167
 Fig. 7 shows that the three-way division unit de-interleaves g) Initializing the image sensor interface control unit;
the interleave data of Y/Cb/Cr in image data of cam_data_i
h) Opening the frame interrupt of the image sensor
into the data of ifo_data_Y/Cb/Cr_o in three channels. The interface control unit;
three-channel data are output to the asynchronous FIFOs. On
the other hand, this unit sets the write enable signals of i) Enabling the image sensor interface control unit;
fifo_wr_Y/Cb_en_o for Y and Cb signals active in odd lines j) Enabling the interrupt signals in INTC for the image
and sets the write enable signals of fifo_wr_Y/Cr_en_o for Y sensor interface control unit;
and Cr active in even lines. Finally, the interlace sampling of
chroma signals is implemented. k) Opening the interrupt signals in INTC for the image
sensor interface control unit;
l) Waiting for interrupting;
When a frame of image data is transmitted completely, the
frame interrupt signal is generated, the program jumps to the
interrupt processing function. In this function, DMA and SD
card model are configured to complete the transmission from
ESRAM to SD card.

Figure 7. Three-way division unit simulation results. C. FPGA Verifications

The Straix II serial chip with the type of EP2S130 is used
In Fig.8, when the write enable signal of write_en_Y_i for FPGA verification. This chip has 12 PLL units, 63 DSP
active, the image data of wdata_Y_i driven by the clock are units, up to 53,016 adaptive logic modules (ALM), and
written to the asynchronous FIFO. When the number of input 6,747,840 bits memories, which provide strong support for the
data reaches the flip-flop stage, the signal of interface verification. The module consisting of OV7670 chip
fifo_level_trigged_Y_o is set to be active by the FIFO. Then, producted in OmniVision company is also used in the
the read enable signal of read_en_Y_i is set to be active by the verification. OV7670 has the merits of small size, low power
AHB master interface. In this case, the data of rdata_Y_o is and high accuracy so that it is suitable for low illumination
output driven by the read clock. intensity in embedded systems [10]. The verification system
based FPGA is shown in Fig.9. The output window is
configured to be in QVGA, the output format is YCbCr422,
and the frame rate is 15fps. A 2G SD card is used to store
processed image data. Finally, the results are shown in PC.
Fig.10 shows the result.

Figure 8. Asynchronous FIFO unit simulation results.

B. System Level Simulation

The interfaces and interworking of each module in the
system are simulated by simulating the real environment of a
system on chip [9]. First, the assemble language and C
language are used to code the test programs for the system Figure 9. The Verification System based on FPGA
simulation. Second, the compiled binary machine codes runs
on the ACORE. Finally, the machine code file is downloaded
by the system task on the test bench into the memories. The
main test functions include:
a) Masking all input interrupt signals;
b) Initializing the SD card model;
c) Initializing I2C unit;
d) Configuring sensor images by I2C;
e) Disabling the image interface control unit;
f) Closing all the interrupts from the image sensor Figure 10. The Verification Result.
interface control unit;

2168
 V. CONCLUSION AND FURTHER WORK [2] M. Alnuaimi, F. Sallabi, and K. Shuaib, “A Survey of Wireless
Multimedia Sensor Networks Challenges and Solutions,” International
This paper proposes a new design of the CMOS image Conference on Innovations in Information Technology, pp. 191-196,
interface controller using the method of top-down design April 2011.
methodology. This controller as a part task of the whole system [3] Cheng Chen, Bin Tian and et al., “Data Aggregation Technologies of
satisfies the logic and timing requirements of the system. Wireless Multimedia Sensor Networks: A Survey,” IEEE International
Conference on ICVES, pp.83-88, July 2011.
Furthermore, in the controller the Asynchronous FIFO and the
three-way division unit are designed to transmit the processed [4] Chao Sha, Ru-chuan Wang, and et al., “A type of healthcare system
based on intelligent wireless sensor networks,” The Journal of China
data to the memory for the fast data transmission and avoid the Universities of Posts and Telecommunications, 2010
conflict with other data transmission on AHB. The simulating [5] C. Arth, H. Bischof, and C. Leistner, “TRICam - An Embedded Platform
and verified results show that the output image is in distortion for Remote Traffic Surveillance,” in Proc. of Conf on Comp Vision and
comparing to the raw data. Pattern Recognition CVPR Workshop on Emb Sys, pp. 125-125, 2006.
[6] David Katz, Rick Gentile. “Embedded Video Processing Princples,”
http://www.eetchina.com. 2008K. Elissa,.
ACKNOWLEDGMENT
[7] Jin Su, “Design of Asynchronous FIFOs,” Doctor Dissertation, Heifei
This paper is partly supported by National Science Industrial University, 2007.
Foundation of China for young researchers (Granted [8] Keith Jack. Video Demystified--A Handbook for the Digital
No.61001104) and partly supported by Research Foundation of Engineer[EB/OL]. http://elsevier.com. 2007.
Southeast University for New Faculty with Doctor Degree [9] Eun-ok Kim and Jang-hyun Park, “Study on Rapid Prototyping
(Granted No.9227000002 ). Methodology of the Lecture Contents for the IT SoC Certificate
Program,” IEEE International Conference on Microelectronic Systems
Education, 2007.
REFERENCES [10] OmniVision Technologies, Inc. OV7670 Preliminary Datasheet[EB/OL].
[1] A. Sharif, V. Potdar, and E. Chang, “Wireless multimedia sensor http://www.ovt.com/products/. 2006.
network technology: A survey, ” 7th IEEE International Conference on
Industrial Informatics, pp. 606-613, 2009.

2169