Autonomous Ripeness Detection Using Image

Processing for an Agricultural Robotic System

Zubaidah Al-Mashhadani Balasubramaniyan Chandrasekaran Ph.D

Department of Electrical & Computer Engineering Department of Electrical & Computer Engineering
Florida Polytechnic University Florida Polytechnic University
Lakeland, Florida Lakeland, Florida
[email protected] [email protected]

II. BACKGROUND
Abstract— This paper discusses the detection of the ripe and According to Florida Tomato Committee (FTC), Florida is
turning tomato fruits using computer vision and image known for producing 50% Of all domestically grown fresh
processing techniques. Field workers can run additional pre-
tomatoes [16]; thus, monitoring the ripeness level of the crops
processing methods when the lighting or weather conditions are
is time-consuming and difficult to maintain precise
not suitable for taking good quality images. The hardware used
in this work is the Raspberry Pi and Pi Camera; the software is
information. Therefore, the proposed work of detection and
Raspbian using python3. The ripeness is detected by using maturity estimation will help the field workers by filing data.
OpenCV and HSV color space. Counting the tomatoes will help This data includes mature and the turning fruit numbers so the
the field workers know how many tomatoes are ready for field workers can see it in real-time as the robot works to
harvesting and how many will be available for harvesting in monitor the fruits' ripeness.
order to prepare labor or equipment. This hardware set up will Fruit maturity is an essential factor in determining the
be interfaced with the Turtlebot and the robot will navigate the
harvesting time. Delays or mistakes in the timing can lead to a
field to achieve ripe detection.
significant loss in crops. This will affect the income rate; thus,
Keywords—ripeness detection; image processing; OpenCV; knowing the number of mature fruits in the real-time system
HSV; Bicubic interpolation; Agricultural robot. would enable the field workers to prepare adequate labor forces
or robots to harvest. The maturity level of crops is usually
estimated by external features such as color, texture, shape and
I. INTRODUCTION size [17]. The ripeness detection is achieved either by image
The focus on using technology in the agricultural industry processing or by using sensors that detect the ripe fruits based
has been increasing in the past decade. The importance of on the emitted gas [18]. This paper focuses on image
adopting technology in this field has raised as it showed that it processing approaches; there have been several methods
saves time, efforts, cost, and more importantly, protect putting introduced to perform this task on different types of crops in
human life in danger for doing the hard work that can exhaust existing work, such as: the estimation of apple ripeness using
and cause health issues for those who work in agriculture. L*a*b color space and K- Nearest Neighbor (KNN) classifier
in [12], tomato maturity prediction based on machine learning
Indeed, the use of technology and robots in agriculture field approach using Support Vector Machine (SVM) and CIELAB
has been taking over many tedious and important tasks such as color space in [11], Multi-Class FRCNN and parallel FRCNN
making a decision whether the farmland is cultivable or not for for detection and ripeness approximation of fruits in [13], apple
a specific type of plants using Zigbee Development Kit [1], ripeness estimation using Artificial Neural Networks (ANN) in
evaluating the health of the crops using the Normalized [17], grapes ripeness evaluation using convolutional neural
Vegetation Index (NDVI) [2], monitoring the state of soil network (CNN) and SVM in [19], and tomato maturity
health and nutrients using Internet of Things (IoT) [3], crop assessment using image processing techniques in MATLAB
disease detection and identification [4] [5] [6] as well as spray [20]. the method used in this work is utilizing OpenCV and
pesticides [7], pest control [8], yield estimation [9], monitoring performing image processing in the HSV color space.
moisture level in soil [10], automatic ripeness detection [11]
[12] [13], crop firmness estimation [14], Human-Robot III. RELATED WORK
Interaction (HRI) for performing harvesting tasks [15]. This
research indicates that technology plays a crucial role in The process of fruit detection and recognition usually is
carrying on the tedious work and is suitable for accurate data demonstrated in a few steps, including: pre-processing
acquisition and monitoring tasks. techniques, image segmentation, color converting and feature
extraction, training / classifying algorithms. Image pre-
processing includes the resizing of the image for faster

978-1-7281-9656-5/20/$31.00 ©2020 IEEE

0744
0743

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on June 17,2021 at 00:55:31 UTC from IEEE Xplore. Restrictions apply.
processing and applying interpolation depending on the Inference System classifier is trained using the gradient
resizing whether it is enlarging or shrinking. In addition to method.
image enhancement options for illumination removal or
contrast adjustment, edge detection and binarization. The Section (II) will cover the background, illustrates the
feature extraction is based on color/shape, and the color space methodology and implementation, section (II) will discuss the
conversion mostly depends on the type of algorithm used for results and the limitations faced.
classifying. Image segmentation involves partitioning the
images into a pixel groups that are homogenous in certain way; IV. ARCHITECTURAL DETAILS
the homogeneity depending on other features such as strength The detection of tomato ripening level and counting in this
and color [14]. In that paper, image segmentation is done by work is accomplished using a computer vision system. The
using KNN algorithm. In [17] and [19] the authors suggest main software and hardware components are clarified in this
using thresholding as the segmentation method, which is one of section.
the easiest and widely used approaches, and in paper [20] fruit
grading is performed based on the image histogram analysis. A. OpenCV
The segmentation in this paper is through using thresholding An open-source software library for computer vision and
and histogram analysis. machine learning. The library consists of more than 2500
In [11] two image sources are used to gather the dataset algorithms. This library can be used in many languages,
from the web and images from special setup photographing including C++, Python, Java, MATLAB. Moreover, it
studio with blue background implemented as computer vision supports different platforms making it easier to use on most
system (CVS). The images from CVS are then converted from software operating systems [21]. OpenCV is implemented for
RGB to L*a*b color space, where L reflects the lightness, computational efficiency and real-time applications. This
black to white, a*, green to red, and b*, blue to yellow. library assesses the user to build a complex vision application
Afterward the image is resized, scaled down and the filters with simple and quick use [22]. This work focuses on the use
were applied. On the contrast, the downloaded images from the of OpenCV in computer vision application, which involves
web went under several pre-processing techniques before data transformation from an image or a video to a decision or
performing the segmentation. These techniques includes: another representation to achieve a specific goal.
histogram equalization, noise reduction, light enhancement and
sharpening. To prepare the images for the segmentation B. Raspberry pi
process and alleviate the removal of the background, the Raspberry pi is a single-board computer with a system on
segmenting stage is done by the use of the thresholding chip (SOC). It contains input and output ports, USB and
method. Equation (1) [11] is used for noise removal from the HDMI ports on it and a memory slot. The Operating Systems
background which will binarize the image to create a mask and
(OS) used to run the Raspberry pi is the Raspbian OS. The
apply the segmentation [11].
programming language used is Python 3. The software used
T(x,y)=m(x,y)+k* δ (x,y) (1) for programming is the Python IDE.
The classification involves using a non-linear SVM C. Raspbian
classifier with color space L*a*b. The accuracy of the final Raspbian is a free Debian OS for Raspberry Pi. An OS is
results was 83.39%, while the accuracy in [17] yielded 98.33%
the collection of simple programs and other utilities to operate
using the ANN classifier, which used the same color space as
the Raspberry Pi and is classified as an operating system that
well, and thresholding is used to extract the fruit from the
background and apply morphological processes on the binary is officially endorsed. The Raspbian OS is the operating
image to reconstruct the RGB image for each maturity level. system used in this work and is run by python language.
The training phase includes the use of ANN, where the inputs D. HSV color space
are the fruits’ colors characteristics. Next, the testing phase is
done by calculating the accuracy in equation (2) [17]. HSV is a circular color pattern that makes the primary
RGB colors more understandable to humans by converting
them into proportions: Hue, Saturation, and Value. Hue
determines the light angle for the RGB filter chain; red results
Accuracy = . 100 (2) in a 0 ° shade, green results in 120 °, and blue results in 240°.
However, the highest accuracy results among the proposed Saturation monitors the volume of color used. A 100 percent
methods in these researches are [12], where the accuracy is saturation color would be the purest possible hue, whereas 0
about 99.33%, which is close to the ideal results. It was percent saturation produces grayscale. The value determines
achieved by using KNN classifier for image segmentation, the color’s illumination, a color with a brightness of 0 percent
training fuzzy system with gradient method, and the cross- is entirely black, and a color with an intensity of 100 percent is
validation method (CV) is used in estimating the machine not blended with black. To demonstrate, the reader is referred
learning model on a specified number of the dataset. The to Fig.1 [24]. In this work, HSV color space is used, and the
feature extraction phase depends on the RGB color as the ratio thresholding is mainly determined by the hue values for the
of each channel pixel number is divided by the total number of ripe and turning fruit colors.
pixels as the color features are the outcome of KNN. The fuzzy

0745
0744

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on June 17,2021 at 00:55:31 UTC from IEEE Xplore. Restrictions apply.
 not. Next, the images are scaled-down and interpolated. The
next step is the image segmentation; this is achieved by using
the thresholding method in HSV color space based mainly on
the Hue parameter. Eventually, the mask is applied to the
original image. Furthermore, the counting of fruits depends on
the contour detection of each ripeness level. The system is
designed to detect and count only the ripe and turning tomato
fruits.

A. Software Setup
Raspberry pi, likewise, any computer requires an operating
system to run. The official OS is chosen to run the board in this
paper. Using Python 3, the programming is done using Python
Figure 1. HSV color space solid cylinder [24] IDE. The VNC is installed on the personal computer and
connected to a personal hotspot or Wi-Fi network. The
Raspberry Pi. On the other hand, the Raspberry Pi is connected
E. VNC viewer [25] to a mouse and keyboard to connect to the same wireless
Virtual Network Computing (VNC) is a graphical desktop- network. The IP address of the Raspberry pi can be displayed
sharing system to control another computer remotely. by using sudo command in the terminal that displays the IP
Moreover, it transmits movement and input of the keyboard configurations, the output should include inet number in the
and mouse to the other computer. In this work, the VNC Wlan section:
viewer program is used to control the Raspberry pi and
perform the programming and testing through the wireless
network for effortless and simple setup as working on the
Raspberry Pi itself require a monitor, keyboard and mouse
similar to any other computer.
Using this IP, the VNC viewer can connect to the
V. EXPERIMENTAL SETUP Raspberry pi after providing the username and password, to
This section will demonstrate the technical setup of the perform the virtual connection successfully and use the Pi
hardware and the software. Additionally, the workflow of this camera, the following settings should be done:
paper is illustrated in Fig.2. Menu → Preference → Raspberry pi configuration →
Interfaces → Enable VNC →Enable Camera.

Next, the Raspberry Pi can be monitored using the

personal computer as soon as the board is connected to the
power. Subsequently, the OpenCV environment is installed;
the version used in this work is “3.4.3”. Then, the
dependencies and libraries download, and installation takes
place. Including NumPy for supporting manipulating the
digital images by performing high-level mathematical
operations as the images are multi-dimensional arrays,
Matplotlib is the plotting library in python and Pandas for data
manipulation and analysis.
B. Hardware Setup
To work on the ripeness detection, we need an input sensor
that is the camera. The First step is to connect the Pi Camera
to the Raspberry Pi board using the camera's provided slot.
The camera has photo capability as well as full HD video
capture capability at 1920x1080. Fig.3 shows the connection
to the Pi Camera.

Figure 2. Vision system workflow general diagram

The detection of ripeness and estimation is done in a

simple, fast method, using OpenCV. The human decides to
apply further pre-processing to enhance the images taken or

0746
0745

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on June 17,2021 at 00:55:31 UTC from IEEE Xplore. Restrictions apply.
 illumination removal and contrast enhancement would be
performed using gamma transformation and histogram
equalization. The former reduces the illumination in the image
that affects the detection, the used gamma value in this paper is
0.9, and the latter improves dark images as it enhances the
contrast and makes it more apparent to the system and user.
When OpenCV reads the image, it will be in BGR color
space. Thus, to apply histogram equalization, the image is
converted to YUV color space. It encodes a color picture or
video that considers human perception. Next, perform
histogram equalization, then the image is converted back to
BGR and to RGB to see the final results.

Figure 3. Raspberry Pi Camera connection

The Raspberry Pi and the Camera is then mounted on the

ground robot to navigate and perform the monitoring and data
acquisition tasks.
VI. IMPLEMENTATION
The level of maturity is determined using computer vision
techniques. Since the input image is of high quality, the size
would be large. Subsequently, the processing time increase.
Thus, the images are scaled down to half the original size and
interpolated using bicubic interpolation. Even though it is
slower than other interpolation techniques such as Nearest
Neighbor (NN) interpolation, bicubic interpolation is more
suitable for shrinking the images and gives a clearer output
image.
The input image used in this test after resizing to 300x300
pixels is shown in Fig. 4 [23]. The system detects and counts
the ripe fruits (stage V and VI) and turning fruit (stage IV).

Figure 5. System block diagram

If the user decided not to proceed with additional

processing, the segmentation and feature extraction for
detection would occur. The block diagram of image processing
and detection is illustrated in figure (5). The images read by
OpenCV are in BGR color space, so it needs to be converted to
Figure 4. Tomato image for maturity detection testing [23]
RGB. After converting to RGB and plotting the histogram, we
The user has the decision to make if the circumstances of do not have precise results to threshold based on the values of
taking the images require performing additional pre-processing the channels for mature and turning fruit, as in Fig.6.
for image enhancement or not. If the user chose to apply it,

0747
0746

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on June 17,2021 at 00:55:31 UTC from IEEE Xplore. Restrictions apply.
 Figure 6. Histogram of RGB channels

Hence, the HSV color space was more suitable for

extracting and thresholding the fruits effectively. The
histogram of the HSV channels shown in Fig.7 demonstrates
the Hue values for ripe (0 - 10) and for turning tomato (11- 20)
Figure 9. Ripe and turning tomato detection
while the Saturation is (170 – 256), and the Value is between
(150 – 256). As the final results show, the tomato is detected with an
ellipse, blue in the case of ripe tomato and green for the turning
tomato. The quantity for each, ripe and turning fruits, are
displayed for the user in a real-time system in the terminal.
Also, exported to a text file saved for data gathering purposes.
Below is the output when performing with the additional image
enhancement.

The following is the output for running the program

without the image enhancement techniques:

Figure 7. HSV channels histogram

Subsequently, performing image segmentation using

thresholding by the specified values leads to having a binary
image result for both ripe and turning tomato, as shown in Fig.
8. The first row shows the ripe fruit while the second row We can conclude that the time difference between both
shows the turning fruits. This result is achieved by removing processes is almost 0.6 second, which is acceptable when
the unwanted shapes, and noise from the image then fill in the performing on images, but in the real-time system, this delay
gaps. would affect monitoring if it is unnecessary. Thus, human
interaction is considered in the system.

VII. RESULTS
The results show that detection and counting were
successfully implemented. The system has been tested on
different images, and there were few limitations. The first
limitation is when the ripe or turning fruits are overlapped, the
system will detect it. However, the counting would not be
accurate as it can count two fruits as one. Moreover, when
testing extensive data set or in the field, this could cause having
less information about the numbers of fruits to be harvested.
Figure 8. Image thresholding and application of morphological processes Another limitation is when the image is taken from an angle
that does not allow the tomato to be identified and could be
Additionally, after having these results, the masking is considered an unwanted item in the picture. Aside from these
performed on the original image, and the contour was detected limitations, this method was simple, reliable and fast to support
to draw an ellipse around each for detection in Fig.9. real-time monitoring systems.

0748
0747

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on June 17,2021 at 00:55:31 UTC from IEEE Xplore. Restrictions apply.
 VIII. CONCLUSION AND FUTURE WORK Industrial and Information Systems (ICIIS), Rupnagar, India, 2018, pp.
83-88, doi: 10.1109/ICIINFS.2018.8721422.
In this work, the estimation of ripeness and counting for [10] P. Senthil and I. S. Akila, "Automated Robotic Moisture Monitoring in
tomato fruit is implemented using computer vision and image Agricultural Fields," 2018 International Seminar on Intelligent
processing techniques. The user decides to run the additional Technology and Its Applications (ISITIA), Bali, Indonesia, 2018, pp.
processing for image improvement to achieve higher accuracy 375-380, doi: 10.1109/ISITIA.2018.8711348.
in uncertain weather conditions. The processing is mainly [11] M. B. Garcia, S. Ambat and R. T. Adao, "Tomayto, Tomahto: A
performed in the HSV color space. Segmentation is applied by Machine Learning Approach for Tomato Ripening Stage Identification
Using Pixel-Based Color Image Classification," 2019 IEEE 11th
thresholding since it is simple, easy to implement, and efficient International Conference on Humanoid, Nanotechnology, Information
to use. The image underwent noise reduction and Technology, Communication and Control, Environment, and
morphological processes to create masking that precisely Management ( HNICEM ), Laoag, Philippines, 2019, pp. 1-6, doi:
detects the ripe and turns tomato and thus counts the detected 10.1109/HNICEM48295.2019.9072892.
fruits. [12] R. Hamza and M. Chtourou, "Design of fuzzy inference system for apple
ripeness estimation using gradient method," in IET Image Processing,
Although the results seem to be satisfying, the counting can vol. 14, no. 3, pp. 561-569, 28 2 2020, doi: 10.1049/iet-ipr.2018.6524.
be easily mistaken if the fruits were overlapping. In future [13] M. Halstead, C. McCool, S. Denman, T. Perez and C. Fookes, "Fruit
work, this work will be interfaced with the Turtlebot using Quantity and Ripeness Estimation Using a Robotic Vision System," in
Robotics Operating Systems (ROS). Also, the detected fruits' IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 2995-3002,
Oct. 2018, doi: 10.1109/LRA.2018.2849514.
location will be sent to the user using GPS that is embedded
[14] P. Sehgal and N. Goel, "Auto-annotation of tomato images based on
with the Turtlebot hardware. Further work can also include: ripeness and firmness classification for multimodal retrieval," 2016
applying haze removal in adverse weather conditions to have International Conference on Advances in Computing, Communications
more explicit images [26] and adding sensors to detect the and Informatics (ICACCI), Jaipur, 2016, pp. 1084-1091, doi:
gases emitted from the fruits, and include this input with the 10.1109/ICACCI.2016.7732189.
camera to create sensor fusion and make the final decision [15] F. A. Cheein et al., "Human-robot interaction in precision agriculture:
based on that. Furthermore, adding more ripeness levels to Sharing the workspace with service units," 2015 IEEE International
Conference on Industrial Technology (ICIT), Seville, 2015, pp. 289-295,
identify and provide the estimated time for harvesting for each doi: 10.1109/ICIT.2015.7125113.
class can improve the results. [16] https://www.floridatomatoes.org/tomato-
101/#:~:text=Growing%20tomatoes%20in%20Florida&text=With%20al
REFERENCES most%20every%20southern%20county,all%20fresh%20tomatoes%20pr
oduced%20domestically. (access date: Aug, 10 ,2020)
[1] P. Singh, P. Pawar, P. Takwale and D. Karia, "Agriculture monitoring [17] R. Hamza and M. Chtourou, "Apple Ripeness Estimation Using
system AMS," 2017 International Conference on Energy, Artificial Neural Network," 2018 International Conference on High
Communication, Data Analytics and Soft Computing (ICECDS), Performance Computing & Simulation (HPCS), Orleans, 2018, pp. 229-
Chennai, 2017, pp. 742-745, doi: 10.1109/ICECDS.2017.8389535. 234, doi: 10.1109/HPCS.2018.00049.
[2] A. M. de Oca, L. Arreola, A. Flores, J. Sanchez and G. Flores, "Low- [18] Kiran, M. S., & Niranjana, G. (2019, April). A Review on Fruit Maturity
cost multispectral imaging system for crop monitoring," 2018 Detection Techniques. Retrieved August 12, 2020, from
International Conference on Unmanned Aircraft Systems (ICUAS), https://www.ijitee.org/wp-
Dallas, TX, 2018, pp. 443-451, doi: 10.1109/ICUAS.2018.8453426. content/uploads/papers/v8i6s/F61300486S19.pdf
[3] R. F. Maia, I. Netto and A. L. H. Tran, "Precision agriculture using [19] H. Zhang, J. Li, L. Li, Y. Li, Q. Zhao and Y. You, "Single image
remote monitoring systems in Brazil," 2017 IEEE Global Humanitarian dehazing based on detail loss compensation and degradation," 2011 4th
Technology Conference (GHTC), San Jose, CA, 2017, pp. 1-6, doi: International Congress on Image and Signal Processing, Shanghai, 2011,
10.1109/GHTC.2017.8239290. pp. 807-811, doi: 10.1109/CISP.2011.6100341.
[4] N. Kitpo and M. Inoue, "Early Rice Disease Detection and Position [20] S. R. Rupanagudi, B. S. Ranjani, P. Nagaraj and V. G. Bhat, "A cost
Mapping System using Drone and IoT Architecture," 2018 12th South effective tomato maturity grading system using image processing for
East Asian Technical University Consortium (SEATUC), Yogyakarta, farmers," 2014 International Conference on Contemporary Computing
Indonesia, 2018, pp. 1-5, doi: 10.1109/SEATUC.2018.8788863. and Informatics (IC3I), Mysore, 2014, pp. 7-12, doi:
[5] K. R. Aravind and P. Raja, "Design and simulation of crop monitoring 10.1109/IC3I.2014.7019591.
robot for green house," 2016 International Conference on Robotics: [21] https://opencv.org/about/ (access date: Aug, 11, 2020)
Current Trends and Future Challenges (RCTFC), Thanjavur, 2016, pp.
1-6, doi: 10.1109/RCTFC.2016.7893419. [22] Adrian Kaehler and Gary Bradski. 2016. Learning OpenCV 3: Computer
Vision in C++ with the OpenCV Library (1st. ed.). O’Reilly Media, Inc.
[6] P. Sakdarat et al., "Copper Oxide Nanorods Pesticide Sensor For Methyl
Parathion Detection," 2019 IEEE 19th International Conference on [23] Thengane, Vishal & Gawande, Mohit. (2018). Arduino Based Tomato
Nanotechnology (IEEE-NANO), Macao, 2019, pp. 113-116, doi: Ripening Stages Monitoring System. 10.15680/IJIRSET.2017.0701013.
10.1109/NANO46743.2019.8993676. [24] Sambyal, P. (2015). Study of HSV Color Space Deviations and
[7] S. K. Pilli, B. Nallathambi, S. J. George and V. Diwanji, "eAGROBOT - Histograms in Image Steganography.
A robot for early crop disease detection using image processing," 2014 [25] https://www.raspberrypi.org/documentation/remote-access/vnc/
International Conference on Electronics and Communication Systems (access date: Aug, 20 ,2020)
(ICECS), Coimbatore, 2014, pp. 1-6, doi: 10.1109/ECS.2014.7090754. [26] Y. Iwamoto, N. Hashimoto and Y. Chen, "Fast Dark Channel Prior
[8] H. Chu, D. Zhang, Y. Shao, Z. Chang, Y. Guo and N. Zhang, "Using Based Haze Removal from a Single Image," 2018 14th International
HOG Descriptors and UAV for Crop Pest Monitoring," 2018 Chinese Conference on Natural Computation, Fuzzy Systems and Knowledge
Automation Congress (CAC), Xi'an, China, 2018, pp. 1516-1519, doi: Discovery (ICNC-FSKD), Huangshan, China, 2018, pp. 458-461, doi:
10.1109/CAC.2018.8623234. 10.1109/FSKD.2018.8686854.
[9] A. Agarwal, A. K. Singh, S. Kumar and D. Singh, "Critical analysis of
classification techniques for precision agriculture monitoring using
satellite and drone," 2018 IEEE 13th International Conference on

0749
0748

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on June 17,2021 at 00:55:31 UTC from IEEE Xplore. Restrictions apply.