Real-Time Tracking of Moving Objects with an Active Camera

1998, Real-Time Imaging

Abstract

Real-Time Tracking of Moving Objects with an Active Camera T his article is concerned with the design and implementation of a system for real-time monocular tracking of a moving object using the two degrees of freedom of a camera platform. Figure-ground segregation is based on motion without making any a priori assumptions about the object form. Using only the first spatiotemporal image derivatives, subtraction of the normal optical flow induced by camera motion yields the object image motion. Closed-loop control is achieved by combining a stationary Kalman estimator with an optimal Linear Quadratic Regulator. The implementation on a pipeline architecture enables a servo rate of 25 Hz. We study the effects of time-recursive filtering and fixed-point arithmetic in image processing and we test the performance of the control algorithm on controlled motion of objects.

Figures (17)

We denote by P,, the 4x 1 vector of homogeneous coordinates with respect to the world coordinate system having origin at the intersection of the pan and the tilt axes. Let P,,. be the vectors with respect to the left and right effector coordinate systems located at the intersection of the tilt and the vergence axes. The transformation between world and effector reads “ Consisting of the TRC BiSight Vergence Head and the TRC UniSight Pan/Tilt Base.

Figure 1. The four degrees of freedom of the camera platform (top) and what it looks like (bottom).

Figure 2. The 20th image of the Diverging Tree sequence (above left), the optical flow field computed with the fixed point implementation of the FIR and IIR filters (above right), and the relative error in the estimation of optical flow of fixed- vs floating-point arithmetic. The relative error as well as the flow vector length are plotted as functions of the distance from the focus of expansion, here the center of the image (below). Key to graph: (—) fixed point; (---) floating point; (----) length. We proceed with studying the differences between The spatial FIR -kernels are binomial approximations to the first derivatives of the Gaussian function [33]. The spatial convolutions are carried out in fixed-point 32 bit arithmetic, with the result stored in 8-bit word length. The inverse of the magnitude of the spatial gradient needed for the computation of normal flow is computed using a LUT table. Fixed-point arithmetic primarily affects the HR filtering, since the binomial coefficient of the FIR filter can be represented by the quotient of powers of two. We use the Diverging Tree sequence [34] as a test-bed for our accuracy investiga- tions. The ground truth optical flow field is known, and we test the filtering effects on the computation of the optical flow field. We use a conventional method [35] that assumes local constancy of the optical flow field. In

Figure 3. Continuous impulse response comparison of the shifted first derivative of a Gaussian (o = 1, continuous curve) and the IIR second order derivative filter (t = 1, dotted curve) (above). In the middle we show the frequency responses of the five-points binomial approximation of the first Gaussian derivative (dashed curve) and the IIR second order derivative filter (t = 1, dotted curve). Below we show the pure differentiation effects, ic. the same spectra divided by the frequency responses of the low-pass prefilters.

Table 2. The average relative error in the Yosemite sequence for different t’s and kernel lengths

Table 1. The average relative error in the Diverging Tree sequence for different t’s and kernel lengths computation performance, the IIR filter guarantees the same motion behavior with much lower space and time complexity.

where J, and O, are 2x2 identity and null matrix, respectively. Assuming a linear control function Au(k) = —KS(k), with § an estimate of the state, we make use of the separation principle stating that optimal control can be obtained by combining the optimum determi- nistic control with the optimal stochastic observer [36].

The minimization of the difference |lo —c|| between object and camera position in the reference coordinate system can be modeled as a Linear Quadratic Reg- ulator problem with the minimizing cost function Si’_o s "(k)Os(k), where Q is a symmetric matrix In steady-state modus a constant control gain K is assumed, resulting in an algebraic Ricatti equation with the simple solution

Figure 4. The tilt @ and vergence 0 angles (left) and the image orbit of the target (right) with the large error due to the hig velocity (1.2 Hz) of the target. Key: left, (-) 6; (-) @. In summary, we do not expect a pixel error better In this experimental study we excluded the image processing effects by moving an easily recognizable light-spot. We controlled the motion of the light-spot by mounting it into the gripper of a robotic manipulator. The control frame rate is equal to the video frame rate (30 Hz). The world trajectory of the light-spot is a circle with radius equal to 20 cm on a plane perpendicular to the optical axis in resting position. The center of the

Figure 5. Image orbit (left) and vergence angle vs. time (right) for the af- and the a-f-y-filter plotted with a continuous anc dashed curve, respectively.

Figure 6. Image orbit of the target for four different velocities v;_4 = (0.17 Hz, 0.35 Hz, 0.52 Hz, and 0.70 Hz) for 4 = 0.1 (left) and A = 0.1 (right). Key to graphs: (-) vy; (-——) Vo; (----) v33 (ee ) V4. We summarize here all the processing steps of the loop: than +10 pixels for the highest maneuvering index if we assume that the object motion trajectory is as smooth as a circle. As we will observe in the experi- ments with usual moving objects instead of light-spots, the trajectory of the detected moving area is so irregular that only a high maneuvering index can lead to smaller tracking errors.

Figure 7. Image orbit of the target for three values of } = 0.01 (-), 0.1 (----), 0.5 (......) eft) and the vergence angle as a function of time (right).

Figure 8. Hardware architecture of the closed-loop. Key: (---) analog signals; (—) digital signals; (J/7) motor control signals. > We use Sony XC-77RR with a frame rate of 30 Hz.

Figure 9. Six frames recorded while the camera is pursuing a Tetrapak moving from right to left. The pixel error (bottom left) shows that the camera remains behind the target and the vergence change (bottom right) shows the turning of the camera from right to left with an average angular velocity of 8.5 degrees per second. Key to graph: right, (—) theta; (----) phi. By means of the setitimerITIMER_REAL...) func- tion of the Solaris 2.4 operating system, we guarantee a

Figure 10. Six frames recorded while the camera is pursuing a rotating target moving from right to left and then again to right, first downwards and then upwards. The average angular velocities for both the vergence and the tilt are 10 degrees per second. Key to graph: right, (—) theta; (----) phi. cycle time of 40 ms. This cycle time consists of 37 ms image processing (steps 2-6 performed on the MaxVi- deo 200 board) and 3 ms control (steps 7-10 performed in the SparcStation). The motion controller guarantees a motion execution time of 40 ms. Considering the effective delay of the temporal derivatives calculations

Figure 11. The camera is pursuing a target attached in the gripper of a manipulator. The target is moving on a circle with frequency 0.35 Hz. The angular velocity is 8 degrees per second for vergence and 5 degrees per second for tilt. Key to graph: right, (—) theta; (----) phi. difference and the gradient magnitudes exceed two thresholds which are the same for all four experiments. The centroid of the detected motion area is marked with a cross. We show the tracking error by drawing the trajectory of the centroid in the image as well as the control values for the tilt and the vergence angle, @ and 6 for the entire time interval of 8s. The performance of the active tracking system in four different object motions is shown here. The images in the figures are chosen out of 20 frames saved “on the fly” during a time of 8s. The images are overlaid with those points on the images where both the normal flow

Figure 12. The camera is pursuing a target mounted on the gripper of a manipulator while the camera is itself translating forwards. The target is moving on a circle with frequency 0.70 Hz. The translation of the camera is shown in the shift of the angle oscillation center. As the camera is approaching on the left side of the manipulator it must turn more to the right (positive shift in vergence) and more downwards (negative shift in tilt). Key to graph: right, (—) theta; (----) phi. corrupted with an error of very high measurement variance. Allowing a high maneuvering index which enables close tracking would result in an extremely jaggy motion of the camera. The estimator would forget the motion model and yield an orbit as irregular as the centroid motion. Therefore, we decrease the maneuver-

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.