Laser and Vision-Based Obstacle Avoidance for A Semi-Autonomous ROV

Remotely operated vehicles (ROV) and Autonomous underwater vehicles (AUV) are increasingly being used for a large spectrum of maritime tasks for which human intervention is either dangerous or impossible. In case of complex tasks such as detailed inspection, difficult environment or even budget shortage, AUV may not be suitable. On the other hand dealing with repetitive actions or having to constantly react to obstacles may be excessively tiring for the ROV operator. In this paper we describe a low-cost Laser-based obstacle sensor which combined with the camera image allow the ROV to autonomously react without the intervention of the operator and even blocking his commands. In order for the ROV to take control, a C++ software was developed to receive ROV sensors data, process it and send commands using the same serial communications port the operator uses. All the calculations related to the Laser sensor are performed by a Matlab program whose results are shared with the former program. Tests were carried out in a swimming pool and the semi-autonomous ROV performed satisfactorily and efficiently. Copyright © VBRI Press.


Introduction
Remotely operated vehicles (ROV) and Autonomous underwater vehicles (AUV) are increasingly being used for a large spectrum of maritime tasks for which human intervention is either dangerous or impossible such as marine life survey, ocean scanning of physical parameters, sea bed evaluation, wreckage inspection or archaeological research. When navigation paths are uncertain or potentially complex, especially for unstructured submarine environment, AUVs may not be able to accomplish their programmed tasks. On the other hand, ROVs may also be difficult to operate by a single operator in very complex environments or even impossible such as in the case of poor visibility, multiple sensorial paradigm or obstacle diversity.
Semi-autonomous ROVs seem to be a good option to overcome difficulties for both vehicle types.
In this poster we describe the implementation and underwater tests of a vision-based system for an inspection-class ROV [1], designed to afford the vehicle with semi-autonomous navigation capabilities. The vehicle equipped with this system is programmed to perform emergency escape maneuvers in situations of imminent collision, while being controlled manually by a human operator. The proposed instrumentation set-up consists of a slant-looking Laser [2,3,4] and an in-water calibrated video camera whose data is processed in real-time, allowing the ROV to detect the vicinity and measure distances to obstacles.
For such, the image pattern projected by the laser beam is exploited by a trilateration-based algorithm to estimate the frontal distance to nearby objects.
As proximity to an obstacle goes below a certain pre-defined value, the running software blocks the operator commands, takes control of the vehicle and performs the necessary emergency maneuvers.
The efficacy of the proposed set-up is demonstrated through a series of underwater tests performed in a swimming pool with a VideoRay Pro 4 ROV.

Overall control
The normal operation of the ROV is manual using a hand controller ( Fig. 1) connected to a computer running Cockpit a graphic program provided by VideoRay. It allows the operator to watch real-time image sent by the camera, rotate the camera, steer the ROV by controlling the speed of the two rear propellers, adjust the intensity of the two front lights and adjust its depth using the vertical thruster. It also collects and displays all the sensors' readings such as heading, temperature, depth and humidity. All this data is also available to the developed C++ program (Fig. 1) which is also connected to the ROV via the same operator-Cockpit communication serial line. Of all the available sensor data, the most valuable is the heading provided by the ROV's internal compass, which is used to correct trajectories. Besides reading sensorial data, this line interference also allows the program to send commands using the same formats of Cockpit's. This communication interference is performed only when an approaching obstacle is detected (Fig. 1).
This detection is performed by another set of programs, in Matlab. Unlike the program in C++, these are acting continuously so that Vision and Laser information are always being computed while the operator drives the VideoRay Pro 4 ROV manually (Fig. 1).
As soon as the navigation program in C++ receives an obstacle warning from the Matlab program, it blocks console communication and takes control of the ROV and performs an escape maneuver.

Camera calibration
The correction of images acquired by the camera requires 9 parameters which can be obtained after a calibration process with a Matlab Toolbox. It measures (x, y) of all significant points of a chess-like board in several positions at a certain distance of interest in front of the ROV. 20 such images were used. The toolbox than acquires the pixel coordinates of all vertices in the image in all submitted images and make the corresponding association to real distances in cm given the previously measured size of the squares and the approximate distance from the camera to the board.
By applying transformation matrices using the camera pin-hole model, the toolbox computes all the intrinsic and extrinsic parameters of the camera for image calibration.
One of the calibration procedures yielded the following parameters: These values are fed to another Matlab which performs adequate compensation for all distortions in each new acquired image.

Image processing
After correcting each image, the RGB components (Red, Green and Blue) are separated, subject to binarization using adequate threshold values and recombined using 2G-R-B for Laser line enhancement.
HSV (Hue, Saturation and Lightness) has also been used. A detailed description will be given in sub-section Laser calibration.

Laser Sensor configuration
In order to detect approaching obstacles, a 10mW Laser line module was attached to the ROV making a certain angle (28º) with the horizontal as shown in Fig. 2 and Fig. 3a.
Its expanded beam projects a Laser line to a planar obstacle (Fig. 3b).
The closer the obstacle is to the ROV, the lower will be the mean y pixel coordinate of the line seen in image (Fig. 3c). Of course, irregular obstacles will yield irregular line shapes; but those shapes can be processed to obtain contour information of obstacles. The previously measured correspondence between y and Distance (Laser calibration) will provide this value. The angle  is a compromise between higher precision (high ) and higher detecting distances (low ). An angle of 28º seemed to be a good compromise to detect and measure obstacles in the range of 20-80 cm.
The Laser beam is always switched on to allow continuous readings. A 532nm Green Laser 10mW Line Diode has been used. Its consumption is about 0.6 A which requires a high capacity battery. The selected battery provides more than one hour of continuous operation which was more than enough to perform tests. However, for future reliable continuous and limitless Laser Line detected operation, the Laser module will have to access the 48 V internal voltage. This is an extremely troublesome task given the dome and cables perfect sealing.
This Laser-based distance sensor is low-cost (as compared to the very expensive Sonar), has much more precision and provides far more measuring points than the Sonar.

Laser calibration
A calibration was carried out based on the correspondence between the measured mean y image coordinate of the projected Laser line (in units of pixel) and the measured actual distance from the ROV dome to the obstacle (in cm). The precision of the latter was approximately 0.5 cm. The planar obstacle was placed at several distances from the ROV, between 10 cm and 100 cm with 5 cm or 10 cm of interval, providing several pairs of values (Fig. 4). A curve fit was then performed to the set of points in the graphic and an equation of Distance = f (mean_y) was obtained. The result of one of those calibrations is shown in Fig. 4. Then, for every mean y of the Laser line projected at the obstacle (in units of pixel), the real distance in cm could rapidly been computed just by using the obtained equation. For a non-planar obstacle, several y values would be measured, the lowest of which indicating the closer part of the obstacle. Furthermore, the several values of y can provide contour information thus allowing the implementation of complex responses to the obstacle approach.
This method was preferred over geometry parameter use, due to faster mathematical processing as well as higher accuracy.
In order to enhance and extract only the Laser line, the image has to be digitally processed: all the image components had to be separated and binarized using predefined threshold values. RGB or HSV components can be selected, whichever yield the better results. Since the Laser line is displayed by pixels having high intensity of the Green component, good results can be obtained by using RGB and selecting only this component for binarization. However, sometimes the HSV components performed better. On the other hand, using just the G component could lead to incorrect measures, especially for high ambient luminosity: in this case, all three components would easily saturate in parts of the image, other than the Laser line itself, yielding false Laser line detections. The solution was recombining all the three RGB components (or HSV components) using I=2G-R-B. By selecting a weight of 2 for the G component and 1 for R and B for each and every pixel, the intensities of saturated pixels would cancel out and the resulting intensity would be zero (black pixel).
In case of non-saturated pixels, the binarization using adequate threshold values would zero the R and B intensity values and the 2G would prevail thus enhancing the green pixels associated with the Laser line.
All this filtering process, performed for the calibration procedure, is ultimately also used for Laser line extraction during normal ROV use in the semiautonomous mode.

Linear path with sensor feedback
A linear trajectory was programed to assess the ROV's capability to navigate in straight lines and maintain the predefined heading.
Heading values were read and stored during navigation (Fig. 5a). As shown in Fig. 5b the ROV executes the planned path as expected (red line). Minor deviations due to confirmed local magnetic anomalies can be observed, but also are the corresponding automatic corrections. In another experiment (blue line) the ROV is intentionally pulled sideways. The program automatically corrects the trajectory.

Square path with sensor feedback
A square trajectory was programed to assess the ROV's capability to perform complex yet precise maneuvers. Heading values were read and stored during navigation (Fig. 6a). As shown in Fig. 6b the ROV executes the planned path as expected. Minor deviations due to inertia can be observed, but also are the corresponding automatic corrections.

Obstacle avoidance
Finally, the whole set-up for obstacle detection, distance measurement and deviation maneuver was subject while the ROV is moving forward.
Heading values were continuously read to ensure a straight line path and distances were measured, also continuously, using the Laser sensor and the running Matlab programs. Fig. 7a shows the heading values collected in such experiment. Fig. 7b shows that the Laser sensor was capable of detecting an approaching obstacle and imminent collision in time, and perform a maneuver to go around it.
Many experiments were carried out showing almost 100% success in detecting the obstacle in time and performing correctly the predefined deviation.
For the time being no lateral distance measures to the obstacle are carried out during the whole deviation maneuver due to absence of a sensor. In the future such a sensor, eventually a smaller version of the Laser-based distance sensor that is presented in this paper, will be added to the ROV. It will then be able to perform a safe deviation whatever the shape and size of the obstacle might be.
The camera+Laser algorithm is always running, providing warnings and distance-to-obstacle to Control.

Conclusion
A low-cost Laser-based sensor has been developed and used efficiently with a ROV to detect and measure distances to obstacles. This extra sensor, along with the heading feedback provided by the ROV internal compass, can then be processed by Matlab and C++ programs to automatically command the ROV in precise deviation maneuvers. The current configuration can even potentially differentiate multiple distinct obstacles and analyze their contour since the projected Laser line crosses the entire image width providing an entire row of distances. The ROV may then autonomously decide and perform equally distinct maneuvers. This added capability allows the ROV to be semi-autonomous providing prompt reactions to approaching obstacles, thus simplifying as well as making less tedious and tiring the operator's task of manual control. This added semiautomation will be extremely useful during manual operations in low visibility or particularly complex environments. This effort towards semi-automation of manually controlled submersible devices is not common perhaps due to difficulties in manufacturing and applying nearly homemade delicate, small, and conveniently well sealed electrical devices to almost closed systems like the commercial ROVs.