Tokyo University of Science Professors Develop Robotic Method to Handling Tricky Objects Using Only RGB Camera

One of the most critical applications of robots is material handling, where grippers are used to move objects, such as automotive parts, logistics packages, food ingredients, and restaurant dishes.

For robots to handle objects autonomously, they must first accurately measure the three-dimensional (3D) shape of the scene using cameras and then plan how to grasp and move each object. However, certain objects pose major challenges for conventional 3D measurement systems. While opaque objects are relatively easy to identify, transparent objects, such as glass and clear plastics, are more difficult, with measurement accuracy decreasing as transparency increases.

That’s why associate Professor Shogo Arai and Mr. Ginga Kennis, both from the Department of Mechanical and Aerospace Engineering at Tokyo University of Science, Japan, developed HEAPGrasp: Hand-Eye Active Perception to Grasp objects with diverse optical properties.

“Traditionally, transparent or mirrored (glossy) objects such as reflective metal parts, transparent trays have been unstable to detect when using depth sensors or conventional 3D measurement techniques, making automatic grasping by robots difficult and ultimately leading to human intervention,” says Arai. “Our approach is based on the idea that even when depth information is unreliable, object shape estimation and grasping are still possible as long as the object’s contours or silhouettes can be captured reliably in images.”

“Our approach achieves accurate 3D measurement of objects while minimizing camera movement and execution time,” adds Kennis. “By reducing the amount of pre-adjustment required, HEAPGrasp simplifies on-site implementation and operation, especially since it can be retrofitted to existing robotic systems.”

Key takeaways:

·        Their study, IEEE Robotics and Automation Letters, outlines how HEAPGrasp relies on analyzing objects from red, green, and blue (RGB) images captured from multiple viewpoints.

·        First, the system identifies and separates objects from the background using a computer vision technique called semantic segmentation, which assigns each pixel in an image to categories such as “object” or “background.” Using a single hand-eye RGB camera, the researchers captured images from different viewpoints and applied semantic segmentation to extract object silhouettes. For this task, the researchers utilized DeepLabv3+ with ResNet-50, a convolutional neural network architecture.

·        The extracted silhouettes are then used in a 3D reconstruction technique known as Shape from Silhouette (SfS). This technique estimates the 3D shape of the object by analyzing its silhouettes from multi-view images. The idea is that each silhouette defines a possible 3D volume where the object could exist. By intersecting these volumes, SfS estimates the object’s shape and position in space. Because this process relies only on object silhouettes, it is unaffected by optical properties such as transparency or reflectivity.

·        In the SfS method, increasing the number of unique viewpoints improves accuracy and therefore can lead to a higher success rate for grasping. However, this also means that the camera must be moved to multiple viewpoints, increasing the computational cost and time burden. To balance this trade-off, the researchers introduced a deep learning-based next pose planning system that determines the most efficient camera movement trajectory, maximizing measurement accuracy while minimizing unnecessary motion.

·        The team evaluated HEAPGrasp using a real robotic system across 20 different scenes, each containing five objects. The scenes included transparent-only objects, opaque-only objects, specular-only objects, and mixed scenes containing all three types of objects. The researchers also compared HEAPGrasp’s performance with existing grasping methods.

·        Using HEAPGrasp, the robot achieved a 96% success rate for grasping objects with diverse optical properties using a single camera, significantly outperforming existing methods. In addition, it reduced the hand-eye RGB camera’s trajectory length by 52% and the execution time by 19% compared to a baseline method that circles around the scene for 3D measurement.