Tutorial using ROS for ver.3

Grasping with TurtleBot3 automatically

This example is a simple example that automatically grasps an object.

TurtleBot3 has OpenManipulator.

Object recognition by darknet_ros (YOLO) and Point Cloud data are used.
So CUDA must be installed in the Ubuntu in order to recognize objects using YOLO.
Although it is possible to check this example scene using CPU without installing CUDA, the object recognition speed is very slow.

The outline of the operation is as follows.

  1. Move the TurtleBot3 to the front of the target object by keyboard operation.
  2. Move the TurtleBot3 so that the target object comes to the center of the color image.
  3. Enter a number to specify the object to be grasped.

When the above operation is performed, TurtleBot3 tries to grasp the object as follows:

  1. The darknet_ros (YOLO) performs object recognition using a color image.
  2. Estimate the 3D position of the target object using color images and Point Cloud information.
  3. Grasp the calculated 3D position.

Please refer to here for the specifications of Open Manipulator.
Please refer to here for camera specifications.

Build Ubuntu environment

In this example, darknet_ros (YOLO) needs to be installed in Ubuntu environment.
It is recommended to check the operation of normal darknet without using ros.

The procedure is as follows:

  1. If you want to use CUDA, install the NVIDIA driver. (It is possible to run using the CPU without installing CUDA.)
  2. If you want to use CUDA, download and install it from here.
    (For details, please adjust according to your PC and NVIDIA Driver environment.)
  3. Git clone darknet_ros.
    Download the commit that have been verified to work.
    $ cd ~/catkin_ws/src
    $ git clone --recursive https://github.com/leggedrobotics/darknet_ros.git
    $ cd darknet_ros
    $ git checkout 1027a280
  4. If you want to use CUDA, modify darknet_ros/darknet/Makefile. (GPU=1, CUDNN=1)
  5. Install darknet_ros.
    $ cd ~/catkin_ws
    $ catkin_make -DCMAKE_BUILD_TYPE=Release

    * Depending on the combination of GPU and CUDA, you may get an error like "Unsupported gpu architecture 'compute_30'" during catkin_make.
    In that case, you can comment out the line related to compute_30 in CMakeLists.txt of darknet_ros to avoid the error.

    * For example, in the case of NVIDIA Tesla T4, delete the line "compute_30" and add "compute_75".

Startup Procedure

Start the Ubuntu side and then the Windows side.

Ubuntu side startup procedure

Open a new terminal and run the following command:

$ roslaunch sigverse_turtlebot3_open_manipulator grasping_auto.launch

Windows side startup procedure

Start the [Assets/SIGVerse/ExampleScenes/Turtlebot3/OpenManipulator(.unity)] scene with reference to here.


You can control TurtleBot3 by keyboard operation on the grasping_auto terminal.

* Check the terminal for details of the operation.

* Because of the high image processing load, the robot's camera image transmission interval is set to 1000 [ms].
(If you want to shorten the sending interval, please reduce the Sending Interval of the ZEDMiniPubCameraImageController attached to turtlebot3_with_open_manipulator/RosBridgeScripts)

If you want to finish, stop the Unity side and then the ROS side.

Ubuntu side (The details may differ from the latest version)


Windows side (The details may differ from the latest version)


Windows side (grasped "clock") (The details may differ from the latest version)


Reference Video (Image recognition is slow because no GPU is used.)

Counter: 2508, today: 1, yesterday: 1

Attach file: fileTurtleBot3GraspAutoWindowsGrasping.png 919 download [Information] fileTurtleBot3GraspAutoWindows.png 915 download [Information] fileTurtleBot3GraspAutoUbuntu.png 915 download [Information]

Front page   Edit Freeze Diff Backup Upload Copy Rename Reload   New List of pages Search Recent changes   Help   RSS of recent changes
Last-modified: 2022-12-26 (Mon) 11:05:32 (513d)