Tutorial using ROS for ver.3

Overview

A human and a robot log in to the same virtual space. The human can give a cleanup instruction to the robot by pointing.

The human logs in to the virtual space using Windows + VR device (Oculus Rift).
The robot logs in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System).

Humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space.

CleanupUsingCloudAndVR_Overview.png

Prerequisites

Common prerequisites

Using networks is a common prerequisite for humans and robots.

Photon Realtime is used for the cloud, and the port numbers listed on the following page are used when connecting to the Internet.
Check whether the port is available beforehand.
https://doc.photonengine.com/en-us/realtime/current/connection-and-authentication/tcp-and-udp-port-numbers

If the port is not available, the cloud cannot be used, but you can use the local network Photon Server instead.
Photon Server can only be installed on Windows.
https://doc.photonengine.com/en-us/server/current/operations/requirements

* Make sure that anti-virus software does not block communication.

The environment constructions when using the cloud and when using the local server are explained in Environment construction.

Prerequisites for Human (VR)

Prerequisites for Robot (ROS)

  • Windows environment and Ubuntu environment
    See here for details.

Environment Construction

The procedure on the human (VR) side and the procedure on the robot (ROS) side are the same.

When using the cloud, follow 4a.Build Cloud environment procedure.
When using a local server, follow 4b. Build Local server environment procedure.

1. Create Photon account

First you need to create a Photon account.
Create it according to the following procedure.

  1. Go to Photon official site.
  2. Create a Photon account by signing in at the top right of the page.

2. Import Oculus Integration

  1. Download Oculus Integration for Unity ver.14.0 from the following link.
    https://developer.oculus.com/downloads/package/unity-integration-archive/14.0/
  2. Open the SIGVerse project in Unity.
  3. Click [Assets]-[Import Package]-[Custom Package...].
  4. Select downloaded OculusIntegration_14.0.unitypackage and open the file.
  5. Click [Import] button.
  6. Click [Upgrade] when "Update Spatializer Plugins" window displayed.
  7. Click [Restart] when "Restart Unity" window displayed.
  8. Click [Yes] when "Update Oculus Utilities Plugin" window displayed.
  9. Click [Restart] when "Restart Unity" window displayed.
  10. Please confirm that no error occurred in Console window.

3. Using Oculus and PUN libraries

  1. Click [SIGVerse]-[SIGVerse Settings] to open the SIGVerse settings.
  2. Check [Use Oculus] and [Use PUN] in [Define symbols Settings].
SIGVerseSettingsOculusPUN.png

4a. Build Cloud environment

The free plan of Photon Cloud can connect up to 20 people at the same time. See the next page for details.
https://www.photonengine.com/en-US/Realtime/pricing

To connect to the cloud, use PUN (Photon Unity Networking).
The PUN environment construction procedure is as follows:

  1. Sign in to Photon official site.
  2. Get application ID.
    1. Click [CREATE A NEW APP] button on the dashboard after signing in.
    2. Set [Photon Type] to "Photon PUN" and set the application name. (For example, the application name is "CleanupVR")
    3. Click [CREATE] button.
    4. Check that the PUN app is added to the dashboard app list, and write down the "App ID".
      (The App ID is 36 digits. Click to display all digits.)
      PhotonAppId.png
  3. Import PUN2 assets into Unity project.
    1. Open the SIGVerse project in Unity.
    2. Access PUN2 page of Unity Asset Store.
    3. Sign in to your Unity account from the top right of the page.
    4. When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity.
    5. Uncheck some libraries.
      Please uncheck "Photon/PhotonLibs/WebSocket/websocket-sharp.dll". Since websocket-sharp.dll is already included in the asset, it will conflict.
      PUN2ExcludeWebsocket-sharp.png
    6. Click [Import] button.
    7. [PUN Wizzard] will be opened. Enter the above "AppID" in the "AppId" field and press the [Setup Project] button.
      PunWizardCloud.png
      The following [PhotonServerSettings] window opens.
      PhotonServerSettingsForCloud.png
      (This window can also be opened from [Window]-[Photon Unity Networking]-[PUN Wizzard]-[Locate PhotonServerSettings]. AppId is set to "App Id Realtime".)
  4. Click [Assets]-[Reimport All] in the Unity menu.
  5. Click [Reimport] button to rebuild.

4b. Build Local server environment

You can also use Photon Server on a local network instead of the cloud.

  1. Sign in to Photon official site.
  2. Install Photon Server.
    1. Access Photon Server page.
    2. Click the version number to the right of "Download SDK" to download the executable file.
    3. Run the downloaded file and extract it to the directory you want to install.
  3. Start Photon Server. (After restarting Windows, Photon Server must also be started.)
    1. Double-click PhotonControl.exe in the "Photon-OnPremise-Server-SDK_vxxxxxxx\deploy\bin_Win64" folder to start the application.
      (App icon appears on the Windows taskbar)
    2. Click the PhotonControl icon on the Windows taskbar, click [Game Server IP Config], and then select the appropriate server IP address.
      PhotonServerIP.png
    3. Click [LoadBalancing (MyCloud)]-[Start as application] to start Photon Server.
    4. Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing] to test the server.
  4. Import PUN2 assets into Unity project.
    1. Open the SIGVerse project in Unity.
    2. Access PUN2 page of Unity Asset Store.
    3. Sign in to your Unity account from the top right of the page.
    4. When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity.
    5. Uncheck some libraries.
      Please uncheck "Photon/PhotonLibs/WebSocket/websocket-sharp.dll". Since websocket-sharp.dll is already included in the asset, it will conflict.
      PUN2ExcludeWebsocket-sharp.png
    6. Click [Import] button.
    7. [PUN Wizzard] will be opened, so press the [Skip] button.
      PunWizardLocalServer.png
    8. [PhotonServerSettings] will be opened. Make the following settings.
      • Uncheck [Use Name Server]
      • Enter the IP address of Photon Server in [Server]
      • Enter "5055" in [Port].
        PhotonServerSettingsForLocalServer.png
        (This window can also be opened from [Window]-[Photon Unity Networking]-[PUN Wizzard]-[Locate PhotonServerSettings].)
  5. Click [Assets]-[Reimport All] in the Unity menu.
  6. Click [Reimport] button to rebuild.

Execution Procedure

If you want to run as a human (VR), please follow here.

If you want to run as a robot (ROS), please follow here.

It doesn't matter which one you run first, but if you want to use a local server, please Launch Photon Server first.

Human (VR) Execution procedure

Startup Procedure

* If you want to use a local server, please Launch Photon Server first.

Windows startup procedure

  1. Make sure that Oculus Rift is connected and ready for use.
  2. Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to here.
  3. Click the [Human] button to log in as a human.
  4. If the robot is not logged in, wait for the robot to log in.

Windows (Login Screen)

HSRCleanupVRLogin.png

Run

Operate the human avatar and give cleanup instructions to the robot.

The basic flow is as follows.

  1. Send the message "Pick it up!" to the robot while pointing at the target object.
  2. Send the message "Clean up!" to the robot while pointing at the destination.

The operation method of Oculus Touch is as follows.

  • Pulling the middle finger trigger puts the avatar's hand in the pointing posture.
  • When the A or X button is pressed, the message "Pick it up!" is sent to the robot.
  • When the B or Y button is pressed, the message "Clean up!" is sent to the robot.
  • When you tilt the left stick, the avatar moves in the tilted direction.
  • When the right stick is pressed, the message "Good!" is sent to the robot.
  • When the left stick is pressed, the message "Bad!" is sent to the robot.

Windows

HSRCleanupVRHuman.png

Robot (ROS) Execution procedure

Startup Procedure

* If you want to use a local server, please Launch Photon Server first.

Start the Ubuntu side and then the Windows side.

Ubuntu side startup procedure

Open a new terminal and run the following command:

$ roslaunch sigverse_hsr_teleop_key teleop_key.launch

To start rviz, execute the following command.

$ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz.launch

Windows startup procedure

  1. Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to here.
  2. Click the [Robot] button to log in as a robot.
  3. If a human is not logged in, wait for the human to log in.

Windows (Login Screen)

HSRCleanupVRLogin.png

Run

You can control HSR by keyboard operation on the teleop_key.launch terminal.

Operate the robot and grasp one of the objects on the desk, then bring it to the place where the human points.

* Check the terminal for details of the operation.

If you want to finish, stop the Unity side and then the ROS side.

Ubuntu side

HSRCleanupVRRobotUbuntu.png

Windows side (Logged in)

HSRCleanupVRRobotWindows.png

Counter: 269, today: 5, yesterday: 0

Tutorial for ver.3


Attach file: fileHSRCleanupVRRobotWindows.png 12 download [Information] fileCleanupUsingCloudAndVR_Overview.png 95 download [Information] filePhotonServerFolder.png 46 download [Information] filePhotonServerIP.png 97 download [Information] filePhotonAppId.png 65 download [Information] filePunWizardLocalServer.png 96 download [Information] filePunWizardCloud.png 64 download [Information] filePhotonServerSettingsForLocalServer.png 72 download [Information] filePhotonServerSettingsForCloud.png 68 download [Information] filePUN2ExcludeWebsocket-sharp.png 69 download [Information] fileOVRPluginFixError.png 40 download [Information] fileHSRCleanupVRRobotUbuntu.png 67 download [Information] fileHSRCleanupVRLogin.png 91 download [Information] fileHSRCleanupVRHuman.png 64 download [Information] fileSIGVerseSettingsOculusPUN.png 77 download [Information]

Front page   Edit Freeze Diff Backup Upload Copy Rename Reload   New List of pages Search Recent changes   Help   RSS of recent changes
Last-modified: 2020-05-18 (Mon) 16:01:25 (20d)