IROS2019/Using Cloud-based VR

Overview †

A human and a robot log in to the same virtual space. The human can give a cleanup instruction to the robot by pointing.

The human logs in to the virtual space using Windows + VR device (Oculus Rift).
The robot logs in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System).

Humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space.

In this example, Photon Realtime is used for networking to synchronize multiple VR scenes.

Environment Construction †

Import Oculus Utilities †

You need to import Oculus Utilities into your Unity project.

1. Open the SIGVerse project on Unity Editor.
2. Click [Assets]-[Import Package]-[Custom Package ...] in Unity menu.
3. Select "OculusUtilities.unitypackage" in the "ovr_unity_utilities_Main\OculusUtilities" folder copied from the USB memory stick.
4. Click [Open] button.
5. Click [Import] button.
6. Click [Yes] button on the "Update Oculus Utilities Plugin" window.
7. Click [Restart] button on the "Restart Unity" window.

Modify SIGVerse Settings †

1. Click [SIGVerse]-[SIGVerse Settings] to open the SIGVerse settings.
2. Check [Use Oculus] and [Use PUN] in [Define symbols Settings].

#ref(): File not found: "SIGVerseSettingsOculusPUN.png" at page "(HSR)Cleanup Task using Cloud and VR"

Setup Photon Local server †

* Make sure that anti-virus software does not block communication.

1. Open [Photon-OnPremise-Server-SDK_v4-0-29-11263\deploy\bin_Win64] folder copied from USB memory stick.
2. Execute "PhotonControl.exe". (App icon appears on the Windows taskbar)
3. Click Photon server application in the taskbar of Windows.
4. Click [Game Server IP Config]-[Set local IP: xxx.xxx.xxx.xxx]. (Select a proper local server IP address)

   #ref(): File not found: "PhotonServerIP.png" at page "(HSR)Cleanup Task using Cloud and VR"

5. Click [LoadBalancing (MyCloud)]-[Start as application] to start Photon Server.
6. Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing)] and check that no error occurred.
7. Close the console window of the test client.

Import PUN2 assets †

1. Open the SIGVerse project on Unity Editor.
2. Access PUN2 page of Unity Asset Store.
3. Sign in to your Unity account from the top right of the page.
4. Click the [Open in Unity] button. "PUN2 - Free" is displayed on the [Asset Store] tab of Unity.
5. Uncheck "Photon/PhotonLibs/WebSocket/websocket-sharp.dll". This dll will conflict with the existing one in the SIGVerse.

   #ref(): File not found: "PUN2ExcludeWebsocket-sharp.png" at page "(HSR)Cleanup Task using Cloud and VR"

6. Click [Import] button.
   
7. Click [Skip] button on the [PUN Wizzard] window.

   #ref(): File not found: "PunWizardLocalServer.png" at page "(HSR)Cleanup Task using Cloud and VR"

8. Click [Close] button.
9. [PhotonServerSettings] will be opened. Modify the settings.
   (You can open the PhotonServerSettings from [Window]-[Photon Unity Networking]-[Highlight Server Settings].)
   • Uncheck [Use Name Server]
   • Enter the IP address of Photon Server in [Server]
   • Enter "5055" in [Port].
     
10. Click [Assets]-[Reimport All] in the Unity menu.
11. Click [Reimport] button to rebuild.

Execution Procedure †

In the case of using the local server, Photon server should be launched on a PC in advance.

Procedure to Login as a Robot (ROS) †

* ROS nodes should be launched on the Ubuntu side before the Unity scene is started.

Ubuntu side startup procedure †

1. Open a new terminal and execute the following command:

   $ roslaunch sigverse_hsr_teleop_key teleop_key.launch

Windows startup procedure †

1. Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene.
2. Click the [Robot] button to log in as a robot.

Windows (Login Screen)

Robot Operation †

You can control HSR by keyboard operation on the teleop_key.launch terminal. Keyboard commands are shown on a terminal.

Operate the robot and grasp one of the objects on the desk, then bring it to the place where the human points.

* When you stop the program, please stop the Unity side fist, then stop ROS nodes.

Ubuntu side

Windows side (Logged in)

Windows side (Pointing)

#ref(): File not found: "HSRCleanupVRRobotWindowsPointing.png" at page "(HSR)Cleanup Task using Cloud and VR"

Procedure to Login as a Human Avatar (VR) †

* This step requires a VR ready PC and a VR headset (Oculus Rift CV1 or Oculus Rift S).
Please see requirements for using Oculus Rift

Oculus Software should be installed on Windows and Oculus Rift should be set up beforehand.

Windows startup procedure †

1. Make sure that Oculus Rift is connected and ready for use.
2. Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene.
3. Click the [Human] button to log in as a human.

Windows (Login Screen)

Instruction to the Robot †

Operate the human avatar and give cleanup instructions to the robot.

The basic flow is as follows.

1. Send the message "Pick it up!" to the robot while pointing at the target object.
2. Send the message "Clean up!" to the robot while pointing at the destination.

The operation method of Oculus Touch is as follows.

• Pulling the middle finger trigger puts the avatar's hand in the pointing posture.
• When the A or X button is pressed, the message "Pick it up!" is sent to the robot.
• When the B or Y button is pressed, the message "Clean up!" is sent to the robot.
• When you tilt the left stick, the avatar moves in the tilted direction.
• When the right stick is pressed, the message "Good!" is sent to the robot.
• When the left stick is pressed, the message "Bad!" is sent to the robot.

Windows

---

Top: Tutorial at IROS2019