Backup of (HSR)Cleanup Task using Cloud and VR (No. 3)

[ Front page ]   [ New | List of pages | Search | Recent changes | Help ]

• List of Backups
• View the diff.
• View the diff current.
• View the source.
• Go to (HSR)Cleanup Task using Cloud and VR.
  • 1 (2019-10-28 (Mon) 18:16:24)
  • 2 (2019-10-28 (Mon) 21:15:44)
  • 3 (2019-10-28 (Mon) 21:43:14)
  • 4 (2019-10-29 (Tue) 17:46:59)
  • 5 (2019-10-29 (Tue) 18:03:32)
  • 6 (2019-10-30 (Wed) 20:48:23)
  • 7 (2019-11-01 (Fri) 16:36:50)
  • 8 (2019-11-05 (Tue) 17:52:27)
  • 9 (2019-11-07 (Thu) 16:46:01)
  • 10 (2019-11-11 (Mon) 15:13:32)
  • 11 (2020-02-04 (Tue) 15:25:52)
  • 12 (2020-03-25 (Wed) 16:02:10)
  • 13 (2020-05-18 (Mon) 16:01:08)
  • 14 (2020-05-18 (Mon) 16:01:25)
  • 15 (2021-08-18 (Wed) 02:28:53)
  • 16 (2021-08-19 (Thu) 04:14:20)
  • 17 (2021-08-19 (Thu) 04:23:47)
  • 18 (2021-12-23 (Thu) 15:13:18)
  • 19 (2022-02-06 (Sun) 05:02:02)
  • 20 (2022-02-07 (Mon) 12:27:31)
  • 21 (2022-06-24 (Fri) 16:21:03)

---

Tutorial using ROS for ver.3

Overview †

A human and a robot log in to the same virtual space, and a human gives a cleanup instruction to the robot by pointing.

The human logs in to the virtual space using a Windows + VR device (Oculus Rift).
The robot log in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System).

Humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space.

Prerequisites †

Common prerequisites †

Using networks is a common prerequisite for humans and robots.

Photon Realtime is used for the cloud, and the port numbers listed on the following page are used when connecting to the Internet.
Check whether the port is available beforehand.
https://doc.photonengine.com/en-us/realtime/current/connection-and-authentication/tcp-and-udp-port-numbers

If the port numbers are not available, the cloud cannot be used, but it is possible to replace Photon Server on the local network.
Photon Server is only compatible with Windows.
https://doc.photonengine.com/en-us/server/current/operations/requirements

The environment constructions when using the cloud and when using the local server are explained in Environment construction.

Prerequisites for Human (VR) †

• Windows environment
  In addition to basic requirements for using SIGVerse assets, requirements for using Oculus Rift is also required.
• Oculus Rift
  Please use Oculus Rift CV1 or Oculus Rift S.
  Please install Oculus Software on Windows and prepare a basic environment to use Oculus Rift.

Prerequisites for Robot (ROS) †

• Windows environment and Ubuntu environment
  See here for details.

Environment Construction †

The procedure on the human (VR) side and the procedure on the robot (ROS) side are the same.

When using the cloud, follow 4a.Build Cloud environment procedure.
When using a local server, follow 4b. Build Local server environment procedure.

1. Create a Photon account †

First you need to create a Photon account.
Create it according to the following procedure.

1. Go to Photon official site.
2. Create a Photon account by signing in at the top right of the page.

2. Import Oculus Utilities †

You need to import Oculus Utilities into your Unity project.

1. Download Oculus Utilities for Unity ver.1.28.0 from the link below.
   https://developer.oculus.com/downloads/package/oculus-utilities-for-unity-5/1.28.0/
2. Unzip the downloaded zip file.
3. Open the SIGVerse project in Unity.
4. Click [Assets]-[Import Package]-[Custom Package ...] in Unity menu.
5. Select "OculusUtilities.unitypackage" in the extracted folder.
6. Click [Import] button.
7. Click [Yes] button on the "Update Oculus Utilities Plugin" window.
8. Click [Restart] button on the "Restart Unity" window.

3. Using Oculus and PUN libraries †

1. Click [SIGVerse]-[SIGVerse Settings] to open the SIGVerse settings.
2. Check [Use Oculus] and [Use PUN] in [Define symbols Settings].

#ref(): File not found: "SIGVerseSettingsOculusPUN.png" at page "(HSR)Cleanup Task using Cloud and VR"

4a. Build Cloud environment †

The free plan of Photon Cloud can connect up to 20 people at the same time. See the next page for details.
https://www.photonengine.com/en-US/Realtime/pricing

To connect to the cloud, use PUN (Photon Unity Networking).
The PUN environment construction procedure is as follows:

1. Sign in to Photon official site.
2. Get application ID.
   1. Click [CREATE A NEW APP] button on the dashboard after signing in.
   2. Set [Photon Type] to "Photon PUN" and set the application name. (For example, the application name is "CleanupVR")
   3. Click [CREATE] button.
   4. Check that the PUN app is added to the dashboard app list, and write down the "App ID".
      (The App ID is 36 digits. Click to display all digits.)
      
3. Import PUN2 assets into Unity project.
   1. Open the SIGVerse project in Unity.
   2. Access PUN2 page of Unity Asset Store.
   3. Sign in to your Unity account from the top right of the page.
   4. When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity.
   5. Import some libraries.
      At this time, please clear the check box of "Photon/PhotonLibs/WebSocket/websocket-sharp.dll". websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated.

      #ref(): File not found: "PUN2ExcludeWebsocket-sharp.png" at page "(HSR)Cleanup Task using Cloud and VR"

   6. [PUN Wizzard] will be opened. Enter the above "Application ID" in the "AppId" field and press the [Setup Project] button.
      (AppId can also be set from [Window]-[Photon Unity Networking]-[PUN Wizzard]-[Locate PhotonServerSettings]. Set AppId to "App Id Realtime".)

      #ref(): File not found: "PunWizardCloud.png" at page "(HSR)Cleanup Task using Cloud and VR"

      The following [PhotonServerSettings] information opens.
      
4. Click [Assets]-[Reimport All] in the Unity menu.
5. Click the [Reimport] button to rebuild.

4b. Build Local server environment †

You can also set up Photon Server on your local network to replace the cloud.

1. Sign in to Photon official site.
2. Obtain a license file.
   1. Click [TRY FREE LICENSE] in Photon Server page.
   2. Click [DOWNLOAD FREE] to get a license file.
      (The license file is named "email address. Photon-vX.free.100-ccu.license".)
3. Install Photon Server.
   1. Access Photon Server page.
   2. Click the version number to the right of "Download SDK" to download the executable file.
   3. Download the executable file and extract it to the directory you want to install.
   4. Open "Photon-OnPremise-Server-SDK_vxxxxxxx \ deploy \ bin_Win64" in the expanded folder in Explorer.
   5. Place the above license file. For example, the following figure.

      #ref(): File not found: "PhotonServerFolder.png" at page "(HSR)Cleanup Task using Cloud and VR"

4. Start Photon Server. ("After restarting Windows, Photon Server must also be started.") & Aname (LaunchPhotonServer);
   1. Double-click PhotonControl.exe in the installed folder to start the application.
      (App icon appears on the taskbar)
   2. Click the PhotonControl icon on the Windows taskbar and select the appropriate server IP address in [Game Server IP Config].

      #ref(): File not found: "PhotonServerIP.png" at page "(HSR)Cleanup Task using Cloud and VR"

   3. Click [LoadBalancing (MyCloud)]-[Start as application] to start Photon Server.
   4. Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing] to check the operation.
5. Import PUN2 assets into Unity project.
   1. Open the SIGVerse project in Unity.
   2. Access PUN2 page of Unity Asset Store.
   3. Sign in to your Unity account from the top right of the page.
   4. When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity. Import it.
      At this time, please clear the check box of "Photon/PhotonLibs/WebSocket/websocket-sharp.dll". websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated.

      #ref(): File not found: "PUN2ExcludeWebsocket-sharp.png" at page "(HSR)Cleanup Task using Cloud and VR"

   5. [PUN Wizzard] will be opened, so press the [Skip] button.

      #ref(): File not found: "PunWizardLocalServer.png" at page "(HSR)Cleanup Task using Cloud and VR"

   6. [PhotonServerSettings] will be opened. Make the following settings.
      • Uncheck [User Name Server]
      • Enter the IP address of Photon Server in [Server]
      • Enter "5055" in [Port].
        
6. Click [Assets]-[Reimport All] in the Unity menu.
7. Click the [Reimport] button to rebuild.

Execution procedure †

If you want to run as a human (VR) side, please follow the here procedure.

When executing as the robot (ROS) side, please execute according to the procedure of here.

It doesn't matter which one you run first, but if you want to use a local server, please Launch Photon Server first.

Human (VR) Execution procedure †

Startup Procedure †

* If you want to use a local server, please Launch Photon Server first.

Windows startup procedure †

1. Make sure that Oculus Rift is connected and ready for use.
2. Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to here.
3. Click the [Human] button to log in as a human.
4. If the robot is not logged in, wait for the robot to log in.

Windows (Login Screen)

Run †

Operate the human avatar and issue cleanup instructions to the robot.

The basic flow is as follows.

1. Send the message “Pick it up!” To the robot while pointing at the object.
2. Send the message “Clean up!” While pointing at the delivery address.

The operation method of Oculus Touch is as follows.

• Pulling the middle finger trigger puts your hand in a pointing posture.
• When the A or X button is pressed, the message "Pick it up!" Is sent to the robot.
• When the B or Y button is pressed, the message “Clean up!” Is sent to the robot.
• If you defeat the left stick, it will move in the direction you defeated.
• When the right stick is pressed, the message "Good!" Is sent to the robot.
• When the left stick is pressed, the message "Bad!" Is sent to the robot.

Windows

Robot (ROS) Execution procedure †

Startup Procedure †

* If you want to use a local server, please [[Launch Photon Server> #LaunchPhotonServer]] first.

Start the Ubuntu side and then the Windows side.

Ubuntu side startup procedure †

Open a new terminal and run the following command:

$ roslaunch sigverse_hsr_teleop_key teleop_key.launch

To start rviz, execute the following command.

$ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz.launch

Windows startup procedure †

1. Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to here.
2. Click the [Robot] button to log in as a robot.
3. If a human is not logged in, wait for the human to log in.

Windows (Login Screen)

Run †

You can operate the HSR key on the teleop_key.launch terminal on Ubuntu side.

Operate the robot, select one of the objects on the desk, hold it, and carry it to the place pointed by the person.

Check the terminal display for details of the operation method. †

When exiting, stop the Unity side before exiting the ROS side.

Ubuntu side

Windows side (Logged in)

Windows side (Pointing)

#ref(): File not found: "HSRCleanupVRRobotWindowsPointing.png" at page "(HSR)Cleanup Task using Cloud and VR"

     

Site admin: inamura

PukiWiki 1.5.0 Copyright © 2001-2006 PukiWiki Developers Team. License is GPL.
Based on "PukiWiki" 1.3 by yu-ji. Powered by PHP 5.5.9-1ubuntu4.29. HTML convert time: 0.043 sec.