Backup source of (HSR)Cleanup Task using Cloud and VR (No. 1)

[ Front page ]   [ New | List of pages | Search | Recent changes | Help ]

• List of Backups
• View the diff.
• View the diff current.
• View the backup.
• Go to (HSR)Cleanup Task using Cloud and VR.
  • 1 (2019-10-28 (Mon) 18:16:24)
  • 2 (2019-10-28 (Mon) 21:15:44)
  • 3 (2019-10-28 (Mon) 21:43:14)
  • 4 (2019-10-29 (Tue) 17:46:59)
  • 5 (2019-10-29 (Tue) 18:03:32)
  • 6 (2019-10-30 (Wed) 20:48:23)
  • 7 (2019-11-01 (Fri) 16:36:50)
  • 8 (2019-11-05 (Tue) 17:52:27)
  • 9 (2019-11-07 (Thu) 16:46:01)
  • 10 (2019-11-11 (Mon) 15:13:32)
  • 11 (2020-02-04 (Tue) 15:25:52)
  • 12 (2020-03-25 (Wed) 16:02:10)
  • 13 (2020-05-18 (Mon) 16:01:08)
  • 14 (2020-05-18 (Mon) 16:01:25)
  • 15 (2021-08-18 (Wed) 02:28:53)
  • 16 (2021-08-19 (Thu) 04:14:20)
  • 17 (2021-08-19 (Thu) 04:23:47)
  • 18 (2021-12-23 (Thu) 15:13:18)
  • 19 (2022-02-06 (Sun) 05:02:02)
  • 20 (2022-02-07 (Mon) 12:27:31)
  • 21 (2022-06-24 (Fri) 16:21:03)

[[Tutorial using ROS for ver.3]]

#contents

* Overview [#me21a0b1]

A human and a robot log in to the same virtual space, and a human gives a cleanup instruction to the robot by pointing.

The human side logs in to the virtual space using a Windows + VR device (Oculus Rift). ~
On the robot side, log in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System).

Remote humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space.

#ref(CleanupUsingCloudAndVR_Overview.png)




* Prerequisites [#r10a7b3e]

** Common prerequisites [#ofa841b3]

Use networks as a common prerequisite for humans and robots.

[[Photon Realtime>https://www.photonengine.com/en-US/Realtime]] is used for the cloud, but the port number on the following page is used when connecting to the Internet. ~
Check beforehand if the port is available. ~
https://doc.photonengine.com/en-us/realtime/current/connection-and-authentication/tcp-and-udp-port-numbers

If the port is not available, the cloud cannot be used, but it is possible to replace Photon Server on the local network. ~
Photon Server is only compatible with Windows. ~
https://doc.photonengine.com/en-us/server/current/operations/requirements

The environment construction method when using the cloud and when using the local server is explained in [[Environment construction>#Environments]].


** Prerequisites for the human (VR) side [#s38cc158]

-Windows environment ~
In addition to [[Basic requirements for using SIGVerse assets>Tutorial for ver.3#SystemRequirements]], [[Requirements for using Oculus Rift>https://support.oculus.com/248749509016567/]] Is also required.
-Oculus Rift ~
Oculus Rift has been tested with Oculus Rift CV1 and Oculus Rift S. ~
Install [[Oculus Software>https://www.oculus.com/setup/]] on Windows and prepare a basic environment to use Oculus Rift.


** Prerequisites for Robot (ROS) [#y682edd3]

-Windows environment and Ubuntu environment ~
See [[here>Tutorial using ROS for ver.3#SystemRequirements]] for details.





* Environment construction [#Environments]

The environment construction procedure is the same for the human (VR) side and the robot (ROS) side.

[[4a.Building a cloud environment>#PhotonCloud]] to use the cloud. ~
When using a local server, perform [[4b. Build local server environment>#PhotonServer]].


** 1. Create a Photon account [#CreatePhotonAccount]

First you need to create a Photon account. ~
Create it according to the following procedure.

+ Go to [[Photon official site>https://www.photonengine.com/]].
+ Create a Photon account by signing in at the top right of the page.


** 2. Importing Oculus Utilities [#k0ec0826]

You need to import Oculus Utilities into your Unity project. .

+ Download Oculus Utilities for Unity ver.1.28.0 from the link below. ~
https://developer.oculus.com/downloads/package/oculus-utilities-for-unity-5/1.28.0/
+ Decompress the downloaded zip file.
Open the SIGVerse project in + Unity.
Click [Assets]-[Import Package]-[Custom Package ...] in the + Unity menu.
+ Select "OculusUtilities.unitypackage" in the extracted folder.
Click the [Import] button.
+ Click the [Yes] button on the "Update Oculus Utilities Plugin" screen.
+ Click the [Restart] button on the "Restart Unity" screen.




** 3. Oculus and PUN library usage settings [#y0decaa9]

+ Open the SIGVerse settings screen from [SIGVerse]-[SIGVerse Settings].
+ Check [Use Oculus] and [Use PUN] in [Define symbols Settings].

#ref(SIGVerseSettingsOculusPUN.png)



** 4a. Build a cloud environment [#PhotonCloud]

The Photon Cloud free plan can connect up to 20 people at the same time. See the next page for details. ~
https://www.photonengine.com/en-US/Realtime/pricing

To connect to the cloud, use [[PUN (Photon Unity Networking)>https://www.photonengine.com/en-US/PUN]]. ~
The PUN environment construction procedure is as follows.

+ Sign in to [[Photon official site>https://www.photonengine.com/]].
+ Get application ID.
++ Click the [Create a new app] button on the dashboard after signing in.
++ Set [Photon Type] to "Photon PUN" and set the application name. (The application name is "CleanupVR", for example)
++ Click the [Create] button.
++ Check that the PUN app is added to the dashboard app list, and write down the "Application ID". ~
(If you click the ID with 36 digits, all digits will be displayed.)
#ref(PhotonAppId.png)
+ Import PUN2 assets into Unity project.
++ Open the SIGVerse project in Unity.
++ Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store.
++ When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity. Import it. ~
At this time, please clear the check box of ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated.
#ref(PUN2ExcludeWebsocket-sharp.png)
++ [PUN Wizzard] will be opened. Enter the above "Application ID" in the "AppId" field and press the [Setup Project] button. ~
(AppId can also be set from [Window]-[Photon Unity Networking]-[PUN Wizzard]-[Locate PhotonServerSettings]. Set AppId to "App Id Realtime".)
#ref(PunWizardCloud.png)
The following [PhotonServerSettings] information opens.
#ref(PhotonServerSettingsForCloud.png)
+ Click [Assets]-[Reimport All] in the Unity menu.
+ Click the [Reimport] button to rebuild.


** 4b. Build local server environment [#PhotonServer]

You can also set up Photon Server on your local network to replace the cloud.

+ Sign in to [[Photon official site>https://www.photonengine.com/]].
+ Obtain a license file.
++ Click [TRY FREE LICENSE] in [[Photon Server page>https://www.photonengine.com/en-US/Server]].
++ Click [DOWNLOAD FREE] to get a license file. ~
(The license file is named "email address. Photon-vX.free.100-ccu.license".)
+ Install Photon Server.
++ Access [[Photon Server page>https://www.photonengine.com/en-US/sdks#server-sdkserverserver]].
++ Click the version number to the right of "Download SDK" to download the executable file.
++ Download the executable file and extract it to the directory you want to install.
++ Open "Photon-OnPremise-Server-SDK_vxxxxxxx \ deploy \ bin_Win64" in the expanded folder in Explorer.
++ Place the above license file. For example, the following figure.
#ref(PhotonServerFolder.png)
+ Start Photon Server. ("After restarting Windows, Photon Server must also be started.") & Aname (LaunchPhotonServer);
++ Double-click PhotonControl.exe in the installed folder to start the application. ~
(App icon appears on the taskbar)
++ Click the PhotonControl icon on the Windows taskbar and select the appropriate server IP address in [Game Server IP Config].
#ref(PhotonServerIP.png)
++ Click [LoadBalancing (MyCloud)]-[Start as application] to start Photon Server.
++ Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing] to check the operation.
+ Import PUN2 assets into Unity project.
++ Open the SIGVerse project in Unity.
++ Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store.
++ When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity. Import it. ~
At this time, please clear the check box of ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated.
#ref(PUN2ExcludeWebsocket-sharp.png)
++ [PUN Wizzard] will be opened, so press the [Skip] button.
#ref(PunWizardLocalServer.png)
++ [PhotonServerSettings] will be opened. Make the following settings.
--- Uncheck [User Name Server]
--- Enter the IP address of Photon Server in [Server]
--- Enter "5055" in [Port].
#ref(PhotonServerSettingsForLocalServer.png)
+ Click [Assets]-[Reimport All] in the Unity menu.
+ Click the [Reimport] button to rebuild.




* Execution procedure [#s9ea839c]

If you want to run as a human (VR) side, please follow the [[here>#human_execution]] procedure.

When executing as the robot (ROS) side, please execute according to the procedure of [[here>#robot_execution]].

It doesn't matter which one you run first, but if you want to use a local server, please [[Launch Photon Server>#LaunchPhotonServer]] first.



* Human (VR) execution procedure [#human_execution]

** Startup procedure [#vafd19d3]

'' * If you want to use a local server, please [[Launch Photon Server>#LaunchPhotonServer]] first. ''

*** Windows startup procedure [#t197903c]

+ Make sure that Oculus Rift is connected and ready for use.
+ Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]].
+ Click the [Human] button to log in as a human.
+ If the robot is not logged in, wait for the robot to log in.

Windows (Login Screen)
#ref(HSRCleanupVRLogin.png)

** Run [# v04386d7] [#d821062e]

Operate the human avatar and issue cleanup instructions to the robot.

The basic flow is as follows.
+ Send the message “Pick it up!” To the robot while pointing at the object.
+ Send the message “Clean up!” While pointing at the delivery address.


The operation method of Oculus Touch is as follows.

-Pulling the middle finger trigger puts your hand in a pointing posture.
-When the A or X button is pressed, the message "Pick it up!" Is sent to the robot.
-When the B or Y button is pressed, the message “Clean up!” Is sent to the robot.
-If you defeat the left stick, it will move in the direction you defeated.
-When the right stick is pressed, the message "Good!" Is sent to the robot.
-When the left stick is pressed, the message "Bad!" Is sent to the robot.

Windows
#ref(HSRCleanupVRHuman.png)





* Robot (ROS) execution procedure [#robot_execution]

** Startup Procedure [#vafd19d3]

~'' * If you want to use a local server, please [[Launch Photon Server> #LaunchPhotonServer]] first. ''

Start the Ubuntu side and then the Windows side.

*** Ubuntu side startup procedure [#qd09b4ab]

Open a new terminal and run the following command:
 $ roslaunch sigverse_hsr_teleop_key teleop_key.launch

To start rviz, execute the following command.
 $ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz.launch


*** Windows startup procedure [#t197903c]

+ Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]].
+ Click the [Robot] button to log in as a robot.
+ If a human is not logged in, wait for the human to log in.

Windows (Login Screen)
#ref(HSRCleanupVRLogin.png)


** Run [#v04386d7]

You can operate the HSR key on the teleop_key.launch terminal on Ubuntu side.

Operate the robot, select one of the objects on the desk, hold it, and carry it to the place pointed by the person.

* Check the terminal display for details of the operation method. [#w91954f2]

When exiting, stop the Unity side before exiting the ROS side.

Ubuntu side
#ref(HSRCleanupVRRobotUbuntu.png)

Windows side (Logged in)
#ref(HSRCleanupVRRobotWindows.png)

Windows side (Pointing)
#ref(HSRCleanupVRRobotWindowsPointing.png)

     

Site admin: inamura

PukiWiki 1.5.0 Copyright © 2001-2006 PukiWiki Developers Team. License is GPL.
Based on "PukiWiki" 1.3 by yu-ji. Powered by PHP 5.5.9-1ubuntu4.29. HTML convert time: 0.010 sec.