(HSR)Cleanup Task using Cloud and VR
[
Front page
] [
New
|
List of pages
|
Search
|
Recent changes
|
Help
]
Start:
[[Tutorial using ROS for ver.3]]
#contents
* Overview [#me21a0b1]
A human and a robot log in to the same virtual space. The...
The human logs in to the virtual space using Windows + VR...
The robot logs in to the virtual space using Windows + Ub...
Humans and robots (ROS) are connected via the Internet an...
#ref(CleanupUsingCloudAndVR_Overview.png)
* Prerequisites [#Prerequisites]
** Common prerequisites [#ofa841b3]
Using networks is a common prerequisite for humans and ro...
[[Photon Unity Networking>https://www.photonengine.com/en...
Check whether the port is available beforehand. ~
https://doc.photonengine.com/en-us/realtime/current/conne...
Instead of the cloud, you can also build a Photon Server ...
Photon Server can only be installed on Windows. ~
https://doc.photonengine.com/en-us/server/current/operati...
~* Make sure that anti-virus software does not block comm...
** Prerequisites for Human (VR) [#s38cc158]
*** Windows environment [#sace77e4]
[[Basic requirements for using SIGVerse assets>Tutorial f...
*** VR Headset [#j62de4b5]
You need to meet the system requirements for the VR heads...
It has been tested with ''Oculus Quest 2'' and ''VIVE Cos...
For VR devices other than these, you may need to do the S...
https://valvesoftware.github.io/steamvr_unity_plugin/tuto...
You will also need to install [[Steam>https://store.steam...
Download the installer from the official website and inst...
Please install [[Oculus Software>https://www.oculus.com/s...
Please install [[VIVE Software>https://www.vive.com/us/se...
** Prerequisites for Robot (ROS) [#y682edd3]
-Windows environment and Ubuntu environment ~
See [[here>Tutorial using ROS for ver.3#SystemRequirement...
* Environment Construction [#Environments]
The procedure on the human (VR) side and the procedure on...
When using the cloud, follow [[4a.Build Cloud environment...
When using a local server, follow [[4b. Build Local serve...
** 1. Create Photon account [#CreatePhotonAccount]
First you need to create a Photon account. ~
Create it according to the following procedure.
+ Go to [[Photon official site>https://www.photonengine.c...
+ Create a Photon account.
** 2. Import Unity Assets [#k0ec0826]
Import the required Unity assets.~
For detailed instructions on how to import, please refer ...
https://docs.unity3d.com/Manual/upm-ui-import.html
*** Import the SteamVR Plugin [#ia30a30b]
+ Go to the following page. ~
https://assetstore.unity.com/packages/tools/integration/s...
+ Download and import the asset.~
Confirmed with version ''2.7.3''.
+ Click [Edit]-[Project Settings...].
+ Check [XR Plug-in Management]-[Plug-in Providers]-[Open...
* If you have checked [Oculus], please uncheck that ...
* Do NOT check the [Initialize XR on Startup] checkb...
*** Import the PUN 2 - FREE (Photon Unity Networking) [#m...
+ Go to the following page. ~
https://assetstore.unity.com/packages/tools/network/pun-2...
+ Download and import the asset.~
Confirmed with version ''2.40''.~
The following should be unchecked when importing because ...
-- Photon/PhotonLibs/WebSocket/websocket-sharp.dll
-- Photon/PhotonUnityNetworking/Demos
-- SteamVR
#ref(ImportPUN.png)
+ In the "PUN Setup" window, please [Skip] and then [Clos...
** 3. Using SteamVR and PUN libraries [#y0decaa9]
+ Click [SIGVerse]-[SIGVerse Settings] to open the SIGVer...
+ Check [Use SteamVR] and [Use PUN] in [Define symbols Se...
#ref(SIGVerseSettingsSteamVRPUN.png)
** 4a. Build Cloud environment [#PhotonCloud]
The free plan of Photon Cloud can connect up to 20 people...
https://www.photonengine.com/en-US/pun/pricing
The cloud environment construction procedure is as follows:
+ Sign in to [[Photon official site>https://www.photoneng...
+ Get application ID.
++ Click [CREATE A NEW APP] button on the dashboard after...
++ Set [Photon Type] to "Photon PUN" and set the applicat...
++ Click [CREATE] button.
++ Check that the PUN app is added to the dashboard app l...
(The App ID is 36 digits. Click to display all digits.)
#ref(PhotonAppId.png)
+ Configure the PUN settings.
++ Open the SIGVerse project in Unity.
++ Click [Window]-[Photon Unity Networking]-[PUN Wizard].
++ Click [Locate PhotonServerSettings].
++ Enter the above "App ID" in the [Server/Cloud Settings...
#ref(PhotonServerSettingsForCloud.png)
+ Click [Assets]-[''Reimport All''] in the Unity menu.
+ Click [Reimport] button to rebuild.
** 4b. Build Local server environment [#PhotonServer]
You can also use [[Photon Server>https://doc.photonengine...
+ Sign in to [[Photon official site>https://www.photoneng...
+ Install Photon Server.
++ Access [[Photon Server page>https://www.photonengine.c...
++ Click the version number to the right of "Download SDK...
++ Extract the downloaded file to the directory you want ...
++ Go to the following page and download the license file...
https://dashboard.photonengine.com/selfhosted
++ Copy the downloaded license file ([Mail Address].Photo...
++ If you want to provide the service to external machine...
(If you are using the service only on your local machine,...
Set the PublicIPAddress in \deploy\LoadBalancing\GameServ...
If you set PublicIPAddress to blank, the IP address will ...
https://doc.photonengine.com/en-us/server/current/applica...
+ Start Photon Server. (''After restarting Windows, you w...
++ Double-click PhotonControl.exe in the "\deploy\bin_Win...
(App icon appears on the Windows taskbar)
++ Click [LoadBalancing (MyCloud)]-[Start as application]...
+ Configure the PUN settings.
++ Open the SIGVerse project in Unity.
++ Click [Window]-[Photon Unity Networking]-[PUN Wizard].
++ Click [Locate PhotonServerSettings].
++ Make the following settings.
--- Uncheck [Use Name Server]
--- Enter the IP address of Photon Server in [Server]
--- Enter "5055" in [Port].
#ref(PhotonServerSettingsForLocalServer.png)
+ Click [Assets]-[Reimport All] in the Unity menu.
+ Click [Reimport] button to rebuild.
* Execution Procedure[#Execution]
If you want to run as a human (VR), please follow [[here>...
If you want to run as a robot (ROS), please follow [[here...
It doesn't matter which one you run first, but if you wan...
*** Note [#o0c0ca14]
If you try to use VMware to run the human side and the ro...
This is because the built executable will try to use VR e...
https://github.com/ValveSoftware/unity-xr-plugin/issues/80
To solve this problem, you may need to run the robot side...
* Human (VR) Execution procedure [#human_execution]
** Startup Procedure [#vafd19d3]
'' * If you want to use a local server, please [[Launch P...
*** Windows startup procedure [#t197903c]
+ Launch SteamVR and the VR software to make the VR heads...
+ Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanup...
+ Click the [Human] button to log in as a human.
+ If the robot is not logged in, wait for the robot to lo...
Windows (Login Screen)
#ref(HSRCleanupVRLogin.png)
** Run [#v04386d7]
Operate the human avatar and give cleanup instructions to...
The basic flow is as follows.
+ Send the message "Pick it up!" to the robot while point...
+ Send the message "Clean up!" to the robot while pointin...
The operation of the VR controller is as follows.
-When pull the trigger, the avatar's hand will be in the ...
-When the front button is pressed, the message "Pick it u...
-When the back button is pressed, the message "Clean up!"...
-When you tilt the left stick, the avatar moves in the ti...
-When the right stick is pressed, the message "Good!" is ...
-When the left stick is pressed, the message "Bad!" is se...
Windows (The details may differ from the latest version)
#ref(HSRCleanupVRHuman.png)
''Reference Video''
#youtube(XhcOVdM-6OI)
* Robot (ROS) Execution procedure [#robot_execution]
** Startup Procedure [#vafd19d3]
~'' * If you want to use a local server, please [[Launch ...
Start the Ubuntu side and then the Windows side.
*** Ubuntu side startup procedure [#qd09b4ab]
Open a new terminal and run the following command:
$ roslaunch sigverse_hsr_teleop_key teleop_key.launch
To start rviz, execute the following command.
$ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz...
*** Windows startup procedure [#t197903c]
+ Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanup...
+ Click the [Robot] button to log in as a robot.
+ If a human is not logged in, wait for the human to log ...
Windows (Login Screen)
#ref(HSRCleanupVRLogin.png)
** Run [#v04386d7]
You can control HSR by keyboard operation on the teleop_k...
Operate the robot and grasp one of the objects on the des...
~* Check the terminal for details of the operation.
If you want to finish, stop the Unity side and then the R...
Ubuntu side (The details may differ from the latest versi...
#ref(HSRCleanupVRRobotUbuntu.png)
Windows side (Logged in) (The details may differ from the...
#ref(HSRCleanupVRRobotWindows.png)
''Reference Video''
#youtube(QE4zfDnHjn8)
----
#counter
[[Tutorial for ver.3]]
End:
[[Tutorial using ROS for ver.3]]
#contents
* Overview [#me21a0b1]
A human and a robot log in to the same virtual space. The...
The human logs in to the virtual space using Windows + VR...
The robot logs in to the virtual space using Windows + Ub...
Humans and robots (ROS) are connected via the Internet an...
#ref(CleanupUsingCloudAndVR_Overview.png)
* Prerequisites [#Prerequisites]
** Common prerequisites [#ofa841b3]
Using networks is a common prerequisite for humans and ro...
[[Photon Unity Networking>https://www.photonengine.com/en...
Check whether the port is available beforehand. ~
https://doc.photonengine.com/en-us/realtime/current/conne...
Instead of the cloud, you can also build a Photon Server ...
Photon Server can only be installed on Windows. ~
https://doc.photonengine.com/en-us/server/current/operati...
~* Make sure that anti-virus software does not block comm...
** Prerequisites for Human (VR) [#s38cc158]
*** Windows environment [#sace77e4]
[[Basic requirements for using SIGVerse assets>Tutorial f...
*** VR Headset [#j62de4b5]
You need to meet the system requirements for the VR heads...
It has been tested with ''Oculus Quest 2'' and ''VIVE Cos...
For VR devices other than these, you may need to do the S...
https://valvesoftware.github.io/steamvr_unity_plugin/tuto...
You will also need to install [[Steam>https://store.steam...
Download the installer from the official website and inst...
Please install [[Oculus Software>https://www.oculus.com/s...
Please install [[VIVE Software>https://www.vive.com/us/se...
** Prerequisites for Robot (ROS) [#y682edd3]
-Windows environment and Ubuntu environment ~
See [[here>Tutorial using ROS for ver.3#SystemRequirement...
* Environment Construction [#Environments]
The procedure on the human (VR) side and the procedure on...
When using the cloud, follow [[4a.Build Cloud environment...
When using a local server, follow [[4b. Build Local serve...
** 1. Create Photon account [#CreatePhotonAccount]
First you need to create a Photon account. ~
Create it according to the following procedure.
+ Go to [[Photon official site>https://www.photonengine.c...
+ Create a Photon account.
** 2. Import Unity Assets [#k0ec0826]
Import the required Unity assets.~
For detailed instructions on how to import, please refer ...
https://docs.unity3d.com/Manual/upm-ui-import.html
*** Import the SteamVR Plugin [#ia30a30b]
+ Go to the following page. ~
https://assetstore.unity.com/packages/tools/integration/s...
+ Download and import the asset.~
Confirmed with version ''2.7.3''.
+ Click [Edit]-[Project Settings...].
+ Check [XR Plug-in Management]-[Plug-in Providers]-[Open...
* If you have checked [Oculus], please uncheck that ...
* Do NOT check the [Initialize XR on Startup] checkb...
*** Import the PUN 2 - FREE (Photon Unity Networking) [#m...
+ Go to the following page. ~
https://assetstore.unity.com/packages/tools/network/pun-2...
+ Download and import the asset.~
Confirmed with version ''2.40''.~
The following should be unchecked when importing because ...
-- Photon/PhotonLibs/WebSocket/websocket-sharp.dll
-- Photon/PhotonUnityNetworking/Demos
-- SteamVR
#ref(ImportPUN.png)
+ In the "PUN Setup" window, please [Skip] and then [Clos...
** 3. Using SteamVR and PUN libraries [#y0decaa9]
+ Click [SIGVerse]-[SIGVerse Settings] to open the SIGVer...
+ Check [Use SteamVR] and [Use PUN] in [Define symbols Se...
#ref(SIGVerseSettingsSteamVRPUN.png)
** 4a. Build Cloud environment [#PhotonCloud]
The free plan of Photon Cloud can connect up to 20 people...
https://www.photonengine.com/en-US/pun/pricing
The cloud environment construction procedure is as follows:
+ Sign in to [[Photon official site>https://www.photoneng...
+ Get application ID.
++ Click [CREATE A NEW APP] button on the dashboard after...
++ Set [Photon Type] to "Photon PUN" and set the applicat...
++ Click [CREATE] button.
++ Check that the PUN app is added to the dashboard app l...
(The App ID is 36 digits. Click to display all digits.)
#ref(PhotonAppId.png)
+ Configure the PUN settings.
++ Open the SIGVerse project in Unity.
++ Click [Window]-[Photon Unity Networking]-[PUN Wizard].
++ Click [Locate PhotonServerSettings].
++ Enter the above "App ID" in the [Server/Cloud Settings...
#ref(PhotonServerSettingsForCloud.png)
+ Click [Assets]-[''Reimport All''] in the Unity menu.
+ Click [Reimport] button to rebuild.
** 4b. Build Local server environment [#PhotonServer]
You can also use [[Photon Server>https://doc.photonengine...
+ Sign in to [[Photon official site>https://www.photoneng...
+ Install Photon Server.
++ Access [[Photon Server page>https://www.photonengine.c...
++ Click the version number to the right of "Download SDK...
++ Extract the downloaded file to the directory you want ...
++ Go to the following page and download the license file...
https://dashboard.photonengine.com/selfhosted
++ Copy the downloaded license file ([Mail Address].Photo...
++ If you want to provide the service to external machine...
(If you are using the service only on your local machine,...
Set the PublicIPAddress in \deploy\LoadBalancing\GameServ...
If you set PublicIPAddress to blank, the IP address will ...
https://doc.photonengine.com/en-us/server/current/applica...
+ Start Photon Server. (''After restarting Windows, you w...
++ Double-click PhotonControl.exe in the "\deploy\bin_Win...
(App icon appears on the Windows taskbar)
++ Click [LoadBalancing (MyCloud)]-[Start as application]...
+ Configure the PUN settings.
++ Open the SIGVerse project in Unity.
++ Click [Window]-[Photon Unity Networking]-[PUN Wizard].
++ Click [Locate PhotonServerSettings].
++ Make the following settings.
--- Uncheck [Use Name Server]
--- Enter the IP address of Photon Server in [Server]
--- Enter "5055" in [Port].
#ref(PhotonServerSettingsForLocalServer.png)
+ Click [Assets]-[Reimport All] in the Unity menu.
+ Click [Reimport] button to rebuild.
* Execution Procedure[#Execution]
If you want to run as a human (VR), please follow [[here>...
If you want to run as a robot (ROS), please follow [[here...
It doesn't matter which one you run first, but if you wan...
*** Note [#o0c0ca14]
If you try to use VMware to run the human side and the ro...
This is because the built executable will try to use VR e...
https://github.com/ValveSoftware/unity-xr-plugin/issues/80
To solve this problem, you may need to run the robot side...
* Human (VR) Execution procedure [#human_execution]
** Startup Procedure [#vafd19d3]
'' * If you want to use a local server, please [[Launch P...
*** Windows startup procedure [#t197903c]
+ Launch SteamVR and the VR software to make the VR heads...
+ Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanup...
+ Click the [Human] button to log in as a human.
+ If the robot is not logged in, wait for the robot to lo...
Windows (Login Screen)
#ref(HSRCleanupVRLogin.png)
** Run [#v04386d7]
Operate the human avatar and give cleanup instructions to...
The basic flow is as follows.
+ Send the message "Pick it up!" to the robot while point...
+ Send the message "Clean up!" to the robot while pointin...
The operation of the VR controller is as follows.
-When pull the trigger, the avatar's hand will be in the ...
-When the front button is pressed, the message "Pick it u...
-When the back button is pressed, the message "Clean up!"...
-When you tilt the left stick, the avatar moves in the ti...
-When the right stick is pressed, the message "Good!" is ...
-When the left stick is pressed, the message "Bad!" is se...
Windows (The details may differ from the latest version)
#ref(HSRCleanupVRHuman.png)
''Reference Video''
#youtube(XhcOVdM-6OI)
* Robot (ROS) Execution procedure [#robot_execution]
** Startup Procedure [#vafd19d3]
~'' * If you want to use a local server, please [[Launch ...
Start the Ubuntu side and then the Windows side.
*** Ubuntu side startup procedure [#qd09b4ab]
Open a new terminal and run the following command:
$ roslaunch sigverse_hsr_teleop_key teleop_key.launch
To start rviz, execute the following command.
$ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz...
*** Windows startup procedure [#t197903c]
+ Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanup...
+ Click the [Robot] button to log in as a robot.
+ If a human is not logged in, wait for the human to log ...
Windows (Login Screen)
#ref(HSRCleanupVRLogin.png)
** Run [#v04386d7]
You can control HSR by keyboard operation on the teleop_k...
Operate the robot and grasp one of the objects on the des...
~* Check the terminal for details of the operation.
If you want to finish, stop the Unity side and then the R...
Ubuntu side (The details may differ from the latest versi...
#ref(HSRCleanupVRRobotUbuntu.png)
Windows side (Logged in) (The details may differ from the...
#ref(HSRCleanupVRRobotWindows.png)
''Reference Video''
#youtube(QE4zfDnHjn8)
----
#counter
[[Tutorial for ver.3]]
Page: