![[PukiWiki]](image/Concept_logo.jpg "[PukiWiki]"){kind=logo}
[[Tutorial using ROS for ver.3]] #contents * Overview [#me21a0b1] A human and a robot log in to the same virtual space, and a human gives a cleanup instruction to the robot by pointing. A human and a robot log in to the same virtual space. The human can give a cleanup instruction to the robot by pointing. The human side logs in to the virtual space using a Windows + VR device (Oculus Rift). ~ On the robot side, log in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System). The human logs in to the virtual space using Windows + VR Headset.~ The robot logs in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System). Remote humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space. Humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space. #ref(CleanupUsingCloudAndVR_Overview.png) * Prerequisites [#r10a7b3e] * Prerequisites [#Prerequisites] ** Common prerequisites [#ofa841b3] Use networks as a common prerequisite for humans and robots. Using networks is a common prerequisite for humans and robots. [[Photon Realtime>https://www.photonengine.com/en-US/Realtime]] is used for the cloud, but the port number on the following page is used when connecting to the Internet. ~ Check beforehand if the port is available. ~ [[Photon Unity Networking>https://www.photonengine.com/en-US/PUN]] is used for the cloud, and the port numbers listed on the following page are used when connecting to the Internet. ~ Check whether the port is available beforehand. ~ https://doc.photonengine.com/en-us/realtime/current/connection-and-authentication/tcp-and-udp-port-numbers If the port is not available, the cloud cannot be used, but it is possible to replace Photon Server on the local network. ~ Photon Server is only compatible with Windows. ~ Instead of the cloud, you can also build a Photon Server on your local network and use that. ~ Photon Server can only be installed on Windows. ~ https://doc.photonengine.com/en-us/server/current/operations/requirements The environment construction method when using the cloud and when using the local server is explained in [[Environment construction>#Environments]]. ~* Make sure that anti-virus software does not block communication. ** Prerequisites for the human (VR) side [#s38cc158] -Windows environment ~ In addition to [[Basic requirements for using SIGVerse assets>Tutorial for ver.3#SystemRequirements]], [[Requirements for using Oculus Rift>https://support.oculus.com/248749509016567/]] Is also required. -Oculus Rift ~ Oculus Rift has been tested with Oculus Rift CV1 and Oculus Rift S. ~ Install [[Oculus Software>https://www.oculus.com/setup/]] on Windows and prepare a basic environment to use Oculus Rift. ** Prerequisites for Human (VR) [#s38cc158] *** Windows environment [#sace77e4] [[Basic requirements for using SIGVerse assets>Tutorial for ver.3#SystemRequirements]] are needed. *** VR Headset [#j62de4b5] You need to meet the system requirements for the VR headset to work. ~ It has been tested with ''Oculus Quest 2'' and ''VIVE Cosmos''. ~ For VR devices other than these, you may need to do the SteamVR binding. ~ https://valvesoftware.github.io/steamvr_unity_plugin/tutorials/SteamVR-Input.html#binding-actions You will also need to install [[Steam>https://store.steampowered.com/about/]] and [[SteamVR>https://store.steampowered.com/app/250820/SteamVR/]] on your PC. ~ Download the installer from the official website and install it. Please install [[Oculus Software>https://www.oculus.com/setup/]] to use Oculus Headsets. [[Oculus Link>https://support.oculus.com/articles/headsets-and-accessories/oculus-link/index-oculus-link]] is also required. ~ Please install [[VIVE Software>https://www.vive.com/us/setup]] to use VIVE Headsets. ** Prerequisites for Robot (ROS) [#y682edd3] -Windows environment and Ubuntu environment ~ See [[here>Tutorial using ROS for ver.3#SystemRequirements]] for details. * Environment construction [#Environments] * Environment Construction [#Environments] The environment construction procedure is the same for the human (VR) side and the robot (ROS) side. The procedure on the human (VR) side and the procedure on the robot (ROS) side are the same. [[4a.Building a cloud environment>#PhotonCloud]] to use the cloud. ~ When using a local server, perform [[4b. Build local server environment>#PhotonServer]]. When using the cloud, follow [[4a.Build Cloud environment>#PhotonCloud]] procedure.~ When using a local server, follow [[4b. Build Local server environment>#PhotonServer]] procedure. ** 1. Create a Photon account [#CreatePhotonAccount] ** 1. Create Photon account [#CreatePhotonAccount] First you need to create a Photon account. ~ Create it according to the following procedure. + Go to [[Photon official site>https://www.photonengine.com/]]. + Create a Photon account by signing in at the top right of the page. + Create a Photon account. ** 2. Importing Oculus Utilities [#k0ec0826] You need to import Oculus Utilities into your Unity project. . ** 2. Import Unity Assets [#k0ec0826] + Download Oculus Utilities for Unity ver.1.28.0 from the link below. ~ https://developer.oculus.com/downloads/package/oculus-utilities-for-unity-5/1.28.0/ + Decompress the downloaded zip file. Open the SIGVerse project in + Unity. Click [Assets]-[Import Package]-[Custom Package ...] in the + Unity menu. + Select "OculusUtilities.unitypackage" in the extracted folder. Click the [Import] button. + Click the [Yes] button on the "Update Oculus Utilities Plugin" screen. + Click the [Restart] button on the "Restart Unity" screen. Import the required Unity assets.~ For detailed instructions on how to import, please refer to the following link.~ https://docs.unity3d.com/Manual/upm-ui-import.html *** Import the SteamVR Plugin [#ia30a30b] + Go to the following page. ~ https://assetstore.unity.com/packages/tools/integration/steamvr-plugin-32647 + Download and import the asset.~ Confirmed with version ''2.7.3''. + Click [Edit]-[Project Settings...]. + Check [XR Plug-in Management]-[Plug-in Providers]-[OpenVR Loader]. ~ * If you have checked [Oculus], please uncheck that checkbox. ~ * Do NOT check the [Initialize XR on Startup] checkbox. *** Import the PUN 2 - FREE (Photon Unity Networking) [#md3beb42] ** 3. Oculus and PUN library usage settings [#y0decaa9] + Go to the following page. ~ https://assetstore.unity.com/packages/tools/network/pun-2-free-119922 + Download and import the asset.~ Confirmed with version ''2.40''.~ The following should be unchecked when importing because these conflicts with other libraries. -- Photon/PhotonLibs/WebSocket/websocket-sharp.dll -- Photon/PhotonUnityNetworking/Demos -- SteamVR #ref(ImportPUN.png) + In the "PUN Setup" window, please [Skip] and then [Close]. + Open the SIGVerse settings screen from [SIGVerse]-[SIGVerse Settings]. + Check [Use Oculus] and [Use PUN] in [Define symbols Settings]. #ref(SIGVerseSettingsOculusPUN.png) ** 3. Using SteamVR and PUN libraries [#y0decaa9] + Click [SIGVerse]-[SIGVerse Settings] to open the SIGVerse settings. + Check [Use SteamVR] and [Use PUN] in [Define symbols Settings]. ** 4a. Build a cloud environment [#PhotonCloud] #ref(SIGVerseSettingsSteamVRPUN.png) The Photon Cloud free plan can connect up to 20 people at the same time. See the next page for details. ~ https://www.photonengine.com/en-US/Realtime/pricing To connect to the cloud, use [[PUN (Photon Unity Networking)>https://www.photonengine.com/en-US/PUN]]. ~ The PUN environment construction procedure is as follows. ** 4a. Build Cloud environment [#PhotonCloud] The free plan of Photon Cloud can connect up to 20 people at the same time. See the next page for details. ~ https://www.photonengine.com/en-US/pun/pricing The cloud environment construction procedure is as follows: + Sign in to [[Photon official site>https://www.photonengine.com/]]. + Get application ID. ++ Click the [Create a new app] button on the dashboard after signing in. ++ Set [Photon Type] to "Photon PUN" and set the application name. (The application name is "CleanupVR", for example) ++ Click the [Create] button. ++ Check that the PUN app is added to the dashboard app list, and write down the "Application ID". ~ (If you click the ID with 36 digits, all digits will be displayed.) ++ Click [CREATE A NEW APP] button on the dashboard after signing in. ++ Set [Photon Type] to "Photon PUN" and set the application name. (For example, the application name is "CleanupVR") ++ Click [CREATE] button. ++ Check that the PUN app is added to the dashboard app list, and write down the "App ID". ~ (The App ID is 36 digits. Click to display all digits.) #ref(PhotonAppId.png) + Import PUN2 assets into Unity project. + Configure the PUN settings. ++ Open the SIGVerse project in Unity. ++ Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store. ++ When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity. Import it. ~ At this time, please clear the check box of ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated. #ref(PUN2ExcludeWebsocket-sharp.png) ++ [PUN Wizzard] will be opened. Enter the above "Application ID" in the "AppId" field and press the [Setup Project] button. ~ (AppId can also be set from [Window]-[Photon Unity Networking]-[PUN Wizzard]-[Locate PhotonServerSettings]. Set AppId to "App Id Realtime".) #ref(PunWizardCloud.png) The following [PhotonServerSettings] information opens. ++ Click [Window]-[Photon Unity Networking]-[PUN Wizard]. ++ Click [Locate PhotonServerSettings]. ++ Enter the above "App ID" in the [Server/Cloud Settings]-[App Id PUN] field. #ref(PhotonServerSettingsForCloud.png) + Click [Assets]-[Reimport All] in the Unity menu. + Click the [Reimport] button to rebuild. + Click [Assets]-[''Reimport All''] in the Unity menu. + Click [Reimport] button to rebuild. ** 4b. Build local server environment [#PhotonServer] You can also set up Photon Server on your local network to replace the cloud. ** 4b. Build Local server environment [#PhotonServer] You can also use [[Photon Server>https://doc.photonengine.com/en-US/server/current/getting-started/photon-server-in-5min]] on a local network instead of the cloud. + Sign in to [[Photon official site>https://www.photonengine.com/]]. + Obtain a license file. ++ Click [TRY FREE LICENSE] in [[Photon Server page>https://www.photonengine.com/en-US/Server]]. ++ Click [DOWNLOAD FREE] to get a license file. ~ (The license file is named "email address. Photon-vX.free.100-ccu.license".) + Install Photon Server. ++ Access [[Photon Server page>https://www.photonengine.com/en-US/sdks#server-sdkserverserver]]. ++ Click the version number to the right of "Download SDK" to download the executable file. ++ Download the executable file and extract it to the directory you want to install. ++ Open "Photon-OnPremise-Server-SDK_vxxxxxxx \ deploy \ bin_Win64" in the expanded folder in Explorer. ++ Place the above license file. For example, the following figure. #ref(PhotonServerFolder.png) + Start Photon Server. ("After restarting Windows, Photon Server must also be started.") & Aname (LaunchPhotonServer); ++ Double-click PhotonControl.exe in the installed folder to start the application. ~ (App icon appears on the taskbar) ++ Click the PhotonControl icon on the Windows taskbar and select the appropriate server IP address in [Game Server IP Config]. #ref(PhotonServerIP.png) ++ Click the version number to the right of "Download SDK" to download the zip file. ++ Extract the downloaded file to the directory you want to install. ++ Go to the following page and download the license file for a "100 CCU" license. ~ https://dashboard.photonengine.com/selfhosted ++ Copy the downloaded license file ([Mail Address].Photon-vX.free.100-ccu.license) to the "\deploy\bin_Win64" folder in the Photon server folder. ++ If you want to provide the service to external machines, please set the PublicIPAddress in GameServer.xml.config. ~ (If you are using the service only on your local machine, do not make the following settings) ~ Set the PublicIPAddress in \deploy\LoadBalancing\GameServer\bin\GameServer.xml.config. ~ If you set PublicIPAddress to blank, the IP address will be recognized automatically. Please refer to the following URL for details. ~ https://doc.photonengine.com/en-us/server/current/applications/loadbalancing/application#deploying_a_game_server + Start Photon Server. (''After restarting Windows, you will need to start the Photon Server again.'') &Aname(LaunchPhotonServer); ++ Double-click PhotonControl.exe in the "\deploy\bin_Win64" folder to start the application. ~ (App icon appears on the Windows taskbar) ++ Click [LoadBalancing (MyCloud)]-[Start as application] to start Photon Server. ++ Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing] to check the operation. + Import PUN2 assets into Unity project. + Configure the PUN settings. ++ Open the SIGVerse project in Unity. ++ Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store. ++ When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity. Import it. ~ At this time, please clear the check box of ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated. #ref(PUN2ExcludeWebsocket-sharp.png) ++ [PUN Wizzard] will be opened, so press the [Skip] button. #ref(PunWizardLocalServer.png) ++ [PhotonServerSettings] will be opened. Make the following settings. --- Uncheck [User Name Server] ++ Click [Window]-[Photon Unity Networking]-[PUN Wizard]. ++ Click [Locate PhotonServerSettings]. ++ Make the following settings. --- Uncheck [Use Name Server] --- Enter the IP address of Photon Server in [Server] --- Enter "5055" in [Port]. #ref(PhotonServerSettingsForLocalServer.png) + Click [Assets]-[Reimport All] in the Unity menu. + Click the [Reimport] button to rebuild. + Click [Reimport] button to rebuild. * Execution Procedure[#Execution] * Execution procedure [#s9ea839c] If you want to run as a human (VR), please follow [[here>#human_execution]]. If you want to run as a human (VR) side, please follow the [[here>#human_execution]] procedure. If you want to run as a robot (ROS), please follow [[here>#robot_execution]]. When executing as the robot (ROS) side, please execute according to the procedure of [[here>#robot_execution]]. It doesn't matter which one you run first, but if you want to use a local server, please [[Launch Photon Server>#LaunchPhotonServer]] first. *** Note [#o0c0ca14] If you try to use VMware to run the human side and the robot side on a single PC, the app may crash. ~ This is because the built executable will try to use VR even when running the robot side. (As of July 15, 2021) ~ https://github.com/ValveSoftware/unity-xr-plugin/issues/80 * Human (VR) execution procedure [#human_execution] To solve this problem, you may need to run the robot side in the Unity editor or remove "xrsdk-pre-init-library=XRSDKOpenVR" from boot.config. ** Startup procedure [#vafd19d3] * Human (VR) Execution procedure [#human_execution] ** Startup Procedure [#vafd19d3] '' * If you want to use a local server, please [[Launch Photon Server>#LaunchPhotonServer]] first. '' *** Windows startup procedure [#t197903c] + Make sure that Oculus Rift is connected and ready for use. + Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]]. + Launch SteamVR and the VR software to make the VR headset ready for use. + Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]]. + Click the [Human] button to log in as a human. + If the robot is not logged in, wait for the robot to log in. Windows (Login Screen) #ref(HSRCleanupVRLogin.png) ** Run [# v04386d7] [#d821062e] ** Run [#v04386d7] Operate the human avatar and issue cleanup instructions to the robot. Operate the human avatar and give cleanup instructions to the robot. The basic flow is as follows. + Send the message “Pick it up!” To the robot while pointing at the object. + Send the message “Clean up!” While pointing at the delivery address. + Send the message "Pick it up!" to the robot while pointing at the target object. + Send the message "Clean up!" to the robot while pointing at the destination. The operation method of Oculus Touch is as follows. -Pulling the middle finger trigger puts your hand in a pointing posture. -When the A or X button is pressed, the message "Pick it up!" Is sent to the robot. -When the B or Y button is pressed, the message “Clean up!” Is sent to the robot. -If you defeat the left stick, it will move in the direction you defeated. -When the right stick is pressed, the message "Good!" Is sent to the robot. -When the left stick is pressed, the message "Bad!" Is sent to the robot. The operation of the VR controller is as follows. Windows -When pull the trigger, the avatar's hand will be in the pointing position. -When the front button is pressed, the message "Pick it up!" is sent to the robot. (A or X button for Oculus Touch) -When the back button is pressed, the message "Clean up!" is sent to the robot. (B or Y button for Oculus Touch) -When you tilt the left stick, the avatar moves in the tilted direction. -When the right stick is pressed, the message "Good!" is sent to the robot. -When the left stick is pressed, the message "Bad!" is sent to the robot. Windows (The details may differ from the latest version) #ref(HSRCleanupVRHuman.png) ''Reference Video'' #youtube(XhcOVdM-6OI) * Robot (ROS) Execution procedure [#robot_execution] * Robot (ROS) execution procedure [#robot_execution] ** Startup Procedure [#vafd19d3] ~'' * If you want to use a local server, please [[Launch Photon Server> #LaunchPhotonServer]] first. '' ~'' * If you want to use a local server, please [[Launch Photon Server>#LaunchPhotonServer]] first. '' Start the Ubuntu side and then the Windows side. *** Ubuntu side startup procedure [#qd09b4ab] Open a new terminal and run the following command: $ roslaunch sigverse_hsr_teleop_key teleop_key.launch To start rviz, execute the following command. $ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz.launch *** Windows startup procedure [#t197903c] + Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]]. + Start the [Assets/SIGVerse/ExampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]]. + Click the [Robot] button to log in as a robot. + If a human is not logged in, wait for the human to log in. Windows (Login Screen) #ref(HSRCleanupVRLogin.png) ** Run [#v04386d7] You can operate the HSR key on the teleop_key.launch terminal on Ubuntu side. You can control HSR by keyboard operation on the teleop_key.launch terminal. Operate the robot, select one of the objects on the desk, hold it, and carry it to the place pointed by the person. Operate the robot and grasp one of the objects on the desk, then bring it to the place where the human points. * Check the terminal display for details of the operation method. [#w91954f2] ~* Check the terminal for details of the operation. When exiting, stop the Unity side before exiting the ROS side. If you want to finish, stop the Unity side and then the ROS side. Ubuntu side Ubuntu side (The details may differ from the latest version) #ref(HSRCleanupVRRobotUbuntu.png) Windows side (Logged in) Windows side (Logged in) (The details may differ from the latest version) #ref(HSRCleanupVRRobotWindows.png) Windows side (Pointing) #ref(HSRCleanupVRRobotWindowsPointing.png) ''Reference Video'' #youtube(QE4zfDnHjn8) ---- #counter [[Tutorial for ver.3]]
Site admin: inamura
PukiWiki 1.5.0 Copyright © 2001-2006 PukiWiki Developers Team. License is GPL.