[[Tutorial using ROS for ver.3]] #contents * Overview [#me21a0b1] A human and a robot log in to the same virtual space, and a human gives a cleanup instruction to the robot by pointing. The human logs in to the virtual space using a Windows + VR device (Oculus Rift).~ The robot log in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System). Humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space. #ref(CleanupUsingCloudAndVR_Overview.png) * Prerequisites [#r10a7b3e] ** Common prerequisites [#ofa841b3] Using networks is a common prerequisite for humans and robots. [[Photon Realtime>https://www.photonengine.com/en-US/Realtime]] is used for the cloud, and the port numbers listed on the following page are used when connecting to the Internet. ~ Check whether the port is available beforehand. ~ https://doc.photonengine.com/en-us/realtime/current/connection-and-authentication/tcp-and-udp-port-numbers If the port numbers are not available, the cloud cannot be used, but it is possible to replace Photon Server on the local network. ~ Photon Server is only compatible with Windows. ~ https://doc.photonengine.com/en-us/server/current/operations/requirements The environment constructions when using the cloud and when using the local server are explained in [[Environment construction>#Environments]]. ** Prerequisites for Human (VR) [#s38cc158] -Windows environment ~ In addition to [[basic requirements for using SIGVerse assets>Tutorial for ver.3#SystemRequirements]], [[requirements for using Oculus Rift>https://support.oculus.com/248749509016567/]] is also required. -Oculus Rift ~ Please use Oculus Rift CV1 or Oculus Rift S. ~ Please install [[Oculus Software>https://www.oculus.com/setup/]] on Windows and prepare a basic environment to use Oculus Rift. ** Prerequisites for Robot (ROS) [#y682edd3] -Windows environment and Ubuntu environment ~ See [[here>Tutorial using ROS for ver.3#SystemRequirements]] for details. * Environment Construction [#Environments] The procedure on the human (VR) side and the procedure on the robot (ROS) side are the same. When using the cloud, follow [[4a.Build Cloud environment>#PhotonCloud]] procedure.~ When using a local server, follow [[4b. Build Local server environment>#PhotonServer]] procedure. ** 1. Create a Photon account [#CreatePhotonAccount] First you need to create a Photon account. ~ Create it according to the following procedure. + Go to [[Photon official site>https://www.photonengine.com/]]. + Create a Photon account by signing in at the top right of the page. ** 2. Import Oculus Utilities [#k0ec0826] You need to import Oculus Utilities into your Unity project. + Download Oculus Utilities for Unity ver.1.28.0 from the link below. ~ https://developer.oculus.com/downloads/package/oculus-utilities-for-unity-5/1.28.0/ + Unzip the downloaded zip file. + Open the SIGVerse project in Unity. + Click [Assets]-[Import Package]-[Custom Package ...] in Unity menu. + Select "OculusUtilities.unitypackage" in the extracted folder. + Click [Import] button. + Click [Yes] button on the "Update Oculus Utilities Plugin" window. + Click [Restart] button on the "Restart Unity" window. ** 3. Using Oculus and PUN libraries [#y0decaa9] + Click [SIGVerse]-[SIGVerse Settings] to open the SIGVerse settings. + Check [Use Oculus] and [Use PUN] in [Define symbols Settings]. #ref(SIGVerseSettingsOculusPUN.png) ** 4a. Build Cloud environment [#PhotonCloud] The free plan of Photon Cloud can connect up to 20 people at the same time. See the next page for details. ~ https://www.photonengine.com/en-US/Realtime/pricing To connect to the cloud, use [[PUN (Photon Unity Networking)>https://www.photonengine.com/en-US/PUN]]. ~ The PUN environment construction procedure is as follows: + Sign in to [[Photon official site>https://www.photonengine.com/]]. + Get application ID. ++ Click [CREATE A NEW APP] button on the dashboard after signing in. ++ Set [Photon Type] to "Photon PUN" and set the application name. (For example, the application name is "CleanupVR") ++ Click [CREATE] button. ++ Check that the PUN app is added to the dashboard app list, and write down the "App ID". ~ (The App ID is 36 digits. Click to display all digits.) #ref(PhotonAppId.png) + Import PUN2 assets into Unity project. ++ Open the SIGVerse project in Unity. ++ Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store. ++ Sign in to your Unity account from the top right of the page. ++ When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity. ++ Import some libraries.~ At this time, please clear the check box of ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated. #ref(PUN2ExcludeWebsocket-sharp.png) ++ [PUN Wizzard] will be opened. Enter the above "Application ID" in the "AppId" field and press the [Setup Project] button. ~ (AppId can also be set from [Window]-[Photon Unity Networking]-[PUN Wizzard]-[Locate PhotonServerSettings]. Set AppId to "App Id Realtime".) #ref(PunWizardCloud.png) The following [PhotonServerSettings] information opens. #ref(PhotonServerSettingsForCloud.png) + Click [Assets]-[Reimport All] in the Unity menu. + Click the [Reimport] button to rebuild. ** 4b. Build Local server environment [#PhotonServer] You can also set up Photon Server on your local network to replace the cloud. + Sign in to [[Photon official site>https://www.photonengine.com/]]. + Obtain a license file. ++ Click [TRY FREE LICENSE] in [[Photon Server page>https://www.photonengine.com/en-US/Server]]. ++ Click [DOWNLOAD FREE] to get a license file. ~ (The license file is named "email address. Photon-vX.free.100-ccu.license".) + Install Photon Server. ++ Access [[Photon Server page>https://www.photonengine.com/en-US/sdks#server-sdkserverserver]]. ++ Click the version number to the right of "Download SDK" to download the executable file. ++ Download the executable file and extract it to the directory you want to install. ++ Open "Photon-OnPremise-Server-SDK_vxxxxxxx \ deploy \ bin_Win64" in the expanded folder in Explorer. ++ Place the above license file. For example, the following figure. #ref(PhotonServerFolder.png) + Start Photon Server. ("After restarting Windows, Photon Server must also be started.") & Aname (LaunchPhotonServer); ++ Double-click PhotonControl.exe in the installed folder to start the application. ~ (App icon appears on the taskbar) ++ Click the PhotonControl icon on the Windows taskbar and select the appropriate server IP address in [Game Server IP Config]. #ref(PhotonServerIP.png) ++ Click [LoadBalancing (MyCloud)]-[Start as application] to start Photon Server. ++ Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing] to check the operation. + Import PUN2 assets into Unity project. ++ Open the SIGVerse project in Unity. ++ Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store. ++ Sign in to your Unity account from the top right of the page. ++ When you click the [Open in Unity] button, PUN2 is displayed on the [Asset Store] tab of Unity. Import it. ~ At this time, please clear the check box of ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' websocket-sharp.dll is already included in the SIGVerse asset and will be duplicated. #ref(PUN2ExcludeWebsocket-sharp.png) ++ [PUN Wizzard] will be opened, so press the [Skip] button. #ref(PunWizardLocalServer.png) ++ [PhotonServerSettings] will be opened. Make the following settings. --- Uncheck [User Name Server] --- Enter the IP address of Photon Server in [Server] --- Enter "5055" in [Port]. #ref(PhotonServerSettingsForLocalServer.png) + Click [Assets]-[Reimport All] in the Unity menu. + Click the [Reimport] button to rebuild. * Execution procedure[#s9ea839c] If you want to run as a human (VR) side, please follow the [[here>#human_execution]] procedure. When executing as the robot (ROS) side, please execute according to the procedure of [[here>#robot_execution]]. It doesn't matter which one you run first, but if you want to use a local server, please [[Launch Photon Server>#LaunchPhotonServer]] first. * Human (VR) Execution procedure [#human_execution] ** Startup Procedure [#vafd19d3] '' * If you want to use a local server, please [[Launch Photon Server>#LaunchPhotonServer]] first. '' *** Windows startup procedure [#t197903c] + Make sure that Oculus Rift is connected and ready for use. + Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]]. + Click the [Human] button to log in as a human. + If the robot is not logged in, wait for the robot to log in. Windows (Login Screen) #ref(HSRCleanupVRLogin.png) ** Run [#v04386d7] Operate the human avatar and issue cleanup instructions to the robot. The basic flow is as follows. + Send the message “Pick it up!” To the robot while pointing at the object. + Send the message “Clean up!” While pointing at the delivery address. The operation method of Oculus Touch is as follows. -Pulling the middle finger trigger puts your hand in a pointing posture. -When the A or X button is pressed, the message "Pick it up!" Is sent to the robot. -When the B or Y button is pressed, the message “Clean up!” Is sent to the robot. -If you defeat the left stick, it will move in the direction you defeated. -When the right stick is pressed, the message "Good!" Is sent to the robot. -When the left stick is pressed, the message "Bad!" Is sent to the robot. Windows #ref(HSRCleanupVRHuman.png) * Robot (ROS) Execution procedure [#robot_execution] ** Startup Procedure [#vafd19d3] ~'' * If you want to use a local server, please [[Launch Photon Server> #LaunchPhotonServer]] first. '' Start the Ubuntu side and then the Windows side. *** Ubuntu side startup procedure [#qd09b4ab] Open a new terminal and run the following command: $ roslaunch sigverse_hsr_teleop_key teleop_key.launch To start rviz, execute the following command. $ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz.launch *** Windows startup procedure [#t197903c] + Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene with reference to [[here>Tutorial using ROS for ver.3#open_scene]]. + Click the [Robot] button to log in as a robot. + If a human is not logged in, wait for the human to log in. Windows (Login Screen) #ref(HSRCleanupVRLogin.png) ** Run [#v04386d7] You can operate the HSR key on the teleop_key.launch terminal on Ubuntu side. Operate the robot, select one of the objects on the desk, hold it, and carry it to the place pointed by the person. * Check the terminal display for details of the operation method. [#w91954f2] When exiting, stop the Unity side before exiting the ROS side. Ubuntu side #ref(HSRCleanupVRRobotUbuntu.png) Windows side (Logged in) #ref(HSRCleanupVRRobotWindows.png) Windows side (Pointing) #ref(HSRCleanupVRRobotWindowsPointing.png)