![[PukiWiki]](image/Concept_logo.jpg "[PukiWiki]"){kind=logo}
#ref(Tutorial at IROS2019/IROS2019_logo.png,70%) * Overview [#me21a0b1] A human and a robot log in to the same virtual space. The human can give a cleanup instruction to the robot by pointing. The human logs in to the virtual space using Windows + VR device (Oculus Rift).~ The robot logs in to the virtual space using Windows + Ubuntu (ROS: Robot Operating System). Humans and robots (ROS) are connected via the Internet and can communicate in the same virtual space. In this example, [[Photon Realtime>https://www.photonengine.com/en-US/Realtime]] is used for networking to synchronize multiple VR scenes. #ref((HSR)Cleanup Task using Cloud and VR/CleanupUsingCloudAndVR_Overview.png) //* Prerequisites [#r10a7b3e] //** Common prerequisites [#ofa841b3] //Using networks is a common prerequisite for humans and robots. //[[Photon Realtime>https://www.photonengine.com/en-US/Realtime]] is used for networking to synchronize multiple VR scenes. //~* Make sure that anti-virus software does not block communication. //The environment constructions when using the cloud and when using the local server are explained in [[Environment construction>#Environments]]. //** Prerequisites for Human (VR) [#s38cc158] //-Windows environment ~ //In addition to [[basic requirements for using SIGVerse assets>Tutorial for ver.3#SystemRequirements]], [[requirements for using Oculus Rift>https://support.oculus.com/248749509016567/]] is also required. //-Oculus Rift ~ //Please use Oculus Rift CV1 or Oculus Rift S. ~ //Please install [[Oculus Software>https://www.oculus.com/setup/]] on Windows //and prepare a basic environment to use Oculus Rift. //** Prerequisites for Robot (ROS) [#y682edd3] //-Windows environment and Ubuntu environment ~ //See [[here>Tutorial using ROS for ver.3#SystemRequirements]] for details. * Environment Construction [#Environments] //The procedure on the human (VR) side and the procedure on the robot (ROS) side are the same. ** Import Oculus Utilities [#k0ec0826] You need to import Oculus Utilities into your Unity project. + Open the SIGVerse project on Unity Editor. + Click [Assets]-[Import Package]-[Custom Package ...] in Unity menu. + Select "OculusUtilities.unitypackage" in the "ovr_unity_utilities_Main\OculusUtilities" folder copied from the USB memory stick. + Click [Open] button. + Click [Import] button. + Click [Yes] button on the "Update Oculus Utilities Plugin" window. + Click [Restart] button on the "Restart Unity" window. ** Import PUN2 assets [#qb0ad89e] + Open the SIGVerse project on Unity Editor. + Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store. + Sign in to your Unity account from the top right of the page. + Click the [Open in Unity] button. "PUN2 - Free" is displayed on the [Asset Store] tab of Unity. + Uncheck ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' This dll will conflict with the existing one in the SIGVerse. #ref((HSR)Cleanup Task using Cloud and VR/PUN2ExcludeWebsocket-sharp.png) + Click [Import] button.~ + Click [Skip] button on the [PUN Wizzard] window. #ref((HSR)Cleanup Task using Cloud and VR/PunWizardLocalServer.png) + Click [Close] button. + [PhotonServerSettings] will be opened. Modify the settings.~ (You can open the PhotonServerSettings from [Window]-[Photon Unity Networking]-[Highlight Server Settings].) --- Uncheck [Use Name Server] --- Enter the IP address of Photon Server in [Server] --- Enter "5055" in [Port]. #ref((HSR)Cleanup Task using Cloud and VR/PhotonServerSettingsForLocalServer.png) + Click [Assets]-[Reimport All] in the Unity menu. + Click [Reimport] button to rebuild. ** Modify SIGVerse Settings [#y0decaa9] + Click [SIGVerse]-[SIGVerse Settings] to open the SIGVerse settings. + Check [Use Oculus] and [Use PUN] in [Define symbols Settings]. #ref((HSR)Cleanup Task using Cloud and VR/SIGVerseSettingsOculusPUN.png) ** Setup Photon Local server [#PhotonServer] ~* Make sure that anti-virus software does not block communication. //A Photon Server should be set in a local network. + Open [Photon-OnPremise-Server-SDK_v4-0-29-11263\deploy\bin_Win64] folder copied from USB memory stick. + Execute "PhotonControl.exe". (App icon appears on the Windows taskbar) + Click Photon server application in the taskbar of Windows. + Click [Game Server IP Config]-[Set local IP: xxx.xxx.xxx.xxx]. (Select a proper local server IP address) #ref((HSR)Cleanup Task using Cloud and VR/PhotonServerIP.png) + Click [LoadBalancing (MyCloud)]-[Start as application] to start Photon Server. + Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing)] and check that no error occurred. + Close the console window of the test client. Please test the connection to the server + Click [LoadBalancing (MyCloud)]-[Run TestClient (LoadBalancing]. ** Import PUN2 assets [#qb0ad89e] + Open the SIGVerse project on Unity Editor. + Access [[PUN2 page>https://assetstore.unity.com/packages/tools/network/pun-2-free-119922]] of Unity Asset Store. + Sign in to your Unity account from the top right of the page. + Click the [Open in Unity] button. "PUN2 - Free" is displayed on the [Asset Store] tab of Unity. + Uncheck ''"Photon/PhotonLibs/WebSocket/websocket-sharp.dll".'' This dll will conflict with the existing one in the SIGVerse. #ref((HSR)Cleanup Task using Cloud and VR/PUN2ExcludeWebsocket-sharp.png) + Click [Import] button.~ + Click [Skip] button on the [PUN Wizzard] window. #ref((HSR)Cleanup Task using Cloud and VR/PunWizardLocalServer.png) + Click [Close] button. + [PhotonServerSettings] will be opened. Modify the settings.~ (You can open the PhotonServerSettings from [Window]-[Photon Unity Networking]-[Highlight Server Settings].) --- Uncheck [Use Name Server] --- Enter the IP address of Photon Server in [Server] --- Enter "5055" in [Port]. #ref((HSR)Cleanup Task using Cloud and VR/PhotonServerSettingsForLocalServer.png) + Click [Assets]-[Reimport All] in the Unity menu. + Click [Reimport] button to rebuild. * Execution Procedure[#s9ea839c] In the case of using the local server, Photon server should be launched on a PC in advance. //You should [[launch Photon Server>#LaunchPhotonServer]] first. //To execute the virtual avatar, please follow thw [[following section>#human_execution]]. //To execute the robot software, please follow the [[following section>#robot_execution]]. ** Procedure to Login as a Robot (ROS) [#robot_execution] ~* ROS nodes should be launched on the Ubuntu side before the Unity scene is started. *** Ubuntu side startup procedure [#qd09b4ab] + Open a new terminal and execute the following command: $ roslaunch sigverse_hsr_teleop_key teleop_key.launch //To start rviz, execute the following command. // $ roslaunch sigverse_hsr_teleop_key teleop_key_with_rviz.launch *** Windows startup procedure [#t197903c] + Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene. + Click the [Robot] button to log in as a robot. //+ If a human is not logged in, wait for the human to log in. Windows (Login Screen) #ref((HSR)Cleanup Task using Cloud and VR/HSRCleanupVRLogin.png) *** Robot Operation [#v04386d7] You can control HSR by keyboard operation on the teleop_key.launch terminal. Keyboard commands are shown on a terminal. Operate the robot and grasp one of the objects on the desk, then bring it to the place where the human points. ~* When you stop the program, please stop the Unity side fist, then stop ROS nodes. Ubuntu side #ref((HSR)Cleanup Task using Cloud and VR/HSRCleanupVRRobotUbuntu.png) Windows side (Logged in) #ref((HSR)Cleanup Task using Cloud and VR/HSRCleanupVRRobotWindows.png) Windows side (Pointing) #ref((HSR)Cleanup Task using Cloud and VR/HSRCleanupVRRobotWindowsPointing.png) ** Procedure to Login as a Human Avatar (VR) [#human_execution] ~* This step requires a VR ready PC and a VR headset (Oculus Rift CV1 or Oculus Rift S).~ Please see [[requirements for using Oculus Rift>https://support.oculus.com/248749509016567/]] [[Oculus Software>https://www.oculus.com/setup/]] should be installed on Windows and Oculus Rift should be set up beforehand. *** Windows startup procedure [#t197903c] + Make sure that Oculus Rift is connected and ready for use. + Start the [Assets/SIGVerse/SampleScenes/HSR/HsrCleanupVR(.unity)] scene. + Click the [Human] button to log in as a human. //+ If the robot is not logged in, wait for the robot to log in. Windows (Login Screen) #ref((HSR)Cleanup Task using Cloud and VR/HSRCleanupVRLogin.png) *** Instruction to the Robot [#v04386d7] Operate the human avatar and give cleanup instructions to the robot. The basic flow is as follows. + Send the message "Pick it up!" to the robot while pointing at the target object. + Send the message "Clean up!" to the robot while pointing at the destination. The operation method of Oculus Touch is as follows. -Pulling the middle finger trigger puts the avatar's hand in the pointing posture. -When the A or X button is pressed, the message "Pick it up!" is sent to the robot. -When the B or Y button is pressed, the message "Clean up!" is sent to the robot. -When you tilt the left stick, the avatar moves in the tilted direction. -When the right stick is pressed, the message "Good!" is sent to the robot. -When the left stick is pressed, the message "Bad!" is sent to the robot. Windows #ref((HSR)Cleanup Task using Cloud and VR/HSRCleanupVRHuman.png) ---- #counter Top: [[Tutorial at IROS2019]]
Site admin: inamura
PukiWiki 1.5.0 Copyright © 2001-2006 PukiWiki Developers Team. License is GPL.