Up:[[Tutorial]] Previous:[[Eye movement]] Next: [[HMD+KINECT]] #contents * Control of an agent by Kinect [#s03fafab] ※This tutorial is valid for SIGVerse 2.1.0 or later. ※This tutorial requires OpenNI in your client computer. Sample program NiUserTracker is also required to be executable. This tutorial introduces a method to control a human avatar using Kinect sensor. ** Version of OpenNI [#qd9610ff] - OpenNI Stable Build for Windows x86 (32-bit) v1.5.2.23 Development Edition -- [[You can download from here>http://75.98.78.94/Downloads/OpenNIModules.aspx]] - PrimeSene NITE Stable Build for Windows x86 (32-bit) v1.5.2.21 Development -- Download to select OpenNI Compliant Middleware Binaries on the same page. - PrimeSense Sensor Module for OpenNI Version 5.1.0.25 -- [[You can download from here>https://github.com/avin2/SensorKinect]] ** Set up at client side [#hc45c4cb] A plug-in module for the SIGViewer receives motion data from Kinect, calculates quaternion of each joint, and send the data to avatar controller on the server. *** Download [#e4ddcefe] Please download SIGNiUserTracker_<version>.zip from [[KINECT Service]] *** Installation [#u458405b] Extract the zip file and find SIGNiUserTracker.sig file. Put the SIGNiUserTracker.sig in a folder which has the OpenNI sample program. The defold folder name is: C:\Program Files (x86)\OpenNI\Samples\Bin\Release (※In the case of Windows7 64bit) Connect your Kinect to the client PC. ** Set up at server side [#vbb0fd2f] A controller receives joint angle data; controls the avatar's joints. *** Controller [#vc7899c9] Create a controller file kinectController.cpp as follows: #highlight(cpp){{ #include <string> #include "Controller.h" #include "Logger.h" #include "ControllerEvent.h" #define PI 3.141592 #define DEG2RAD(DEG) ( (PI) * (DEG) / 180.0 ) using namespace std; class AgentController : public Controller { public: double onAction(ActionEvent &evt); void onRecvMsg(RecvMsgEvent &evt); void onInit(InitEvent &evt); private: double m_posx, m_posy, m_posz; double m_yrot; double m_range; int m_maxsize; // maximum number of joints BaseService *m_kinect; }; void AgentController::onInit(InitEvent &evt) { m_kinect = NULL; SimObj *my = getObj(myname()); // Get initial position Vector3d pos; my->getPosition(pos); m_posx = pos.x(); m_posy = pos.y(); m_posz = pos.z(); // Get initial rotation Rotation rot; my->getRotation(rot); double qw = rot.qw(); double qy = rot.qy(); m_yrot = acos(fabs(qw))*2; if(qw*qy > 0) m_yrot = -1*m_yrot; m_range = 0.1; m_maxsize = 15; } double AgentController::onAction(ActionEvent &evt) { // Frequent check whether service is available or not bool available = checkService("SIGKINECT"); // If service is available if(available && m_kinect == NULL){ // Connect to the service m_kinect = connectToService("SIGKINECT"); } // If service is not working else if (!available && m_kinect != NULL){ m_kinect = NULL; } return 1.0; } void AgentController::onRecvMsg(RecvMsgEvent &evt) { std::string sender = evt.getSender(); SimObj *my = getObj(myname()); // Receiving message char *all_msg = (char*)evt.getMsg(); char *msg = strtok(all_msg," "); if(strcmp(msg,"KINECT_DATA") == 0) { int i = 0; while(true) { i++; if(i == m_maxsize+1) break; char *type = strtok(NULL,":"); // Position of the avatar if(strcmp(type,"POSITION") == 0) { double x = atof(strtok(NULL,",")); double y = atof(strtok(NULL,",")); double z = atof(strtok(NULL," ")); // Coordinate transform from Kinect coordinate to SIGVerse coordinate double gx = cos(m_yrot)*x - sin(m_yrot)*z; double gz = sin(m_yrot)*x + cos(m_yrot)*z; my->setPosition(m_posx+gx,m_posy+y,m_posz+gz); continue; } // Orientation of the whole body else if(strcmp(type,"WAIST") == 0) { double w = atof(strtok(NULL,",")); double x = atof(strtok(NULL,",")); double y = atof(strtok(NULL,",")); double z = atof(strtok(NULL," ")); my->setJointQuaternion("ROOT_JOINT0",w,x,y,z); continue; } else if(strcmp(type,"END") == 0) break; // Rotation of joints else { double w = atof(strtok(NULL,",")); double x = atof(strtok(NULL,",")); double y = atof(strtok(NULL,",")); double z = atof(strtok(NULL," ")); double angle = acos(w)*2; double tmp = sin(angle/2); double vx = x/tmp; double vy = y/tmp; double vz = z/tmp; double len = sqrt(vx*vx+vy*vy+vz*vz); if(len < (1 - m_range) || (1 + m_range) < len) continue; my->setJointQuaternion(type,w,x,y,z); continue; } } } } extern "C" Controller * createController () { return new AgentController; } }} This controller receives messages from Kinect; rotates each joint of an avatar. Compile with the following command. $ ./sigmake.sh kinectController.cpp *** World file [#kcdef075] Create world file kinectWorld.xml as follows: #highlight(xml){{ <?xml version="1.0" encoding="utf8"?> <world name="myworld2"> <!--Gravity--> <gravity x="0.0" y="-980.7" z="0.0"/> <!--Initialization of Avatar, Man-nii--> <instanciate class="Man-nii.xml"> <!--Name of the avatar--> <set-attr-value name="name" value="kinect_man"/> <!--Set C++ as controller language--> <set-attr-value name="language" value="c++"/> <!--Set kinectController.so as a controller--> <set-attr-value name="implementation" value="./kinectController.so"/> <!--Do not use dynamics simulation--> <set-attr-value name="dynamics" value="false"/> <!--Position of the avatar(x,y,z)--> <set-attr-value name="x" value="0.0"/> <set-attr-value name="y" value="60.0"/> <set-attr-value name="z" value="0.0"/> </instanciate> </world> }} Just an avator will be appeared in this world. This sample controls only one human avatar by the Kinect. ** Execution [#s58961b9] *** SIGVerse server [#fe0a59dc] Start the SIGVerse server $ sigserver.sh -w ./kinectWorld.xml Next, run the SIGViewer and connect to the sigserver. You can see an avatar is standing. *** Registration of user interface plug-in [#i3bdc209] Click the [Service]-->[Add] menu on the top-left of the SIGViewer window bar. #ref(SIGNi_1.PNG) You then can see the following window. Click the Add button to registrate SIGNiUserTracker.sig #ref(./SIGNi_2.PNG,80%) After the registration of SIGNiUserTracker.sig to the Service list, click the OK button to close the window. Next, click the [Service]-->[Start] menu; then select the SIGNiUserTracker.sig. If the Kinect's depth map window is appeared and you see the following message on the server, it means the application is successfully started. [SYS] Service: "SIGKINECT" is available Click the "START" button to start the simulation. You can control the human avatar with the Kinect. #ref(./SIGNi_3.PNG,40%) ** Compilation of source code of user interface plug-in [#n9886846] In this tutorial Visual C++ 2008 is used to compile the user interface plug-in. *** Download of the source code of user interface plug-in [#g1622ce0] Please download the source code from [[KINECT Service]] //#ref(SIGNiUserTracker_1-0-0.zip) Move the SIGNiUserTracker folder, which is contained in the ZIP file, to a folder which has sample codes of OpenNI. C:\Program Files (x86)\OpenNI\Samples *** Compile [#od39f64a] Next, double-click the solution file in the SIGNiUserTracker folder with administrative right. ※You have to install boost library in the case of the version number is 1.0.0. //VC++が起動したらソリューションエクスプローラの中のソリューション名を右クリックしてビルドを選択します。ビルドが成功すると以下に実行ファイルSIGNiUserTracker.exeが作成されます。 Right-click the solution name in the solution explorer of VC++ to build the source code. If the build succeeds, SIGNiUserTracker.exe will be created at C:\Program Files (x86)\OpenNI\Samples\Bin\Release You can registrate the plug-in module to SIGViewer after changing the suffix from .exe to .sig. Up:[[Tutorial]] Previous:[[Eye movement]] Next: [[HMD+KINECT]] #highlight(end)