Import and use of user defined python modules

The aim of this tutorial is to show:

how to import a system-define and user-define python modules to be used inside Sigverse functions written in C++
how to read the values returned by Python modules (e.g. double., list etc.) into C++ data types like double, arrays etc/

We saw in the previous tutorial that the "exec" function runs the arbitrary code in the string parameter within the specified namespace. We showed the following examples for that, in which we showed the use of normal, non-imported code.

Of course, we would like to import modules (both user defined and pre-designed system modules) and extract values for them.

Here, I will show one example of standard module "random". The following example is taken from the blog by Joseph Turner

Here we imported the random module by executing the corresponding Python statement within the __main__ namespace, bringing the module into the namespace. After the module is available, we can use functions, objects, and variables within the namespace. In this example, we use the "eval" function, which returns the result of the passed-in Python statement, to create a boost::python object containing a random value as returned by the random() function in the random module. Finally, we extract the value as a C++ double type and print it.

As suggested in the blog, there is a more object oriented way to do this:

In the above example, random module is imported, but this time using the boost::python import function, which loads the module into a boost Python object. Next, the random function object is extracted from the random module and stored in a boost::python object. The function is called, returning a Python object containing the random number. Finally, the double value is extracted and printed. In general, all Python objects can be handled in this way – functions, classes, built-in types.

Now, please copy the following code and name it as "ControllerSample.cpp".

The code as following functionalities

it executes normal, non-imported code
imports a system defined module "random", call its attribute functions and reads the returned values into C++ data type
shows an object-oriented way to import a module into Sigverse function, call its attribute functions reads the returned values into C++ data type
using a user-defined module called sample.py which returns a list and list into a C++ array data type
use of error parsing

#include "ControllerEvent.h"
#include "Controller.h"
#include "Logger.h"
#include <iostream>
#include <boost/python.hpp>
#include <Python.h>
#include <dlfcn.h>

namespace py = boost::python; // create namespace variable for boost::python
using namespace std;

std::string parse_python_exception(); // functional declaration for exception handling

template <typename T> string tostr(const T& t) { ostringstream os; os<<t; return os.str(); } // template to convert double variables to string

class MyController : public Controller {  
public:  
  void onInit(InitEvent &evt);  
  double onAction(ActionEvent&);  
  void onRecvMsg(RecvMsgEvent &evt); 
  void onCollision(CollisionEvent &evt); 
};  
  
void MyController::onInit(InitEvent &evt) {

SimObj *my = getObj(myname());

dlopen("libpython2.7.so", RTLD_LAZY | RTLD_GLOBAL);

Py_Initialize();  //initialization of the python interpreter

}

double MyController::onAction(ActionEvent &evt) {

double x=2.0;

Py_Initialize();
    try{

py::object main_module = py::import("__main__");
        // load the dictionary object out of the main module
        py::object main_namespace = main_module.attr("__dict__");
        // run simple code within the main namespace using the boost::python::exec

py::exec("print 'Executing normal, non-imported code'", main_namespace);

//  function
        py::exec("print 'Hello, world'", main_namespace);
        // any valid Python will execute
        py::exec("print 'Hello, world'[3:5]", main_namespace);
        // it is as if you opened a new Python file and wrote the commands
        // all of the normal functionality is available
        py::exec("print '.'.join(['1','2','3'])", main_namespace);

py::exec("print 'Importing a module that prints random numbers'", main_namespace);

// of course, you can also import functionality from the standard library
        py::exec("import random", main_namespace);
        // boost::python::eval will return the result of an expression 
        //  as a boost::python::object
        py::object rand = py::eval("random.random()", main_namespace);
        // the boost::python::extract function can then convert to C++ data types
        //  (only as appropriate, of course)
        std::cout << py::extract<double>(rand) << std::endl;

//  or, if you'd prefer, you can extract the functions themselves into objects
        //  then call them as you wish from C++. this is akin to what we did at the top
        //  of this file. you can't execute code with the exec function, though, without
        //  extracting the namespace object as we did with __main__ above.
        //  this method ultimately provides a much cleaner way of interacting with Python
        //  objects over their lifetimes

py::exec("print 'A more object oriented way to  import a module that prints random numbers'", main_namespace);

// import the random module
        py::object rand_mod = py::import("random");
        // extract the random function from the random module (random.random())
        py::object rand_func = rand_mod.attr("random");
        // call the function and extract the result
        //  [sidenote: this sort of syntax is what makes boost::python so valuable]
        py::object rand2 = rand_func();
        std::cout << py::extract<double>(rand2) << std::endl;

// Here is an example, of reading a list to C++ data types.

py::exec("print 'Converting a list to C++ data types'", main_namespace);

py::object randList = py::eval("random.sample(range(30), 3) ", main_namespace);

// call the function and extract the result
        // the boost::python::extract function can then read to C++ data types
        //  (only as appropriate, of course)
        int n = len((randList));
        for(unsigned int i=0; i<n; i++){
            std::cout << py::extract<double>((randList)[i])<< std::endl;
        }

// Here is an example, of importing a user define module called sample.py which returns a list
        // The list is extracted into a C++ Data type.

py::exec("print ' User defined python module that returns list to C++ data types'", main_namespace);

py::object sample_module =    py::import("sample");
       py::object sample_func   =    sample_module.attr("simpleList");
       py::object func_list     =    sample_func();

int x = len((func_list));

for(unsigned int i=0; i<x; i++){
            std::cout << py::extract<double>((func_list)[i])<< std::endl;
        }

}catch(boost::python::error_already_set const &){
        // Parse and output the exception
        std::string perror_str = parse_python_exception();
        std::cout << "Error in Python: " << perror_str << std::endl;
    }

return 1.0;      
}  
  
void MyController::onRecvMsg(RecvMsgEvent &evt) {  
}

void MyController::onCollision(CollisionEvent &evt) { 
}
  
extern "C" Controller * createController() {  
  return new MyController;  
}

std::string parse_python_exception(){
    PyObject *type_ptr = NULL, *value_ptr = NULL, *traceback_ptr = NULL;
    // Fetch the exception info from the Python C API
    PyErr_Fetch(&type_ptr, &value_ptr, &traceback_ptr);

// Fallback error
    std::string ret("Unfetchable Python error");
    // If the fetch got a type pointer, parse the type into the exception string
    if(type_ptr != NULL){
        py::handle<> h_type(type_ptr);
        py::str type_pstr(h_type);
        // Extract the string from the boost::python object
        py::extract<std::string> e_type_pstr(type_pstr);
        // If a valid string extraction is available, use it 
        //  otherwise use fallback
        if(e_type_pstr.check())
            ret = e_type_pstr();
        else
            ret = "Unknown exception type";
    }
    // Do the same for the exception value (the stringification of the exception)
    if(value_ptr != NULL){
        py::handle<> h_val(value_ptr);
        py::str a(h_val);
        py::extract<std::string> returned(a);
        if(returned.check())
            ret +=  ": " + returned();
        else
            ret += std::string(": Unparseable Python error: ");
    }
    // Parse lines from the traceback using the Python traceback module
    if(traceback_ptr != NULL){
        py::handle<> h_tb(traceback_ptr);
        // Load the traceback module and the format_tb function
        py::object tb(py::import("traceback"));
        py::object fmt_tb(tb.attr("format_tb"));
        // Call format_tb to get a list of traceback strings
        py::object tb_list(fmt_tb(h_tb));
        // Join the traceback strings into a single string
        py::object tb_str(py::str("\n").join(tb_list));
        // Extract the string, check the extraction, and fallback in necessary
        py::extract<std::string> returned(tb_str);
        if(returned.check())
            ret += ": " + returned();
        else
            ret += std::string(": Unparseable Python traceback");
    }
    return ret;
}

Please set the following environment variables to set the path to boost library.

         export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/pool/lib

Now, execute "make" it will create a ControllerSample.so file.

        make

Please copy the following xml file and name it as "WorldSample.xml"

Please copy the following python file and name it as "sample.py". It imports numpy and returns a list.

Please copy the following Makefile file and name it as "Makefile"

#sigverse header
SIG_SRC  = $(SIGVERSE_PATH)/include/sigverse
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/raghav/pool/lib

CC          = gcc
CPP     = g++
AS          = as
LD          = g++
AR          = ar
RANLIB     = ranlib
OBJCOPY     = objcopy

# external libraries and headers

# change the next line to point to the location of your boost installation
EXTERN_DIR = /home/raghav/pool
EXTERN_LIBDIR = $(EXTERN_DIR)/lib
EXTERN_INCDIR = $(EXTERN_DIR)/include
EXTERN_BINDIR = $(EXTERN_DIR)/bin

BOOST_PYTHON_LIB = $(EXTERN_LIBDIR)/libboost_python.a

INCDIRS = . 
INCDIRS += $(EXTERN_INCDIR)
# you may also need to change this directory if you want to pin to a specific
#   python version
INCDIRS += /usr/include/python2.7
INCDIRS += /usr/lib/python2.7/dist-packages/numpy/core/include

INCDIR = $(foreach dir, $(INCDIRS), -I$(dir))

%.o: %.cpp
     $(CPP) $(INCDIR) $(CFLAGS) $(CPPFLAGS) -c $< -o $@

#specifying of object file
OBJS =  ControllerSample.so

ControllerSample.so: ControllerSample.cpp
     g++ -DCONTROLLER -DNDEBUG -DUSE_ODE -DdDOUBLE  -I$(SIG_SRC) -I$(SIG_SRC)/comm/controller -I$(INCDIR) -fPIC -shared -o $@ $^ $(LDFLAGS) -L$(EXTERN_LIBDIR) -lboost_python -lpython2.7

#$^ , $+     The names of all the prerequisites, with spaces between them. The value of $^ omits duplicate prerequisites, 
# while $+ retains them and preserves their order

clean:
     rm -rf *.so *.o $(HWOBJS) hello_world

Also set the PYTHONPATH

        export PYTHONPATH=$PYTHONPATH:/path_to_your_workspace

Now, please load the world file into Sigverse and push "Start" Button.

       ./sigverse.sh -w ./WorldSample.xml -p your_port_number

Counter: 6370, today: 2, yesterday: 0

Import and use of user defined python modules

recent(20)