PyRunner

PyRunner is a tool to run Python code from Java. The PyRunner takes a ready-to-use python tar.gz file that was generated using the setuptool in python. This file contains all dependencies. The PyRunner then creates a virtual environment, and installs your tar.gz file and all dependencies in the virtual environment.

At this moment PyRunner is compatible with python 3.8, 3.9 and 3.10.

Network

PyRunner in many cases is dependent on the network, because most systems will use pip to install 3rd party libraries, even for fundamental things like wheel for installation support or multipledispatch for implementing multiple same-name methods with different arity. Python does not have a caching mechanism nor a build-with-dependencies.

Preparing Python

This is a tool to run python code from Java. The tool requires FULL python to be installed, including pip and venv.

Some python installations require manual post-installation of these. To do this you need something like

sudo apt-get update
sudo apt install python3-pip
sudo apt install python3.<VERSION>-venv

where <VERSION> is 8,9 etc depending on your python version.

To check whether your python is ready for use, run this command

python -m venv venv

If you get any errors, check the error and do further installations. Unfortunately this seems the python way of getting a working installation.

PYTHON3EXE

After installing python, you have to set the PYTHON3EXE environment variable to point to the python3 executable. On ubuntu, you can find the location with which python3. How you set this varies depending on how you want to use the PythonVenv (from Eclipse, from command line, double clicking etc; and which OS).

Usage

The PythonVenv usage is roughly like this:

• create a PythonVenv E, possibly already with the targz code you want to run. You can use the usual python3 setup.py sdist, or another mechanism such as the java2python compiler.
  • Pass your targz directly into the PythonVenv constructor
  • or if you dont have a targz, copy the code into E.
• call the code you want to run eg from PythonVenvTest​:

   PythonVenv env = new PythonVenv(
  	new File("src/test/resources/helloworld-1.0.0.tar.gz"), logger);
   String res = env.call(Arrays.asList("-c","import helloworld; helloworld.helloworld()"), 40000);
  

• call E.remove to remove the venv

The PythonVenv will run completely in its own venv. The zip file is installed with pip and thus respects all the setup including pip installing dependencies (which may be downloaded from the web).

results

To determine the result(s), stdin and stderr are collected and checked.

• If the deadline is exceeded, an InterruptedException is thrown
• If there is content in stderr, the result is a PythonError containing the stderr message.
• otherwise, the call was a success and the value in stdin is returned. This will normally contain all the text that was printed to stdout in python. Your program can decide then what to do with that.

Running python unit tests

PythonVenv also has support to run unit tests with discovery. Your java code is assumed to have ...Test.java files that are compiled into your zip file, If you use java2python, you can keep the src and test java code separate and use something like

PyProgram tests = PyProgram.fromDirectories(Arrays.asList(src, test));

The src and test dir can have overlapping modules as usual in java (eg, you have geniuswebsrc/issuevalue/Bid.java and a matching geniuswebtest/issuevalue/BidTest.java. This results in the tests in the same directory in the compiled code, while being separate in the sources. Having it all in a single zip file allows for all dependencies, both for the code and for the testing, to be installed in one go.

Then you run the test using

PythonVenv py = new PythonVenv(tests.getZip(), logger);
TestResult res = py.test("geniusweb", 60000);

When using a zipped python resource you can not use python unittest directly (using python -m unittest discover) because python's pip install procedure will remove all the __init__.py files that are essential for discover.

results

When a test is run, the stdin and stderr are used to collect the test results. Therefore any message to stderr will be interpreted as a test failure. When the python program finishes, it wraps the collected python TestResult in a json-stringified object which is then sent to stdout, which is then collected on the java side and converted into a java Testresult.

Your tests should NOT write to stdout, as this will mess up the TestResult