PyRunner

PyRunner is a tool to run Python code from Java. The PyRunner can take a ready-to-use python pip-installable tar.gz file. Such file is usually generated using the setuptool in python. This file contains references to all dependencies, those need to be downloaded from the web during installation of the file. The PyRunner creates a virtual environment, and installs your tar.gz file and all dependencies in the virtual environment.

At this moment PyRunner is compatible with python 3.8, 3.9 3.10 and 3.11.

Network

PyRunner in many cases is dependent on the network, because most systems will use pip to install 3rd party libraries, even for fundamental things like wheel for installation support or plum for implementing method overloading. Python (pip) may cache some dependencies but cached data may expire any time.

Preparing Python

This is a tool to run python code from Java. The tool requires FULL python to be installed, including pip and venv.

Some python installations require manual post-installation of these. To do this you need something like

sudo apt-get update
sudo apt install python3.<VERSION>-venv
sudo apt install python3-pip
sudo apt install python3-virtualenv

where <VERSION> is 8,9 etc depending on your python version. Sometimes this installation fails. Using a different system seems to fix that. For example, you may want to use Debian 12.6 instead of 12.7.

To check whether your python is ready for use, run this command

python -m venv venv

If you get any errors, check the error and do further installations. Unfortunately this seems the python way of getting a working installation.

PYTHON3EXE

After installing python, you have to set the PYTHON3EXE environment variable to point to the python3 executable. On ubuntu, you can find the location with which python3. How you set this varies depending on how you want to use the PythonVenv (from Eclipse, from command line, double clicking etc; and which OS).

If you want to use a venv, PYTHON3EXE usually points to YOURVENV/bin/python3. That version of python automatically uses the libraries that were pip-installed there.

Usage

The PythonVenv usage is roughly like this:

• create a PythonVenv E, possibly already with the targz code you want to run. You can use the usual python3 setup.py sdist, or another mechanism such as the java2python compiler.
  • Pass your targz directly into the PythonVenv constructor
  • or if you dont have a targz, copy the code into E.
• call the code you want to run eg from PythonVenvTest​:

   PythonVenv env = new PythonVenv(
  	new File("src/test/resources/helloworld-1.0.0.tar.gz"), logger);
   String res = env.call(Arrays.asList("-c","import helloworld; helloworld.helloworld()"), 40000);
  

• call E.remove to remove the venv

The PythonVenv will run completely in its own venv. The zip file is installed with pip and thus respects all the setup including pip installing dependencies (which may be downloaded from the web).

results

To determine the result(s), stdin and stderr are collected and checked.

• If the deadline is exceeded, an InterruptedException is thrown
• If there is content in stderr, the result is a PythonError containing the stderr message.
• otherwise, the call was a success and the value in stdin is returned. This will normally contain all the text that was printed to stdout in python. Your program can decide then what to do with that.

Run all tests programatically

PythonVenv also has support to run unit tests with discovery.

For discovery, test classes must have a filename ending on Test.py

If you use java2python, you can keep the src and test java code separate and use something like

PyProgram tests = PyProgram.fromDirectories(Arrays.asList(src, test));

The src and test dir can have overlapping modules as usual in java (eg, you have geniuswebsrc/issuevalue/Bid.java and a matching geniuswebtest/issuevalue/BidTest.java. This results in the tests in the same directory in the compiled code, while being separate in the sources. Having it all in a single zip file allows for all dependencies, both for the code and for the testing, to be installed in one go.

Then you run the test using

PythonVenv py = new PythonVenv(tests.getZip(), logger);
TestResult res = py.test("geniusweb", 60000);

When using a zipped python resource you can not use python unittest directly (using python -m unittest discover) because python's pip install procedure will remove all the __init__.py files that are essential for discover.

results

When a test is run, the stdin and stderr are used to collect the test results. Therefore any message to stderr will be interpreted as a test failure. When the python program finishes, it wraps the collected python TestResult in a json-stringified object which is then sent to stdout, which is then collected on the java side and converted into a java Testresult.

Your tests should NOT write to stdout, as this will mess up the TestResult

Running/Debugging unitpy tests

If the unittests are using unitpy, the test class does not need to extend TestCase and is by itself not discoverable. If the test class was compiled with the j2p and the junit-t translator, then a bit of glue code has been added to your module. This glue code is an additional class like this:

class Test(TestCase):
 def test(self):
  runTests(YOURTESTCLASS)

where YOURTESTCLASS is the name of the class containing your unityp tests.

If this glue code is there, or if you can add this glue code manually, you can use the normal procedure with python -m unittest path.to.YOURTESTCLASS. This only works for single classes because pip install removes all __init__.py files, breaking auto discovery of the tests.

Either with or without glue code, you can use runTests directly, using python -c "from unitpy.Runner import runTests; from path.to.YOURTESTCLASS import YOURTESTCLASS; runTests(YOURTESTCLASS)" .

For debug, activate your venv and use

python -c "import pdb; from unitpy.Runner import runTests; from path.to.YOURTESTCLASS import YOURTESTCLASS;  pdb.set_trace(); runTests(YOURTESTCLASS)" -m pdb

We recommend then to set a breakpoint using b /ABSOLUTE/PATH/TO/YOURMODULE.py:LINENR instead of debugging the entire runTest code. And then use 'c' to continue to the breakpoint.

Theoretically, for debugging you would need to install pytest (sudo apt install python3-pytest) after which you can do pytest-3 --trace path/to/YOURTESTCLASS.py. Unfortunately there seems a bug in pytest-3 and it does not break at the first test. This makes this approach mostly useless.