Java to Python (j2p)

This tool can translate java programs into python programs. This is a complex mechanism and still experimental/in development. This documentation is not complete yet. This code is not yet published on the artifactory.

Some features:

• "core translator" for translating java and the built-in libraries.
• Handles java method overloading using the @dispatch annotation from ​plum.
• Minimalistic code generation: if your java code does not use external libraries, no additional libraries are needed to run generated python code, except maybe the plum library if you use java method overloading.
• both single-file as multi-file projects
• translates calls to external libraries
• "translator plugins": a translator plugin is a plugin on the java side that handles translating calls to external java libraries such as jackson, junit etc. You select the required translator plugins as needed.
• There may be different translator plugins available for the same java library, translating to a different python library, depending on your needs.
• can generate pip-installable zip file that automatically (pip) installs all required dependencies (external python libraries) depending on your chosen library translators.
• PyRunner supports running python zip files from java.

The repo contains the translator in the module named "core". The other modules, ending with "-t", are translator plug-ins described below.

Core Translation

For normal use, you do not need to know exactly how the translation is done. But the way the translator works becomes highly relevant if you want to inject your own python translations into the code (see the following section).

Translation is done on a per-java-file basis.

There are two main translation components in the core:

1. The translator that parses java file with ​Javaparser and creates equivalent python code.
2. per-java-class translators that know how to translate all calls to any java class method or field into equivalent python code. This code is using java-side code introspection to determine argument types, and therefore requires all referred java classes are actually available in compiled form, either through libraries or from the java compiler.

(1) is generic and used for all java programs. It is currently pretty complete but details will be filled in over time as the need arises to support more java syntax.

(2) is currently very partial. The reason is that there are a huge number of java classes and even more 3rd party libraries, and almost every field and function in it will need a specialized translator. This will grow slowly over time as needed.

The mechanism is flexible in its mapping from java classes to python classes. The general approach is to assume a fixed mapping from java classes to python classes, as in the table below. Also it is assumed that the equivalent classes have roughly the same functionality, even if the actual function names and arguments may differ. This fixed mapping simplifies translation.

But the translator can intercept special classes and make dedicated translations. For instance if a class C is implementing Iterator and it is of class C, translating it.next() gives next(it), and the definition void iterator() translates to __iter__(self). These are all handled in the translator classes, in this example in tudelft.utilities.j2p.t.java.util.Iterator.

Comments

The exact location of comments is very important as it determines to which java element the comment is attached to. And where it is attached to determines how it is translated, and which element it modifies if you use #PY annotations to override the translation (see below).

Generally, a comment attaches to the element that comes next to it.

element after comment	placement of comment in translation
@Annotation	before the annotation
method declaration, method modifier	first item in the method
class declaration	before the class

Therefore, we recommend to place method @Annotations before the javadoc, like this

@NonNull @Override
/** @return  a string representation of the object ....*/
public String toString() { ... }

If you place the comment before the @Annotation, the comment will end up before the method, instead of inside it as it custom in python.

This allows you to override just the annotation of an element, like this:

//#PY #annotation not needed in python
@JavaSpecificAnnotation
private final String values...

constructs

The private keyword is reflected by the python convention of prefixing private fields function names and method names with __.

The static keyword results in fields being set in the class definition object.

Fields in Java have to be initialized in the __init__ function in the python translation.

Overloaded methods

Overloaded methods can not be handled by default python. To support it, an external library plum-dispatch==2.2.2 is used. If you use overloading, you need to have plum installed to run your code. Check also the #@NonNull section.

When a method is overloaded, the translator adds @dispatch annotations in the translated code as required by plum.

If you have two methods overloading each other, and both methods end up with the same signature in python, then plum will NOT warn you and just use the last definition only. Therefore your translated code will not work as expected. An example is if your code has both get(BigInteger x) and get(long x), both will be translated to @dispatch def get(self, x:int) and plum will use only the last one defined.

built-in "primitives"

java	python	remarks
String	str
int	int
float,Float,double,Double	float	in Java, float and double overflow at different places. Translation may fail if code depends on overflow behaviour
BigDecimal	Decimal
Map, HashMap	dict
Set	set

Classes

Classes are by default translated using the "Stub" translator which assumes assumes the class names and modules are identical in Python and Java, and that all functions have the same functions with the same arguments and types.

References to classes in the code (SomeClass.class in java) translates to SomeClass in Python. It is assumed the proper imports are done on the python side to enable this.

This default translation can be overridden using custom translators.

Custom translators

The number of custom translated classes is too big to even list here. Many built in java classes are already supported. Please check the source code

If function calls in python deliver a wrong object (not matching the above mapping), the translator has to inject additional code to convert it to the proper object. For example, Map.getKeys() in python would be set(map.keys()) where the extra set converts the dict_keys into a proper set.

Stream

The Stream class is currently not translated. It might be possible to translate it automatically but it looks pretty complex because the python way to deal with streams is list comprehension and the translation is quite far from straightforward.

Inner classes

Inner classes (classes inside classes) have very limited support.

When an inner class translator is needed, eg p.A$B where B is an inner class of A, the translator searches for a translator for class p.a/B where p/a is used as a *directory* and is the lower case version of A. If it is found, then that translator B is used.

Note that that translated class will be in package p.a, rather than inside class p.A. Therefore this mechanism, has limited use in practice.

Otherwise, no support is available. This includes anonymous inner classes. You can not import, call or use inner classes. This is for many reasons but the brief summary:

• The useful variant of inner class is the non-static inner class. This class can access fields in the enclosing class.
• Python does not support accessing fields of the enclosing class unless some tricks are applied. But these tricks would break uniformity of constructors.
• Many deep technical reasons #175 that largely complicate dealing with these.

We recommend now:

• manually override translation when you really need to call an inner class
• use package-private classes to replace your own inner classes.
• For the non-static inner class: pass the enclosing class as argument to the child class constructor. You may have to use more tricks to get around cyclic dependencies.

Workaround

If some particular inner classes, particularly constants, are used in very specific translators, these translators may opt to attempt to recognise these and process them separately (thus avoiding calling the translator for them). One example is the JsonDeserializer (jackson-t package) that recognises use of "com.fasterxml.jackson.databind.JsonDeserializer.None".

Auto Boxing

Autoboxing is the automatic conversion that the Java compiler makes between the primitive types and their corresponding object wrapper classes. For example, converting an int to an Integer, a double to a Double, and so on. If the conversion goes the other way, this is called unboxing.

The translator currently does not support auto boxing. That means you will get an error on the python side if you for instance do this on the java side

int n=1;
return "number "+ n;

You will have to call an explicit converter, typically toString(), on the arguments that need to be converted into the proper argument type.

Another example is when varargs are involved:

public sum(int... values) {
  return sumlist(values);
}

public sumList(int[] values) {
  return ....;
}

The call sumlist(values) uses auto boxing, but here the translation is from int... to int[]. A similar situation is when you do a for loop over varargs, like

public sum(int... values) {
  for (v: values) { .... }
}

All these complications around varargs, including more eg ​around dispatching vararg-typed methods led us to currently not support varargs.

@NonNull

Java variables like String val can contain a null value, and functions like String f() { ... } can return null. Therefore they are translated to val:Optional[str] and def f(self)->Optional[str]. You can annotate the java code with @NonNull (from org.eclipse.jdt.annotation.NonNull) to indicate the value/return value will not be null, like this @NonNull String val or @NonNull String f() { ... } . Java primitive types like boolean and int can never be null and do not need @NonNull

The dependency needed for this is

<dependency>
    <groupId>org.eclipse.jdt</groupId>
    <artifactId>org.eclipse.jdt.annotation</artifactId>
    <version>2.3.0</version>
</dependency>

Unfortunately this annotation is not built in anymore in the JRE. But this dependency is very lightweight (the jar is 11kb) and is completely separate from the Eclipse IDE platform.

Java code X instanceof C is translated as in Java, so null/None is not an instance of C - C is not "Optional" and you do not need to write X instanceof @NonNull C.

Also be aware of the subtleties of this notation. For instance @NonNull Set<String> is a set that can not be null but that contain null values. @NonNull Set<@Nonnull String> is a set that can not be null and also can not contain null values.

@NonNull can be used by the java compiler to do extra checks on your code. Eclipse uses this annotation by default, but you may need to enable or customize it in Preferences/Java/Compiler/Errors/Warnings and scroll down to "Null Analysis".

@NonNull is NOT inherited by subclasses. Therefore @NonNull has to be repeated in the derived classes.

Reserved keywords

Python has a number of reserved keywords:

and except lambda with as finally nonlocal while assert false None yield break for not class from or continue global pass def if raise del import return elif in True else is try

If you have these in your java program, as method name or variable name, then j2p will change the name by (de)capitalizing the first letter of the name. Do not use both the lower and upper case variant at the same time, because the translated names will then collide.

More notes:

• some classes like Arrays or Override do not have a direct python equivalent. calls to static functions in these classes can still be translated.
• Different java classes, eg Map AbstractMap and HashMap, may translate to the same python class (map). This can be done because translation is only 1 way.
• The imported libraries in python have a different name than those in java. If the python library name equals the name of some variable in your code, this will cause a runtime error, typically something like local variable 'json' referenced before assignment. We can not (yet?) detect these issues at compile time. To solve it, change the variable name in java.

Usage

For usage you have the choice between translating a single file or an entire package. We generally recommend translating the entire package.

Single File

If you want to translate a single java file or even just a string containing your program, that does not need any specialized libraries, you can do the approach as many tests in core do. Check the examples. The heart of the code will look like this, where most boilerplate is about setting up the javaparser:

		ParserConfiguration conf = new ParserConfiguration();
		CombinedTypeSolver typeSolver = new CombinedTypeSolver();
		typeSolver.add(new ReflectionTypeSolver(false));
		JavaSymbolSolver symbolSolver = new JavaSymbolSolver(typeSolver);
		conf.setSymbolResolver(symbolSolver);
		JavaParser parser = new JavaParser(conf);
		ParseResult<CompilationUnit> res = parser.parse(new File(javaFile));

		Block translation = Translator.translate(res.getResult().get());

After you get the translation, you can print it or put it in a .py file.

Entire Package

If you want to translate an entire package, first create a separate source directory containing all the java code that you want to translate. Also make sure that your code is compiled (eg, using the build-helper-maven-plugin , check the pom files). Then you just do

PyProgram program = PyProgram.fromDirectories(Arrays.asList(Paths.get("src/test/myprogram")));

This gives you a fully translated program that you can print or get a zip file from. To get a zip file, call program.getZip()

The zip file is ready for a pip install (eg in your own virtual environment) or running from java through our PythonVenv.

Resources

If you use the PyProgram translator, all non-java files are copied verbatim to the python program, to the same directory. Just make sure that if you have a separate resources directory, that you add that aswell to the list of root directories.

THe resource can be accessed like usual in java:

	final InputStream stream = getClass()
				.getResourceAsStream("yourresource.txt");
	...stream.read()...

Overriding the translation

Comments can contain python code to override the automatic code, if the block starts with #PY. This code replaces the entire object (if/case/while block; statement) that follows the comment.

Place the #PY block directly before the object you want to override. For example if you place a #PY block before an annotation, you override only the annotation, not the actual method/class after the annotation. This unfortunately will look a bit non-java but it seems the best way to accurately target the override.

Python has strict requirements regarding indentation. To make this possible, we need to be strict about indentation as well. In a single line python comment, the code must look exactly like

 //#PY single_python_line 

Note the single whitespace after the #PY. Your code starts after this single whitespace.

In a multi line comment the code must look exactly like

/*#PY
 * codeline1
 * codeline2
 * ...
 */

Your code lines each start with "* ", note the whitespace after the star. You are free to indent before the "*".

Your code is automatically indented to the level needed at the insertion place in the code.

Code must be placed in either a standard block comment or a single line comment. Starting a javadoc with #PY is not allowed. This is to encourage proper use of javadoc.

A comment block overrides also annotations.

If the code block contains no code at all, it is translated as pass, to ensure that the code is a proper statement.

Any line containing an import (from ... import ... or import ... ) without leading whitespace is interpreted as a global import. If you place an INDENT before the import, it's not a global import anymore and kept in-line with the code.

You can also annotate import statements with any python code. Note that in the translated python code, these snips will be placed after the imports.

#PY always goes before a java object. If you really have to add #PY after your last line of java code, you will have to add a dummy class at the end, and annotate that.

Auto formatting (eg when using "save actions" in Eclipse) may corrupt the format of #PY blocks. We recommend turning off auto comment formatting.

Translator plugins

J2P has a modular translation mechanism. A translator plugin can be plugged in as needed, to add support for translating external libraries that are used in the java code. Also this allows to customize the translation process, for instance to use another python library for the translation.

A number of translation modules are already available

module	what it translates	details	limitations
jackson-t	jackson serialization annotations	translates jackson to pyson annotations	Covers what we need for translation of ​GeniusWeb
junit-t	junit calls assertTrue,assertEquals		Very limited
tudutils-t	translates calls to utilities package		limited, currently mainly to support parts of immutablelist
mockito-t	mockito translations		very limited

The translators are all in standardized directories. If there is a translator for class a.b.C then this must be in the class tudelft.utilities.j2p.t.a.b.C. This makes it easy to find the translator class, makes everything pluggable through the maven dependency plugins, and requires minimum extra naming conventions.

The translators generally follow the same inheritance hierarchy as the original classes. So if a.b.P is the parent class of a.b.C, then there usually is a glass tudelft.utilities.j2p.t.a.b.P that is the parent translator of tudelft.utilities.j2p.t.a.b.C. The latter forwards shared translation matters to the parent class as much as possible. A simple example can be seen in tudelft.utilities.j2p.t.java.lang.ArithmeticException, which forwards almost everything to the RuntimeException translator which in turn forwards everything to Exception etc.

Translators should all be public classes, even if the real class is abstract. This is because java will handle calls to abstract classes and interfaces and these will be translated with the translator for that abstract class. For example, if we have

List<String> l=...some expression or function call...
l.get(0)

then l is of type List (the Interface) and l.get is a function in the interface which needs to be translated. Note that in general any type of List can end up in l at runtime, the compiler has to compile it such that the compiled code will work regardless (thus using only functions available in python's equivalent of the List class).

Unit Tests, @Test

The junit-t plugin helps translation of org.junit components. In python, a class with tests must extend unittest.TestCase. This allows discovery by the py test tools. In Python the mechanism is slightly different, requiring the test class to extend TestCase. To support the translation mechanism and make our lives a bit easier, j2p requires all tests to be annotates with @RunWith. Use the following, depending on what kind of test you want to run:

test type in java	use java annotation
normal @Test methods	@RunWith(JUnit4ClassRunner.class)
parametric test	@RunWith(Parameterized.class)

FAQs

Question	Explanation
I'm getting "No translator found for X". But X is a class that I'm trying to translate	The java files you are trying to translate are probably not compiled by the java compiler. When the translator finds a method call, it needs the compiled java to determine the proper signature (function name and arguments). You can use the build-helper-maven-plugin to add your additional sources to the standard maven build path.
I get an error that casting primitives is not supported.	Casting of primitives is different from normal casting, it fundamentally changes the data. Instead use Double.valueOf(), Integer.valueOf(), Character.toChars() etc
I get an error UnboundLocalError: local variable ... referenced before assignment (where ... is a variable name)	Usually this happens if the variable has a name that collides with a method name or library in python, eg "list". Rename the variable so that it is not colliding
I get "Python can not handle overloaded methods that use type vars"	This happens if you overloaded a method (you have two or more methods with the same method name) and one of them uses a type var (you have <T> and use T in the method declaration). Python does not support overloading. none of the existing libraries to add this functionality can handle type variables.
It seems the python code uses the wrong values in my lambda expression	There is a subtle but essential scoping difference between java and python. In python, a local variable continues living even if it goes outside of scope and can be changed. If such a change happen, the lambda will use the changed value. Python only keeps the current value if a new variable is created, eg by a method call