[[PageOutline]]

= Java to Python (j2p)

This tool can translate java programs into python programs.
This is a complex mechanism and still experimental/in development. This documentation is not complete yet. This code is not yet published on the artifactory.

Some features:
* "core translator" for translating java and the built-in libraries.
* Handles java method overloading using the @dispatch annotation from [https://pypi.org/project/plum-dispatch/ plum].
* Minimalistic code generation: only those external libraries are needed that are actually needed for the specific translated code. Also we attempt to make minimal use of external libraries.
* both single-file as multi-file projects
* translates calls to external java libraries
* translator plugins: plugins on the java side that handle translating calls to external java libraries such as jackson, junit etc. You select the required translator plugins as needed.  
* There may be different translator plugins available for the same java library, translating to a different python library, depending on your needs.
* can generate pip-installable zip file that automatically (pip) installs all required dependencies (external python libraries) depending on your chosen library translators.
* [https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/wiki/pyrunner PyRunner] supports running python zip files from java.
The repo contains the translator in the module named "core". The other modules, ending with "-t", are translator plug-ins described below.
* Tested compatibility with python 3.8 3.9 3.10 and 3.11.

== Core Translation
For normal use, you do not need to know exactly how the translation is done. But the way the translator works becomes highly relevant if you want to inject your own python translations into the code (see the following section). 

Translation is done on a per-java-file basis. 

There are two main translation components in the core:
1. The translator that parses java file with [https://github.com/javaparser/javaparser Javaparser] and creates equivalent python code.
2. per-java-class translators that know how to translate all calls to any java class method or field into equivalent python code. This code is using java-side code introspection to determine argument types, and therefore requires all referred java classes are actually available in compiled form, either through libraries or from the java compiler.

(1) is generic and used for all java programs. It is currently pretty complete but details will be filled in over time as the need arises to support more java syntax.

(2) is currently very partial. The reason is that there are a huge number of java classes and even more 3rd party libraries, and almost every field and function in it will need a specialized translator. This will grow slowly over time as needed.


The mechanism is flexible in its mapping from java classes to python classes. 
The general approach is to assume a fixed mapping from java classes to python classes, as in the table below. Also it is assumed that the equivalent classes have roughly the same functionality, even if the actual function names and arguments may differ. This fixed mapping simplifies translation.

But the translator can intercept special classes and make dedicated translations. For instance if a class C is implementing {{{Iterator}}} and it is of class C, translating {{{it.next()}}} gives {{{next(it)}}}, and the definition {{{void iterator()}}} translates to {{{__iter__(self)}}}. These are all handled in the translator classes, in this example in {{{tudelft.utilities.j2p.t.java.util.Iterator}}}.

=== Comments
The exact location of comments is very important as it determines to which java element the comment is attached to. And where it is attached to determines how it is translated, and which element it modifies if you use #PY annotations to override the translation (see below). 

Generally, a comment attaches to the element that comes next to it. 

||element after comment||placement of comment in translation||
||@Annotation||before the annotation||
||method declaration, method modifier||first item in the method||
||class declaration||before the class||

Therefore, we recommend to place method @Annotations **before** the javadoc, like this
{{{
@NonNull @Override
/** @return  a string representation of the object ....*/
public String toString() { ... }
}}} 

If you place the comment before the @Annotation, the comment will end up before the method, instead of inside it as it custom in python.

This allows you to override just the annotation of an element, like this:
{{{
//#PY #annotation not needed in python
@JavaSpecificAnnotation
private final String values...
}}}


=== constructs
The {{{private}}} keyword is reflected by the python convention of prefixing private fields function names and method names with {{{__}}}.

The {{{static}}} keyword results in fields being set in the class definition object. 

Fields in Java have to be initialized in the {{{__init__}}} function in the python translation.

=== Overloaded methods
Overloaded methods can not be handled by default python. To support it, an external library {{{plum-dispatch==2.2.2}}} is used. If you use overloading, you need to have plum installed to run your code. Check also the #@NonNull section.

When a method is overloaded, the translator adds {{{@dispatch}}} annotations  in the translated code as required by plum.


{{{#!td style="background: #fee"
If you have two methods overloading each other, and both methods end up with the same signature in python, then plum will NOT warn you and just use the last definition only. Therefore your translated code will not work as expected. An example is if your code has both {{{get(BigInteger x)}}} and {{{get(long x)}}}, both will be translated to {{{@dispatch def get(self, x:int)}}} and plum will use only the last one defined.
}}}



=== built-in "primitives"
||=java=||=python=||=remarks=||
||String||str||
||int||int||
||float,Float,double,Double||float||in Java, float and double overflow at different places. Translation may fail if code depends on overflow behaviour||
||BigDecimal||Decimal||
||Map, HashMap||dict||
||Set||set||

=== Classes
Classes are by default translated using the "Stub" translator which assumes
assumes the class names and modules are identical in Python and Java,
and that all functions have the same functions with the same arguments and types.

References to classes in the code (SomeClass.class in java) translates to SomeClass in Python. It is assumed the proper imports are done on the python side to enable this.

This default translation can be overridden using custom translators. 

==== Custom translators
The number of custom translated classes is too big to even list here. Many built in java classes are already supported. Please check the  [source:java2python/core/src/main/java/tudelft/utilities/j2p/t/java source code]


If function calls in python deliver a wrong object (not matching the above mapping), the translator has to inject additional code to convert it to the proper object. For example, {{{Map.getKeys()}}} in python would be {{{set(map.keys())}}} where the extra {{{set}}} converts the dict_keys into a proper set.

===== Stream
The {{{Stream}}} class is currently not translated. It might be possible to translate it automatically but it looks pretty complex because the python way to deal with streams is list comprehension and the translation is quite far from straightforward.

==== Inner classes
Inner classes (classes inside classes) have very limited support. 

When an inner class translator is needed, eg p.A$B where B is an inner class of A, the translator searches for a translator for class p.a/B where p/a is used as a *directory* and is the lower case version of A. If it is found, then that translator B is used. 

Note that that translated class will be in package p.a, rather than inside class p.A. Therefore this mechanism, has limited use in practice.

Otherwise, no support is available. This includes anonymous inner classes. You can not import, call or use inner classes. This is for many reasons but the brief summary:
* The useful variant of inner class is the non-static inner class. This class can access fields in the enclosing class.
* Python does not support accessing fields of the enclosing class unless some tricks are applied. But these tricks would break uniformity of constructors.
* Many deep technical reasons #175 that largely complicate dealing with these.

We recommend now:
* manually override translation when you really need to call an inner class
* use package-private classes to replace your own inner classes.
* For the non-static inner class: pass the enclosing class as argument to the child class constructor. You may have to use more tricks to get around cyclic dependencies.

===== Workaround
If some particular inner classes, particularly constants, are used in very specific translators, these translators may opt to attempt to recognise these and process them separately (thus avoiding calling the translator for them). One example is the JsonDeserializer (jackson-t package) that recognises use of "com.fasterxml.jackson.databind.JsonDeserializer.None". 

==== Enum
Enum classes are translated to normal Python classes. For instance if you have a class MyEnum and it has items P and Q, we assume on the python side that a class MyEnum is there with static public fields P and Q. As usual the translator initializes P and Q in the {{{_static_init}} method. This approach matches with the traditional way python supported enumerations, for example {{{math.pi}}} directly delivers the value of pi, not some placeholder.

This approach ignores the more recent Python Enum class. This new approach is incompatible with the traditional approach, it gives an extra layer of indirection, it does not allow the custom constructors that we need, it is very complex, and all this complicates our translation job. 

=== Auto Boxing
Autoboxing is the automatic conversion that the Java compiler makes between the primitive types and their corresponding object wrapper classes. For example, converting an int to an Integer, a double to a Double, and so on. If the conversion goes the other way, this is called unboxing.

The translator currently does not support auto boxing. That means you will get an error on the python side if you for instance do this on the java side

{{{
int n=1;
return "number "+ n;
}}}

You will have to call an explicit converter, typically toString(), on the arguments that need to be converted into the proper argument type.


Another example is when varargs are involved:
{{{
public sum(int... values) {
  return sumlist(values);
}

public sumList(int[] values) {
  return ....;
}
}}}

The call sumlist(values) uses auto boxing, but here the translation is from {{{int...}}} to {{{int[]}}}. A similar situation is when you do a for loop over varargs, like 
{{{
public sum(int... values) {
  for (v: values) { .... }
}
}}}

All these complications around varargs, including more eg [https://github.com/mrocklin/multipledispatch/issues/72 around dispatching vararg-typed methods] led us to currently not support varargs.

=== Implicit casting, narrowing
Java does automatic casting to silently convert between bytes, int, long etc. These can also involve narrowing, loss of precision etc. There is partial support for these. The translator recognises when such conversion is done and tries to insert extra code to implement the expected narrowing as needed. This is not yet implemented for conversions between double and float, as the translator translates both to python floats.

=== @NonNull
Java variables like {{{String val}}} can contain a null value, and functions like {{{String f() { ... } }}} can return null. Therefore they are translated to {{{val:Optional[str]}}} and {{{def f(self)->Optional[str]}}}.
You can annotate the java code with {{{@NonNull}}} (from {{{org.eclipse.jdt.annotation.NonNull}}}) to indicate the value/return value will not be null, like this
{{{@NonNull String val}}} or {{{@NonNull String f() { ... } }}}. Java primitive types like {{{boolean}}} and {{{int}}} can never be null and do not need {{{@NonNull}}}

The dependency needed for this is
{{{
<dependency>
    <groupId>org.eclipse.jdt</groupId>
    <artifactId>org.eclipse.jdt.annotation</artifactId>
    <version>2.3.0</version>
</dependency>
}}}

Unfortunately this annotation is not built in anymore in the JRE. But this dependency is very lightweight (the jar is 11kb) and is completely separate from the Eclipse IDE platform.

Java code {{{X instanceof C}}} is translated as in Java, so null/None is not an instance of C - C is not "Optional" and you do not need to write {{{X instanceof @NonNull C}}}.

Also be aware of the subtleties of this notation. For instance
{{{@NonNull Set<String>}}} is a set that can not be null but that contain null values.
{{{@NonNull Set<@Nonnull String>}}} is a set that can not be null and also can not contain null values. 


{{{#!td style="background: #efe"
{{{@NonNull}}} can be used by the java compiler to do extra checks on your code. Eclipse uses this annotation by default, but you may need to enable or customize it in Preferences/Java/Compiler/Errors/Warnings and scroll down to "Null Analysis".
}}}

{{{#!td style="background: #fee"
{{{@NonNull}}} is NOT inherited by subclasses. Therefore {{{@NonNull}}} has to be repeated in the derived classes.
}}}

=== @Defer
{{{@Defer}}} is a qualifier for Type specifications. The type that has this annotation is suggested to be imported later.  The translator will double quote such annotated types and suppress import of that class.

For example, consider the cyclicref testcode in the core:
{{{
public class P {
	private @Defer Q q = null;
	public void join(@Defer Q q) {
		this.q = q;
	}
}
}}}

The class Q that is referred from P refers back to Q. Without the @Defer annotations, class Q would have to be imported, resulting in a cyclic import. With the annotation however, the P class is translated as 
{{{
class P:
 def join(self,q:"Optional[Q]") -> None:
  self.__q=q
 def __init__(self):
  self.__q:"Optional[Q]" = None
}}}

This avoids the cyclic import, at the expense of some loss in python typing. Of course  the type checking is still done at the java side.


=== Reserved keywords
Python has a number of reserved keywords:

and
except
lambda
with
as
finally
nonlocal
while
assert
false
None
yield
break
for
not
class
from
or
continue
global
pass
def
if
raise
del
import
return
elif
in
True
else
is
try

If you have these in your java program, as method name or variable name, then j2p will change the name by (de)capitalizing the first letter of the name. Do not use both the lower and upper case variant at the same time, because the translated names will then collide.


=== More notes:
* some classes like {{{Arrays}}} or {{{Override}}} do not have a direct python equivalent. calls to static functions in these classes can still be translated. 
* Different java classes, eg {{{Map}}} {{{AbstractMap}}} and {{{HashMap}}}, may translate to the same python class ({{{map}}}). This can be done because  translation is only 1 way.
* The imported libraries in python have a different name than those in java. If the python library name equals the name of some variable in your code, this will cause a runtime error, typically something like {{{local variable 'json' referenced before assignment}}}. We can not (yet?) detect these issues at compile time. To solve it, change the variable name in java.

== Usage
For usage you have the choice between translating a single file or an entire package. We generally recommend translating the entire package.

=== Single File
If you want to translate a single java file or even just a string containing your program, that does not need any specialized libraries, you can do the approach as many tests in core do. Check the examples. The heart of the code will look like this, where most boilerplate is about setting up the javaparser:

{{{
		ParserConfiguration conf = new ParserConfiguration();
		CombinedTypeSolver typeSolver = new CombinedTypeSolver();
		typeSolver.add(new ReflectionTypeSolver(false));
		JavaSymbolSolver symbolSolver = new JavaSymbolSolver(typeSolver);
		conf.setSymbolResolver(symbolSolver);
		JavaParser parser = new JavaParser(conf);
		ParseResult<CompilationUnit> res = parser.parse(new File(javaFile));

		Block translation = Translator.translate(res.getResult().get());

}}}

After you get the translation, you can print it or put it in a {{{.py}}} file.

=== Entire Package
If you want to translate an entire package, first create a separate source directory containing all the java code that you want to translate. Also make sure that your code is compiled (eg, using the build-helper-maven-plugin , check the pom files). Then you just do
{{{
PyProgram program = PyProgram.fromDirectories(Arrays.asList(Paths.get("src/test/myprogram")));
}}}

This gives you a fully translated program that you can print or get a zip file from. To get a zip file, call
{{{program.getZip() }}}

The zip file is ready for a pip install (eg in your own virtual environment) or running from java through our PythonVenv.

==== Resources
If you use the PyProgram translator, all non-java files are copied verbatim to the python program, to the same directory. Just make sure that if you have a separate resources directory, that you add that aswell to the list of root directories.

THe resource can be accessed like usual in java:
{{{
	final InputStream stream = getClass()
				.getResourceAsStream("yourresource.txt");
	...stream.read()...
}}}


== Overriding the translation
Comments can contain python code to override the automatic code, if the block starts with {{{#PY}}}. This code replaces the entire object (if/case/while block; statement) that follows the comment.

Place the #PY block directly before the object you want to override. For example if you place a #PY block before an annotation, you override only the annotation, not the actual method/class after the annotation.  This unfortunately will look a bit non-java but it seems the best way to accurately target the  override.

Python has strict requirements regarding indentation. To make this possible, we need to be strict about indentation as well. In a single line python comment, the code must look exactly like

{{{
 //#PY single_python_line 
}}}

  Note the single whitespace after the #PY. Your code starts after this single whitespace.

 In a multi line comment the code must look exactly like
{{{
/*#PY
 * codeline1
 * codeline2
 * ...
 */
}}}

Your code lines each start with "* ", note the whitespace after the star. You are free to indent before the "*".

Your code is automatically indented to the level needed at the insertion place in the code.

Code must be placed in either a standard block comment or a single line
comment. Starting a javadoc with {{{#PY}}} is not
allowed. This is to encourage proper use of javadoc.
 
  A comment block overrides also annotations.

 If the code block contains no code at all, it is translated as
 {{{pass}}}, to ensure that the code is a proper statement.

Any line containing an import ({{{from ... import ...}}} or {{{import ...}}} )
without leading whitespace is interpreted as a global import. If you place an INDENT before the import, it's not a global import anymore and kept in-line        with the code.

You can also annotate import statements with any python code. Note that in the translated python code, these snips will be placed after the imports. 

{{{#!td style="background: #efe"
Avoid using manual translation where possible.
Manual translations are more prone to errors.
They are also not refactored if you refactor
the java code.
}}}

{{{#!td style="background: #eef"
#PY always goes **before** a java object. If you really have to add #PY after your last line of java code, you will have to add a dummy class at the end, and annotate that.
}}}

{{{#!td style="background: #fee"
Auto formatting (eg when using "save actions" in Eclipse) may corrupt the format of #PY blocks. We recommend turning off auto comment formatting.
}}}

== Translator plugins
J2P has a modular translation mechanism. A translator plugin can be plugged in as needed, to add support for translating external libraries that are used in the java code. Also this allows to customize the translation process, for instance to use another python library for the translation.

A number of translation modules are already available

||=module=||=what it translates=||=details=||=limitations=||
||jackson-t||jackson serialization annotations||translates jackson to [https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/wiki/pyson pyson] annotations||Covers what we need for translation of [https://tracinsy.ewi.tudelft.nl/pubtrac/GeniusWeb GeniusWeb]||
||junit-t||junit calls assertTrue,assertEquals|| ||Very limited||
||tudutils-t||translates calls to [https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/wiki/utilities utilities] package|| ||limited, currently mainly to support parts of immutablelist||
||mockito-t||mockito translations|| ||very limited||


The translators are  all in standardized directories. If there is a translator for class {{{a.b.C}}} then this must be in the class {{{tudelft.utilities.j2p.t.a.b.C}}}. This makes it easy to find the translator class, makes everything pluggable through the maven dependency plugins, and requires minimum extra naming conventions. 

The translators generally follow the same inheritance hierarchy as the original classes. So if {{{a.b.P}}} is the parent class of {{{a.b.C}}}, then there usually is a glass {{{tudelft.utilities.j2p.t.a.b.P}}} that is the parent translator of {{{tudelft.utilities.j2p.t.a.b.C}}}. The latter forwards shared translation matters to the parent class as much as possible. A simple example can be seen in {{{tudelft.utilities.j2p.t.java.lang.ArithmeticException}}}, which forwards almost everything to the RuntimeException translator which in turn forwards everything to Exception etc.

Translators should all be public classes, even if the real class is abstract. This is because java will handle calls to abstract classes and interfaces and these will be translated with the translator for that abstract class. For example, if we have
{{{
List<String> l=...some expression or function call...
l.get(0)
}}}

then l is of type List (the Interface) and l.get is a function in the interface which needs to be translated. Note that in general any type of List can end up in l at runtime, the compiler has to compile it such that the compiled code will work regardless (thus using only functions available in python's equivalent of the List class).



=== Unit Tests, @Test
The junit-t plugin helps translation of org.junit components. 

All tests **must** be annotates with {{{@RunWith}}} if this plugin is used. Use the following, depending on what kind of test you want to run:

||= test type in java =||= use java annotation =||
||normal @Test methods||{{{@RunWith(JUnit4ClassRunner.class)}}}||
||parametric test     ||{{{@RunWith(Parameterized.class)}}}||

The junit-t translator depends on,  and uses  [wiki:unitpy]. It also adds some extra code to each test class, so that the standard python test discovery works as usual. 

== GeniusWebTranslator
The GeniusWebTranslator module is at this moment a demo project. It shows that the translator can automatically translate a big part of GeniusWeb. The GeniusWeb code has been modified in a number of ways
* at a few  places, a manual translation has been added. Most often this involves a single line of code, particularly because the translator does not support java stream.
* A lot of @NonNull annotations have been added to make the state of Java variables more clear, also cleaning up the resulting python code.
* various small fixes, to work around a few restrictions of the translator.

== FAQs
||=Question=||=Explanation=||
||I'm getting "No translator found for X". But X is a class that I'm trying to translate||The java files you are trying to translate are probably not compiled by the java compiler. When the translator finds a method call, it needs the compiled java to determine the proper signature (function name and arguments). You can use the build-helper-maven-plugin to add your additional sources to the standard maven build path.|| 
||I get an error {{{UnboundLocalError: local variable ... referenced before assignment}}} (where ... is a variable name)||Usually this happens if the variable has a name that collides with a method name or library in python, eg "list". Rename the variable so that it is not colliding||
||I get "Python can not handle overloaded methods that use type vars"||This happens if you overloaded a method (you have two or more methods with the same method name) and one of them uses a type var (you have <T> and use T in the method declaration). Python does not support overloading. none of the existing libraries to add this functionality can handle type variables.||
||It seems the python code uses the wrong values in my lambda expression||There is a subtle but essential scoping difference between java and python. In python, a local variable continues living even if it goes outside of scope and can be changed. The lambda will use the latest value assigned to the variable. To copy the current value, consider creating the lambda in a dedicated method||
||I get UnsupportedOperationException ... ResolvedWildcard||This happens sometimes in complex method calls with complex parameters. Usually this happens in a stream() context. We suggest using manual translation in such case||
||I get {{{Inner classes can not be translated}}}||Check the section on [wiki:j2p#Innerclasses inner classes]||
||I get something like {{{from com.X.Y.Z import Z: ModuleNotFoundError: No module named 'com'}}}. {{{com.X.Y.Z}}} is a java package that does not exist in python. ||You did not provide  a library or code translate {{{com.X.Y.Z}}} and the translator now uses a "stub" translator that assumes the package on the python side is identical. To fix, you have to add a translator for the {{{com.X.Y.Z}}} package. Or avoid using the package in java||
||I get POSTFIX_INCREMENT: not supported and the same for POSTFIX_DECREMENT.  Why is this not supported?||The general prefix- and postfix-increments are tricky, because they postpone the actual change of the variable. In python multiple statements would have to be used. However this can also occur in situations where python does not allow such multiple statements. We recommend to manually unroll and use extra statements +=1 and -=1 before or after the statement. The general case may require more advanced refactoring.||
||Everything compiles fine but I get {{{ImportError: cannot import name ... from partially initialized module}}} when running in python ||Typically this happened if your java code has a cyclic reference: class A refers to B and class B refers to A. Python can not handle such cyclic references. We suggest using the {{{@Defer}}} annotation to help the translator, or to refactor the code.||