time javac Main.java --> 0m1.050s time javac Main.java & javac Main.java --> 0m1.808s time javac Main.java & javac Main.java & javac Main.java --> 0m2.690s time javac Main.java & ... 8 time --> 0m8.309s
When we run
javac command in parallel and with each increase in
~1 sec gets added for all the
javac command to complete.
Why is there a linear growth is time ?
javac process while running involved in some kind on
locks, if yes how to overcome it so as not to have a linear growth in time
PS: I have tried above on
single core machine,
double core machine,
4 core machine all showed same behaviour.
The java compiler already handles dividing its work across available processors, even when only compiling a single file. Therefore running separate compiler instances in parallel yourself won’t yield the performance gains you are expecting.
To demonstrate this, I generated a large (1 million lines, 10,000 methods) java program in a single file called
Main1.java. Then made additional copies as
Main8.java. Compile times are as follows:
Single file compile:
time javac Main1.java & --> (real) 11.6 sec
Watching this single file compile in
top revealed processor usage mostly in the 200-400% range (indicating multiple CPU usage, 100% per CPU), with occasional spikes in the 700% range (the max on this machine is 800% since there are 8 processors).
Next, two files simultaneously:
time javac Main1.java & --> (real) 14.5 sec time javac Main2.java & --> (real) 14.8 sec
So it only took 14.8 seconds to compile two, when it took 11.6 seconds to compile one. That’s definitely non-linear. It was clear by looking at
top while these were running that again each java compiler was only taking advantage of at most four CPUs at once (with occasional spikes higher). Because of this, the two compilers ran across eight CPUs mostly in parallel with each other.
Next, four files simultaneously:
time javac Main1.java & --> (real) 24.2 sec time javac Main2.java & --> (real) 24.6 sec time javac Main3.java & --> (real) 25.0 sec time javac Main4.java & --> (real) 25.0 sec
Okay, here we’ve hit the wall. We can no longer out-parallelize the compiler. Four files took 25 seconds when two took 14.8. There’s a little optimization there but it’s mostly a linear time increase.
Finally, eight simultaneously:
time javac Main1.java & --> (real) 51.9 sec time javac Main2.java & --> (real) 52.3 sec time javac Main3.java & --> (real) 52.5 sec time javac Main4.java & --> (real) 53.0 sec time javac Main5.java & --> (real) 53.4 sec time javac Main6.java & --> (real) 53.5 sec time javac Main7.java & --> (real) 53.6 sec time javac Main8.java & --> (real) 54.6 sec
This was actually a little worse than linear, as eight took 54.6 seconds while four only took 25.0.
So I think the takeaway from all this is to have faith that the compiler will do a decent job trying to optimize the work you give it across the available CPU resources, and that trying to add additional parallelization by hand will have limited (if any) benefit.
For reference, there are two entries I found in Oracle’s bug database regarding enhancing javac to take advantage of multiple processors:
- Bug ID: JDK-6629150 — The original complaint, this was eventually marked as a duplicate of:
- Bug ID: JDK-6713663 — Suggests the resolution, and based on the “Resolved Date” it appears that multi-processor support in javac was added on 2008-06-12.