ImgLib benchmarks

Last update: 2012 February 21

This page compares the time performance of image processing operations using raw byte arrays, ImageJ 1.x, ImgLib2 (used in ImageJ 2.x), and Dimiter Prodanov's PixLib library. The benchmark tests these various methods for a "cheap" per-pixel operation (inverting an 8-bit image) as well as an "expensive" operation (some calls to java.util.Math) at several image resolutions.

Some of the charts plot results at several iterations, meaning the test was performed repeatedly in a loop. This is important because the just-in-time compiler (JIT) is able to optimize performance increasingly well as the same code is executed more than once. Hence, we show results after both a single iteration, as well as ten iterations.

Dataset: Cheap operation on 1Mpx image (1000 x 1000), by iteration Expensive operation on 1Mpx image (1000 x 1000), by iteration Cheap operation on 25Mpx image (5000 x 5000), by iteration Expensive operation on 25Mpx image (5000 x 5000), by iteration Cheap operation on various image resolutions, 1st iteration (fresh JVM) Expensive operation on various image resolutions, 1st iteration (fresh JVM) Cheap operation on various image resolutions, 10th iteration Expensive operation on various image resolutions, 10th iteration

System configuration

The data above was collected with an ImgLib2 snapshot from 2012-Feb-21, and PixLib from 2010-Dec-31. The version of ImageJ used was 1.45s. The benchmarks were run on a 27" iMac running Mac OS X 10.6.8, with 3.06 GHz Intel Core 2 Duo processor, 16 GB 1067 MHz RAM, and the Java SE Runtime Environment (build 1.6.0_29-b11-402-10M3527) with Java HotSpot 64-Bit Server VM (build 20.4-b02-402, mixed mode).

Analysis of time performance

For cheap operations, time performance is dominated by the overhead of looping itself, meaning several methods are significantly slower—though in the case of the ImgLib2 Array method, the JIT quickly optimizes it down to raw performance. However, this loop overhead is very small for all methods except PixLib Reflection. Hence, in the expensive case, performance converges across all methods except PixLib Reflection.

Looking at trends as image resolution increases (the "various image resolutions" charts), most methods have less than 1/8th second overhead even for relatively large 25Mpx (5000 x 5000) images. And again, for non-trivial image processing operations, performance is extremely comparable. One oddity is that the JIT appears to optimize ImgLib2 Array performance for most, but not all, image resolutions. However, this discrepancy is noticeable only when loop overhead dominates the algorithm (i.e., for the cheap operation).

In conclusion, we believe there is little reason for concern regarding time performance of any of these libraries, with the exception of the PixLib Reflection approach. And the advantages of ImgLib2's type- and container-agnostic algorithm development certainly outweigh any minor differences in time performance—especially since the flexible containers provide a mechanism for optimizing space performance based on the data type.

Source code

The main benchmark code can be found at:

PerformanceBenchmark.java

The script that runs the benchmark at various image resolutions is:

benchmark.sh

The shell script also uses a Python script to transform the CSV output into Flot data:

chart-gen.py