The benchmarks follow the criteria:

• They are written as the average software developer would write them, i.e.

  • The algorithms are implemented as cited in public sources;
  • The libraries are used as described in the tutorials, documentation and examples;
  • Used data structures are idiomatic.

• The used algorithms are similar between the languages (reference implementations), variants are acceptable if the reference implementation exists.

• All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

• JIT warming up is applied when necessary, and the actual measurements are taken only after the test signals the runner with the TCP request.

2019-12-13

Testing brainfuck implementations using two code samples (bench.b and mandel.b).

Brainfuck v2.0 Brainfuck v1.0

Mandel in Brainfuck

Testing large blob base64 encoding/decoding into newly allocated buffers.

Base64

Testing parsing and simple calculating of values from a big JSON file.

Json

Testing allocating and multiplying matrices.

Matmul

Testing Havlak's loop finder implementations.

Havlak

CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz

Base Docker image: Debian GNU/Linux bullseye/sid

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where is:

• versions (print installed language versions)
• shell (start the shell)
• brainfuck2 bench (build and run Brainfuck2 bench.b benchmarks)
• brainfuck2 mandel (build and run Brainfuck2 mandel.b benchmarks)
• base64 (build and run Base64 benchmarks)
• json (build and run Json benchmarks)
• matmul (build and run Matmul benchmarks)
• havlak (build and run Havlak benchmarks)

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

The tests should be built first (using build.sh) and after that executed (using run.sh and run2.sh where applicable). The measurements are taken using analyze.rb script:

$ cd <test>
$ ../analyze.rb ./run.sh
$ ../analyze.rb ../xtime.rb <single test>

Please note that the measurements could take hours (it uses 10 iterations by default).

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

• Powercap for reading energy counters in Linux (Debian package powercap-utils)

For Python:

• NumPy for matmul tests (Debian package python3-numpy)
• UltraJSON for JSON tests (Debian package python3-ujson)

For C++:

• Boost for JSON tests (Debian package libboost-dev)
• JSON-C for JSON tests (Debian package libjson-c-dev)

For Rust:

• libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/)

For Java, Scala:

• Coursier for downloading Maven artifacts

For Lua:

• LuaSocket for TCP connectivity between the tests and the test runner (Debian package lua-socket)

For Haskell:

• network-simple for TCP connectivity between the tests and the test runner

For C, C++, Chez Scheme:

• libsocket for TCP connectivity between the tests and the test runner