LmdbJava Benchmarks

Just want the latest results?

• Library Comparisons: Full Benchmark | Smoketest
• Version Regressions: Full Benchmark | Smoketest
• LMDB Benchmarks: Full Benchmark | Smoketest
• Historical: 2016 Results

This is a JMH benchmark of open source, embedded, memory-mapped, key-value stores available from Java:

• LmdbJava (with fast ByteBuffer, safe ByteBuffer and an Agrona buffer)
• LMDBJNI
• Lightweight Java Game Library (LMDB API)
• LevelDBJNI
• RocksDB
• MVStore (pure Java)
• MapDB (pure Java)
• Xodus (pure Java)
• Chroncile Map (pure Java) (**)

(**) does not support ordered keys, so iteration benchmarks not performed

The benchmark itself is adapted from LMDB's db_bench_mdb.cc, which in turn is adapted from LevelDB's benchmark.

The benchmark includes:

• Writing data
• Reading all data via each key
• Reading all data via a reverse iterator
• Reading all data via a forward iterator
• Reading all data via a forward iterator and computing a CRC32 (via JDK API)
• Reading all data via a forward iterator and computing a XXH32 hash

Byte arrays (byte[]) are always used for the keys and values, avoiding any serialization library overhead. For those libraries that support compression, it is disabled in the benchmark. In general any special library features that decrease latency (eg batch modes, disable auto-commit, disable journals, hint at expected data sizes etc) were used. While we have tried to be fair and consistent, some libraries offer non-obvious tuning settings or usage patterns that might further reduce their latency. We do not claim we have exhausted every tuning option every library exposes, but pull requests are most welcome.

Build

Clone this repository and build:

mvn clean package

Usage

This benchmark uses POSIX calls to accurately determine consumed disk space and only depends on Linux-specific native library wrappers where a range of such wrappers exists. Operation on non-Linux operating systems is unsupported.

Running Benchmarks

Library Comparison Benchmarks

Use the run-libs.sh script to compare different key-value store libraries:

# Quick smoke test (1K entries, fast verification)
./run-libs.sh smoketest

# Full benchmark using 25% of system RAM (default)
./run-libs.sh benchmark

# Full benchmark using 50% of system RAM
./run-libs.sh benchmark 50

The benchmark auto-scales based on available RAM and caps at 1 million entries. Results are written to target/benchmark-libs/.

File handle limit: Benchmark mode requires at least 1 million file handles for RocksDB and LevelDB LSM operations:

ulimit -n 1000000
./run-libs.sh benchmark

The script will check and abort if the limit is too low.

Resumption: The script skips completed runs by checking for existing result files. For a fresh run, remove the output directory:

rm -rf target/benchmark-libs
./run-libs.sh benchmark

Long-running benchmarks: For multi-hour benchmarks, redirect output to a log file and monitor with tail:

# Start the benchmark with output redirected
./run-libs.sh benchmark > log.txt 2>&1

# Monitor progress in another terminal
tail -f log.txt

Version Regression Testing

Use the run-vers.sh script to test LmdbJava performance across versions:

# Quick smoke test (1K entries, fast verification)
./run-vers.sh smoketest

# Full benchmark using 25% of system RAM (default)
./run-vers.sh benchmark

# Full benchmark using 50% of system RAM
./run-vers.sh benchmark 50

This tests selected LmdbJava versions from Maven Central plus current development branches to identify performance regressions. Results are written to target/benchmark-vers/.

Resumption: The script skips completed versions by checking for existing result files. For a fresh run, remove the output directory:

rm -rf target/benchmark-vers
./run-vers.sh benchmark

LMDB Benchmarks

Use the run-lmdb.sh script to test LmdbJava (master) performance across different LMDB library versions:

# Quick smoke test (1K entries, fast verification)
./run-lmdb.sh smoketest

# Full benchmark using 25% of system RAM (default)
./run-lmdb.sh benchmark

# Full benchmark using 50% of system RAM
./run-lmdb.sh benchmark 50

This tests LmdbJava master against 9 different LMDB library versions (0.9.18-0.9.33) to isolate LMDB native library performance characteristics from LmdbJava wrapper code. Results are written to target/benchmark-lmdb/.

Resumption: The script skips completed LMDB versions by checking for existing result files. For a fresh run, remove the output directory:

rm -rf target/benchmark-lmdb
./run-lmdb.sh benchmark

Running Both Benchmark Suites

Use the run-both.sh script to run both library and version benchmarks sequentially (designed for overnight runs):

# Run both benchmarks using 25% of system RAM (default)
./run-both.sh

# Run both benchmarks using 50% of system RAM
./run-both.sh 50

This will run library comparison benchmarks followed by version regression benchmarks, both in full benchmark mode with 120s iterations. Expect several hours of runtime depending on your system.

Generating Reports

After running library comparison benchmarks, generate a comprehensive report:

./report-libs.sh

After running version regression tests, generate a version comparison report:

./report-vers.sh

After running LMDB benchmarks, generate an LMDB performance report:

./report-lmdb.sh

Reports generate:

• target/benchmark/README.md - Full markdown report with charts
• target/benchmark/index.html - HTML viewer with embedded charts (open in browser)
• Various SVG charts and supporting files

Publishing Reports

After generating a report, you can publish it to Cloudflare Pages:

export CLOUDFLARE_API_TOKEN="your-token"
export CLOUDFLARE_ACCOUNT_ID="your-account-id"
./publish-results.sh

The script automatically detects:

• Type: Library comparison or version regression (from README heading)
• Mode: Smoketest (3s) or benchmark (120s) (from smoketest warning)

Reports are published to:

• libraries-smoketest.lmdbjava.org - Fast daily library verification
• libraries-benchmark.lmdbjava.org - Full library comparison analysis
• versions-smoketest.lmdbjava.org - Fast daily version monitoring
• versions-benchmark.lmdbjava.org - Full version regression analysis

Workflow for curated reports:

1. Run full benchmarks: ./run-both.sh (or individually with ./run-libs.sh benchmark and ./run-vers.sh benchmark)
2. Generate reports: ./report-libs.sh and ./report-vers.sh
3. Review and edit commentary in report scripts if needed, then re-run
4. Publish: ./publish-results.sh (run twice, once after each report generation)

Performance Bisection

The run-bisect.sh script uses bisection to find the commit that introduced a performance regression:

./run-bisect.sh

Configure the bisection by editing variables at the top of the script:

• START_COMMIT: Known good commit (full hash)
• END_COMMIT: Known bad commit (full hash)
• BENCHMARK_NAME: Full benchmark qualifier (eg LmdbJavaAgrona.write)
• MAX_BISECTIONS: Maximum bisection iterations (default 10)

The script:

• Tests endpoint commits to confirm regression
• Bisects the commit range at 50% intervals
• Uses distance-based decision making (closer to good or bad endpoint)
• Uses the system LMDB library (/usr/lib/liblmdb.so) for all tests
• Caches built JARs and benchmark results to avoid rebuilding
• Generates a summary report with an asterisk marking the regression commit

Results are saved in bisect/results/ with a chronologically-sorted summary report showing scores and percentage changes. To re-run with fresh benchmarks, delete bisect/results/*.json.

Development Testing

The run-dev.sh script enables rapid testing of local LmdbJava changes without full bisection overhead:

./run-dev.sh

This script:

• Builds LmdbJava from dev/lmdbjava/ (clone or symlink your working copy here)
• Patches the benchmark pom.xml to use the local SNAPSHOT version
• Runs a quick benchmark (10K entries, 3s runtime by default)
• Saves results to dev/output/ with commit hash and timestamp

To match full bisection configuration, edit the script and change:

• JMH_RUNTIME=30 (30 second iterations)
• NUM_ENTRIES=1000000 (1 million entries)

Use this for iterative development and verification of performance fixes before committing.

Version Management

Update all dependency and plugin versions:

mvn versions:update-properties

Support

Issues are disabled for this repository. Please report any issues or questions on the LmdbJava issue tracker.

Contributing

Contributions are welcome! Please see the LmdbJava project's Contributing Guidelines.

License

This project is licensed under the Apache License, Version 2.0.