From 207dc260b5edda98836eece2fb6510af3d842503 Mon Sep 17 00:00:00 2001 From: Arnav Kapoor Date: Tue, 21 Oct 2025 15:28:30 +0530 Subject: [PATCH] docs(benchmarkprofiles): add detailed explanations of parameters and interpretation (fixes #35) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Explain what τ (performance ratio) and ρ (fraction solved) represent - Add key interpretation guidelines for reading performance profiles - Document customization options (lw, color, linestyles, etc.) - List additional parameters (logscale, sampletol, title) - Add reference to Dolan & Moré 2002 paper - Improve tutorial structure with clearer sections --- .../index.jmd | 37 ++++++++++++++++++- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/tutorials/introduction-to-benchmarkprofiles/index.jmd b/tutorials/introduction-to-benchmarkprofiles/index.jmd index 6c50ada..40ec0ca 100644 --- a/tutorials/introduction-to-benchmarkprofiles/index.jmd +++ b/tutorials/introduction-to-benchmarkprofiles/index.jmd @@ -4,11 +4,24 @@ tags: ["benchmarking"] author: "Abel Soares Siqueira and Dominique Orban" --- -This tutorial is essentially a collection of examples. +This tutorial demonstrates how to use BenchmarkProfiles.jl to visualize and compare solver performance across multiple test problems. ## Performance Profile -Performance profiles are straightforward to use. The input is a matrix `T` with entries `T[i,j]` indicating the cost to solve problem `i` using solver `j`. Cost can be, for instance, elapsed time, or number of evaluations. The cost should be positive. If any cost is zero, all measures will be shifted by 1. +Performance profiles, introduced by Dolan and Moré (2002), provide a graphical way to compare the performance of multiple solvers across a test set. They show the fraction of problems solved by each solver as a function of a performance tolerance. + +### Understanding Performance Profiles + +The input is a matrix `T` with entries `T[i,j]` indicating the cost to solve problem `i` using solver `j`. Cost can be, for instance, elapsed time, number of iterations, or function evaluations. The cost should be positive. If any cost is zero, all measures will be shifted by 1. + +The performance profile plots: +- **x-axis (τ)**: Performance ratio - how much slower a solver is compared to the best solver for each problem. τ=1 means the solver was fastest, τ=2 means it took twice as long as the fastest solver. +- **y-axis (ρ(τ))**: Fraction of problems solved within the performance ratio τ. ρ(2) = 0.8 means the solver solved 80% of problems within twice the time of the best solver. + +**Key interpretations**: +- The height at τ=1 (left side) shows the fraction of problems where the solver was fastest +- The right-side height (as τ→∞) shows the fraction of problems successfully solved (robustness) +- Higher curves are better - the solver solves more problems with smaller performance ratios Basic usage: @@ -35,9 +48,19 @@ T[:,2] = 100T[:,3] performance_profile(PlotsBackend(), T, ["Solver 1", "Solver 2", "Solver 3"]) ``` +### Customization Options + `Plots` arguments can be passed to `performance_profile()` or used as they normally would be with `Plots`. In the example below, we pass `xlabel` to `performance_profile` and set `ylabel` through `ylabel!`. +Common customization options: +- `lw`: Line width +- `c` or `color`: Line colors +- `linestyles`: Line styles (`:solid`, `:dash`, `:dot`, etc.) +- `xlabel`, `ylabel`: Axis labels +- `title`: Plot title +- `legend`: Legend position (e.g., `:bottomright`, `:topleft`) + ```julia using Plots @@ -46,3 +69,13 @@ performance_profile(PlotsBackend(), T, ["Solver 1", "Solver 2", "Solver 3"], lw=2, c=:black, linestyles=[:solid, :dash, :dot], xlabel="τ") ylabel!("ρ(τ)") ``` + +### Additional Parameters + +The `performance_profile` function accepts several optional keyword arguments: + +- `logscale::Bool=true`: Use logarithmic scale on the x-axis (default: true). Useful for viewing performance across a wide range of ratios. +- `sampletol::Number=0`: Tolerance for sampling data points. Can reduce plot complexity for large datasets. +- `title::String=""`: Title for the plot + +For more details on performance profiles, see: Dolan, E. D., & Moré, J. J. (2002). Benchmarking optimization software with performance profiles. Mathematical Programming, 91(2), 201-213.