Elixir Benchfella & Benchwarmer的结果截然不同

Question

我正在尝试对快速排序算法与Enum.sort进行基准测试。我从benchfella和benchwarmer得到了截然不同的结果。

简介:

               Benchfella        Benchwarmer 
Enum.sort       8920.47 µs/op    2418767.00 μs/op
QuickSort      16660.45 µs/op      15745.04 μs/op

---

细节:

这是我的基准:

defmodule Thing do
  defstruct [:key]
end
defmodule QuickSort do
  def qsort([]), do: []
  def qsort([pivot | rest]) do
    {left, right} = Enum.partition(rest, fn(x) -> x.key < pivot.key end)
    qsort(left) ++ [pivot] ++ qsort(right)
  end
end
defmodule Bench do
  use Benchfella
  Benchfella.start
  bench "QuickSort", [list: gen()] do
    QuickSort.qsort(list)
  end
  bench "Enum.sort", [list: gen()] do
    Enum.sort(list, fn(x, y) -> x.key > y.key end)
  end
 def gen, do: for _ <- 1..10000, do: %Thing{key: :rand.uniform}
  # Tests
  list = for _ <- 1..10000, do: %Thing{key: :rand.uniform}
  sorted = Enum.sort_by(list, &(&1.key))
  true = sorted == QuickSort.qsort(list)
end

Benchfella的输出:

$ mix bench
Settings:
  duration:      1.0 s
## Bench
[20:57:40] 1/2: Enum.sort
[20:57:43] 2/2: QuickSort
Finished in 4.71 seconds
## Bench
benchmark  iterations   average time
Enum.sort         200   8920.47 µs/op
QuickSort         100   16660.45 µs/op

下面是Benchwarmer的输出:

$ iex -S mix
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Interactive Elixir (1.3.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> list = Bench.gen
[%Thing{key: 0.29976857621751096}, %Thing{key: 0.42956956935048163},
 %Thing{key: 0.8682735084348573}, %Thing{key: 0.13149039866062429},
 %Thing{key: 0.5315758481143932}, %Thing{...}, ...]
iex(2)> Benchwarmer.benchmark fn -> QuickSort.qsort list end
*** #Function<20.52032458/0 in :erl_eval.expr/5> ***
1.9 sec    127 iterations   15745.04 μs/op
[%Benchwarmer.Results{args: [], duration: 1999619,
  function: #Function<20.52032458/0 in :erl_eval.expr/5>, n: 127, prev_n: 64}]
iex(3)> Benchwarmer.benchmark fn -> Enum.sort(list, fn(x, y) -> x.key > y.key end) end
*** #Function<20.52032458/0 in :erl_eval.expr/5> ***
2.4 sec      1 iterations   2418767.0 μs/op
[%Benchwarmer.Results{args: [], duration: 2418767,
  function: #Function<20.52032458/0 in :erl_eval.expr/5>, n: 1, prev_n: 1}]

Answer 1

这很可能是由于测试shell中定义的函数引起的。在shell中，代码被求值，而不是编译。除了编译过的模块，不要对任何东西进行基准测试。在编译模块之外进行的任何基准测试或多或少都是无用的。