Ruby基准测试异常的来源

Question

运行此代码:

require 'benchmark'
Benchmark.bm do |x|
  x.report("1+1") {15_000_000.times {1+1}}
  x.report("1+1") {15_000_000.times {1+1}}
  x.report("1+1") {15_000_000.times {1+1}}
  x.report("1+1") {15_000_000.times {1+1}}
  x.report("1+1") {15_000_000.times {1+1}}
end

输出这些结果:

       user     system      total        real
1+1  2.188000   0.000000   2.188000 (  2.250000)
1+1  2.250000   0.000000   2.250000 (  2.265625)
1+1  2.234000   0.000000   2.234000 (  2.250000)
1+1  2.203000   0.000000   2.203000 (  2.250000)
1+1  2.266000   0.000000   2.266000 (  2.281250)

猜测变化是系统环境的结果，但想确认这种情况。

Answer 1

"猜测变异是系统环境的结果"，你是对的。

基准测试不可能总是精确的。你没有一个完美的常规机器总是在同一时间运行某些东西。如果两个数字太接近，则从基准中取相同的数字，如在本例中。

Answer 2

我尝试使用eval部分展开循环，虽然它使它更快，但它使执行时间不太一致!

$VERBOSE &&= false # You do not want 15 thousand "warning: useless use of + in void context" warnings
# large_number = 15_000_000 # Too large! Caused eval to take too long, so I gave up
somewhat_large_number = 15_000
unrolled = "def do_additionn" + ("1+1n" * somewhat_large_number) + "endn" ; nil
eval(unrolled)
require 'benchmark'
Benchmark.bm do |x|
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
  x.report("1+1 partially unrolled") { i = 0; while i < 1000; do_addition; i += 1; end}
end

给我

      user     system      total        real
1+1 partially unrolled  0.750000   0.000000   0.750000 (  0.765586)
1+1 partially unrolled  0.765000   0.000000   0.765000 (  0.765586)
1+1 partially unrolled  0.688000   0.000000   0.688000 (  0.703089)
1+1 partially unrolled  0.797000   0.000000   0.797000 (  0.796834)
1+1 partially unrolled  0.750000   0.000000   0.750000 (  0.749962)
1+1 partially unrolled  0.781000   0.000000   0.781000 (  0.781210)
1+1 partially unrolled  0.719000   0.000000   0.719000 (  0.718713)
1+1 partially unrolled  0.750000   0.000000   0.750000 (  0.749962)
1+1 partially unrolled  0.765000   0.000000   0.765000 (  0.765585)
1+1 partially unrolled  0.781000   0.000000   0.781000 (  0.781210)

为了比较，您在我的计算机上的基准测试给出了

      user     system      total        real
1+1  2.406000   0.000000   2.406000 (  2.406497)
1+1  2.407000   0.000000   2.407000 (  2.484629)
1+1  2.500000   0.000000   2.500000 (  2.734655)
1+1  2.515000   0.000000   2.515000 (  2.765908)
1+1  2.703000   0.000000   2.703000 (  4.391075)

(real time在最后一行变化，但不包括user或total)