Go developers who have written benchmarks using the testing package might have encountered some of its various pitfalls. Go 1.24 introduces a new way to write benchmarks that’s just as easy to use, but at the same time far more robust: testing.B.Loop. Traditionally, Go benchmarks are written using a loop from 0 to b.N: func Benchmark(b *testing.B) {
  for range b.N {
    ... code to measure ...
  }
} Using b.Loop instead is a trivial change: func Benchmark(b *testing.B) {
  for b.Loop() {
    ... code to measure ...
  }
} testing.B.Loop has many benefits: It prevents unwanted compiler optimizations within the benchmark loop.
It automatically excludes setup and cleanup code from benchmark timing.
Code can’t accidentally depend on the total number of iterations or the current iteration. These were all easy mistakes to make with b.N-style benchmarks that would silently result in bogus benchmark results. As an added bonus, b.Loop-style benchmarks even complete in less time! Let’s explore the advantages of testing.B.Loop and how to effectively utilize it. Old benchmark loop problems Before Go 1.24, while the basic structure of a benchmark was simple, more sophisticated benchmarks required more care: func Benchmark(b *testing.B) {
  ... setup ...
  b.ResetTimer() // if setup may be expensive
  for range b.N {
    ... code to measure ...
    ... use sinks or accumulation to prevent dead-code elimination ...
  }
  b.StopTimer() // if cleanup or reporting may be expensive
  ... cleanup ...
  ... report ...
} If setup or cleanup are non-trivial, the developer needs to surround the benchmark loop with ResetTimer and/or StopTimer calls. These are easy to forget, and even if the developer remembers they may be necessary, it can be difficult to judge whether setup or cleanup are “expensive enough” to require them. Without these, the testing package can only time the entire benchmark function. If a benchmark function omits them, the setup and cleanup code will be included in the overall time measurement, silently skewing the final benchmark result. There is another, more subtle pitfall that requires deeper understanding: (Example source) func isCond(b byte) bool {
  if b%3 == 1 && b%7 == 2 && b%17 == 11 && b%31 == 9 {
    return true
  }
  return false
}

func BenchmarkIsCondWrong(b *testing.B) {
  for range b.N {
    isCond(201)
  }
} In this example, the user might observe isCond executing in sub-nanosecond time. CPUs are fast, but not that fast! This seemingly anomalous result stems from the fact that isCond is inlined, and since its result is never used, the compiler eliminates it as dead code. As a result, this benchmark doesn’t measure isCond at all; it measures how long it takes to do nothing. In this case, the sub-nanosecond result is a clear red flag, but in more complex benchmarks, partial dead-code elimination can lead to results that look reasonable but still aren’t measuring what was intended. How testing.B.Loop helps Unlike a b.N-style benchmark, testing.B.Loop is able to track when it is first called in a benchmark when the final iteration ends. The b.ResetTimer at the loop’s start and b.StopTimer at its end are integrated into testing.B.Loop, eliminating the need to manually manage the benchmark timer for setup and cleanup code. Furthermore, the Go compiler now detects loops where the condition is just a call to testing.B.Loop and prevents dead code elimination within the loop. In Go 1.24, this is implemented by disallowing inlining into the body of such a loop, but we plan to improve this in the future. Another nice feature of testing.B.Loop is its one-shot ramp-up approach. With a b.N-style benchmark, the testing package must call the benchmark function several times with different values of b.N, ramping up until the measured time reached a threshold. In contrast, b.Loop can simply run the benchmark loop until it reaches the time threshold, and only needs to call the benchmark function once. Internally, b.Loop still uses a ramp-up process to amortize measurement overhead, but this is hidden from the caller and can be more efficient. Certain constraints of the b.N-style loop still apply to the b.Loop-style loop. It remains the user’s responsibility to manage the timer within the benchmark loop, when necessary: (Example source) func BenchmarkSortInts(b *testing.B) {
  ints := make([]int, N)
  for b.Loop() {
    b.StopTimer()
    fillRandomInts(ints)
    b.StartTimer()
    slices.Sort(ints)
  }
} In this example, to benchmark the in-place sorting performance of slices.Sort,a randomly initialized array is required for each iteration. The user must still manually manage the timer in such cases. Furthermore, there still needs to be exactly one such loop in the benchmark function body (a b.N-style loop cannot coexist with a b.Loop-style loop), and every iteration of the loop should do the same thing. When to use The testing.B.Loop method is now the preferred way to write benchmarks: func Benchmark(b *testing.B) {
  ... setup ...
  for b.Loop() {
    // optional timer control for in-loop setup/cleanup
    ... code to measure ...
  }
  ... cleanup ...
} testing.B.Loop offers faster, more accurate, and more intuitive benchmarking. Acknowledgements A huge thank you to everyone in the community who provided feedback on the proposal issue and reported bugs as this feature was released! I’m also grateful to Eli Bendersky for his helpful blog summaries. And finally a big thank you to Austin Clements, Cherry Mui and Michael Pratt for their review, thoughtful work on the design options and documentation improvements. Thank you all for your contributions! Credits: Junyang Shao Photo by Will Paterson on Unsplash This article is available on The Go Blog under a CC BY 4.0 DEED license. Go developers who have written benchmarks using the testing package might have encountered some of its various pitfalls. Go 1.24 introduces a new way to write benchmarks that’s just as easy to use, but at the same time far more robust: testing.B.Loop . testing testing.B.Loop Traditionally, Go benchmarks are written using a loop from 0 to b.N : b.N func Benchmark(b *testing.B) {
  for range b.N {
    ... code to measure ...
  }
} func Benchmark(b *testing.B) {
  for range b.N {
    ... code to measure ...
  }
} Using b.Loop instead is a trivial change: b.Loop func Benchmark(b *testing.B) {
  for b.Loop() {
    ... code to measure ...
  }
} func Benchmark(b *testing.B) {
  for b.Loop() {
    ... code to measure ...
  }
} testing.B.Loop has many benefits: testing.B.Loop It prevents unwanted compiler optimizations within the benchmark loop. It automatically excludes setup and cleanup code from benchmark timing. Code can’t accidentally depend on the total number of iterations or the current iteration. It prevents unwanted compiler optimizations within the benchmark loop. It automatically excludes setup and cleanup code from benchmark timing. Code can’t accidentally depend on the total number of iterations or the current iteration. These were all easy mistakes to make with b.N -style benchmarks that would silently result in bogus benchmark results. As an added bonus, b.Loop -style benchmarks even complete in less time! b.N b.Loop Let’s explore the advantages of testing.B.Loop and how to effectively utilize it. testing.B.Loop Old benchmark loop problems Before Go 1.24, while the basic structure of a benchmark was simple, more sophisticated benchmarks required more care: func Benchmark(b *testing.B) {
  ... setup ...
  b.ResetTimer() // if setup may be expensive
  for range b.N {
    ... code to measure ...
    ... use sinks or accumulation to prevent dead-code elimination ...
  }
  b.StopTimer() // if cleanup or reporting may be expensive
  ... cleanup ...
  ... report ...
} func Benchmark(b *testing.B) {
  ... setup ...
  b.ResetTimer() // if setup may be expensive
  for range b.N {
    ... code to measure ...
    ... use sinks or accumulation to prevent dead-code elimination ...
  }
  b.StopTimer() // if cleanup or reporting may be expensive
  ... cleanup ...
  ... report ...
} If setup or cleanup are non-trivial, the developer needs to surround the benchmark loop with ResetTimer and/or StopTimer calls. These are easy to forget, and even if the developer remembers they may be necessary, it can be difficult to judge whether setup or cleanup are “expensive enough” to require them. ResetTimer StopTimer Without these, the testing package can only time the entire benchmark function. If a benchmark function omits them, the setup and cleanup code will be included in the overall time measurement, silently skewing the final benchmark result. testing There is another, more subtle pitfall that requires deeper understanding: ( Example source ) Example source func isCond(b byte) bool {
  if b%3 == 1 && b%7 == 2 && b%17 == 11 && b%31 == 9 {
    return true
  }
  return false
}

func BenchmarkIsCondWrong(b *testing.B) {
  for range b.N {
    isCond(201)
  }
} func isCond(b byte) bool {
  if b%3 == 1 && b%7 == 2 && b%17 == 11 && b%31 == 9 {
    return true
  }
  return false
}

func BenchmarkIsCondWrong(b *testing.B) {
  for range b.N {
    isCond(201)
  }
} In this example, the user might observe isCond executing in sub-nanosecond time. CPUs are fast, but not that fast! This seemingly anomalous result stems from the fact that isCond is inlined, and since its result is never used, the compiler eliminates it as dead code. As a result, this benchmark doesn’t measure isCond at all; it measures how long it takes to do nothing. isCond isCond isCond In this case, the sub-nanosecond result is a clear red flag, but in more complex benchmarks, partial dead-code elimination can lead to results that look reasonable but still aren’t measuring what was intended. How testing.B.Loop helps testing.B.Loop Unlike a b.N -style benchmark, testing.B.Loop is able to track when it is first called in a benchmark when the final iteration ends. The b.ResetTimer at the loop’s start and b.StopTimer at its end are integrated into testing.B.Loop , eliminating the need to manually manage the benchmark timer for setup and cleanup code. b.N testing.B.Loop b.ResetTimer b.StopTimer testing.B.Loop Furthermore, the Go compiler now detects loops where the condition is just a call to testing.B.Loop and prevents dead code elimination within the loop. In Go 1.24, this is implemented by disallowing inlining into the body of such a loop, but we plan to improve this in the future. testing.B.Loop improve Another nice feature of testing.B.Loop is its one-shot ramp-up approach. With a b.N -style benchmark, the testing package must call the benchmark function several times with different values of b.N , ramping up until the measured time reached a threshold. testing.B.Loop b.N b.N In contrast, b.Loop can simply run the benchmark loop until it reaches the time threshold, and only needs to call the benchmark function once. Internally, b.Loop still uses a ramp-up process to amortize measurement overhead, but this is hidden from the caller and can be more efficient. b.Loop b.Loop Certain constraints of the b.N -style loop still apply to the b.Loop -style loop. It remains the user’s responsibility to manage the timer within the benchmark loop, when necessary: ( Example source ) b.N b.Loop Example source func BenchmarkSortInts(b *testing.B) {
  ints := make([]int, N)
  for b.Loop() {
    b.StopTimer()
    fillRandomInts(ints)
    b.StartTimer()
    slices.Sort(ints)
  }
} func BenchmarkSortInts(b *testing.B) {
  ints := make([]int, N)
  for b.Loop() {
    b.StopTimer()
    fillRandomInts(ints)
    b.StartTimer()
    slices.Sort(ints)
  }
} In this example, to benchmark the in-place sorting performance of slices.Sort ,a randomly initialized array is required for each iteration. The user must still manually manage the timer in such cases. slices.Sort Furthermore, there still needs to be exactly one such loop in the benchmark function body (a b.N -style loop cannot coexist with a b.Loop -style loop), and every iteration of the loop should do the same thing. b.N b.Loop When to use The testing.B.Loop method is now the preferred way to write benchmarks: testing.B.Loop func Benchmark(b *testing.B) {
  ... setup ...
  for b.Loop() {
    // optional timer control for in-loop setup/cleanup
    ... code to measure ...
  }
  ... cleanup ...
} func Benchmark(b *testing.B) {
  ... setup ...
  for b.Loop() {
    // optional timer control for in-loop setup/cleanup
    ... code to measure ...
  }
  ... cleanup ...
} testing.B.Loop offers faster, more accurate, and more intuitive benchmarking. testing.B.Loop Acknowledgements A huge thank you to everyone in the community who provided feedback on the proposal issue and reported bugs as this feature was released! I’m also grateful to Eli Bendersky for his helpful blog summaries. And finally a big thank you to Austin Clements, Cherry Mui and Michael Pratt for their review, thoughtful work on the design options and documentation improvements. Thank you all for your contributions! Credits: Junyang Shao Credits: Junyang Shao Photo by Will Paterson on Unsplash Photo by Will Paterson on Unsplash Will Paterson Unsplash This article is available on The Go Blog under a CC BY 4.0 DEED license. This article is available on The Go Blog under a CC BY 4.0 DEED license. The Go Blog The Go Blog

Certain

Happy 15th Birthday Go!

testing.B.Loop: Some More Predictable Benchmarking for You

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Beginner's Guide to Code Coverage for Go Integration Tests

104 Stories To Learn About Go

The Noonification: How to Use React to Replace useEffect (12/31/2022)

3 Golang Pitfalls Every Developer Needs to Know

207 Stories To Learn About Golang

2 Error-Free Options for Decimal handling in Golang

A Beginner's Guide to Code Coverage for Go Integration Tests

104 Stories To Learn About Go

The Noonification: How to Use React to Replace useEffect (12/31/2022)

3 Golang Pitfalls Every Developer Needs to Know

207 Stories To Learn About Golang

2 Error-Free Options for Decimal handling in Golang

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps