{"id":6071,"date":"2018-03-27T18:38:11","date_gmt":"2018-03-27T18:38:11","guid":{"rendered":"http:\/\/putridparrot.com\/blog\/?p=6071"},"modified":"2018-03-27T18:38:11","modified_gmt":"2018-03-27T18:38:11","slug":"benchmarking-my-java-code-using-junitbenchmarks","status":"publish","type":"post","link":"https:\/\/putridparrot.com\/blog\/benchmarking-my-java-code-using-junitbenchmarks\/","title":{"rendered":"Benchmarking my Java code using JUnitBenchmarks"},"content":{"rendered":"

As part of building some integration tests for my Java service code, I wanted to get some (micro-)benchmarks run against the tests. In C# we have the likes of NBenchmark (see my post Using NBench for performance testing<\/a>), so it comes as no surprise to find libraries such as JUnitBenchmarks<\/a> in Java.<\/p>\n

Note: the JUnitBenchmarks site states it’s now deprecated in favour of using JMH, but I will cover it here anyway as it’s very simple to use and get started with and fits nicely in with existing JUnit code.<\/em><\/p>\n

JUnitBenchmarks<\/strong><\/p>\n

First off we need to add the required dependency to our pom.xml, so add the following<\/p>\n

\r\n<dependency>\r\n   <groupId>com.carrotsearch<\/groupId>\r\n   <artifactId>junit-benchmarks<\/artifactId>\r\n   <version>0.7.2<\/version>\r\n   <scope>test<\/scope>\r\n<\/dependency>\r\n<\/pre>\n
JUnitBenchmarks, as the name suggests, integrates with JUnit. To enable our tests within the test runner we simply add a rule to the unit test, like this<\/p>\n
\r\npublic class SampleVerticleIntegrationTests {\r\n    @Rule\r\n    public TestRule benchmarkRule = new BenchmarkRule();\r\n\r\n   \/\/ tests\r\n}\r\n<\/pre>\n
This will report information on the test, like this<\/p>\n
\r\n[measured 10 out of 15 rounds, threads: 1 (sequential)]\r\nround: 1.26 [+- 1.04], round.block: 0.00 [+- 0.00], \r\nround.gc: 0.00 [+- 0.00], \r\nGC.calls: 5, GC.time: 0.19, \r\ntime.total: 25.10, time.warmup: 0.00, \r\ntime.bench: 25.10\r\n<\/pre>\n
The first line tells us that the test was actually executed 15 times (or rounds), but only 10 times was it “measured” the other 5 times were warm-ups all on a single thread – this is obviously the default for benchmarking, however what if we want to change these parameters… <\/p>\n
If we want to be more specific about the benchmarking of various test methods we add the annotation @BenchmarkOptions<\/em>, for example<\/p>\n
\r\n@BenchmarkOptions(benchmarkRounds = 20, warmupRounds = 0)\r\n@Test\r\npublic void testSave() {\r\n   \/\/ our code\r\n}\r\n<\/pre>\n
As can be seen, this is a standard test but the annotation tells JUnitBenchmarks will run the test 20 times (with no warm-up runs) and then report the benchmark information, for example<\/p>\n
\r\n[measured 20 out of 20 rounds, threads: 1 (sequential)]\r\nround: 1.21 [+- 0.97], round.block: 0.00 [+- 0.00], \r\nround.gc: 0.00 [+- 0.00], \r\nGC.calls: 4, GC.time: 0.22, \r\ntime.total: 24.27, time.warmup: 0.00, \r\ntime.bench: 24.27\r\n<\/pre>\n
As you can see the first line tells us the code was measured 20 times on a single thread with no warm-ups (as we specified).<\/p>\n
I’m not going to cover build integration here, but checkout JUnitBenchmarks: Build Integration<\/a> for such information.<\/p>\n
What do the results actually mean?<\/strong><\/p>\n
I’ll pretty much recreate what’s on Class Result<\/a> here.<\/p>\n
Let’s look at these results…<\/p>\n
\r\n[measured 20 out of 20 rounds, threads: 1 (sequential)]\r\nround: 1.21 [+- 0.97], round.block: 0.00 [+- 0.00], \r\nround.gc: 0.00 [+- 0.00], \r\nGC.calls: 4, GC.time: 0.22, \r\ntime.total: 24.27, time.warmup: 0.00, \r\ntime.bench: 24.27\r\n<\/pre>\n
We’ve already seen that the first line tells us how many times the test was run, and how many of those runs were warm-ups. It also tells us how many threads were used in this benchmark.<\/p>\n
round<\/strong> tells us the average round time in seconds (hence the example took 1.21 seconds with a stddev of +\/- 0.97 seconds).
\nround.block<\/strong> tells us the average (and stddev) of blocked threads, in this example there’s no concurrency hence 0.00.
\nround.gc<\/strong> tells us the average and stddev of the round’s GC time.
\nGC.calls<\/strong> tells us the number of times GC was invoked (in this example 4 times).
\nGC.time<\/strong> tels us the accumulated time take invoking the GC (0.22 seconds in this example).
\ntime.total<\/strong> tells us the total benchmark time which includes benchmarking and GC overhead.
\ntime.warmup<\/strong> tells us the total warmup time which includes benchmarking and GC overhead.<\/p>\n
Caveats<\/strong><\/p>\n
Apart from the obvious caveat that this library has been marked as deprecated (but I feel it’s still useful), when benchmarking you have to be aware that, the results may be dependent upon outside factors, such as memory available, maybe hard disk\/SSD speed if tests include any file I\/O, network latency etc. So such figures are best seen as approximates of performance etc. <\/p>\n
Also there’s seems to be no way to “fail” a test, for example if the test exceeds a specified time or more GC’s than x<\/em> are seen, so treat these more as informational.<\/p>\n","protected":false},"excerpt":{"rendered":"
As part of building some integration tests for my Java service code, I wanted to get some (micro-)benchmarks run against the tests. In C# we have the likes of NBenchmark (see my post Using NBench for performance testing), so it comes as no surprise to find libraries such as JUnitBenchmarks in Java. Note: the JUnitBenchmarks […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[161],"tags":[],"class_list":["post-6071","post","type-post","status-publish","format-standard","hentry","category-java"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts\/6071","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/comments?post=6071"}],"version-history":[{"count":7,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts\/6071\/revisions"}],"predecessor-version":[{"id":6078,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts\/6071\/revisions\/6078"}],"wp:attachment":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/media?parent=6071"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/categories?post=6071"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/tags?post=6071"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}