Sandmark: Boosting Multicore Projects with Performance Benchmarking

Introduction

In the realm of software development, continuous improvement is paramount. When it comes to Multicore projects, the need for thorough benchmarking becomes even more critical. This is where Sandmark comes into play. Sandmark, developed for the OCaml programming language, has proven to be an invaluable tool for optimising performance and aiding in upstreaming efforts. In this blog post, we will explore the benefits of using Sandmark and its role in the development of Multicore projects.

Enhancing Upstreaming Efforts

Sandmark has been extensively used in Multicore projects to assist with upstreaming. Its impact can be witnessed in the OCaml community, where it helped demonstrate that sequential programs running on OCaml 5 performed nearly as efficiently as those running on OCaml 4. For instance, the results achieved in the Multicore PR (pull request) merge were accomplished using Sandmark. Additionally, the findings presented in the ICFP'20 paper were all obtained through the utilisation of Sandmark. This tool has played a crucial role in showcasing the progress made tracking performance regressions in the OCaml compiler.

Ongoing Compiler Development

Even after the Multicore merge, Sandmark remains actively employed with a dashboard for the compiler development. Its significance is evident in the multitude of pull requests related to Sandmark in the OCaml repository. For example, consider the issue #11589, where an idle processing domain slows down major garbage collection (GC) cycles. The comparison of parallel benchmarks between the fix in the PR and the current development version of the compiler in the illustration shows the speedup comparisons. This highlights the continued reliance on Sandmark as a vital tool in the compiler development process.

Nightly Benchmarking

One of the key aspects of Sandmark is its nightly benchmarking feature. Sandmark ensures that benchmarks are run regularly on diverse x86 servers, namely Turing (Intel Xeon Gold with 56 cores) and Navajo (AMD EPYC 7551 with 128 cores). This practice serves as a proactive measure to continuously identify and address performance regressions promptly. The nightly runs cover both sequential and parallel benchmarks, providing comprehensive insights into the program's behaviour under different scenarios for different inputs.

Sandmark Nightly Config

To simplify the process of requesting development branches for nightly benchmarking, Sandmark offers a convenient service called "Sandmark Nightly Config". This service streamlines simplifies the configuration setup for benchmarking, thereby reducing the steps required to initiate the benchmark runs. Compiler developers only need to provide their development branch URL for the configuration, and the nightly service will execute both the sequential and parallel benchmarks. By automating this process, developers can focus on their core tasks while still benefiting from gaining insights from the regular benchmark runsing insights.

Permalinks for Easy Sharing and Discussion

A remarkable feature of Sandmark is the provision of permalinks. These permalinks enable users to easily share benchmark results and engage in meaningful discussions. You can specify more than two development branches across dates, and even different hosts for comparison. This capability is a game-changer for collaborative development, as it facilitates efficient communication and fosters a deeper understanding of the pull request changes using the benchmarking outcomes. The permalinks in Sandmark allow for specific results to be referenced and examined in detail.

Importance of Perfstat Output

Sandmark offers perfstat output, which plays a vital role in accurately evaluating program performance. Modern machines exhibit varying raw running times due to their complex nature. However, “instructions retired” provide a more stable and reliable metric, especially when assessing the impact of compiler optimisations. This feature ensures that performance analysis is based on consistent and meaningful measurements.

Looking Towards the Future

Sandmark continues to evolve, with ongoing developments in the Multicore release. The efforts put into enhancing Sandmark reflect the commitment to improving Multicore programming in OCaml. As the OCaml community pushes the boundaries of Multicore development, Sandmark will undoubtedly play a crucial role in optimising performance, tracking regressions, and ensuring the stability of the language.

Conclusion

Sandmark has emerged as an indispensable tool for the OCaml community, particularly in the realm of Multicore projects. Its ability to benchmark performance, catch regressions, simplify configuration, and facilitate discussions through permalinks has greatly contributed to the OCaml compiler development process. The commitment to ongoing improvements and enhancements will help measure, monitor and track the compiler development as the OCaml language evolves. We encourage you to try the above services, and share any feedback or file new feature requests or GitHub issues for the Sandmark project.

References

• Sandmark. https://github.com/ocaml-bench/sandmark
• Multicore OCaml PR merge. https://github.com/ocaml/ocaml/pull/10831
• Retrofitting Parallelism onto OCaml. https://kcsrk.info/papers/retro-parallel_icfp_20.pdf
• Sandmark Dashboard. https://sandmark.tarides.com/
• Idle Domain Slows Down GC Cycles. https://github.com/ocaml/ocaml/issues/11589
• Sandmark-nightly-config. https://github.com/ocaml-bench/sandmark-nightly-config