-
Notifications
You must be signed in to change notification settings - Fork 6
Add support for HPT results #25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
bench_runner/hpt.py
Outdated
a3 = 1.781477937 | ||
a4 = -1.821255978 | ||
a5 = 1.330274429 | ||
pi = 3.141592653589793238462643 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you just use math.pi
here? These constants in general feel a little 'magic', is there a CDF function in the stdlib or numpy that could be used instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a CDF function in Scipy, with different behavior as to how the wraparound is handled. So it's both too big a dependency for this little thing, IMHO, and it doesn't do what we need anyway.
But, yes, could use math.pi
.
import functools | ||
import json | ||
import os | ||
from typing import Dict, Optional, Tuple |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could use from __future__ import annotations
and dispense with these typing imports by using PEP 585 types
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TIL!
Support for the Heirarchical Performance Testing (HPT) method in this paper: T. Chen, Y. Chen, Q. Guo, O. Temam, Y. Wu and W. Hu, "Statistical performance comparisons of computers," IEEE International Symposium on High-Performance Comp Architecture, New Orleans, LA, USA, 2012, pp. 1-12, doi: 10.1109/HPCA.2012.6169043. This is largely a direct port of the bash implementation available here: https://github.com/cirosantilli/parsec-benchmark/tree/master/toolkit/hpt This approach is a more robust way to measure overall effectiveness across a number of benchmarks. It is still biased in that the benchmarks should be a representative sample, but it accounts for the fact that some benchmarks are more reproducible and reliable than others. It has been modified so that each benchmark can have a different number of samples (the original code assumed the matrix was rectangular, but there is nothing about the method itself that should require that).
Support for the Heirarchical Performance Testing (HPT) method in this paper:
T. Chen, Y. Chen, Q. Guo, O. Temam, Y. Wu and W. Hu,
"Statistical performance comparisons of computers,"
IEEE International Symposium on High-Performance Comp Architecture,
New Orleans, LA, USA, 2012, pp. 1-12,
doi: 10.1109/HPCA.2012.6169043.
This is largely a direct port of the bash implementation available here:
https://github.com/cirosantilli/parsec-benchmark/tree/master/toolkit/hpt
This approach is a more robust way to measure overall effectiveness across a number of benchmarks. It is still biased in that the benchmarks should be a representative sample, but it accounts for the fact that some benchmarks are more reproducible and reliable than others.
It has been modified so that each benchmark can have a different number of samples (the original code assumed the matrix was rectangular, but there is nothing about the method itself that should require that).