Skip to content

Add support for HPT results #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 30, 2023
Merged

Add support for HPT results #25

merged 4 commits into from
Aug 30, 2023

Conversation

mdboom
Copy link
Contributor

@mdboom mdboom commented Aug 25, 2023

Support for the Heirarchical Performance Testing (HPT) method in this paper:

T. Chen, Y. Chen, Q. Guo, O. Temam, Y. Wu and W. Hu,
"Statistical performance comparisons of computers,"
IEEE International Symposium on High-Performance Comp Architecture,
New Orleans, LA, USA, 2012, pp. 1-12,
doi: 10.1109/HPCA.2012.6169043.

This is largely a direct port of the bash implementation available here:

https://github.com/cirosantilli/parsec-benchmark/tree/master/toolkit/hpt

This approach is a more robust way to measure overall effectiveness across a number of benchmarks. It is still biased in that the benchmarks should be a representative sample, but it accounts for the fact that some benchmarks are more reproducible and reliable than others.

It has been modified so that each benchmark can have a different number of samples (the original code assumed the matrix was rectangular, but there is nothing about the method itself that should require that).

a3 = 1.781477937
a4 = -1.821255978
a5 = 1.330274429
pi = 3.141592653589793238462643

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you just use math.pi here? These constants in general feel a little 'magic', is there a CDF function in the stdlib or numpy that could be used instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a CDF function in Scipy, with different behavior as to how the wraparound is handled. So it's both too big a dependency for this little thing, IMHO, and it doesn't do what we need anyway.

But, yes, could use math.pi.

import functools
import json
import os
from typing import Dict, Optional, Tuple

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use from __future__ import annotations and dispense with these typing imports by using PEP 585 types

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL!

mdboom added 4 commits August 30, 2023 09:41
Support for the Heirarchical Performance Testing (HPT) method in this paper:

  T. Chen, Y. Chen, Q. Guo, O. Temam, Y. Wu and W. Hu,
  "Statistical performance comparisons of computers,"
  IEEE International Symposium on High-Performance Comp Architecture,
  New Orleans, LA, USA, 2012, pp. 1-12,
  doi: 10.1109/HPCA.2012.6169043.

This is largely a direct port of the bash implementation available here:

  https://github.com/cirosantilli/parsec-benchmark/tree/master/toolkit/hpt

This approach is a more robust way to measure overall effectiveness across a
number of benchmarks. It is still biased in that the benchmarks should be a
representative sample, but it accounts for the fact that some benchmarks are
more reproducible and reliable than others.

It has been modified so that each benchmark can have a different number of
samples (the original code assumed the matrix was rectangular, but there is
nothing about the method itself that should require that).
@mdboom mdboom merged commit 23bacda into main Aug 30, 2023
@mdboom mdboom deleted the hpt branch September 12, 2023 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants