Algo Benchmark Library card catalog #235

mpharrigan · 2021-12-03T00:33:49Z

This "card catalog" will be the centralized record of algorithmic benchmarks that are highly organized and use the new cirq_google.workflow cirqflow style of execution.

This PR does not commit any benchmarks, but establishes (via the dataclasses and the tests) a set of conventions for organizing things.

Change-Id: I4ca692cd92ddbee95d0d2aeed5a4027fcd0f2aa4

dstrain115 · 2021-12-03T10:18:00Z

recirq/algo_benchmark_library.py

+
+
+@dataclass
+class AlgoBenchmark:


Should we call this AlgorithmBenchmark? It is more verbose, but abbreviations are generally discouraged:
cirq-google/cirq_google/engine/engine_program.py

done 77e48f9

dstrain115 · 2021-12-03T10:19:43Z

recirq/algo_benchmark_library.py

+@dataclass
+class AlgoBenchmark:
+    domain: str
+    """The problem domain. Usually a high-level ReCirq module."""


Is this the proper way to document this class? I thought the usual way to document these was to have a docstring at the beginning of the class with an "Args" section that describes each parameter.

done 9481205

dstrain115 · 2021-12-03T10:21:42Z

recirq/algo_benchmark_library.py

+    data_class: Type
+    """The class which can contain ETL-ed data for this AlgoBenchmark."""
+
+    gen_func: Callable[[...], QuantumExecutableGroup]


This is also an abbreviation. Maybe generator would be a better name?

done 6f9224c

dstrain115 · 2021-12-03T10:23:59Z

recirq/algo_benchmark_library.py

+
+
+@lru_cache()
+def get_all_algo_configs() -> List[Tuple[AlgoBenchmark, BenchmarkConfig]]:


Missing docstring.

done 8f139ec

dstrain115 · 2021-12-03T10:24:39Z

recirq/algo_benchmark_library.py

+
+
+def main():
+    df = pd.DataFrame([algo.as_strings() for algo in BENCHMARKS]).set_index('executable_family')


Heh. This is a funny way of printing out the benchmark names. You are using pandas just for text formatting?

sortof forgot I left this in here. Changed the name of this function and removed the __main__ entry point 1514b20

The nice thing about pandas is you can use the same technique in a juypter noteook and get a nice html table that you can paste into sheets or a doc

dstrain115 · 2021-12-03T10:26:14Z

recirq/algo_benchmark_library.py

+@dataclass
+class AlgoBenchmark:
+    domain: str
+    """The problem domain. Usually a high-level ReCirq module."""


"Usually a high-level ReCirq module." This is actually enforced by the tests, so I would adjust the wording to make it mandatory and also make clear that it should include the "recirq." prefix.

done 7c7faa1

dstrain115 · 2021-12-03T10:28:10Z

recirq/algo_benchmark_library_test.py

+@pytest.mark.parametrize('algo', BENCHMARKS)
+def test_executable_family_is_formulaic(algo):
+    # Check consistency in the AlgoBenchmark dataclass:
+    assert algo.executable_family == algo.spec_class.executable_family


Since these are checking consistency for future benchmarks, it might be good to add messages to this assert and other asserts in this file so that developers of benchmarks know why their tests are failing.

done 3308c5e

dstrain115 · 2021-12-03T10:29:41Z

recirq/algo_benchmark_library.py

+    """A globally unique name for this config."""
+
+    gen_script: str
+    """The script filename that generates the QuantumExecutableGroup for this Config."""


We should include that the gen-script should begin with "gen-". Same for other test-enforced conventions.

done 69b6b65

Change-Id: Ic95e07f4690f4f10bf2634eda7a66ac81649774a

Change-Id: Ic1213396a235f6e40e46d2f3472d8380bb653719

Change-Id: I97e19c6544def983cadde8c959fbc83b5d9f867e

Change-Id: I7c23fa8eac6d0b04cc7a480ac6a81cf4cb29f308

Change-Id: I21a65e3e126c42bb227582d335d24a3fca287015

Change-Id: Ic393c5bb90b6df5b50b9a70ae0b3386d43a89e6f

Change-Id: I95696873269c5e66dcbffedce7814c4488ac6d47

Change-Id: I04cd53462b7190e071cd6ec680aaec37e08e7064

Change-Id: I7a78669ee18d47a4e909783663c2da210d1db699

mpharrigan

ptal

mpharrigan · 2021-12-04T00:18:03Z

recirq/algo_benchmark_library.py

+
+
+@dataclass
+class AlgoBenchmark:


done 77e48f9

mpharrigan · 2021-12-04T00:22:06Z

recirq/algo_benchmark_library.py

+@dataclass
+class AlgoBenchmark:
+    domain: str
+    """The problem domain. Usually a high-level ReCirq module."""


done 9481205

mpharrigan · 2021-12-04T00:23:36Z

recirq/algo_benchmark_library.py

+@dataclass
+class AlgoBenchmark:
+    domain: str
+    """The problem domain. Usually a high-level ReCirq module."""


done 7c7faa1

mpharrigan · 2021-12-04T00:26:30Z

recirq/algo_benchmark_library.py

+    data_class: Type
+    """The class which can contain ETL-ed data for this AlgoBenchmark."""
+
+    gen_func: Callable[[...], QuantumExecutableGroup]


done 6f9224c

mpharrigan · 2021-12-04T00:30:21Z

recirq/algo_benchmark_library.py

+    """A globally unique name for this config."""
+
+    gen_script: str
+    """The script filename that generates the QuantumExecutableGroup for this Config."""


done 69b6b65

mpharrigan · 2021-12-04T00:31:54Z

recirq/algo_benchmark_library.py

+
+
+@lru_cache()
+def get_all_algo_configs() -> List[Tuple[AlgoBenchmark, BenchmarkConfig]]:


done 8f139ec

mpharrigan · 2021-12-04T00:33:40Z

recirq/algo_benchmark_library.py

+
+
+def main():
+    df = pd.DataFrame([algo.as_strings() for algo in BENCHMARKS]).set_index('executable_family')


sortof forgot I left this in here. Changed the name of this function and removed the __main__ entry point 1514b20

The nice thing about pandas is you can use the same technique in a juypter noteook and get a nice html table that you can paste into sheets or a doc

mpharrigan · 2021-12-04T00:39:06Z

recirq/algo_benchmark_library_test.py

+@pytest.mark.parametrize('algo', BENCHMARKS)
+def test_executable_family_is_formulaic(algo):
+    # Check consistency in the AlgoBenchmark dataclass:
+    assert algo.executable_family == algo.spec_class.executable_family


done 3308c5e

dstrain115

LGTM once the tests resolve.

dstrain115 · 2021-12-05T12:48:54Z

recirq/algorithmic_benchmark_library.py

+
+import pandas as pd
+
+from cirq_google.workflow import ExecutableSpec, QuantumExecutableGroup


From the pytest:

ModuleNotFoundError: No module named 'cirq_google.workflow'

Is this waiting on another PR?

This and other forthcoming code relies on the currently-unreleased cirq_google.workflow module. I'll have to figure out a way to make this work in the mean time.

release openfermion that doesn't pin to a particular cirq version

enable testing against multiple versions of cirq including prerelease in ReCirq CI

put import guards and conditional testing so these tests only run if a recent enough cirq version is available

Following #237 , this is now implemented. The tests get run on the current and next config. They were failing on cirq 0.12. Looks like the executable spec stuff got included in v0.13 but future functionality may only be tested on next until the relevant cirq release is cut.

Change-Id: I1ae995491202c2b2f00c1614cb3c5623080ac685

Change-Id: I55a4e61783f337c7220f85b36fef61391138e0d9

Change-Id: I350ce16324e64d0c1e8d14d56737ea090b3fc039

Algo Benchmark Library card catalog

a8c0cf7

Change-Id: I4ca692cd92ddbee95d0d2aeed5a4027fcd0f2aa4

mpharrigan added the devops label Dec 3, 2021

google-cla bot added the cla: yes label Dec 3, 2021

mpharrigan requested review from dabacon, dstrain115, MichaelBroughton and tanujkhattar December 3, 2021 00:35

dstrain115 reviewed Dec 3, 2021

View reviewed changes

mpharrigan added 9 commits December 3, 2021 16:17

Algorithmic benchmark - no abbrev

77e48f9

Change-Id: Ic95e07f4690f4f10bf2634eda7a66ac81649774a

Docstring format

9481205

Change-Id: Ic1213396a235f6e40e46d2f3472d8380bb653719

domain constraints

7c7faa1

Change-Id: I97e19c6544def983cadde8c959fbc83b5d9f867e

gen_func -> executable_generator_func

6f9224c

Change-Id: I7c23fa8eac6d0b04cc7a480ac6a81cf4cb29f308

docstring format

d267c9e

Change-Id: I21a65e3e126c42bb227582d335d24a3fca287015

document gen_script formula

69b6b65

Change-Id: Ic393c5bb90b6df5b50b9a70ae0b3386d43a89e6f

docstring get_all_algo_configs

8f139ec

Change-Id: I95696873269c5e66dcbffedce7814c4488ac6d47

print_table_of_benchmarks

1514b20

Change-Id: I04cd53462b7190e071cd6ec680aaec37e08e7064

assert messages

3308c5e

Change-Id: I7a78669ee18d47a4e909783663c2da210d1db699

mpharrigan commented Dec 4, 2021

View reviewed changes

mpharrigan mentioned this pull request Dec 4, 2021

loschmidt.tilted_square_lattice Benchmark #236

Merged

mpharrigan added the ftb/bl label Dec 4, 2021

dstrain115 approved these changes Dec 5, 2021

View reviewed changes

mpharrigan added 3 commits December 7, 2021 14:17

Merge remote-tracking branch 'github/master' into 2021-12-algo-library

9ee6547

Change-Id: I1ae995491202c2b2f00c1614cb3c5623080ac685

import failsafe

bf0ee59

Change-Id: I55a4e61783f337c7220f85b36fef61391138e0d9

fixup skip

4592ab3

Change-Id: I350ce16324e64d0c1e8d14d56737ea090b3fc039

mpharrigan merged commit fee620b into master Dec 8, 2021

mpharrigan deleted the 2021-12-algo-library branch December 8, 2021 18:27



		@lru_cache()
		def get_all_algo_configs() -> List[Tuple[AlgoBenchmark, BenchmarkConfig]]:



		def main():
		df = pd.DataFrame([algo.as_strings() for algo in BENCHMARKS]).set_index('executable_family')


		import pandas as pd

		from cirq_google.workflow import ExecutableSpec, QuantumExecutableGroup

Algo Benchmark Library card catalog #235

Algo Benchmark Library card catalog #235

Uh oh!

Conversation

mpharrigan commented Dec 3, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpharrigan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dstrain115 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpharrigan Dec 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mpharrigan Dec 7, 2021 •

edited

Loading