provide a mechanism for public verification of release binaries

[grenade]

currently the quilibrium node binary is built and released using a process which is not transparent or auditable by the general public. there is a binary signatory check at startup but as far as i am aware there is no mechanism whereby the public can establish a trust relationship with the signatories. as such there is no obvious mechanism whereby a node maintainer may verify that the release binary they install and run on their own hardware, is built from an unmodified version of the source code which is publicly accessible.
many open source blockchains and other open source tools which provide release binaries make use of github actions and the github release mechanism to provide an auditable trail of the build and release process whereby a node maintainer may review build steps in order to satisfy themselves that the release binary is derived from the same source code as the publicly accessible source. the ceremonyclient repository contains no such actions or releases and no notes or explanations are provided in the repository readme to explain how node binaries are built and released, so there is no obvious or transparent correlation between release binaries and source code.
node maintainers may work around this by building their own binaries and skipping signature checks but a first class solution by a trustworthy project which provides release binaries, should include an obvious mechanism by which those downloading release binaries may validate the build and release process of those binaries.
if i am mistaken and an auditable mechanism for release binary verification does exist, it would be helpful if a note about the same could be included in the main repository readme.

[CassOnMars]

I agree completely – there are some hurdles:

• we can't rely on github actions, github previously nuked our account without warning and relying on them is dangerous
• reproducible builds would be amazing! we use CGO/Rust FFI though, which to my understanding isn't verifiable

If you have suggestions or implementations on how to address those two specific hurdles, we would welcome it greatly

[grenade]

ok. i'll put some effort into a few pr's to try to tackle these hurdles if i can get a little more direction or input on what might be acceptable.

constraint 1: no gh actions

the no gh actions constraint will require decisions and input from those of you internal to the team because i'm just an outside observer here and if you want a build system that is not github, i will need direction about who you are happy to use, since somebody's computer has to perform the build. github's system has a few benefits that much of the ecosystem has come to trust and depend on and we'll need to find tooling that is not github but does provide these benefits:

• transparent: all of the fetch, build and upload logs should be publicly visible
• history: logs and artifacts should remain publicly available indefinitely
• availability: build artifacts and logs need to be hosted on a service whose uptime record is strong, capable and resistant to ddos or unforseen issues
• modern: some archaic systems exist (remember sourceforge?) but using anything like that will limit participation and contribution from the open source community
• trust: whoever provides these benefits should or must fit into one of the two categories below
  • a recognisable entity which has a history of providing reliable services (there are many, most cost money)
  • a trustless entity whose credentials can be audited and verified by anyone (there are few, i am aware of none)

it's forseeable and understandable that the solution to this constraint will be a financial decision.

1. do we want to spend money on a non-github solution?
2. do we want to find or build a distributed trustless software artifact building and hosting platform?
3. do we want to host all services on in-house open source tools and pay someone to maintain them?
4. do we want to remove the constraint and take advantage of github's free provision of all of the above for open source projects?

constraint 2: reproducible builds

afaik, and as is acknowledged above, it is not possible to create reproducible builds for the rust artifacts (see rust-lang/rust#34902 and rust-lang/rust#75362). as long as this is the case, including this as a constraint for auditable builds will mean we won't get auditable or observable builds until the rust compiler is fixed upstream. since we don't need reproducible builds in order to provide auditable or observable builds, i propose removing this as a constraint to making progress on the less challenging aspects of just being transparent, auditable and observable, as proposed.

---

note: i followed the events surrounding the nuking of the github repository when it happened. it was my understanding that gh took issue with release binaries being included in the repository rather than as release artifacts. there are good technical reasons why gh struggles with repos doing this and it is documented. it's understandable that it felt like no warning was given before the repo was nuked, especially considering the number of people relying on the repo when it happened, but it was an avoidable situation if artifact distribution had been handled as releases rather than being included in the repo source. gh releases have infrastructure built around them that is designed to handle the considerations of distributing large binary files whereas including binaries in repository sources do not. i propose that we take advantage of the github release mechanism as many thousands of other open source projects do.

[VishvaWang]

• we can't rely on github actions, github previously nuked our account without warning and relying on them is dangerous
• reproducible builds would be amazing! we use CGO/Rust FFI though, which to my understanding isn't verifiable

If you think GitHub and its automation tools are unreliable, you can try self-hosting version control and build tools.
I don't understand exactly what 'unverifiable' means in this context.
However, I believe that at minimum, teams should make their build steps as public as possible to help those who want to build from source, rather than simply stating 'Running production nodes from source is no longer recommended given build complexity. Please refer to our release information to obtain the latest version.'