openssl/v2 module panic: openssl: OpenSSL version: 0.0.0 with formerly working libcrypto.so version

[jfkw]

On SUSE Linux Enterprise Server running in FIPS mode, Go binaries built with go1.22 and the openssl/v2 module panic with panic: openssl: OpenSSL version: 0.0.0. Go binaries built with go1.21 or earlier and the original openssl module work as expected and dlopen() the appropriate libcrypto.so.

It would help with troubleshooting dlopen() issues if panic(errUnsupportedVersion()) returned additional state information.

Expected Behavior

All examples below are running in FIPS mode:

# sysctl -a | grep fips_enabled
crypto.fips_enabled = 1

Using a toolchain built at tag go1.21.13-1-openssl-fips (or any earlier tag in the go1.21 series or previous go1.x series) with module github.com/golang-fips/openssl, the produced Go application binary runs as expected:

# go build -gcflags="all=-N -l" https-server.go
# ./https-server
Starting server on :4000

Tracing confirms that this binary dlopens /usr/lib64/libcrypto.so.3 -> libcrypto.so.3.1.4. On an earlier distro version, dlopen succeeds with /usr/lib64/libcrypto.so -> /lib64/libcrypto.so.1.0.0.

Demonstration of Problem Behavior

Using a toolchain built at tag go1.22.5-3-openssl-fips (or any earlier tag in the go1.22 series) with the new module github.com/golang-fips/openssl/v2, the produced Go application binary panics with panic: openssl: OpenSSL version: 0.0.0.

# go build -gcflags="all=-N -l" https-server.go
# ./https-server
panic: openssl: OpenSSL version: 0.0.0

goroutine 1 [running]:
vendor/github.com/golang-fips/openssl/v2.FIPS()
        /usr/lib64/go/1.22-openssl/src/vendor/github.com/golang-fips/openssl/v2/openssl.go:110 +0x125
crypto/internal/backend.init.0()
        /usr/lib64/go/1.22-openssl/src/crypto/internal/backend/openssl.go:22 +0xf

The code path raising the panic() during fall-through to the default case:

// FIPS returns true if OpenSSL is running in FIPS mode, else returns false.
func FIPS() bool {
	switch vMajor {
	case 1:
		return C.go_openssl_FIPS_mode() == 1
	case 3:
		// If FIPS is not enabled via default properties, then we are sure FIPS is not used.
		if C.go_openssl_EVP_default_properties_is_fips_enabled(nil) == 0 {
			return false
		}
		// EVP_default_properties_is_fips_enabled can return true even if the FIPS provider isn't loaded,
		// it is only based on the default properties.
		// We can be sure that the FIPS provider is available if we can fetch an algorithm, e.g., SHA2-256,
		// explicitly setting `fips=yes`.
		return C.go_openssl_OSSL_PROVIDER_available(nil, providerNameFips) == 1
	default:
		panic(errUnsupportedVersion())
	}
}

It is not yet apparent what is causing the openssl/v2 fall-through to the default case when loading these specific libcrypto.so(s). The 0.0.0 designation from the panic leaves open the question of whether any libcrypto.sois being found in this case. The other possibility is that a libcrypto.so is found but its signatures are incompatible with certain checks.

I will describe ongoing runtime debug troubleshooting efforts using the delve debugger in subsequent comments.

[karianna]

@gdams - Might be able to triage this on a SUSE host on Azure?

[dagood]

tag go1.21.13-1-openssl-fips

To clarify for my fellow microsoft/go folks listening in, this refers to https://github.com/golang-fips/go/tree/go1.21.13-1-openssl-fips, which we aren't using.

Observing 0.0.0 in the call to openssl.FIPS() is odd--I wouldn't think a properly initialized openssl would be able to get that far. It should have already returned an error from inside openssl.Init, here:

openssl/init.go

Lines 33 to 43 in 97b3bd9

	major, minor, patch = uint(imajor), uint(iminor), uint(ipatch)
	var supported bool
	if major == 1 {
	supported = minor == 0 || minor == 1
	} else if major == 3 {
	// OpenSSL guarantees API and ABI compatibility within the same major version since OpenSSL 3.
	supported = true
	}
	if !supported {
	return 0, 0, 0, errUnsupportedVersion()
	}

Another possibility I'd add to the pile is that for some reason, openssl.FIPS is being called before openssl.Init is. Or, the caller of openssl.Init isn't processing/emitting the error correctly. Maybe parallel calls are involved? (Although openssl.Init itself does use a sync.Once.)

I would expect that if you modify the src/vendor directory in your Go installation, you can add more diagnostic code yourself that will take effect in the next build. (Not that I think more diagnostic code would be a bad idea in golang-fips/openssl, it's just hard to know what to add without working directly on a repro.)

[qmuntal]

@jfkw I would recommend transferring this issue to https://github.com/golang-fips/go. As @dagood explained, it looks like an issue integrating golang-fips/openssl into the Go standard library rather than an OpenSSL interop issue.

[jfkw]

@qmuntal. Thanks, will do. I can refile the issue there and close here. Alternatively, someone with write access to the repository could use the UI action to transfer the issue which would preserve the helpful comments. (edit: transfer of the issue would be preferable, I'll hold off on refiling)

[jfkw]

Closing as invalid. On my build one or more of the higher-numbered patches were not being applied. Ensuring that all patches are applied resolves the panic and incorrect OpenSSL version detection. Thanks everyone for the advice given here.