"error reading server preface: EOF" when connecting to containerd socket on mac

[aran]

Description

When attempting to connect to the containerd socket via gRPC or containerd APIs, we encounter the error "error reading server preface: EOF". colima nerdctl works. The socket is owned by the same user running the test program.

Version

colima version && limactl --version && qemu-img --version
colima version 0.8.1
git commit: 96598cc5b64e5e9e1e64891642b91edc8ac49d16
limactl version 1.0.6
qemu-img version 9.2.2
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

colima version && limactl --version && qemu-img --version
colima version 0.8.1
git commit: 96598cc
limactl version 1.0.6
qemu-img version 9.2.2
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

Reproduction Steps

colima start -p contd --runtime containerd
export CONTAINERD_ADDRESS=/Users/me/.colima/contd/containerd.sock

Example program to exercise it, via raw gRPC or containerd API (commented-out):

package main

import (
        "context"
        "log"
        "os"

        // "github.com/containerd/containerd/v2/client"
        "github.com/containerd/containerd/v2/defaults"

        "google.golang.org/protobuf/types/known/emptypb"
        versionservice "github.com/containerd/containerd/api/services/version/v1"
        "google.golang.org/grpc"
        "google.golang.org/grpc/credentials/insecure"
)

func main() {
        address := os.Getenv("CONTAINERD_ADDRESS")
        if address == "" {
                address = defaults.DefaultAddress
        }
        log.Printf("Using containerd socket: %s", address)

        gopts := []grpc.DialOption{
                grpc.WithTransportCredentials(insecure.NewCredentials()),
        }

        conn, err := grpc.NewClient("unix://"+address, gopts...)
        if err != nil {
                log.Fatalf("Failed to create gRPC connection: %v", err)
        }
        defer conn.Close()

        client := versionservice.NewVersionClient(conn)
        v, err := client.Version(context.Background(), &emptypb.Empty{})
        if err != nil {
                log.Fatalf("Failed to get containerd version: %v", err)
        }
        log.Printf("Containerd version: %s", v.Version)

        // // Create a client with default namespace
        // containerdClient, err := client.New(
        //      address,
        // )
        // if err != nil {
        //      log.Fatalf("Failed to create containerd client: %v", err)
        // }

        // ctx := context.Background()

        // version, err := containerdClient.Version(ctx)
        // if err != nil {
        //      log.Fatalf("Failed to get containerd version: %v", err)
        // }

        // log.Printf("Containerd version: %s", version)

        // defer containerdClient.Close()
}

Expected behaviour

Test program should run successfully displaying the version of the containerd runtime.

Additional context

As far as I can tell, the traffic from the test program is being sent to the mac side socket, but it is not being forwarded to the linux side.

If I do:

colima -p contd ssh
    # (in the vm)
    sudo chmod a+rwx /run/containerd/
    sudo chmod a+rwx /run/containerd/containerd.sock
colima ssh-config -p contd > ssh.conf
ssh -F ssh.conf -L /tmp/containerd.sock:/run/containerd/containerd.sock colima-contd -N

The resulting socket at /tmp/containerd.sock works with the given test program.