io/fs: add ReadLinkFS interface

[zombiezen]

What version of Go are you using (go version)?

$ go version
go version go1.17.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

Walked a directory with fs.WalkDir and encountered a symlink that I wanted to read.

What did you expect to see?

A function fs.ReadLink that behaves like os.Readlink, but operates on an fs.FS. Design sketch:

package fs

// ReadLink returns the destination of the named symbolic link.
//
// If fsys does not implement ReadLinkFS, then ReadLink returns an error.
func ReadLink(fsys FS, name string) (string, error)

// ReadLinkFS is the interface implemented by a file system that supports symbolic links.
type ReadLinkFS interface {
  FS

  // ReadLink returns the destination of the named symbolic link.
  ReadLink(name string) (string, error)
}

I would also want the file system returned by os.DirFS to have an implementation that calls os.Readlink. IIUC archive/zip.Reader would probably also benefit from an implementation.

An open question in my mind is whether the returned destination should be a slash-separated path or kept as-is. I think for consistency it probably should convert to a slash-separated path, but I'm not sure if this has problems on Windows.

What did you see instead?

No such API exists.

Other details

I have bandwidth to contribute an implementation of this, but I understand we're in the freeze and the earliest this could go in is Go 1.19.

This is somewhat related to #45470, but I'm not proposing changing any existing semantics, just adding a new method.

[bcmills]

An open question in my mind is whether the returned destination should be a slash-separated path or kept as-is.

I would say it should be slash-separated and also relative to the same FS: links to absolute paths should be made relative, and links to paths above the FS root (or on an entirely different volume) should be rejected.

[rsc]

Rewriting the link is tricky and not rewriting the link is also tricky. It's unclear to me what we should do here, if anything.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[zombiezen]

As a data point, for the application I was writing, rewriting the link to be relative is effectively what I did anyway. I wanted to create a zip archive of an on-disk directory, so absolute paths were rewritten to be relative to the DirFS root. Returning an error if the path could not be represented was acceptable.

[rsc]

So the API for ReadLink is that it must return a path relative to the link, and if it can't do that, then it returns an error?
It seems like most symlinks are absolute, though, and most DirFS(foo) will not use foo = "/", so that will make most symlinks result in errors?

I'm trying to understand how useful this will be in practice, to balance against the cost. Will it be useful in practice? Or will people just be frustrated that 99% of symlinks aren't usable?

[bcmills]

So the API for ReadLink is that it must return a path relative to the link, and if it can't do that, then it returns an error?

Yes.

It seems like most symlinks are absolute, though, and most DirFS(foo) will not use foo = "/", so that will make most symlinks result in errors?

I would expect the ReadLink method to internally transform absolute link paths to relative ones (for example, by using filepath.Rel and then checking that the returned path does not begin with ../.) That would allow symlinks to succeed when they are to other locations under the FS root — they would only fail if they are absolute symlinks that would otherwise refer to locations completely outside the FS tree.

So, for example, os.DirFS("/") on Unix would be able to resolve symlinks anywhere on the filesystem, even if those symlinks are absolute.

[hherman1]

I would expect the ReadLink method to internally transform absolute link paths to relative ones (for example, by using filepath.Rel and then checking that the returned path does not begin with ../.) That would allow symlinks to succeed when they are to other locations under the FS root — they would only fail if they are absolute symlinks that would otherwise refer to locations completely outside the FS tree.

Does this extend to non local-disk filesystems?

[zombiezen]

@rsc:

So the API for ReadLink is that it must return a path relative to the link, and if it can't do that, then it returns an error?

That was sufficient for my application. The root of my DirFS could contain multiple projects (think a GOPATH-like setup), so symlinks would usually resolve within the io/fs filesystem.

It seems like most symlinks are absolute, though, [...]

I'm not 100% convinced of that, but I don't have evidence to dispute your claim.

FWIW the cases that I was concerned with in a DevTools context:

• Symlinks within the same directory. A common case is multiple aliases for a single on-disk program, like Busybox.
• IIRC old versions of Git would use symlinks to represent symbolic references like HEAD instead of a file.

I'm trying to understand how useful this will be in practice, to balance against the cost. Will it be useful in practice? Or will people just be frustrated that 99% of symlinks aren't usable?

Agreed, I think weighing this tradeoff is the trickiest part of this proposal.

Relative links are IMO the most useful for a consumer of this API. I could imagine an implementation of ReadLink also returning absolute paths in the case where the returned path is above the root io/fs directory, but this might add complexity when the paths aren't slash-separated. However, I tend to agree with @bcmills that being strict about this is probably better.

You're probably already considering this, but it's just FS.ReadLink that would be picky about the link target. The implicit interface of DirFS.Open is to follow symlinks (again, #45470 tracks spelling out that behavior for other filesystems), and symlinks are already visible in directory listings.

---

@hherman1:

Does this extend to non local-disk filesystems?

I'm proposing the slash-separated relative path restriction would extend to non-local-disk filesystems, yes. How each implementation meets this contract is up to the individual filesystem.

[rsc]

@zombiezen, thanks for the use cases. It seems fairly unobtrusive to add ReadLinkFS and fs.ReadLink, so the cost seems low and the benefit > 0.

@bcmills:

I would expect the ReadLink method to internally transform absolute link paths to relative ones (for example, by using filepath.Rel and then checking that the returned path does not begin with ../.) That would allow symlinks to succeed when they are to other locations under the FS root — they would only fail if they are absolute symlinks that would otherwise refer to locations completely outside the FS tree.

So, for example, os.DirFS("/") on Unix would be able to resolve symlinks anywhere on the filesystem, even if those symlinks are absolute.

Rewriting symlinks may run into problems. I've been burned enough that I'm a bit wary about that. Should we start with just erroring out on the absolute ones?

[bcmills]

I think it would be fine to start by erroring out on absolute links, and perhaps define a specific error (or error type) for symlinks that refer to locations outside of the FS.

From the perspective of fs.ReadLinkFS, the requirement would be that every returned path is relative to the passed-in name and below the FS root. (It would be up to the specific FS implementation to decide whether to achieve that by rewriting absolute links or rejecting them.)

[rsc]

OK, so it sounds like we agree on ReadLink and ReadLinkFS but with the restriction that the returned link must be relative, and absolute symlinks return errors instead. Do I have that right?

[zombiezen]

IIUC the full constraint for the returned link is relative and underneath the root of FS. Otherwise, yes, that matches my understanding.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group