copy directory contains symbolic to subdir to directory twice bigger

[pyfreyr]

src dir:

data/2025-05-16
├── sparse_embeddings -> model/sparse_embeddings
└── model/sparse_embeddings/

when use local fs to copy, the sparse_embeddings symbolic link will copy model/sparse_embeddings data duplicated. As a result, the data/2025-05-16 directory will be twice bigger than origin.

from fsspec.implementations.local import LocalFileSystem

fs = LocalFileSystem()
fs.copy("data/2025-05-16/", "models/2025-05-16/", recursive=True)

[martindurant]

cp() for localFS uses shutil.copyfile. You should be able to pass follow_symlinks=False to get the behaviour you are after.

[pyfreyr]

In my scene, I want to copy model to remote repo so must keep directory data the same as local dir, include soft link. shutil.copytree work well by setting symlinks true, but fsspec glob dir first, and then copyfile one by one, the symlink to dir become real dir and then data in symlink link to will be copied.

In the example above, symlink sparse_embeddings in 16 becomes real dir in 17 and model/sparse_embeddings data be copied into it, and model/sparse_embeddings dir copy to 17 after.

[martindurant]

In the example above, symlink sparse_embeddings in 16 becomes real dir in 17 and model/sparse_embeddings data be copied into it, and model/sparse_embeddings dir copy to 17 after.

Yes, I understand; did you try with the additional kwarg I suggested?

[pyfreyr]

I add follow_symlinks=False to copy or cp, symlink always become real dir