didChangeWatchedFiles does not work recursively, LSP server goes out of sync for new directories

[stapelberg]

Steps to reproduce:

1. I compiled Emacs at git revision 6a9f1b504a5a3c096afd17f3b8f8ebeca6a03ed5
2. Then I started emacs -q ~/repro-emacs/hello1 (a directory)
3. (Only relevant for diagnosis) I switched to the scratch buffer and evaluated the following LISP code:

   (defun my-file-notify-add-watch-logger (file event &rest args)
     "log a message whenever file-notify-add-watch is called"
     (message "file-notify-add-watch called with file: %s, event: %s" file event))
   
   (advice-add 'file-notify-add-watch :before #'my-file-notify-add-watch-logger)

4. Open hello1.go
5. M-x eglot gopls RET
6. In a terminal, I run touch extra.go and I see a didChangeWatchedFiles event in the *EGLOT …* buffer — good!
7. In a terminal, I run mkdir newinternal, then touch newinternal/extra.go and I see no event — this is unexpected.
8. In the *Messages* buffer, I see:

file-notify-add-watch called with file: /home/stapelberg/repro-emacs/hello1/, event: (change)
file-notify-add-watch called with file: /home/stapelberg/repro-emacs/hello1/internal/, event: (change)

9. Notably, when creating the newinternal directory from my terminal, I see no file-notify-add-watch. I conclude that Eglot does not re-evaluate recursive glob patterns when new directories are created.
   • The Emacs “File Notifications” documentation states:

     If file is a directory, change watches for file creation and deletion in that directory. Some of the native file notification libraries also report file changes in that case. This does not work recursively.

   • …which I think is the reason why Eglot expands recursive glob patterns to directories in the first place.

[joaotavora]

and I see no event

Showing fs events to the user is not Eglot's core business. It's an implementation detail that some servers need to do certain things. Can you explain what the problem to the user really is, from a non-Elisp perspective? Also, where is the event log for the short experiment?

[stapelberg]

The problem to the user is that LSP functionality like organizeImports for Go code does not work correctly, as the language server (gopls in this case) is not aware of the newly created files (newly created Go packages).

Specifically, after adding the foo.Bar() function call to my func main(), I use eglot-code-action-organize-imports to add the project/path/to/foo import to my code.

When foo refers to a package I just created, gopls is unable to locate the package unless Eglot correctly tells gopls about files that were added.

Here is the EGLOT buffer log for the steps to reproduce provided above:

[jsonrpc] D[10:31:41.504] Running language server: gopls
[jsonrpc] e[10:31:41.524] --> initialize[1] {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"processId":3079337,"clientInfo":{"name":"Eglot","version":"1.17"},"rootPath":"/usr/local/google/home/stapelberg/repro-emacs/hello1/","rootUri":"file:///usr/local/google/home/stapelberg/repro-emacs/hello1","initializationOptions":{},"capabilities":{"workspace":{"applyEdit":true,"executeCommand":{"dynamicRegistration":false},"workspaceEdit":{"documentChanges":true},"didChangeWatchedFiles":{"dynamicRegistration":true},"symbol":{"dynamicRegistration":false},"configuration":true,"workspaceFolders":true},"textDocument":{"synchronization":{"dynamicRegistration":false,"willSave":true,"willSaveWaitUntil":true,"didSave":true},"completion":{"dynamicRegistration":false,"completionItem":{"snippetSupport":false,"deprecatedSupport":true,"resolveSupport":{"properties":["documentation","details","additionalTextEdits"]},"tagSupport":{"valueSet":[1]}},"contextSupport":true},"hover":{"dynamicRegistration":false,"contentFormat":["plaintext"]},"signatureHelp":{"dynamicRegistration":false,"signatureInformation":{"parameterInformation":{"labelOffsetSupport":true},"documentationFormat":["plaintext"],"activeParameterSupport":true}},"references":{"dynamicRegistration":false},"definition":{"dynamicRegistration":false,"linkSupport":true},"declaration":{"dynamicRegistration":false,"linkSupport":true},"implementation":{"dynamicRegistration":false,"linkSupport":true},"typeDefinition":{"dynamicRegistration":false,"linkSupport":true},"documentSymbol":{"dynamicRegistration":false,"hierarchicalDocumentSymbolSupport":true,"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]}},"documentHighlight":{"dynamicRegistration":false},"codeAction":{"dynamicRegistration":false,"resolveSupport":{"properties":["edit","command"]},"dataSupport":true,"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"isPreferredSupport":true},"formatting":{"dynamicRegistration":false},"rangeFormatting":{"dynamicRegistration":false},"rename":{"dynamicRegistration":false},"inlayHint":{"dynamicRegistration":false},"publishDiagnostics":{"relatedInformation":false,"codeDescriptionSupport":false,"tagSupport":{"valueSet":[1,2]}}},"window":{"showDocument":{"support":true},"workDoneProgress":true},"general":{"positionEncodings":["utf-32","utf-8","utf-16"]},"experimental":{}},"workspaceFolders":[{"uri":"file:///usr/local/google/home/stapelberg/repro-emacs/hello1","name":"~/repro-emacs/hello1/"}]}}
[jsonrpc] e[10:31:41.571] <-- initialize[1] {"jsonrpc":"2.0","result":{"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"save":{}},"completionProvider":{"triggerCharacters":["."]},"hoverProvider":true,"signatureHelpProvider":{"triggerCharacters":["(",","]},"definitionProvider":true,"typeDefinitionProvider":true,"implementationProvider":true,"referencesProvider":true,"documentHighlightProvider":true,"documentSymbolProvider":true,"codeActionProvider":{"codeActionKinds":["quickfix","refactor.extract","refactor.inline","refactor.rewrite","source.fixAll","source.organizeImports"],"resolveProvider":true},"codeLensProvider":{},"documentLinkProvider":{},"workspaceSymbolProvider":true,"documentFormattingProvider":true,"renameProvider":true,"foldingRangeProvider":true,"selectionRangeProvider":true,"executeCommandProvider":{"commands":["gopls.add_dependency","gopls.add_import","gopls.add_telemetry_counters","gopls.apply_fix","gopls.change_signature","gopls.check_upgrades","gopls.diagnose_files","gopls.edit_go_directive","gopls.fetch_vulncheck_result","gopls.gc_details","gopls.generate","gopls.go_get_package","gopls.list_imports","gopls.list_known_packages","gopls.maybe_prompt_for_telemetry","gopls.mem_stats","gopls.regenerate_cgo","gopls.remove_dependency","gopls.reset_go_mod_diagnostics","gopls.run_go_work_command","gopls.run_govulncheck","gopls.run_tests","gopls.start_debugging","gopls.start_profile","gopls.stop_profile","gopls.test","gopls.tidy","gopls.toggle_gc_details","gopls.update_go_sum","gopls.upgrade_dependency","gopls.vendor","gopls.views","gopls.workspace_stats"]},"callHierarchyProvider":true,"semanticTokensProvider":{"legend":{"tokenTypes":[],"tokenModifiers":[]},"range":true,"full":true},"inlayHintProvider":{},"workspace":{"workspaceFolders":{"supported":true,"changeNotifications":"workspace/didChangeWorkspaceFolders"}}},"serverInfo":{"name":"gopls","version":"{\"GoVersion\":\"go1.23-20240317-RC00 cl/616607620 +0a6f05e30f X:fieldtrack,boringcrypto\",\"Path\":\"golang.org/x/tools/gopls\",\"Main\":{\"Path\":\"golang.org/x/tools/gopls\",\"Version\":\"v0.15.2\",\"Sum\":\"h1:4JKt4inO8JaFW3l/Fh9X1k/5JQn+iUOpdc4/Lpi0mOs=\",\"Replace\":null},\"Deps\":[{\"Path\":\"github.com/BurntSushi/toml\",\"Version\":\"v1.2.1\",\"Sum\":\"h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=\",\"Replace\":null},{\"Path\":\"github.com/google/go-cmp\",\"Version\":\"v0.6.0\",\"Sum\":\"h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=\",\"Replace\":null},{\"Path\":\"golang.org/x/exp/typeparams\",\"Version\":\"v0.0.0-20221212164502-fae10dda9338\",\"Sum\":\"h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y=\",\"Replace\":null},{\"Path\":\"golang.org/x/mod\",\"Version\":\"v0.15.0\",\"Sum\":\"h1:SernR4v+D55NyBH2QiEQrlBAnj1ECL6AGrA5+dPaMY8=\",\"Replace\":null},{\"Path\":\"golang.org/x/sync\",\"Version\":\"v0.6.0\",\"Sum\":\"h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ=\",\"Replace\":null},{\"Path\":\"golang.org/x/telemetry\",\"Version\":\"v0.0.0-20240209200032-7b892fcb8a78\",\"Sum\":\"h1:vcVnuftN4J4UKLRcgetjzfU9FjjgXUUYUc3JhFplgV4=\",\"Replace\":null},{\"Path\":\"golang.org/x/text\",\"Version\":\"v0.14.0\",\"Sum\":\"h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=\",\"Replace\":null},{\"Path\":\"golang.org/x/tools\",\"Version\":\"v0.18.1-0.20240311201521-78fbdeb61842\",\"Sum\":\"h1:No0LMXYFkp3j4oEsPdtY8LUQz33gu79Rm9DE+izMeGQ=\",\"Replace\":null},{\"Path\":\"golang.org/x/vuln\",\"Version\":\"v1.0.1\",\"Sum\":\"h1:KUas02EjQK5LTuIx1OylBQdKKZ9jeugs+HiqO5HormU=\",\"Replace\":null},{\"Path\":\"honnef.co/go/tools\",\"Version\":\"v0.4.6\",\"Sum\":\"h1:oFEHCKeID7to/3autwsWfnuv69j3NsfcXbvJKuIcep8=\",\"Replace\":null},{\"Path\":\"mvdan.cc/gofumpt\",\"Version\":\"v0.6.0\",\"Sum\":\"h1:G3QvahNDmpD+Aek/bNOLrFR2XC6ZAdo62dZu65gmwGo=\",\"Replace\":null},{\"Path\":\"mvdan.cc/xurls/v2\",\"Version\":\"v2.5.0\",\"Sum\":\"h1:lyBNOm8Wo71UknhUs4QTFUNNMyxy2JEIaKKo0RWOh+8=\",\"Replace\":null}],\"Settings\":[{\"Key\":\"-buildmode\",\"Value\":\"exe\"},{\"Key\":\"-compiler\",\"Value\":\"gc\"},{\"Key\":\"DefaultGODEBUG\",\"Value\":\"asynctimerchan=1,httplaxcontentlength=1,httpmuxgo121=1,panicnil=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1,winreadlinkvolume=0,winsymlink=0\"},{\"Key\":\"CGO_ENABLED\",\"Value\":\"1\"},{\"Key\":\"CGO_CFLAGS\",\"Value\":\"\"},{\"Key\":\"CGO_CPPFLAGS\",\"Value\":\"\"},{\"Key\":\"CGO_CXXFLAGS\",\"Value\":\"\"},{\"Key\":\"CGO_LDFLAGS\",\"Value\":\"\"},{\"Key\":\"GOARCH\",\"Value\":\"amd64\"},{\"Key\":\"GOEXPERIMENT\",\"Value\":\"fieldtrack,boringcrypto\"},{\"Key\":\"GOOS\",\"Value\":\"linux\"},{\"Key\":\"GOAMD64\",\"Value\":\"v1\"}],\"Version\":\"v0.15.2\"}"}},"id":1}
[jsonrpc] e[10:31:41.571] --> initialized {"jsonrpc":"2.0","method":"initialized","params":{}}
[jsonrpc] e[10:31:41.572] --> workspace/didChangeConfiguration {"jsonrpc":"2.0","method":"workspace/didChangeConfiguration","params":{"settings":{}}}
[jsonrpc] e[10:31:41.578] <-- window/workDoneProgress/create[1] {"jsonrpc":"2.0","method":"window/workDoneProgress/create","params":{"token":"3597244683180143804"},"id":1}
[jsonrpc] e[10:31:41.578] --> window/workDoneProgress/create[1] {"jsonrpc":"2.0","id":1,"result":null}
[jsonrpc] e[10:31:41.579] <-- $/progress {"jsonrpc":"2.0","method":"$/progress","params":{"token":"3597244683180143804","value":{"kind":"begin","title":"Setting up workspace","message":"Loading packages..."}}}
[jsonrpc] e[10:31:41.579] <-- workspace/configuration[2] {"jsonrpc":"2.0","method":"workspace/configuration","params":{"items":[{"scopeUri":"file:///usr/local/google/home/stapelberg/repro-emacs/hello1","section":"gopls"}]},"id":2}
[jsonrpc] e[10:31:41.579] --> workspace/configuration[2] {"jsonrpc":"2.0","id":2,"result":[null]}
[jsonrpc] e[10:31:41.599] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:31:41 go info for /usr/local/google/home/stapelberg/repro-emacs/hello1\n(view type GoPackagesDriverView)\n(root dir /usr/local/google/home/stapelberg/repro-emacs/hello1)\n(go version go version go1.23-20240419-RC02 cl/626470163 +7f76c00fc5 X:fieldtrack,boringcrypto linux/amd64)\n(build flags: [])\n(go env: {GOOS:linux GOARCH:amd64 GOCACHE:/usr/local/google/home/stapelberg/.cache/go-build GOMODCACHE:/usr/local/google/home/stapelberg/go/pkg/mod GOPATH:/usr/local/google/home/stapelberg/go GOPRIVATE: GOFLAGS: GO111MODULE: GoVersion:23 GoVersionOutput:go version go1.23-20240419-RC02 cl/626470163 +7f76c00fc5 X:fieldtrack,boringcrypto linux/amd64\n GOWORK: GOPACKAGESDRIVER:/usr/bin/gopackagesdriver})\n(env overlay: map[])\n\n"}}
[jsonrpc] e[10:31:42.151] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:31:42 go/packages.Load #1\n\tsnapshot=0\n\tdirectory=file:///usr/local/google/home/stapelberg/repro-emacs/hello1\n\tquery=[./... builtin]\n\tpackages=2\n"}}
[jsonrpc] e[10:31:42.153] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:31:42 go/packages.Load #1: updating metadata for 50 packages\n"}}
[jsonrpc] e[10:31:42.174] <-- $/progress {"jsonrpc":"2.0","method":"$/progress","params":{"token":"3597244683180143804","value":{"kind":"end","message":"Finished loading packages."}}}
[jsonrpc] e[10:31:42.174] <-- client/registerCapability[3] {"jsonrpc":"2.0","method":"client/registerCapability","params":{"registrations":[{"id":"workspace/didChangeWatchedFiles-0","method":"workspace/didChangeWatchedFiles","registerOptions":{"watchers":[{"globPattern":"**/*.{mod,work}","kind":7},{"globPattern":"**/*.{go,mod,sum,work}","kind":7}]}}]},"id":3}
[jsonrpc] e[10:31:42.206] --> client/registerCapability[3] {"jsonrpc":"2.0","id":3,"result":null}
[jsonrpc] e[10:31:42.211] <-- workspace/configuration[4] {"jsonrpc":"2.0","method":"workspace/configuration","params":{"items":[{"section":"gopls"}]},"id":4}
[jsonrpc] e[10:31:42.211] --> workspace/configuration[4] {"jsonrpc":"2.0","id":4,"result":[null]}
[jsonrpc] e[10:31:42.232] <-- workspace/configuration[5] {"jsonrpc":"2.0","method":"workspace/configuration","params":{"items":[{"scopeUri":"file:///usr/local/google/home/stapelberg/repro-emacs/hello1","section":"gopls"}]},"id":5}
[jsonrpc] e[10:31:42.232] --> workspace/configuration[5] {"jsonrpc":"2.0","id":5,"result":[null]}
[jsonrpc] e[10:31:42.252] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:31:42 go info for /usr/local/google/home/stapelberg/repro-emacs/hello1\n(view type GoPackagesDriverView)\n(root dir /usr/local/google/home/stapelberg/repro-emacs/hello1)\n(go version go version go1.23-20240419-RC02 cl/626470163 +7f76c00fc5 X:fieldtrack,boringcrypto linux/amd64)\n(build flags: [])\n(go env: {GOOS:linux GOARCH:amd64 GOCACHE:/usr/local/google/home/stapelberg/.cache/go-build GOMODCACHE:/usr/local/google/home/stapelberg/go/pkg/mod GOPATH:/usr/local/google/home/stapelberg/go GOPRIVATE: GOFLAGS: GO111MODULE: GoVersion:23 GoVersionOutput:go version go1.23-20240419-RC02 cl/626470163 +7f76c00fc5 X:fieldtrack,boringcrypto linux/amd64\n GOWORK: GOPACKAGESDRIVER:/usr/bin/gopackagesdriver})\n(env overlay: map[])\n\n"}}
[jsonrpc] e[10:31:42.800] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:31:42 go/packages.Load #2\n\tsnapshot=0\n\tdirectory=file:///usr/local/google/home/stapelberg/repro-emacs/hello1\n\tquery=[./... builtin]\n\tpackages=2\n"}}
[jsonrpc] e[10:31:42.802] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:31:42 go/packages.Load #2: updating metadata for 50 packages\n"}}
[jsonrpc] e[10:32:00.756] --> workspace/didChangeWatchedFiles {"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":[{"uri":"file:///usr/local/google/home/stapelberg/repro-emacs/hello1/extra.go","type":1}]}}
[jsonrpc] e[10:32:02.328] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:32:02 go/packages.Load #3\n\tsnapshot=1\n\tdirectory=file:///usr/local/google/home/stapelberg/repro-emacs/hello1\n\tquery=[hello]\n\tpackages=1\n"}}
[jsonrpc] e[10:32:02.329] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:32:02 go/packages.Load #3\n\tsnapshot=1\n\tdirectory=file:///usr/local/google/home/stapelberg/repro-emacs/hello1\n\tpackage=\"hello\"\n\tfiles=[/usr/local/google/home/stapelberg/repro-emacs/hello1/extra.go /usr/local/google/home/stapelberg/repro-emacs/hello1/hello1.go]\n"}}
[jsonrpc] e[10:32:02.330] <-- window/logMessage {"jsonrpc":"2.0","method":"window/logMessage","params":{"type":3,"message":"2024/06/17 10:32:02 go/packages.Load #3: updating metadata for 1 packages\n"}}
[jsonrpc] e[10:32:02.351] <-- textDocument/publishDiagnostics {"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"uri":"file:///usr/local/google/home/stapelberg/repro-emacs/hello1/extra.go","diagnostics":[{"range":{"start":{"line":0,"character":0},"end":{"line":0,"character":0}},"severity":1,"source":"syntax","message":"expected ';', found 'EOF'"}]}}

I hope this makes more sense now, let me know if you need additional details.

[joaotavora]

I hope this makes more sense now, let me know if you need additional details.

Hmmm, no. I'm more confused now :-) What has organize imports have to do with creating directories? Is creating these directories and files within these directories a part of what the server does when being asked to organize imports? if so, why does it need to be notified about the files it itself is creating?

[joaotavora]

Can you give me a recipe that doesn't require me to add Elisp advice to things and where I as a Go newbie that just has Emacs with Eglot would see the unexpected erroring behavior? Start from some kind of hello world example and describe the steps I should take, like type this here, click there, etc, please.