SourceKit document structure/key.substructure missing Swift attribute information

[J-MR-T]

Motivation

To be able to access information about Swift attributes through SourceKit requests such as editor.open, its key.substructure or document structure part of the response should include more information on attributes.

Missing attribute arguments

Currently, only the basic type of attribute, and its offset and length are provided in the response.

E.g. for this function

@available(swift 5.10)
func foo(){}

the key.substructure response includes this

        {
          key.offset: 0,
          key.length: 22,
          key.attribute: source.decl.attribute.available
        }

which does not include an information about the arguments of the attribute.

(reproduce using this request in sourcekitd-repl)

{
  key.request: source.request.editor.open,
  key.name: "myname",
  key.enablesubstructure: 1,
  key.sourcetext: "@available(swift 5.10) func foo(){}"
}

Missing custom attribute information

For custom attributes, the name of the custom attribute isn't available, it's just shown as source.decl.attribute._custom.

Take this file:

@propertyWrapper
struct MyCustomAttribute{
    var wrappedValue: Int 
}

struct MyTopLevel {
    @MyCustomAttribute
    var i:Int
}  

The key.substructure response for the i property of the MyTopLevel struct includes

            {
              key.offset: 97,
              key.length: 18,
              key.attribute: source.decl.attribute._custom
            }

(reproduce using this request in sourcekitd-repl)

{
  key.request: source.request.editor.open,
  key.name: "myname",
  key.enablesubstructure: 1,
  key.sourcetext: "@propertyWrapper\nstruct MyCustomAttribute{\n    var wrappedValue: Int \n}\n\nstruct MyTopLevel {\n    @MyCustomAttribute\n    var i:Int\n}\n  \n"
}

Little distinction between attributes and declaration modifiers

Currently, attributes and declaration modifiers are not distinguished in the key.substructure response. They both live under the key.attributes list and their UIDs both contain .attribute..

E.g. for
@available(swift 5.10) public func foo(){}

This is contained in the key.substructure response:

      key.attributes: [
        {
          key.offset: 23,
          key.length: 6,
          key.attribute: source.decl.attribute.public
        },
        {
          key.offset: 0,
          key.length: 22,
          key.attribute: source.decl.attribute.available
        }
      ]

(reproduce using this request in sourcekitd-repl)

{
  key.request: source.request.editor.open,
  key.name: "myname",
  key.enablesubstructure: 1,
  key.sourcetext: "@available(swift 5.10) public func foo(){}"
}

Unlike the first two kinds of missing information in this proposal, this one can be inferred, but I still feel it would be cleaner to mark these distinct ways of annotating functions.

Proposed solution

Missing attribute arguments

To support attribute arguments, two things need to be considered:

1. How to structure a response about attribute arguments
2. How to identify attribute arguments themselves

How to structure a response about attribute arguments

• One option would be to have the key.attribute list include an attribute with the same key.attribute kind multiple times, but a different argument each time, resulting in something like this:

  key.attributes: [
      {
        key.offset: 0,
        key.length: 22,
        key.attribute: source.decl.attribute.available,
        key.argument: ...,
        key.argumentoffset: ...,
        key.argumentlength: ...
      },
      {
        key.offset: 0,
        key.length: 22,
        key.attribute: source.decl.attribute.available,
        key.argument: ...,
        key.argumentoffset: ...,
        key.argumentlength: ...
      }
    ]
  

  (this seems more consistent with the current implementation, but not very clean)
• My preferred alternative would be a separate key.arguments list attached to an attribute:

  key.attributes: [
      {
        key.offset: 0,
        key.length: 22,
        key.attribute: source.decl.attribute.available,
        key.arguments: [
          {
            key.argument: ...,
            key.offset: ...,
            key.length: ...
          },
          {
            key.argument: ...,
            key.offset: ...,
            key.length: ...
          }
        ]
      }
    ]
  

  In my opinion, this seems cleaner and easier to access.

How to identify attribute arguments themselves

I deliberately omitted the key.argument values above, because it would need to be decided how these are to be represented. Options include:

• strings, possibly with spaces in them
• a list of strings
• UIDs, like attributes themselves
• etc.

As I'm not very familiar with the Swift language itself and don't know all the cases and trade-offs here I would like to leave this fully open to discussion.

Missing custom attribute information

I think the cleanest solution would be to add a key.name or key.customattributename for attributes with key.attribute: source.decl.attribute._custom

            {
              key.offset: 97,
              key.length: 18,
              key.attribute: source.decl.attribute._custom,
              key.customattributename: "MyCustomAttribute"
            }

Little distinction between attributes and declaration modifiers

Here there are again a few options:

1. splitting the key.attributes list into a key.declarationmodifiers and key.attributes (breaking change)
2. adding some kind of distinction UID key, e.g. key.kind: attribute/key.kind: modifier/key.kind: declarationmodifier (or an entirely new key.xyz) (non-breaking change)
3. Changing the key.attribute UIDs for declaration modifiers, e.g. (for the example above):

     key.attributes: [
       {
         key.offset: 23,
         key.length: 6,
         key.attribute: source.decl.modifier.public
       },
       {
         key.offset: 0,
         key.length: 22,
         key.attribute: source.decl.attribute.available
       }
     ]
   

   (breaking change, but probably wouldn't break as much as 1.)

Alternatives considered

For the Missing attribute arguments and Missing custom attribute information there aren't really any alternatives, except parsing the returned key.sourcetext yourself, which I wouldn't consider structured information.

For Little distinction between attributes and declaration modifiers, it's currently possible to find out whether something is an attribute or a declaration modifiers by simply mapping the source.decl.attribute.x UIDs to whether they are attributes/declaration modifiers respectively, as these are known ahead of time. But this isn't very clean, and doesn't allow for easy handling of just one of the two, which is why more explicit information in the response would be desirable.

Additional information

If these ideas are too varied and this issue too big, I'd be happy to split it up into multiple smaller issues and convert this one to a map of those issues.

[rintaro]

In general, document structure is only meant for providing "structure" of the source code. Clients can use it for implementing some IDE functionalities, such as code navigation, code folding, etc.
For retrieving such detailed syntactic information, I recommend to use SwiftSyntax. Ideally, clients should not use SourceKit's document structure at all, it only provide some syntactic information, and all that information is easily retrievable from the SwiftSyntax tree.

[J-MR-T]

Ah, thank you for the recommendation! We'll definitely look into SwiftSyntax as an alternative.

We're looking into whether the document structure part of the response could be used for IDE support, so as a follow-up question: Is this really generally discouraged in favor of something like SwiftSyntax? I'm asking because we might also have a look at adding more information to the document structure, not only syntactic information as described here, but e.g. USRs as well. But as I understand you, this wouldn't be welcome, as it's not purely structural information, correct?

[rintaro]

Correct, I don't think we would consider adding more information to SourceKit's document structure.
USR in particular cannot be added to document structure as it's a semantic information. I.e. to generate a USR, the decl must be type checked.