[Bug][Devlake][GitHub] GitHub app connection data source usability issues

[petercort]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Creating a GitHub App connection isn't working as expected.

1. No organizations or repositories are found 'naturally' after the connection is created
2. Can't multiselect to add repositories when they are specified into the datasources search

What do you expect to happen

I would expect that the repositories in the organization where the app has been installed would auto populate and I could select multiples to add as a data source.

How to reproduce

1. Configure a GitHub App as a connection
2. Add data sources - no

Anything else

I was able to reproduce this behavior with a GitHub app installed into an organization and my personal account.

I'm willing to take a crack at a PR if it's not a massive thing. I'll spend some time setting up a debugger and seeing if I can get more info. Might need someone more skilled than me (cough cough copilot) to fix this.

I also tested this with the v0.21.0@ac4f7fe and it's reproducible there.

Here's the error message formatted nicely:

level=error msg="HTTP 400 error
    caused by: attached stack trace
     -- stack trace:
        | github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*DsScopeApiHelper[...]).PutMultiple
        | /app/helpers/pluginhelper/api/ds_scope_api_helper.go:114  
        | github.com/apache/incubator-devlake/plugins/github/api.PutScopes  
        | /app/plugins/github/api/scope_api.go:42
        | github.com/apache/incubator-devlake/server/api.handlePluginCall.func1
        | /app/server/api/router.go:135
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/apache/incubator-devlake/server/api.OAuth2ProxyAuthentication.func1
        | /app/server/api/middlewares.go:95
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/apache/incubator-devlake/server/api.RestAuthentication.func1
        | /app/server/api/middlewares.go:117
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/gin-gonic/gin.LoggerWithConfig.func1
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
        | github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620
        | github.com/gin-gonic/gin.(*Engine).ServeHTTP
        | /go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576
        | net/http.serverHandler.ServeHTTP
        | /usr/local/go/src/net/http/server.go:2936
        | net/http.(*conn).serve
        | /usr/local/go/src/net/http/server.go:1995
        | runtime.goexit
        | /usr/local/go/src/runtime/asm_arm64.s:1172
    Wraps: (2) invalid data row (400)
    Error types: (1) *withstack.withStack (2) *errutil.leafError"

Version

v1.0.0-beta11@15c4c09

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hello, @petercort! I'm here to assist you with any bugs, questions, or contribute to the repository. Let me know how I can help you while we wait for a human maintainer.

To resolve the issue where creating a GitHub App connection does not auto-populate organizations or repositories and does not allow multiselect to add repositories, ensure the following:

1. Correct Permissions: Verify that the GitHub App has the necessary permissions. The required permissions for collecting data from repositories are:

   • Repository
     • Actions
     • Administration
     • Checks
     • Commit statuses
     • Contents
     • Deployments
     • Issues
     • Metadata
     • Pull requests
   • Organization
     • Members

2. Endpoint URL: Ensure the Endpoint URL is correctly set to https://api.github.com/ and ends with a /.

3. Personal Access Token: If using a Personal Access Token (PAT), ensure it has the required permissions:

   • For public repositories:
     • repo:status
     • repo_deployment
     • read:user
     • read:org
   • For private repositories:
     • repo
     • read:user
     • read:org

4. GraphQL API: If using fine-grained PATs, ensure the Use GraphQL APIs option is disabled as fine-grained PATs do not support GraphQL APIs.

5. Proxy Settings: If behind a corporate firewall or VPN, ensure the Proxy URL is correctly configured.

6. Rate Limits: Ensure the rate limit settings do not exceed the maximum allowed by GitHub.

7. Add Data Scopes: After creating the connection, ensure you add the data scopes correctly. Choose the GitHub repositories you wish to collect either by finding them in the miller column or searching. Note that you can only add public repositories through the search box.

8. Test and Save Connection: Click Test Connection, if the connection is successful, click Save Connection to add the connection.

If all the above settings are correct and the issue persists, refer to the GitHub Configuration Guide for detailed instructions ^{[1][2][3][4][5]}.

Additionally, specific error messages or logs can indicate why the organizations or repositories are not being auto-populated. Functions like listGithubOrgRepos, listGithubAppInstalledRepos, and listGithubUserOrgs return errors if they encounter issues during the API calls. These errors are propagated back to the caller, which can then log or display them to indicate the issue ^[6].

To continue talking to Dosu, mention @dosu.

[klesh]

Can you share the request payload from your browser under the Network tab?
Please remove any sensitive information before you post but keep the overall structure.

[petercort]

For the connection being created and scopes loading:

Once the connection is created, I've got 2 calls
http://localhost:4000/api/plugins/github/connections/11/scopes?page=1&pageSize=10&blueprints=true
Response:

{"count":0,"scopes":[]}

http://localhost:4000/api/plugins/github/connections/11/remote-scopes
Response:

{
  "children": [
      {
          "type": "scope",
          "parentId": "petes-fre-shavocado-stand",
          "id": "UID",
          "name": "roadwork",
          "fullName": "petes-fre-shavocado-stand/roadwork",
          "data": null
      }
  ],
  "nextPageToken": ""
}

Multi selecting scope on a manual search

http://localhost:4000/api/plugins/github/connections/11/scopes

[Mantesh-Globant]

@petercort We are facing same issue while integrating Github app with devlake , we have validated all the required permissions and end point url and still facing same issue
@klesh

[klesh]

Seems like there are multiple problems to me:

1. http://localhost:4000/api/plugins/github/connections/11/remote-scopes is not responding correctly: it doesn't return repositories under the org.
2. the config-ui is sending null to the PUT http://localhost:4000/api/plugins/github/connections/11/scopes API

Am I right?

[petercort]

@klesh from what I can see that's correct.

I've been running into a lot of issues setting up a debug environment to get more info about the calls, so I can't offer much more at the moment :(

[klesh]

There are a couple of bugs in the Github App implementation.
I suggest that you use tokens as a workaround for the time being, It might take quite a while to fix all those problems.

[sanjchhoker]

I had same issue , I did a workaround to add scopes to the connection programmatically using first github API to fetch repository id's and then add them to connection using devlake API.
However pipelines are generally flaky with the setup and have got a weird error on the 1st and last step of the pipeline without any actual error in the logs , with pipeline status set to Partially Successful.

[d4x1]

@sanjchhoker Can you help submit a new issue and describe what happened in detail ? It seems a differcent problem.