Prometheus Metrics return "server returned HTTP status 404 Not Found"

[FLeven]

• Gitea version (or commit ref): v1.14.3 or v1.13.7

• Git version: 2.32.0

• Operating system: Windows Server Core 2019 or Win10 local

  • gitea-1.14.3-windows-4.0-amd64
  • Docker and / or local

• Database (use [x]):

  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite

• Can you reproduce the bug at https://try.gitea.io:

  • Yes (provide example URL)
  • No

• Log gist (local):

2021/06/29 11:34:49 ...ters/user/profile.go:29:GetUserByName() [E] GetUserByName: user redirect does not exist [name: metrics]
/source/routers/user/profile.go:29 (0x1feba32)
/source/routers/user/profile.go:68 (0x1febcf8)
/source/modules/web/route.go:64 (0x1fa1ffb)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/github.com/go-chi/chi/mux.go:436 (0x1b4ba6a)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/modules/web/route.go:103 (0x1fa26e9)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/modules/web/route.go:103 (0x1fa26e9)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/modules/web/route.go:103 (0x1fa26e9)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/modules/web/route.go:103 (0x1fa26e9)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/github.com/go-chi/chi/middleware/get_head.go:37 (0x21d3fe1)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/modules/context/context.go:704 (0x1b6b121)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/routers/routes/base.go:94 (0x21db6ac)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/routers/routes/base.go:94 (0x21db6ac)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/modules/public/public.go:86 (0x13d800c)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/modules/public/public.go:86 (0x13d800c)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/routers/routes/base.go:199 (0x21dd317)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/gitea.com/go-chi/session/session.go:256 (0x152ed6e)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/github.com/go-chi/chi/mux.go:70 (0x1b4956a)
/source/vendor/github.com/go-chi/chi/mux.go:311 (0x1b4fa1b)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/github.com/go-chi/chi/mux.go:436 (0x1b4ba6a)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/routers/routes/web.go:107 (0x21de7a4)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/routers/routes/base.go:38 (0x21da308)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/github.com/go-chi/chi/middleware/strip.go:30 (0x21d4927)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/github.com/chi-middleware/proxy/middleware.go:37 (0x21cfe53)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/routers/routes/web.go:63 (0x21de2e7)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/source/vendor/github.com/go-chi/chi/mux.go:87 (0x1b492f0)
/source/modules/web/route.go:298 (0x1fa151a)
/source/vendor/github.com/gorilla/context/context.go:141 (0x11980fa)
/usr/local/go/src/net/http/server.go:2069 (0x7858aa)
/usr/local/go/src/net/http/server.go:2887 (0x788ee9)
/usr/local/go/src/net/http/server.go:1952 (0x78436c)
/usr/local/go/src/runtime/asm_amd64.s:1371 (0x426640)

Description

Prometheus reports status "server returned HTTP status 404 Not Found", accessing the site in a browser, same result. I first thought it might be my change from redirecting ports, because I switched to https only without a port redirect, but it won't work with redirect turned on or http only. It was working with the first 1.14.x release(s).

[noerw]

What URL is prometheus trying to access here?

[FLeven]

https://server:3081/metrics
I also reverted back to the port redirection of 3080 -> 3081 and tried everything without SSL and certificates.
Same result

[noerw]

I can't reproduce this on an 1.14 instance.

The 404 user redirect does not exist is likely an erroneous response that should actually be a 404 user not found
(for 1.15.0 this was fixed in 89255e9#diff-5b3f30d78073b3263d7d49f791b6f26a19f8ed007de001cbb535aba63d1ff34c)

But this user lookup shouldn't even be made; there's far too much middleware running on the /metrics route.

A failure condition I can think of:

• maybe the /metrics handler isn't triggered at all, as it is confused with a username
• there's inconsistent state in your DB, triggering the exception by the (unrelated) middleware looking up user redirects

Not sure where to further debug this without being able to reproduce

[FLeven]

What tables will I have to check for inconsistencys in the DB ?

[6543]

@FLeven are you sure you did enable metrics in app.ini?

[metrics]
ENABLED=true

[lunny]

@FLeven are you sure you did enable metrics in app.ini?

[metrics]
ENABLED=true

I cannot reproduce this if enabled metrics

[FLeven]

Yes and it was working before

[FLeven]

I cleaned up my app.ini a bit and now I get a clean 404 in the logs:

`2021/07/08 10:04:59 Started GET /metrics for 192.168.1.2:49872
/source/routers/routes/base.go:36 (0x31fa2ca)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/vendor/github.com/go-chi/chi/middleware/strip.go:30 (0x31f4927)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/vendor/github.com/chi-middleware/proxy/middleware.go:37 (0x31efe53)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/routers/routes/web.go:63 (0x31fe2e7)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/vendor/github.com/go-chi/chi/mux.go:87 (0x2b692f0)
/source/modules/web/route.go:298 (0x2fc151a)
/source/vendor/github.com/gorilla/context/context.go:141 (0x21b80fa)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/usr/local/go/src/net/http/server.go:2887 (0x17a8ee9)
/usr/local/go/src/net/http/server.go:1952 (0x17a436c)
/usr/local/go/src/runtime/asm_amd64.s:1371 (0x1446640)

2021/07/08 10:04:59 Completed GET /metrics 404 Not Found in 10.0839ms
/source/routers/routes/base.go:45 (0x31fa705)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/vendor/github.com/go-chi/chi/middleware/strip.go:30 (0x31f4927)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/vendor/github.com/chi-middleware/proxy/middleware.go:37 (0x31efe53)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/routers/routes/web.go:63 (0x31fe2e7)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/source/vendor/github.com/go-chi/chi/mux.go:87 (0x2b692f0)
/source/modules/web/route.go:298 (0x2fc151a)
/source/vendor/github.com/gorilla/context/context.go:141 (0x21b80fa)
/usr/local/go/src/net/http/server.go:2069 (0x17a58aa)
/usr/local/go/src/net/http/server.go:2887 (0x17a8ee9)
/usr/local/go/src/net/http/server.go:1952 (0x17a436c)
/usr/local/go/src/runtime/asm_amd64.s:1371 (0x1446640)
`

[FLeven]

Ok, it was my fault, [metrics] was un-commented, in the correct app.ini, sry. Still I don't understand why the log had different error messages, the last one was the definitiv hint, that it has to be me, who caused the error. thx for looking into it anyway.