Merge branch 'main' into feature/clear-merge-commit-content-button

Author: lunny

docs/content/doc/advanced/config-cheat-sheet.en-us.md

@@ -289,8 +289,13 @@ The following configuration set `Content-Type: application/vnd.android.package-a
    This includes CSS files, images, JS files and web fonts.
    Avatar images are dynamic resources and still served by Gitea.
    The option can be just a different path, as in `/static`, or another domain, as in `https://cdn.example.com`.
-   Requests are then made as `%(ROOT_URL)s/static/css/index.css` and `https://cdn.example.com/css/index.css` respective.
+   Requests are then made as `%(ROOT_URL)s/static/assets/css/index.css` or `https://cdn.example.com/assets/css/index.css` respectively.
    The static files are located in the `public/` directory of the Gitea source repository.
+   You can proxy the STATIC_URL_PREFIX requests to Gitea server to serve the static
+   assets, or copy the manually built Gitea assets from `$GITEA_BUILD/public` to
+   the assets location, eg: `/var/www/assets`, make sure `$STATIC_URL_PREFIX/assets/css/index.css`
+   points to `/var/www/assets/css/index.css`.
+
 - `HTTP_ADDR`: **0.0.0.0**: HTTP listen address.
   - If `PROTOCOL` is set to `fcgi`, Gitea will listen for FastCGI requests on TCP socket
      defined by `HTTP_ADDR` and `HTTP_PORT` configuration settings.

docs/content/doc/features/authentication.en-us.md

@@ -166,11 +166,47 @@ Uses the following fields:
 
 ## PAM (Pluggable Authentication Module)
 
-To configure PAM, set the 'PAM Service Name' to a filename in `/etc/pam.d/`. To
-work with normal Linux passwords, the user running Gitea must have read access
-to `/etc/shadow`.
-
-**Note**: PAM support is added via [build-time flags](https://docs.gitea.io/en-us/install-from-source/#build), and the official binaries provided do not have this enabled.
+This procedure enables PAM authentication.  Users may still be added to the
+system manually using the user administration.  PAM provides a mechanism to
+automatically add users to the current database by testing them against PAM
+authentication.  To work with normal Linux passwords, the user running Gitea
+must also have read access to `/etc/shadow` in order to check the validity of
+the account when logging in using a public key.
+
+**Note**: If a user has added SSH public keys into Gitea, the use of these
+keys _may_ bypass the login check system.  Therefore, if you wish to disable a user who
+authenticates with PAM, you _should_ also manually disable the account in Gitea using the
+built-in user manager.
+
+1. Configure and prepare the installation.
+    - It is recommended that you create an administrative user.
+    - Deselecting automatic sign-up may also be desired.
+1. Once the database has been initialized, log in as the newly created
+administrative user.
+1. Navigate to the user setting (icon in top-right corner), and select
+`Site Administration` -> `Authentication Sources`, and select
+`Add Authentication Source`.
+1. Fill out the field as follows:
+    - `Authentication Type` : `PAM`
+    - `Name` : Any value should be valid here, use "System Authentication" if
+    you'd like.
+    - `PAM Service Name` : Select the appropriate file listed under `/etc/pam.d/`
+    that performs the authentication desired.[^1]
+    - `PAM Email Domain` : The e-mail suffix to append to user authentication.
+    For example, if the login system expects a user called `gituser`, and this
+    field is set to `mail.com`, then Gitea will expect the `user email` field
+    for an authenticated GIT instance to be `[email protected]`.[^2]
+
+**Note**: PAM support is added via [build-time flags](https://docs.gitea.io/en-us/install-from-source/#build),
+and the official binaries provided do not have this enabled.  PAM requires that
+the necessary libpam dynamic library be available and the necessary PAM
+development headers be accessible to the compiler.
+
+[^1]: For example, using standard Linux log-in on Debian "Bullseye" use
+`common-session-noninteractive` - this value may be valid for other flavors of
+Debian including Ubuntu and Mint, consult your distribution's documentation.
+[^2]: **This is a required field for PAM**.  Be aware: In the above example, the
+user will log into the Gitea web interface as `gituser` and not `[email protected]`
 
 ## SMTP (Simple Mail Transfer Protocol)
 

docs/content/doc/packages/storage.en-us.md

@@ -0,0 +1,84 @@
+---
+date: "2022-11-01T00:00:00+00:00"
+title: "Storage"
+slug: "packages/storage"
+draft: false
+toc: false
+menu:
+  sidebar:
+    parent: "packages"
+    name: "storage"
+    weight: 5
+    identifier: "storage"
+---
+
+# Storage
+
+This document describes the storage of the package registry and how it can be managed.
+
+**Table of Contents**
+
+{{< toc >}}
+
+## Deduplication
+
+The package registry has a build-in deduplication of uploaded blobs.
+If two identical files are uploaded only one blob is saved on the filesystem.
+This ensures no space is wasted for duplicated files.
+
+If two packages are uploaded with identical files, both packages will display the same size but on the filesystem they require only half of the size.
+Whenever a package gets deleted only the references to the underlaying blobs are removed.
+The blobs get not removed at this moment, so they still require space on the filesystem.
+When a new package gets uploaded the existing blobs may get referenced again.
+
+These unreferenced blobs get deleted by a [clean up job]({{< relref "doc/advanced/config-cheat-sheet.en-us.md#cron---cleanup-expired-packages-croncleanup_packages" >}}).
+The config setting `OLDER_THAN` configures how long unreferenced blobs are kept before they get deleted.
+
+## Cleanup Rules
+
+Package registries can become large over time without cleanup.
+It's recommended to delete unnecessary packages and set up cleanup rules to automatically manage the package registry usage.
+Every package owner (user or organization) manages the cleanup rules which are applied to their packages.
+
+|Setting|Description|
+|-|-|
+|Enabled|Turn the cleanup rule on or off.|
+|Type|Every rule manages a specific package type.|
+|Apply pattern to full package name|If enabled, the patterns below are applied to the full package name (`package/version`). Otherwise only the version (`version`) is used.|
+|Keep the most recent|How many versions to *always* keep for each package.|
+|Keep versions matching|The regex pattern that determines which versions to keep. An empty pattern keeps no version while `.+` keeps all versions. The container registry will always keep the `latest` version even if not configured.|
+|Remove versions older than|Remove only versions older than the selected days.|
+|Remove versions matching|The regex pattern that determines which versions to remove. An empty pattern or `.+` leads to the removal of every package if no other setting tells otherwise.|
+
+Every cleanup rule can show a preview of the affected packages.
+This can be used to check if the cleanup rules is proper configured.
+
+### Regex examples
+
+Regex patterns are automatically surrounded with `\A` and `\z` anchors.
+Do not include any `\A`, `\z`, `^` or `$` token in the regex patterns as they are not necessary.
+The patterns are case-insensitive which matches the behaviour of the package registry in Gitea.
+
+|Pattern|Description|
+|-|-|
+|`.*`|Match every possible version.|
+|`v.+`|Match versions that start with `v`.|
+|`release`|Match only the version `release`.|
+|`release.*`|Match versions that are either named or start with `release`.|
+|`.+-temp-.+`|Match versions that contain `-temp-`.|
+|`v.+\|release`|Match versions that either start with `v` or are named `release`.|
+|`package/v.+\|other/release`|Match versions of the package `package` that start with `v` or the version `release` of the package `other`. This needs the setting *Apply pattern to full package name* enabled.|
+
+### How the cleanup rules work
+
+The cleanup rules are part of the [clean up job]({{< relref "doc/advanced/config-cheat-sheet.en-us.md#cron---cleanup-expired-packages-croncleanup_packages" >}}) and run periodicly.
+
+The cleanup rule:
+
+1. Collects all packages of the package type for the owners registry.
+1. For every package it collects all versions.
+1. Excludes from the list the # versions based on the *Keep the most recent* value.
+1. Excludes from the list any versions matching the *Keep versions matching* value.
+1. Excludes from the list the versions more recent than the *Remove versions older than* value.
+1. Excludes from the list any versions not matching the *Remove versions matching* value.
+1. Deletes the remaining versions.

models/migrations/migrations.go

@@ -439,6 +439,8 @@ var migrations = []Migration{
 	NewMigration("Alter package_version.metadata_json to LONGTEXT", v1_19.AlterPackageVersionMetadataToLongText),
 	// v233 -> v234
 	NewMigration("Add header_authorization_encrypted column to webhook table", v1_19.AddHeaderAuthorizationEncryptedColWebhook),
+	// v234 -> v235
+	NewMigration("Add package cleanup rule table", v1_19.CreatePackageCleanupRuleTable),
 }
 
 // GetCurrentDBVersion returns the current db version

models/migrations/v1_19/v234.go

@@ -0,0 +1,29 @@
+// Copyright 2022 The Gitea Authors. All rights reserved.
+// Use of this source code is governed by a MIT-style
+// license that can be found in the LICENSE file.
+
+package v1_19 //nolint
+
+import (
+	"code.gitea.io/gitea/modules/timeutil"
+
+	"xorm.io/xorm"
+)
+
+func CreatePackageCleanupRuleTable(x *xorm.Engine) error {
+	type PackageCleanupRule struct {
+		ID            int64              `xorm:"pk autoincr"`
+		Enabled       bool               `xorm:"INDEX NOT NULL DEFAULT false"`
+		OwnerID       int64              `xorm:"UNIQUE(s) INDEX NOT NULL DEFAULT 0"`
+		Type          string             `xorm:"UNIQUE(s) INDEX NOT NULL"`
+		KeepCount     int                `xorm:"NOT NULL DEFAULT 0"`
+		KeepPattern   string             `xorm:"NOT NULL DEFAULT ''"`
+		RemoveDays    int                `xorm:"NOT NULL DEFAULT 0"`
+		RemovePattern string             `xorm:"NOT NULL DEFAULT ''"`
+		MatchFullName bool               `xorm:"NOT NULL DEFAULT false"`
+		CreatedUnix   timeutil.TimeStamp `xorm:"created NOT NULL DEFAULT 0"`
+		UpdatedUnix   timeutil.TimeStamp `xorm:"updated NOT NULL DEFAULT 0"`
+	}
+
+	return x.Sync2(new(PackageCleanupRule))
+}

models/packages/package.go

@@ -45,6 +45,21 @@ const (
 	TypeVagrant   Type = "vagrant"
 )
 
+var TypeList = []Type{
+	TypeComposer,
+	TypeConan,
+	TypeContainer,
+	TypeGeneric,
+	TypeHelm,
+	TypeMaven,
+	TypeNpm,
+	TypeNuGet,
+	TypePub,
+	TypePyPI,
+	TypeRubyGems,
+	TypeVagrant,
+}
+
 // Name gets the name of the package type
 func (pt Type) Name() string {
 	switch pt {

models/packages/package_cleanup_rule.go

@@ -0,0 +1,110 @@
+// Copyright 2022 The Gitea Authors. All rights reserved.
+// Use of this source code is governed by a MIT-style
+// license that can be found in the LICENSE file.
+
+package packages
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"regexp"
+
+	"code.gitea.io/gitea/models/db"
+	"code.gitea.io/gitea/modules/timeutil"
+
+	"xorm.io/builder"
+)
+
+var ErrPackageCleanupRuleNotExist = errors.New("Package blob does not exist")
+
+func init() {
+	db.RegisterModel(new(PackageCleanupRule))
+}
+
+// PackageCleanupRule represents a rule which describes when to clean up package versions
+type PackageCleanupRule struct {
+	ID                   int64              `xorm:"pk autoincr"`
+	Enabled              bool               `xorm:"INDEX NOT NULL DEFAULT false"`
+	OwnerID              int64              `xorm:"UNIQUE(s) INDEX NOT NULL DEFAULT 0"`
+	Type                 Type               `xorm:"UNIQUE(s) INDEX NOT NULL"`
+	KeepCount            int                `xorm:"NOT NULL DEFAULT 0"`
+	KeepPattern          string             `xorm:"NOT NULL DEFAULT ''"`
+	KeepPatternMatcher   *regexp.Regexp     `xorm:"-"`
+	RemoveDays           int                `xorm:"NOT NULL DEFAULT 0"`
+	RemovePattern        string             `xorm:"NOT NULL DEFAULT ''"`
+	RemovePatternMatcher *regexp.Regexp     `xorm:"-"`
+	MatchFullName        bool               `xorm:"NOT NULL DEFAULT false"`
+	CreatedUnix          timeutil.TimeStamp `xorm:"created NOT NULL DEFAULT 0"`
+	UpdatedUnix          timeutil.TimeStamp `xorm:"updated NOT NULL DEFAULT 0"`
+}
+
+func (pcr *PackageCleanupRule) CompiledPattern() error {
+	if pcr.KeepPatternMatcher != nil || pcr.RemovePatternMatcher != nil {
+		return nil
+	}
+
+	if pcr.KeepPattern != "" {
+		var err error
+		pcr.KeepPatternMatcher, err = regexp.Compile(fmt.Sprintf(`(?i)\A%s\z`, pcr.KeepPattern))
+		if err != nil {
+			return err
+		}
+	}
+
+	if pcr.RemovePattern != "" {
+		var err error
+		pcr.RemovePatternMatcher, err = regexp.Compile(fmt.Sprintf(`(?i)\A%s\z`, pcr.RemovePattern))
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func InsertCleanupRule(ctx context.Context, pcr *PackageCleanupRule) (*PackageCleanupRule, error) {
+	return pcr, db.Insert(ctx, pcr)
+}
+
+func GetCleanupRuleByID(ctx context.Context, id int64) (*PackageCleanupRule, error) {
+	pcr := &PackageCleanupRule{}
+
+	has, err := db.GetEngine(ctx).ID(id).Get(pcr)
+	if err != nil {
+		return nil, err
+	}
+	if !has {
+		return nil, ErrPackageCleanupRuleNotExist
+	}
+	return pcr, nil
+}
+
+func UpdateCleanupRule(ctx context.Context, pcr *PackageCleanupRule) error {
+	_, err := db.GetEngine(ctx).ID(pcr.ID).AllCols().Update(pcr)
+	return err
+}
+
+func GetCleanupRulesByOwner(ctx context.Context, ownerID int64) ([]*PackageCleanupRule, error) {
+	pcrs := make([]*PackageCleanupRule, 0, 10)
+	return pcrs, db.GetEngine(ctx).Where("owner_id = ?", ownerID).Find(&pcrs)
+}
+
+func DeleteCleanupRuleByID(ctx context.Context, ruleID int64) error {
+	_, err := db.GetEngine(ctx).ID(ruleID).Delete(&PackageCleanupRule{})
+	return err
+}
+
+func HasOwnerCleanupRuleForPackageType(ctx context.Context, ownerID int64, packageType Type) (bool, error) {
+	return db.GetEngine(ctx).
+		Where("owner_id = ? AND type = ?", ownerID, packageType).
+		Exist(&PackageCleanupRule{})
+}
+
+func IterateEnabledCleanupRules(ctx context.Context, callback func(context.Context, *PackageCleanupRule) error) error {
+	return db.Iterate(
+		ctx,
+		builder.Eq{"enabled": true},
+		callback,
+	)
+}

models/packages/package_version.go

@@ -320,6 +320,15 @@ func SearchLatestVersions(ctx context.Context, opts *PackageSearchOptions) ([]*P
 	return pvs, count, err
 }
 
+// ExistVersion checks if a version matching the search options exist
+func ExistVersion(ctx context.Context, opts *PackageSearchOptions) (bool, error) {
+	return db.GetEngine(ctx).
+		Where(opts.toConds()).
+		Table("package_version").
+		Join("INNER", "package", "package.id = package_version.package_id").
+		Exist(new(PackageVersion))
+}
+
 // CountVersions counts all versions of packages matching the search options
 func CountVersions(ctx context.Context, opts *PackageSearchOptions) (int64, error) {
 	return db.GetEngine(ctx).