Skip to content

Supports forced use of S3 virtual-hosted style #30969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
May 15, 2024
Merged

Supports forced use of S3 virtual-hosted style #30969

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
9fdc34b
Support minio virtual host pathlike
dicarne May 13, 2024
fb80fa5
Modify the default lookup method to be the same as before
dicarne May 14, 2024
3727384
Merge branch 'go-gitea:main' into main
dicarne May 14, 2024
5756446
fix incorrect fmt
dicarne May 14, 2024
697982a
Add `VIRTUAL_HOST` to document and app.example.ini.
dicarne May 14, 2024
1306ff6
Change Minio lookup type to enum.
dicarne May 14, 2024
016a884
Fix the error of MINIO_LOOKUP_TYPE in app.example.ini.
dicarne May 14, 2024
792a9c3
Change MINIO_LOOKUP_TYPE to MINIO_BUCKET_LOOKUP_TYPE, make it more pr…
dicarne May 14, 2024
2996267
Fix document incorrect formatting.
dicarne May 14, 2024
d84e92e
Added default value of `MINIO_BUCKET_LOOKUP_TYPE` to getDefaultStorag…
dicarne May 14, 2024
cdbc997
Simplfy BucketLookUpType check.
dicarne May 14, 2024
96171b8
Add option to migrate_storage
dicarne May 15, 2024
49a4f9e
Added MINIO_BUCKET_LOOKUP_TYPE processing of empty strings.
dicarne May 15, 2024
b770c9b
Merge pull request #2 from dicarne/fix_bucketlookup
dicarne May 15, 2024
06c0b0b
Merge branch 'main' into main
KN4CK3R May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions custom/conf/app.example.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1895,6 +1895,9 @@ LEVEL = Info
;;
;; Minio checksum algorithm: default (for MinIO or AWS S3) or md5 (for Cloudflare or Backblaze)
;MINIO_CHECKSUM_ALGORITHM = default
;;
;; Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`
;MINIO_BUCKET_LOOKUP_TYPE = auto

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Expand Down Expand Up @@ -2576,6 +2579,9 @@ LEVEL = Info
;;
;; Minio skip SSL verification available when STORAGE_TYPE is `minio`
;MINIO_INSECURE_SKIP_VERIFY = false
;;
;; Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`
;MINIO_BUCKET_LOOKUP_TYPE = auto

;[proxy]
;; Enable the proxy, all requests to external via HTTP will be affected
Expand Down
8 changes: 8 additions & 0 deletions docs/content/administration/config-cheat-sheet.en-us.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -851,6 +851,7 @@ Default templates for project boards:
- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when STORAGE_TYPE is `minio`
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio`
- `MINIO_CHECKSUM_ALGORITHM`: **default**: Minio checksum algorithm: `default` (for MinIO or AWS S3) or `md5` (for Cloudflare or Backblaze)
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`

## Log (`log`)

Expand Down Expand Up @@ -1272,6 +1273,7 @@ is `data/lfs` and the default of `MINIO_BASE_PATH` is `lfs/`.
- `MINIO_BASE_PATH`: **lfs/**: Minio base path on the bucket only available when `STORAGE_TYPE` is `minio`
- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when `STORAGE_TYPE` is `minio`
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio`
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`

## Storage (`storage`)

Expand All @@ -1286,6 +1288,7 @@ Default storage configuration for attachments, lfs, avatars, repo-avatars, repo-
- `MINIO_LOCATION`: **us-east-1**: Minio location to create bucket only available when `STORAGE_TYPE` is `minio`
- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when `STORAGE_TYPE` is `minio`
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio`
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`

The recommended storage configuration for minio like below:

Expand All @@ -1307,6 +1310,8 @@ MINIO_USE_SSL = false
; Minio skip SSL verification available when STORAGE_TYPE is `minio`
MINIO_INSECURE_SKIP_VERIFY = false
SERVE_DIRECT = true
; Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`
MINIO_BUCKET_LOOKUP_TYPE = auto
```

Defaultly every storage has their default base path like below
Expand Down Expand Up @@ -1353,6 +1358,8 @@ MINIO_LOCATION = us-east-1
MINIO_USE_SSL = false
; Minio skip SSL verification available when STORAGE_TYPE is `minio`
MINIO_INSECURE_SKIP_VERIFY = false
; Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`
MINIO_BUCKET_LOOKUP_TYPE = auto
```

## Repository Archive Storage (`storage.repo-archive`)
Expand All @@ -1372,6 +1379,7 @@ is `data/repo-archive` and the default of `MINIO_BASE_PATH` is `repo-archive/`.
- `MINIO_BASE_PATH`: **repo-archive/**: Minio base path on the bucket only available when `STORAGE_TYPE` is `minio`
- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when `STORAGE_TYPE` is `minio`
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio`
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`

## Repository Archives (`repo-archive`)

Expand Down
8 changes: 8 additions & 0 deletions docs/content/administration/config-cheat-sheet.zh-cn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -796,6 +796,7 @@ Gitea 创建以下非唯一队列：
- `MINIO_USE_SSL`: **false**: Minio 启用 SSL，仅当 STORAGE_TYPE 为 `minio` 时可用。
- `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio 跳过 SSL 验证，仅当 STORAGE_TYPE 为 `minio` 时可用。
- `MINIO_CHECKSUM_ALGORITHM`: **default**: Minio 校验算法：`default`（适用于 MinIO 或 AWS S3）或 `md5`（适用于 Cloudflare 或 Backblaze）
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio的bucket查找方式默认为`auto`模式，可将其设置为`dns`（虚拟托管样式）或`path`（路径样式），仅当`STORAGE_TYPE`为`minio`时可用。

## 日志 (`log`)

Expand Down Expand Up @@ -1201,6 +1202,7 @@ ALLOW_DATA_URI_IMAGES = true
- `MINIO_BASE_PATH`：**lfs/**：桶上的 Minio 基本路径，仅在 `STORAGE_TYPE` 为 `minio` 时可用。
- `MINIO_USE_SSL`：**false**：Minio 启用 ssl，仅在 `STORAGE_TYPE` 为 `minio` 时可用。
- `MINIO_INSECURE_SKIP_VERIFY`：**false**：Minio 跳过 SSL 验证，仅在 `STORAGE_TYPE` 为 `minio` 时可用。
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio的bucket查找方式默认为`auto`模式，可将其设置为`dns`（虚拟托管样式）或`path`（路径样式），仅当`STORAGE_TYPE`为`minio`时可用。

## 存储 (`storage`)

Expand All @@ -1215,6 +1217,7 @@ ALLOW_DATA_URI_IMAGES = true
- `MINIO_LOCATION`：**us-east-1**：创建桶的 Minio 位置，仅在 `STORAGE_TYPE` 为 `minio` 时可用。
- `MINIO_USE_SSL`：**false**：Minio 启用 ssl，仅在 `STORAGE_TYPE` 为 `minio` 时可用。
- `MINIO_INSECURE_SKIP_VERIFY`：**false**：Minio 跳过 SSL 验证，仅在 `STORAGE_TYPE` 为 `minio` 时可用。
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio的bucket查找方式默认为`auto`模式，可将其设置为`dns`（虚拟托管样式）或`path`（路径样式），仅当`STORAGE_TYPE`为`minio`时可用。

建议的 minio 存储配置如下：

Expand All @@ -1236,6 +1239,8 @@ MINIO_USE_SSL = false
; Minio skip SSL verification available when STORAGE_TYPE is `minio`
MINIO_INSECURE_SKIP_VERIFY = false
SERVE_DIRECT = true
; Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`
MINIO_BUCKET_LOOKUP_TYPE = auto
```

默认情况下，每个存储都有其默认的基本路径，如下所示：
Expand Down Expand Up @@ -1282,6 +1287,8 @@ MINIO_LOCATION = us-east-1
MINIO_USE_SSL = false
; Minio skip SSL verification available when STORAGE_TYPE is `minio`
MINIO_INSECURE_SKIP_VERIFY = false
; Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio`
MINIO_BUCKET_LOOKUP_TYPE = auto
```

### 存储库归档存储 (`storage.repo-archive`)
Expand All @@ -1299,6 +1306,7 @@ MINIO_INSECURE_SKIP_VERIFY = false
- `MINIO_BASE_PATH`: **repo-archive/**：存储桶上的Minio基本路径，仅在`STORAGE_TYPE`为`minio`时可用。
- `MINIO_USE_SSL`: **false**：启用Minio的SSL，仅在`STORAGE_TYPE`为`minio`时可用。
- `MINIO_INSECURE_SKIP_VERIFY`: **false**：跳过Minio的SSL验证，仅在`STORAGE_TYPE`为`minio`时可用。
- `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio的bucket查找方式默认为`auto`模式，可将其设置为`dns`（虚拟托管样式）或`path`（路径样式），仅当`STORAGE_TYPE`为`minio`时可用。

### 存储库归档 (`repo-archive`)

Expand Down
2 changes: 2 additions & 0 deletions modules/setting/storage.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ type MinioStorageConfig struct {
InsecureSkipVerify bool `ini:"MINIO_INSECURE_SKIP_VERIFY"`
ChecksumAlgorithm string `ini:"MINIO_CHECKSUM_ALGORITHM" json:",omitempty"`
ServeDirect bool `ini:"SERVE_DIRECT"`
BucketLookUpType string `ini:"MINIO_BUCKET_LOOKUP_TYPE"`
}

// Storage represents configuration of storages
Expand Down Expand Up @@ -82,6 +83,7 @@ func getDefaultStorageSection(rootCfg ConfigProvider) ConfigSection {
storageSec.Key("MINIO_USE_SSL").MustBool(false)
storageSec.Key("MINIO_INSECURE_SKIP_VERIFY").MustBool(false)
storageSec.Key("MINIO_CHECKSUM_ALGORITHM").MustString("default")
storageSec.Key("MINIO_BUCKET_LOOKUP_TYPE").MustString("auto")
return storageSec
}

Expand Down
18 changes: 14 additions & 4 deletions modules/storage/minio.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,21 @@ func NewMinioStorage(ctx context.Context, cfg *setting.Storage) (ObjectStorage,

log.Info("Creating Minio storage at %s:%s with base path %s", config.Endpoint, config.Bucket, config.BasePath)

lookup := minio.BucketLookupAuto
if config.BucketLookUpType == "dns" {
lookup = minio.BucketLookupDNS
} else if config.BucketLookUpType == "path" {
lookup = minio.BucketLookupPath
} else if config.BucketLookUpType != "auto" {
return nil, fmt.Errorf("invalid minio bucket lookup type: %s", config.BucketLookUpType)
}

minioClient, err := minio.New(config.Endpoint, &minio.Options{
Creds: credentials.NewStaticV4(config.AccessKeyID, config.SecretAccessKey, ""),
Secure: config.UseSSL,
Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: config.InsecureSkipVerify}},
Region: config.Location,
Creds: credentials.NewStaticV4(config.AccessKeyID, config.SecretAccessKey, ""),
Secure: config.UseSSL,
Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: config.InsecureSkipVerify}},
Region: config.Location,
BucketLookup: lookup,
})
if err != nil {
return nil, convertMinioErr(err)
Expand Down
Loading