Description
Current Behavior
We have provisioned Milvus Cluster-1 in AWS (us-east-2 region) and successfully set up the backup and restore tool using the bucket backup-bucket-1.
We also provisioned Milvus Cluster-2 in AWS (us-west-2 region) and configured the backup and restore tool using the same backup bucket (backup-bucket-1).
In Cluster-2, I configured the backup.yaml file as shown below. While I’m able to list the backups from the bucket, I’m unable to restore them and encountering the following error:
backup.yaml:
minio:
storageType: "aws"
address: s3.us-east-2.amazonaws.com
port: 443
useSSL: true
useIAM: true
iamEndpoint: "iam.amazonaws.com"
bucketName: "bucket-2"
rootPath: "milvus/s3-access"
# Backup storage configs, the storage you want to put the backup data
backupStorageType: "aws"
backupAddress: s3.us-west-2.amazonaws.com
backupPort: 443
backupBucketName: "backup-bucket-1"
backupRootPath: "milvus/s3-backup-access"
backupUseSSL: true
crossStorage: "true"
Restore Error:
[2025/04/18 22:23:17.652 +00:00] [INFO] [core/backup_impl_restore_backup.go:24] ["receive RestoreBackupRequest"] [request="requestId:"ba22045b-1ca3-11f0-91c8-4a35b3b7b2cb" backup_name:"complete_backup_20250417000002" collection_names:"book" collection_suffix:"_restore" async:true"]
[2025/04/18 22:23:17.652 +00:00] [INFO] [core/backup_context.go:291] ["receive GetBackupRequest"] [requestId=ba22142c-1ca3-11f0-91c8-4a35b3b7b2cb] [backupName=complete_backup_20250417000002] [backupId=] [bucketName=] [path=]
[2025/04/18 22:23:18.075 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=isds-milvus-dev-west2-backup] [root=milvus/s3-backup-access]
[2025/04/18 22:23:19.625 +00:00] [INFO] [core/backup_context.go:368] ["finish GetBackupRequest"] [requestId=ba22142c-1ca3-11f0-91c8-4a35b3b7b2cb] [backupName=complete_backup_20250417000002] [backupId=] [bucketName=] [path=]
[2025/04/18 22:23:19.626 +00:00] [INFO] [core/backup_impl_restore_backup.go:111] ["skip restore RBAC"]
[2025/04/18 22:23:19.626 +00:00] [INFO] [core/backup_impl_restore_backup.go:180] ["Collections to restore"] [collection_num=1]
[GIN] 2025/04/18 - 22:23:19 | 200 | 1.974461101s | 17.51.226.9 | POST "/api/v1/restore"
[2025/04/18 22:23:19.657 +00:00] [WARN] [storage/minio_chunk_manager.go:115] ["failed to check blob bucket exist"] [bucket=bucket-2] [error="Get "https://bucket-2.s3.us-east-2.amazonaws.com/?location=": Forbidden"]
[2025/04/18 22:23:19.870 +00:00] [WARN] [storage/minio_chunk_manager.go:115] ["failed to check blob bucket exist"] [bucket=bucket-2] [error="Get "https://bucket-2.s3.us-east-2.amazonaws.com/?location=": Forbidden"]
[2025/04/18 22:23:20.280 +00:00] [WARN] [storage/minio_chunk_manager.go:115] ["failed to check blob bucket exist"] [bucket=bucket-2] [error="Get "https://bucket-2.s3.us-east-2.amazonaws.com/?location=": Forbidden"]
[2025/04/18 22:23:21.097 +00:00] [WARN] [storage/minio_chunk_manager.go:115] ["failed to check blob bucket exist"] [bucket=bucket-2] [error="Get "https://bucket-2.s3.us-east-2.amazonaws.com/?location=": Forbidden"]
Expected Behavior
No response
Steps To Reproduce
Environment
Anything else?
No response