[Bug]: parse backup collections from request failed

[iurii-stepanov]

Current Behavior

Milvus 2.4.10
milvus-backup 0.5.3

When I backup the default database, which consists of one collection, everything goes well (milvus-backup create -d deafault --config configs.yaml).

When I backup another database of more than 2500 collections, I get an error (milvus-backup create -d rag_platform_mvp --config configs.yaml)
The same thing if I backup all the databases together (milvus-backup create --config configs.yaml).

0.5.3 (Built on 2025-04-02T05:27:27Z from Git SHA f20e3ba532d7daf8807b5e384c3284da9f3ad9f3)
config:./configs/backup_yc_new.yaml
[2025/04/04 17:34:25.565 +03:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2025/04/04 17:34:25.567 +03:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2025/04/04 17:34:25.568 +03:00] [INFO] [core/backup_impl_create_backup.go:44] ["receive CreateBackupRequest"] [requestId=e85231ac-1161-11f0-b297-5a6ecce8fcb5] [backupName=] [collections="[]"] [databaseCollections="{\"rag_platform_mvp\":[]}"] [async=false] [force=false] [metaOnly=false]
[2025/04/04 17:34:25.568 +03:00] [INFO] [core/backup_context.go:87] ["{Base:0x1400003cc08 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:false GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091}"]
[2025/04/04 17:34:25.568 +03:00] [INFO] [core/backup_context.go:88] ["{Base:0x1400003cc08 Enabled:true DebugMode:false SimpleResponse:true}"]
[2025/04/04 17:34:25.985 +03:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=llm-platform] [root=milvus-backup]
[2025/04/04 17:34:26.048 +03:00] [INFO] [client/grpc.go:241] ["New milvus grpc client"] [host=llm-platform-milvus.tcsbank.ru:19530]
[2025/04/04 17:34:26.049 +03:00] [WARN] [grpclog/grpclog.go:155] ["[core]Adjusting keepalive ping interval to minimum period of 10s"]
[2025/04/04 17:34:26.488 +03:00] [ERROR] [core/backup_impl_create_backup.go:168] ["fail in ListCollections"] [error="client: list collections failed: rpc error: code = Internal desc = unexpected EOF"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).parseBackupCollections\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:168\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:672\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).CreateBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:125\ngithub.com/zilliztech/milvus-backup/cmd.init.func3\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/create.go:62\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:989\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:19\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.7/x64/src/runtime/proc.go:272"]
[2025/04/04 17:34:26.489 +03:00] [ERROR] [core/backup_impl_create_backup.go:674] ["parse backup collections from request failed"] [error="client: list collections failed: rpc error: code = Internal desc = unexpected EOF"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:674\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).CreateBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:125\ngithub.com/zilliztech/milvus-backup/cmd.init.func3\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/create.go:62\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:989\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:19\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.7/x64/src/runtime/proc.go:272"]
client: list collections failed: rpc error: code = Internal desc = unexpected EOF
duration:1 s

Expected Behavior

No response

Steps To Reproduce

Environment

Anything else?

No response

[iurii-stepanov]

update: there is 3700 collections in db

[huanghaoyuanhhy]

It looks like the issue is caused by a gRPC connection being interrupted. Could you please check if there is a load balancer (LB) between the backup tool and Milvus? If so, does the LB have any limitations on response size or connection duration?

Since the database contains more than 2500 collections, the metadata response from Milvus could be quite large, which may trigger such limitations.