Improve Backup User Experience

[huanghaoyuanhhy]

The current backup tool has several usability issues that negatively affect the user experience. We'd like to propose improvements to make the process more intuitive, informative, and user-friendly.

Problems Identified:

• Overly Complex Configuration File

• The configuration file contains many fields and is difficult to complete correctly, especially for new users. There’s a steep learning curve to get a valid configuration up and running.

• Additionally, the tool reads configuration values from both environment variables and the config file, often without clear precedence rules or documentation. This causes confusion and unintended behavior when values conflict or are overridden unexpectedly.

• Lack of Validation for minio.rootPath

• If the minio.rootPath field is misconfigured, it may lead to silent failures where the generated binlog files are empty. There is currently no error message or warning to alert the user of this issue.

• No Progress Indicator or Time Estimation

• During backup operations, users receive no feedback about the current progress or how much time is remaining. This lack of transparency can be frustrating, especially for large-scale backups.

• Incorrect Exit Code on Failure

• In some failure scenarios, the tool exits with code 0 (success) even when the operation has failed. This is problematic for scripts or tools that rely on the exit code to determine success or failure, as it can lead to silent and undetected errors.

• Confusing Parameters

• The use of overlapping or ambiguous CLI flags like --database, --collections, and --database_collections makes it difficult to understand how to specify the target of a backup or restore operation. This leads to confusion and misuse.

• Unclear Permission Requirements

• The tool requires certain permissions for accessing object storage, but these requirements are not clearly documented. Users encounter permission-denied errors without knowing what exact access policies are needed.

[LoveEachDay]

@huanghaoyuanhhy Should we add a verify check for the backup file such as collection meta file lack, data file lack or incomplete?

[huanghaoyuanhhy]

@huanghaoyuanhhy Should we add a verify check for the backup file such as collection meta file lack, data file lack or incomplete?

Based on the feedback we've received, the root cause is mostly due to incorrect minio.rootPath configuration, which leads to missing binlog files. We will prioritize adding a check for this configuration.

[Andy6132024]

If I may, I'd like to request a small feature which checks if the number of binlog files to be copied in the source bucket matches the number of copied files in the destination bucket. Thank you @huanghaoyuanhhy.