Skip to content

[SYNPY-1488] Patch nested tqdm progress bars and messages to logger #1177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 7, 2025

Conversation

BryanFauble
Copy link
Member

@BryanFauble BryanFauble commented Mar 6, 2025

Problem:

  1. Nested TQDM progress bars by default overwrite other progress bars. This leads to loss of information and poor looking logs.
  2. There are a significant number of print, and sysout messages getting printed to the console. When messages are printed in this way then it will break the formatting of the tqdm progress bar.

Solution:

  1. Updating the TQDM progress bars to include leave=None. This makes it so the behavior for the progress bars it to be removed if it isn't the highest level parent progress bar. It cleans up the bars significantly.
  2. Updating many print and sysout messages to use the synapse class logger to print out exceptions, warnings, and info messages.

Testing:

  1. The following are 2 short clips that show the console log during the creation, querying, and update of a FileView within Synapse during an integration test run:

Before Changes: https://www.loom.com/share/b4594dc4f83142c784f43aa7561d9652?sid=99de514e-9177-4453-b161-54c928af2cec
After Changes: https://www.loom.com/share/0dead7ad188a4bd191a8dfd2cf31ee0c?sid=365c3fd9-1a14-4c65-9499-7e120834d646

  1. I also verified that the syncFromSynapse and syncToSynapse looks correct:
from synapseclient import Synapse
from synapseutils import syncFromSynapse, syncToSynapse

syn = Synapse(debug=False)
syn.login()

syncFromSynapse(syn=syn, entity="syn60584018", path=".")
syncToSynapse(syn=syn, manifestFile="/home/ec2-user/SYNAPSE_METADATA_MANIFEST.tsv", dryRun=True)

image

image

  1. Finally, I verified that exceptions and the messages related to those exceptions occur as a result of running CLI commands:
    (Before, only the message without any stacktrace would print)
    image

(After, it will also print the stack trace along with the given exception)
image

@BryanFauble BryanFauble marked this pull request as ready for review March 6, 2025 20:42
@BryanFauble BryanFauble requested a review from a team as a code owner March 6, 2025 20:42
Copy link
Contributor

@BWMac BWMac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, progress bar looks good in the video and when I run this branch locally.

@@ -5129,22 +5130,27 @@ def _waitForAsync(self, uri, request, endpoint=None):
# https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/asynch/AsynchronousJobStatus.html
sleep = self.table_query_sleep
start_time = time.time()
lastMessage, lastProgress, lastTotal, progressed = "", 0, 1, False
lastMessage, lastProgress, lastTotal = "", 0, 1
Copy link
Contributor

@BWMac BWMac Mar 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an opportunity to switch some variables/functions over to camel-case? or are we just leaving all of the older code alone with respect to formatting/style?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this PR, my focus was solely on ensuring that things use tqdm and the logger.

This code will eventually need to be patched, but a significant portion of it will also be deprecated or removed. I have a ticket to go through and deprecate everything that has a replacement, as well as to take stock of the items that don’t.

@BryanFauble BryanFauble merged commit 403ea13 into develop Mar 7, 2025
28 checks passed
@BryanFauble BryanFauble deleted the synpy-1488-progress-bars-and-log-messages branch March 7, 2025 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants