[SYNPY-1578] DatasetCollection OOP Model

[BWMac]

Problem:

We do not currently have an OOP model representing the Synapse DatasetCollection model. We should follow the example of the Dataset class and use the table component mixins to create one.

Solution:

This PR follows the Dataset example to create the DatasetCollection model. Users are now able to:

• Create DatasetCollections in Synapse
• Delete DatasetCollections
• Get existing DatasetCollections
• Add Datasets to their collection
• Remove Datasets from their collection
• Annotate Datasets within a collection using custom columns
• Save snapshots of DatasetCollections

Needed documentation pages and a tutorial to help users get started using DatasetCollections were added to the mkdocs site.

Testing:

• New methods were manually tested as they were implemented and as part of the process of creating the tutorial
• Unit and Integration tests were created for all new logic introduced into with the DatasetCollection class

Notes:

• The _cast_into_class_type function in entity_factory.py was getting pretty unwieldy with the amount if/elif lines that were included, but a lot of logic was repeated across many different concrete types. I refactored that function to reduce complexity and hopefully make it more maintainable. I will need to add MaterializedView to this new mapping method and resolve conflicts once [SYNPY-1579] Introduce the materialized view #1190 is merged.
• During testing, we discovered that there is a condition in the Synapse backend when you create a snapshot of a View that causes additional async jobs to run subsequent to the initial snapshot job itself. I added retry logic in the async communicator class to account for in progress async jobs that we are not in control of.

[BryanFauble]

The _cast_into_class_type function in entity_factory.py was getting pretty unwieldy with the amount if/elif lines that were included, but a lot of logic was repeated across many different concrete types. I refactored that function to reduce complexity and hopefully make it more maintainable. I will need to add MaterializedView to this new mapping method/resolve conflicts once #1190 is merged.

@BWMac I merged in the changes to develop. This branch is ready for your resolution to the merge conflicts

[Copilot]

Copilot reviewed 18 out of 18 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

tests/unit/synapseclient/models/synchronous/unit_test_dataset.py:121

• [nitpick] Consider using a less brittle error message matcher (for example, checking for a substring like 'Dataset or EntityRef') to reduce test fragility if the error message formatting changes.

with pytest.raises(ValueError, match="item must be a Dataset or EntityRef. 1 is a <class 'int'>",):

synapseclient/api/entity_factory.py:383

• [nitpick] It might help to add an inline comment clarifying that entity_to_update is reused when provided, which is particularly relevant now that the mapping includes DatasetCollection.

entity_instance = entity_to_update or entity_class()

[BryanFauble]

I left my last round of comments, great changes here. Thanks for your attention to detail on the changes.

[thomasyu888]

🔥 nice! i took a look at the tests and thanks for tackling this!