Skip to content

Commit 5b36421

Browse files
BryanFaubleBryanFauble
authored
[SYNPY-1577] Adds a new SubmissionView model class that represents Synapse Submission Views (#1192)
* Adds a new SubmissionView model class that represents Synapse Submission Views, allowing users to create, manage, and query collections of evaluation queue submissions. This provides a tabular view of submissions similar to other view types in Synapse.
1 parent a33dc9a commit 5b36421

File tree

21 files changed

+3084
-48
lines changed

21 files changed

+3084
-48
lines changed

‎.readthedocs.yaml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,4 +22,6 @@ python:
2222
# See https://docs.readthedocs.com/platform/stable/config-file/v2.html#search for more options
2323
search:
2424
ranking:
25-
reference/*: -1
25+
reference/*: -3
26+
reference/experimental/*: -2
27+
reference/experimental/sync/*: -1

‎docs/reference/experimental/async/submissionview.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# SubmissionView
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API reference
8+
9+
::: synapseclient.models.SubmissionView
10+
options:
11+
inherited_members: true
12+
members:
13+
- store_async
14+
- get_async
15+
- delete_async
16+
- query_async
17+
- query_part_mask_async
18+
- snapshot_async
19+
- add_column
20+
- reorder_column
21+
- delete_column
22+
- get_acl_async
23+
- get_permissions_async
24+
- set_permissions_async
25+
---

‎docs/reference/experimental/sync/submissionview.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# SubmissionView
2+
3+
Contained within this file are experimental interfaces for working with the Synapse Python
4+
Client. Unless otherwise noted these interfaces are subject to change at any time. Use
5+
at your own risk.
6+
7+
## API reference
8+
9+
::: synapseclient.models.SubmissionView
10+
options:
11+
inherited_members: true
12+
members:
13+
- store
14+
- get
15+
- delete
16+
- query
17+
- query_part_mask
18+
- snapshot
19+
- add_column
20+
- reorder_column
21+
- delete_column
22+
- get_acl
23+
- get_permissions
24+
- set_permissions
25+
---

‎docs/tutorials/python/dataset.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,3 +123,4 @@ Finally, let's save a snapshot of the dataset. This creates a read-only version
123123
- [Column][synapseclient.models.Column]
124124
- [syn.login][synapseclient.Synapse.login]
125125
- [Project](../../reference/experimental/sync/project.md)
126+
- [query examples](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/web/controller/TableExamples.html)

‎docs/tutorials/python/entityview.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,3 +138,4 @@ Synapse web UI. It should look similar to:
138138
- [Column][synapseclient.models.Column]
139139
- [syn.login][synapseclient.Synapse.login]
140140
- [Project](../../reference/experimental/sync/project.md)
141+
- [query examples](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/web/controller/TableExamples.html)

‎docs/tutorials/python/materializedview.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,3 +169,4 @@ Results from the materialized view with UNION:
169169
- [Column][synapseclient.models.Column]
170170
- [Project][synapseclient.models.Project]
171171
- [syn.login][synapseclient.Synapse.login]
172+
- [query examples](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/web/controller/TableExamples.html)

‎docs/tutorials/python/submissionview.md

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# SubmissionView
2+
3+
SubmissionViews in Synapse allow you to aggregate and query submissions from one or more
4+
evaluation queues in a tabular format. These views are useful for managing submissions in
5+
challenges, tracking scoring progress, and analyzing submission data across different
6+
evaluation queues.
7+
8+
This tutorial will walk you through the basics of working with SubmissionViews
9+
using the Synapse Python client.
10+
11+
## Tutorial Purpose
12+
In this tutorial, you will:
13+
14+
1. Set up and create an evaluation queue
15+
2. Create a SubmissionView for the evaluation queue
16+
3. Create and submit a file to the evaluation queue
17+
4. Query and update the submission status
18+
5. Modify the SubmissionView scope
19+
6. Create a snapshot of the view
20+
7. Query the snapshot
21+
22+
## Prerequisites
23+
* This tutorial assumes that you have a Synapse project.
24+
* Pandas must also be installed as shown in the [installation documentation](../installation.md).
25+
26+
## 1. Set up and create an evaluation queue
27+
28+
Before creating a SubmissionView, we need to log in to Synapse, retrieve your project,
29+
and create an evaluation queue that will be used in the view.
30+
31+
You will want to replace `"My uniquely named project about Alzheimer's Disease"` with
32+
the name of your project.
33+
34+
```python
35+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!lines=13-44}
36+
```
37+
38+
## 2. Create a SubmissionView for the evaluation queue
39+
40+
Next, we will create a SubmissionView that includes the evaluation queue we just created
41+
in its scope. We'll also add custom columns for metrics that will be used for scoring
42+
submissions.
43+
44+
```python
45+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!lines=46-82}
46+
```
47+
48+
## 3. Create and submit a file to the evaluation queue
49+
50+
Now let's create a test file and submit it to our evaluation queue. For convenience,
51+
we'll use a temporary file that will be automatically cleaned up after execution.
52+
53+
```python
54+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!lines=84-105}
55+
```
56+
57+
## 4. Query and update the submission status
58+
59+
After submitting a file, we can query the SubmissionView to see our submission and update
60+
its status with scoring metrics.
61+
62+
Note: Due to Synapse's eventual consistency model, we need to wait briefly for the
63+
submission to appear in the view.
64+
65+
```python
66+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!lines=107-126}
67+
```
68+
69+
<details class="example">
70+
<summary>The result of querying your SubmissionView should include your submission with its details:</summary>
71+
```
72+
Query results:
73+
ROW_ID ROW_VERSION ROW_ETAG metric_A metric_B ... submitteralias entityid entityversion dockerrepositoryname dockerdigest
74+
0 9751779 0 e7c37ec7-e5e8-435d-b378-dc2d6bddad21 NaN NaN ... Participant 1 syn66272627 1 NaN NaN
75+
```
76+
77+
After updating the submission status:
78+
79+
```
80+
Submission status: SCORED
81+
```
82+
</details>
83+
84+
## 5. Modify the SubmissionView scope
85+
86+
As your challenge evolves, you might need to add more evaluation queues to your SubmissionView.
87+
Here's how to create another evaluation queue and add it to your view's scope.
88+
89+
```python
90+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!lines=128-143}
91+
```
92+
93+
## 6. Create a snapshot of the view
94+
95+
SubmissionViews support creating snapshots, which capture the state of all submissions at a
96+
specific point in time. This is useful for archiving or comparing submission states.
97+
98+
```python
99+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!lines=145-152}
100+
```
101+
102+
## 7. Query the snapshot
103+
104+
After creating a snapshot, you can query it to retrieve the state of submissions at the
105+
time the snapshot was created. This is useful for historical analysis or auditing.
106+
107+
```python
108+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!lines=153-162}
109+
```
110+
111+
## Source Code for this Tutorial
112+
113+
<details class="quote">
114+
<summary>Click to show me</summary>
115+
116+
```python
117+
{!docs/tutorials/python/tutorial_scripts/submissionview.py!}
118+
```
119+
</details>
120+
121+
## References
122+
- [SubmissionView][synapseclient.models.SubmissionView]
123+
- [Evaluation][synapseclient.evaluation]
124+
- [syn.submit][synapseclient.Synapse.submit]
125+
- [syn.getSubmissionStatus][synapseclient.Synapse.getSubmissionStatus]
126+
- [syn.getSubmission][synapseclient.Synapse.getSubmission]
127+
- [syn.getSubmissions][synapseclient.Synapse.getSubmissions]
128+
- [syn.getSubmissionBundles][synapseclient.Synapse.getSubmissionBundles]
129+
- [syn.store][synapseclient.Synapse.store]
130+
- [Column][synapseclient.models.Column]
131+
- [Project][synapseclient.models.Project]
132+
- [syn.login][synapseclient.Synapse.login]
133+
- [query examples](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/web/controller/TableExamples.html)

‎docs/tutorials/python/tutorial_scripts/submissionview.py

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
"""
2+
Here is where you'll find the code for the SubmissionView tutorial.
3+
4+
A SubmissionView allows you to aggregate and query submissions from one or more
5+
evaluation queues in a tabular format, similar to other view types in Synapse.
6+
7+
This tutorial also shows how to create an evaluation queue, submit a file to it, and
8+
update the submission status. The example uses a temporary file for the submission,
9+
which is automatically cleaned up after the script runs. You can modify the file
10+
path and name as needed.
11+
12+
"""
13+
14+
import tempfile
15+
16+
import pandas as pd
17+
18+
from synapseclient import Evaluation, Synapse
19+
from synapseclient.models import (
20+
Activity,
21+
Column,
22+
ColumnType,
23+
File,
24+
Project,
25+
SubmissionView,
26+
UsedURL,
27+
)
28+
29+
syn = Synapse()
30+
syn.login()
31+
32+
# Retrieve the project ID
33+
my_project = Project(name="My uniquely named project about Alzheimer's Disease").get()
34+
project_id = my_project.id
35+
print(f"My project ID is: {project_id}")
36+
37+
# Step 1: Set up and create an evaluation queue
38+
evaluation_name = "Test Evaluation Queue for Alzheimer conference"
39+
evaluation_description = "Evaluation queue for testing submission view"
40+
evaluation = Evaluation(
41+
name=evaluation_name, description=evaluation_description, contentSource=project_id
42+
)
43+
evaluation = syn.store(evaluation)
44+
print(f"Created evaluation queue with ID: {evaluation.id}")
45+
46+
# Step 2: Create a SubmissionView for the evaluation queue
47+
view = SubmissionView(
48+
name="SubmissionView for Alzheimer conference",
49+
parent_id=project_id,
50+
scope_ids=[evaluation.id],
51+
include_default_columns=True,
52+
columns=[
53+
Column(
54+
name="metric_A",
55+
column_type=ColumnType.DOUBLE,
56+
),
57+
Column(
58+
name="metric_B",
59+
column_type=ColumnType.DOUBLE,
60+
),
61+
],
62+
activity=Activity(
63+
name="Submission Review Analysis",
64+
description="Analysis of Q1 2025 challenge submissions",
65+
used=[
66+
UsedURL(
67+
name="Challenge Homepage",
68+
url="https://sagebionetworks.org/community/challenges-portal",
69+
)
70+
],
71+
),
72+
).store()
73+
74+
print(f"My SubmissionView ID is: {view.id}")
75+
76+
# Reorder columns for better display
77+
view.reorder_column(name="name", index=2)
78+
view.reorder_column(name="status", index=3)
79+
view.reorder_column(name="evaluationid", index=4)
80+
view.store()
81+
82+
print("Available columns in the view:", list(view.columns.keys()))
83+
84+
# Step 3: Create and submit a file to the evaluation queue
85+
with tempfile.NamedTemporaryFile(
86+
mode="w", suffix=".txt", delete=True, delete_on_close=False
87+
) as temp_file:
88+
with open(temp_file.name, "w") as opened_temp_file:
89+
opened_temp_file.write("This is a test submission file.")
90+
temp_file_path = temp_file.name
91+
92+
# Upload the temporary file to Synapse
93+
submission_file = File(
94+
path=temp_file_path, parent_id=project_id, name="Test Submission"
95+
).store()
96+
97+
# Submit the file to the evaluation queue
98+
submission = syn.submit(
99+
evaluation=evaluation,
100+
entity=submission_file,
101+
name="Test Submission",
102+
submitterAlias="Participant 1",
103+
)
104+
105+
print(f"Created submission with ID: {submission.id}")
106+
107+
# Step 4: Query and update the submission status
108+
# Query the SubmissionView to see our submission
109+
query = f"SELECT * FROM {view.id} WHERE id = '{submission.id}'"
110+
results_as_dataframe: pd.DataFrame = view.query(query=query)
111+
# Due to the eventual consistency of the system, we need to perform 2 queries
112+
results_as_dataframe: pd.DataFrame = view.query(query=query)
113+
114+
print("Query results:")
115+
print(results_as_dataframe)
116+
117+
# Update the status to indicate it's been scored
118+
submission_status = syn.getSubmissionStatus(submission=submission.id)
119+
print(f"Submission status: {submission_status.status}")
120+
submission_status.status = "SCORED"
121+
submission_status.submissionAnnotations["metric_A"] = 90
122+
submission_status.submissionAnnotations["metric_B"] = 80
123+
submission_status.score = 0.7
124+
submission_status = syn.store(submission_status)
125+
print(f"Updated submission status to: {submission_status.status}")
126+
127+
# Step 5: Modify the SubmissionView scope
128+
# First let's make sure we have the latest view from Synapse:
129+
view.get()
130+
131+
# Create another evaluation queue to demonstrate adding to the scope
132+
second_evaluation = Evaluation(
133+
name="Second Test Evaluation Queue for Alzheimer conference",
134+
description="Another evaluation queue for testing submission view",
135+
contentSource=project_id,
136+
)
137+
second_evaluation = syn.store(second_evaluation)
138+
print(f"Created second evaluation queue with ID: {second_evaluation.id}")
139+
140+
# Add the new evaluation queue to the view's scope
141+
view.scope_ids.append(second_evaluation.id)
142+
view.store() # Store the updated view
143+
print("Updated SubmissionView scope. Current scope IDs:", view.scope_ids)
144+
145+
# Step 6: Create a snapshot of the view
146+
snapshot_info = view.snapshot(
147+
comment="Initial submission review snapshot",
148+
)
149+
print("Created snapshot of the SubmissionView:")
150+
print(snapshot_info)
151+
snapshot_version = snapshot_info.snapshot_version_number
152+
print(f"Snapshot version number: {snapshot_version}")
153+
154+
# Step 7: Query the snapshot we just created
155+
# You may also get the snapshot version from the view object directly by looking at the version number
156+
# (which is the latest version of the view) and subtracting 1.
157+
# snapshot_version = view.version_number - 1
158+
159+
snapshot_query = f"SELECT * FROM {view.id}.{snapshot_version}"
160+
snapshot_results = view.query(snapshot_query)
161+
print("Query results from the snapshot:")
162+
print(snapshot_results)

‎mkdocs.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ nav:
3636
- Dataset: tutorials/python/dataset.md
3737
- Dataset Collection: tutorials/python/dataset_collection.md
3838
- Materialized View: tutorials/python/materializedview.md
39+
- Submission View: tutorials/python/submissionview.md
3940
# - Sharing Settings: tutorials/python/sharing_settings.md
4041
# - Wiki: tutorials/python/wiki.md
4142
# - Team: tutorials/python/team.md
@@ -86,6 +87,7 @@ nav:
8687
- Dataset Collection: reference/experimental/sync/dataset_collection.md
8788
- EntityView: reference/experimental/sync/entityview.md
8889
- MaterializedView: reference/experimental/sync/materializedview.md
90+
- SubmissionView: reference/experimental/sync/submissionview.md
8991
- Activity: reference/experimental/sync/activity.md
9092
- Team: reference/experimental/sync/team.md
9193
- UserProfile: reference/experimental/sync/user_profile.md
@@ -100,6 +102,7 @@ nav:
100102
- Dataset Collection: reference/experimental/async/dataset_collection.md
101103
- EntityView: reference/experimental/async/entityview.md
102104
- MaterializedView: reference/experimental/async/materializedview.md
105+
- SubmissionView: reference/experimental/async/submissionview.md
103106
- Activity: reference/experimental/async/activity.md
104107
- Team: reference/experimental/async/team.md
105108
- UserProfile: reference/experimental/async/user_profile.md

0 commit comments

Comments
 (0)