Skip to content

Commit 009d4e8

Browse files
pflookypflooky
authored
Merge pull request #124 from bitol-io/dev
v3.0.2 release
2 parents f884de4 + acddac5 commit 009d4e8

24 files changed

+2521
-42
lines changed

‎CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,14 @@ image: "https://raw.githubusercontent.com/bitol-io/artwork/main/horizontal/color
66

77
This document tracks the history and evolution of the **Open Data Contract Standard**.
88

9+
# v3.0.2 - 2024-03-31 - APPROVED
10+
11+
* Added field `physicalName` for the properties in JSON schema.
12+
* Explicitly specifies `YYYY-MM-DDTHH:mm:ss.SSSZ` for default date format.
13+
* Added field `name` team members in JSON schema and docs.
14+
* Added field `description` team members in JSON schema and docs.
15+
* Fixed Athena Server required property name from `staging_dir` to `stagingDir`
16+
917
# v3.0.1 - 2024-12-22 - APPROVED
1018

1119
* Added field `authoritativeDefinitions` into JSON schema

‎README.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Welcome!
1313
Thanks for your interest and for taking the time to come here! ❤️
1414

1515
## Executive summary
16-
This standard describes a structure for a **data contract**. Its current version is v3.0.1. It is available for you as an Apache 2.0 license. Contributions are welcome!
16+
This standard describes a structure for a **data contract**. Its current version is v3.0.2. It is available for you as an Apache 2.0 license. Contributions are welcome!
1717

1818
## Discover the open standard
1919
A reader-friendly version of the standard can be found on its [dedicated site](https://bitol-io.github.io/open-data-contract-standard/).
@@ -56,6 +56,20 @@ Check out the [CONTRIBUTING](./CONTRIBUTING.md) page.
5656

5757
## More
5858

59+
### Citation
60+
61+
If you need to cite this standard, you can use the following BibTeX entry:
62+
63+
```bibtex
64+
@manual{ODCS2025,
65+
title = {Open Data Contract Standard (ODCS)},
66+
author = {{Bitol}},
67+
organization = {LF AI \& Data Foundation},
68+
year = {2025},
69+
url = {https://bitol-io.github.io/open-data-contract-standard}
70+
}
71+
```
72+
5973
### History
6074
Formerly known as the data contract template, this standard is used to implement Data Mesh at [PayPal](https://about.pypl.com/). Starting with v2.2.0, it is maintained by a 501c6 non-profit organization called [AIDA User Group (Artificial Intelligence, Data, and Analytics User Group)](https://aidaug.org). On November 30th, 2023, [AIDA User Group](https://aidaug.org) and the [Linux Foundation AI & Data](https://lfaidata.foundation/) joined forces to create [Bitol](https://bitol.io). Bitol englobes ODCS and future standards & tools.
6175

‎docs/README.md

Lines changed: 65 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ This section contains general information about the contract.
4343
### Example
4444

4545
```YAML
46-
apiVersion: v3.0.1 # Standard version
46+
apiVersion: v3.0.2 # Standard version
4747
kind: DataContract
4848

4949
id: 53581432-6c55-4ba2-a65f-72344a91553a
@@ -66,7 +66,7 @@ tags: ['finance']
6666
6767
| Key | UX label | Required | Description |
6868
|--------------------------------------|---------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
69-
| apiVersion | Standard version | Yes | Version of the standard used to build data contract. Default value is `v3.0.1`. |
69+
| apiVersion | Standard version | Yes | Version of the standard used to build data contract. Default value is `v3.0.2`. |
7070
| kind | Kind | Yes | The kind of file this is. Valid value is `DataContract`. |
7171
| id | ID | Yes | A unique identifier used to reduce the risk of dataset name collisions, such as a UUID. |
7272
| name | Name | No | Name of the data contract. |
@@ -243,9 +243,9 @@ Some keys are more applicable when the described property is a column.
243243
|--------------------------|------------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
244244
| primaryKey | Primary Key | No | Boolean value specifying whether the field is primary or not. Default is false. |
245245
| primaryKeyPosition | Primary Key Position | No | If field is a primary key, the position of the primary key element. Starts from 1. Example of `account_id, name` being primary key columns, `account_id` has primaryKeyPosition 1 and `name` primaryKeyPosition 2. Default to -1. |
246-
| logicalType | Logical Type | Yes | The logical field datatype. One of `string`, `date`, `number`, `integer`, `object`, `array` or `boolean`. |
246+
| logicalType | Logical Type | No | The logical field datatype. One of `string`, `date`, `number`, `integer`, `object`, `array` or `boolean`. |
247247
| logicalTypeOptions | Logical Type Options | No | Additional optional metadata to describe the logical type. See [here](#logical-type-options) for more details about supported options for each `logicalType`. |
248-
| physicalType | Physical Type | Yes | The physical element data type in the data source. For example, VARCHAR(2), DOUBLE, INT. |
248+
| physicalType | Physical Type | No | The physical element data type in the data source. For example, VARCHAR(2), DOUBLE, INT. |
249249
| description | Description | No | Description of the element. |
250250
| required | Required | No | Indicates if the element may contain Null values; possible values are true and false. Default is false. |
251251
| unique | Unique | No | Indicates if the element contains unique values; possible values are true and false. Default is false. |
@@ -270,7 +270,7 @@ Additional metadata options to more accurately define the data type.
270270
| array | maxItems | Maximum Items | No | Maximum number of items. |
271271
| array | minItems | Minimum Items | No | Minimum number of items. |
272272
| array | uniqueItems | Unique Items | No | If set to true, all items in the array are unique. |
273-
| date | format | Format | No | Format of the date. Follows the format as prescribed by [JDK DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html). For example, format 'yyyy-MM-dd'. |
273+
| date | format | Format | No | Format of the date. Follows the format as prescribed by [JDK DateTimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html). Default value is using ISO 8601: 'YYYY-MM-DDTHH:mm:ss.SSSZ'. For example, format 'yyyy-MM-dd'. |
274274
| date | exclusiveMaximum | Exclusive Maximum | No | If set to true, all values are strictly less than the maximum value (values < maximum). Otherwise, less than or equal to the maximum value (values <= maximum). |
275275
| date | exclusiveMinimum | Exclusive Minimum | No | If set to true, all values are strictly greater than the minimum value (values > minimum). Otherwise, greater than or equal to the minimum value (values >= minimum). |
276276
| date | maximum | Maximum | No | All date values are less than or equal to this value (values <= maximum). |
@@ -289,6 +289,53 @@ Additional metadata options to more accurately define the data type.
289289
| string | minLength | Minimum Length | No | Minimum length of the string. |
290290
| string | pattern | Pattern | No | Regular expression pattern to define valid value. Follows regular expression syntax from ECMA-262 (https://262.ecma-international.org/5.1/#sec-15.10.1). |
291291

292+
#### Expressing Date / Datetime / Timezone information
293+
294+
Given the complexity of handling various date and time formats (e.g., date, datetime, time, timestamp, timestamp with and without timezone), the existing `logicalType` options currently support only `date`. To specify additional temporal details, `logicalType` should be used in conjunction with `logicalTypeOptions.format` or `physicalType` to define the desired format. Using `physicalType` allows for definition of your data-source specific data type.
295+
296+
``` yaml
297+
version: 1.0.0
298+
kind: DataContract
299+
id: 53581432-6c55-4ba2-a65f-72344a91553a
300+
status: active
301+
name: date_example
302+
apiVersion: v3.0.2
303+
schema:
304+
# Date Only
305+
- name: event_date
306+
logicalType: date
307+
logicalTypeOptions:
308+
- format: "yyyy-MM-dd"
309+
examples:
310+
- "2024-07-10"
311+
312+
# Date & Time (UTC)
313+
- name: created_at
314+
logicalType: date
315+
logicalTypeOptions:
316+
- format: "yyyy-MM-ddTHH:mm:ssZ"
317+
examples:
318+
- "2024-03-10T14:22:35Z"
319+
320+
# Time Only
321+
- name: event_start_time
322+
logicalType: date
323+
logicalTypeOptions:
324+
- format: "HH:mm:ss"
325+
examples:
326+
- "08:30:00"
327+
328+
# Physical Type with Date & Time (UTC)
329+
- name: event_date
330+
logicalType: date
331+
physicalType: DATETIME
332+
logicalTypeOptions:
333+
- format: yyyy-MM-ddTHH:mm:ssZ"
334+
examples:
335+
- "2024-03-10T14:22:35Z"
336+
337+
```
338+
292339
### Authoritative definitions
293340

294341
Reference to an external definition on element logic or values.
@@ -595,7 +642,7 @@ team:
595642
dateIn: 2022-10-01
596643
- username: daustin
597644
role: Owner
598-
comment: Keeper of the grail
645+
description: Keeper of the grail
599646
name: David Austin
600647
dateIn: 2022-10-01
601648
```
@@ -607,6 +654,8 @@ The UX label is the label used in the UI and other user experiences.
607654
|-------------------------|----------------------|----------|--------------------------------------------------------------------------------------------|
608655
| team | Team | No | Object |
609656
| team.username | Username | No | The user's username or email. |
657+
| team.name | Name | No | The user's name. |
658+
| team.description | Description | No | The user's name. |
610659
| team.role | Role | No | The user's job role; Examples might be owner, data steward. There is no limit on the role. |
611660
| team.dateIn | Date In | No | The date when the user joined the team. |
612661
| team.dateOut | Date Out | No | The date when the user ceased to be part of the team. |
@@ -722,7 +771,8 @@ Each server in the schema has the following structure:
722771

723772
```yaml
724773
servers:
725-
- type: <server-type>
774+
- server: my-server-name
775+
type: <server-type>
726776
description: <server-description>
727777
environment: <server-environment>
728778
<server-type-specific-fields> # according to the server type
@@ -734,11 +784,14 @@ servers:
734784

735785
#### Common Server Properties
736786

737-
- **type**: The type of server. Valid values include various server technologies like `athena`, `bigquery`, `postgresql`, etc.
738-
- **description**: A description of the server.
739-
- **environment**: The environment where the server operates (e.g., `prod`, `dev`, `uat`). There are no set values.
740-
- **roles**: An optional array of roles that have access to the server.
741-
- **customProperties**: Any additional custom properties specific to the server that are not part of the standard.
787+
| Key | UX label | Required | Description |
788+
|------------------|-------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
789+
| server | Server | Yes | Identifier of the server. |
790+
| type | Type | Yes | Type of the server. Can be one of: api, athena, azure, bigquery, clickhouse, databricks, denodo, dremio, duckdb, glue, cloudsql, db2, informix, kafka, kinesis, local, mysql, oracle, postgresql, postgres, presto, pubsub, redshift, s3, sftp, snowflake, sqlserver, synapse, trino, vertica, custom. |
791+
| description | Description | No | Description of the server. |
792+
| environment | Environment | No | Environment of the server. Examples includes: prod, preprod, dev, uat. |
793+
| roles | Roles | No | List of roles that have access to the server. Check [roles](#roles) section for more details. |
794+
| customProperties | Custom Properties | No | Custom properties that are not part of the standard. |
742795

743796
### Specific Server Properties
744797

‎docs/examples/all/full-example.odcs.yaml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ description:
1616
tenant: ClimateQuantumInc
1717

1818
kind: DataContract
19-
apiVersion: v3.0.1 # Standard version (follows semantic versioning)
19+
apiVersion: v3.0.2 # Standard version (follows semantic versioning)
2020

2121
# Infrastructure & servers
2222
servers:
@@ -32,6 +32,7 @@ schema:
3232
- name: tbl
3333
physicalName: tbl_1
3434
physicalType: table
35+
businessName: Core Payment Metrics
3536
description: Provides core payment metrics
3637
authoritativeDefinitions:
3738
- url: https://catalog.data.gov/dataset/air-quality
@@ -41,7 +42,8 @@ schema:
4142
tags: [ 'finance', 'payments']
4243
dataGranularityDescription: Aggregation on columns txn_ref_dt, pmt_txn_id
4344
properties:
44-
- name: txn_ref_dt
45+
- name: transaction_reference_date
46+
physicalName: txn_ref_dt
4547
primaryKey: false
4648
primaryKeyPosition: -1
4749
businessName: transaction reference date
@@ -152,7 +154,7 @@ team:
152154
dateIn: "2022-10-01"
153155
- username: daustin
154156
role: Owner
155-
comment: Keeper of the grail
157+
description: Keeper of the grail
156158
dateIn: "2022-10-01"
157159

158160

‎docs/examples/data-types/all-data-types.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ id: 53581432-6c55-4ba2-a65f-72344a91553a
44
status: active
55
name: my_table
66
dataProduct: my_quantum
7-
apiVersion: v3.0.1
7+
apiVersion: v3.0.2
88
schema:
99
- name: transactions_tbl
1010
description: Provides core payment metrics

‎docs/examples/fundamentals/table-column-description.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: DataContract
33
id: 53581432-6c55-4ba2-a65f-72344a91553a
44
status: active
55
name: my_quantum
6-
apiVersion: v3.0.1
6+
apiVersion: v3.0.2
77
schema:
88
- name: tbl
99
description: Provides core payment metrics

‎docs/examples/quality/column-accuracy.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ id: 53581432-6c55-4ba2-a65f-72344a91553a
44
status: active
55
name: my_table
66
dataProduct: my_quantum
7-
apiVersion: v3.0.1
7+
apiVersion: v3.0.2
88
schema:
99
- name: Air_Quality
1010
description: Air quality of the city of New York

‎docs/examples/quality/column-completeness.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
version: 1.0.0
2-
apiVersion: v3.0.1
2+
apiVersion: v3.0.2
33
kind: DataContract
44
id: 53581432-6c55-4ba2-a65f-72344a91553a
55
status: active

‎docs/examples/quality/column-custom.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
version: 1.0.0
2-
apiVersion: v3.0.1
2+
apiVersion: v3.0.2
33
kind: DataContract
44
id: 53581432-6c55-4ba2-a65f-72344a91553a
55
status: active

‎docs/examples/quality/column-validity.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
version: 1.0.0
2-
apiVersion: v3.0.1
2+
apiVersion: v3.0.2
33
kind: DataContract
44
id: 53581432-6c55-4ba2-a65f-72344a91553a
55
status: active

‎docs/examples/roles/service-and-operational-roles.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ status: active
55
name: my_table
66
dataProduct: my_quantum
77
schema: []
8-
apiVersion: v3.0.1
8+
apiVersion: v3.0.2
99
roles:
1010
- role: microstrategy_user_opr
1111
access: read

‎docs/examples/schema/all-schema-types.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: DataContract
33
id: 53581432-6c55-4ba2-a65f-72344a91553a
44
status: active
55
name: my_quantum
6-
apiVersion: v3.0.1
6+
apiVersion: v3.0.2
77
schema:
88
- name: tbl
99
logicalType: object

‎docs/examples/schema/kafka-schema.odcs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
apiVersion: v3.0.1
1+
apiVersion: v3.0.2
22
kind: DataContract
33
id: orders
44
status: development

0 commit comments

Comments
 (0)