Skip to content

Commit 258297f

Browse files
authored
chore: Update doc tests (#135)
This PR just denotes the changes that were made to filetypes. See cloudquery/cloudquery#16404 for `file` destination. I think in the future we'll need to update the filetypes schema to distinguish between schemas better (possibly just rename the structs from `pkg.Spec` to `pkg.JSONSpec` and so on)
1 parent ef44afa commit 258297f

File tree

5 files changed

+400
-44
lines changed

5 files changed

+400
-44
lines changed
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Table of contents
2+
3+
* [`Spec`](#Spec)
4+
* [`Spec`](#Spec-1)
5+
* [`Spec`](#Spec-2)
6+
* [`Spec`](#Spec-3)
7+
* [`Duration`](#Duration)
8+
9+
## <a name="Spec"></a>Spec
10+
11+
* `format` (`string`) (required) (possible values: `csv`, `json`, `parquet`)
12+
13+
Output format.
14+
15+
* `format_spec` ([`Spec`](#Spec-1), [`Spec`](#Spec-2) or [`Spec`](#Spec-3)) (nullable)
16+
17+
* `compression` (`string`) (possible values: ` `, `gzip`)
18+
19+
Compression type.
20+
Empty or missing stands for no compression.
21+
22+
* `path` (`string`) (required)
23+
24+
Path template string that determines where files will be written.
25+
26+
The path supports the following placeholder variables:
27+
- `{{TABLE}}` will be replaced with the table name
28+
- `{{FORMAT}}` will be replaced with the file format, such as `csv`, `json` or `parquet`. If compression is enabled, the format will be `csv.gz`, `json.gz` etc.
29+
- `{{UUID}}` will be replaced with a random UUID to uniquely identify each file
30+
- `{{YEAR}}` will be replaced with the current year in `YYYY` format
31+
- `{{MONTH}}` will be replaced with the current month in `MM` format
32+
- `{{DAY}}` will be replaced with the current day in `DD` format
33+
- `{{HOUR}}` will be replaced with the current hour in `HH` format
34+
- `{{MINUTE}}` will be replaced with the current minute in `mm` format
35+
36+
**Note** that timestamps are in `UTC` and will be the current time at the time the file is written, not when the sync started.
37+
38+
* `no_rotate` (`boolean`) (default: `false`)
39+
40+
If set to `true`, the plugin will write to one file per table.
41+
Otherwise, for every batch a new file will be created with a different `.<UUID>` suffix.
42+
43+
* `batch_size` (`integer`) (nullable) (range: `[1,+∞)`) (default: `10000`)
44+
45+
This parameter controls the maximum amount of items may be grouped together to be written in a single write.
46+
47+
Defaults to `10000` unless `no_rotate` is `true` (will be `0` then).
48+
49+
* `batch_size_bytes` (`integer`) (nullable) (range: `[1,+∞)`) (default: `52428800`)
50+
51+
This parameter controls the maximum size of items that may be grouped together to be written in a single write.
52+
53+
Defaults to `52428800` (50 MiB) unless `no_rotate` is `true` (will be `0` then).
54+
55+
* `batch_timeout` ([`Duration`](#Duration)) (nullable) (default: `30s`)
56+
57+
This parameter controls the maximum interval between batch writes.
58+
59+
Defaults to `30s` unless `no_rotate` is `true` (will be `0s` then).
60+
61+
### <a name="Spec-1"></a>Spec
62+
63+
CloudQuery CSV file output spec.
64+
65+
* `skip_header` (`boolean`) (default: `false`)
66+
67+
Specifies if the first line of a file should be the header.
68+
69+
* `delimiter` (`string`) ([pattern](https://json-schema.org/draft/2020-12/json-schema-validation#section-6.3.3): `^.$`) (default: `,`)
70+
71+
Character that will be used as the delimiter.
72+
73+
### <a name="Spec-2"></a>Spec
74+
75+
CloudQuery JSON file output spec.
76+
77+
(`object`)
78+
79+
### <a name="Spec-3"></a>Spec
80+
81+
CloudQuery Parquet file output spec.
82+
83+
(`object`)
84+
85+
### <a name="Duration"></a>Duration
86+
87+
CloudQuery configtype.Duration
88+
89+
(`string`) ([pattern](https://json-schema.org/draft/2020-12/json-schema-validation#section-6.3.3): `^[-+]?([0-9]*(\\.[0-9]*)?[a-z]+)+$`)

jsonschema/docs/.snapshots/TestFiletypes.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,24 @@
11
# Table of contents
22

33
* [`FileSpec`](#FileSpec)
4-
* [`CSVSpec`](#CSVSpec)
5-
* [`JSONSpec`](#JSONSpec)
6-
* [`ParquetSpec`](#ParquetSpec)
4+
* [`Spec`](#Spec)
5+
* [`Spec`](#Spec-1)
6+
* [`Spec`](#Spec-2)
77

88
## <a name="FileSpec"></a>FileSpec
99

1010
* `format` (`string`) (required) (possible values: `csv`, `json`, `parquet`)
1111

1212
Output format.
1313

14-
* `format_spec` ([`CSVSpec`](#CSVSpec), [`JSONSpec`](#JSONSpec) or [`ParquetSpec`](#ParquetSpec)) (nullable)
14+
* `format_spec` ([`Spec`](#Spec), [`Spec`](#Spec-1) or [`Spec`](#Spec-2)) (nullable)
1515

1616
* `compression` (`string`) (possible values: ` `, `gzip`)
1717

1818
Compression type.
1919
Empty or missing stands for no compression.
2020

21-
### <a name="CSVSpec"></a>CSVSpec
21+
### <a name="Spec"></a>Spec
2222

2323
CloudQuery CSV file output spec.
2424

@@ -30,13 +30,13 @@
3030

3131
Character that will be used as the delimiter.
3232

33-
### <a name="JSONSpec"></a>JSONSpec
33+
### <a name="Spec-1"></a>Spec
3434

3535
CloudQuery JSON file output spec.
3636

3737
(`object`)
3838

39-
### <a name="ParquetSpec"></a>ParquetSpec
39+
### <a name="Spec-2"></a>Spec
4040

4141
CloudQuery Parquet file output spec.
4242

jsonschema/docs/docs_test.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,7 @@ func TestClickHouse(t *testing.T) {
4444
func TestFiletypes(t *testing.T) {
4545
genSnapshot(t, "testdata/filetypes.json")
4646
}
47+
48+
func TestFileDestination(t *testing.T) {
49+
genSnapshot(t, "testdata/file-destination.json")
50+
}

0 commit comments

Comments
 (0)