Skip to content

Commit 23b9b16

Browse files
Ericson2314Ericson2314
committed
Introduce top-level structuredAttrs field in JSON derivation format
Makes the behavoral change of NixOS#13263 without the underlying refactor. Hopefully this clearly safe from a perf and GC perspective, and will make it easier to benchmark NixOS#13263.
1 parent b3c1b70 commit 23b9b16

File tree

10 files changed

+173
-25
lines changed

10 files changed

+173
-25
lines changed

‎doc/manual/source/language/advanced-attributes.md

Lines changed: 7 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -53,23 +53,13 @@ Derivations can declare some infrequently used optional attributes.
5353
5454
- [`__structuredAttrs`]{#adv-attr-structuredAttrs}\
5555
If the special attribute `__structuredAttrs` is set to `true`, the other derivation
56-
attributes are serialised into a file in JSON format. The environment variable
57-
`NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build
58-
and a [`nix-shell`](../command-ref/nix-shell.md). This obviates the need for
59-
[`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions,
60-
unlike process environments.
61-
62-
It also makes it possible to tweak derivation settings in a structured way; see
63-
[`outputChecks`](#adv-attr-outputChecks) for example.
64-
65-
As a convenience to Bash builders,
66-
Nix writes a script that initialises shell variables
67-
corresponding to all attributes that are representable in Bash. The
68-
environment variable `NIX_ATTRS_SH_FILE` points to the exact
69-
location of the script, both in a build and a
70-
[`nix-shell`](../command-ref/nix-shell.md). This includes non-nested
71-
(associative) arrays. For example, the attribute `hardening.format = true`
72-
ends up as the Bash associative array element `${hardening[format]}`.
56+
attributes are serialised into a file in JSON format.
57+
58+
This obviates the need for [`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions, unlike process environments.
59+
It also makes it possible to tweak derivation settings in a structured way;
60+
see [`outputChecks`](#adv-attr-outputChecks) for example.
61+
62+
See the [corresponding section in the derivation page](@docroot@/store/derivation/index.md#structured-attrs) for further details.
7363
7464
> **Warning**
7565
>

‎doc/manual/source/protocols/json/derivation.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,3 +91,7 @@ is a JSON object with the following fields:
9191
9292
* `env`:
9393
The environment passed to the `builder`.
94+
95+
* `structuredAttrs`:
96+
[Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
97+
Structured attributes are JSON, and thus embedded as-is.

‎doc/manual/source/store/derivation/index.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,17 @@ See [Wikipedia](https://en.wikipedia.org/wiki/Argv) for details.
138138

139139
Environment variables which will be passed to the [builder](#builder) executable.
140140

141+
#### Structured Attributes {#structured-attrs}
142+
143+
Nix also has special support for embedding JSON in the derivations.
144+
145+
The environment variable `NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
146+
147+
As a convenience to Bash builders, Nix writes a script that initialises shell variables corresponding to all attributes that are representable in Bash.
148+
The environment variable `NIX_ATTRS_SH_FILE` points to the exact location of the script, both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
149+
This includes non-nested (associative) arrays.
150+
For example, the attribute `hardening.format = true` ends up as the Bash associative array element `${hardening[format]}`.
151+
141152
### Placeholders
142153

143154
Placeholders are opaque values used within the [process creation fields] to [store objects] for which we don't yet know [store path]s.

‎src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs-defaults.json

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
],
66
"builder": "/bin/bash",
77
"env": {
8-
"__json": "{\"builder\":\"/bin/bash\",\"name\":\"advanced-attributes-structured-attrs-defaults\",\"outputHashAlgo\":\"sha256\",\"outputHashMode\":\"recursive\",\"outputs\":[\"out\",\"dev\"],\"system\":\"my-system\"}",
98
"dev": "/02qcpld1y6xhs5gz9bchpxaw0xdhmsp5dv88lh25r2ss44kh8dxz",
109
"out": "/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9"
1110
},
@@ -22,5 +21,16 @@
2221
"method": "nar"
2322
}
2423
},
24+
"structuredAttrs": {
25+
"builder": "/bin/bash",
26+
"name": "advanced-attributes-structured-attrs-defaults",
27+
"outputHashAlgo": "sha256",
28+
"outputHashMode": "recursive",
29+
"outputs": [
30+
"out",
31+
"dev"
32+
],
33+
"system": "my-system"
34+
},
2535
"system": "my-system"
2636
}

‎src/libstore-tests/data/derivation/ca/advanced-attributes-structured-attrs.json

Lines changed: 57 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
],
66
"builder": "/bin/bash",
77
"env": {
8-
"__json": "{\"__darwinAllowLocalNetworking\":true,\"__impureHostDeps\":[\"/usr/bin/ditto\"],\"__noChroot\":true,\"__sandboxProfile\":\"sandcastle\",\"allowSubstitutes\":false,\"builder\":\"/bin/bash\",\"exportReferencesGraph\":{\"refs1\":[\"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9\"],\"refs2\":[\"/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv\"]},\"impureEnvVars\":[\"UNICORN\"],\"name\":\"advanced-attributes-structured-attrs\",\"outputChecks\":{\"bin\":{\"disallowedReferences\":[\"/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g\"],\"disallowedRequisites\":[\"/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8\"]},\"dev\":{\"maxClosureSize\":5909,\"maxSize\":789},\"out\":{\"allowedReferences\":[\"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9\"],\"allowedRequisites\":[\"/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z\"]}},\"outputHashAlgo\":\"sha256\",\"outputHashMode\":\"recursive\",\"outputs\":[\"out\",\"bin\",\"dev\"],\"preferLocalBuild\":true,\"requiredSystemFeatures\":[\"rainbow\",\"uid-range\"],\"system\":\"my-system\"}",
98
"bin": "/04f3da1kmbr67m3gzxikmsl4vjz5zf777sv6m14ahv22r65aac9m",
109
"dev": "/02qcpld1y6xhs5gz9bchpxaw0xdhmsp5dv88lh25r2ss44kh8dxz",
1110
"out": "/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9"
@@ -44,5 +43,62 @@
4443
"method": "nar"
4544
}
4645
},
46+
"structuredAttrs": {
47+
"__darwinAllowLocalNetworking": true,
48+
"__impureHostDeps": [
49+
"/usr/bin/ditto"
50+
],
51+
"__noChroot": true,
52+
"__sandboxProfile": "sandcastle",
53+
"allowSubstitutes": false,
54+
"builder": "/bin/bash",
55+
"exportReferencesGraph": {
56+
"refs1": [
57+
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9"
58+
],
59+
"refs2": [
60+
"/nix/store/qnml92yh97a6fbrs2m5qg5cqlc8vni58-bar.drv"
61+
]
62+
},
63+
"impureEnvVars": [
64+
"UNICORN"
65+
],
66+
"name": "advanced-attributes-structured-attrs",
67+
"outputChecks": {
68+
"bin": {
69+
"disallowedReferences": [
70+
"/0nyw57wm2iicnm9rglvjmbci3ikmcp823czdqdzdcgsnnwqps71g"
71+
],
72+
"disallowedRequisites": [
73+
"/07f301yqyz8c6wf6bbbavb2q39j4n8kmcly1s09xadyhgy6x2wr8"
74+
]
75+
},
76+
"dev": {
77+
"maxClosureSize": 5909,
78+
"maxSize": 789
79+
},
80+
"out": {
81+
"allowedReferences": [
82+
"/164j69y6zir9z0339n8pjigg3rckinlr77bxsavzizdaaljb7nh9"
83+
],
84+
"allowedRequisites": [
85+
"/0nr45p69vn6izw9446wsh9bng9nndhvn19kpsm4n96a5mycw0s4z"
86+
]
87+
}
88+
},
89+
"outputHashAlgo": "sha256",
90+
"outputHashMode": "recursive",
91+
"outputs": [
92+
"out",
93+
"bin",
94+
"dev"
95+
],
96+
"preferLocalBuild": true,
97+
"requiredSystemFeatures": [
98+
"rainbow",
99+
"uid-range"
100+
],
101+
"system": "my-system"
102+
},
47103
"system": "my-system"
48104
}

‎src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs-defaults.json

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
],
66
"builder": "/bin/bash",
77
"env": {
8-
"__json": "{\"builder\":\"/bin/bash\",\"name\":\"advanced-attributes-structured-attrs-defaults\",\"outputs\":[\"out\",\"dev\"],\"system\":\"my-system\"}",
98
"dev": "/nix/store/8bazivnbipbyi569623skw5zm91z6kc2-advanced-attributes-structured-attrs-defaults-dev",
109
"out": "/nix/store/f8f8nvnx32bxvyxyx2ff7akbvwhwd9dw-advanced-attributes-structured-attrs-defaults"
1110
},
@@ -20,5 +19,14 @@
2019
"path": "/nix/store/f8f8nvnx32bxvyxyx2ff7akbvwhwd9dw-advanced-attributes-structured-attrs-defaults"
2120
}
2221
},
22+
"structuredAttrs": {
23+
"builder": "/bin/bash",
24+
"name": "advanced-attributes-structured-attrs-defaults",
25+
"outputs": [
26+
"out",
27+
"dev"
28+
],
29+
"system": "my-system"
30+
},
2331
"system": "my-system"
2432
}

‎src/libstore-tests/data/derivation/ia/advanced-attributes-structured-attrs.json

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
],
66
"builder": "/bin/bash",
77
"env": {
8-
"__json": "{\"__darwinAllowLocalNetworking\":true,\"__impureHostDeps\":[\"/usr/bin/ditto\"],\"__noChroot\":true,\"__sandboxProfile\":\"sandcastle\",\"allowSubstitutes\":false,\"builder\":\"/bin/bash\",\"exportReferencesGraph\":{\"refs1\":[\"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo\"],\"refs2\":[\"/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv\"]},\"impureEnvVars\":[\"UNICORN\"],\"name\":\"advanced-attributes-structured-attrs\",\"outputChecks\":{\"bin\":{\"disallowedReferences\":[\"/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar\"],\"disallowedRequisites\":[\"/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev\"]},\"dev\":{\"maxClosureSize\":5909,\"maxSize\":789},\"out\":{\"allowedReferences\":[\"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo\"],\"allowedRequisites\":[\"/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev\"]}},\"outputs\":[\"out\",\"bin\",\"dev\"],\"preferLocalBuild\":true,\"requiredSystemFeatures\":[\"rainbow\",\"uid-range\"],\"system\":\"my-system\"}",
98
"bin": "/nix/store/33qms3h55wlaspzba3brlzlrm8m2239g-advanced-attributes-structured-attrs-bin",
109
"dev": "/nix/store/wyfgwsdi8rs851wmy1xfzdxy7y5vrg5l-advanced-attributes-structured-attrs-dev",
1110
"out": "/nix/store/7cxy4zx1vqc885r4jl2l64pymqbdmhii-advanced-attributes-structured-attrs"
@@ -41,5 +40,60 @@
4140
"path": "/nix/store/7cxy4zx1vqc885r4jl2l64pymqbdmhii-advanced-attributes-structured-attrs"
4241
}
4342
},
43+
"structuredAttrs": {
44+
"__darwinAllowLocalNetworking": true,
45+
"__impureHostDeps": [
46+
"/usr/bin/ditto"
47+
],
48+
"__noChroot": true,
49+
"__sandboxProfile": "sandcastle",
50+
"allowSubstitutes": false,
51+
"builder": "/bin/bash",
52+
"exportReferencesGraph": {
53+
"refs1": [
54+
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
55+
],
56+
"refs2": [
57+
"/nix/store/vj2i49jm2868j2fmqvxm70vlzmzvgv14-bar.drv"
58+
]
59+
},
60+
"impureEnvVars": [
61+
"UNICORN"
62+
],
63+
"name": "advanced-attributes-structured-attrs",
64+
"outputChecks": {
65+
"bin": {
66+
"disallowedReferences": [
67+
"/nix/store/r5cff30838majxk5mp3ip2diffi8vpaj-bar"
68+
],
69+
"disallowedRequisites": [
70+
"/nix/store/9b61w26b4avv870dw0ymb6rw4r1hzpws-bar-dev"
71+
]
72+
},
73+
"dev": {
74+
"maxClosureSize": 5909,
75+
"maxSize": 789
76+
},
77+
"out": {
78+
"allowedReferences": [
79+
"/nix/store/p0hax2lzvjpfc2gwkk62xdglz0fcqfzn-foo"
80+
],
81+
"allowedRequisites": [
82+
"/nix/store/z0rjzy29v9k5qa4nqpykrbzirj7sd43v-foo-dev"
83+
]
84+
}
85+
},
86+
"outputs": [
87+
"out",
88+
"bin",
89+
"dev"
90+
],
91+
"preferLocalBuild": true,
92+
"requiredSystemFeatures": [
93+
"rainbow",
94+
"uid-range"
95+
],
96+
"system": "my-system"
97+
},
4498
"system": "my-system"
4599
}

‎src/libstore/derivations.cc

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1333,6 +1333,11 @@ nlohmann::json Derivation::toJSON(const StoreDirConfig & store) const
13331333
res["args"] = args;
13341334
res["env"] = env;
13351335

1336+
if (auto it = env.find("__json"); it != env.end()) {
1337+
res["env"].erase("__json");
1338+
res["structuredAttrs"] = nlohmann::json::parse(it->second);
1339+
}
1340+
13361341
return res;
13371342
}
13381343

@@ -1396,7 +1401,17 @@ Derivation Derivation::fromJSON(
13961401
res.platform = getString(valueAt(json, "system"));
13971402
res.builder = getString(valueAt(json, "builder"));
13981403
res.args = getStringList(valueAt(json, "args"));
1399-
res.env = getStringMap(valueAt(json, "env"));
1404+
1405+
auto envJson = valueAt(json, "env");
1406+
try {
1407+
res.env = getStringMap(envJson);
1408+
} catch (Error & e) {
1409+
e.addTrace({}, "while reading key 'env'");
1410+
throw;
1411+
}
1412+
1413+
if (auto structuredAttrs = get(json, "structuredAttrs"))
1414+
res.env.insert_or_assign("__json", structuredAttrs->dump());
14001415

14011416
return res;
14021417
}

‎src/libstore/unix/build/derivation-builder.cc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -589,10 +589,10 @@ static void replaceValidPath(const Path & storePath, const Path & tmpPath)
589589
way first. We'd better not be interrupted here, because if
590590
we're repairing (say) Glibc, we end up with a broken system. */
591591
Path oldPath;
592-
592+
593593
if (pathExists(storePath)) {
594594
// why do we loop here?
595-
// although makeTempPath should be unique, we can't
595+
// although makeTempPath should be unique, we can't
596596
// guarantee that.
597597
do {
598598
oldPath = makeTempPath(storePath, ".old");

‎tests/functional/structured-attrs.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,4 +50,4 @@ expectStderr 0 nix-instantiate --expr "$hackyExpr" --eval --strict | grepQuiet "
5050

5151
# Check it works with the expected structured attrs
5252
hacky=$(nix-instantiate --expr "$hackyExpr")
53-
nix derivation show "$hacky" | jq --exit-status '."'"$hacky"'".env.__json | fromjson | . == {"a": 1}'
53+
nix derivation show "$hacky" | jq --exit-status '."'"$hacky"'".structuredAttrs | . == {"a": 1}'

0 commit comments

Comments
 (0)