Description
What is the problem this feature will solve?
Incremental rebuilds of Node.js are slow. This is annoying for developers, especially those with less powerful machines.
I am fortunate enough to have a very powerful machine, and it's still pretty slow for me. It takes 22 seconds to do an incremental rebuild where I've changed a single Javascript file:
$ TERM=dumb ninja -j 1 -C out/Release node | ts -i '%.s'
0.020594 ninja: Entering directory `out/Release'
1.410648 [1/12] ACTION libnode: node_js2c_1613b577312726d6acf41fb14bd20601
0.001637 [2/12] STAMP obj/libnode.actions_rules_copies.stamp
9.011218 [3/12] CXX obj/gen/libnode.node_javascript.o
0.101600 [4/12] AR obj/libnode.a
0.001407 [5/12] STAMP obj/node_mksnapshot.compile_depends.stamp
0.001513 [6/12] STAMP obj/node_mksnapshot.actions_depends.stamp
2.838031 [7/12] LINK node_mksnapshot
0.001629 [8/12] STAMP obj/node.actions_depends.stamp
0.283595 [9/12] ACTION node: node_mksnapshot_9b7a2d2290b02e76d66661df74749f56
0.001580 [10/12] STAMP obj/node.actions_rules_copies.stamp
5.806897 [11/12] CXX obj/gen/node.node_snapshot.o
2.939010 [12/12] LINK nodeWhat is the feature you are proposing to solve the problem?
Not sure, some ideas:
For gen/node_javascript.cc and gen/node_snapshot.cc, we have a bunch of large generated C++ array literals, which is known to be slow. Currently compiling those two files takes 9s + 6s ~ 15s on my powerful machine. I hacked together a version which makes js2c and node_mksnapshot generate the data via inline assembly, and it made that incremental recompile ~2x as fast (12s):
$ TERM=dumb ninja -j 1 -C out/Release node | ts -i '%.s'
0.020747 ninja: Entering directory `out/Release'
0.213469 [1/12] ACTION libnode: node_js2c_1613b577312726d6acf41fb14bd20601
0.001563 [2/12] STAMP obj/libnode.actions_rules_copies.stamp
3.576886 [3/12] CXX obj/gen/libnode.node_javascript.o
0.102211 [4/12] AR obj/libnode.a
0.002206 [5/12] STAMP obj/node_mksnapshot.compile_depends.stamp
0.001503 [6/12] STAMP obj/node_mksnapshot.actions_depends.stamp
2.811231 [7/12] LINK node_mksnapshot
0.001377 [8/12] STAMP obj/node.actions_depends.stamp
0.191197 [9/12] ACTION node: node_mksnapshot_9b7a2d2290b02e76d66661df74749f56
0.001393 [10/12] STAMP obj/node.actions_rules_copies.stamp
2.070298 [11/12] CXX obj/gen/node.node_snapshot.o
2.889826 [12/12] LINK nodeI plan to upstream this change, at least for gcc/Linux. However this is platform-dependent & we'd likely need different inline assembly for each platform (or use an existing tool that abstracts this, like incbin.h).
We could try having postject inject the js2c files/snapshot data. You could also imagine building a single node executable rather than node_mksnapshot/node, and using postject to inject "hydrate" an existing node with the generated snapshot data. This could avoid doing the linking twice. It would also let us centralize on a single path for snapshot deserialization, however it may regress startup performance/memory as we'd need to parse the snapshot into our in-memory format.
And perhaps -fvisibility=hidden would help with link times? Not sure.
What alternatives have you considered?
There are some ways to improve incremental rebuild time for specific sorts of changes (like --node-builtin-modules-path or ccache), but ideally it wouldn't be slow.