Parcel MemoryFS significantly slower than default config (importing & large files)

[davidjb]

🐛 bug report

When implementing an in-memory file system for Parcel (as per https://parceljs.org/features/parcel-api/#file-system), Parcel builds become significantly slower than the default configuration, even when caching is disabled to make for a fair comparison. There is a worked example in a repo below; I've attempted to follow the documentation as much as possible (noting the documentation needs fixing as per #9050 (comment)).

It appears that the issue relates to the use of import, which may in turn be related to the module loader or else file access under the hood. If an empty or basic file (e.g. console.log('foobar')) is built without any imports, then the in-memory implementation is vastly quicker (177ms vs 345ms). However, with each import, the in-memory implementation gets slower and slower. In the example below, just 1 import from a plugin from a library called amCharts takes 4x longer (7.3s vs 1.8s) in memory compared to the default.

As I mention below though, the issue isn't isolated to this one library. Mocking out a single JavaScript file full of noop lines of 1;, the default implementation is quicker than using MemoryFS.

🎛 Configuration (.babelrc, package.json, cli command)

See https://gist.github.com/davidjb/1f3c090b7bae52fecc0a6eac57e32db2 for a worked example.
To run:

git clone https://gist.github.com/1f3c090b7bae52fecc0a6eac57e32db2.git example
cd example
npm i
npm t

It boils down to:

const bundler = new Parcel({
  entries: 'a.js',
  defaultConfig: '@parcel/config-default',
  shouldDisableCache: true,
});

await bundler.run();

running a lot quicker than:

const workerFarm = createWorkerFarm();
const outputFS = new MemoryFS(workerFarm);

const bundler = new Parcel({
  entries: 'a.js',
  defaultConfig: '@parcel/config-default',
  shouldDisableCache: true,
  workerFarm,
  outputFS,
});

await bundler.run();
await workerFarm.end();

🤔 Expected Behavior

Ideally, the in-memory implementation would be faster, otherwise it negates its reason for being. Otherwise, if there are situations where the MemoryFS implementation is known to be slower, these should be documented so it's clear when MemoryFS is a better choice.

😯 Current Behavior

$ npm t

> test
> NODE_OPTIONS=--experimental-vm-modules npx jest

(node:52151) ExperimentalWarning: VM Modules is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:52151) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
 PASS  ./build.test.mjs (9.304 s)
  ✓ run() - default options (1806 ms)
  ✓ run() - in memory (7342 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        9.318 s, estimated 10 s
Ran all test suites.

That said, this is time to first build in a cold environment. Once having built initially and bundler/workers/FS are already in existence, a re-build is faster on the in-memory implementation by a measurable amount compared to the defaults (e.g. 650ms vs 750ms).

💁 Possible Solution

🔦 Context

I'm attempting to speed up builds by using memory compared to using the file system.

💻 Code Sample

See https://gist.github.com/davidjb/1f3c090b7bae52fecc0a6eac57e32db2

As a more contrived test, by replacing a.js with just a lot of lines like so: 1;, the default configuration's build time increases somewhat linearly, but the in-memory implementation is consistently worse after a.js reaches a few 1000 lines of code, and gets progressively worse. For a 10MB .js file full of lines of 1;, the results look this:

./build.test.mjs (7.162 s)
  ✓ run() - default options (2742 ms)
  ✓ run() - in memory (4233 ms)

🌍 Your Environment

Software	Version(s)
Parcel	2.10.3
Node	21.1.0
npm/Yarn	10.2.0
Operating System	macOS but also observed on AL2023

[mischnic]

Some rust-based operations are optimized to access the FS directly if possible (= no memory fs):

parcel/packages/utils/node-resolver-core/src/Wrapper.js

Line 87 in dc3d92b

this.options.fs instanceof NodeFS &&

parcel/packages/core/package-manager/src/NodePackageManager.js

Line 81 in dc3d92b

this.fs instanceof NodeFS && process.versions.pnp == null

Apart from that, we mostly use the MemoryFS for the tests. So there might be some potential for optimization.

[davidjb]

/unstale

[davidjb]

/unstale

[davidjb]

/unstale