Can't deploy convex config under linux with node >= v20

[jedimonkey]

There is an issue under linux and node greater than v19. The convex deploy command can't seem to connect to the convex ecosystem to deploy. This works fine under v18 and v19, however if you are using more modern node functionality in the project (not necessarily deployed to convex) this fails. Note this works under mac (any node versions AFAIK), haven't tested windows.

node -v
v19.9.0
pnpm convex deploy

looksee-platform@1.0.0 convex /home/marcus/dev/proj
convex deploy

You're currently developing against your dev deployment

dev-project-233 (set in CONVEX_DEPLOYMENT)

Your prod deployment your-prod-292 serves traffic at:

CONVEX_URL=https://your-prod-292.convex.cloud

Make sure that your published client is configured with this URL (for instructions see https://docs.convex.dev/hosting)

? Do you want to push your code to your prod deployment your-prod-292 now? Yes
✔ Deployed Convex functions to https://your-prod-292.convex.cloud

However, under newer versions node v20 (also tested under v22 & v24):

nvm use 20
Now using node v20.19.2 (npm v10.8.2)
pnpm convex deploy

looksee-platform@1.0.0 convex /home/marcus/dev/proj
convex deploy

You're currently developing against your dev deployment

dev-project-233 (set in CONVEX_DEPLOYMENT)

Your prod deployment your-prod-292 serves traffic at:

CONVEX_URL=https://your-prod-292.convex.cloud

Make sure that your published client is configured with this URL (for instructions see https://docs.convex.dev/hosting)

? Do you want to push your code to your prod deployment your-prod-292 now? Yes
Retrying request (attempt 3/6)...
Retrying request (attempt 4/6)...
Retrying request (attempt 5/6)...
Retrying request (attempt 6/6)...
✖ Error: Unable to run schema validation on https://your-prod-292.convex.cloud
✖ TypeError: fetch failed
 ELIFECYCLE  Command failed with exit code 1.

Quite frustrating, as this means this is difficult to integrate with github actions.

[thomasballinger]

Sounds quite frustrating. I haven't reproduced this, what are you using and is there an image I can run in docker or AWS to test? I just tried on recent ubuntu

~/convex/npm-packages/demos/tutorial$ uname -v
#25-Ubuntu SMP Mon Jan 27 20:44:03 UTC 2025
~/convex/npm-packages/demos/tutorial$ node --version
v20.19.3
~/convex/npm-packages/demos/tutorial$ npx convex deploy
You're currently developing against your dev deployment

  colorful-squirrel-921 (set in CONVEX_DEPLOYMENT)

Your prod deployment unique-horse-499 serves traffic at:

  VITE_CONVEX_URL=https://unique-horse-499.convex.cloud

Make sure that your published client is configured with this URL (for instructions
see https://docs.convex.dev/hosting)

? Do you want to push your code to your prod deployment unique-horse-499 now? Yes
✔ Deployed Convex functions to https://unique-horse-499.convex.cloud
~/convex/npm-packages/demos/tutorial$

I don't know what changed in Node.js 20, the LLMs I ask say maybe OpenSSL/TLS getting stricter. You might try with CONVEX_VERBOSE=1 set.

[jedimonkey]

There is a post on discord about this as well - so it must affect a bunch of linux versions.
So on my current situation - I'm running debian 12.11 in a LXC container.
On this linux box, I'm running nvm to swap between node versions.
Kernel: Linux dev 6.8.12-10-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-10 (2025-04-18T07:39Z) x86_64 GNU/Linux

Running CONVEX_VERBOSE=1 dumps out a bunch of info - nothing bad in the file processing (just preparing/skipping files) - and then after the failed 6 attempts:

TypeError: fetch failed
[verbose] 2025-06-30T01:33:49.924Z Flushing and exiting, error: TypeError: fetch failed
    at node:internal/deps/undici/undici:13502:13
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  [cause]: AggregateError [ETIMEDOUT]:
      at internalConnectMultiple (node:net:1139:18)
      at internalConnectMultiple (node:net:1215:5)
      at Timeout.internalConnectMultipleTimeout (node:net:1739:5)
      at listOnTimeout (node:internal/timers:596:11)
      at process.processTimers (node:internal/timers:529:7) {
    code: 'ETIMEDOUT',
    [errors]: [ [Error], [Error], [Error], [Error], [Error], [Error] ]
  }
}
[verbose] 2025-06-30T01:33:49.924Z TypeError: fetch failed
    at node:internal/deps/undici/undici:13502:13
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
[verbose] 2025-06-30T01:33:49.924Z Running 0 cleanup functions
[verbose] 2025-06-30T01:33:49.924Z All cleanup functions ran

I'm sure it was failing under github actions, I'll restore that functionality and post what I find shortly. Will see if I can get it running (well failing) on something more publicly accessible for you @thomasballinger to test on as well.

[thomasballinger]

Ugh these aggregate errors, I just changed the formatting for some others.
Could you run a network test on the newest client version? npx convex@latest network-test

[thomasballinger]

I just tried with docker on a mac where this seems to work, is anyone using a remote setup that I could create something like? Can anyone confirm that something like this (you'll need to create a CONVEX_DEPLOY_KEY in one of your projects) doesn't work for you?

docker run -it debian:12.11 bash
apt update
apt install -y curl build-essential
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
source ~/.bashrc
nvm install 18
nvm install 20
nvm use 18
mkdir testing
cd testing
npm init -y
npm i convex@latest
CONVEX_DEPLOY_KEY='prod:aromatic-mole-986|eyXXXXXXXXX0=' npx convex dev
nvm use 20
CONVEX_DEPLOY_KEY='prod:aromatic-mole-986|eyXXXXXXXXX0=' npx convex dev

[thomasballinger]

Just tried an EC2 machine with image debian-12-amd64-20250316-2053 on AWS, no luck yet.

admin@ip:~$ uname -a
Linux ip-X.X.X.X 6.1.0-32-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.129-1 (2025-03-06) x86_64
GNU/Linux

[jedimonkey]

Will spin up a docker container now on my mac, but I went down a rabbit hole thinking that ipv6 may be the issue (I had enabled it in my network, just to disable it as my router wasn't allowing custom dns entries), but as a result I still have some stale ipv6 ips across the network.
I recall ipv6 priority swapping in a node change... but that doesn't appear to be the issue here:

pnpx convex@latest network-test
✔ Deployment URL: https://my-site-222.convex.cloud
✔ OK: DNS lookup => 52.44.230.118:ipv4 (10.02ms)
✖ FAIL: TCP connect (AggregateError with 6 errors:
  1. Error: connect ETIMEDOUT 52.54.48.130:443
  2. Error: connect ENETUNREACH 2600:1f18:77e:2300:4929:e49b:6cb2:75e6:443 - Local (:::0)
  3. Error: connect ETIMEDOUT 52.44.230.118:443
  4. Error: connect ENETUNREACH 2600:1f18:77e:2301:685b:995f:b5bb:9cbf:443 - Local (:::0)
  5. Error: connect ETIMEDOUT 52.200.179.149:443
  6. Error: connect ENETUNREACH 2600:1f18:77e:2302:ddda:86fa:5687:daaa:443 - Local (:::0))

was also trying with NODE_OPTIONS="--dns-result-order ipv4first" (handy way to prefer ipv4 priority).

[thomasballinger]

oh good thought, I wondered about ipv4/6 stuff but only changes I saw were from Node.js 16 -> 18.
Another thing to run for more info:

NODE_DEBUG=net,http,https,tls,dns DEBUG=* npx convex network-test

[thomasballinger]

I can repro something on my mac:

current hypothesis is "if it takes longer than 250ms to connection on Node.js >20 it doesn't work"

nodejs/node#52216

[thomasballinger]

If NODE_OPTIONS=--no-network-family-autoselection npx convex network-test or NODE_OPTIONS=--no-network-family-autoselection npx convex deploy fixes this that's promising

[thomasballinger]

fix: 2cf8dc7

try convex@1.25.2 and let me know if it's fixed somewhere where you can repro the issue with convex@1.25.1

[jedimonkey]

Apologies for the delay - got sidetracked with work and then every time I remembered I was no where near a computer! thanks @thomasballinger this appears to be working - will do a bit more testing in the next couple of days. Tested with node v22 using convex 1.25.2.

[jedimonkey]

Closing this - tested it also under Github Actions and it's working as expected. Thanks again @thomasballinger

[thomasballinger]

Thanks for confirming!