feat(web): add web generator #285

avivkeller · 2025-05-28T21:53:14Z

Fixes #7.

This PR adds the web generator.

Tasks / Issues

P1 – Must Complete Before Merge

Add more items (anyone can do this as they review¹)

P2 – Must complete before migration

ChangeHistory component doesn’t render Markdown correctly. (See entry.changes aren't converted to HTML #326)
Add more items (anyone can do this as they review¹)

P3 – Can Be Done in a Follow-up

Remove mustache dependency
Add more items (anyone can do this as they review¹)

Get a preview

node bin/cli.mjs generate -t orama-db -i "/node/doc/api/*.md" -o "./out" # If you want search functionality
node bin/cli.mjs generate -t web -i "/node/doc/api/*.md" -o "./out" --index "/node/doc/api/index.md" # Specifying `--index` is optional, but recommended
npx serve out # You can serve the output, or just open one of the files in your browser. Serving is required for using search.

Add things as they appear, or leave review comments. ↩ ↩² ↩³

codecov-commenter · 2025-05-28T21:53:59Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
171	2	169	0

View the top 2 failed test(s) by shortest run time

test::should process entries and include JSX wrapper elements

Stack Traces | 0.00654s run time

Error [ERR_TEST_FAILURE]: Missing "NavBar". Found: 
    at new Promise (<anonymous>)
    at Array.map (<anonymous>) {
  code: 'ERR_TEST_FAILURE',
  failureType: 'testCodeFailure',
  cause: AssertionError [ERR_ASSERTION]: Missing "NavBar". Found: 
      at file:.../utils/__tests__/buildContent.test.mjs:31:7
      at Array.forEach (<anonymous>)
      at TestContext.<anonymous> (file:.../utils/__tests__/buildContent.test.mjs:30:19)
      at Test.runInAsyncScope (node:async_hooks:214:14)
      at Test.run (node:internal/test_runner/test:1047:25)
      at Test.start (node:internal/test_runner/test:944:17)
      at node:internal/test_runner/test:1440:71
      at node:internal/per_context/primordials:483:82
      at new Promise (<anonymous>)
      at new SafePromise (node:internal/per_context/primordials:451:29) {
    generatedMessage: false,
    code: 'ERR_ASSERTION',
    actual: false,
    expected: true,
    operator: '=='
  }
}

test::should return HTML and CSS

Stack Traces | 0.0346s run time

Error [ERR_TEST_FAILURE]: result.html.startsWith is not a function
    at async Promise.all (index 0) {
  code: 'ERR_TEST_FAILURE',
  failureType: 'testCodeFailure',
  cause: TypeError [Error]: result.html.startsWith is not a function
      at TestContext.<anonymous> (file:.../web/__tests__/index.test.mjs:43:27)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async Test.run (node:internal/test_runner/test:1054:7)
      at async Promise.all (index 0)
      at async Suite.run (node:internal/test_runner/test:1442:7)
      at async Test.processPendingSubtests (node:internal/test_runner/test:744:7)
}

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

src/generators/web/components/SearchBox/constants.mjs

AugustinMauroy · 2025-05-29T18:42:20Z

src/generators/web/components/CodeBox.jsx

+  const language = matches?.groups?.language ?? '';
+  const [copyText, setCopyText] = useState('Copy to clipboard');
+
+  const handleCopy = async text => {


Why not including useCopyToClipord in ui lib ?

IIRC I didn't include it since it didn't fall under the "UI Component" category, it's a hook, however, if you disagree, I won't block.

src/generators/legacy-html/utils/tableOfContents.mjs

src/generators/web/components/NavBar.jsx

ovflowd · 2025-05-29T19:51:14Z

package.json

    "estree-util-value-to-estree": "^3.4.0",
    "estree-util-visit": "^2.0.0",
    "github-slugger": "^2.0.0",
    "glob": "^11.0.2",
    "hast-util-to-string": "^3.0.1",
    "hastscript": "^9.0.1",
    "html-minifier-terser": "^7.2.0",
+    "mustache": "^4.2.0",


I tried to use __<VARIABLE>__ like in the other generators, however, the JavaScript I was trying to add to the HTML broke the structure of the file. Using a library like Mustache properly escapes anything injected.

export const TEMPLATE = ({...props}) => dedent` <html> ${props.foo} </html> `;

this can work

Sorry if I wasn’t clear, the compiled JavaScript broke the HTML structure because it contained characters that need to be escaped to be in an HTML script.

For example, in your scenario, if props.foo contains a </html>, it’ll escape the sandbox.

Mustache performs all the needed escaping.

I don't like that we're using yet another layer of whatever freakery here. Mustache is great, but it feels like you're throing a nuke at an ant's sized problem. Not to mention adding yet another engine here will make things even more complex and costly-performance wise. Let's try to avoid throwing npm packages to any problem we encounter, shall we?

avivkeller · 2025-05-30T14:54:47Z

🎉 The code now dehydrates to the client so it can render without JavaScript!

AugustinMauroy · 2025-05-30T15:28:54Z

🎉 The code now dehydrates to the client so it run without JavaScript!

Wow 😵‍💫 and what about codetab

avivkeller · 2025-05-30T15:43:36Z

It rehydrates and runs with JS, but if you don't have JS, you can still view the docs. I used React's SSRing

src/constants.mjs

avivkeller · 2025-05-31T21:54:29Z

@AugustinMauroy and I got search to finally work 🎉

src/generators/web/client/components/MetaBar.jsx

src/generators/web/index.mjs

src/generators/web/client/components/MetaBar.jsx

src/generators/web/client/components/SearchBox/index.jsx

avivkeller · 2025-06-03T23:05:06Z

@nodejs/nodejs-website @nodejs/web-infra ChangeHistory and SideBar aren't implemented yet, so this is still a draft, but it's ready for you to take a look, so feel free to review :-)

avivkeller · 2025-06-29T12:55:21Z

Can we add syntax highlighting on these entries.

Fixed.

Can we update the table so that the <code> tags are not in bold
Is it just me or is the font size changing for code tags within the table? And yes, feels like the styling is a bit broken not gonna lie.

The <code /> tags use the same styling everywhere, it's inherited from @node-core/ui-components/styles/markdown.css.

I also feel the heading sections don't need to be a <code> tag anymore as they're just a method name.

Fixed.

Class Constructors should it be "new ClassName" or just "ClassName" Constructor?

I've changed it to ${name} Constructor

We should document what C, M, E (and the different versions of "C") mean somewhere. Either by a hover tooltip or "about this documentation"
Otherwise let's rename the Pills to be like "Class", "Nethod", "Event", "Class Constructor"

If we want to add them to "About this documentation", that's a nodejs/node issue. If we want to add a tooltip, we can PR nodejs/nodejs.org's DataTag. I'm also fine with the pill.

Ah and some stuff seems to not be using the new table yet

Fixed.

I feel we need to add "tags" labels or something; as it upsets me so much "write file" and none ofn the fs: write file methods are even showing up on these results.

I've updated the orama-db generator to also index the first paragraph of a given header.

It'd be great if we could also have a navbar 🤔 but maybe not (and not a blocker can be a folow-up)

We could add a static (backup) navbar, and then also use #327 to load any potential changes?

Select shows major + minor; But options show major with .x; Shouldn't select options label show the full version?

That's an issue with the releases array: It only includes majors.

ovflowd · 2025-06-29T19:49:03Z

The <code /> tags use the same styling everywhere, it's inherited from @node-core/ui-components/styles/markdown.css.

Yes, but that's not the <code> styling, these elements are written with <bold><code>ffff</code></bold> we could sanitize something or change the font for small code blocks, or the antialising because it looks quite weird and contrasting the table/theme

ovflowd · 2025-06-29T19:50:25Z

I've updated the orama-db generator to also index the first paragraph of a given header.

Orama also supports tags and labels, hence we could for fs.writeFileSync add a label of write file sync (direct human counterpart) or something; Trying to be ingenius here to increase the likelihood of a match based on what matters versus textual match.

cc @micheleriva any idea here?

ovflowd · 2025-06-29T19:57:43Z

We could add a static (backup) navbar, and then also use #327 to load any potential changes?

We can keep it empty for now. Would be cool to have at least one entry "Go back to the Website" (i18n)

BTW, have we integrated i18n on the new web gen?

ovflowd · 2025-06-29T20:10:57Z

@avivkeller latest commit build failing:

avivkeller · 2025-06-29T20:52:07Z

On my machine, performance tanked, I need to investigate.

ovflowd · 2025-06-29T21:01:41Z

Noticed the same.

ovflowd · 2025-06-29T21:10:10Z

I wonder if for these sort of properties (such as an object), we can do something like

(Just a rough idea) but allows better understanding that these are pieces of options?

Also I noticed what's wrong, the text on the table is 14px but the <code> is 16px when it should also be 14px!

avivkeller · 2025-06-29T21:15:15Z

I actually think the perf regression is a bug in tailwind. Maybe the patch they just released contained a bug.

avivkeller · 2025-06-29T21:24:24Z

Indeed. Downgrading to @tailwind/[email protected] resolves the perf regression

avivkeller · 2025-06-29T21:47:28Z

Yes, but that's not the <code> styling, these elements are written with ffff we could sanitize something or change the font for small code blocks, or the antialising because it looks quite weird and contrasting the table/theme

I've adjusted the code tags to inherit their size.

Orama also supports tags and labels

We could add a "tags" in the YAML metadata in nodejs/node?

I wonder if for these sort of properties (such as an object), we can do something like

Done.

BTW, have we integrated i18n on the new web gen?

#280

ovflowd · 2025-06-29T22:00:53Z

We could add a "tags" in the YAML metadata in nodejs/node?

too much upstream work. -1

ovflowd · 2025-06-29T22:04:23Z

Is it still supposed to take this much?:

node bin/cli.mjs generate -t web -i "../node/doc/api/*.md" -o "./out" --index  106.07s user 34.23s system 293% cpu 47.827 total

ovflowd · 2025-06-29T22:05:25Z

I feel we need to fix styles upstream on the Table component?

(missing borders, and maybe making the paddings be equally correct top/bottom)

ovflowd · 2025-06-29T22:06:56Z

Also, OOC: Do we need line numbers on these that are from the definitions?

ovflowd · 2025-06-29T22:08:11Z

Also missing Description column for top level items?

I also think when there's no description can we either write N/A or just -

avivkeller · 2025-06-30T01:00:55Z

Is it still supposed to take this much?:

106.07s isn't great, we have several improvements planned, including:

Using a faster minifier
Precompiling ui-components (So no custom ESBuild resolver, PostCSS, or Tailwind)

I feel we need to fix styles upstream on the Table component?
(missing borders, and maybe making the paddings be equally correct top/bottom)

nodejs/nodejs.org#7911

Also, OOC: Do we need line numbers on these that are from the definitions?

No, I can update the CSS to remove them.

Also missing Description column for top level items?

It's not a bug, if there aren't any descriptions on any of the entries, the column is omitted. Ditto for name and type. However, perhaps we should give the subtable headings if it has different columns than the parent table.

avivkeller · 2025-06-30T01:07:38Z

We are also Promise.all-ing the generation, which, while faster, can be very intensive for the host machine, so I want make that use a for loop, like the legacy generator.

It prevents memory crashes from happening, but it exponentially slower, so I really need to improve performance.

(We can also p-limit the Promise.all, what what would the limit be? --threads is for the amount of workers, should it also dictate how many promises the worker can spawn?)

ovflowd · 2025-06-30T01:08:43Z

It's not a bug, if there aren't any descriptions on any of the entries, the column is omitted. Ditto for name and type. However, perhaps we should give the subtable headings if it has different columns than the parent table.

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

ovflowd · 2025-06-30T01:09:46Z

We are also Promise.all-ing the generation, which, while faster, can be very intensive for the host machine, so I want make that use a for loop, like the legacy generator.

It prevents memory crashes from happening, but it exponentially slower, so I really need to improve performance.

(We can also p-limit the Promise.all, what what would the limit be? --threads is for the amount of workers, should it also dictate how many promises the worker can spawn?)

Let's think of that if all other performance improvements aren't enough

avivkeller · 2025-06-30T01:10:53Z

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

I feel like there are cases where that's not beneficial, like

Let's think of that if all other performance improvements aren't enough

That (being using Promise.all) is what we are using right now... with the 100s+ execution time 😬

ovflowd · 2025-06-30T10:00:20Z

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

I feel like there are cases where that's not beneficial, like

Let's think of that if all other performance improvements aren't enough

That (being using Promise.all) is what we are using right now... with the 100s+ execution time 😬

I feel that we should always have the same columns, to make it consistent. Otherwise it might be confusing. Simply set the values to said empty values as N/A or - I'm 100% inclined into consistency here (please 🙇)

ovflowd · 2025-06-30T10:43:06Z

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

I feel like there are cases where that's not beneficial, like

Let's think of that if all other performance improvements aren't enough

That (being using Promise.all) is what we are using right now... with the 100s+ execution time 😬

I feel that we should always have the same columns, to make it consistent. Otherwise it might be confusing. Simply set the values to said empty values as N/A or - I'm 100% inclined into consistency here (please 🙇)

Even more as just this Type , doesn't tell anything to the users. Is this a parameter? Return type? Argument of the method? Also where's the snippet on that example?

avivkeller · 2025-06-30T12:07:27Z

Even more as just this Type, doesn't tell anything to the users. Is this a parameter? Return type? Argument of the method? Also where's the snippet on that example?

That example was generated from the following markdown:

#### `dir.path`

<!-- YAML
added: v12.12.0
-->

* {string}

The read-only path of this directory as was provided to [`fs.opendir()`][],
[`fs.opendirSync()`][], or [`fsPromises.opendir()`][].

Maybe it's just me, but it feels weird to have the table below when only Type is specified:

Property	Type	Description
-	`<string>`	-

avivkeller · 2025-07-01T00:36:18Z

Moving to draft until the following performance improvements are completed (dynamic list, feel free to add more items):

PostCSS / Tailwind removed (This depends on https://github.com/nodejs/nodejs.org, so this PR is blocked )
ESBuild replaced with Rolldown¹ (Speed comparison: https://x.com/youyuxi/status/1869608132386922720)
Hast compiled directly to Babel OR Babel replaced with ESTree alternative²
OOM Crashes on less powerful machines³

Ideally, we'd use a pluginless implementation due to the removal of PostCSS, making everything even faster ↩
Currently, we MDX -> ESTree -> Babel transforms -> Text. I would like to either MDX -> ESTree -> Transforms -> Text OR MDX -> ESTree -> Text -> String Transforms. ↩
My A2179 crashes (OOM) when generating doc/api/*.md, but maybe it's just me? ↩

avivkeller marked this pull request as ready for review May 28, 2025 22:13

avivkeller requested a review from a team as a code owner May 28, 2025 22:13

avivkeller commented May 29, 2025

View reviewed changes

src/generators/web/components/SearchBox/constants.mjs Outdated Show resolved Hide resolved

avivkeller marked this pull request as draft May 29, 2025 16:36

AugustinMauroy reviewed May 29, 2025

View reviewed changes

ovflowd reviewed May 29, 2025

View reviewed changes

avivkeller force-pushed the feat/web/gen branch from 1eb144a to 492a696 Compare May 29, 2025 20:30

avivkeller force-pushed the feat/web/gen branch from d8ab24c to 9ebcb95 Compare May 30, 2025 14:57

avivkeller mentioned this pull request May 30, 2025

chore(ui): simplify CodeBox-like UI components nodejs/nodejs.org#7793

Merged

avivkeller added the enhancement New feature or request label May 30, 2025

avivkeller added this to Node.js API Documentation Tooling May 31, 2025

avivkeller moved this to In Progress in Node.js API Documentation Tooling May 31, 2025

avivkeller removed this from Node.js API Documentation Tooling May 31, 2025

AugustinMauroy reviewed May 31, 2025

View reviewed changes

src/constants.mjs Outdated Show resolved Hide resolved

flakey5 reviewed May 31, 2025

View reviewed changes

src/generators/web/client/components/MetaBar.jsx Outdated Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch from 705f9e2 to 76ef14a Compare May 31, 2025 22:18

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/index.mjs Show resolved Hide resolved

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/client/components/MetaBar.jsx Outdated Show resolved Hide resolved

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/client/components/SearchBox/index.jsx Outdated Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch 2 times, most recently from 024bbea to dbfe55d Compare June 3, 2025 21:40

avivkeller force-pushed the feat/web/gen branch from e814300 to ba07721 Compare June 5, 2025 20:27

avivkeller linked an issue Jun 6, 2025 that may be closed by this pull request

Write React Components #10

Closed

avivkeller force-pushed the feat/web/gen branch from ca72ed0 to 99176ba Compare June 6, 2025 20:33

avivkeller force-pushed the feat/web/gen branch from 0be0b84 to a38b047 Compare June 29, 2025 21:57

feat(web): add web generator

f621054

avivkeller force-pushed the feat/web/gen branch from a38b047 to f621054 Compare June 30, 2025 12:20

avivkeller marked this pull request as draft July 1, 2025 00:34

remove parallelism to debug performance

a97aa71

feat(web): add web generator #285

Are you sure you want to change the base?

feat(web): add web generator #285

Conversation

avivkeller commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks / Issues

P1 – Must Complete Before Merge

P2 – Must complete before migration

P3 – Can Be Done in a Follow-up

Get a preview

Footnotes

Uh oh!

codecov-commenter commented May 28, 2025 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

Uh oh!

AugustinMauroy May 29, 2025

Choose a reason for hiding this comment

Uh oh!

avivkeller Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ovflowd May 29, 2025

Choose a reason for hiding this comment

Uh oh!

avivkeller May 29, 2025

Choose a reason for hiding this comment

Uh oh!

AugustinMauroy May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

avivkeller May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ovflowd May 30, 2025

Choose a reason for hiding this comment

Uh oh!

avivkeller commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AugustinMauroy commented May 30, 2025

Uh oh!

avivkeller commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

avivkeller commented May 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

avivkeller commented Jun 3, 2025

Uh oh!

avivkeller commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

avivkeller commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avivkeller commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

avivkeller commented May 28, 2025 •

edited

Loading

codecov-commenter commented May 28, 2025 •

edited by codecov bot

Loading

AugustinMauroy May 29, 2025 •

edited

Loading

avivkeller May 30, 2025 •

edited

Loading

avivkeller commented May 30, 2025 •

edited

Loading

avivkeller commented May 30, 2025 •

edited

Loading

ovflowd commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

ovflowd commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 30, 2025 •

edited

Loading

avivkeller commented Jun 30, 2025 •

edited

Loading

avivkeller commented Jul 1, 2025 •

edited

Loading