Taking it to the Next Level!

[softworkz]

Hi @FlorianRappl and @GregorBiswanger,

after reading the "Reloaded Discussion", I thought it's time for some positive vibe and try to move things forward a bit.
Before I'm going to shock you with the intentions and ideas I have, I think I should drop the magic words, saying that I'm willing to actively help in these regards rather than demanding anything. Essentially, I'm kind of forced into it, because the slow debugging startup times (and a few other limitations) are preventing me from finishing my work. Now I could do just the minimum required changes privately, but I thought it would be nice and fair to provide some return for the benefits that we gain from your project.

Proposed Changes

In the other conversation, you mentioned that you made some plans for future evolvement, so I hope this is not too far off, even though it may appear a bit radical ☢

Anway, here's what I'm up to:

• Eliminate the CLI tool - do everything with MSBuild instead
• Eliminate electron.manifest.json - use MSBuild (Project-) properties instead
  • Then create package.json files dynamically from the MSBuild properties
  • This will also create a consistent appearance
• Greatly Improve the debugging experience
  • Minimize startup times
  • Debug the ASP.Net process, by default not the Electron process
  • Enable Hot Reload (at least edit+continue) for the ASP.Net code
    Ideally also get it working like with Browser Link
  • Make sure that all processes get terminated in case of non-graceful exit during debugging
  • Enable node.js javascript debugging in Visual Studio (probably requires a secondary project in the solution)
  • Enable debugging of (browser-)javascript in Visual Studio (probably requires a secondary project in the solution)
• Establish an "unpackaged" run-mode
  This means to use a regular build (to MSIL, without publish) at the .NET side and an unpackaged configuration at the Electron side (just npm install, not invoking electron-builder)
  • The Electron side should use MSBuild's incremental build abilities, i.e. re-build it only when changes are made that may affect the Electron output
  • ,NET side is using incremental build anyway
• Allow freely choosing the version of Electron and the version of electron-builder
• Make RID selection part of the project
• Re-structure the build folder layout
  Putting the electron files into the same subfolder path like the .net build output, e.g. bin\net8.0\win-x64, because bin\Desktop is ambiguous when building for multiple target RIDs.
• Enable building of Linux packages on Windows via WSL
  Only the electron build commands need to be run on WSL, the .net side can be built on Windows
• Enable running and debugging Linux outputs on Windows (by running them on WSL)
• Use VS/MSBuild publishing mechanisms for creating Electron packages
  • Use "Folder Publishing" type - try to hide the others
  • All those parameters which are currently passed through by the CLI, can then be set just directly/naturally, then
• Remove distracting elements from the project (e.g.: no need for 'pack')
• etc.

Let me know what you think about it!

sw

[softworkz]

I'm not coming empty-handed...:

[FlorianRappl]

Yeah that sounds super awesome and goes in the direction (but honestly already quite a lot further) what I had originally in mind.

So you definitely have my blessing!

[softworkz]

How about setting up a new branch for this, like nextgen or so?

My typical approach for such things is to "race" over all the tough points to get to a point where I got it roughly working - at least enough for me to know that it's doable. That phase is already done, so we're not talking just about ideas, but most of the core points are already confrimed and kind-of working.
I could create a PR against a new branch, to allow you guys and everybody else to try it out, and check out the details.

I'll follow-up with a few posts, showing what I got working already and how, so that we can discuss things rather than me just dictating the result.

Sounds good?

[GregorBiswanger]

At first glance, I really like it too!

It would be exciting to see if everything works when it's in use...

[softworkz]

I'm starting a series of posts, documenting and explaining the changes I've made so far, Let's start with this:

.NET Project System Extension & Manifest Replacement

In the screenshot above you can see an "Electron.NET" section. This is made possible through extensibility of the .NET/VS project system. It is working without the need for developing a Visual Studio extension like in case of the older project system. It is working working through the nuget package. So that part doesn't change: You add the Electron.NET nuget package to an ASP.Net Core project just like usual. There's also no requirement to use Visual Studio, but if you do, you get this nice UI tab in the project designer as shown above.
Changes made there, go directly into the .csproj file:

That's essentially the same plus a little bit more than what is currently in electron.manifest.json. The motivations for this change are:

• Allow configuration vie project designer
• For the intended deep integration with MSBuild, it's much easier to have this in XML rather than JSON as MSBuild is all XML based
• It provides a single source for this information. Currently, it is disjoint between:
  • .NET assembly information (properties of .net executables and dlls)
  • Electron runtime app info (what's currently in electron.manifest.json)
  • Package building app info (what's currently under the 'build' node tree in electron.manifest.json)
  • It also resolves the app icon path confusion (which is hard to figure out how it needs to be set)

The very first building block now, is to automatically generate a package.json file (for the Electron app) at design time.
This is done by MSBuild targets which are run as part of the design-time builds.
The package.json file is generated under /obj/<Configuration>/<runtime>/<rid>/package.json.

Here you can see it being updated automatically while editing the values in the app designer:

ElectronNet_PackageJson.mp4

[softworkz]

Building

Prerequisites

RID-Specific Builds

Since we always end up having RID-specific output, we can do so right away, even when not publishing. This allows pinpointing platform-specific issues early (like unsupported APIs), but most importantly, it allows a clean separation of outputs, because the RID gets included in the output paths when specificed (this happens automatically by the build system).

To make this more explicit, I've added a selection - which also enforces that one is specified (if not using VS, there's still one set as default).

Now we get output paths like this: bin\Debug\net8.0\win-x64\

This allows to have a separate Windows and a Linux output (like for running on WSL) without conflicting.

Unpackaged App Layout

For running and debugging, we also need the Electron part. This is platform-specific as well (at least when using Electron), so this part should go together with the output under the same folder.

For simplicity and avoiding ambiguities, I'm using a dotted subfolder at the moment: .electron

So, the full output tree looks like this:

Populating the .electron folder

Next, we need to fill the content of the .electron folder. This is not much different (with regards to the result) as currently, except the following:

• ElectronNet core files
  The files in the folders (api, ElectronHostHook, splashscreen, .vscode) are included in the .nuget as plain files
  This allows letting MSBuild handle copying and change detection. Even though these never change, it needs still be determined whether they need to be copied or whether all files are there already - that would be difficult when extracting them from embedded resource like it's done currently
• Splash screen image and icon files
  While the icons are only needed when packaging, the splash screen image is needed for unpackaged builds as well.
  As seen above, the path of the splash screen image gets stripped when creating the package.json file. This is because it is always copied into the .electron folder, no matter where it is - which avoids the current path problems.
  => it could also be put into a predetermined subfolder of course
• package.json
  This generated file (see above) needs to be copied to the .electron folder as well

After building, the .electron folder will look like this:

Build Targets

MSBuild is a creature of its own and its inner workings not directly obvious at first sight, and even worse when trying to figure out how to do certain things in the right way. But luckily I had to jump over this hurdle before and what we need here is not a big thing. I've added comments in the files which should allow to follow what's happening. The following targets are defined for building (I'll get to the publishing part later on).
What helps to understand a bit better is this: In MSBuild, there's almost always a separation between "planning" and "execution" when it's about processing files and generating output.
Practically speaking: Before any actual compilation of file-copying is performed there many targets/tasks being executed for the sole purpose of collecting and calculating all the input and output files before something is done.
This provides the benefit of building incrementally: only doing what's needed without repeating anything that has been done already and is still valid. And we're leveraging this for the Electron side as well.
The following targets are defined for it:

• ElectronResolvePaths
  This is a helper target which sets properties for the Electorn-specific paths. It needs to be in a target so that it can be executed at a point in time when the base properties it is relying on are properly set
• ElectronPrepareForBuild
  This runs during build but also in design-time builds. Currently, it does the following:
  • Replace the placeholders in the package.json template with actual values from the project
  • Compare it with the current package.json (under obj/xxx/xxx/xxx/package.json)
  • Only save it when there's a change
    => This is IMPORTANT, as it allows to avoid running npm when there's no change
  • It further creates an item group ElectronFiles and sets the target path for each item. The group includes:
    • package.json
    • splash image
    • icon
    • the ElectronNet core files
• ElectronConfigureApp
  This is executed later (after the CopyFilesToOutputDirectory target), and only during actual builds.
  It specifies the ElectronFiles items as inputs and their target paths as outputs. This has the effect that the target will be executed only when any of the source files is never thant its target counterpart. This gives us the "only build when necessary" effect. It does the following:
  • Copies all input files to the target locations
  • Runs npm install in the .electron folder (build output)
• ElectronBeforeClean
  This runs during a clean or a rebuild. All it does is adding the files in the .electron folder to the Clean item group, so that they'll get properly removed when cleaning

That's all for building. (we'll get to publishing later)

[softworkz]

Incremental Build Demo

Note that "Building Electron" here means to copy (update) the changed files and run npm install in the .electron folder.

ElectronNet_Building.mp4

[GregorBiswanger]

One challenge I see here, which is why I didn't make it freely selectable by design in the past. For example, there are many breaking changes between the native Electron versions.

Events are called something different, functions have been removed, different data structures, etc.
The same applies to Electron builds, too. This is then only compatible with certain native Electron versions.

It would be interesting to know how you can stabilize this here?

[GregorBiswanger]

For support native Electron extensions, we have integrated a feature, called Electron Hooks, typically require an Electron CLI again. This is because it manages triggering actions like "npm install," starting TypeScript, etc.

However, this might also work via MSBuild?!

[softworkz]

One challenge I see here, which is why I didn't make it freely selectable by design in the past. For example, there are many breaking changes between the native Electron versions.

Events are called something different, functions have been removed, different data structures, etc. The same applies to Electron builds, too. This is then only compatible with certain native Electron versions.

Yes, I've seen your earlier comments in this direction. Yet I've seen that people are using more recent Electron versions with ElectronNet and there's currently 30.0.3 in the devlop branch vs. 23.2.0 in the main branch, plus an open PR for switching to 31.

This makes me assume that these newer Electron versions are at least partially tested by users.
I also have the suspicion that the stability of the Electron APIs has improved in the more recent history (= since 23). Last year I had updated an older (non-.net) Electron app of ours without any issues even though it's using quite a number of Electron APIs.

Currently in Electron.NET, it's hard to change the preconfigured version. Of course this has the advantage of keeping everybody on the safe side, but it has also prevented most users from trying different versions, so there's little to no feedback about compatibility with newer versions except those PRs to the develop branch.

My thought is that allowing to switch the version freely will bring more feedback and possibly also PRs from users with appropriate fixes. Since there's UI now to switch the version, it would also be possible to show a warning ("unsupported/untested") when selecting any other version than 23.x.x - until others are confirmed working.

It would be interesting to know how you can stabilize this here?

Generally, I see two approaches for this:

Writing Integration Tests

Other than unit tests (which might not help much for this), integration tests would mean to create a test app and run coded tests for the various APIs in the context of the running app.
The whole thing could then be run with at least all the major Electron versions since 23.
AI assistance could be used for avoiding to code all the tests by hand.

Autogeneration of Electron API Projection

This would be a bit more advanced, but I've done something very similar before. Electron.NET is just a small puzzle piece for us for covering Linux platforms. We have a kind of cross-platform meta-framework which allows us to bring the same app to many platforms. It works on top of various other frameworks, namely WinUI3, UWP (xbox), MAUI and Electron.NET (plus generic web), using always the same html/js core code but extended by many platform-specific features done in C# (a shared base, plus platform-specific C# with P/Invoke etc. and a range of plugins).
The core of interaction is a messaging-based remoting which works in both directions:
The "native" (C#) side can expose objects to the JS side in the browser. This causes the creation of proxy objects inside the browser engine which have the same methods, properties and events(!) like the C# object. Method invocation can be synchronous or async. For subscribing to C# events, there's a special method provided. Essentially, you can transparently work with C# objects almost (few caveats only) as if they were JS objects.
For the other direction, it works a bit different. Since C# is a type language, the typings need to exist in advance. There's a helper JS module which you can give a list of JS modules. It then retrieves or instantiates their exports and generates C# interfaces for those. As it cannot know the return types and parameter types, those are of type object, but well - that's how JS works.
At runtime, when the C# side needs to work with a JS object, the framework is creating C# proxy objects implementing those generated interfaces. It also works for subscribing to events from JS objects (event emitters). It's probably the most comprehensive JS-C# interaction framework that exists (last time I looked around).

Just that you know about the background of where I'm coming from. So, a similar autogeneration would also be possible here. Since TypeScrip declarations are available for Electron, this also wouldn't need to end up having object as parameter and return types. Once it's auto-generated, you can easily generate this for every major Electron version.

But I'm not sure whether I got that much time to spend, so probably aim for a simpler solution first. :-)

[softworkz]

For support native Electron extensions, we have integrated a feature, called Electron Hooks, typically require an Electron CLI again. This is because it manages triggering actions like "npm install," starting TypeScript, etc.

However, this might also work via MSBuild?!

I haven't forgotten that part, but here I'm not sure yet about the way. I see these two options:

• Do it same like before:
  • If present in the project, copy the ElectronHostHook contents to the .electron/ElectronHostHook output folder
  • then run npm install and npx tsc there
    or
• Use the TypeScript compilation support from the ASP.NET tooling
  I haven't tried this yet, but the idea would be to
  • Change the TypeScript root folder from ./ to ./ElectronHostHook
  • Have it compiled and npm-restored as part of the ASP.NET project compilation
  • Copy only the compiled output to the .electron/ElectronHostHook folder

I'll look into feasibility of the second approach, when it doesn't work nicely, the former approach should always do it.

Either way it's clear this this should work just like before.

[softworkz]

ElectronHostHook

That part was a bit mind-bending to figure it out. Not about how to handle it but rather about understanding the way how it's done and how it rather should be done, because the current approach is - err - not quite by the books.
The ElectronHostHook part is currently treated like a separate application, means it has its own dependencies and a separate package restore (npm install) is run for it - which makes it two separate and independent things. But then - at runtime - it gets loaded by main.js as if it were part of the main nodejs app - even though it has its own node_modules and it might not even be compatible in its dependencies.

Now I understand that this was also done in order to perform TypeScript compilation of a custom ElectronHostHook folder. But that's not really necessary in ASP.NET Core projects. TypeScript compilation happens automatically (maybe it has been different in earlier days).

The Solution

Okay - so what we need is just a single npm install as it's all a single node.js application. But the ElectronHostHook implementation can still use dependencies that are not included in the main Electron app - what to do? Merge the dependencies from ElectronHostHook into the package.json of the main app? But there might be conflicts or duplications, that doesn't sound solid either.
This is what took me some time, albeit the solution is pretty simple: We need to treat ElectronHostHook just like another dependency of the Electron node.js app.
And that's what it already is: it has its package.json and a number of compiled js files. Now we need to add it as a dependency in the package.json of the Electron app. Like this:

When we run npm install now, the host hook "package" gets installed under node_modules/electron-host-hook - just like any other package. And npm also reads the dependencies from that package.json file - this means for example, when the host hook code has exceljs as a dependency then it gets installed as well.

In turn, we are creating a correct app config and we need to run npm install just once - not twice.

The best part: nobody with a custom host hook will need to change anything.

Additional Notes

Electron Types in HostHook

index.ts and connector.ts are currently using Electron.App in their constructors. This is not valid because the host hook package.json doesn't import Electron as a dependency. And I don't think it should, because this would just blow up everything and it's purely a design-time thing - the generated js files don't have typing anyway.
Hence I would just change Electron.App to Any and it's all good (and users won't be confronted with those errors anymore.

(in the development solution it doesn't error because it is imported elsewhere and Intellisense adds all dependencies to a common pool, but for Electron.NET consumers, these errors do show up - have seen it as well)

Package Diet

All generated packages will become 30 MB smaller in size.

That's because all packages currently are including the TypeScript dev dependency package which isn't actually needed:

[softworkz]

Host Hook is all good now. I've changed the first post into a task list for tracking progress. Next point is:

Debugging

I belive this is a point where many interested developers have turned around, looking for other possible options. The current behavior is indeed somewhat hard to sell: normally in ASP.NET projects, you can easily work and debug the project, but when they come to try Electron.NET and start debugging, they realize that debugging doesn't work as expected. Instead of debugging their code, the debugger is debugging an Electron process where there's nothing interesting to see with a managed code debugger: There just isn't any managed code running in Electron/Chromium processes.

And then - well, you can of course attach to the ASP.NET process (with the new "Re-attach to process" feature in VS, this has become a bit less tedious) or you can add Debugger.Launch() to your C# code - still this is slowing down development.
But the worst point here, is that when the debugger doesn't get attached right on launch, it is not possible to make code changes during runtime. With the slow startup, two things are coming together. No edit-and-continue might be ok when startup is fast, or a long startup might be acceptable when you can work while it's running (edit-and-continue).
When both are coming together - that's a real pain-point. Even though we got the slow startup fixed already, there's more to gain.

Launch Sequence

In order to debug the ASP.NET project, we need to have the debugger launch it directly. VS has the ability to debug multiple processes in parallel, but - unfortunately - not from a single project. Setting up an additional project just for debugging is possible but ugly and complicated. The only way to get what is needed here is to have the debugger launch the ASP.NET project and let the ASP.NET project then launch the Electron app.

But we can't easily change this for packaged applications. Since this is all Electron, they are relying on the Electron app to be started first and I don't think it's reasonable to change this.

Long story short: both ways are needed!

• The current way which replicates the exact behaviour of the packaged and released app
• A new way which launches the ASP.NET project first for easy and convenient debugging without anything in the way

Launching ASP.NET First

Pretty much straightforward, just the other way round:

• Debugger launches ASP.NET
• The UseElectron() middleware launches the Electron app
• It watches the process
• it reads the stdout and stderr streams and logs it as part of its own output
• When it exits, it shuts down the ASP.NET instance

Debug Launch Profiles

So that's the result:

The Electron (unpacked) profile launches Electron and the other one the ASP.NET project for debugging.
(there will be more...)

I'm getting tired of writing - time for another demo!

[softworkz]

At first glance, I really like it too!
It would be exciting to see if everything works when it's in use...

Yea, well - when something sounds like too good to be true, it's hardly ever true. Exceptions are rare, talk is cheap - time for truth

🚀

Debugging Demo Pt 1

ElectronNet_Debugging_1.mp4

Debugging Demo Pt 2

ElectronNet_Debugging_2.mp4

[FlorianRappl]

I love it!

I think we should go that path.

Regarding selecting an Electron version; my idea was to have an adapter here - something like (names just to illustrate the idea)

• Electron.NET.Core (core framework)
• Electron.NET.Api (API definitions as the API to consume from Electron .NET)
• Electron.NET.BridgeX (x is the Electron carrier, e.g., 30 - Electron.NET.Bridge30)

Bridge implements Api, Core references / knows Api. One selects a version, which installs / uses a variant of Bridge that is compatible with the Electron version. This way Electron.NET is "independent" (to some degree) on Electron.

My idea was to start with a single / or two (at most) of the most popular Electron versions for the Bridge. Most of the time - as already written - a new Bridge / support for a new Electron version is really easy to establish. This could also be backwards compatible, i.e., if Api gets a new feature that is only there in a new version of Electron, then older Bridges might still implement it as a stub / just to be compatible.