Greatly improve code sharing and reuse options

[atrauzzi]

Creating and sharing pure typescript code has quite a few barriers that seem like the kind of complexity that TypeScript should be helping to alleviate.

I'm going through a situation right now where I want to share a pure typescript library and - unless I'm mistaken - it seems like I'm going to have to go through the pain of adding AMD to my project. Even despite the fact that it's all internal modules.

With the overall goal of making life easier for developers, I think it would be nice if:

• There was some way to reference typescript files statically such that they are used in the compile process. This could even be used to eliminate the /// syntax used to load .d.ts files.
• There was some way to tell typescript to prepend regular javascript files to the entire build output (dependencies with .d.ts definitions for example).

While AMD/CommonJS are module-loading systems, they are very confusing and cluttered build systems, especially once you bring TypeScript into the mix.

It's conceivable that with these changes, you'd also reap the ability to create build profiles implicitly by defining entrypoint scripts and having typescript crawl the dependency graph. Similar to how r.js works, except at a TypeScript level.

[atrauzzi]

It should probably be mentioned that any solution really needs to be sensitive to the inevitability that the location of the node_modules and typescript sources that the compiler is currently targeting will not be consistent between projects.

This might be one of the major pain points of coming up with a build system that knows how to integrate code from packages. Not only are people conflicted about npm vs bower, but what if what we really need are multiple node_modules directories in a project based on what's being built (front end, cli tools, server side)?

[danquirk]

For your first bullet, is tsconfig not a solution to the problem you're experiencing?

For the second bullet, this is something we've generally stayed out of. People have very varied build systems with a variety of linters, concatentation steps, minifiers, etc and it seems difficult for us to cover the majority of those scenarios in a satisfactory way (and convince someone to break their existing, working system to migrate to the provided TS one). Maybe the solution is to make it easier to customize pre and post build steps. We generally don't want to be in the business of creating some robust, general purpose, cross platform build system. Obviously it's important that the end to end experience be as good as possible though, so maybe we need to make changes to slot into existing workflows more easily or be better at detecting errors that might have occurred as a result of that system.

For node_modules #247 is a priority and hopefully helps alleviate some of the mismatch with using TS + npm modules now. Could you elaborate on what you mean by node_modules and TS sources not being consistent beyond that?

[atrauzzi]

I know I can't speak for others, but I think a build system integrated with typescript would be a welcome change to getting mired in endless build system hacking and configuration.

In an ideal world, I should be able to do this:

tsc path/to/built.js path/to/script.ts

BUT

TS does the legwork to resolve and include in the build everything that script.ts references.

What's preventing this from being a possibility?

[danquirk]

I think a build system integrated with typescript would be a welcome change to getting mired in endless build system hacking and configuration.

Sure, but this is really a complaint about the state of JS build systems and wishing for a great 'one build system to rule them all.' In an ideal world that would exist :) In reality peoples' needs are extremely varied which is why we have many different options (many tried to create the one true build system, now we just have a lot of different ones with different pros and cons). We don't feel like we're in any better situation to suddenly solve that problem better than all the other attempts (especially prior to ES6 having a finalized, cohesive module story). And then there's the fact that even if we actually did succeed at that, many people are not going to switch from their working build system to something new, even if the new thing is better in the abstract ('if it ain't broke...'). That said, we do want to try to slot into what exists well, and we do want to enable others to make good tools around us (like interacting with grunt/gulp/etc).

What's preventing this from being a possibility?

What do you mean by 'everything that script.ts references'? You mean a file without /// or imports? Or a file with ES6 style imports? Obviously right now the compiler does chase through your /// references and imports or will use a tsconfig related to that script. It sounds like you mostly want to avoid /// reference pain which is what tsconfig should help with.

[atrauzzi]

So for tsconfig, I have to manually create one for every graph of source files I want to use? Why not just have the compiler automatically infer this? r.js does this when you tell it to build a file. It will recurse through all the defines and requires and concatenate all those files together.

If I was able to statically reference other .ts files from one root-level .ts script, wouldn't that obivate the need for various mechanisms to tell the compiler about where files are? The compiler would know everything being used because it would have crawled the full import-graph (my root script, it's dependencies, their dependencies, and so on) before going through the code.

[danquirk]

So for tsconfig, I have to manually create one for every graph of source files I want to use?

Not for every graph necessarily, it's up to you whether you want to explicitly list everything or let reference chasing happen depending on what references are in relevant files. Think of it like a basic project file.

Why not just have the compiler automatically infer this? r.js does this when you tell it to build a file. It will recurse through all the defines and requires and concatenate all those files together.

This is exactly what the compiler currently does with .ts files based on the /// references and import/require in the file. With RequireJS you explicitly list your module's dependencies, and each dependency has its own dependency list, and the loader walks through this graph. That's the same thing tsc does when you explicitly list dependencies in the form of /// or import/require. The problem is that explicitly managing /// references in this way can be painful and sometimes things get pulled in in an unexpected order and now you have to manually try to track down how the graph was traversed and what happened. Using a tsconfig to manage the project ordering can alleviate some of this pain.

[atrauzzi]

So general advice for the time being is that I should be making everything external and not really try to make internal modules? That's a distinction I'm still trying to internalize, but I was under the impression that there might be a way for me to do all this without having to set up AMD....

[danquirk]

It's really up to you as far as how you want people to consume your code. Given that ES6 only has external modules it's certainly possible that's the right choice for you going forward though. The TypeScript compiler is only internal modules but you can find small npm packages that provide access to it through external modules. Likewise you'll see some .d.ts files written in such a way as to work with both internal and external (I believe knockout is one example like this).

[atrauzzi]

I'm still at a loss to really understand what the difference is between internal and external. Some of it flies over my head.

[danquirk]

I assume you've read https://github.com/Microsoft/TypeScript/wiki/Modules and http://www.typescriptlang.org/Handbook#modules ? Don't feel bad, it is unfortunately confusing and a situation we definitely can improve. I'm reticent to try to give an off the cuff primer since we've already spent a bunch of time trying to wordsmith the content in those two places since this confusion definitely occurs with some frequency.

The simplest distinction is that external modules are intended to be used with a module loader (either AMD or CommonJS) for all the reasons module loaders are good (plenty written on that topic for JS already). Think of internal modules simply as namespaces. Internal modules leave you with some JS (either many files or a single concatenated output files) that someone consumes by simply putting your file in scope (ex a <script> tag or concatenating with their own JS) without a separate module loader.

[atrauzzi]

So if I'm making the actual entrypoint script for a page, is it possible to author an internal module while also gaining the ability to consume javascript modules? Or am I required to be producing a module so that I can consume other modules?

[rotemdan]

Hi, here's the pattern I've been using for more than a year and a half now (for very large TS apps with more than a hundred source files). Though it may appear to stand in contrast to much of the design recommendations and advice you'll get from the TypeScript team, it has served me very well:

• Although a large part of my development is for Node.js, I use internal modules _only_ (I compile to a single file that works both on the web and in Node). I declare a single namespace/internal module, let's call it App.
• Every .ts file usually contains a single class and looks like this:

module App {
  export class SomeClass {
  ...
  }
}

• I compile all files to a single .js output file (external module preference set to "CommonJS" but I don't think it matters as I never declare any).
• I enable the compiled file to serve as a node module (when the node environment is detected) in this simple way:

module App {
  var runningInNodeJS = () => (typeof require == "function") && (typeof module == "object")

  if (runningInNodeJS())
    module.exports = App;
}

• I dynamically load external CommonJS modules (in this example the Node file system module), when needed, in this way:

import fs = require("fs");

module App {
  if (runningInNodeJS()) {
    var NodeFileSystem: typeof fs = require("fs");
  }
}

EDIT: The compiler will not emit anything for the import statement as fs is only use in a type position, see http://www.typescriptlang.org/Handbook#modules-optional-module-loading-and-other-advanced-loading-scenarios

EDIT: Removed solved problem: solution is in #2346 (comment)

I published a library using exactly this pattern:
https://github.com/rotemdan/lzutf8.js

Here's the compiled release file, I linked to the specific line where the internal module CommonJS export pattern is used:
https://github.com/rotemdan/lzutf8.js/blob/master/ReleaseBuild/lzutf8.js#L15

BTW I'm aware that the next version of TS will have ES6 modules but I haven't really looked deeply into them, though from what I've seen so far I don't feel I'd eventually make use of them.

[rotemdan]

As for build automation, I agree that there's no "one" optimal build system, but here's my own current workflow:

• All my compilation and builds (including compile-on-save) are now done through Grunt, which can be conveniently integrated into visual studio with the Task Runner Explorer VS extension: https://visualstudiogallery.msdn.microsoft.com/8e1b4368-4afb-467a-bc13-9650572db708
• Recently I even disabled VS initiated TypeScript compilation entirely, instructions can be found here: Visual Studio "No Compile" option #2294 (comment)
• In the Gruntfile I have a script (based on the Grunt API) that scans a set of glob paths for .ts files and generates a single references.ts file for the files found (it hasn't been published as a grunt plugin yet but may in the future):

function generateTypeScriptReferencesFile(filters, outFilename)
    {
        var fileNames = [];

        for (var i=0; i < filters.length; i++)
            fileNames = fileNames.concat(grunt.file.expand(filters[i]))

        function getPriorityByFileExtension(filePath)
        {
            if (/.+\.d.ts$/.test(filePath))
                return 0;
            else if (/.+\.ext.ts$/.test(filePath))
                return 1;
            else if (/.+\.spec.ts$/.test(filePath))
                return 3;
            else
                return 2;
        }

        fileNames.sort(function (fileName1, fileName2) { return getPriorityByFileExtension(fileName1) - getPriorityByFileExtension(fileName2) });
        fileContent = fileNames.reduce(function(result, filePath) { return result + "/// <reference path=\"" + filePath + "\"/>\n"; }, "");

        grunt.file.write(outFilename, fileContent);
        grunt.log.writeln(fileNames.join("\n"));
    }
...
generateTypeScriptReferencesFile(["**/*.ts"], "references.ts");

• The script also orders the files by priority according to their extension (.d.ts. and .ext.ts having the highest priority and .spec.ts the lowest, this scheme would probably be extended in the future).
• I compile the generated references.ts file with grunt-ts to a single outfile, where the debug output file will also contain Jasmine tests (originating from .spec.ts files) so the resulting file, having all its test suites integrated, can be tested both in the browser and in Node.
• Note that for a release build, for example, I simply don't include Tests/ and Benchmarks/ paths for the references file generator. In this way many different build configurations (debug, release, tests only etc.) can be made for a single project, simply based on inclusion and exclusion of directories. This cannot be currently done from within visual studio (though with the new tsconfig file it might).
• ..Other project specific build steps (such as minification, running unit tests, concatenation, adding banners, updating package versions etc.)..

An example Gruntfile.js (for LZ-UTF8) can be found here:
https://github.com/rotemdan/lzutf8.js/blob/master/Gruntfile.js

(Note this is a truly cross-platform solution and can also be run in Linux - I have it successfully building on a remote Linux container with Travis-CI)

Hope that was useful :)

[atrauzzi]

I guess I'll spend the next 5 days inventing my own build setup... 👎

[atrauzzi]

@danquirk - It seems like every explanation I read only gives a partial understanding, but I never get a full picture that I can put into practice.

For example: I'm trying to publish just some typescript interfaces as a package for two projects to communicate along. Literally no JavaScript will be produced (for now).

The top of my files read:

declare module MyProject {
    interface OneOfSeveral {
        ...

When I leave declare in, my .d.ts file contains the interface definitions. But I can't understand why when I remove declare, suddenly my .d.ts file is empty. Similarly, if I remove the module declaration all together, my interfaces are back in the .d.ts file. What is going on that's different here?

To compound the confusion, I'm normally used to structuring my files so that the namespaces are represented in a directory structure. There's a contradiction in my mind with explicitly naming modules and the way AMD works where the namespacing is usually just expressed through the filesystem. At which point, should I even bother namespacing any of my typescript code?