Unable to compile a simple Swift file (maybe not possible?)

[geelen]

Hi team, playing around with Emscripten in earnest for the first time, not my usual area of work so apologies if this is clearly never going to work. What I'd like to do is compile a simple Swift program to JS:

println("Hello, world!")

So far what I've tried is:

xcrun swift hello_world.swift -emit-bc -o hello_world.bc
emcc hello_world.bc

but that fails because it can't find certain symbols (looks like the standard library isn't present):

Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Drilling a bit further, and looking at how xcrun swift -v hello_world.swift actually works, I'm able to use this linking command to compile the bitcode to an executable using normal ld.

cp /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libswift_stdlib_core.dylib .
/usr/bin/ld hello_world.bc \
              -lSystem -arch x86_64 \
              -L . -rpath . \
              -o hello_world
./hello_world
=> Hello, world!

So, it seems like libswift_stdlib_core.dylib is the only dependency for hello_world.bc to be properly linked to an executable.

But I'm stuck - is there some equivalent between the -L and -rpath flags on ld that I should be passing to emcc? Or is a dylib like that not possible to be used in emscripten? The final command I tried was:

> emcc libswift_stdlib_core.dylib hello_world.bc                                 ~/src/experiments/swift.js • 2.1.0p0
WARNING  root: emcc: cannot find library "swift_stdlib_core"
Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Hopefully something's possible from here - Swift is a neat high-level language with a nice type system and no garbage collector, which makes me think it's a good fit for compiling to JS.

[juj]

Unfortunately it is not possible to link .dylib files to Emscripten. The reason for that is that the .dylibs already contain native machine code for x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form again. What one would have to do is implement the standard library for Swift and compile that in. The unrecognized struct value errors sounds like something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions with your own implementation?

[geelen]

Interesting, I suspected as much, but had hoped dylibs might have llvm
bitcode in there.

I'll take a look to see how much of the Swift standard library source is
available to compile from, but I fear it won't be enough.

Appreciate the response, thanks!

On Saturday, June 14, 2014, juj notifications@github.com wrote:

Unfortunately it is not possible to link .dylib files to Emscripten. The
reason for that is that the .dylibs already contain native machine code for
x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form
again. What one would have to do is implement the standard library for
Swift and compile that in. The unrecognized struct value errors sounds like
something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions
with your own implementation?

—
Reply to this email directly or view it on GitHub
#2427 (comment).

[iongion]

What if we implement our own standard lib, according to http://practicalswift.com/2014/06/14/the-swift-standard-library-list-of-built-in-functions there are just 74 functions

[JoshCheek]

If the println command was omitted, would this work? (it didn't for me, I still got the same result, but I'm very ignorant) Then one could create a library and just call it from js without it depending on anything.

Also, an update, the commands in the first post seem deprecated, here is what I tried:

Versions

$ xcrun swiftc --version
Swift version 1.0 (swift-600.0.45.3.2)
Target: x86_64-apple-darwin13.2.0

$ emcc --version
emcc (Emscripten GCC-like replacement) 1.25.0 ()
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The file to compile:

$ cat f.swift
var message = 1

The IR

$ xcrun swiftc -emit-ir f.swift
; ModuleID = '-'
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

%Si = type <{ i64 }>

@_Tv1f7messageSi = global %Si zeroinitializer, align 8

define internal void @top_level_code() {
entry:
  store i64 1, i64* getelementptr inbounds (%Si* @_Tv1f7messageSi, i32 0, i32 0), align 8
  ret void
}

define i32 @main(i32 %argc, i8** %argv) {
entry:
  %0 = call i8* @_TFSsa6C_ARGCVSs5Int32()
  %1 = bitcast i8* %0 to i32*
  store i32 %argc, i32* %1
  %2 = call i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()
  %3 = bitcast i8* %2 to i8***
  store i8** %argv, i8*** %3
  call void @top_level_code()
  ret i32 0
}

declare i8* @_TFSsa6C_ARGCVSs5Int32()

declare i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()

!llvm.module.flags = !{!0, !1, !2, !5, !6, !7, !8}

!0 = metadata !{i32 2, metadata !"Dwarf Version", i32 3}
!1 = metadata !{i32 1, metadata !"Debug Info Version", i32 1}
!2 = metadata !{i32 6, metadata !"Linker Options", metadata !3}
!3 = metadata !{metadata !4}
!4 = metadata !{metadata !"-lswiftCore"}
!5 = metadata !{i32 1, metadata !"Objective-C Version", i32 2}
!6 = metadata !{i32 1, metadata !"Objective-C Image Info Version", i32 0}
!7 = metadata !{i32 1, metadata !"Objective-C Image Info Section", metadata !"__DATA, __objc_imageinfo, regular, no_dead_strip"}
!8 = metadata !{i32 4, metadata !"Objective-C Garbage Collection", i32 0}

The SIL (IDK if it's useful, but can't hurt):

$ xcrun swiftc -emit-sil f.swift
sil_stage canonical

import Builtin
import Swift
import SwiftShims

var message: Int

// top_level_code
sil private @top_level_code : $@thin () -> () {
bb0:
  %0 = global_addr #message : $*Int               // user: %3
  %1 = integer_literal $Builtin.Word, 1           // user: %2
  %2 = struct $Int (%1 : $Builtin.Word)           // user: %3
  store %2 to %0 : $*Int                          // id: %3
  %4 = tuple ()                                   // user: %5
  return %4 : $()                                 // id: %5
}

// Swift.Int._convertFromBuiltinIntegerLiteral (Swift.Int.Type)(Builtin.Int2048) -> Swift.Int
sil public_external [transparent] @_TFSi33_convertFromBuiltinIntegerLiteralfMSiFBi2048_Si : $@thin (Builtin.Int2048, @thin Int.Type) -> Int {
bb0(%0 : $Builtin.Int2048, %1 : $@thin Int.Type):
  %2 = builtin_function_ref "s_to_s_checked_trunc_Int2048_Word" : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %3
  %3 = apply %2(%0) : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %4
  %4 = tuple_extract %3 : $(Builtin.Word, Builtin.Int1), 0 // user: %5
  %5 = struct $Int (%4 : $Builtin.Word)           // user: %6
  return %5 : $Int                                // id: %6
}

Trying to compile with emcc:

$ xcrun swiftc -emit-bc f.swift -o f.bc
$ emcc f.bc
WARNING: Linking two modules of different data layouts: '/Users/josh/.emscripten_cache/libc.bc' is 'e-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-p:32:32:32-v128:32:128-n32-S128' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'e-m:o-i64:64-f80:128-n8:16:32:64-S128'
WARNING: Linking two modules of different target triples: /Users/josh/.emscripten_cache/libc.bc' is 'asmjs-unknown-emscripten' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'x86_64-apple-darwin13.2.0'
Unknown specifier in datalayout string
UNREACHABLE executed at /Users/clb/emscripten-fastcomp/lib/IR/DataLayout.cpp:300!
0  opt                      0x00000001086d04ae llvm::sys::PrintStackTrace(__sFILE*) + 46
1  opt                      0x00000001086d07bb PrintStackTraceSignalHandler(void*) + 27
2  opt                      0x00000001086d0b4c SignalHandler(int) + 412
3  libsystem_platform.dylib 0x00007fff8b0e35aa _sigtramp + 26
4  libsystem_platform.dylib 0x00007fff6492d380 _sigtramp + 3649347056
5  opt                      0x00000001086d07eb raise + 27
6  opt                      0x00000001086d08a2 abort + 18
7  opt                      0x000000010865a7a6 llvm::llvm_unreachable_internal(char const*, char const*, unsigned int) + 198
8  opt                      0x0000000108416b74 llvm::DataLayout::parseSpecifier(llvm::StringRef) + 2804
9  opt                      0x0000000108415c57 llvm::DataLayout::init(llvm::StringRef) + 471
10 opt                      0x000000010749b47e llvm::DataLayout::DataLayout(llvm::StringRef) + 158
11 opt                      0x0000000107482ba5 llvm::DataLayout::DataLayout(llvm::StringRef) + 37
12 opt                      0x000000010747943c main + 3756
13 libdyld.dylib            0x00007fff8a2865fd start + 1
Stack dump:
0.  Program arguments: /Users/josh/code/emsdk_portable/clang/e1.25.0_64bit/opt /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc -strip-debug -internalize -internalize-public-api-list=main,malloc,free -globaldce -pnacl-abi-simplify-preopt -pnacl-abi-simplify-postopt -enable-emscripten-cxx-exceptions -o /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc.opt.bc
Traceback (most recent call last):
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/emcc", line 1224, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/tools/shared.py", line 1357, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

[JoshCheek]

If anyone can look at the above and give me an idea of what specifically it's missing, ie "it's trying to link stdlib.swift" and a link to somewhere that someone documents how they solved a similar problem, then I'll put a few hours into trying to get it. If there's any reasonable progress in that time, I'll report it here and probably keep at it.

Right now I'm just so ignorant that I don't even know what what the problem is or what to look for in terms of resolving it. But I have like 50 options I'm considering for my project, including things like Elm which I've already put a few days into playing with, so I need some guidance to not wander in circles for a months.

[kripken]

The targets need to match. As the IR shows, the swift frontend emits

target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

Emscripten uses its own triple (asmjs-unknown-emscripten). You might be able to force the frontend to use it? If not, you might be able to just edit the LLVM IR file in text format to fix it (create one using emcc, to see the right values for those two lines). But, this is risky, as the triple affects stuff that is baked into the output in various places...

[JoshCheek]

So, I'm ignorant enough that I don't know how to interpret "the targets need to match" (I don't know what "target" means, or what a "triple" is -- kinda sounds like a fixed-length set of memory, like a "tuple" in a functional language). So it's unclear to me if it's something that I fucked up (e.g. did I compile it wrong, or with my own version of some binary, since emscripten uses its own set of binaries -- e.g. python2 was not in the PATH, and I couldn't find it in the emsdk_portable, so I just made a bash script named python2 that then invoked python and forwarded the args. Seemed reasonable since I had Python 2.7.5 installed, which the readme alleged was compatible https://github.com/kripken/emscripten/blob/06961a0ef6e3d8d92d5e36ff904262fefec62bec/tests/python/readme.md, and xcrun is some binary that I don't understand, which is just sort of generally available, but didn't come from emscripten) or is it some issue caused by Swift being overly zealous about an OSX environment, or is it some flag to find to tell it to chill the fuck out and forget about being execuatble and just compile as if some other program will load it and deal with all the integration stuff? I mean, you'll notice all I do in this script is declare a variable and assign it the value 1, so what is it trying to dynamically load? And does that have anything to do with the "targets" not matching?

I'll create the file using emcc and diff them and spend an hour poring over the differences, confused and trying and failing repeatedly, to figure out what I need to do, but I don't know how. The bc format si the only one I've found so far that emcc will even consider trying to compile.

The reality is that I have no particularly good model for figuring out how this stuff works. I once spent 8 hours getting a C program to compile and load a library I was interested in, that's about the extent of my experience. Code that operates at this level operates under a model that I am simply ignorant of, so while we're in this domain, it's reasonable to me like I've never programmed before.

I'm actually pretty good at guessing and cursing repeatedly trying and failing and guessing and cursing again, but I need some sort of model or context to iterate upon. And there's a sufficiently large amount of information available out there, that I need someone who's familiar with this domain to clue me in, and imply to me that this isn't a giant waste of my time (b/c, lets be honest, I could just suck it up and write my shit in JavaScript).

[kripken]

A target is what clang/LLVM is emitting code for. When you build normally, on OSX your target triple is something like darwin-intel-apple or some other combo that contains the information that you are building for intel hardware, to run on a darwin kernel on an apple userspace. This affects codegen in various ways, and the LLVM IR is not portable because of it.

Emscripten has its own target, and emcc tells clang and LLVM to use it, -triple=asmjs-unknown-emscripten or something like that.

[JoshCheek]

Hmm. Guess I thought LLVM was the target.

Anyway, asmjs-unknown-emscripten looks correct:

$ ag asmjs-unknown-emscripten | wc -l
     115

-triple isn't the correct flag, apparently, and it tells me the wrong help file:

$ xcrun swiftc -Xllvm "-triple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '-triple=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-spiller=asmjs-unknown-emscripten'?

$ swift '(LLVM option parsing)' -help
fish: Unknown command 'swift'

$ xcrun swift '(LLVM option parsing)' -help
<unknown>:0: error: no such file or directory: '(LLVM option parsing)'

$ xcrun swift 'LLVM option parsing' -help
<unknown>:0: error: no such file or directory: 'LLVM option parsing'

$ xcrun swift 'option parsing' -help
<unknown>:0: error: no such file or directory: 'option parsing'

llvm isn't a binary, but tab-complete suggestions include llvm-gcc and llvm-g++, their help screens don't mention a triple, but do talk about a --target, but that doesn't work either. Which is strange, because it's listed in the help output, so it seems like the error should be something about using it wrong, rather than "Unknown command line argument", so maybe this isn't the binary that it invokes for llvm:

$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '--target=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-stats=asmjs-unknown-emscripten'?

# same output for each of these potential variations:
$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target" -Xllvm "asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# for that last one, it says "swift (LLVM option parsing): Did you mean '-stats'?"
# so lets verify that llvm-gcc is the right binary by seeing if `-stats` is one of its options:
$ llvm-gcc --help | grep stats

# ...nope, what the fuck is llvm?
# searching implies its binary is named "clang" for some reason,
# but no dice here either:
$ clang --help | grep stats

# man pages have any ideas?
$ man llvm-gcc
No manual entry for llvm-gcc

$ man llvm-g++
No manual entry for llvm-g++

$ man clang # this one works!

# After looking through this man page, I try -arch, which doesn't do shit
# was going to try setting MACOSX_DEPLOYMENT_TARGET env var
# but then I realized there's some way to search all man pages for "-stats"
# some stupid searching (man man) eventually I figure out:
$ man -K stats

# go through this list, and `gcc-4.8` shows up, so maybe llvm is just gcc? *shrug* lets try it
# it turns out to be fruitless
# at this point, I'm considering trying to find in the emscripten code where they invoke it
# figure I'll try googling a bit, find some docs for `llc`, which isn't a binary on my system, but does have a `-mtriple` flag, so lets try that:
$ xcrun swiftc -Xllvm "-mtriple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# run the gamut of possible ways that I'm supposed to pass this thing
# (why can't I find a fucking example of how to do this?)

# eventually, one of the options suggests "-spiller" as a correction possibility
$ man -K spiller  # ...just some thing about crypto

At this point, documenting in code would add another hour, but many searches, fancy -k and -K uses of the man flag. A google search for llvm -spiller and -stats takes me back to that llc page, look through it a second time, this time notice that they have two dashes before them, even though the suggestion when I got it wrong only has 1 dash. So, try the permutations of possible ways to pass an argument to -Xllvm with two dashes. None of them work.

Get pissed, internet search like the fourth time for an example of how to use -Xllvm flag, no results, cd to xcode and search for the string 'Xllvm', nothing. Try emsdk_portable, nothing.

Look one more time at llvm-gcc, llvm-g++, lldb, apropos llvm, clang, nothing.

Decide to go see if I can find where this happens in emsdk_portable, looks like the makefiles set a build_triple but never use it. Probably some makefile magic, and I don't feel like reading a book about makefiles just to find out how this gets persisted. Notice some of the files use LLVM_TARGET, I don't know Python, maybe that's an env var, so try compiling with that set env LLVM_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc, but it doesn't do anything (triple is still set to "x86_64-apple-darwin13.2.0"), remember the clang man pages had an env var that seemed like it had potential, so env MACOSX_DEPLOYMENT_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc. Nope.

So, I'm like 3 hours in, and literally no further than when I started. Looks like someone got Rust compiling to asm.js, maybe that's a better choice. Someone got Go compiling directly to JavaScript. Might be sufficient. Could go suffer through the Elm type system for another week or two (it was basically like this: hours of slamming myself into the wall, but I kept at it for like 3 times longer), or maybe write the thing in C, since that at least compiles, but some part of me thinks that's masochism. If I did it in Opal, I wouldn't have to learn a new language... but that's part of the appeal, and I'm not sure it'll be able to scale in directions I might want to go with my project.

Going to go for a drink and weigh the shittiness of C as a language against the fact that I'll almost certainly succeed.

[endash]

I was able to get very, very simple programs to build with the 6.3 beta at least. Unfortunately even something as simple as a string literal will not work, and regardless the resulting file has a bunch of missing symbols. Some of the symbols (swift_once) are defined in libswiftCore.dylib, so it's likely using Swift will be impossible until the standard library is open sourced.

[guidobouman]

So, luckily, this will help later on: https://developer.apple.com/swift/blog/?id=29

[nmn]

So I was trying to do the same thing, and I just stumbled upon this Issue. So glad that I won't waste my time again.

By the way did you try again with XCode 7 and the swiftc tool??

[stepanhruda]

Since Apple now supports LLVM Bitcode, would that enable including dylibs that contain it?

[kripken]

LLVM bitcode is not portable. Bitcode for the App Store would be specific to Apple's hardware and OS.

[stepanhruda]

Thanks, that makes sense. In case anyone is interested in more details as I was, LLVM FAQ: Can I compile C or C++ code to platform-independent LLVM bitcode?