Using WebAssembly in Dart web apps

I maintain the sqlite3 Dart package which provides convenient Dart bindings to sqlite on the Dart VM with dart:ffi. To use sqlite3 on the web, I've previously used a wrapper around sql.js. While this worked fine for simple cases, I wanted a pure-Dart implementation that makes it easy to use the sqlite3 package on all platforms without any compromises. Recently, I've finally managed to port that package to the web with a custom sqlite library. In this article, I describe which changes I had to make to sqlite3 and how the compilation is set up. I'll link relevant parts of the Dart implementation on the way, so it might be helpful to others also interested in running existing C libraries on the web.

In recent versions, the LLVM toolchain supports emitting WebAssembly code for LLVM IR. So in theory, every C library can be compiled to WebAssembly through clang. In practice, this is much harder of course: The web assembly runtime has no access to any file system or the network, so most functions from the C standard library just can't be supported. An easy solution to this problem is Emscripten, which essentially implements these C functions in JavaScript. Emscripten also generates JavaScript glue code to access native APIs.

In my case, Emscripten was not the right tool though:

• I'm writing a Dart package to access sqlite3 on the web, not an application. So I can't include any JavaScript tags, it needs to be all Dart.
• I found it very hard to use parts of Emscripten that could be interesting for me. My initial idea was to use Emscripten for the native compilation and then re-write the generated glue code in Dart. But the documentation on the internal ABI between WebAssembly modules and Emscripten glue code was so limited it was easier to just avoid Emscripten entirely.

In my case, I was really lucky that I have a C library designed with cross-platform support in mind! sqlite3 supports Windows and POSIX platforms out-of-the-box, but is flexible enough to support custom targets too. A SQLite OS Interface, or "VFS", is a collection of methods sqlite uses to access functionality from the operating system. In sqlite3, the os_unix.c or os_win.c source files are responsible for defining the default VFS used. To make sqlite3 support the web, I just had to compile it without these files and with a os_web.c file that I wrote myself.

Implementing and compiling the VFS

To compile sqlite3 with clang, I essentially chose these options:

clang \
  --target=wasm32-unknown-wasi --sysroot $(WASI_SYSROOT) \
  -nostartfiles -Ofast \
  -Wl,--import-memory -Wl,--no-entry -Wl,--export-dynamic \
  -DSQLITE_API='__attribute__((visibility("default")))' \
  -o sqlite3.wasm
  sqlite3.c os_web.c helpers.c

Here:

• -nostartfiles and -Wl,--no-entry tell the compiler and the linker that there isn't a main function (since we're compiling a library)
• sqlite3 uses the SQLITE_API macro on functions it intends to export. For those to be visible in the generated WASM module, I had to add the __attribute__((visibility("default"))) attribute onto them.
• -Wl,--import-memory means that, instead of having the compiled module manage its own memory, the web assembly memory instance is injected when the module is instantiated. By injecting the instance, it can be shared between Dart and sqlite3. Shared memory is necessary to, for instance, read Dart strings out of char* pointers returned by sqlite3 library calls.
• -Wl,--export-dynamic makes the linker export all functions that haven't been hidden. By default, only functions reachable from the entrypoint are exported. As there is no entrypoint for libraries, this option is needed to export symbols.
• --sysroot points to an installation of the wasi SDK, which implements parts of the libc that can be implemented in WebAssembly (it really just wraps muslc for that).

With that taken care of, I still had to write the actual VFS. For this, I defined C function definitions that I then need to implement in Dart and inject into the module. Let's take the xFullPathname function which a VFS needs to support for example. Its definition is essentially int (*xFullPathname)(const char *zName, int nOut, char *zOut), where zName is the input path and nOut / zOut is an output buffer. The VFS is responsible for normalizing the zName input path (resolving ../ and whatnot). In Dart, we can use package:path for this! But how do we access that package from C? Let's see.

LLVM's linker for WebAssembly supports the import_module and import_name functions to inform it that some C functions are to be injected into the module when instantiating it. So I can define this function in a header:

#define import_dart(name) \
  __attribute__((import_module("dart"), import_name(name)))

import_dart("path_normalize") extern int dartNormalizePath(const char *zPath,
                                                           char *zOut,
                                                           int nOut);

This informs the linker that, when the dartNormalizePath function is called in C, we're actually calling an injected WebAssembly function called path_normalize in an injected module named dart. In my os_web.c implementation, I provide this function to sqlite3 for normalizing paths (source).

I used this approach to implement all VFS functions through a Dart bridge. For the file system, I've written two Dart implementations: One in-memory file system and a simple one persisting data in IndexedDB.

WebAssembly interop from Dart

Next, the compiled web assembly module needs to be loaded. I wrote a simple Dart wrapper around the WebAssembly JavaScript APIs using package:js. When instantiating a module, one needs to pass a Map<String, Map<String, Object>> for injected value. The outer map describes the modules to inject, the inner map is from symbol identifiers to the actual values.

Considering the path_normalize example, the map may look like this:

typedef Pointer = int;

final memory = Memory(MemoryDescriptor(initial: 16));

final injectedValues = {
    'env': {
      'memory': memory,
    },
    'dart': {
      'path_normalize': allowInterop((Pointer zPath, Pointer zOut, int nOut) {
          final input = memory.readString(zPath);
          final output = p.normalize(input);
          final encoded = utf8.encode(normalize);

          if (encoded.length >= length) {
            return NOMEM;
          } else {
            memory.buffer.asUint8List(zOut, encoded.length).setAll(0, encoded);
            return OK;
          }
      }),
    }
};

With the --import-memory linker flag, the compiled module will read memory from an env environment which we need to inject. The dart module with the path_normalize function is used to implement the dartNormalizePath function. Some helpers for reading strings from memory are defined here.

Once the module is loaded, exported symbols are available through Module.functions or Module.globals. For the sqlite3 implementation, all the relevant symbols are loaded as functions which are then used to implement convenient Dart sqlite3 API.

Right now, this approach in package:sqlite3 looks really promising. I can share parts of the implementation between the FFI-based native wrappers and the WASM-based web wrappers.

Future ideas

The sqlite3 package now exports three separate libraries: The common platform interface, a dart:ffi based implementation and the new WASM implementation. There's a fair bit of duplicate code between the two implementations because the C functions are essentially the same. I can't easily generalize the code to work on both platforms though, since definitions from dart:ffi aren't available in the browser.

Refactoring both implementations into a separate, very low-level wrapper around sqlite3's C APIs and shared functionality to map sqlite3 to Dart would be interesting. While we're at it, having ffigen generate JS-interop code for C libraries compiled to WebAssembly could be very helpful too.