Simon Binder

Using native assets in existing FFI packages.

In Dart, the FFI library can be used to call functions written in other languages that compile to a shared library with a C ABI. Traditionally, this binding code was written with definitions that mirror dlopen and dlsym in C (which makes sense given that the Dart program is responsible for loading these shared libraries dynamically). For instance, the SQLite database is one such C library providing a function like this:

int sqlite3_libversion_number();

(that is perhaps the most boring function SQLite provides, but it's a good and simple example). On Linux, using it in Dart works like this:

final library = DynamicLibrary.open('libsqlite3.so');
final libversion = library.lookupFunction<Int Function(), int Function()>(
  'sqlite3_libversion_number',
);

print('SQLite version is: ${libversion()}');

Immediately, a fair bit syntactic overhead stands out. That's why FFI bindings are typically generated automatically with the ffigen package. That package can generate a Dart class exposing functions and globals from a shared library, making native code easier to use:

final library = DynamicLibrary.open('libsqlite3.so');
final wrapped = NativeLibrary(library); // Class generated by ffigen

print('SQLite version is: ${wrapped.sqlite3_libversion_number()}');

However, a few pain points remain:

  1. Opening the dynamic libraries requires OS-specific Dart code (as the names of libraries are different between Windows, Linux and macOS).
  2. dart:ffi alone provides no mechanisms to actually deliver these libraries (DynamicLibrary.open fails if the library is unavailable). This is especially annoying for authors of Dart libraries, since they have no way of controlling how the app runs. For Flutter (which allows plugins to define a CMake/CocoaPods/SwiftPM/Gradle buildscript), I've come up with the pattern of releasing a _flutter_libs package in addition to each FFI package that does nothing more than adding buildscripts to make libraries available! Authors of Dart packages now need to deal with these native and highly platform-dependent setups. Also, not everything in Dart is Flutter, and even Flutter plugins don't work with e.g. flutter test - so the problem is far from solved.
  3. It's hard to link libraries statically (we can look up symbols with DynamicLibrary.executable), but that means symbols can't be stripped and, since the lookups happen at runtime, we can't use linker scripts to remove unused C code.

Native assets to the rescue

To solve all of these problems, the Dart team has been working on native assets. The main change is how Dart bindings to C code are written. Instead of opening a library and looking up the symbol there, we just write it as an external Dart function now:

@Native<Int Function()>()
external int sqlite3_libversion_number();

void main() {
  print('SQLite version is: ${sqlite3_libversion_number()}');
}

This alone already fixes problems one (no more DynamicLibrary to open) and three (tree-shaking in Dart can figure out which @Native functions are actually called in Dart, the compiler would then only emit undefined symbols for those, so a linker could strip out unused C functions. Or so I think it would work, I've always taken linkers for granted and don't fully understand them).

But the really big upgrade is that Dart finally has a cross-platform way to include build scripts that work without Flutter. A package can provide a hook/build.dart file responsible for providing the native libraries, and both Dart and Flutter run those files when needed. Because these scripts are just Dart, even complex builds are fairly easy to express. For instance, this is what I'm doing to download SQLite source files and then compile them (granted, I had a small issue with my setup because downloading source files and then passing them to the package responsible for invoking C compilers trips up the dependency tracker. I've found a hacky workaround, but I hope use cases like mine are eventually covered properly).

Native assets are an experimental feature at the moment (and require an opt-in flag), but I hope they'll become generally available soon.

Migrating existing packages

Native assets are easy to set up, but for packages already based on the older, DynamicLibrary-based interop, there's a question of how to migrate:

  1. As package authors, we don't want to break existing users! Just the act of adding a hook/build.dart file to a package forces dependents to enable the native assets feature though. Ideally, we want to support both styles of calling C code.
  2. The older interop method used a class wrapping a DynamicLibrary to provide C functions as instance methods. @Natives are global functions though, so we essentially have to duplicate the implementation effort.

I think I've found a pretty neat way to fix those problems, and I wanted to share it with other package authors who may have the same issue. First, there's a question of generating code: ffigen supports generating both styles of interop code, but only supports one per configuration. I've considered using multiple configuration files, but a small script to generate code was easier in the end. To enable maximum code reuse later, we want to invoke ffigen three times:

  1. First, stuff that doesn't change beetwen DynamicLibrary and @Native should be reused. This includes structs and types. So one ffigen invocation should generate those alone. I'm writing these into a shared.dart file.
  2. Then, we run ffigen with native assets enabled. I'm writing this into native.dart and instruct ffigen to import shared.dart for structs.
  3. And finally again with native assets disabled (written into dynamic_library.dart, again importing shared.dart for structs).

In the case of my sqlite3 package, that looks like this (full code here):

void main() {
  final generator = FfiGen();
  generator.run(_GenerationMode.shared.resolveConfig());
  generator.run(_GenerationMode.native.resolveConfig());
  generator.run(_GenerationMode.dynamicLibrary.resolveConfig());
}

enum _GenerationMode {
  dynamicLibrary('dynamic_library.dart'),
  native('native.dart'),
  shared('shared.dart');

  final String output;
  const _GenerationMode(this.output);
  Uri get outputDartUri => Uri.parse('lib/src/ffi/generated/').resolve(output);

  Config resolveConfig() {
    final libraryImports = <String, LibraryImport>{};
    Map<String, ImportedType> typeMappings = <String, ImportedType>{};

    if (this != shared) {
      typeMappings = symbolFileImportExtractor(
          [symbolUri.toString()], libraryImports, null, null);
    }

    final config = Config(
      preamble: '// ignore_for_file: type=lint',
      ffiNativeConfig: FfiNativeConfig(
          enabled: this == native,
          assetId: 'package:sqlite3_native_assets/sqlite3_native_assets.dart'),
      output: outputDartUri,
      entryPoints: [Uri.parse('assets/sqlite3.h')],
      symbolFile: this == shared
          ? SymbolFile(Uri.parse('shared.dart'), symbolUri)
          : null,
      structDecl: _includeSqlite3Only,
      functionDecl: this == shared ? _includeNothing : _includeSqlite3Only,
      globals: this == shared ? _includeNothing : _includeSqlite3Only,
    ) as ConfigImpl;

    // Adding these throug the constructor unmangles their keys, which causes
    // the header parser to no longer recognize them.
    config.libraryImports.addAll(libraryImports);
    config.usrTypeMappings.addAll(typeMappings);
    return config;
  }

  static Uri symbolUri = Uri.parse('lib/src/ffi/generated/shared_symbols.yml');
}

There are a bunch of workarounds in there I wish I didn't have to use, but hey, it works! Code that only interacts with structs can now import shared.dart and work with both with native assets and the older style of bindings.

Now comes the fun part. I've written a post-processing step for the three files that does the following:

  1. Using the analyzer package, parse the dynamic_library.dart file and extract all the type of all the methods (e.g. int sqlite3_libversion_number();).
  2. Then, patch shared.dart to add a new interface class consisting of all these extracted methods.
  3. Patch dynamic_library.dart to make the generated class extend that interface.
  4. Create a new file, native_library.dart with a class that implements the shared interface by calling the top-level native functions!

That's it, that's the whole trick! Once that's done, your code just has to replace usages of the ffigen-generated class for the dynamic library with the shared interface. Only the part actually responsible for loading the dyamic library and creating instances of these classes needs to be adapted to support both the native_library and the dynamic_library implementation. In my case of the sqlite3 package, virtually all of the existing FFI code remained unchanged! I hereby declare the second problem officially solved.

Releasing native assets packages

The first problem (writing a build hook forcing users to enable shared assets) is understandable as a design decision, but unfortunate since it will prevent packages supporting both interop styles. I'm using a workaround that only requires a bit of maintenance overhead and works perfectly fine for me:

  1. In my FFI package (package:sqlite3), I configure ffigen to use a different package as an "asset id" for the native code assets (the SQLite native library): package:sqlite3_native_assets.
  2. I then move the build scripts into that additional package.

That way, users can continue to use package:sqlite3 without native assets. But if they want to try it out, they can simply add package:sqlite3_native_assets as a dependency and change a few lines of code.