Simon Binder

Generating files across directories with build_runner

Dart's build system with the standard build_runner implementation is a popular mechanism to auto-generate code for Dart projects. Thanks to a large ecosystem of builders, tedious tasks like serialization, persistence or writing data classes (among many others) can be automated. Even this website was generated with build_runner!

Dart's build system is designed to enable fast and incremental rebuilds. It does this by limiting which files a generator is allowed to write. For each input, all possible outputs must be known beforehand, and there may only be a fixed number of them. This limitation allows reasoning about how edits must propagate through builders, enabling incremental (yet correct!) rebuilds.

Until recently, these outputs (essentially) had to be in the same directory as the input. By convention, most Dart builders only generated a .g.dart file next to their primary input. With lots of files, seeing all the .g.dart files flying around can become distracting. In other build tools like Gradle, we prefer to generate all sources into a single directory. Thanks to a little help from my humble self, we can now do this with Dart's build system!

Generating into a directory with source_gen

This is probably the most relevant section for end users of build_runner. When you're seeing generated .g.dart files, they are most likely coming from the source_gen package. You might be depending on built_value, json_serializable or moor, but all these packages do is create hidden files for source_gen to bundle up into a .g.dart file.

Luckily, source_gen supports generating all files into a single directory! To enable this, create a build.yaml file in the root of your project. For now, it should have the following content:

targets:
  $default:
    builders:
      source_gen|combining_builder:
        options:
          build_extensions:
            '^lib/{{}}.dart': 'lib/generated/{{}}.g.dart'

This tells source_gen to consider all files in lib/, and redirect all outputs into lib/generated. The full syntax is described later in this post.

To try it out, let's add a generator. Pretty much all of them work, but json_serializable is the easiest one to setup so that's the one we're using here.

dependencies:
  json_annotation: ^4.1.0

dev_dependencies:
  json_serializable: ^5.0.0
  build_runner: ^2.1.0
  source_gen: ^1.1.0

As an example, we create a Dart file, lib/persons.dart:

import 'package:json_annotation/json_annotation.dart';

part 'generated/person.g.dart';

@JsonSerializable()
class Person {
  final String firstName;
  final String lastName;
  final DateTime? dateOfBirth;
  Person({required this.firstName, required this.lastName, this.dateOfBirth});
  factory Person.fromJson(Map<String, dynamic> json) => _$PersonFromJson(json);
  Map<String, dynamic> toJson() => _$PersonToJson(this);
}

Note that, instead of person.g.dart, we're instead declaring a part in generated/, as that's where we want outputs to be.

After a quick dart run build_runner build, we find this directory structure:

lib/
├── generated/
│   └── person.g.dart
└── person.dart

Nice! The generated file goes into generated/, and we finally got rid of all the .g.dart outputs cluttering up the rest of our code. Files in generated/ mirror the structure of inputs, so a file lib/model/person.dart would get a generated part file in lib/generated/model/person.g.dart.

Important note: When you're configuring source_gen in a build.yaml file, you are using its public API! For this reason, always add source_gen: ^1.1.0 to your pubspec before using this feature.

Understanding build outputs

The {{}} syntax used in build extensions can be a bit daunting at first. It is helpful to understand how builders are working in general. They declare build extensions as a Map<String, List<String>>. Each key in the map matches as a suffix in all files. For instance, a .dart key would enable a builder to run on all files ending with .dart. The matching list of strings in that map entry form valid outputs that builder is allowed to emit when running on a matched input. Outputs are formed by removing the matched suffix and then appending the declared output. For instance, a builder declaring {'.dart': ['.g.dart']} would emit a .g.dart file for each Dart input (as the .dart is removed and replaced with .g.dart).

To generate outputs in a different directory, builders must match more than just an extension. Instead, they need to match a whole directory that they can then change with something else in an output extension. For this, the build system allows builders to declare capture groups, which always match as many characters as possible. For instance, lib/{{}}.dart would match all Dart files somewhere below a folder ending with lib. A file lib/src/nested/input/file.dart would match this input. Here, the input matches the entire path (which is still technically a suffix) and the capture group expands to src/nested/input/file. By using say lib/generated/{{}}/.g.dart as an output extension, the builer is allowed to write lib/generated/src/nested/input/file.dart, which enables the directory moves we want.

There is one problem with that input matcher though. Consider the file lib/foo/lib/model.dart. As build extensions match suffixes, it would actually only replace lib/model.dart here, with an expected output in lib/foo/generated/model.dart. Since we want to generate into a top-level lib/generated directory, this is undesirable. A ^ at the start of an input tell's the build system that we always want to match full paths (which again, is still technically a suffix). This makes it unambiguous and enables the behavior we want.

More information on capture groups is available here.

Tips for builder authors

If you're writing a builder, you should consider making your outputs configurable so that users can choose which files your builder generates. If you're writing a shared part builder for source_gen, there's nothing more you have to do - it just works already.

For other builders, first stop using hard-coded outputs. You may be using something like buildStep.input.changeExtension() to construct an output today. As users configure your builder, this stops working! Instead, use the allowedOutputs getter on BuildStep. If you know you'll only have a single output, you can use allowedOutputs.single.

Next, you can make outputs configurable and parse them from BuilderOptions in your factory. You can take a look at the implementation in source_gen for inspiration.