I’ve been developing Gradle plugins for 7 years now. Sometimes I love it, sometimes I hate it. But even when I hate it, I love hating it!

I just realized some of the Gradle APIs are like those metal wire brain teasers games:

• Are they easy? No.

• Do you have to turn them upside down for hours (or months…) trying to understand what you’re supposed to do? Yup, most probably.

• Do you get a nice feeling of completion once you figure out how to untangle them? Absolutely!

I just had a couple of “Aha!” moment very recently. This post is about the “Aha!” moment of untangling dependency resolution and project isolation.
A follow up one will be about input files.
Warning, spoilers ahead, do not read if you want to play the game yourself.

The problem

If you have a multi-project build, there are a number of occasions where aggregating artifacts is desirable. Dokka is an example. Maven publication is another one. Each project (also known as module) creates their artifacts and you want to collect them in the root project to upload everything all at once.

# aggregate all publications accross all modules and publish them
# to the Maven Central Portal
./gradlew publishToMavenCentral

Sounds easy, right?

Well… Not that much. There are a lot of traps along the way.

Project isolation

Project isolation is an initiative to make builds faster by parallelizing the creation of the task graph. You enable it with the Dorg.gradle.unsafe.isolated-projects flag.

Up until recently it wasn’t really an option to enable it but it’s been picking up steam lately!

Let’s take a simple build:


By using project isolation, building the task graph for project1 and project2can be executed in parallel. That’s right, actually using all the CPUs in your machine 🎉.

If you’ve read the documentation about sharing outputs between projects, you know that you shouldn’t do things like this:

// build.gradle.kts
dependencies {
   // Don't do this!
   add("aggregate", project(":project1").tasks.named("generatePublication")
}

From the documentation:

This publication model is unsafe and can lead to non-reproducible and hard to parallelize builds.

If the projects are evaluated concurrently then accessing the mutable Project.tasks is prone to race conditions.

If we want to support project isolation, we need to do this instead in our root (consumer) project:

// build.gradle.kts
dependencies {
   // Do this
   // This is project isolation compatible
   // (PS: this doesn't use attributes for REASONs!)
   add("aggregate", project(":project1", "outgoingPublication")
}

To expose the artifacts in our producer project, we use configurations and the artifacts {} API:

// project1/build.gradle.kts
val configuration = configurations.consumable("outgoingPublication").get()

val m2Dir = layout.buildDirectory.dir("m2/").get()
publishing {
  // Add a local repository
  repositories {
    maven {
      name = "m2"
      setUrl(uri(m2Dir.asFile.path))
    }
  }
}
artifacts {
  artifacts {
    // expose the repository to other projects
    add(configuration.name, m2Dir) {
      builtBy("publishAllPublicationsToM2Repository")
    }
  }
}

(There’s a lot in there and if you’re curious about how that works in details, take a look at this other post about Gradle configurations).

Using the configurations and artifacts API, we can express cross-project dependencies. But it’s a bit verbose.

Avoiding repetition

The above works fine but assuming you have 50 modules, you now have to do something like this:

// build.gradle.kts
dependencies {
   add("aggregate", project(":project1", "outgoingPublication")
   add("aggregate", project(":project2", "outgoingPublication")
   // ...
   // Pfewwwww, that's a long list of projects to maintain
   add("aggregate", project(":project50", "outgoingPublication")
}

Wouldn’t it be nice if instead we could register the dependency automatically? After all, the subprojects have some build logic that runs already. Could they automatically register themselves as dependencies of the root project?

To this day I haven’t found a way to add cross-project dependencies in a project isolation compatible way (Gradle issue).

But luckily there is another solution!

Lenient artifact views

The trick is in using subprojects and lenient artifact views:

// Create the configuration
val aggregate = configurations.create("aggregate")
// Add all subprojects as dependency
// ℹ️ subprojects {} is not PI compatible
// (but subprojects.forEach {} is)
subprojects.forEach {
  aggregate.dependencies.add(
    dependencies.project(":${it.name}", "outgoingPublication")
  )
}

tasks.register("zipAggregate", Zip::class.java) {
  // Use a lenient artifact view to avoid failing if some subprojects
  // do not publish anything.
  from(aggregate.incoming.artifactView { lenient(true) }.files.asFileTree)
  destinationDirectory.set(layout.buildDirectory.dir("zip"))
  archiveBaseName.set("aggregate")
}

By using a lenient artifact view, the root project can still collect all subprojects. Because the dependency is added on the explicit "outgoingPublication" configuration, there is no risk of resolving other (wrong) artifacts. And because the resolution is lenient, it will fail silently if a project doesn’t publish anything. SUCCESS!


Wrap-up

I’ve been toying with publishing and Maven Central APIs for a while now. Project isolation was something that has been itching me for the last 6 months or so.
Using lenient artifact views provides a simple solution that would otherwise require a lot of fragile code.

With this itch gone, new opportunities for publishing arise, stay tuned!

PS: this only works for cases where a single root projects depends on all the other ones but modeling a more complex project graph is still itching me…

Brainteasers pictures from mtairymd on Instructables