How to override the classpath order on launch & on assembly

[kaepaokaawela]

In my (currently sbt) project, I sometimes have cases where I need to modify the behavior of a library, or some code I don't control.

When I was using sbt, on local run, it seems to specify the project build output classpath before any library jars. This way, if my compiler outputs a class with the same name as one in a jar, the jar ones were shadowed and not used. (I know this feels not nice for some, but there are no alternatives in my case.)

Also, sbt-assembly plugin has a strategy called PreferProject, that allows me to include the ones from my compiler, not from imported jar, on conflict. So I could rely on this behavior without worrying which classfile ends up in the docker image.

Is it possible to get sbt's behavior here for local runs and assembly? I'm in the middle of a build tool migration and this came up as an unexpected blocker.

[lefou]

Looks like you already know that depending on the order of classpath entries is a call for trouble. I don't recommend to rely on classpath ordering in the long term.

That aside, I know from experience that it's sometimes necessary to fiddle with such details. Welcome to Mill, where such customizations are possible and probably much easier in comparison to other build tools. Nevertheless, you should aim to fix these workaround by addressing the root causes. For example by patching and re-packaging the respective thirdparty-library and using these customized artifacts in a standardized and streamlined way.

The classpath used to run your module is assembled in runClasspath.

mill/scalalib/src/mill/scalalib/JavaModule.scala

Lines 557 to 566 in 4c3d2cb

	/**
	* All classfiles and resources from upstream modules and dependencies
	* necessary to run this module's code after compilation
	*/
	override def runClasspath: T[Seq[PathRef]] = Task {
	super.runClasspath() ++
	resolvedRunIvyDeps().toSeq ++
	transitiveLocalClasspath() ++
	localClasspath()
	}

You can customize it or follow it's dependencies (navigation features of your IDE should work) and customize those. Either replace the implementation or just apply selective changes with some costom sorter applied to super.runClasspath().

The classpath used for assembly is split into upstream dependencies upstreamAssembly and local dependencies localClasspath. Rules for collision handling and shading are configured in assemblyRules.

Keep in mind that once you customize such integral targets, you should watch Mill changes closely to be aware of any upstream changes you risk to miss when using your own implementations. Therefore I recommend to first try to customize the output (e.g. by re-ordering the classpath elements based on known paths) instead of replacing the original target definition.

[kaepaokaawela]

Thanks a lot. This did the trick for local. Let me try update assembly...

  override final def runClasspath = T {
    super.runClasspath().sortBy(_.path.toString.endsWith(".jar"))
  }

[kaepaokaawela]

For assembly, I can't access the jar name or the classpath from inside the rule.
I need a/Foo.class in my a.jar to be excluded, and the one from my classpath included.

The rules only match against the path inside the jar. I can't distinguish these where they are from.

[kaepaokaawela]

Ah, I figured it out. Because mill's assembly doesn't keep any state while writing into the jar, if I make these compilation outputs appended after the jar ones, I can effectively make the them to override anything that originates from a jar. (This is opposite from the run classpath, where I want these classpath to be appended earlier)

  override def upstreamAssemblyClasspath: T[Agg[PathRef]] = T {
    Agg.from(
      super
        .upstreamAssemblyClasspath()
        .items
        .toVector
        .sortBy(cp => !cp.path.toString.endsWith(".jar"))
    )
  }

Please don't try this at home, unless you know what you're doing 😅

(this is wrong, updated below)

[kaepaokaawela]

Actually, the above worked once and I touched something and it didn't work. I am a bit confused, still trying to figure out what's going on...

[kaepaokaawela]

mill/scalalib/src/mill/scalalib/Assembly.scala

Lines 119 to 122 in 0e50ff7

	case _ if !entries.contains(mapping) =>
	entries + (mapping -> WriteOnceEntry(entry))
	case _ =>
	entries

Ah, it needs to be specified first, same as the runClasspath ordering. I am not sure what happened in that one time it looked it was working; maybe it was something else.

  override def upstreamAssemblyClasspath: T[Agg[PathRef]] = T {
    Agg.from(
      super
        .upstreamAssemblyClasspath()
        .items
        .toVector
        .sortBy(_.path.toString.endsWith(".jar"))
    )
  }

[kaepaokaawela]

My questions are all answered, but I actually have a discussion point: if there's classpath conflict, shouldn't the one that originate from the project be preferred? I feel the dependencies having higher priority than compilation output is a bad default.

I don't know how many people rely on this (probably undefined) behavior though, so I would of course be careful to change this.

[lefou]

This is difficult. In general I agree with you and I also expressed this in past PR reviews regarding the classpath ordering, but in the end, we settled with the current logic. I guess duplicate entries are most likely always an issue.

[lefou]

• FTR, Fix conflicting dependencies between upstream JavaModules #2735