Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use shorter partially-qualified names when labelling named tasks during planning #3418

Merged
merged 5 commits into from
Aug 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions build.sc
Original file line number Diff line number Diff line change
Expand Up @@ -1549,6 +1549,7 @@ object dist0 extends MillPublishJavaModule {
def moduleDeps = Seq(runner, idea)

def testTransitiveDeps = runner.testTransitiveDeps() ++ Seq(
main.graphviz.testDep(),
runner.linenumbers.testDep(),
scalalib.backgroundwrapper.testDep(),
contrib.bloop.testDep(),
Expand Down
643 changes: 0 additions & 643 deletions docs/modules/ROOT/images/VisualizeCompile.svg

This file was deleted.

739 changes: 0 additions & 739 deletions docs/modules/ROOT/images/VisualizeFoo.svg

This file was deleted.

1,213 changes: 1,213 additions & 0 deletions docs/modules/ROOT/images/VisualizeJava.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
325 changes: 0 additions & 325 deletions docs/modules/ROOT/images/VisualizePlan.svg

This file was deleted.

541 changes: 541 additions & 0 deletions docs/modules/ROOT/images/VisualizePlanJava.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
769 changes: 769 additions & 0 deletions docs/modules/ROOT/images/VisualizePlanScala.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
73 changes: 0 additions & 73 deletions docs/modules/ROOT/pages/Java_Builtin_Commands.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,76 +10,3 @@ application code, but instead are utilities that help you understand and work
with your Mill build.

include::example/javalib/basic/4-builtin-commands.adoc[]

== visualize

[source,bash]
----
$ mill show visualize foo._
[
".../out/visualize.dest/out.txt",
".../out/visualize.dest/out.dot",
".../out/visualize.dest/out.json",
".../out/visualize.dest/out.png",
".../out/visualize.dest/out.svg"
]
----

`mill show visualize` takes a subset of the Mill build graph (e.g. `+core._+`
is every task directly under the `core` module) and draws out their
relationships in `.svg` and `.png` form for you to inspect. It also generates
`.txt`, `.dot` and `.json` for easy processing by downstream tools.

The above command generates the following diagram:

image::VisualizeFoo.svg[VisualizeFoo.svg]

`visualize` can be very handy for trying to understand the dependency graph of
tasks within your Mill build.

== visualizePlan

[source,bash]
----
$ mill show visualizePlan foo.compile
[
".../out/visualizePlan.dest/out.txt",
".../out/visualizePlan.dest/out.dot",
".../out/visualizePlan.dest/out.json",
".../out/visualizePlan.dest/out.png",
".../out/visualizePlan.dest/out.svg"
]
----

`mill show visualizePlan` is similar to `mill show visualize` except that it
shows a graph of the entire build plan, including tasks not directly resolved
by the query. Tasks directly resolved are shown with a solid border, and
dependencies are shown with a dotted border.

The above command generates the following diagram:

image::VisualizeCompile.svg[VisualizeCompile.svg]

== Running Mill with custom JVM options

It's possible to pass JVM options to the Mill launcher. To do this you need to
create a `.mill-jvm-opts` file in your project's root. This file should contain
JVM options, one per line.

For example, if your build requires a lot of memory and bigger stack size, your
`.mill-jvm-opts` could look like this:

----
-Xss10m
-Xmx10G
----

The file name for passing JVM options to the Mill launcher is configurable. If
for some reason you don't want to use `.mill-jvm-opts` file name, add
`MILL_JVM_OPTS_PATH` environment variable with any other file name.


---

Come by our https://discord.com/channels/632150470000902164/940067748103487558[Discord Channel]
if you want to ask questions or say hi!
73 changes: 0 additions & 73 deletions docs/modules/ROOT/pages/Scala_Builtin_Commands.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -31,76 +31,3 @@ It prompts you to enter project name and creates a folder with that name.
You can use it to quickly generate a starter project.
There are lots of templates out there for many frameworks and tools!


== visualize

[source,bash]
----
$ mill show visualize foo._
[
".../out/visualize.dest/out.txt",
".../out/visualize.dest/out.dot",
".../out/visualize.dest/out.json",
".../out/visualize.dest/out.png",
".../out/visualize.dest/out.svg"
]
----

`mill show visualize` takes a subset of the Mill build graph (e.g. `+core._+`
is every task directly under the `core` module) and draws out their
relationships in `.svg` and `.png` form for you to inspect. It also generates
`.txt`, `.dot` and `.json` for easy processing by downstream tools.

The above command generates the following diagram:

image::VisualizeFoo.svg[VisualizeFoo.svg]

`visualize` can be very handy for trying to understand the dependency graph of
tasks within your Mill build.

== visualizePlan

[source,bash]
----
$ mill show visualizePlan foo.compile
[
".../out/visualizePlan.dest/out.txt",
".../out/visualizePlan.dest/out.dot",
".../out/visualizePlan.dest/out.json",
".../out/visualizePlan.dest/out.png",
".../out/visualizePlan.dest/out.svg"
]
----

`mill show visualizePlan` is similar to `mill show visualize` except that it
shows a graph of the entire build plan, including tasks not directly resolved
by the query. Tasks directly resolved are shown with a solid border, and
dependencies are shown with a dotted border.

The above command generates the following diagram:

image::VisualizeCompile.svg[VisualizeCompile.svg]

== Running Mill with custom JVM options

It's possible to pass JVM options to the Mill launcher. To do this you need to
create a `.mill-jvm-opts` file in your project's root. This file should contain
JVM options, one per line.

For example, if your build requires a lot of memory and bigger stack size, your
`.mill-jvm-opts` could look like this:

----
-Xss10m
-Xmx10G
----

The file name for passing JVM options to the Mill launcher is configurable. If
for some reason you don't want to use `.mill-jvm-opts` file name, add
`MILL_JVM_OPTS_PATH` environment variable with any other file name.


---

Come by our https://discord.com/channels/632150470000902164/940067748103487558[Discord Channel]
if you want to ask questions or say hi!
9 changes: 8 additions & 1 deletion example/javalib/builds/1-common-config/build.sc
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,13 @@ object foo extends RootModule with JavaModule {
// this does not apply to running externally via `.assembly
def forkEnv: T[Map[String, String]] = Map("MY_CUSTOM_ENV" -> "my-env-value")
}

// If you want to better understand how the various upstream tasks feed into
// a task of interest, such as `run`, you can visualize their relationships via
/** Usage
> mill show visualizePlan run
*/
//
// image::VisualizePlanJava.svg[VisualizePlanJava.svg]
//
//// SNIPPET:FATAL_WARNINGS
//
71 changes: 71 additions & 0 deletions example/scalalib/basic/4-builtin-commands/build.sc
Original file line number Diff line number Diff line change
Expand Up @@ -288,3 +288,74 @@ foo.compileClasspath
// * Always applies to all modules in the build.
// * Doesn't apply to `$ivy` dependencies used in the build definition itself.
//
// == visualize
//
/** Usage
> mill show visualize foo._
[
".../out/visualize.dest/out.txt",
".../out/visualize.dest/out.dot",
".../out/visualize.dest/out.json",
".../out/visualize.dest/out.png",
".../out/visualize.dest/out.svg"
]
*/
//
// `mill show visualize` takes a subset of the Mill build graph (e.g. `+core._+`
// is every task directly under the `core` module) and draws out their
// relationships in `.svg` and `.png` form for you to inspect. It also generates
// `.txt`, `.dot` and `.json` for easy processing by downstream tools.
//
// The above command generates the following diagram:
//
// image::VisualizeJava.svg[VisualizeJava.svg]
//
// `visualize` can be very handy for trying to understand the dependency graph of
// tasks within your Mill build.
//
// == visualizePlan
//
/** Usage
> mill show visualizePlan foo.run
[
".../out/visualizePlan.dest/out.txt",
".../out/visualizePlan.dest/out.dot",
".../out/visualizePlan.dest/out.json",
".../out/visualizePlan.dest/out.png",
".../out/visualizePlan.dest/out.svg"
]
*/
//
// `mill show visualizePlan` is similar to `mill show visualize` except that it
// shows a graph of the entire build plan, including tasks not directly resolved
// by the query. Tasks directly resolved are shown with a solid border, and
// dependencies are shown with a dotted border.
//
// The above command generates the following diagram:
//
// image::VisualizePlanJava.svg[VisualizePlanJava.svg]
//
// == Running Mill with custom JVM options
//
// It's possible to pass JVM options to the Mill launcher. To do this you need to
// create a `.mill-jvm-opts` file in your project's root. This file should contain
// JVM options, one per line.
//
// For example, if your build requires a lot of memory and bigger stack size, your
// `.mill-jvm-opts` could look like this:
//
// ----
// -Xss10m
// -Xmx10G
// ----
//
// The file name for passing JVM options to the Mill launcher is configurable. If
// for some reason you don't want to use `.mill-jvm-opts` file name, add
// `MILL_JVM_OPTS_PATH` environment variable with any other file name.
//
//
// ---
//
// Come by our https://discord.com/channels/632150470000902164/940067748103487558[Discord Channel]
// if you want to ask questions or say hi!
//
10 changes: 9 additions & 1 deletion example/scalalib/builds/1-common-config/build.sc
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,16 @@ object foo extends RootModule with ScalaModule {
// Additional Scala compiler options, e.g. to turn warnings into errors
def scalacOptions: T[Seq[String]] = Seq("-deprecation", "-Xfatal-warnings")
}
// If you want to better understand how the various upstream tasks feed into
// a task of interest, such as `run`, you can visualize their relationships via
/** Usage
> mill show visualizePlan run
*/
//
// image::VisualizePlanScala.svg[VisualizePlanScala.svg]
//
//// SNIPPET:END

//
//
// Note the use of `millSourcePath`, `T.dest`, and `PathRef` when preforming
// various filesystem operations:
Expand Down
67 changes: 49 additions & 18 deletions main/eval/src/mill/eval/Plan.scala
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,22 @@ import mill.api.Strict
import mill.define.{NamedTask, Segment, Segments, Task}
import mill.util.MultiBiMap

private object Plan {
private[mill] object Plan {
def plan(goals: Agg[Task[_]]): (MultiBiMap[Terminal, Task[_]], Strict.Agg[Task[_]]) = {
val transitive = Graph.transitiveTargets(goals)
val topoSorted = Graph.topoSorted(transitive)
val seen = collection.mutable.Set.empty[Segments]
val overridden = collection.mutable.Set.empty[Task[_]]
val overridden = collection.mutable.Map.empty[Segments, Seq[String]]
topoSorted.values.reverse.iterator.foreach {
case x: NamedTask[_] if x.isPrivate == Some(true) =>
// we always need to store them in the super-path
overridden.add(x)
case x: NamedTask[_] =>
if (!seen.contains(x.ctx.segments)) seen.add(x.ctx.segments)
else overridden.add(x)
case t: NamedTask[_] =>
val segments = t.ctx.segments
// we always store private tasks in the super-path to avoid collisions with
// subclass implementations with the same name
if (t.isPrivate == Some(true) || overridden.contains(segments)) {
overridden.updateWith(segments)(o => Some(o.getOrElse(Nil) ++ Seq(t.ctx.enclosing)))
} else {
overridden.updateWith(segments)(o => Some(o.getOrElse(Nil)))
}

case _ => // donothing
}

Expand All @@ -26,20 +29,48 @@ private object Plan {
case t: NamedTask[Any] =>
val segments = t.ctx.segments
val augmentedSegments =
if (!overridden(t)) segments
else {
val Segment.Label(tName) = segments.value.last
Segments(
segments.value.init ++
Seq(Segment.Label(tName + ".super")) ++
t.ctx.enclosing.split("[.# ]").filter(_ != "<empty>").map(Segment.Label)
)
}
if (!overridden(segments).contains(t.ctx.enclosing)) segments
else assignOverridenTaskSegments(overridden(segments), t)
Terminal.Labelled(t, augmentedSegments)

case t if goals.contains(t) => Terminal.Task(t)
}

(sortedGroups, transitive)
}

/**
* If a task has been overriden, give it a name by looking at all the
* other overriden tasks of the same name, and finding the shortest
* suffix that uniquely distinguishes them.
*/
private def assignOverridenTaskSegments(overriddenEnclosings: Seq[String], t: NamedTask[Any]) = {
def splitEnclosing(s: String) = s.split("[.# ]").filter(_ != "<empty>")
val segments = t.ctx.segments
val superSegmentStrings = overriddenEnclosings.map(splitEnclosing)

// Find out how many segments of the enclosing strings are identical
// among all overriden tasks, so we can drop them
val shortestSuperLength = superSegmentStrings.map(_.length).min
val dropLeft = Range(0, shortestSuperLength)
.find(i => superSegmentStrings.distinctBy(_(i)).size != 1)
.getOrElse(shortestSuperLength)

val splitted = splitEnclosing(t.ctx.enclosing)
// `dropRight(1)` to always drop the task name, which has to be
// the same for all overriden tasks with the same segments
val superSuffix0 = splitted.drop(dropLeft).dropRight(1)

// If there are no different segments between the enclosing strings,
// preserve at least one path segment which is the class name
val superSuffix =
if (superSuffix0.nonEmpty) superSuffix0.toSeq
else Seq(splitted(splitted.length - 2))

val Segment.Label(tName) = segments.value.last
Segments(
segments.value.init ++
Seq(Segment.Label(tName + ".super")) ++ superSuffix.map(Segment.Label)
)
}
}
Loading
Loading