Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid unnecessary dependency downloading by providing fetches per cache policy #494

Merged
merged 4 commits into from
Dec 20, 2018

Conversation

gehnaphore
Copy link
Contributor

Also adds basic ticker logging to indicate when they are downloading.

@alexarchambault
Copy link
Contributor

I confirm this is the right thing to do (~chaining the cache policies of coursier.CachePolicy.default), the methods on coursier.Cache should soon accept a sequence of CachePolicy rather than just one CachePolicy, with coursier.CachePolicy.default as a default value.

@alexarchambault
Copy link
Contributor

One can get unnecessary network requests with the current default parameters for cachePolicy in the methods of coursier.Cache.

Copy link
Member

@lefou lefou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I personally would love to see more ScalaDoc, at least on the newly introduced classes.

@lefou lefou merged commit 056cd88 into com-lihaoyi:master Dec 20, 2018
@lefou
Copy link
Member

lefou commented Dec 20, 2018

Thank you!

ucbjrl added a commit to ucbjrl/mill that referenced this pull request Mar 19, 2019
* bump Ammonite version

* update-readme

* Add Gitlab to VersionControl (com-lihaoyi#489)

* Added info about skipped mill server in `-i` cmdline option

See com-lihaoyi#497

* bump ammonite version

* fix tests

* Added note about experimental Evaluator commands

See com-lihaoyi#502

* First pass at splitting out worker-api from mill core. (com-lihaoyi#504)

This reduces the {scala,scalajs,scalanative}-worker dependency from the entirety of Mill to a much narrower `mill.api` module. This reduces the amount of classpath pollution within these workers, should mean they're much faster to download the first time, and reduces the amount of random junk they would pull in if they were to be used outside of the Mill project.

The interactions between the various *Modules and their *WorkerImpls has been narrowed down to the `*.api` modules, which only depend on other `*.api` modules.

A lot of things have been moved around; user code is unlikely to break, but it's possible some will if it references classes that have been moved around. Forwarders have been left for the few internal classes that Mill uses in it's own `build.sc`, to support bootstrapping. Third-party code which breaks should be a straightforward to fix just by updating imports

The `*.api` modules have minimal dependencies (mostly uPickle and os-lib) and minimal code. There is still a bunch of implementation code in there: some of it defining data-types that are commonly sent across the module/worker interface (`Agg`, `PathRef`, ...), and some of it just general helper functions that are needed both in modules and workers. The latter code isn't strictly API definitions, but for now is small enough it's not worth splitting into it's own module

* collapse boilerplate folder structure within src/ folders (com-lihaoyi#505)

* collapse boilerplate folder structure within src/ folders

* .

* Bump Mill Scala version to 2.12.8 (com-lihaoyi#507)

* Added doc to version target

* Fixed plugin usage docs for mill-osgi

* Discover - break overridesRoutes into fixed size chunks (com-lihaoyi#509)

* Discover - break overridesRoutes into fixed size chunks

* Discover - simplify lambda creation

* add LargeProjectTests

* LargeProjectTests: remove Ydelambdafy

* fix GenIdea to create required folders (com-lihaoyi#510)

* Generalize Zinc Worker (com-lihaoyi#514)

* Generalize Zinc worker

- Compiler bridges can now be either pre-compiled or on-demand-compiled
- Scala library/compiler jar discovery is now configurable
- Zinc compiler cache is now configurable, rather than being hardcoded at n=1

* .

* update constructor args

* remove duplicate util/AggWrapper.scala file

* fix

* fix

* fix

* cleanup

* bump zinc 1.2.1 -> 1.2.5

* bump Ammonite version

* Avoid unnecessary dependency downloading by providing fetches per cache policy (com-lihaoyi#494)

* Avoid unnecessary dependency downloading by providing fetches per cache policy; add ticker logging when they are downloading

* Fix GenIdeaTests by making the Log context Option[]al

* Add some comments

* Rebase and resolve

* Fixed typo and added class link

* Added some docs to result type

* Detailed the difference between Failure and Exception

* Updated changelog

* Introduced new failFast parameter to Evaluator with default `true`

Added Aborted result type.

* Added cmdline option -k/--keep-going

See com-lihaoyi#477

* Also added failFast test cases

* Added changelog entry

* adding support for customizable constructor annotations, codec and "inclusive dot" (com-lihaoyi#481)

* Allow usage of ZincWorkerImpl without hashing files (com-lihaoyi#525)

* Allow usage of ZincWorkerImpl without hashing files

This is to better support non-Mill build tools like Bazel or Make who might do their own file hashing/mtiming for change-detection

* Update ZincWorkerImpl.scala

* Update ZincWorkerImpl.scala

* Update ZincWorkerImpl.scala

* Update ZincWorkerImpl.scala

* More improvements to ZincWorkerImpl (com-lihaoyi#526)

* More improvements to ZincWorkerImpl

- Cache classloaders separately from `ScalaInstance`s
- Pre-compute `analysisMap` to speed up lookups
- Allow compile-to-jar using sbt/zinc 1.3.0-m1

* Update build.sc

* Update ZincWorkerModule.scala

* Update ZincWorkerImpl.scala

* Fixed GenIdea library path to use file:// if it is not a jar file (otherwise dest/classes and resources dirs are not properly recognized).

* Fixed com-lihaoyi#511.

* 0.3.6

* PublishModule: add --gpgKeyName flag (com-lihaoyi#530)

* PublishModule: adds gpgKeyName flag

* 1 - Intro to Mill.md: usage sample of publish updates for gpgKeyName

* Update readme.md

* Update readme.md

* Added compiler-classpath properties to generated IntelliJ xml files (com-lihaoyi#531)

* Also export compileIvyDeps as provided scope

Fixes com-lihaoyi#535

* Updated Changelog

* Fixed typo

* Fixed typo

* Corrected Java example: sources -> allSources

Fixes com-lihaoyi#538

Thanks to Jim Kleckner

* Only hold weak references to classloaders in internal cache

That way, we do not prevent class loader unloading / garbage collection.
Also, we reduce the chance to use an outdated class loader.

* Properly strip scaladoc end tags

Fixes com-lihaoyi#451

* Create a Play! module to compile the router

* Load/convert instances from classloader

* Keep twirllib module

* Add playlib (we do not want to replace twirllib)

* rebase on master

* Reworks the toplevel playlib module

This is the first commit of a redesign of the play lib module. The new
design was massively inspired from the `scalajslib` module.

It adds a specialized worker for each version of play, both workers
implement a common api from an `api`. The main module delegates to a
`loader` which dynamically looks up the bridge instance through
reflection then triggers the generation.

- adds a `RouteCompilerWorkerApi` trait which establishes the bridge to
the actual `RouteCompilerWorker`.
- drops the existing `RouterGeneratorWorker` (it is specialized by
versions of play and extracted to its own submodule).
- updates the `RouterModule` with improved settings and documentation.

* Adds the api submodule

This is the second commit of a redesign of the play lib module.

This module contains only the common `api` which is implemented by the
actual workers. It also defines a specific ADT to configure the type of
routes generator to be used for the project. The ADT feels cleaner than
a simple string but may be too restrictive.

* Adds the specialized workers

This module adds workers specialized for play 2.6.0. and 2.7.0, these
modules actually depend on playframework artifacts. They are dynamically
loaded from the `RoutesCompilerWorkerApi`.

* Updates the test of the play lib module

The tests have been updated to check the returned error if the route
file cannot be parsed properly.

* Adjusts the main build file to match the new playlib module design

* Documentation section for the Play! module

* switch to T.sources for the routesFiles setting

* update scaladoc to leverage list markers

* better aligns RouterModule with mill's naming conventions

* Added mill-jbake plugin

* Also add main-api to build library paths

* Added log statement to BuildInfo generator

* Moved BuildInfo into package mill.contrib.buildinfo

This fixes com-lihaoyi#548

See also com-lihaoyi#547

* Added proper imports to contrib module code snippets

Fixes com-lihaoyi#547

* Version bump to more recent JBake version

* Documented some mill.api classes

* Splitted IO.scala to avoid cyclic references

* Added mill-integrationtest plugin

* Search tests annotations also on inherited public methods

Fixes com-lihaoyi#553

* Add FreeBSD installation instructions (com-lihaoyi#559)

* Fixed incorrect comment

* Added fixme

* Added support for typelevel scala libraries

Fixes com-lihaoyi#534

* Reviewed exitCode handling

Propagate server stop because of version change as exit code.

* Introduced JavaModule.docOptions to add extra (Java)Doc procesing options

E.g. disabling picky JavaDoc linter with `-Xdoclint:none`.

The method is called `docOptions` instead of `javadocOptions` so that other modules like `ScalaModule` can reuse it.

* Renamed JavaModule.docOptions to javadocOptions

* Adds individual modules to configure mill to playframework's defaults

This adds :
- version configuration
- dependencies
- directory layout
- runtime server

* Tightens visibility of the internal workers

* Integrate Router module and add a trait to configure it for play

* Cleanup useless play version mappings

* Add trait to configure twirl integration for playframework

* Play modules : api only, full play and mono module project.

* Adds tests for the PlayModule and SingleModule

* Adds test for PlayApi single module

* Documents play module usage

* fixes route compiler worker selector key

The version was correctly derived into a worker key but the key was
left unused and the full version was used instead which didn't work for
non .0 versions

* Provides additionnal dependency keys for easier configuration

* Reworks version specific keys and extraction

Route compilation workers are specific to play minor version but still
added a patch level version in the key.
The playMinorVersion used to compute the version part of the key had to
add a useless trailing `.0`, this is now fixed.

* add static assets including WebJars

Signed-off-by: Jean Helou <jhe@codamens.fr>

* Extracts static asset and webjar handling to a specific trait

* Relaxes the visibility on the Twil trait.

This makes it easier for users to customize their build.

* Update ZincWorkerImpl.scala

* Fixed some formatting issues in markdown files

* Add section to documentation about ivy classifiers

* Added changes to the MillClientMain to make it only use workers running
the JVM the client was launched with.

* Properly respect JAVA_HOME env var
@lefou lefou added this to the 0.3.6 milestone Apr 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants