Fix #1875, #3744, #3735, #2432: Add support for app bundles via Bazel, Proguard, and build flavors

[BenHenning]

Explanation

Fixes #1875
Fixes #3744
Fixes #3735
Fixes #2432

At a high-level, this PR accomplishes three things:

1. It introduces support for building Android app bundles using Bazel
2. It introduces build flavors
3. It introduces full Proguard & resource shrinking support

Each of these have various caveats mentioned in the subsections below.

App bundle support

I initially tried using the android_application Starlark macro introduced as part of the pre-release alpha version of rules_android that was suggested by the Bazel team. However, it has some issues with wrangling resources that led to a dead-end set of errors that we couldn't work around. With help from the Bazel team, a lot of research of various bundle-tool & Bazel resources, and some referencing the rules_proto macros, I was able to build a medium-term custom version of android_application that can successfully build an app bundle. This has been tested both on local devices & via a Play Store deployment (using the internal channel). See below section for how to build & run these new targets.

Building app bundles effectively requires four steps:

1. Build a base APK (in our case, using android_binary)
2. Convert the APK to an AAB (this is technically a module AAB, not a deployable bundle)
3. Extract the module AAB and fix the directory structure (this would also be the time we'd merge it with other modules--we don't support that currently, but rules_android will out of the box)
4. Use Android bundletool to build the deployable AAB

There's a fair amount of complexity hidden behind the new Starlark rules being introduced (in oppia_android_application.bzl), especially around linking to the aapt2 & bundletool binaries (the former of which is part of the user's local Android SDK & the latter which is conveniently downloaded from GitHub). Android's public documentation for bundletool (which is linked in a few places in code) was really helpful in understanding the steps, but rules_android was also critical since Android's documentation doesn't really describe how to properly fix the directory structure when using aapt2 to convert an APK to an AAB (it actually suggests an alternative strategy of linking directly from AAPT2 rather than going via an APK). I chose to follow the APK route since it's closer to rules_android which I'm hoping will mean an easier migration in the future.

Please note the new bundle_config.pb.json file. This is actually really important: it allows us to tweak how bundletool assembles the final AAB. Since bundles are templates for Play Store to generate APKs, we needed to make sure that bundles don't split on language (which they do by default) since this would require users to be connected to the internet in order to switch languages (& it also inconveniently ruins custom language support since it strips unrecognized language values from the resources list).

Regarding CI, two new workflows are added for each of the two new build flavors (see below), both of which are AABs. These are individually uploaded in the same way that oppia.apk is uploaded. Note that we intentionally keep both the APK and AAB builds intentionally since:

1. It's easier to download and install oppia.apk to quickly test a PR
2. Our long-term app deployment strategy will probably include both deploying APKs and AABs, so we should ensure CI is set up to move toward that strategy

Finally, this PR also introduces a convenience bazel run command for running AABs by using the bundletool to deploy them to the device (this is helpful since there are two commands that need to be run to actually deploy an AAB to a device). See below section for examples of this new command. Note that this command does not support having multiple devices/emulators present at the same time, and it's actually not known how the command will behave in these circumstances.

Multiple build flavors

build_flavors.bzl introduces a macro for defining custom build flavors. In order to do this "properly" we're also adding support for dynamically transforming the app manifest during compile time to generate a version name & embed other version information within it. This simplifies the release process & consolidates version tracking to a new version.bzl file (which we conceivably may automatically update in the future if we introduce automatic releases from GitHub CI).

Some details on this:

• Computing the version name is a bit hacky: it requires the latest commit hash relative to the develop branch & this information is impossible to get within Bazel. There is support for "stamping" builds but I couldn't actually find if this was supported by android_binary. To work around this, I decided just to query the local Git repository for the hash (using a new script). However, Bazel intentionally doesn't provide access to the local repository during build time, so we are working around this by copying the .git folder of the repository to the sandbox environment when running the script. There are some issues with this, but the most noteworthy are potential performance issues when building AABs & potential caching issues since Bazel won't be properly tracking the .git folder for changes (the latter of which can be avoided by always clean building for new AABs being deployed). Remote caching will also not like these AAB files, so we may see some weird version issues in our CI environment (that should be innocuous).
• New build flavors have specific metadata defined for each version. While this is pretty flexible, it doesn't actually allow for easy plug-and-play Dagger modularization until Add support for Dagger Hilt [Blocked: #59] #1720 is addressed. Manual work is needed to support build flavors at the Dagger level & is outside the scope of both this PR & Create multiple build flavors in Bazel #2432.

To start, two flavors are being introduced: dev and alpha. 'dev' is meant to be the version of the app that has parity with //:oppia (i.e. has fully debugging support), whereas 'alpha' is meant to be our current production flavor of the app.

The following targets can be used to build the new AABs, respectively:

• bazel build //:oppia_dev
• bazel build //:oppia_alpha

As mentioned above, each of these can also be installed conveniently directly from Bazel:

• bazel run //:install_oppia_dev
• bazel run //:install_oppia_alpha

Proguard & resource shrinking

In order to have proper production flavors (such as oppia_alpha), it's important to also have a properly optimized, obfuscated, and shrunk version of the app. This is done through a series of new Proguard configuration files (.pro) which were compiled through research & trail and error (in a way where I actively tried to reduce broad -dontwarn patterns). I've played through most of the app to verify that everything works, but we should more extensively test before deploying to a broad audience. These configurations will not apply to Gradle.

Resource shrinking has also been enabled however it seems to like to strip away some critical resources that actually broke Firebase. To mitigate, this PR introduces a new raw resource file for shrink exemptions & explicitly prohibits those resources from being removed. This configuration will apply to Gradle, but I think Gradle already works around this issue.

We saw a pretty substantial reduction in AAB size: ~13.2MB (oppia_dev) -> ~7.4MB (oppia_alpha) vs. ~14.2MB (oppia.apk). This is with Proguard 3-pass optimization and resource shrinking. I think we could see much more substantial reductions in the future (partially related: #3381 & #3751).

Also, unsurprisingly there is a high cost to running Proguard (especially for the 3 configured optimization passes). On my machine it takes about an extra ~2.5 minutes just for the optimization step, so it's not recommended to use oppia_alpha for regular development (oppia_dev is intentionally not Proguarded so that builds are fast).

Note that, as mentioned above, we are now building oppia_alpha in CI as of this PR. This means that new dependencies may break Proguard and require updates to the various .pro files. This is by design--there's work needed to make sure that new or updated dependencies play nice with Proguard, and these CI builds will help us find those situations.

Miscellaneous

• Re-add support for legacy multidex / introduce app binary flavors #1875 is tentatively fixed here. We're keeping native multidex for //:oppia & //:oppia_dev, but switched back to legacy for //:oppia_alpha. This seems to be working well despite needing 2 dex files after Proguard finishes. This still needs to be tested on an API 19 image.
• This PR was tested with downstream localization demo code & has been verified to work correctly with custom languages (which is important since bundles majorly change how resources are handled when APKs are shipped to end users).
• This PR may not be able to submit as-is since it introduces a new test (for transforming manifests) that breaks the CI workflow by bringing the total test count to >256. I'm looking into ways of solving this in the meantime.
• The release flavor of the app introduced in this PR (oppia_alpha) needs to be built with compilation_mode=opt in order to produce a deployable AAB for the Play Store. This is demonstrated in CI, but I haven't yet figured out a way to define this automatically via bazelrc or the target definition yet. Test suite surpasses max number of parallel tasks supported by GitHub #3752 is tracking.

Essential Checklist

• The PR title and explanation each start with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".)
• Any changes to scripts/assets files have their rationale included in the PR explanation.
• The PR follows the style guide.
• The PR does not contain any unnecessary code changes from Android Studio (reference).
• The PR is made from a branch that's not called "develop" and is up-to-date with "develop".
• The PR is assigned to the appropriate reviewers (reference).

For UI-specific PRs only

N/A

[BenHenning]

Moving this into review for now despite it being blocked on #3752.

[BenHenning]

@rt4914 PTAL for the bits that affect UI layer & if you have any thoughts on the build aspects, @anandwana001 PTAL for Bazel-specific pieces, and @seanlip PTAL for codeowners.

[BenHenning]

Keeping assignment to myself per #3752 as well.

[seanlip]

Lgtm for codeowners.

[oppiabot]

Unassigning @seanlip since they have already approved the PR.

[BenHenning]

@anandwana001 you need to use Bazel 4.0.0 with build tools 29.0.2. This is actually why I recently pinned the version but that hasn't made it into this branch yet. I'll update it once I pull in #3757.

[BenHenning]

@anandwana001 just updated the branch--can you try again? It should give a clearer error now.

Also, hoping that CI will run this time.

[BenHenning]

Aha, it appears sharding is successfully unblocking CI for this PR. :) I think all that's needed to unblock the merge now is:

• Get @anandwana001's approval once he verifies that the targets are working locally
• Merge [RunAllTests] Fix #3752: Introduce Bazel test batching in CI #3757 as a prerequisite
• Ensure all CI checks are passing

[anandwana001]

I'm running locally now. Waiting for the result, I will update here once build is complete.

[anandwana001]

Tried from the second last commit, it failed locally for me.

As it is passing on CI, I am assuming there is something wrong locally for me.
My bazel version is 4.0.0 though.

[anandwana001]

LGTM on CI.

[oppiabot]

Hi @BenHenning, this PR is ready to be merged. Please address any remaining comments prior to merging, and feel free to merge this PR once the CI checks pass and you're happy with it. Thanks!

[BenHenning]

@anandwana001 are you building from a clean directory? If not, I suggest running bazel clean and specifically trying oppia_dev to start (since oppia_alpha runs Proguard & takes much longer). It seems like you have an extra artifact that I don't & I suspect this may be coming from old builds.

Also, are you building on Mac? That could also potentially cause differences I suppose.

[BenHenning]

To close the loop: there was an issue with one of the unpack commands on OSX--this has been patched in the latest version.