1.9.2

Bug Fixes

• Remove incorrect manual action for adding Configuration Store to Tiltfile

Known Issues

Refer to the release notes for 1.9

How to Upgrade

Refer to the release notes for 1.9

What's Changed

Full Changelog: aissemble-root-1.9.1...aissemble-root-1.9.2

1.9

Major Additions

Universal Configuration Store

The Configuration Store is a tool that enables the various configurations for a project to be centrally defined and managed. It then provides a standardized way of accessing them, allowing the environment specific configurations to be dynamically provided to the their respective resources within the project at runtime. See the official documentation for more details on leveraging the configuration store.

aiSSEMBLE Infrastructure Helm Chart

The aiSSEMBLE Infrastructure Helm Chart contains the necessary infrastructure for deploying your project within a single umbrella chart. This chart includes support for Argo CD, Jenkins, and Nginx Ingress. See the chart README for more details.

Spark Infrastructure v2 Helm Chart

The following Helm charts have been migrated to the v2 structure and combined into a single spark-infrastructure chart. To migrate your Helm charts to use the v2 pattern, follow the instructions in the technical documentation.

• Spark Infrastructure
• Hive Metastore Service
• Hive Metastore Database

With this new chart, the aissemble-hive-mysql image is no longer being used. As a result, the image is deprecated and will not be updated or maintained moving forward. If you choose to remain on an older version of the Spark Infrastructure charts, you can continue to use the 1.8 version of the aissemble-hive-mysql image. However, we recommend upgrading to the new v2 spark-infrastructure chart to take full advantage of future fixes and improvements.

Helm Chart Updates

MLFlow Helm Chart parent version upgraded from 0.2.1 to 1.4.22. This includes an update to use the community docker image bitnami/mlflow:2.15.1-debian-12-r0 instead of the deprecated boozallen/aissemble-mlflow:1.7.0 image. This new image updates the MLFlow version from 2.3.1 to 2.15.1.

Airflow Helm Chart parent version upgraded from 1.10.0 to 1.15.0. This includes an update to use the community docker image apache/airflow:2.9.3 instead of the deprecated boozallen/aissemble-airflow:1.7.0 image. This new image updates the Airflow version from 2.6.2 to 2.9.3.

Kafka Helm Chart updated to use the community docker image bitnami/kafka:3.5.1-debian-11-r1 instead of the deprecated boozallen/aissemble-kafka:1.7.0. This new image remains on the same Kafka version 3.5.1.

ArgoCD Deployment Branch

Resolved issue when deploying with ArgoCD where apps would fail to utilize the current deploy job branch parameter. Now ArgoCD deployments will use the correct deploy branch when performing test deployments on branches other than the default dev.

Breaking Changes

There are no breaking changes in the 1.9 release.

Known Issues

There are no known issues with the 1.9 release.

Known Vulnerabilities

Date identified	Vulnerability	Severity	Package	Affected versions	CVE	Fixed in

Recommended Kubernetes Version

aiSSEMBLE recommends any consumer be on a minimum Kubernetes version of 1.30 due to security findings in 1.29. For more information on Kubernetes current security findings, view their CVE feed. If using AWS EKS, please follow AWS documentation on upgrading your clusters and node groups.

How to Upgrade

The following steps will upgrade your project to 1.9. These instructions consist of multiple phases:

• Automatic Upgrades - no manual action required
• Precondition Steps - needed in all situations
• Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
• Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name	Description
upgrade-tiltfile-aissemble-version-migration	Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration	Updates the Helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration	Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
ml-flow-dockerfile-migration	Updates the MLFlow's Dockerfile to use the bitnami/mlflow image as a base instead of the deprecated boozallen/aissemble-mlflow image
airflow-dockerfile-migration	Updates the Airflow's Dockerfile to use the bitnami/airflow image as a base instead of the deprecated boozallen/aissemble-airflow image
update-data-access-thrift-endpoint-migration	For projects using the default data-access thrift endpoint, updates to the new endpoint associated with v2 spark-infrastructure
argocd-value-file-sync-policy-configuration-migration	Updates the ArgoCD values files (<YOUR_PROJECT>-deploy/src/main/resources/) to include the syncPolicy values to enable the Configuration Store to deploy first on your cluster.
argocd-template-sync-policy-configuration-migration	Updates the ArgoCD template files (<YOUR_PROJECT>-deploy/src/main/resources/templates/) to include the syncPolicy helm function to enable the Configuration Store to deploy first on your cluster.

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+             </deactivateMigrations>
+        </configuration>
    </plugin>

Precondition Steps - Required for All Projects

Beginning the Upgrade

To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.9.2 version of the build-parent:

<parent>
    <groupId>com.boozallen.aissemble</groupId>
    <artifactId>build-parent</artifactId>
    <version>1.9.2</version>
</parent>

Conditional Steps

For Projects with Data Delivery Pipelines

It is strongly recommended that projects migrate from the old Spark infrastructure charts to the new Spark Infrastructure umbrella chart. The previous charts are now deprecated and will not receive any updates in future releases. To migrate to the new chart follow the upgrade instructions outlined on the Path to Production > Containers page in the technical documentation. Because this change is more significant than p...

1.8.2

Bug Fixes

• Fixed bug affecting the Pipeline Invocation service due to permissions issues
• Fixed the PDP migration of properties files to apply more broadly to catch more cases

Known Issues

Refer to the release notes for 1.8.0

How to Upgrade

Refer to the release notes for 1.8.0

What's Changed

• 61352ba Broaden PDP rename migration to catch more cases
• 44d816c cherry-pick ce08e16

Full Changelog: aissemble-root-1.8.1...aissemble-root-1.8.2

1.8.1

Bug Fixes

• Fixed the group name for the new Docker Maven plugin
• Fixed the CI configuration for Docker modules
• Fixed the <project>-tests-docker module's base image
• Fixed Spark Operator functionality for projects created before 1.2 and provided instructions migrating to the new Spark Operator

Known Issues

Refer to the release notes for 1.8.0

How to Upgrade

Refer to the release notes for 1.8.0

What's Changed

• #284 Create migration instructions for transitioning from v1 Spark Operator helm chart to v2 by @carter-cundiff in #299
• #284 Update hive metastore service to use 1.7.0 image only due to 1.8 incompatibilities by @carter-cundiff in #303
• #306 Resolve migration issues from Orphedomos to Fabric8 by @carter-cundiff in #315
• [#320] fix docker plugin config to account for change from orphedomos by @ewilkins-csi in #326

Full Changelog: aissemble-root-1.8.0...aissemble-root-1.8.1

1.8

Major Additions

Python Code Linter

We incorporated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's configuration.

There is a known issue with using PyLint and importing modules from the Behave package that will require a modification to Python test scripts. For existing projects, you will need to change the imports by:

- from behave import *
+ from behave import given, when, then  # pylint: disable=no-name-in-module

Transition from Orphedomos to Fabric8

aiSSEMBLE has changed its Docker management plugin from the orphedomos-maven-plugin to Fabric8's docker-maven-plugin. This will enable developers to leverage a Docker management tool that is maintained and supported by a broader community.

Helm Migration

The following Helm charts have been migrated to the v2 structure. To migrate your helm charts to use the v2 pattern, please follow the instruction in the technical documentation.

• Spark Operator (projects created after 1.2 will have this chart by default)

Breaking Changes

Note: instructions for adapting to these changes are outlined in the upgrade instructions below.

• Transitioning from orphedomos to fabric8 may result in breaking changes, see the Upgrade Steps for Projects with Customized Orphedomos Configurations for further details.
• The following docker modules have been deprecated. You need to continue to use the 1.7.0 versions rather than
  upgrade them. Please note, these will only be breaking IF you try to use the 1.8.0 versions, which would require manual
  intervention. The helm charts referencing these containers have already been updated to use 1.7.0.
  • aissemble-airflow
  • aissemble-kafka
  • aissemble-mlflow
• Upgraded Pydantic v1.10.x to 2.8.x to incorporate performance improvements and incorporate the availability of new features for future. See here for the guide on how to migrate pydantic V1 into V2.

Known Issues

• There is currently a bug with the Sagemaker training Docker image generated by the aissemble-sagemaker-training-docker Fermenter profile. The installation of the logistic-training module's requirements.txt fails, due to an unresolvable set of dependencies.
• Running a downstream project build with -Pintegration-test is currently broken due to incompatibilities that have remained from before the open-sourcing.
• If you have updated your project to include the docker registry in image names when built locally, the orphedomos-to-fabric8-migration may remove the registry. It can simply be added back without issue.
• ArgoCD apps fail to utilize the current deploy job branch parameter, see the Upgrade Steps for ArgoCD Branch Deployment below for resolution.
• For project deploying the Spark Operator V2 chart with ArgoCd, there is an issue with metadata being too long resulting in the CRD failing to deploy. See the Upgrade Steps for Spark Operator v2 ArgoCD CRD Deployment section below for resolution.

Known Vulnerabilities

Date identified	Vulnerability	Severity	Package	Affected versions	CVE	Fixed in

How to Upgrade

The following steps will upgrade your project to 1.8. These instructions consist of multiple phases:

• Automatic Upgrades - no manual action required
• Precondition Steps - needed in all situations
• Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
• Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name	Description
upgrade-tiltfile-aissemble-version-migration	Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration	Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration	Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
python-linting-migration	Updates Habushu configuration in root pom.xml to disable build failures when linting issues are detected
orphedomos-to-fabric8-migration	Updates orphedomos-maven-plugin usages to leverage fabric8's docker-maven-plugin
aiops-reference-python-migration	Updates the python packages which were renamed from aiops to aissemble.
aiops-reference-pdp-python-migration	Updates the policy decision point python packages which were renamed from aiops to aissemble.
aiops-reference-java-migration	Updates the java packages which were renamed from aiops to aissemble.
add-fabric8-to-build-migration	Ensures all Docker modules have the Docker Maven plugin defined in project/build/plugins.
fabric8-location-migration	For all aggregator projects (packaging type pom), moves plugin definitions for the Docker Maven plugin from build/plugins to build/pluginManagement/plugins.

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+             </deactivateMigrations>
+        </configuration>
    </plugin>

Precondition Steps - Required for All Projects

Beginning the Upgrade

To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.8.0 version of the build-parent:

<parent>
    <groupId>com.boozallen.aissemble</groupId>
    <artifactId>build-parent</artifactId>
    <version>1.8.2</version>
</parent>

Conditional Steps

Upgrade Steps for Projects Leveraging aiSSEMBLE Inference Pipeline

With the enablement of linting, there is a known issue with the Pylint package when linting on Python modules using the Pydantic package or Python files generated by Protobuff. As a result, when linting is enabled, the project build will error.

To resolve this error, within the <project-name>-pipelines/<pipeline-name>/<inference-step-name>/pyproject.toml file, add the configuration:

[tool.pylint.'MESSAGES CONTROL']
extension-pkg-whitelist = "pydantic"

[tool.pylint.MASTER]
ignore-patterns = '.*pb2[\S]*.py'

Upgrade Steps for Projects with Customized Orphedomos Configurations

If any extra configurations were added to the orphedomos-maven-plugin in addition to the generated defaults, executing the migration will result in loss of these extra configurations. To facilitate the upgrade, the following steps should be taken:

1. Before executing the baton migration, it is recommended to ensure some form of version control is in place to preserve your existing orphedomos-maven-plugin configurations.
2. After executing the baton migration, please see the docker-maven-plugin docs to add back any extra configurations, via corresponding configuration analogs.

Steps for Projects leveraging Pipeline Invocation Service

For projects that downgraded the Pipeline Invocation Service to 1.6.1 to workaround the known issue with the 1.7.0 service, the workaround can be removed to upgrade the se...

1.7

Major Additions

OpenLineage Namespace Conventions

Conventions for setting namespaces when leveraging Data Lineage has been updated to better follow OpenLineage's guidelines. Moving forward, namespaces should be defined in the data-lineage.properties file, such that Jobs are tied to pipelines and Datasets are tied to data sources. This is a departure from the old pattern of one single namespace property (data.lineage.namespace) being leveraged for an entire project. Refer to the GitHub docs for updated guidance. Usage of the data.lineage.namespace property in a project's data-lineage.properties file will be supported as a fallback but should not be used in practice.

Maven Build Cache

The Maven Build Cache is now enabled by default for new projects. Existing projects can reference the generation template to enable this functionality in their own projects.

Kafka Docker Image

The baseline Kafka Docker image has moved away from using the wurstmeister/kafka image (which was outdated and is no longer available) to using Bitnami's Kafka image as its base. If you are using the v2 Kafka chart managed by aiSSEMBLE, it will now pull the baseline Kafka image instead of directly using the Bitnami image. If you are still on the older v1 chart, it is already using the baseline image and will be the Bitnami flavor in 1.7.0. Kafka Connect support is still included in the baseline image.

Package Renaming

• Python modules were renamed to reflect aiSSEMBLE. These include the following.

Old Python Module	New Python Module
foundation-core-python	aissemble-core-python
foundation-model-training-api	aissemble-foundation-model-training-api
foundation-versioning-service	aissemble-foundation-versioning-service
foundation-drift-detection-client	aissemble-foundation-drift-detection-client
foundation-encryption-policy-python	aissemble-foundation-encryption-policy-python
foundation-model-lineage	aissemble-foundation-model-lineage
foundation-data-lineage-python	aissemble-foundation-data-lineage-python
foundation-messaging-python-client	aissemble-foundation-messaging-python-client
foundation-pdp-client-python	aissemble-foundation-pdp-client-python
foundation-transform-core-python	aissemble-Foundation-transform-core-python
extensions-model-training-api-sagemaker	aissemble-extensions-model-training-api-sagemaker
extensions-data-delivery-spark-py	aissemble-extensions-data-delivery-spark-py
extensions-encryption-vault-python	aissemble-extensions-encryption-vault-python
extensions-transform-spark-python	aissemble-extensions-transform-spark-python
test-data-delivery-pyspark-model	aissemble-test-data-delivery-pyspark-model
test-data-delivery-pyspark-model-basic	aissemble-test-data-delivery-pyspark-model-basic
machine-learning-inference	aissemble-machine-learning-inference
machine-learning-training	aissemble-machine-learning-training
machine-learning-training-base	aissemble-machine-learning-training-base
machine-learning-sagemaker-training	aissemble-machine-learning-sagemaker-training

• Helm Charts and their relevant modules have been renamed to the following:

Old Module Name	New Helm Chart and Module Name
extensions-helm-airflow	aissemble-airflow-chart
extensions-helm-data-access	aissemble-data-access-chart
extensions-helm-elasticsearch	aissemble-elasticsearch-chart
extensions-helm-elasticsearch-operator	aissemble-elasticsearch-operator-chart
extensions-helm-fastapi	aissemble-fastapi-chart
extensions-helm-hive-metastore-db	aissemble-hive-metastore-db-chart
extensions-helm-hive-metastore-service	aissemble-hive-metastore-service-chart
extensions-helm-inference	aissemble-inference-chart
extensions-helm-jenkins	aissemble-jenkins-chart
extensions-helm-kafka	aissemble-kafka-chart
extensions-helm-keycloak	aissemble-keycloak-chart
extensions-helm-lineage-http-consumer	aissemble-lineage-http-consumer-chart
extensions-helm-localstack	aissemble-localstack-chart
extensions-helm-metadata	aissemble-metadata-chart
extensions-helm-mlflow	aissemble-mlflow-chart
extensions-helm-pipeline-invocation	aissemble-pipeline-invocation-chart
extensions-helm-pipeline-invocation-lib	aissemble-pipeline-invocation-lib-chart
extensions-helm-policy-decision-point	aissemble-policy-decision-point-chart
extensions-helm-quarkus	aissemble-quarkus-chart
extensions-helm-sealed-secrets	aissemble-sealed-secrets-chart
extensions-helm-spark-application	aissemble-spark-application-chart
extensions-helm-spark-operator	aissemble-spark-operator-chart
extensions-helm-vault	aissemble-vault-chart
extensions-helm-versioning	aissemble-versioning-chart

Breaking Changes

Note instructions for adapting to these changes are outlined in the upgrade instructions below.

• The maven property version.clean.plugin was changed to version.maven.clean.plugin causing the *-deploy/pom.xml to be invalid.
• The specification of private maven repositories has been changed from prior releases.
• The specification of private PyPI repositories has been changed from prior releases.
• The specification of private docker repository has been changed from prior releases.
• The specification of Helm publishing repositories has been changed from prior releases.
• The Kafka home directory in the aissemble-kafka image has changed from /opt/kafka to /opt/bitnami/kafka

Known Issues

• There is currently a bug with the way the pipeline-invocation-service is accessing the spark application helm chart. The chart cannot be accessed using the --repo flag and must instead use a direct reference to the chart.

Known Vulnerabilities

Date identified	Vulnerability	Severity	Package	Affected versions	CVE	Fixed in

How to Upgrade

The following steps will upgrade your project to 1.7. These instructions consist of multiple phases:

• Automatic Upgrades - no manual action required
• Precondition Steps - needed in all situations
• Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
• Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name	Description
upgrade-tiltfile-aissemble-version-migration	Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration	Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration	Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-mlflow-v2-external-s3-migration ...