Revamp CI/CD Infrastructure, Simplify Build #1857

nikitawootten-nist · 2023-07-14T23:01:04Z

Committer Notes

Fixes #1847

Schema gen pipeline
Converter gen pipeline
Markdown validation pipeline (linkcheck)
Metaschema validation pipeline (schematron)
stretch Generated content (schemas, converters) validation pipeline
New pipelines wired in CI/CD
stretch XSpec tests NOTE: tests are outdated so their results will be ignored until a future PR fixes them
Archive artifacts

Also:

metaschema-xslt points at the correct revision of metaschema (avoid duplicate submodules)
New build/ folder README
JSON->XML conversion adds nm:ERROR when using json instead of file input #1849

Note: we considering replacing markdown-link-check.

All Submissions:

Have you selected the correct base branch per Contributing guidance?
Have you set "Allow edits and access to secrets by maintainers"?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Have you squashed any non-relevant commits and commit messages? [instructions]
Do all automated CI/CD checks pass?

By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.

(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you included examples of how to use your new feature(s)?
Have you updated all OSCAL website and readme documentation affected by the changes you made? Changes to the OSCAL website can be made in the docs/content directory of your branch.

* Replace metaschema submodule with metaschema-xslt * Remove extraneous dependencies (pom.xml, some package.json entries) Co-authored-by: Wendell Piez <wendell.piez@nist.gov> Co-authored-by: Dmitry Cousin <dmitry.cousin@nist.gov>

* Remove generate ci documentation workflow * Create `artifacts` and `checks` makefile targets

* Started a `build/` README * Added libxml2-utils to status action setup

* Advanced metaschema-xslt submodule * Added more details to the OSCAL build readme

* Downgraded MLC to 3.10.2 to mitigate intermittent "Callback was already called" errors * Artifact CI job run multiple jobs

build/Makefile

Co-authored-by: Chris Compton <daniel.compton@nist.gov>

nikitawootten-nist · 2023-07-21T21:04:10Z

Looks like we're ready for review!

nikitawootten-nist · 2023-07-21T21:10:57Z

@wendellpiez may have found some regressions in the metaschemas via the new schematron tests, however any fixes will have to happen in the upstream #1856 branch after this PR is merged (to avoid diverging histories).

aj-stein-nist

Initial feedback, more to follow.

.github/workflows/status.yml

build/Makefile

build/README.md

Co-authored-by: A.J. Stein <alexander.stein@nist.gov>

Compton-US · 2023-07-24T17:05:59Z

We talked in the standup on the diff findings so far. One other thing is that there is no directory structure in the newly generated folder, which may be ok, but we should check the filenames because some of the naming is inconsistent and that may cause headaches for integrators. On the convertors there is a dash ('-') after the model name, and we probably want to adjust that to an underscore ('_'). e.g. oscal_ssp_ vs oscal_ssp-

nikitawootten-nist · 2023-07-24T18:02:29Z

We talked in the standup on the diff findings so far. One other thing is that there is no directory structure in the newly generated folder, which may be ok, but we should check the filenames because some of the naming is inconsistent and that may cause headaches for integrators. On the convertors there is a dash ('-') after the model name, and we probably want to adjust that to an underscore ('_'). e.g. oscal_ssp_ vs oscal_ssp-

This has been solved in usnistgov/metaschema-xslt#50.

@aj-stein-nist would you be comfortable reviewing this PR in Wendell's absence? It also aligns the metaschema-xslt with Kyle and your changes to the JSON schema generation stylesheet.

nikitawootten-nist · 2023-07-24T20:58:03Z

The newest commit also generates archive artifacts, as seen here in this test release: https://github.com/nikitawootten-nist/OSCAL/releases/tag/untagged-94aa5a273145cbd339e9

aj-stein-nist · 2023-07-24T21:09:29Z

The newest commit also generates archive artifacts, as seen here in this test release: https://github.com/nikitawootten-nist/OSCAL/releases/tag/untagged-94aa5a273145cbd339e9

So we will move from gzip from bz2? I am not preferential but we did the latter before (I pick an example release that was not one of those in the list I "backported" by manually uploading artifacts we split out).

https://github.com/usnistgov/OSCAL/releases/tag/v1.0.0-milestone3

nikitawootten-nist · 2023-07-24T21:17:51Z

So we will move from gzip from bz2? I am not preferential but we did the latter before (I pick an example release that was not one of those in the list I "backported" by manually uploading artifacts we split out).

https://github.com/usnistgov/OSCAL/releases/tag/v1.0.0-milestone3

I think you are referring to the "Source code (tar.gz)" link that GitHub generates with all releases.

The "oscal-${RELEASE}.tar.bz2" and "oscal-${RELEASE}.zip" release assets have been generated by the package release action and contain the zipped up artifacts as well as a special release readme (see https://github.com/usnistgov/OSCAL/releases/tag/v1.0.4).

The archive format has been unchanged.

aj-stein-nist · 2023-07-24T21:26:17Z

So we will move from gzip from bz2? I am not preferential but we did the latter before (I pick an example release that was not one of those in the list I "backported" by manually uploading artifacts we split out).
https://github.com/usnistgov/OSCAL/releases/tag/v1.0.0-milestone3

I think you are referring to the "Source code (tar.gz)" link that GitHub generates with all releases.

The "oscal-${RELEASE}.tar.bz2" and "oscal-${RELEASE}.zip" release assets have been generated by the package release action and contain the zipped up artifacts as well as a special release readme (see https://github.com/usnistgov/OSCAL/releases/tag/v1.0.4).

The archive format has been unchanged.

My bad, the untagged- prefix tag on your fork was gone so I found one of the latest tag releases and quickly skimmed. Then I am happy, please proceed.

…-oscal-version Prepare models, bump schema version to 1.1.0

Compton-US

Based on actions yesterday, I think we're in good shape.

…`nm:ERROR` when using `json` instead of `file` input #1849

nikitawootten-nist · 2023-07-25T15:21:51Z

One last update, Wendell and I recreated a PR made by @kylelaker that will fix #1849.

…nverter (#1849) * Replaced `metaschema` submodule with `metaschema-xslt` * The `metaschema-xslt` version also fixes #1849 * Removed all bespoke scripts and replaced them with Makefile targets * Changed CI infrastructure to use the new Makefile scripts * Removed Dockerfile and infrastructure as it is no longer needed * Changed model version to `1.1.0` Co-authored-by: Wendell Piez <wendell.piez@nist.gov> Co-authored-by: Dmitry Cousin <dmitry.cousin@nist.gov> Co-authored-by: Chris Compton <daniel.compton@nist.gov> Co-authored-by: A.J. Stein <alexander.stein@nist.gov>

nikitawootten-nist and others added 5 commits July 14, 2023 18:59

Remove old build/ci-cd, create makefile

25457b6

* Replace metaschema submodule with metaschema-xslt * Remove extraneous dependencies (pom.xml, some package.json entries) Co-authored-by: Wendell Piez <wendell.piez@nist.gov> Co-authored-by: Dmitry Cousin <dmitry.cousin@nist.gov>

Rewrite validate repo markdown workflow

0e7fc1a

Replace GHA workflows

3b459db

* Remove generate ci documentation workflow * Create `artifacts` and `checks` makefile targets

Added JSONSCHEMA/XSD validation targets

50e54dd

* Started a `build/` README * Added libxml2-utils to status action setup

Added XSpec and converter gen makefile targets

c6c98ff

* Advanced metaschema-xslt submodule * Added more details to the OSCAL build readme

nikitawootten-nist requested a review from wendellpiez as a code owner July 19, 2023 18:27

Properly capture MLC errors in Makefile target

37137f0

* Downgraded MLC to 3.10.2 to mitigate intermittent "Callback was already called" errors * Artifact CI job run multiple jobs

nikitawootten-nist linked an issue Jul 20, 2023 that may be closed by this pull request

Revamp CI/CD to use the new metaschema-xslt, generate artifacts on release #1847

Closed

8 tasks

Compton-US reviewed Jul 21, 2023

View reviewed changes

build/Makefile Show resolved Hide resolved

Added schematron compose validation targets

ae22717

Co-authored-by: Chris Compton <daniel.compton@nist.gov>

nikitawootten-nist changed the title ~~[DRAFT] Revamp CI/CD Infrastructure, Simplify Build~~ Revamp CI/CD Infrastructure, Simplify Build Jul 21, 2023

nikitawootten-nist requested review from Compton-US and aj-stein-nist July 21, 2023 21:03

aj-stein-nist reviewed Jul 22, 2023

View reviewed changes

.github/workflows/status.yml Show resolved Hide resolved

build/Makefile Show resolved Hide resolved

build/README.md Outdated Show resolved Hide resolved

build/README.md Show resolved Hide resolved

build/README.md Outdated Show resolved Hide resolved

build/README.md Show resolved Hide resolved

Update build/ README with version numbers

98f3897

Co-authored-by: A.J. Stein <alexander.stein@nist.gov>

aj-stein-nist approved these changes Jul 24, 2023

View reviewed changes

nikitawootten-nist added 2 commits July 24, 2023 14:53

Advance metaschema-xslt aligning with 1.0.6 schema

847b2b5

Generate release archive artifacts

c0476d9

aj-stein-nist and others added 2 commits July 24, 2023 17:34

Prepare updated models, bump version to 1.1.0.

faed550

Merge pull request #46 from aj-stein-nist/issue1847-release-prep-bump…

67ad172

…-oscal-version Prepare models, bump schema version to 1.1.0

Compton-US approved these changes Jul 25, 2023

View reviewed changes

Advanced metaschema-xslt submodule, fixing JSON->XML conversion adds …

8fe2138

…`nm:ERROR` when using `json` instead of `file` input #1849

nikitawootten-nist linked an issue Jul 25, 2023 that may be closed by this pull request

JSON->XML conversion adds nm:ERROR when using json instead of file input #1849

Closed

nikitawootten-nist merged commit d4e5c5c into usnistgov:feature-1802-oscal-pages-nosquash Jul 25, 2023
1 check passed

This was referenced Aug 23, 2023

Remove deprecated set-state and set-output usage from all GHA workflows by June 2023 #1522

Closed

Add metadata to each GitHub Action workflow to improve documentation #1679

Closed

Prevent certain GitHub actions from running on forks #1686

Closed

aj-stein-nist mentioned this pull request Aug 31, 2023

Mix of dashes and underscore in json schema's file names #1090

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revamp CI/CD Infrastructure, Simplify Build #1857

Revamp CI/CD Infrastructure, Simplify Build #1857

nikitawootten-nist commented Jul 14, 2023 •

edited

Loading

nikitawootten-nist commented Jul 21, 2023

nikitawootten-nist commented Jul 21, 2023

aj-stein-nist left a comment

Compton-US commented Jul 24, 2023 •

edited

Loading

nikitawootten-nist commented Jul 24, 2023

nikitawootten-nist commented Jul 24, 2023

aj-stein-nist commented Jul 24, 2023 •

edited

Loading

nikitawootten-nist commented Jul 24, 2023 •

edited

Loading

aj-stein-nist commented Jul 24, 2023

Compton-US left a comment

nikitawootten-nist commented Jul 25, 2023

Revamp CI/CD Infrastructure, Simplify Build #1857

Revamp CI/CD Infrastructure, Simplify Build #1857

Conversation

nikitawootten-nist commented Jul 14, 2023 • edited Loading

Committer Notes

All Submissions:

Changes to Core Features:

nikitawootten-nist commented Jul 21, 2023

nikitawootten-nist commented Jul 21, 2023

aj-stein-nist left a comment

Choose a reason for hiding this comment

Compton-US commented Jul 24, 2023 • edited Loading

nikitawootten-nist commented Jul 24, 2023

nikitawootten-nist commented Jul 24, 2023

aj-stein-nist commented Jul 24, 2023 • edited Loading

nikitawootten-nist commented Jul 24, 2023 • edited Loading

aj-stein-nist commented Jul 24, 2023

Compton-US left a comment

Choose a reason for hiding this comment

nikitawootten-nist commented Jul 25, 2023

nikitawootten-nist commented Jul 14, 2023 •

edited

Loading

Compton-US commented Jul 24, 2023 •

edited

Loading

aj-stein-nist commented Jul 24, 2023 •

edited

Loading

nikitawootten-nist commented Jul 24, 2023 •

edited

Loading