Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Infra] Check builds for runtime #3783

Closed
hs-apotell opened this issue Jul 12, 2022 · 29 comments
Closed

[Infra] Check builds for runtime #3783

hs-apotell opened this issue Jul 12, 2022 · 29 comments

Comments

@hs-apotell
Copy link
Collaborator

@parrt I am attempting to get the CI builds working for the different runtime targets. I noticed that number of runtime builds are disabled for one reason or other.

Also, the workflow is using a self-hosted runner. Does it have to work with self-hosted machines? Looks like lot of the installation for needed software is happening as part of the workflow which is already installed on Github hosted machines. Builds will be a lot faster if GitHub hosted machines are used if things like VS isn't required to install on every build. If self-hosted is a requirement, I am going to need access to be able to kick off builds. At the moment, the builds are waiting forever because there are no self-hosted machines available to pick up the pending jobs.

I have already upgraded all the actions in the the Windows workflow to the latest and also got a few runtime tests working but all on Github hosted machines.

@hs-apotell
Copy link
Collaborator Author

@ericvergnaud
Copy link
Contributor

The reason we're using self hosted is because at the time we built this, we could not get free hosting on circle-ci for macOS and windows

Despite the looks of it, the software is not installed on every build since the installer skips the install if it's already installed.

@KvanTTT
Copy link
Member

KvanTTT commented Jul 12, 2022

The reason we're using self hosted is because at the time we built this, we could not get free hosting on circle-ci for macOS and windows

But self hosted CI works unstable. For instance, the latest build of dev failed: https://github.com/antlr/antlr4/runs/7222752630?check_suite_focus=true It hardly differs from not working solution. On the other hand, it looks like circle-ci works well and stable for the latest time.

@hs-apotell great! Please suggest a pull request to review changes.

@ericvergnaud
Copy link
Contributor

It was also working well with travis-ci until they pulled the plug.
I'd suggest not jumping on conclusions here.
It's true that from time to time the self-hosted CI encounters network issues.
But if we consume too much free time from circle-ci, then we'll have no CI at all.

@hs-apotell
Copy link
Collaborator Author

I can try creating a PR to see if the same setup would work for self-hosted. I have only verified my changes on GitHub hosted machines.

The longest build is about 40 minutes (cpp) which is well under limit of 6 hours. AFAIK for public repositories there is no maximum allotted time each month. Plus, if you are paying to host it, you might as well pay for time on GitHub and let others take care of keeping the machines up to date. Optionally, you can always have both and let them run in parallel.

@parrt
Copy link
Member

parrt commented Jul 12, 2022

hi. Eric has two physical boxes so it's free so far :) minus power / internet. We'd very quickly hit the 2000 minute free limit I think. Multiplier for mac is 10x so only 200 total minutes. I turned on circlei and they they hit me with $75/month until I turned off. Given the time requirements of 40 minutes for cpp, that's only 5 builds for a month and just linux.

https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions

Any idea if we can get a free open-ended one as open source proj? Actually, that page says:

"GitHub Actions usage is free for both public repositories and self-hosted runners."

Hmm...really? then it says: "GitHub Free 500 MB 2,000minutes"

@KvanTTT
Copy link
Member

KvanTTT commented Jul 12, 2022

I'm wondering, are mac and win tests highly required? There are no system-related functionality in ANTLR runtimes.

hs-apotell added a commit to hs-apotell/antlr4 that referenced this issue Jul 12, 2022
hs-apotell added a commit to hs-apotell/antlr4 that referenced this issue Jul 12, 2022
@hs-apotell
Copy link
Collaborator Author

Please suggest a pull request to review changes.

#3785

hi. Eric has two physical boxes so it's free so far :) minus power / internet. We'd very quickly hit the 2000 minute free limit I think. Multiplier for mac is 10x so only 200 total minutes. I turned on circlei and they they hit me with $75/month until I turned off. Given the time requirements of 40 minutes for cpp, that's only 5 builds for a month and just linux.

https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions

Any idea if we can get a free open-ended one as open source proj? Actually, that page says:

"GitHub Actions usage is free for both public repositories and self-hosted runners."

Hmm...really? then it says: "GitHub Free 500 MB 2,000minutes"

This is ambiguous at best and not my personal experience with working on open-source projects. I have been contributing to chipsalliance project for over two years now and I haven't paid a penny for using Github Actions. All projects under this domain are open source and I primarily work on forks of the main repository. Each build is about an hour long (some close to 2 hours long) and runs across at all three platforms (Windows, Linux, and MacOS). You might want to get some clarification on the billing.

My understanding is Github Free is not the same as Github for open source projects. The former is for individual account holders who are using private repositories. For open source projects Github is free.

@KvanTTT
Copy link
Member

KvanTTT commented Jul 12, 2022

Maybe it makes sense to give another try to cloud GitHub CI after my big refactoring of tests (I've fixed multithreading issues).

hs-apotell added a commit to hs-apotell/antlr4 that referenced this issue Jul 13, 2022
@parrt
Copy link
Member

parrt commented Jul 13, 2022

Yeah, i'm ok trying the free web again. I gotta figure out if we're classified as "open source" or just freebie public. Weird. we're on the "$25/month" plan for antlr org but billing says "You have not made any payments.". :)

@parrt
Copy link
Member

parrt commented Jul 13, 2022

Hmm... From what I can tell, public repositories get free CI. I think we should turn it on and see what happens, @KvanTTT. For one thing it would be a reduced burden on @ericvergnaud

@parrt
Copy link
Member

parrt commented Jul 13, 2022

I'm wondering, are mac and win tests highly required? There are no system-related functionality in ANTLR runtimes.

True but it's still helpful just to see if there are bugs in the supporting programming language implementations that are exposed by antlr stuff.

@parrt
Copy link
Member

parrt commented Jul 14, 2022

We don't have a linux config for self-hosted to copy and looks like a hassle to create. haha.

@hs-apotell
Copy link
Collaborator Author

One could configure all supported configurations and platforms on GitHub cloud. It should be pretty straightforward. Self-hosted is an entirely different story.

@parrt
Copy link
Member

parrt commented Jul 15, 2022

Well we have like 9 languages to support during testing...haha. configs are going to be as complex I'd think, but it looked scary and I got bored and went back to working on antlr-as-a-service project.

@parrt
Copy link
Member

parrt commented Jul 15, 2022

@KvanTTT if you want to take a whack at this, happy to help.

@KvanTTT
Copy link
Member

KvanTTT commented Jul 15, 2022

Well we have like 9 languages to support during testing...haha

Actually 10 (including Python 2 and Python 3).

@KvanTTT if you want to take a whack at this, happy to help.

The pull request #3785 looks good to me.

@parrt
Copy link
Member

parrt commented Jul 15, 2022

Looks like @ericvergnaud has some issues that we need to resolve.

@hs-apotell
Copy link
Collaborator Author

Clarification - #3785 will work only for GitHub cloud and not for a self hosted server unless it is managed by the GitHub service. The workflow changes sets up different tools (including Python2 and Python3) using public actions which do not assume any constant paths like expected by some of the scripts. Each action however provides a way to query for the installation path for use.

Also, Python2 and Python3 are currently disabled on the self hosted server so technically they are non-functional.

@parrt
Copy link
Member

parrt commented Jul 15, 2022

Ok, can we enable this for github cloud w/o disabling self-hosted for a trial run?

@hs-apotell
Copy link
Collaborator Author

I can create an entirely separate workflow so we don't even bother changing any existing files. I would need an empty workflow file in master (GitHub doesn't pickup a new workflow unless it is introduced in master branch).

@parrt
Copy link
Member

parrt commented Jul 15, 2022

@hs-apotell
Copy link
Collaborator Author

That works. Will fill in the meat and deliver!

hs-apotell added a commit to hs-apotell/antlr4 that referenced this issue Jul 16, 2022
Adding workflow for Github hosted CI check builds. Supports all three
supported platforms with all runtime languages except swift.

Also, includes cpp native builds across all supported platforms i.e.
independent on the test run builds.

Every congiguration build generates an artifact (archive of the entire
repository) and is uploaded as a result of the build (regardless of
success or failure).
hs-apotell added a commit to hs-apotell/antlr4 that referenced this issue Jul 16, 2022
Adding workflow for Github hosted CI check builds. Supports all three
supported platforms with all runtime languages except swift.

Also, includes cpp native builds across all supported platforms i.e.
independent on the test run builds.

Every congiguration build generates an artifact (archive of the entire
repository) and is uploaded as a result of the build (regardless of
success or failure).
hs-apotell added a commit to hs-apotell/antlr4 that referenced this issue Jul 16, 2022
Adding workflow for Github hosted CI check builds. Supports all three
supported platforms with all runtime languages except swift.

Also, includes cpp native builds across all supported platforms i.e.
independent on the test run builds.

Every congiguration build generates an artifact (archive of the entire
repository) and is uploaded as a result of the build (regardless of
success or failure).
hs-apotell added a commit to hs-apotell/antlr4 that referenced this issue Jul 16, 2022
Adding workflow for Github hosted CI check builds. Supports all three
supported platforms with all runtime languages except swift.

Also, includes cpp native builds across all supported platforms i.e.
independent on the test run builds.

Every congiguration build generates an artifact (archive of the entire
repository) and is uploaded as a result of the build (regardless of
success or failure).

Signed-off-by: HS <hs@apotell.com>
@hs-apotell
Copy link
Collaborator Author

@parrt @KvanTTT #3792 Includes 33 different build configurations across all supported platforms and languages (except swift). This workflow doesn't use any existing scripts. The entire workflow is standalone.

And, here's the first successful build - https://github.com/hs-apotell/antlr4/actions/runs/2681510006

@parrt
Copy link
Member

parrt commented Jul 16, 2022

@hs-apotell WOW!

parrt pushed a commit that referenced this issue Jul 16, 2022
Adding workflow for Github hosted CI check builds. Supports all three
supported platforms with all runtime languages except swift.

Also, includes cpp native builds across all supported platforms i.e.
independent on the test run builds.

Every congiguration build generates an artifact (archive of the entire
repository) and is uploaded as a result of the build (regardless of
success or failure).

Signed-off-by: HS <hs@apotell.com>
@parrt
Copy link
Member

parrt commented Jul 16, 2022

Closed by #3792

@ericvergnaud
Copy link
Contributor

I would still merge the windows cpp for self hosted
I fear that free GitHub is not going to be free for very long

@parrt
Copy link
Member

parrt commented Jul 16, 2022

Yeah, it's unclear but I think public repos get free forever minutes. haha. BTW, this repo is ranked 1230 by stars out of all repos at github! they should let it run and run!

@hs-apotell
Copy link
Collaborator Author

I would still merge the windows cpp for self hosted

@ericvergnaud Please feel free to cherry pick whatever you need from the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants