New config framework for the different schedulers

[na--]

This is a (very WIP) subset with the config-related changes from the massive refactoring before we can implement the arrival-rate based execution. I haven't committed any of the unfinished tests, since this is mostly intended to serve as a starting point for discussions about the different TODO statements that litter the current code. Still, it should be able to parse (and mostly validate) something like this:

export let options = {
    execution: {
        someKey: {
            type: "constant-looping-vus",
            vus: 10,
            duration: "60s",
            startTime: "10s",
            exec: "someExportedFunctionThatsNotTheDefault",
        },
        otherKey: {
            type: "variable-looping-vus",
            startVUs: 10,
            stages: [
                { target: 50, duration: "1m" },
                { target: 50, duration: "3m30s" },
                { target: 0, duration: "30s" },
            ],
        },
        yetAnotherKey: {
            type: "shared-iterations",
            vus: 50,
            iterations: 1000,
        },
        newStuff: {
            type: "constant-arrival-rate",
            rate: 20,
            timeUnit: "1m",
            duration: "1h",
            preAllocatedVUs: 50,
            maxVUs: 100,
        },
        maybeAlsoNewStuff: {
            type: "variable-arrival-rate",
            startRate: 20,
            // Initialize 50 VUs, if more than 50 are needed, they will be dynamically allocated
            preAllocatedVUs: 50,
            // Hard limit, no more than 100 VUs will be run, no matter what
            maxVUs: 100,

            stages: [
                { /* something something */ },
                //TODO: figure out stages? same as variable-looping-vus?
            ],
        },
    }
};

[mstoykov]

I am pretty sure we will have to either use a validation library or write ourself one. Even if it means adding code generation. Maybe we can leave it until we actually redo the configuration but ... I don't like code like this that will most definitely miss something

[na--]

I looked at several validation libraries, and none of them seemed like a big improvement. The biggest obstacle were the nullable types we use - we'd have to make a bunch of adapters to support them properly, which isn't very worth it...

I think we can consolidate some of the repeating checks we'll have into helper functions, or reuse some of the validator functions (but not struct tags and reflection-based stuff) from some library for things like IP addresses, urls, etc., but I'm not sure if will make sense to do more than that. Code generation for sure feels an extreme response to this problem...

And I doubt any other solution will allow us the current flexibility. Consider this validation code below:

if !carc.Rate.Valid {
	errors = append(errors, fmt.Errorf("the iteration rate isn't specified"))
} else if carc.Rate.Int64 <= 0 {
	errors = append(errors, fmt.Errorf("the iteration rate should be positive"))
}

Notice how only one of the errors will be specified, since it doesn't make sense to show both the not specified and should be positive errors at the same time. Not sure how libraries will handle that correctly. Also, we have the flexibility to write very user-friendly and specific error messages. And we can have subtly different validations between the different configs, for example the validation for the variable arrival-rate doesn't require the value and allows it to be 0, but not negative.

I'm sure we can create something with all of the flexibility, but with less boilerplate, it's just likely going to be very difficult, i.e. I'm not sure if it'd end up worth it.

[codecov-io]

Codecov Report

Merging #913 into master will increase coverage by 1.35%.
The diff coverage is 86.88%.

@@            Coverage Diff             @@
##           master     #913      +/-   ##
==========================================
+ Coverage   70.72%   72.07%   +1.35%     
==========================================
  Files         121      131      +10     
  Lines        9188     9598     +410     
==========================================
+ Hits         6498     6918     +420     
+ Misses       2283     2267      -16     
- Partials      407      413       +6

Impacted Files	Coverage Δ
cmd/common.go	40% <ø> (ø)	⬆️
cmd/login_cloud.go	9.25% <0%> (ø)	⬆️
cmd/login_influxdb.go	3.77% <0%> (ø)	⬆️
lib/scheduler/shared_iterations.go	100% <100%> (ø)
cmd/root.go	34% <100%> (+7.33%)	⬆️
lib/scheduler/helpers.go	100% <100%> (ø)
lib/scheduler/variable_arrival_rate.go	100% <100%> (ø)
js/bundle.go	83.33% <100%> (ø)	⬆️
converter/har/converter.go	64.36% <100%> (ø)	⬆️
lib/scheduler/constant_arrival_rate.go	100% <100%> (ø)
... and 26 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf8bccc...d88be34. Read the comment docs.

[mstoykov]

Mostly LGTM, just some minor questions and 2 ❤️ ;)

[na--]

I'm mostly pausing work here until #935 is done and merged back in this branch. The main remaining work there involves adding more config tests, especially ones involving different config layers. And cleaning up the JSON config file support by fixing at least some of the issues described in this comment #883 (comment)

I still plan to add a separate error type for execution config conflicts ( #913 (comment) ), but still not sure if that will be in the other branch or in this one

[mstoykov]

LGTM
although we should probably do some pretty significant testing before releasing this. I suppose you have tested it manually as well, as the test currently can't test this accurately, as you have mentioned

[na--]

@mstoykov, yeah, manual testing is definitely still necessary. Unfortunately, despite all of the refactoring efforts to make the config more testable, there are huge gaps... I've done some manual testing already, but I want to do quite a bit more today and tomorrow, so and I won't merge this until I'm fairly confident there aren't any obvious issues

[mstoykov]

LGTM