Annotate fails with large volume of test failure output

[rmtobin]

If you manage to catastrophically break something and cause a large number of tests to fail which have several lines of output, at some point, the plugin will fail due to limits on the size of Buildkite annotations:

The annotation body must be less than 50 kilobytes

It would be nice in this case to possibly truncate the annotation body so the annotation can still be made instead of failing out.

[tessereth]

Yeah, this is definitely a thing we should handle. Thanks for bringing it up. Unfortunately it's not entirely obvious the best way to handle it. For example, I can't just truncate the final markdown string because it's all html and it'll be missing closing tags. I made a token effort in #13 (released in v1.3.0) to at least report something but it's pretty clunky and just changes the format without even telling you why.

My wish list for a good solution to this is something like:

1. It always generates valid markdown/html (including all closing tags etc)
2. The user is notified of any truncation that occurs
3. The number of failures for each input type is always reported
4. As far as is sensible, the least amount of data is dropped
5. There's some sense of "fairness" around what's dropped (ie it's not just the last input type that's dropped)
6. The annotation is still generated quickly

1 and 3 are currently supported but not the rest (well, I guess 6 is supported because it doesn't do anything smart so there's nothing to take time...). The outstanding question I have though, is:

• Is it better to include the full details of a few errors or a summary of ~4^[1] times as many?

I literally have no idea which of those is better. Do you have any thoughts/opinions on how a good truncation strategy should behave?

^[1] Multiplier based on a tiny, non-representative sample, YMMV

[tessereth]

If you want to give v1.4.0 a go, it should have a more sensible truncation strategy 🙂