New filter strategies

[andrews05]

This PR adds 4 new heuristic filter strategies. These are all generally better than the standard number 5 strategy (MinSum) and can save even more when combined together.

• 6: Shannon entropy algorithm, as seen in LodePNG/Zopfli.
• 7: Count of distinct bigrams, as seen in pngwolf.
• 8: Shannon entropy of bigrams, inspired by the two above.
• 9: Brute force by compressing each filter on each line. This is currently configured to compress 4 lines at a time (the current filter attempt plus 3 previous lines to provide context). It's very effective and not as slow as one might expect, thanks to libdeflate's high-speed level 1.

Benchmarks on a set of test files with -o4 -a -i0 -s:

Filters	Total size	Time	Filters	Total size	Time
0,5	27572705	1:24	0,1,2,3,4,5	27411540	1:49
0,6	27404150	1:24	0,1,2,3,4,6	27321509	1:50
0,7	27376244	1:23	0,1,2,3,4,7	27292046	1:51
0,8	27406071	1:30	0,1,2,3,4,8	27305417	1:57
0,9	27348063	1:38	0,1,2,3,4,9	27170547	2:06
			0-9	27147885	3:29*

*My machine has 8 threads, so using all 10 filters pushed me over my thread count.

Additional changes made in this PR:

• Filter types now use an enum, making it much easier to keep track of them all.
• The filter byte is included in the output from filter_line, which means this is included in all heuristic evaluations. This is how LodePNG does it with the entropy strategy, and I think it makes sense.
• bit-vec has been replaced with bitvec, which includes a fast bitset necessary for the bigrams strategy. As a bonus, reductions are now about 10% faster.
• rustc-hash has been added, so the bigram entropy strategy can use the fast FxHasher (this strategy is still relatively slow though, I'm open to ideas about faster approaches that might be used). Color to palette reductions now also use this hasher for a 30% speed improvement.
• Additional tests added for changing interlacing, which helped me ensure the interlacing was correct when I replaced the bitvec.
• List of all filters shown in help.

Note that these filters are currently all opt-in. I will investigate including these in the default optimisation levels in future.

Closes #183

[andrews05]

@kornelski I'm not sure if the bitvec opt-level = 3 setting is still useful with the new crate? I didn't experience any performance concerns in debug either way.
P.S. Feel free to review.

[AlexTMjugador]

It's good to see all of these PRs coming from you lately. Thank you for your great work! ❤️

rustc-hash has been added, so the bigram entropy strategy can use the fast FxHasher (this strategy is still relatively slow though, I'm open to ideas about faster approaches that might be used). Color to palette reductions now also use this hasher for a 30% speed improvement.

My two cents here: this may be a bit stupid and you probably have thought about it already, but have you considered using aHash instead? According to its benchmarks, its may be faster than the vanilla FxHash, so it might be faster than the custom hasher that rustc-hash uses too.

[andrews05]

@AlexTMjugador Thanks for your comment!

I tried a number of hashers, which generally weren't as good as I was expecting them to be.

• The default (SipHash): cryptographic hashers are obviously not the fastest
• nohash: as the keys are just u16s I was expecting this to be a great option but it was only a small improvement
• twox-hash: this was okay but not great - the xxh3 implementation was actually horribly slow
• wyhash: this was decent, though also not as good as I was expecting based on the claims it makes
• ahash: don't recall specific results but wasn't as good as fxhash
• metrohash: as above

Other ideas that weren't as fast:

• Plain vec (65k long), like the base Entropy strategy - fast to construct but slow to iterate
• IndexMap (with FxHasher)

Thought I just had:

• Sparse vec?

Note oxipng is my first experience with rust, so any feedback on my code is welcome :)

[kornelski]

@shssoichiro I think it's good to merge

[shssoichiro]

Thanks, this looks amazing! I apologize for the slow response in getting it merged. That's completely on me.