Performance issue: @DataPoints called N+1 times for each theory

[bloritsch]

I created a set of theories to verify the equals(), hashCode()
relationship. While I had the equals and reciprocal equals working
properly, I was surprised at the unintended hashCode collisions I had in
my code. More importantly I was surprised at how often the @DataPoints
method was called to set up the tests. It was called no less than 61
times for 60 discrete data points. That seems a bit excessive to me.

I set up a bunch of data with the assumption that it was set up once,
and then the contents were copied as needed. Essentially the expensive
set up work is done 61 times, accounting for a large portion of the 37
seconds it takes to run all my tests. If the @DataPoints method is only
called once, and the values are copied as necessary, then the
implementation would be much faster.

[dsaff]

@bloritsch,

Sorry that no progress has been made on this issue in 3 years! Are you still interested in a solution?

[dsaff]

@bloritsch, I'm assuming it became no longer an issue. Closing for now.

[dev930018]

This is still an issue as of 4.12 and can cause a huge unexpected performance hit.

Minimal example

For N=5 datapoints of type T=int and M=3 arguments of type T to a Theory

import org.junit.runner.RunWith;
import org.junit.experimental.theories.Theory;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.DataPoints;
import static org.junit.Assert.assertTrue;

@RunWith(Theories.class)
public class DataPointsBug {

    @DataPoints
    public static int[] generateDataPoints() {
        System.out.println("generateDataPoints()");
        // Expensive operation
        try {
            Thread.sleep(1000);
        } catch (InterruptedException ex) {
        }
        return new int[] {1,2,3,4,5};
    }

    @Theory
    public void manyArguments(int a, int b, int c) {
        // Cheap test
        assertTrue(a > 0 && b > 0 && c > 0);
    }
}

Output

generateDataPoints()
<line repeated 30 more times>

Finished in 31s 45ms

In general, for N datapoints of type T and a Theory taking M arguments of type T, this requires S = sum(1 + N + N^2 + ... + N^M) ~ O(N^M), which seems pretty excessive...

Workaround

It is an easy-enough workaround to store the generator output once and annotate the variable with @DataPoints:

import org.junit.runner.RunWith;
import org.junit.experimental.theories.Theory;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.DataPoints;
import static org.junit.Assert.assertTrue;

@RunWith(Theories.class)
public class DataPointsBug {

    private static int[] generateDataPoints() {
        System.out.println("generateDataPoints()");
        // Expensive operation
        try {
            Thread.sleep(1000);
        } catch (InterruptedException ex) {
        }
        return new int[] {1,2,3,4,5};
    }
    @DataPoints
    public static int[] dataPoints = generateDataPoints();

    @Theory
    public void manyArguments(int a, int b, int c) {
        // Cheap test
        assertTrue(a > 0 && b > 0 && c > 0);
    }
}

Output

generateDataPoints()

Finished in 1s 37ms

Should this be re-opened and addressed?