Experimental Java API and example #499

julienledem · 2014-05-02T20:31:45Z

A Java API to make summingbird easier to use from java (including java 8).

johnynek · 2014-05-02T20:53:18Z

summingbird-core/src/main/java/com/twitter/summingbird/javaapi/JProducer.java

+
+  JProducer<P, T> filter(Function1<T, Boolean> f);
+
+  // TODO java does not allow U super T here. used T


if you are not doing variance in this API, and you are keeping JProducer invariant, I think it is fine to just T.

…uctor

julienledem · 2014-05-06T00:54:41Z

here are some examples with different versions of java:
https://gist.github.com/julienledem/fa2127752cc4965e16da

julienledem · 2014-05-06T01:22:54Z

summingbird-example/src/main/java/com/twitter/summingbird/example/javaapi/WordCount.java

+   })).sumByKey(store, sg); // store is typed Object
+  }
+
+  public static <P extends Platform<P>> void wordCount2(JProducer<P, Status> source, Store<P, ?, String, Long> store) {


I'm thinking of removing that 2nd type parameter in Store, Service, Sink, ...
This is the actual type used by the platform, but the Producer API does not care at that point because those types have been erased by the scala compiler and replaced by Object.
The downside of that parameter is the summingbird job code must use "?" to say that is is not using it. The upside is that otherwise the Wrapper class would just contain Object when possibly we may want to access the actual type to interact with it.

This also allows people to write ill-formed jobs, and that will happen frequently. The confusion around why online stores are different from offline stores will be amplified.

This kind of type parameter is called a phantom type, and it is there to ensure that everything is consistent. Without this, you will just have to fail at runtime.

I know java users are used to that, but our goal in summingbird (scala) is that if it compiles, it runs.

julienledem · 2014-05-07T17:14:50Z

the build for scala 2.9 fails because Either has moved from the scala package to the scala.util package between 2.9 and 2.10.
This is hidden in scala because a type Either has been added in the scala package resolving to scala.util.Either but the resulting compiled class uses a different type in 2.9 and 2.10:
signature of Producer.either in 2.9:
public abstract <U> Producer<P, scala.Either<T, U>> either(Producer<P, U>);
signature of Producer.either in 2.10:
public abstract <U> Producer<P, scala.util.Either<T, U>> either(Producer<P, U>);
which means we can't cross compile this in java. The java source needs to depend on the actual type.
As either can be emulated by use of map and merge (p.map(Left(_)).merge(other.map(Right(_)))), I will just remove it from the java API.

johnynek · 2014-05-07T18:00:46Z

summingbird-core/src/main/java/com/twitter/summingbird/javaapi/JProducers.java

+    }
+  }
+
+  public static <T> Semigroup<T> semigroup(Class<T> c) {


add issue on algebird linking to this that we can move to algebird-java

twitter/algebird#316

and pull request:
twitter/algebird#317

ianoc · 2014-05-12T21:56:46Z

We should probably move this code into a summingbird-java target?

jcoveney · 2014-05-12T22:15:22Z

summingbird-core/src/main/java/com/twitter/summingbird/javaapi/Buffer.java

+
+import com.twitter.summingbird.Platform;
+
+public class Buffer<P extends Platform<P>, BUFFER, K, V> extends Wrapper<BUFFER> {


where does K and V come into play?

They are the type parameters defined in Platform#Buffer[K,V]

summingbird/summingbird-core/src/main/scala/com/twitter/summingbird/Platform.scala

Line 32 in 8d16fad

type Buffer[k, v] = Service[k, v] with Sink[(k, v)]

K is the key we do the join (or lookup) on.

Right, but I mean, we never use them on our side of the code, right? There's no connection?

It is checked here:
https://github.com/julienledem/summingbird/blob/d88553b193c24b39175bb19e3bab46deb5006997/summingbird-core/src/main/java/com/twitter/summingbird/javaapi/JKeyedProducer.java#L28
The platform will provide the right types. Those parameters are here to be carried to the Producer.

I can not express that in java but BUFFER is SOMETHING<K,V> and the Platform provides the right type.
See for example with Store:
https://github.com/julienledem/summingbird/blob/d88553b193c24b39175bb19e3bab46deb5006997/summingbird-core/src/main/java/com/twitter/summingbird/memory/javaapi/JMemory.java#L38

jcoveney · 2014-05-12T22:58:29Z

summingbird-core/src/main/java/com/twitter/summingbird/javaapi/JProducers.java

+    return (Option<T>)scala.None$.MODULE$;
+  }
+
+  public static <T> Option<T> some(T t) {


what happens if they pass null? This may trip up. In scala this shouldn't be possible but from Java it may be. If you call Option.apply, a null will become a None. Or you can throw an exception.

I will fix this.

jcoveney · 2014-05-13T00:05:54Z

Except for some nits, looks reasonable. Let's make a big fat EXTREMELY EXPERIMENTAL on this and try to get people banging on it. Also: should be in a separate package, per our discussions. Make sure that when you do that you add it to the aggregate so the tests are run.

…ample Conflicts: project/Build.scala

Experimental Java API and example

jcoveney · 2014-05-21T01:17:00Z

Thanks Julien! And god help anyone who wants to use it ;)

julienledem added 4 commits April 30, 2014 10:13

example word count in java

7724829

first stab at java API

1915471

rework

d2c1bab

cleanup

e20cad0

johnynek reviewed May 2, 2014
View reviewed changes

julienledem added 2 commits May 2, 2014 18:26

when we can't have U super T, make invariant; fix example; add constr…

e74acd3

…uctor

rework API for java 8

f18d9d0

julienledem reviewed May 6, 2014
View reviewed changes

julienledem added 3 commits May 6, 2014 17:20

add tests; improve tests

890b61f

fix usage of java8 api

8b5eb66

add proper test dependency

47688cb

removing either as it does not cross compile in java

e065068

johnynek reviewed May 7, 2014
View reviewed changes

remove unused KeyValue; cleanup tests

d88553b

jcoveney reviewed May 12, 2014
View reviewed changes

julienledem added 8 commits May 13, 2014 12:24

adress review comments; add javadoc

194e2e8

fix example

ca1a484

make separate summingbird-core-java module

c45a88a

create summingbird-storm-java

ed01e34

JStorm platform

77084af

make java tests run; fix storm test

674c2e6

fix storm test; add helper method

7ee5f75

fix flatmap test

1c3b72c

julienledem changed the title ~~Java API and example~~ Experimental Java API and example May 20, 2014

julienledem added 2 commits May 20, 2014 17:31

whitespace

8109d85

Merge branch 'develop' of github.com:twitter/summingbird into java_ex…

b3360e2

…ample Conflicts: project/Build.scala

jcoveney added a commit that referenced this pull request May 21, 2014

Merge pull request #499 from julienledem/java_example

b824dc8

Experimental Java API and example

jcoveney merged commit b824dc8 into twitter:develop May 21, 2014

johnynek mentioned this pull request Nov 22, 2015

summingbird-java #314

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental Java API and example #499

Experimental Java API and example #499

julienledem commented May 2, 2014

johnynek May 2, 2014

julienledem commented May 6, 2014

julienledem May 6, 2014

johnynek May 6, 2014

julienledem commented May 7, 2014

johnynek May 7, 2014

julienledem May 7, 2014

julienledem May 8, 2014

ianoc commented May 12, 2014

jcoveney May 12, 2014

julienledem May 12, 2014

jcoveney May 13, 2014

julienledem May 13, 2014

julienledem May 13, 2014

jcoveney May 12, 2014

julienledem May 12, 2014

jcoveney commented May 13, 2014

jcoveney commented May 21, 2014


		JProducer<P, T> filter(Function1<T, Boolean> f);

		// TODO java does not allow U super T here. used T


		import com.twitter.summingbird.Platform;

		public class Buffer<P extends Platform<P>, BUFFER, K, V> extends Wrapper<BUFFER> {

Experimental Java API and example #499

Experimental Java API and example #499

Conversation

julienledem commented May 2, 2014

Choose a reason for hiding this comment

julienledem commented May 6, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienledem commented May 7, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ianoc commented May 12, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcoveney commented May 13, 2014

jcoveney commented May 21, 2014