Make Differentials more Numbery

[oxinabox]

See #35 (comment)
and JuliaDiff/ChainRules.jl#55 (comment)

Things to consider:

• Subtype Number
• define iszero
• define -
• define /
• define convert/promote using extern
• boardcasting ? Like scalar?

I don't know that we should do all of them but with considering.
I don't particular like the idea of things implictly calling extern
But maybe it is ok.

[tkf]

Looking at AbstractDifferential subtypes, I got an impression that a lot of machinery can be replaced with LazyArrays.jl. One advantage for using LazyArrays.jl may be that it can optimize computation across multiple pullbacks. For example, the pullback for x -> x + A * x could use gemm! etc. by fusing + and *. Do you think it make sense to use LazyArrays.jl here?

(I wish there are BLAS-like kernels with diagonal β. We could then use it for ResNet-like layer...)

[oxinabox]

maybe.
I've not used LazyArrays before.
it is worth looking into.
I will report back

[nickrobinson251]

related discussion in FluxML/Zygote.jl#603

[oxinabox]

My latest thoughts in particular is the operations it makes sense to define on differentials are linear operators.

Conceptually, a differential being a vector spacish type.
(if one really wants to weaken it, it is an additive groupish type but I like my scaling)
so there really should be a common super-type of Number, AbstractArray, and AbstractDifferential.
And we should define the things that are require for that.
But not the things that make arrays and number distrinct