[DataFrame] Inconsistent null handling in DataFrameColumn arithmetic

[rhysparry]

Suppose that I have the following DataFrame:

var df = new DataFrame(
    new PrimitiveDataFrameColumn<int>("Foo", 10),
    new PrimitiveDataFrameColumn<int>("Bar", Enumerable.Range(1, 10))
);

When performing mathematical operations where either side is null I would expect the null to be propagated to the resulting column.

Indeed, that is what happens when the null value is the left-hand operand. E.g.

df.Columns["Foo"] + df.Columns["Bar"]

Here the result is a column of nulls, but if we reverse the operands:

df.Columns["Bar"] + df.Columns["Foo"]

The nulls in the Foo column are effectively treated as 0.

It looks like this occurs because the Arithmetic classes are working on the underlying buffers which don't keep track of the null values (that seems to be tracked in a separate NullBitMapBuffers property on the container).

[pgovind]

Yup, you're right. I repro'd it with "Foo" have a null value and "Bar" having no null values. This is a bug in all our arithmetic methods. In the first case, we clone Foo first, so the null information gets carried over to the result correctly. In the reversed case, we clone Bar, but since it doesn't have any nulls, the null information doesn't make it through. The fix is to set the right bits in result.NullBitMapBuffers.