Spherical Harmonics incorrect for some environment maps?

[MiiBond]

Describe the bug
The spherical harmonics coefficients produced by cmgen usually appear correctly when rendered in Babylon.js or other engines. However, the ennis.exr (http://gl.ict.usc.edu/data/highresprobes/) environment map produces SH coeffs that result in very dark lighting on the positive Z axis (perhaps even negative lighting). This occurs in both Babylon.js as well as this online SH visualizer:
SH visualization of ennis.exr
I'm wondering if this is somehow related to there being such a large range in values in this particular image.

To Reproduce
Steps to reproduce the behavior:

1. Generate the SH coeffs for ennis.exr using the cmgen command line tool.
2. Enter them on the http://ed.ilogues.com/2016/02/19/interactive-spherical-harmonic-visualization to visualize the results.

Expected behavior
The dark side of the sphere should at least have some lighting, especially with high exposure but it always appears black. Perhaps I have a fundamental misunderstanding of SH's that's affecting my expectations.

Screenshots

[romainguy]

@pixelflinger is our SH specialist.

A couple of things to note:

• We pre-divide the SH by π so we don't have to do it in the shaders. I don't know if those other engines do this or re-divide by π at runtime (which wouldn't help with the darkening you're seeing for ennis.exr)
• Our SH reconstruction in the shaders guards against negative lighting by doing a max(…, 0.0)

[romainguy]

We also premultiply the SH by the basis (we basically output colors), not sure what Babylon, etc. expect.

[MiiBond]

I actually multiply by π when exporting from Dimension to get things to look correct in Babylon but I wouldn't expect any colours near (and definitely not below) zero for this image so it seems like more than just a scale factor issue, no?

[MiiBond]

Okay, so using ./cmgen -f .hdr -i --sh-output=ennisIr.hdr ennis.exr to reconstruct the irradiance from the SH coeffs, I get a reasonable result.

So, this issue is due to my lack of understanding in how these SH coeffs should be translated for use by Babylon or that online visualizer.

[MiiBond]

Using the output of this call as .txt as @romainguy suggested to me, gives SH coeffs that work beautifully in Babylon. So, problem solved!
Thanks, @romainguy !

[MiiBond]

I've changed my code from computeIrradianceSH3Bands to computeSH to get the correct SH coefficients for use in Babylon.js (and indeed, the colour values look much better). However, now the irradiance lighting looks flipped. I had expected the orientation to not change...
I can get a matching orientation if I change computeSH to negate the x and y coords of the direction vector. Obviously, this doesn't seem like a correct solution. Any ideas what might cause this, @pixelflinger?

[romainguy]

Looking at the code of the two methods it seems to be doing exactly the same thing. The difference could be in computeShBasis() invoked by computeSH().

[romainguy]

Interestingly, when I render the SH using cmgen and the two techniques, I get the same result both times:

./cmgen --sh-shader --sh-output=test1.png Runyon.exr

./cmgen --sh-irradiance --sh-output=test2.png Runyon.exr

[MiiBond]

Hmmm, I must be doing something wrong. I'll take another look tomorrow.
Thanks for checking.

[romainguy]

Could still be on our side if we mirror twice.

[TimvanScherpenzeel]

Perhaps I'm missing something very basic but when using the SH visualizer the Y-axis appears to be flipped (with and without --no-mirror). The blue and white is the sky, the floor should be green. It has been generated from https://hdrihaven.com/hdri/?c=outdoor&h=green_point_park. Does this also have to do with the difference in reconstruction in Filament compared to this SH visualizer?

[romainguy]

The mirroring option is only horizontal. I don't know how that visualizer works. Is lighting flipped in other engines too? What's the coordinates system of that visualizer? Filament uses right handed, Y up.

[TimvanScherpenzeel]

Got it, the visualizer appears to use Three.js so it's right handed, Y-up.
I just tested it on Cesium.js and it appears to be flipped there as well (see the white chess piece).

This is the case for me on the nightly build as well as the latest pre-release and stable release. I am using the 4K .hdr version as input.

I've used the following command: ./cmgen -f .hdr -i --sh-output=sh.txt green_point_park_4k.hdr

[romainguy]

Our SH reconstruction is pretty simple:

vec3 Irradiance_SphericalHarmonics(const vec3 n) {
    return max(
          frameUniforms.iblSH[0]
#if SPHERICAL_HARMONICS_BANDS >= 2
        + frameUniforms.iblSH[1] * (n.y)
        + frameUniforms.iblSH[2] * (n.z)
        + frameUniforms.iblSH[3] * (n.x)
#endif
#if SPHERICAL_HARMONICS_BANDS >= 3
        + frameUniforms.iblSH[4] * (n.y * n.x)
        + frameUniforms.iblSH[5] * (n.y * n.z)
        + frameUniforms.iblSH[6] * (3.0 * n.z * n.z - 1.0)
        + frameUniforms.iblSH[7] * (n.z * n.x)
        + frameUniforms.iblSH[8] * (n.x * n.x - n.y * n.y)
#endif
        , 0.0);
}

Where n is the surface normal.

[romainguy]

I wonder if the issue could come from here: https://github.com/google/filament/blob/master/tools/cmgen/src/Cubemap.h#L118

I don't know if the direction is supposed to be the direction of the light or the direction to a point on the cubemap. @pixelflinger would know.

[TimvanScherpenzeel]

I just remembered that Cesium.js is using the wrong reconstruction because their current implementation looks like this: https://github.com/AnalyticalGraphicsInc/cesium/blob/master/Source/Shaders/Builtin/Functions/sphericalHarmonics.glsl and should be using the implementation you mentioned above (as they also generate SH through cmgen).

I swapped out the reconstruction in the SH visualizer with the implementation in Filament which now renders correctly (using CubemapSH::computeSH instead of CubemapSH::computeIrradianceSH3Bands, so without the --sh-shader flag). By doing this it will now render with the correct orientation. However when I do use the --sh-shader flag I experience the same exposure issues as mentioned by @MiiBond (the orientation is correct however, without swapping it out with the Filament implementation).

[pixelflinger]

I'm having trouble tracking which problems are still problems :-)
I'm finally having some cycles to look at cmgen, will do this this week.

[pixelflinger]

I can't speak for Babylon.js -- I have no idea how they implemented SH reconstruction. But I've run some tests with filament, and generating the SH with --sh=3 and --sh-output=file gives the correct result. It also works with --sh-shader.

--sh-shader means "whatever format filament needs", so you can't really use them in another engine. However, in practice, our output seems to match what ed.ilogues.com expects.

cmgen always folds the Lambert BRDF into the irradiance (-i) SH coefficients, this happens with both --sh-shader and --sh=bands, and that's why our coefficients are pre-multiplied by 1/pi.

--sh-shader currently also premultiplies the SH coefficients by the basis scale factor, which means that sh[0] has the dimension of a color, i.e.: it is the average irradiance of the scene. You can use it directly as a linear color in the shader. You can verify this by running cmgen on the white_furnace.exr environment, --sh-shader will output SH[0]=1 as expected, and this seems to match ed.ilogues.com.

When I use --sh=3 and I plug that into ed.ilogues.com, I do get the wrong exposure and vertically inverted environment, but that's just because we're not feeding it what it expects.

We need a visualizer that takes raw SHs to test this.

I don't think there is a problem with cmgen, unless I'm not understanding what's not looking right.

Here is the image I used for my tests:

And the result I got with both --sh=3 and --sh-shader using --no-mirror:

And this is what the visualizer shows for --sh-shader --no-mirror:

The Y "inversion" comes from the fact --sh-shader pre-scales the SH coefficient with the reconstruction scale factors, so using those coefficients with the wrong reconstruction (or vice-versa) will appear scaled/inverted.

[romainguy]

So it seems that cmgen is fine, it's just that we pre-scale the coefficients for runtime efficiency. We could have a "raw" mode that outputs SH without any optimization for compatibility with other tools/engines.

@MiiBond Is that what you're asking for?