E3SM UXarray and Dask Examples

[rytam2]

Closes #822

Overview

Added user guide notebook on parallel loading with Dask; and workflow examples with E3SM output with visualization and calculations.

PR Checklist

General

• An issue is linked created and linked
• Add appropriate labels
• Filled out Overview and Expected Usage (if applicable) sections

Examples

• Any new notebook examples added to docs/examples/ folder
• Clear the output of all cells before committing
• New notebook files added to docs/examples.rst toctree
• New notebook files added to new entry in docs/gallery.yml with appropriate thumbnail photo in docs/_static/thumbnails/

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-26T17:50:07Z
----------------------------------------------------------------

Line #2.    # Parallel load

Try to avoid standard comments in these notebooks and include these either as markdown or headings.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-26T17:50:08Z
----------------------------------------------------------------

Make sure to emphasize that the data is lazy-loaded. If you index a data variable (i.e. soa_c1DDF), show that you don't actually see the data.

I would add another block, such as the following

Python

uxds_e3sm_parallel_load['soa_c1DDF'].compute()

to show how to get the data in me memory (can also show the .values) too

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

philipc2 commented on 2024-07-26T17:50:08Z
----------------------------------------------------------------

Add a small markdown comment above to make sure to close your dask client, stating that it cleanly closes all dask workers and the scheduler

[rajeeja]

We might want to think about changing the name of Gallery - that houses two of the notebooks in this PR.

The formatting of equations for the below: don't look good://

//

[philipc2]

We might want to think about changing the name of Gallery - that houses two of the notebooks in this PR.

The formatting of equations for the below: don't look good:// //

The Gallery can house both Workflow & Visualization examples. Do you think this sepperation is appropriate?

The equations look like this for me in the rendered docs in light mode.

Looks like they are broken with the Dark Mode.

Have you ever encountered this @anissa111 ?

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

rajeeja commented on 2024-08-12T19:13:53Z
----------------------------------------------------------------

A little mention of what all can go as chunks?, here you do chunks={"lev": 4}

Explain what is ".Q" and why this results in Dask graph "18 chunks in 2 graph layers", the more details on how to choose chunk size and info around what happens when you choose the better.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

rajeeja commented on 2024-08-12T19:13:53Z
----------------------------------------------------------------

the only change to ux.open..call is parallel=True

A lot happens under the hood that needs explaining.

In this case you create a client object that using LocalCluster from Dask, . It would be good to reference how it interacts with our open_dataset call. Something along the lines here:

https://docs.xarray.dev/en/stable/user-guide/dask.html (specifically see the mention of dask.delayed)

[anissa111]

I haven't actually.

I poked around for a bit and suggest starting at this stackoverflow post and here in mathjax documentation.

Let me know what you find!

[rljacob]

I think of "gallery" as just being images/viz.

[philipc2]

I think of "gallery" as just being images/viz.

I've tried to closely follow what Xarray does with their gallery. , which is a combination of visualization and workflow examples.

[rajeeja]

see comments in e3sm-load notebook.

[philipc2]

Apologies for the long gap in reviews! I've left a few suggestions here. Once we've resolved all the ReviewNB comments, we should be good to go!

[philipc2]

An idea for a more descriptive title, any thoughts?

Suggested change

	- title: Calculations on E3SMv2 Output with UXarray
	- title: Radiative Feedback Analysis on E3SMv2 Model Output

[rytam2]

Now updated with link.

---

View entire conversation on ReviewNB

[rytam2]

Updated

---

View entire conversation on ReviewNB

[rytam2]

Related to bug in .plot.rasterize that requires a name to plot; will keep this line as a temporary solution.

---

View entire conversation on ReviewNB