Authors:
(1) Simone Silvestri, Massachusetts Institute of Technology, Cambridge, MA, USA;
(2) Gregory Wagner, Massachusetts Institute of Technology, Cambridge, MA, USA;
(3) Christopher Hill, Massachusetts Institute of Technology, Cambridge, MA, USA;
(4) Matin Raayai Ardakani, Northeastern University, Boston, MA, USA;
(5) Johannes Blaschke, Lawrence Berkeley National Laboratory, Berkeley, CA, USA;
(6) Valentin Churavy, Massachusetts Institute of Technology, Cambridge, MA, USA;
(7) Jean-Michel Campin, Massachusetts Institute of Technology, Cambridge, MA, USA;
(8) Navid Constantinou, Australian National University, Canberra, ACT, Australia;
(9) Alan Edelman, Massachusetts Institute of Technology, Cambridge, MA, USA;
(10) John Marshall, Massachusetts Institute of Technology, Cambridge, MA, USA;
(11) Ali Ramadhan, Massachusetts Institute of Technology, Cambridge, MA, USA;
(12) Andre Souza, Massachusetts Institute of Technology, Cambridge, MA, USA;
(13) Raffaele Ferrari, Massachusetts Institute of Technology, Cambridge, MA, USA.
Table of Links
5.1 Starting from scratch with Julia
5.2 New numerical methods for finite volume fluid dynamics on the sphere
5.3 Optimization of ocean free surface dynamics for unprecedented GPU scalability
6 How performance was measured
7 Performance Results and 7.1 Scaling Results
9 Acknowledgments and References
9 Acknowledgments
This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231 using NERSC award DDRERCAP0025591. This work is partly supported by the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program and by NSF grant AGS-1835576. N.C.C. is supported by the Australian Research Council DECRA Fellowship DE210100749.
References
[1] HIP: C++ Heterogeneous-Compute Interface for Portability. https://github.com/ ROCm-Developer-Tools/HIP.
[2] MIT Satori User Documentation. https://mit-satori.github.io/index.html.
[3] A. Arakawa and V.R. Lamb. Computational design of the basic dynamical processes of the UCLA general circulation model. General Circulation Models of the Atmosphere, 17, 1977.
[4] V. Balaji, E. Maisonnave, N. Zadeh, B.N. Lawrence, J. Biercamp, U. Fladrich, G. Aloisio, R. Benson, A. Caubel, J. Durachta, et al. CPMIP: measurements of real computational performance of Earth system models in CMIP6. Geoscientific Model Development, 10(1):19–34, 2017.
[5] T. Besard, V. Churavy, A. Edelman, and B. De Sutter. Rapid software prototyping for heterogeneous and distributed platforms. Advances in engineering software, 132:29–46, June 2019.
[6] T. Besard, C. Foket, and B. De Sutter. Effective Extensible Programming: Unleashing Julia on GPUs. IEEE Trans. Parallel Distrib. Syst., (4):827–841, December 2017.
[7] J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah. Julia: A fresh approach to numerical computing. SIAM Review, 59(1):65–98, January 2017.
[8] E.P. Chassignet, S.G. Yeager, B. Fox-Kemper, A. Bozec, F. Castruccio, G. Danabasoglu, C. Horvat, W.M. Kim, N. Koldunov, Y. Li, et al. Impact of horizontal resolution on global ocean–sea ice model simulations based on the experimental protocols of the Ocean Model Intercomparison Project phase 2 (OMIP-2). Geoscientific Model Development, 13(9):4595–4637, 2020.
[9] T.R. Christian, D. Lebrun-Grandie, D. Arndt, J. Ciesko, V. Dang, N. Ellingwood, R. Gayatri, E. Harvey, D.S. Hollman, D. Ibanez, et al. Kokkos 3: Programming model extensions for the exascale era. IEEE Transactions on Parallel and Distributed Systems, 33(4):805–817, 2021.
[10] V. Churavy, D. Aluthge, J. Samaroo, A. Smirnov, J. Schloss, L. C Wilcox, S. Byrne, M. Waruszewski, A. Ramadhan, S. Schaub, N. C Constantinou, J. Bolewski, M. Ng, T. Besard, B. Arthur, C. Kawczynski, C. Hill, C. Rackauckas, J. Cook, J. Liu, M. Schanen, O. Schulz, P. Haraldsson, T. Arakaki, and T. Chor. JuliaGPU/KernelAbstractions.jl: v0.9.1, March 2023.
[11] M. Ding, H. Liu, P. Lin, Y. Meng, W. Zheng, B. An, Y. Luan, Y. Yu, Z. Yu, Y. Li, J. Ma, J. Chen, and K. Chen. A century-long eddy-resolving simulation of global oceanic large- and mesoscale state. Sci. Data, 9:661, 2022.
[12] C. Donlon, B. Berruti, S. Mecklenberg, J. Nieke, H. Rebhan, U. Klein, A. Buongiorno, C. Mavrocordatos, J. Frerick, and B. Seitz. The sentinel-3 mission: Overview and status. In 2012 IEEE International Geoscience and Remote Sensing Symposium, pages 1711–1714. IEEE, 2012.
[13] D. Ferreira, J. Marshall, and J.-M. Campin. Localization of deep water formation: Role of atmospheric moisture transport and geometrical constraints on ocean circulation. J. Climate, 23:1456–1476, 2010.
[14] T. Fiedler, A.J. Pitman, K. Mackenzie, N. Wood, C. Jakob, and S.E. Perkins-Kirkpatrick. Business risk and the emergence of climate analytics. Nature Climate Change, 11:87–94, 2021.
[15] O. Fuhrer, T. Chadha, T. Hoefler, G. Kwasniewski, X. Lapillonne, D. Leutwyler, D. Lüth, C. Osuna, C. Schär, T.C. Schulthess, and H. Vogt. Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0. Geosci. Model Dev., 14:2781–2799, 2021.
[16] M. Giordano, M. Klöwer, and V. Churavy. Productivity meets performance: Julia on A64FX. In 2022 IEEE International Conference on Cluster Computing (CLUSTER), pages 549–555, September 2022.
[17] W. F. Godoy, P. Valero-Lara, E. T. Dettling, C. Trefftz, I. Jorquera, T. Sheehy, R. G. Miller, M. Gonzalez-Tallada, J. S Vetter, and V. Churavy. Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and kokkos on exascale nodes. March 2023.
[18] J M. Gutiérrez and A.-M. Tréguier. IPCC, 2021: Annex II: Models. Climate Change 2021: The Physical Science Basis, page 2087–2138, 2021.
[19] R. Haarsma, M. Acosta R., Bakhshi, P.-A. Bretonnière, L.-P. Caron, M. Castrillo, S. Corti, P. Davini, E. Exarchou, F. Fabiano, U. Fladrich, R. Fuentes Franco, J. García-Serrano, J. von Hardenberg, T. Koenig, X. Levine, V.L. Meccia, T. van Noije, G. van den Oord, F.M. Palmeiro, M. Rodrigo, Y. Ruprich-Robert, P. Le Sager, E. Tourigny, S. Wang, M. van Weele, and K. Wyser. HighResMIP versions of EC-Earth: EC-Earth3P and EC-Earth3P-HR – description, model computational performance and basic validation. Geoscientific Model Development, 13(8):3507–3527, 2020.
[20] C. Hope. The $10 trillion value of better information about the transient climate response. Phil. Trans. R. Soc. A, 3:20140429, 2015.
[21] S. Hunold and S. Steiner. Benchmarking Julia’s communication performance: Is Julia HPC ready or full HPC? In 2020 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pages 20–25. IEEE, November 2020.
[22] D. Häfner, R. Nuterman, and M. Jochum. Fast, Cheap, and Turbulent—Global Ocean Modeling With GPU Acceleration in Python. JAMES, 13(12):e2021MS002717, 2021.
[23] E.J. Kendo, N.M. Roberts, H.J. Fowler, M.J. Roberts, S.C. Chan, and C.A. Senior. Heavier summer downpours with climate change revealed by weather forecast resolution model. Nature Climate Change, 4(7):570–576, 2014.
[24] C Lattner and V Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generation and Optimization, 2004. CGO 2004. IEEE, 2004.
[25] H. Lee and J. Romero (eds.). IPCC, 2023: Climate Change 2023: Synthesis Report. A Report of the Intergovernmental Panel on Climate Change. Contribution of Working Groups I, II and III to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change, 2023.
[26] Q. Li, M.H. England, A. McC Hogg, S.R. Rintoul, and A.K. Morrison. Abyssal ocean overturning slowdown and warming driven by Antarctic meltwater. Nature, 615(7954):841–847, 2023.
[27] W. C. Lin and S. McIntosh-Smith. Comparing julia to performance portable parallel programming models for HPC. In 2021 International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), pages 94–105. ieeexplore.ieee.org, November 2021.
[28] J. Ma, S. Xu, and B. Wang. Warm bias of sea surface temperature in eastern boundary current regions—a study of effects of horizontal resolution in CESM. Ocean Dynamics, 69:939–954, 2019.
[29] J. Marshall, A. Adcroft, C. Hill, L. Perelman, and C. Heisey. A finite-volume, incompressible Navier Stokes model for studies of the ocean on parallel computers. Journal of Geophysical Research: Oceans, 102(C3):5753–5766, 1997.
[30] D.S. Medina, A. St-Cyr, and T. Warburton. OCCA: A unified approach to multi-threading languages. arXiv preprint arXiv:1403.0968, 2014.
[31] Dimitris Menemenlis, Jean-Michel Campin, Patrick Heimbach, Chris Hill, Tong Lee, An Nguyen, Michael Schodlok, and Hong Zhang. Ecco2: High resolution global ocean and sea ice data synthesis. Mercator Ocean Quarterly Newsletter, 31(October):13–21, 2008.
[32] R. Morrow, L-L. Fu, F. Ardhuin, M. Benkiran, B. Chapron, E. Cosme, F. d’Ovidio, J.T. Farrar, S.T. Gille, G. Lapeyre, P.-Y. Le Traon, A. Pascual, A. Ponte, B. Qiu, Rascle N, C. Ubelmann, J. Wang, and E.D. Zaron. Global Observations of Fine-Scale Ocean Surface Topography with the Surface Water and Ocean topography (SWOT) Mission. 2019.
[33] D.R. Munday, H.L. Johnson, and D.P. Marshall. Impacts and effects of mesoscale ocean eddies on ocean carbon storage and atmospheric pCO2. Global Biogeochemical Cycles, 28(8):877–896, 2014.
[34] T. Palmer. Build high-resolution global climate models. Nature, 515:338–339, 2014.
[35] A. Ramadhan, G. Wagner, C. Hill, J.-M. Campin, V. Churavy, T. Besard, A. Souza, A. Edelman, R. Ferrari, and J. Marshall. Oceananigans.jl: Fast and friendly geophysical fluid dynamics on GPUs. Journal of Open Source Software, 5(53), 2020.
[36] T. Ringler, M. Petersen, R.L. Higdon, D. Jacobsen, P.W. Jones, and M. Maltrud. A multiresolution approach to global ocean modeling. Ocean Modelling, 69:211–232, 2013.
[37] M.J. Roberts, A. Baker, E.W. Blockley, D. Calvert, A. Coward, H.T. Hewitt, L.C. Jackson, T. Kuhlbrodt, P. Mathiot, C.D. Roberts, R. Schiemann, J. Seddon, B. Vannière, and P.L. Vidale. Description of the resolution hierarchy of the global coupled hadgem3-gc3.1 model as used in cmip6 highresmip experiments. Geosci. Model Dev., 12(12):4999–5028, 2019.
[38] V. Churavy S. Byrne, L.C. Wilcox. MPI.jl: Julia bindings for the message passing interface. Proceedings of the JuliaCon Conferences, 2021.
[39] O.A. Saenko, D. Yang, and J.M. Gregory. Impact of mesoscale eddy transfer on heat uptake in an eddy-parameterizing ocean model. Journal of Climate, 31(20):8589 – 8606, 2018.
[40] T. Schneider, S. Behera, G. Boccaletti, C. Deser, K. Emanuel, R. Ferrari, L.R. Leung, N. Lin, T. Müller, A. Navarra, O. Ndiaye, A. Stuart, J. Tribbia, and T. Yamagata. Harnessing AI, Data, and Computing to Advance Climate Modeling and Prediction. Nature Climate Change, submitted.
[41] T. Schneider, S. Lan, A. Stuart, and J. Teixeira. Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High-Resolution Simulations. Geophysical Research Letters, 2017.
[42] C. Shu. Essentially non-oscillatory and weighted essentially non-oscillatory schemes for hyperbolic conservation laws. Icase report no. 97-65, Institute for Computer Applications in Science and Engineering, NASA Langley Research Center, 1997.
[43] S. Silvestri, G.L. Wagner, A. Souza, N. Constantinou, C. Hill, J.-M. Campin, and R. Ferrari. A WENO-based Vector Invariant advection scheme for Implicit Ocean LES. In preparation.
[44] D. Struck. Meet the team shaking up climate models. The Christian Science Monitor, 2021.
[45] Z. Su, J. Wang, P. Klein, A. Thompson, and D. Menemenlis. Ocean submesoscales as a key component of the global heat budget. Nat. Commun., 9:775, 2018.
[46] J.R. Taylor and A.F. Thompson. Submesoscale Dynamics in the Upper Ocean. Annual Review of Fluid Mechanics, 55(1):103–127, 2023.
[47] MPI Team. MPI: A Message-Passing interface standard. Technical report, USA, 1994.
[48] P. Wang, J. Jiang, P. Lin, M. Ding, J. Wei, F. Zhang, L. Zhao, Y. Li, Z. Yu, W. Zheng, Y. Yu, X. Chi, and H. Liu. The GPU version of LASG/IAP climate system ocean model version 3 (LICOM3) under the heterogeneous-compute interface for portability (HIP) framework and its large-scale application. Geosci. Model Dev., 14(5):2781–2799, 2021.
[49] N.P. Wedi, I. Polichtchouk, P. Dueben, V.G. Anantharaj, P. Bauer, S. Boussetta, P. Browne, W. Deconinck, W. Gaudin, I. Hadade, et al. A baseline for global weather and climate simulations at 1 km resolution. Journal of Advances in Modeling Earth Systems, 12(11):e2020MS002192, 2020.
[50] B. Xiao, F. Qiao, Q. Shu, X. Yin, G. Wang, and S. Wang. The development and validation of a global 1/32° surface-wave-tide-circulation coupled ocean model: FIO-COM32. Geosci. Model Dev., 2022.
[51] S. Zhang, H. Fu, L. Wu, Y. Li, H. Wang, Y. Zeng, X. Duan, W. Wan, L. Wang, Y. Zhuang, et al. Optimizing high-resolution community earth system model on a heterogeneous many-core supercomputing platform. Geoscientific Model Development, 13(10):4809–4829, 2020.
This paper is available on arxiv under CC BY 4.0 DEED license.
