Cross-scale Ocean Modeling Benchmark

The emergence of ocean observing systems poses novel opportunities for observation-based benchmarks for numerical ocean models.

A self-formed working group proposes one such benchmark, and invites broad community participation in its development. A discussion forum was set-up for community feedback.

Our envisioned approach involves the creation of reference benchmarks that can remain “fresh” for a decade or more, hence informing current and next generations of models. Four such benchmarks – possibly at different stages of development – will be presented at the conference. The first will be focused on an estuary (Delaware Bay), the second on a coupled estuary-plume-shelf system (Columbia River), the third on a geometrically complex shelf region (to be defined), and the fourth on an ocean basin, sub-basin or internal sea (to be defined).

The two benchmarks already in preparation will illustrate the key elements of our planned approach. For Delaware Bay, we will leverage an inter-model comparison currently being conducted by NOAA's Coast Survey Development Laboratory. For Columbia River, we will leverage the CORIE coastal-margin observatory maintained by the Science and Technology Center for Coastal Margin Observation and Prediction, at Oregon Health & Science University

While each benchmark will be prepared and maintained by a different team and might be supported by a different set of reporting web technologies, all benchmarks will share the following characteristics:

• Benchmarks will include at least two retrospective simulations (one week and one year, respectively), and at least one near real-time forecast. Observations will be available ahead of time for the retrospective simulations, and will be “blind” for the near real-time forecasts.
• Observations – extensive in space and time and diverse in nature – will provide the basis for skill assessment, with model skill measured through both quantitative metrics of model-data differences and ability to represent distinctive circulation processes and features.
• Besides skill, performance metrics will measure numerical integrity (such as conservation properties and positive definiteness) and computational efficiency.
• Two mandatory “reference discretizations” in time and space will be provided for each benchmark, one for unstructured- and the other for structured-grid models. However, for each solution using the reference discretization, one or more solutions can also be submitted that represent some form of discretization optimization for that particular model.
• Data assimilation solutions will be allowed only when a corresponding non-assimilated solution is also provided.
• Inter-model skill comparisons will be scaled against computationally efficiency, rather than surrogate characteristics such as number of nodes.
• As a part of the benchmark solutions, we plan to ask participants to supply a combination of (source or compiled) model code, model input files, and model output results – which together would enable the organizers to create consistent, replicable calculations of performance metrics for cross-model comparison.
• Benchmark results will be kept electronically – as an openly available community resource – and inter-model skill assessments will be conducted periodically, typically through specialty sessions associated with major conferences or workshops.

We believe that the proposed benchmarks will fill a current void within the tools available to the ocean modeling community, and that they will prove valuable to drive future model development.

Frank Aikman, National Oceanic and Atmospheric Administration
Antonio M. Baptista, Oregon Health & Science University
Cheryl Ann Blain, Naval Research Laboratory
Mohamed Iskandarani, University of Miami