CHANGELOG.md

CHANGELOG

unreleased

2021-06-19: v0.3 release

• Recreate biweekly renku datasets with float32 single precision and newly uploaded data to climetlab_s2s_ai_challenge. Before lead_time in tp had a one day shift. These new renku datasets will be used by the s2saichallengescorer. (#3, #5, !14, https://renkulab.io/gitlab/aaron.spring/s2s-ai-challenge/-/issues/22, Aaron Spring). Averaged recalibrated ECMWF RPSS skill value to beat at least: RPSS = -0.043, see RPSS_verification.ipynb. You should also beat climatology, i.e. RPSS = 0.
• Order of processing gridded RPSS to final score: (#7, !9, s2s-ai-competition-scoring-image!2, Aaron Spring)
  1. RPSS
  2. penalize where NaN submitted but value expected #7
  3. clip(-10,1): prevent too negative values
  4. mean over forecast_time
  5. spatially weighted mean [90N-60S]
  6. mean over lead_time and data_vars
• Dont forget to git add current_notebook.ipynb also to ensure that consistent training pipeline and submission file are tagged, added to notebooks. (!9, Aaron Spring)
• Rerun ML_train_and_predict.ipynb (!9, Aaron Spring)
• Fix typo in safeguards in ML_forecast_template.ipynb: "We did NOT use test explicitly in training or implicitly in incrementally adjusting parameters." (!8, Aaron Spring)
• Add notebooks showcasing accessing output of different models from different sources: (!2, Aaron Spring)
  • S2S-Project models:
    • from from European Weather Cloud:
      • climetlab-s2s-ai-challenge [recommended], see climetlab-s2s-ai-challenge notebooks
      • curl & wget, see wget_curl.ipynb
      • intake, see intake.ipynb
    • IRIDL including overview, see IRIDL.ipynb
    • intake catalogs for IRIDL SubX and S2S see IRIDL.ipynb (#9 !10, Aaron Spring)
  • SubX-Project models: IRIDL including overview, see IRIDL.ipynb
  • How to access password-protected S2S-Project output from IRIDL with xarray? see IRIDL.ipynb
  • Access NOAA CPC observations via opendap from IRIDL, see IRIDL.ipynb (#9 !10, Aaron Spring)
• fix netcdf4 version to 1.5.4 for opendap to work lazily with xarray (!2, !7, Aaron Spring)

2021-05-31: v0.2 release

After this v0.2 release, this CHANGELOG.md will describe all changes made in this template repository.

• update README how to join competition, please git pull if you forked before
• find status of your submission in s2s-ai-competition-scoring-image https://renkulab.io/gitlab/tasko.olevski/s2s-ai-competition-scoring-image/-/blob/master/README.md
• calculate RPSS with respect to climatology (not ECMWF anymore) (Aaron Spring)
  • update RPSS_verification.ipynb
  • update scorer: https://renkulab.io/gitlab/tasko.olevski/s2s-ai-competition-scoring-image (Tasko Olevski, Aaron Spring)
  • Averaged ECMWF RPSS skill value to beat at least: -0.0070

2021-05-26: v0.1 pre-release

• update README how to join competition (!4, Aaron Spring)
• git lfs track zarr: git lfs track "**/*.zarr/**" (!4, Aaron Spring)
• add notebooks: (!4, Aaron Spring)
  • create renku datasets: renku_datasets_biweekly.ipynb
  • RPSS verification: RPSS_verification.ipynb
  • ML train and predict based on weatherbench: ML_train_and_predict.ipynb
  • mean bias reduction: mean_bias_reduction.ipynb
  • template for training and predictions: ML_forecast_template.ipynb
• add renku dataset s2s-ai-challenge with files: (!4, Aaron Spring)
  • hindcast-like-observations_2000-2019_biweekly_deterministic.zarr
  • forecast-like-observations_2020_biweekly_deterministic.zarr
  • hindcast-like-observations_2000-2019_biweekly_tercile-edges.nc
  • hindcast-like-observations_2000-2019_biweekly_terciled.zarr
  • forecast-like-observations_2020_biweekly_terciled.nc
  • ecmwf_forecast-input_2020_biweekly_deterministic.zarr
  • ecmwf_hindcast-input_2000-2019_biweekly_deterministic.zarr
  • ecmwf_recalibrated_benchmark_2020_biweekly_terciled.nc
• add reproducibility section below in training (!4, Aaron Spring)
• how to deal with this dry mask? provide as renku dataset? now implicitly masked in categorized observations obs_p
• justify if training takes more than a week (!4, Aaron Spring)
• show RPS for all years. ToDo: take RPSS (!4, Aaron Spring)

2021-05-10

• git lfs track “submissions/*.nc”, so submission netcdf files are not carried in git but git lfs
• add notebooks/ML_prediction.ipynb as a template for submission notebooks with safeguards.