renku datasets and initial notebooks

• git lfs track nc and zarr

• add renku datasets notebook

• add renku datasets:

  • hindcast-like-observations_2000-2019_biweekly_deterministic
  • forecast-like-observations_2020_biweekly_deterministic
  • hindcast-like-observations_2000-2019_biweekly_tercile-edges
  • hindcast-like-observations_2000-2019_biweekly_terciled
  • forecast-like-observations_2020_biweekly_terciled
  • ecmwf_forecast-input_2020_biweekly_deterministic
  • ecmwf_hindcast-input_2000-2019_biweekly_deterministic

• add verification notebook

• add ML notebook based on weatherbench

• link submission file to website

• add reproducibility section below in training

• how to deal with this dry mask? provide as renku dataset? now implicitly masked in categorized observations obs_p

• benchmark working on renku: wait for climetlab release: week 5-6, tp and 2000-2019 dont wait

• dont commit nc and zarr, only with git lfs

• dont commit figures

• create nc for files <100MB, zarr if >100MB?

• justify if training takes more than a week

• show RPS for all years

• test & dry run

• create one renku dataset with all files: s2s-ai-challenge

• add mean bias reduction dataset

added 4 commits

• c590920c - renku dataset: committing 1 newly added files
• c5eafb2c - renku dataset add --create...
• b751cb21 - renku dataset: committing 32 newly added files
• 8f28a5a5 - renku dataset add --create...

Compare with previous version

added 11 commits

• 7fdb5ada - renku dataset: committing 200 newly added files
• d45078a6 - renku dataset add --create...
• 9e20beca - renku dataset: committing 203 newly added files
• ad02f7fa - renku dataset add --create...
• 9f53563f - renku dataset: committing 45 newly added files
• dd7d8526 - renku dataset add --create ecmwf_forecast-input_2020_biweekly_deterministic...
• c03859a5 - renku dataset: committing 1 newly added files
• 98d0f297 - renku dataset add --create forecast-like-observations_2020_biweekly_terciled...
• e0fc1970 - renku dataset: committing 52 newly added files
• b02274a8 - renku dataset add --create...
• 09c38b02 - add notebooks

Compare with previous version

marked the checklist item dont commit nc and zarr, only with git lfs as completed

added 8 commits

• 4d49bbf6 - renku dataset rm: final mutation
• 6468f1ec - renku dataset rm hindcast-like-observations_2000_2019_biweekly_terciled
• 66bda0a2 - renku dataset: committing 52 newly added files
• 2bdc1121 - renku dataset add --create...
• b6f974ff - clean notebooks, add RPS training, fix obs_p renku ds
• a3e0a870 - template_test commit message
• 6fa33d3f - fixes
• d5e6ad0d - minor

Compare with previous version

marked the checklist item link submission file to website as completed

changed the description

added 2 commits

• 2861890c - uncomment renku, assert func, cleanup
• 0b1bdd5f - fix climetlab install from source

Compare with previous version

marked the checklist item create nc for files <100MB, zarr if >100MB? as completed

changed the description

added 30 commits

• 95f8a65b - renku dataset create s2s-ai-challenge
• b4d56712 - renku dataset rm: final mutation
• 235c5f11 - renku dataset rm ecmwf_forecast-input_2020_biweekly_deterministic
• 3be1608a - renku dataset rm: final mutation
• 86e70ddd - renku dataset rm forecast-like-observations_2020_biweekly_terciled
• e075edb4 - renku dataset rm: final mutation
• 19a12f69 - renku dataset rm hindcast-like-observations_2000-2019_biweekly_deterministic
• 27d0c390 - renku dataset rm: final mutation
• 6eb08710 - renku dataset rm forecast-like-observations_2020_biweekly_deterministic
• 3dfb2f4f - renku dataset rm: final mutation
• c58d61df - renku dataset rm hindcast-like-observations_2000-2019_biweekly_terciled
• 358e258f - renku dataset rm: final mutation
• fc0180e0 - renku dataset rm ecmwf_hindcast-input_2000-2019_biweekly_deterministic
• c7f80bb0 - renku dataset rm: final mutation
• 133434a4 - renku dataset rm hindcast-like-observations_2000-2019_biweekly_tercile-edges
• cd8a0c3b - renku dataset add s2s-ai-challenge...
• 9e1e0c80 - renku dataset add s2s-ai-challenge data/forecast-like-observations_2020_biweekly_deterministic.zarr
• faf9a469 - renku dataset add s2s-ai-challenge data/ecmwf_hindcast-input_2000-2019_biweekly_deterministic.zarr
• cf3515ac - renku dataset add s2s-ai-challenge data/ecmwf_forecast-input_2020_biweekly_deterministic.zarr
• 63d91aec - renku dataset add s2s-ai-challenge...
• 6db84722 - renku dataset add s2s-ai-challenge data/forecast-like-observations_2020_biweekly_terciled.nc
• d885e6b9 - renku dataset add s2s-ai-challenge data/hindcast-like-observations_2000-2019_biweekly_terciled.zarr
• 581b4a36 - renku dataset tag s2s-ai-challenge 0.1_initial_t2m_tp_ecmwf_obs
• e6ef313e - renku dataset tag -d initial_t2m_tp_ecmwf_obs s2s-ai-challenge v0.1
• 2a804760 - renku dataset tag s2s-ai-challenge 0.1 -d v0.1: initial_t2m_tp_ecmwf_obs
• 3adeaea1 - renku dataset rm-tags s2s-ai-challenge v0.1
• 56dd499d - renku dataset rm-tags s2s-ai-challenge 0.1_initial_t2m_tp_ecmwf_obs
• f1421e70 - notebooks again
• 62f4de3e - template_test no ML mean bias reduction
• 9e853672 - mean_bias_reduction notebook

Compare with previous version

added 3 commits

• 228e5dd8 - renku dataset: committing 1 newly added files
• 091fd4c5 - renku dataset add s2s-ai-challenge data/ecmwf_recalibrated_benchmark_2020_biweekly_terciled.nc
• 988b6084 - add benchmark renku and adapt notebooks

Compare with previous version

added 1 commit

• 29071da3 - Update environment.yml

Compare with previous version

added 2 commits

• dec740d0 - Update instructions for scorer
• fb042e58 - Merge branch 'update-instructions-for-checker' into 'AS_initial'

Compare with previous version

marked this merge request as ready

added 1 commit

• eba64509 - Update CHANGELOG.md

Compare with previous version

added 1 commit

• f0324ac9 - Update CHANGELOG.md

Compare with previous version

added 1 commit

• a2c0bb3c - Update README.md

Compare with previous version

added 1 commit

• e04fb4e2 - Update environment.yml

Compare with previous version

added 1 commit

• 7d57ea31 - cleanup RPSS verification

Compare with previous version

approved this merge request