Forecast and Verification times
email by @awr Tu 27 to be discussed Th 29th april:
- Which forecast starts/target periods (weeks 3-4 & 5-6) to require to be submitted?
- It may make sense to take all Thursday starts in 2020 since there are available from all S2S models, including our ECMWF benchmark
- In that case, the first forecast will start S=2 Jan 2020, for the week 3-4 target 16-29 Jan.
- The list of 52 forecasts and targets could be tabulated for clarity
- Which data to “allow” to be used to make a specific ML forecast?
- for the first forecast S=2 Jan 2020, any observational data up to the day before (or of?) the forecast start, ie 1 Jan 2020
- any S2S forecasts up to and including S=2 Jan 2020
- daily data
BUT how to make it transparent that more recent obs and forecast data are not being used to make the forecast, and thus cheating? Given what we said last time about there being only about a month after the competition ends to the end of Aaron's contract, there won’t be much time to verify that the protocol was followed. How can we design the pipeline and curated datasets to make it transparent to cheating? In addition, the leaderboard could be rendered effectively meaningless if it’s trivial to cheat.