Review Stephan Rasp
Review of 2nd placed entry to WMO S2S AI competition
First, I would like to thank the authors for their detailed documentation and especially for the clear notebook. The authors train 4 methods, climatology, raw (de-biased) ECMWF, logistic regression and random forest. For each each grid-point and lead time the best method - based on a leave-one-year-out cross validation from 2000 to 2019 - is chosen for the final 2020 prediction.
The method and code are easy to understand, reproducible and the train/valid split is well executed, so that no overfitting it to be suspected. Therefore, I give this submission a definite pass!
I do have two follow on questions though that, while not critical, would be very interesting:
- You mention that for precipitation you only used climatology and raw ECMWF due to time constraints. In one evaluation figure I think I saw that your method is very good on temperature but basically has close to zero skill on precipitation. Did you since have time to rerun all the 4 methods for precipitation? I would love to know what the "real skill" of your method is.
- Could you produce a map showing which model was picked for which location/lead time? Or also statistics showing the percentages for how often each method was chosen? Related to this, did you think about merging the four methods in some way rather than choosing the best?
Best, Stephan Rasp