getting renku ready for competition: scorer
working together with @tasko.olevski
So the steps to join the competition would be:
- Register/login to renkulab
- Fork the starter project (from my end this is not mandatory, as long as the steps below are done the scorer will work on any repo in renkulab - even if the starter project was not used)
- Make the project private (so that other people do not steal your idea/code)
- Add the scorer user to your repo with Reporter permissions (this is the lowest view-only permission so that the scorer can check results)
- Add a gitlab variable with key COMPETITION and name S2S-AI (we can set whatever name we want here)
two steps provided by @aaron.spring:
- participants create the ML predictions into file rb_prediction_2020.nc to be git lfsed with https://renkulab.io/gitlab/aaron.spring/s2s-ai-competition-bootstrap/-/blob/master/notebooks/bias_reduction_forecast.ipynb
- the scorer https://renkulab.io/gitlab/aaron.spring/s2s-ai-competition-bootstrap/-/blob/master/notebooks/verification_RPSS.ipynb then calcs RPSS and appends to leaderboard (very rudimentary implemented only)
Todos:
-
implement scorer
@tasko.olevski -
s2s-ai-competition-template @aaron.spring