Only 2 phases:
- Feed-back phase: You get to practice (with code or result submission) on 5 datasets and get immediate feed-back on a leaderboard.
- AutoML phase: The code gets blind tested on 5 new datasets.
The setting is similar to AutoML1, so you can get a head start by studying what was done.
The First ChaLearn AutoML challenge included 5 rounds, 30 datasets (available for download), and $30,000 in prizes. The goal was to design the perfect machine learning “black box” capable of performing all model selection and hyper-parameter tuning without any human intervention. Post-challenge submissions can be made on the clone websites:
- Round 0: Sample data (practice).
- Round 1: Novice level. binary classification problems, dense data (practice).
- Round 2: Intermediate level. Multi-class classification, dense data (practice).
- Round 3: Advanced level. Multi-class and multi-label classification, dense and sparse data (practice).
- Round 4: Expert. Regression + all the other cases seen so far.
- Round 5: Master. Final blind testing. We are not releasing the data, which will be used in further benchmarks.
The results of top ranking participants of the first round were discussed at the AutoML workshop at ICML 2015. See summary paper. A more detailed paper was presented at IJCNN 2015. The NIPS 2016 [slides] give an update. We also have published the results of a short survey on methods used (fact sheets). Thanks for citing our papers, see citations for the BibTex entries.
- AutoML and Final, all rounds: AAD Freiburg team. [NIPS2015 paper]. [NIPS2015 slides][supplementary material].
- Final, all rounds: Eugene Tuv: How we ranked at the top in all Tweakathons! [key paper].
- Final and AutoML, rounds 2-end: Damir Jajetic [summary].
- AutoML 1&2: James Lloyd [summary].
- Final4 CPU and GPU: Abhishek Thakur: Keras Neural Networks to win NVIDIA Titan X. [ICML2015 workshop paper][ICML2016 workshop paper]
- Third in AutoML4: Marc Boulle: AutoML: an appealing challenge to stimulate research in data mining automation.[method summary]
The competition started December 8, 2014 and ended May 1, 2016.