Winning Software

AutoML2

Only 2 phases:

1. Feed-back phase: You get to practice (with code or result submission) on 5 datasets and get immediate feed-back on a leaderboard.
2. AutoML phase: The code gets blind tested on 5 new datasets.

The setting is similar to AutoML1, so you can get a head start by studying what was done.

AutoML1

The First ChaLearn AutoML challenge included 5 rounds, 30 datasets (available for download), and $30,000 in prizes. The goal was to design the perfect machine learning “black box” capable of performing all model selection and hyper-parameter tuning without any human intervention. Post-challenge submissions can be made on the clone websites:

• Round 0: Sample data (practice).
• Round 1: Novice level. binary classification problems, dense data (practice).
• Round 2: Intermediate level. Multi-class classification, dense data (practice).
• Round 3: Advanced level. Multi-class and multi-label classification, dense and sparse data (practice).
• Round 4: Expert. Regression + all the other cases seen so far.
• Round 5: Master. Final blind testing. We are not releasing the data, which will be used in further benchmarks.

In phase Final4, we also had a GPU track.

RESULTS

We have released a short analysis of the challenge results for the ICML 2016 workshop on AutoML.

The results of top ranking participants of the first round were discussed at the AutoML workshop at ICML 2015. See summary paper. A more detailed paper was presented at IJCNN 2015. The NIPS 2016 [slides] give an update. We also have published the results of a short survey on methods used (fact sheets). Thanks for citing our papers, see citations for the BibTex entries.

The top ranking participants described briefly their methods in the FACT SHEETS and wrote blogs:

• AutoML and Final, all rounds: AAD Freiburg team. [NIPS2015 paper]. [NIPS2015 slides][supplementary material].
• Final, all rounds: Eugene Tuv: How we ranked at the top in all Tweakathons! [key paper].
• Final and AutoML, rounds 2-end: Damir Jajetic [summary].
• AutoML 1&2: James Lloyd [summary].
• Final4 CPU and GPU: Abhishek Thakur: Keras Neural Networks to win NVIDIA Titan X. [ICML2015 workshop paper][ICML2016 workshop paper]
• Third in AutoML4: Marc Boulle: AutoML: an appealing challenge to stimulate research in data mining automation.[method summary]
  • Third in AutoML5 (post-challenge): Lisheng Sun: Right after the challenge was over, Lisheng Sun made a minor correction to he code (without ever seeing the datasets) and place 3rd. The entry called reference_ls does not count towards the prizes, still her entry is worthy of attention. [fact_sheet]

WINNERS

The competition started December 8, 2014 and ended May 1, 2016.