If you have been keeping an eye on the leaderboard, then you will notice there has been apparently little activity since the milestone 1 deadline. On some occasions there has been no change in the top 40 positions for over one week. This is quite an eerie silence and I suspect there may be a lack of submissions so teams can merge (the total submissions of teams merging has to be less than the number of days the comp has been running).
There have also been some very interesting movements if you look closer - more to come on this in a later post.
If you look further down though, the pack is catching up. In the past six weeks more teams are heading towards the 0.461 mark, which is the point the early leaders got to straight away and was the score to beat. Now it is only a top 50 place.
This score is a good single model. To improve dramatically from there though it is probably necessary to ensemble various models. What is pretty clear though is that the benchmark of 0.40 for the 3 million is impossible (hopefully this might be adjusted?).
I tried to put some nice colours in the chart below, which is generated in R, but could not find any up to date listing of colour codes in R. This is one of the disadvantages of the open source movement - documentation is very low on the contributors list of priorities (and what documentation there is leads a lot to be desired if R is to be used by 'regular' types of people).
I did find the following link though, which is where I got the colours for the plot,
http://colorbrewer2.org/
click on the image to enlarge
Thanks for the post. I think the fact that the benchmark of 0.40 for the 3 million seems impossible shows that they have put a lot of thinking into it. Whatever team can beat this benchmark would then be definitely worth $3 million. Otherwise $500k is a pretty good prize as well.
ReplyDeleteIt's also striking ... if I'm reading this correctly ... that the "Sub .460 Club" had about 25 members as of Oct 1, and 2 1/2 months later, essentially it's the same teams.
ReplyDeleteSo it appears that, helped no doubt by the Market-Maker code (Sali Mali's team), there has been a major surge in the number of teams/people who have been able to make a competitive quality single candidate model.
But the Art of the Ensemble apparently remains illusive to most participants at this stage, and continues to be the separator between the Best and the Rest.
Has anyone seen any published material on how much "extra lift" so to speak is (practically or probably) achievable, given a portfolio of candidate models that seem to be suggesting a limit on how far single models can go? I have looked, but I have not found anything. 2% bump? 5% bump? 10% ?
I'm thinking 5% is a useful hope/guideline for the extra ooomph in the realm of predictions, but that's really just a guess. (And of course, a majority of the slim collection of published work on ensembles has involved classifiers, it's unclear if that extrapolates to predictive models.)