Chris R has now opened up a gap at the top of the leaderboard. It may look like nothing, but when you plot it, it is quite significant.
![]()
If you plot the number of entries per team, we see that progress seems to come with more effort. Chris R and a few other teams are almost doing daily submissions.
![]()
I am now progressing steadily as I investigate more of the data and more algorithms. I have gone from 40th to 19th by just refining the algorithm, rather than adding any different data.
The technique I am using to determine if things are improving are quite simple. Rather than have an independent set to test the errors against, I randomly split the set in half, build a model on one set and then calculate the error on the rest. The error on the building set is the 'in bag error' and the error on the unseen set is 'out of bag' error. This random splitting is repeated many times with the predictions just averaged. What I am looking for is the OOB error to go down with improvements in the algorithm. This seems to work, and the leaderboard scores improve as by OOB scores improve, although the leaderboard error is about 0.01 higher than my OOB error.
For a sanity check, you can plot the distributions of the predictions by in bag, out of bag and leaderboard set. The plot below shows a remarkable similarity, meaning the data distributions in the sets are very similar.
![]()
If you plot the number of entries per team, we see that progress seems to come with more effort. Chris R and a few other teams are almost doing daily submissions.
I am now progressing steadily as I investigate more of the data and more algorithms. I have gone from 40th to 19th by just refining the algorithm, rather than adding any different data.
The technique I am using to determine if things are improving are quite simple. Rather than have an independent set to test the errors against, I randomly split the set in half, build a model on one set and then calculate the error on the rest. The error on the building set is the 'in bag error' and the error on the unseen set is 'out of bag' error. This random splitting is repeated many times with the predictions just averaged. What I am looking for is the OOB error to go down with improvements in the algorithm. This seems to work, and the leaderboard scores improve as by OOB scores improve, although the leaderboard error is about 0.01 higher than my OOB error.
For a sanity check, you can plot the distributions of the predictions by in bag, out of bag and leaderboard set. The plot below shows a remarkable similarity, meaning the data distributions in the sets are very similar.
Nice Blog - hope to see more stuff like this. FWIW - I have made it a personal goal to try and make one submission a day. Usually I underestimate how long stuff will take - and if you look at the time of my submissions - you will see a trend :)
ReplyDeleteI wold say about a third/half of mine involve a very small change as I was running out of time...