Sunday, 29 May 2011

Hunting the Hidden Dimensions

Last week I came across a documentary on TV called 'Hunting the Hidden Dimensions'. Now generally there is nothing worth watching on Australian TV (unless you are into cooking or talent contests) but this one actually got me interested (although I must admit to falling asleep half way through due to the late for me time slot).

The program was about fractals and the hidden order that appears in everything in nature. I didn't realise that fractal geometry was involved in film animation to generate mountain ranges and volcanic lava flow!

One thing that did strike an accord was how Mandelbrot, when he first published his work, got a bit of a thumbs down from the academic mainstream because the ideas didn't conform to the nice orderly nature of the world people were used to (ie the type of mathematics the Egyptians used to build the pyramids). The thing is, fractals are perfectly orderly, but looked at in a way that was different to the norm.

This is in a data mining context is similar to neural networks, with many people who should know better saying they are uninterpretable black boxes. When neural networks give improved predictions, they have found 'hidden dimensions' that are very easy to find if you are willing to look. Like Mandelbrot, you will need to look at data visually rather than mathematically in order to figure things out.

You can watch the documentary online. 

1 comment:

  1. Statistical machine learning and fractal geometry are certainly fields of exploration that we couldn't examine easily before the invention of the datascope several decades ago. I like your analogy and agree visual representation of results is very important.

    btw, the sbs link didn't work for me but this one did: