The Ivory Tower

Why do they call academia an ivory tower? My theory is that either the world sits on the head of an elephantine organism made of dark matter, or that academia enhances one's memory to elephantlike levels, or that academia produces a well-calibrated mind with likeness to a perfectly tuned piano. It could be an interesting study in the etymology of a phrase to investigate what exactly its source is, as I'm sure you philosophical professionals have pondered this on occasion. Not a gravely serious topic, I realize, but could be an amusing case study in philology.