The British use of "side" for what Americans call "team" explains this, I believe. Taking sides is the point at which players in a casual game choose teams. To Americans this sounds more like an abstraction, but it's actually a very concrete metaphor.