The phrase "to leave the side" means to separate or move away from a particular group or team that you were part of.