A senior Conservative former minister has defected from the Leave campaign days before the referendum, accusing them of spreading “hate and xenophobia”.