"The only reason I can explain it to you is I am not a fan of our president. I didn't send it as racist, although that's what it is. I sent it out because it's anti-Obama."
His apology seems sincere enough, except that he's apologizing for how the email was perceived, rather than it's substance. Still, it's not really the point. This man's sole job is, essentially, to use good judgement. In this case (and perhaps many others), he failed to do so.
He admitted that he read the email, and knew it was racist, but decided to forward it on to some "old buddies" anyway. A part of his job description is to maintain an appearance of impartiality. To me, his failure to do so makes every one of his rulings suspect.
This isn't about free speech, as the Republican senators in his state would have you believe. He is free to have opinions, and discuss them privately. But this man holds the life and liberty of American citizens in his hands daily. To use such poor judgement precludes having a job with that kind of responsibility, and for that reason alone, he should step down.
If he can't see this, it's just another indication of his poor judgement.