Will cancelling my bad credit card affect my credit score?

I have other cards that I have excellent credit history on, but this one card is really bothering me because I had more than a couple of late payments on it in the past. Is it safer for me to just cancel it out, rather than keeping it open?