I think he definitely had something coming to him, but what he actually got is more than anybody deserves, in my mind. D: I wouldn't wish it on the worst of the worst. (But then I've always had a particular aversion to fingernail/finger trauma >< )
What I find sad about Theon is that all the terrible things he did, he did trying to please his father. The father he hadn't seen since he was a kid, had built up in his head as some great guy. He was a cocky a-hole and kept telling people he was going to be Lord of Pyke, welcomed back with a parade as his house's last son and heir, etc. only to come home and find out that his dad thought he was weak and soft, and basically hated him. So he went out to try and change his mind, and turned out to make some really stupid and loathsome decisions that got him captured in the end. But he was never going to succeed. His father was going to resent him and see him as a symbol of the Greyjoys' defeat no matter what.
So, everything he did was for nothing, and wound up putting him in a massively worse position than he would've been in had he just lived with his father's hatred (or gone back to the Starks, who he remarked were more like his true family).
Theon's story kind of sucks all around.