This is going to be our first serious conversation! xD
Anyways back to the point. In the end, no ones opinions matter, just take some peoples advice and make the decision on your own. You can let people change your decision a bit, but in the end it's all up to you.
Think about this, if your family doesn't like him, what would they think when you marry him if you do? Long story short, I'm going to say a short story of my friends family. Ok so lets say there's a sister named Angie, and another sister, and the husband of Angie is Bob. So Angie's side of the family loved Bob, and everyone in the family, and my friend who was their daughter. Until one day Angie's sister asked for something medical (won't get into detail to much) which would put Angie's life at risk, but her sister wouldn't be and never has been. Angie turned her sister down, and now Angie's family hates her, they never speak to her, and they don't even include her as a family member. My friend hasn't seen them in years.
Either way, see one little mistake with family members can ruin a whole entire family. And most the time it's over marriages, babies, and/or money. Which is the saddest part. Families are supposed to accept each other no matter what.
Anyways back on your situation, I understand that you've been together with him for awhile, but the reason that you noticed there was an issue will only get worse in the future. Has he even done anything violent to you? If so what would he do if you had kids with him?
In my opinion with the quote "It is better to love and lost, than to never have loved at all." I think if he is violent, and had problems, then take your separate ways. I know it would be very very tough, but it would probably be easier than getting serious with him.
Another thing is, is you say he doesn't care about you, then what is he there for? If he doesn't support you, and isn't your shoulder to cry on, then leave him. Why does he want kids? So he can marry you and claim to be a family member, then get a divorce and get half of your money? I don't know, in the end it's all your decision because you know him better than anyone in the world. Other than his mother of course! Just think what it would be like in the future, does he have/plan to have a stable job in the future? Why does he want kids so desperately? Think of all the questions, and ask yourself "Why is he here for me?"