Good question; I never thought about it... maybe because he lived with the Dursleys (however you spell it) for 10 years? 'Cause Lily was technically a witch...