It's not that people don't consider it art, it's that people learn stylization, especially over exaggerated features without understanding the anatomy and other basics and it tends to look bad and inconsistent.
Despite what some people say, you do need to know anatomy to draw nice anime and manga inspired art. Even if exaggerated and disproportional, you have to know what you are doing and about the human body to make it work. That is the problem.
Also anime just means animation, there is no set style. There are often common traits, but each artist has their own style and skill level. Some are more realistic than others.
I love anime but my first picture was bad. Why? I didn't know anatomy and the character I drew had no joints, I didn't know how to make clothing realistic, didn't know much about shading, highlighting, perspective, looked flat,etc...I am actually trying to learn that stuff now. I been working on gesture drawing and drawing random crap I see in my house lately. I need to study and practice a lot more as I have a lot to learn (though I am a hobbyist, not trying to become professional, I still want it to look good and improve), but I can see improve though compared to last year.
Most books and tutorials on anime and manga just cover the style but none of the basics mostly (though a few are getting a bit better at that and talk about the anatomy and how it should look realistically first, proportions, different fabric and how they should move and look, etc...though they still leave out a lot and hardly go in depth with the basics as that isn't the focus). People usually jump in without knowing what they are really doing.
Basically it is just a style, you should develop your own style, though it can be influenced by someone as not all artist are the same or should be the same and need a solid understanding of how things work.
I would suggest taking a class if you can, though the classes I took turn me off from drawing for awhile. All we did was copy in one class why the teacher hardly really said anything. Just mindless busy work where we weren't really learning anything. The second teacher had us draw those boxes, but he didn't talk about the vanishing point and all that really, which didn't help me with perspective. So make sure it is a good class first.
I am assuming you are talking about in general and on the forums and as a hobby since you didn't mention schools, being a professional. portfolios, etc...As for schools and stuff, it usually doesn't impress many people (though I know some good artist, though they have their own style and solid grasp of the basics), probably because people aren't really developing their skills and style when drawing or really expressing yourself through the art and looks more like a copy. They want to see what you know more than anything else first. Schools focus on making sure the student understands things properly and help them to correct mistakes.
People didn't say you can't draw anime or that it isn't real art, just that they highly recommend drawing from life first and building upon what you know before stylizing.
Just my thoughts from a hobbyist and going through the forums.