This whole arguement isn't valid at all to begin with.
First of all, I don't think that seeing or not seeing people walking around with anime paraphenalia is an indication of it's popularity, here or in Japan. You don't see people walking around with it here much, either.
Second, when it comes to finding anime stuff in stores, you are much more likely to find an anime store, cosplay cafe, or anime merchandise in general at a store in Japan than in the U.S. Phone cards, cell phone straps, plushies in arcades, figurines, manga books, doujinshi, even advertisements...you name it, you can find anime easier in Japan than the US. It's not that hard to find, at least in Tokyo in the places I visited; you might have more trouble in smaller cities in Japan, but then again, you'd have trouble finding anime in rural Kentucky, too).
Anime is produced for all audiences in Japan; shows like Gantz tend to attract an audience of over 16 due to their graphic/adult content. Adults (30+) don't watch much anime on TV, but they are a large portion of viewers who go see Miyazaki films. Besides, you don't see a lot of 30+ anime fans in the U.S. either.
Saying anime is more "adult" that American animation is both true and false. It is true that because of the cultural differences between Japan and the US, some anime content would be considered inappropriate for younger audiences in the US. Therefore, yes, anime can be considered "adult" in the US, because a large portion of anime fans in the US are over 18.
At the same time, it is not really "adult" because anime is a broad category of animated entertainment with shows targeting all audiences. Saying anime is adult (or not adult, for that matter) is like saying all PG-13 movies are made for teens.
The Simpsons, Family Guy, Futurama...these appeal to adults just as much as kids. The Flintstones and the Jetsons were created originally for adults (complete with laugh track to make it feel like a sitcom).
Just because anime is, for lack of a better word, a cartoon, does not make it non-adult. It's just another form of entertainment, and has many audiences. It is neither adult nor non-adult, unless it is intended to be one or the other (i.e. hentai).
The point is moot. Now shaddap.