Well, I only was part of the AA for '07, which I'll get to in a minute, but here's a basic breakdown of each year.
2005 - Was utterly awesome. I'm a bit jaded, because it was my first convention and I walked in without any expectations. I didn't participate in many major events, but it was pretty easy to get around in and the dealer's room was nice.
2006 - Went with my friends rather than being alone, and it was honestly disastrous. We did group registration and the convention lost our reg information. There wasn't any wireless internet accessible, so we couldn't give them our confirmation email. (We had to take a 4 or 5 hour train ride in order to get there, so it's not like we could get to anyone's house or anything. D: ) We all had to register again, if I recall correctly, and it meant hours of waiting and searching and processing. The convention attendance had grown quite a bit from the year before, while the staff had not. So all of the staff members seemed tightly stretched. Most of the major events were badly organised. The AMV competition, for instance, had tickets handed out the day prior to the event. But the fact that the tickets were handed out was not publicised, so most people weren't aware of this. The people who didn't get tickets were made to wait for hours in a non-air-conditioned room (in L.A. heat) with no explanation of what was going on until about ten out of the hundreds of people who were waiting were allowed into the competition and everyone else was turned away. (Again, without an explanation.) I swear, a riot nearly took place. The convention was overall okay, good enough to go back to the next year.
2007 - The convention moved from the Anaheim convention center to the Long Beach convention center. The LB con center was larger, but the layout wasn't as easy to move about in. Again, more people attended but the staff seemed to have grown to match this time around. I can't say exactly what was wrong with it that year... but things simply weren't enjoyable. I've spoken to a lot of other people who attended, and got similar reviews. The con just wasn't fun. Although I had spent $50 on a table in the AA, I actually only wound up spending one day at the table. They were facing gigantic glass walls, and the heat and light of the sun coming through was simply unbearable. When I was actually trying to find where to get my table, none of the staff knew where the AA was supposed to be.
I think that AX just got too big, and the staff didn't know how to manage that many people and still keep it fun. :