The prison was the best possible thing to happen to the group.
Everyone likes to think Woodbury is the best option. It isn't. Sure, it looks like home, it looks normal, it gives the impression that things will be okay. That they'll go back to how it used to be soon. But assuming that things are safe within the walls can lead to complacency, an idea that nothing can hurt you because you don't SEE the danger. Another thing is that Woodbury was just too big to defend. Even if you block off every single street leading to the neighborhood, that's a LOT of streets to block off and constantly keep watch over. On top of it, there didn't seem to be enough land to grow food, which would be completely necessary to sustain a large population.
Another point is weapons and protection: the prison is full of both, from arms and ammunition to body armor and shield use by riot police. There is enough room in the prison for each individual to have their own room, even if it isn't the most luxurious place. If necessary, someone can be locked in a cell or locked OUT of one, if things get out of hand. There's a mess hall, medical facilities, places to shower, to use the bathroom... hell, they even had a library. And having a bit of downtime is 100% necessary to staying sane. There are guard towers very high in the air so people can keep a look out over large areas. There were double fences, topped with barbed wire, and very few means to enter. If there had been full power, it's even possible those fences might be electric.
Honestly, the prison was the best possible place to keep a population protected. All Woodbury had was that it LOOKS pretty.