Basically, any place where the crystal has time to grow, undisturbed. Other than that, it depends largely on what type of mineral is doing the crystalizing. Some crystals grow faster than others, meaning that its crystals could form in a place that is more likely to be disturbed. Others require a certain amount of heat or pressure.
Here's a few examples.
This is a photograph from a mine near Naica, Mexico. The crystals are selenite, a form of gypsum, and they are some of the largest crystals in the world at as much as 40 feet long. The person in the picture is wearing a protective suit that insulates him from the extreme heat in the cave. The cavern where they were found was originally flooded with water before the connected mine drained it. The minerals formed crystals as it precipitated out of its suspension in the water. Naica is hardly the only place that selenite has formed, and it was well known that it is a crystal dependent on water for form before the discover of the Naica cave. But it is apparent that it likes heat as well.
This is a picture of a pair of amethyst cathedrals we have where I work. They are about five feet tall and are two halves of the same geode. They were found in Brazil in an area where there had once been a great deal of volcanic activity. Geodes form as pockets of gas in magma and lava. After they harden, water carries other minerals into the space and deposits them, forming crystals. This particular set was found in a long tube of volcanic stone.
It's important to remember that a crystal's shape is an outward manifestation of its molecular structure. The more disturbed the crystal is while it is forming or the more other "stuff" gets in the way, the more distorted it will be or the smaller the crystals will be. So you're going to want to find places that will be undisturbed by civilization, flora and fauna, and recent seismic activity. This is why caves tend to be some of the best places to find crystals.
It's also important to note that the faster a crystal forms, in general, the softer it will be. Selenite, for example, forms relatively quickly in comparison to crystals in the quartz family. However, it dissolves in water quite readily. So the really durable crystals, the ones that might in a fantasy setting be a magical artifact passed from person to person over a number of generations, probably took a very long time to form.
Don't know if any of that rambling helped at all. But it's the light I can shed on it.
