Right, the Silence work by using post-hypnotic suggestion. Not only will you forget them when you look away, they can make you do things if they want. For one, it would work on the other person as well and they'd forget entirely. The Doctor uses a method to find out what they look like, putting micro-corders into the palms of his companions' hands as well as his own. Seeing as the Silence can't erase the recordings, they used that method to learn their appearance. They also managed to store a picture on a cell phone, but every time they looked at the image and looked away, they forgot again. So, a recording of a description is the only way you'd ever be able to get their appearance.