The point is that babies aren't just a combination of their parents. It is possible for two creatures to have a child that is larger or smaller than either of them through mutation. Probably not much larger or smaller than them, but not necessarily the exact average. And then selection takes over.
Suppose you have 2 pairs of creatures, each of size 6. So our population looks like {6, 6, 6, 6}.
And each pair has four children. Some of the children are smaller, say, size 5, while others are larger, say, size 7. So our sample of kids is {5, 5, 5, 5, 7, 7, 7, 7}.
Now suppose all the small kids die. So we're left with {7, 7, 7, 7} as our population. Now all of our creatures are larger!
And now our population has kids. Since the parents are larger, the new batch of kids will be larger on average; some will be size 6 and some will be of size 8, so we get {6, 6, 6, 6, 8, 8, 8, 8}.
And then the small kids die, leaving us with {8, 8, 8, 8}. And again, our creatures are larger than their parents were.