Grow and smaller aren’t linked or related (described further here) as they are interchanging. If a fish has grew to a certain size in adulthood, it is shrinking further than their predecessors, so they are bigger than them.