The Empire was being depopulated long before the end of the period of peace and prosperity which stretched from Augustus to Marcus Aurelius.

Does not Augustus himself summon the poor man of Fiesole who has a family of eight children, thirty-six grandchildren and eighteen great grandchildren, and organize in his honour a fete in the Capitol, accompanied by a great deal of publicity?
Does not Tacitus, half-anthropologist and half-Rousseau, describing the noble savage with his eye on fellow citizens, remark that among the Germans it is accounted a shameful thing to limit the number of your children?
The long duration of Augustus's legislation to raise the birthrate is significant; successful it was not, but the fact that it was maintained on the statute book and systematically revised and developed for three centuries shows that it was at least accounted necessary.

It is true of course that the mortality rate was a far more important factor in those days than it is in our own, and the mortality from pestilence and civil war from Marcus Aurelius onwards was exceptional.

And it is plain that the proportion of celibates was high in the Roman empire and that the fall in the fertility of marriages was going on.

It is the childless marriage, the small family system that contemporary writers deplore.