They cleared away mound after mound from the garden, turning them once more into solid walls, for they were formed mainly of excellent stones, which had just been waiting to be put up again.

The only evil consequence was that the garden became a little less picturesque by their removal, although, on the other hand, a good deal more productive.
Yes, there was a second apparently bad consequence--the Priory spread as well as grew, until it encroached not a little upon the garden.

But for this a remedy soon appeared.
The next house and garden, although called the Manse, because the clergyman of the parish lived there, were Mr Shepherd's own property.
The ruins formed a great part of the boundary between the two, and it was plain to see that the Priory had extended a good way into what was now the other garden.

Indeed Mr Shepherd's house, as well as Mr MacMichael's, had been built out of the ruins.

Mr Shepherd offered to have the wall thrown down and the building extended on his side as well--so that it should stand in the middle of one large garden.
My readers need not put a question as to what would have become of it if the two proprietors had quarrelled; for it had become less likely than ever that such a thing should happen.