What was originally a prosperous market town with its Saxon priory and 16th century grammar school didn't decay, neither was it destroyed by a natural disaster; it was too busy, especially the Grammar School, for the likings of Lord Milton who, rather than move himself to a more amenable location, moved the town instead.

The old town of Milton Abbas had grown up as a flourishing market town outside the gates of the abbey founded here by King Athelstan in 938. The town's Grammar School was founded by Abbot Middleton in 1521 and it became one of the most reknowned schools in Dorset (Admiral Hardy, whose monument stands on Blackdown, was a boarder here).

By the 17th century it had more than one hundred houses, four inns and a brewery which exported its ale not only throughout Dorset but to the capital as well. In 1674, an almshouse was built and endowed for six poor widows. When small change was scarce, four of the town's tradesmen were even prosperous enough to issue their own money tokens.

Over the space of twenty years, Lord Milton bought up the leases of the townhouses and, one-by-one, pulled them down until nothing remained by 1786 and he had an unobstructed view. The church bells were sold, the building demolished and, regardless of the graves and monuments, the churchyard converted to a lawn.

The Grammar School was relocated to Blandford and a new village built in a valley, out of site of his Lordship, for those folk for whome he had a use. The remainder were left to find a living wherever they could.