In 1312 Robert the Bruce is said to have given Ascog to the Bute family of Glass. In 1594 the estate, including a mill, Loch Ascog and Nether and Over Ascog, was bought by John Stewart of Kilchattan, a distant kinsman of the Stewarts of Bute who later became Earls and later Marquesses of Bute.
John Stewart may have built the first house at Ascog, replacing an older tower. Despite the date of 1678 above one of the dormer windows, the original Ascog House was earlier than that. With its stair tower and cap-house, it is of a type commonly built around 1600. Moreover, in the wall of the present kitchen is part of a grand chimneypiece. This belonged to a great hall whose floor and ceiling were both at a higher level than today. In 1673 John Stewart of Ascog, grandson of the first John Stewart, married Margaret Cunningharn and it is their initials that are engraved on the house. They must have carried out a major reconstruction, lowering the floors to create two main storeys, and adding the dormer windows. John Stewart was rich enough to lend the Earl of Bute £9,385 to help re-build Rothesay Castle after damage in the Civil War. He was also crowner or coroner of Bute from 1666-98.
During the 18th century, the original mullion windows of Ascog House were enlarged and fitted with sash and casement frames. In 1773 another John Stewart, who had no children, made a complicated will intended to ensure that Ascog would always be owned by a Stewart. His heir, a cousin named Archibald McArthur, had therefore to change his name. Archibald Stewart was said to be both mean and eccentric – he kept pigs in his drawing room in Edinburgh – but he helped pay for a road from Rothesay to Ascog in 1813. He too had no children. The next heir was a distant American cousin, Frederick Campbell. He tried to sell Ascog but the terms of the old will defeated him. His brother Ferdinand, a professor of mathematics in Virginia, succeeded where he had failed and sold Ascog in 1831 to the eminent engineer Robert Thom.
Ascog passed through various hands until in 1939 when it was bought by the Earl of Dumfries, later 5th Marquess of Bute. Meikle Ascog was lived in by Lord Rhidian Crichton Stuart, then let to a Mr Collins and finally to Patrick Crichton, who left in 1988. Ascog House was divided into several dwellings for estate employees, but structural problems began to appear and the house gradually emptied. To secure its future the late Lord Bute approached the Landmark Trust, as a charity which rescues and cares for historic buildings. In 1989 Ascog House, its gardens and the nearby Meikle Ascog were placed in Landmark’s care