The house was built in 1718 on the site of an Elizabethan house by a London builder, Thomas Blagden, for Elizabeth Cornelisen of Camberwell, London. The detailed articles of agreement survive in the Colchester and North East Essex branch of the Essex Record Office [D/DR T28/21]. Nothing of the earlier house survives, but some of the old timbers were reused within the new building. The new house was square in plan with three storeys of four rooms and a basement containing the kitchen and store rooms. The west elevation included a small projecting bay containing the staircase, half landings and closets. Because of the considerable social changes in the 200 years since construction, it is difficult to be precise about the original uses of most of the rooms.
The building was completed by March 1719 but sadly Elizabeth Cornelisen died before she could live in it. The house passed to Sarah Creffeild who later married Charles Gray (1696 - 1782) in 1727, when the house passed to him and he lived there until his death. He was a significant figure in the town, being an attorney, a Justice of the Peace and one of its Members of Parliament between 1742 and 1780. He was also a prominent antiquary, being a Trustee of the British Museum and a friend and correspondent of the Essex historian Philip Morant and the famous antiquary, William Stukeley. These interests may have stimulated his mother-in-law, Mary Webster, to buy him Colchester Castle as a wedding present, adding it to Hollytrees estate which already included the eastern half of what has become Castle Park. Much of Gray’s energies went into the improvement of his house and estate. He restored the Castle, landscaped the grounds, built the arcade at the north east corner of Hollytrees and planted the trees which give the house its name.
In 1748 Gray commissioned the local architect, James Deane, to add an extension to the west side of Hollytrees. The description and drawings for this work survive in the Essex Record Office [D/DRe E8]. It made use of the existing staircase so that the rooms in the extension are at mezzanine levels opening onto the half landings. The extension contained a basement scullery, servants’ room and arcade to the garden on the ground floor, a library and two attic rooms. A large fragment of the original wallpaper in one of the attic rooms was rediscovered in 1928 and is preserved in the museum store.
Little is known of alterations to the house during the 19th century when it was owned by the Round family who were major local landowners and prominent members of Colchester society. However, a diary written by Charles Gray Round (1797 - 1867) gives some idea of life in the house in the first half of the 19th century. Census returns show that there were a considerable number of people living in the house. In 1881 there were nine members of the Round family and the same number of servants. It is likely that some, perhaps most, of the servants were accommodated in a block to the west that also contained stabling and which was demolished in 1926 to complete the setting for the war memorial.
In the early 20th century the house was rented out. Tenants included the organising committee for the 1909 Colchester Pageant which ranks alongside the construction of the Town Hall as the apotheosis of flamboyant civic pride in Colchester.
In 1920 Hollytrees was sold to Colchester Corporation who had already bought the western half of the grounds in 1892 to create Castle Park. Once the sitting tenant had vacated the house in 1926, Colchester Corporation employed the local architects, Duncan Clark and Beckett, to convert it to a museum with accommodation for a member of staff on the upper floors. This resulted in substantial alterations to the building. Hollytrees opened to the public in 1929 as a museum of mediaeval antiquities and later bygones.
The second floor of the west wing was used as a library for the Essex Society for Archaeology and History from 1929 until 2000. A substantial redisplay of the museum, including the addition of studwork partitions on the first floor, was carried out in 1964 and there were major repairs to the roof in 1992.
In August 2001 the museum opened after fourteen months of major redevelopment funded by the Heritage Lottery Fund. This saw the installation of a lift, the re-orientation of the museum entrance, new visitor facilities and the creation of the present displays on the themes of domestic life and childhood in Colchester over the last 300 years.