A motte and bailey castle probably built by Bishop William Gifford between 1107-1129, replaced in the early 13th century by a shell keep castle. From the later 13th century the castle was the primary adminstrative centre for the manor of Taunton Dene, it was also used to hold Assizes from 1280. The castle defences were improved in 1575 in preparation of an anticipated Spanish invasion, and in 1645 by the parliamentarians Sir Robert Pye and General Blake. An order to destroy the fortifications of Taunton in 1662 lead to the infilling of the moat and demolition of the Keep. A prison was in use at the castle until the late 17th century and the Great Hall was used for public meetings during the 18th and 19th century. In 1873 the Great Hall was bought by the Somerset Natural History and Archaeology Society and it houses the County Museum and local History Society.
The castle comprised a keep, inner ward and outer bailey enclosed by an outer moat 12 metres wide and 3 metres deep.which has been identified from excavations. The main castle buildings, situated within the inner ward, included the keep and extant buildings such as the Great Hall, Camera and Constable's Tower. The outer bailey, which contained auxillary buildings accessed by an eastern garden, has been extensively built over.