Castle Bytham is a Medieval motte and bailey castle, believed to have been built in the 11th century by Drogo, brother-in-law of William I. In 1220-21 William de Fortibus fortified the castle against King Henry III. Henry successfully besieged the castle and ordered its destruction. The manor was rebuilt. In the 15th century the castle fell into decline and by 1544 was in ruins.
The motte is a conical flat topped mound about 18m above the river valley and covering an area of 100m by 80m. The summit is occupied by the remains of a quadrangular keep including ranges of rooms around a central courtyard. Basements in the south range are thought to have been part of a strong tower or donjon. Remains of other structures include a postern gate, the main gateway, and two substantial stone towers. The motte is surrounded by a deep ditch and on three sides with the remains of a stone curtain wall with mural towers and the remains of an internal barbican. The earthwork remains of the bailey are to the south and east of the motte. The bailey is approximately 155m by 80m and is divided into two parts by the remains of a stone wall. To the north is a walled inner courtyard, and to the south a larger outer courtyard which contain remains of ranges of stone buildings. These are believed to have included domestic, agricultural and service buildings such as stables, barns, brewhouses and lodgings.
To the north and west of the motte and bailey earthworks is a low, flat area known as Castle Yard. This is the remains of a large pool associated with an elaborate water control system built on the original course of the river and consisting of a series of artificial channels and ponds linked by dams and sluices.