The now ruined St Olave's Priory was founded circa 1216 by Roger Fitz Osbert. Following the dissolution of this Augustinian priory in 1537 by Henry VIII, the site was purchased by Sir Henry Jerringham. In 1547, Jerringham constructed a three storey mansion situated north of cloisters which incorporated monastic remains. This mansion was demolished in 1784. Stone was removed from the site in 1823 to carry out repairs to Herringfleet church. In 1825 the refectory undercroft was converted to a cottage and used as such until 1902.
Part of the church, cloister and refectory survive and to the north, fragments of Jerringham's house can still be deciphered. The remains of the church, which is located to the south of the cloister, consists of five bays to the east of the west end and is of early 13th century date. The church was initially constructed without aisles, however a south aisle was added in 1300-10. The remains of the cloister include early 13th century brick faced piers which supported the roof of the cloister; the bases of three piers survive to the north wall and four to the south. The west wall of the cloister range also survives, pierced at the north end by an early 14th century ashlar doorway which would have led to the Prior's lodgings, guest rooms, kitchen and Cellarer's rooms. The north range, built in 1300-10, comprises part of the refectory buildings, of which the undercroft survives but the refectory which was on the upper floor has now been demolished.