The now ruined St Mary's Abbey was originally founded as the minster of St Olave at Galmanho before 1055, but had been re-established by 1068 as a Benedictine monastery as part of an exchange of land between the Archbishop of York and monk Stephen of Whitby. Following a visit by William Rufus circa 1086-9, the church was found to be too small for the brethren and William granted land adjacent to the church to expand the abbey. A new church was built and rededicated to St Mary. It was the first monastic establishment founded in Yorkshire after the Conquest and became one of the wealthiest abbeys of the order. The monastery also had a mitred Abbot who sat in the House of Lords. In 1539 the abbey was dissolved by Henry VIII and acquired by the Crown for the northern headquarters of the King's Council.
The chief portions remaining are the late 13th century north aisle with arcading and traceried windows, the adjoining west wall and doorway, most of the 13th century precinct wall, towers and gatehouse. The present hospitium is a modern restoration.