An previous castle on this site was owned by the Siwards family, the present castle was built by the Oliphants, who held the land from 1360 to 1613, when the 5th Lord Oliphant was forced to sell the property.
By the late 15 century, the original tower and barmkin was on the site, since replaced by the remarkable walled gardens. The L-plan tower forming the east end of the castle is dated 1573, and bears the initials of Margaret Hay and Laurence, the 4th earl. The stone may be reset, but it is likely that the tower was built then for Margaret as a jointure-house.The upper works on the towers probably date from 1603-6, when the new main block was built and the older 15th century tower was heightened by a new top storey with round bartizans. (Bar"ti*zan` (?), n. [Cf. Brettice.] (Arch.) A small, overhanging structure for lookout or defense, usually projecting at an angle of a building or near an entrance gateway.)
The bartizans are mostly gone and the bartizan roofs are mostly gone, roofed over by the main tower roofThe 5th lord Oliphant suffered from financial problems and in 1617 sold Kellie to Thomas Erskine. The hall and dining room windows were enlarged just before 1661. Kellie passed to the 9th earl of Mar in 1829, but was mostly used as a farmhouse.
Kellie was abandoned in 1829, but by 1878 James Lorimer leased Kellie as a nearly roofless ruin and proceeded to restore it. Robert Lorimer, his son and a well-known architect, spent most of his childhood here. In 1970, Kellie came into the hands of the National Trust for Scotland.
Kellie may be one of the finest castles in Scotland. A very large E-plan, it consists of a three story main block with three large square towers that form the arms of the E. The towers are five stories, and two have bartizans.Two smaller stair towers project from the side of the main block.
Kellie was the home of the sculptor Hew Lorimer, Sir Robert's son. The stable block houses his studio, plus an audio-visual presentation about his life and work.