There is no sign left and little description of the castle that Caileen Oig is said to have built in 1396 but there is a note of repairs in 1586. In July 1563 Mary Queen of Scots in reported to have stayed at Ardkinglas on a journey to Inveraray. The First Statistical Account 1791 to 1799 mentions the old residence of the family of Ardkinglas of which the ruins can now scarcely be traced, was at a small distance from the present castle but in a more commanding position. This site can possibly be identified by a massive stone lined tank or well on a ridge to the north-northeast. A report of 1792 by Sir John Sinclair tells of three separate towers and stout walls about 15 feet high. In the course of this wall is the great gate, which is defended by small round turrets in flank, with apertures, through which those who assailed the gate might be annoyed with arrows or with small firearms. A small tower immediately above it, the gate-tower, also defended the gate. The courtyard measured 98 feet in each direction and within the walls were smaller buildings for lodging servants, for holding arms and for storehouses and cellars. The castle was reported destroyed in 1769.
During the 18th century there had been several plans to build a country house at Ardkinglas and some famous architects were commissioned to produce designs. Among these were Colen Campbell, Robert Adam and later James Playfair. A house was built in 1795 and was described as a classical house of two storeys above a basement. In Richardson’s Guide to Loch Lomond & the Clyde 1799 it is described as Ardkinglass, the seat of Sir Alexander Campbell, Bart. This is a handsome new building, of modern architecture, finely situated upon the banks of Loch Fine, surrounded with pretty extensive plantations. However Stoddart (Remarks on Local Scenery and Manners in Scotland during the years 1799 & 1800) says the modern house is not in a happy style of architecture and Gernett agrees ..but the architecture by no means elegant. The old castle of Ardkinglas, a large and venerable pile of ruins, has been entirely demolished to make way for offices, a circumstance surely to be regretted by the lovers of picturesque scenery. In 1797 the house was recorded as having 66 windows. The mansion stood north of the Caspian and was nearly derelict in 1822 when Dorothy Wordsworth wandered through the estate. It was completely destroyed by fire in 1831 and the nearby stable block (probably also by Playfair) was converted to serve as the main house. Nearly half of this house was taken down when the current house was built and the rest converted to workshops. The first floor and roof of the 'Old House' were demolished in October 1996.