Within the castle wall was the outer ballium or court, containing some small, low-roofed dwellings, the residence of many feudal retainers of the baron.

A rude church or chapel was also within this court, holding a communication with the keep or principal part of the castle by means of a passage in the third wall, which divided the ballium from the inner court.

In very large castles there were in general a second fosse, wall, gate, and towers guarding the keep, and thus making a complete division between it and the ballium; but the original owners of Kildrummie, less rich and powerful suzerains than their equals in South Britain, were probably contented with merely a stout wall to divide their own sovereign residence from their more plebeian followers.

The keep itself, constructed like all other similar buildings of the age, was a massive tower, covering but a small square, and four or five stories high.

There were attempts at luxury in the chambers within, but to modern taste the Norman luxury was little better than rudeness; and certainly though the cushions were soft and richly embroidered, the arras in some of the apartments splendid specimens of needlework, and the beautifully carved and often inlaid oaken walls of others, gave evidence of both taste and talent, yet the dim light seemed to shed a gloom and heaviness over the whole range of rooms and passages, which no skill of workmanship or richness of material could remove.