The oldest part of the hall is the Tudor building, which dates back to around 1500 and was once the scene of a bloody battle between the English and Scots. The hall was extended and enlarged between 1830 and 1847, to designs by the architect Lewis Nockalls Cottingham. In the early twentieth century there was a secret tank development facility at Brougham Hall. The project was known as Canal Defence Light (CDL). A plaque at the Hall remembers the men who worked there during the war. There is also a bunker, which was used during World War II.
The hall was broken up by the 4th Baron Brougham and Vaux, Victor Henry Peter Brougham, in the 1950s so that he could pay his many debts. Today, the hall is the subject of a renovation project by volunteers and is open to visitors throughout the year.
The hall and grounds are reportedly haunted by sounds of soldiers battling in the middle night, world war soldiers marching and various people who were employed at the property including a woman called Emily and a boy who died there. Henry Brougham who lived there in the nineteenth century was reported to be highly interested in spirituality and his spirit and that of his brother William are also said to have been felt by visitors. The hall was investigated in the Living TV series Most Haunted where the team supposedly communicated with the spirits of Emily, the boy and Henry Brougham during a séance and moved a heavy table across the room without explanation.