With headquarters in London it was always probable that the Society's buildings would be affected by the Luftwaffe's raids. Two bombs hit the building in 1941, the most devastating on the night of April 16th. One member was killed, three people injured and the building severely damaged. Areas of the library were ruined and an estimated 35,000 volumes and 5,000 pamphlets and documents were lost. Recovery was difficult; appeals were made in the press and collection centres were established across the Empire. The War Damage Commission eventually awarded the library almost £22,000 but various decisions by the Society's Council led to the purchase of new volumes rather than the replacement of the lost stock (Rees 1968, pp.198-200).

	In 1980 a flood caused considerable damage to the Library.

The post-war years saw growing uncertainty for the library. Financial troubles once again loomed large on the horizon. In 1968 a major collection was sold to the National Library of Australia. However, it was the 1988 Tysoe Radley Report that marked the beginning of the end. Commissioned by the Society to advise on achieving financial viability, the report suggested the library was "a dead thing" (Barringer, 1994, pp.3-4). Debate and dispute followed but by 1991 it had become evident that the Society's financial problems were far more pressing than generally envisioned. Apart from the building itself, the library was the Society's major asset. In May 1991 the Society's Council agreed that, were it to become necessary, they would sell the library. By late October postulation had turned to reality and it was announced that the library was to close by the end of the year (Barringer 1994, p.5).