Preparing the Ground for Room 40

There was nothing recognisable as Sigint in the UK before the First World War started, but we know a fair amount about interception before the war started and can make reasonable assumptions about why there was no permanent organisation created to monitor the wireless traffic of foreign armed forces. The idea that there was no serious thinking about the value of interception, and, its corollary, that there was no thinking about how an enemy might exploit UK communications, is not tenable. What follows is about the Admiralty – I’ll perhaps look at the War Office in another post.

The earliest indication I have seen so far of somebody thinking about a potential downside to the wireless telegraphy (WT) systems first being trialled by RN ships at the start of the last century comes from a 1900 report. Between March and May 1900 four ships on the Cape of Good Hope Station, HMS Forte, HMS Magicienne, HMS Raccoon and HMS Thetis, were equipped with Marconi WT equipment and carried out trials. Their reports, forwarded to the Admiralty by the Senior Naval Officer on the Delagoa Bay and Natal Division, concentrated on the performance of the equipment and its potential, but the report from the Captain of HMS Raccoon added:

‘I am strongly in favour of its suitability for communications between Her Majesty’s ships especially during patrolling operations of a similar character as those existing here. Had the presence of an enemy’s fleet to be considered, it would be a question if the possession of this instrument would not tend to disclose the proximity of your presence to the enemy.’

At its most basic, those thinking about foreign communication usage thought at the message level, rather than at the carrier level, and saw any potential value as deriving from messages transmitted during actions. Thus, in the 1902 Royal Navy fleet exercises, the Admiralty issued an instruction forbidding either fleet from acting on information derived from intercepted messages alone, because ‘in a real action, the signals would probably be unintelligible to the other side’; and in 1904, the British squadron in Chinese waters intercepted messages being passed between a Russian Admiral and his staff. The RN Commander reported to the Admiralty that ‘the character of the messages was not important, but the fact of interception showed possibilities that a belligerent's messages could be and probably would be intercepted’.

Radio procedures of foreign navies (French, German, Japanese, American and Italian) were studied to provide recognition: in 1904 British warships were instructed to transmit to the Admiralty copies of all foreign radio messages which they intercepted and in 1904 and 1905 the NID was informed of Intercept of Russian naval traffic by ships at Suez, the intercept of German naval traffic by ships at Corfu; and of Dutch stations by ships in the Baltic. Furthermore, the RN had acquired the French Naval Wireless Telegraphy Handbook of 1904; and by 1906, procedures used by the American and Japanese Navies had been added to the list.

The accepted view was that though wireless messages might be intercepted, encryption would render any intercepted message unreadable. In 1906 the commander of the Home Fleet (and later First Sea Lord) Arthur Wilson, stated that even if ‘the enemy has our signal books’, all messages in this compromised codebook would remain entirely secret so long as they were superenciphered by means of an elementary transposition cipher. In 1907 the head of the navy’s radio service and the Director of Naval Intelligence (DNI) both doubted that encoded radio traffic could ever be broken, since ‘under most circumstances it will only be necessary to cypher a few words in each message to render it quite unintelligible’.

The belief that encryption would render any message unreadable led to thinking in another direction: how to deny an enemy the possibility of wireless communication. Again, this thinking was not developed: jamming was not a possibility because it would mean jamming of own comms as well as of the enemy's. There was little thinking beyond the destruction from the sea of enemy coastal wireless stations and the cutting of enemy telegraphic cables.

Counter-jamming was studied: already by 1909 the first experiments in directional wireless telegraphy were being carried out by the Admiralty, at both shore stations and in ships at sea.

There had been no cryptanalytic organisation in the UK since 1844 but crucially, the pre-1844 Secret Offices looked only at civil and diplomatic material, not military. The Indian Army, following its experiences in South Africa, had set up a two man cryptanalytic bureau in India, but its function was to look at diplomatic and attaché traffic acquired from telegraph companies, something that, while legal in India, was illegal in the UK, and the product of which was of no interest to the British government.

In 1908 Fleet Paymaster Charles Rotter of the Naval Intelligence Division asked that any German Naval messages intercepted should be logged and forwarded to the NID. This work had certainly begun by 1909 and a file of this material certainly existed in 1973 in GCHQ though it was probably subsequently returned to the MOD and is now waiting to be discovered in an ADM piece in The National Archives.

The file contained raw traffic and work-sheets. Some of the raw traffic was recorded on specially printed forms headed ‘Return of Intercepted Wireless Telegraph Messages for the month of ...’ These returns were prepared at the point of interception, presumably by transcription from the actual logs, and sent to ‘Admiral Commanding Reserves’ by the ‘Chief Officer’.

The worksheets showed successful decryption of one encryption system, and work on callsigns and the collation of versions received from the different interception points. These interception points were: Whitehall (presumably the Admiralty), Aberdeen, Ipswich, Dover, Rosyth, Port Patrick, Pembroke Dock, St. Helier, Essex Hill, Hunstanton, and, at least from 1912, Scarborough. At the Admiralty end, monthly batches of returns as received from the stations were ‘referred for information’ to DID in person. What the latter made of the raw intercepts is not clear.

Callsigns were two-, three-, or four-letter. It was easy to identify many: for example RO, NA, AKO and AHO stood respectively for the ships ROON, NAUEN, KOENIGSBERG and HOHENZOLLERN. Frequencies were reported as ‘Wavelengths in feet’. They included 5000, 6000 and 6500 (the roundness of the figures suggests strongly that the measurements were pretty rough).

The monthly return forms had a column for ‘Remarks, Strength of Signals; If musical, etc.’ It was not much used, but some transmissions were described as ‘musical’ or ‘high musical’; others were described as ‘whistling note’ and ‘quenched spark’. Strengths were given in a scale up to 10 and ‘rates’ – presumably words-per-minute – were of the order of 12 to 15.

The traffic that was read at the time was a 10-letter code; each group consisted of two more or less pronounceable 5-letter ‘words’. There were clear indications that a German codebook had been acquired physically by some means, rather than that it had been reconstructed cryptanalytically. All the messages were originated by ‘Admiralstab’, callsign KMD (probably Kriegsmarinedienststelle – Naval HQ), using the main German naval transmitter at Norddeich. The messages refer to movements of German ships or to the deployment of British and other warships in the Mediterranean and home waters – and on one occasion to an eruption of Mount Etna! The readable sequence started in March 1910 and finished with the following message dated 11 July 1910: ‘This is the last telegram from the Admiralstab’. (Or might this have been a comment?)

There is no evidence that any other cypher was read at the time. Most of the unread traffic is in 5-letter transposition systems.

Conventional wisdom is that the Naval Intelligence Division had arranged for the Secret Service to purchase a copy of a German Naval codebook from an agent. £600 went into this venture but the codebook which eventually turned up was a forgery. I wonder, however, whether it isn’t just as likely that the codebook was used until 11 July 1910 at which point it was superseded.

Though this activity by Rotter was tolerated, he was acting in isolation. At his request, during the naval manoeuvres of 1912 and 1913 a separate division of the Admiralty War Room, known as Section C, had been assigned to work on intercepted wireless traffic, but nevertheless when the War Room was mobilised on 30 July 1914 it was simply decided that such a unit was unnecessary, and Section C was dropped.

Just before the advent of war, Cdr Frederick Loring RN, probably the Navy's expert in wireless telegraphy at the time, summed up why interception would not prove a serious threat to naval wireless in wartime.

‘... the more skilled the organisation, the more difficult it is for a strange operator to take down with the necessary accuracy the groups of a code message: he cannot ask for the repetition of doubtful groups, and he has no intimate and daily familiarity with the methods of his opponent to assist him in his task. And, after all, giving the enemy every advantage, giving him a perfect record of the signals, the key of the code to his hand and equal facility of skill and language to translate it for use—a most improbable combination, it must be admitted—he has still failed to prevent the all-important information reaching its destination.’

That Loring was wrong in 1914, just as the FSL and the DNI had been wrong in 1906 and 1907, shows how different Sigint is from any other sort of intelligence that existed in 1914 and how difficult it was to imagine it. A combination of the traffic analysis carried out in the War Office by Room 40 staff in September and October 1914, the capture of the Magdeburg Codebook and its arrival in London at the end of October, Rotter’s working out the superencryption system used by the German High Seas Fleet, and Winston Churchill’s imaginative insight into this totally new source meant that by 8 November Room 40 had been established as the first cryptanalytic bureau in the UK since 1844 and was ready to become the UK’s first Sigint organisation.

Collection Management

Collection Management evolved during the Second World War and was honed to a precision instrument during the Cold War. What I describe belongs to the HF era (and I’m only looking in detail at the Second World War here). The same principles apply today, but the way in which they are exercised is as totally different as are the comms in use in the twenty-first century. 

Even within the bounds of the laws of physics and the permissions a Sigint agency has, it can’t collect everything it would like to or even everything its customers say they would like to see; even so, it will always collect more than can be processed; what filters through the processing won’t all be intelligible to its analysts; and a significant amount of what is intelligible won’t actually be relevant.

The city on the hill towards which the collection management system is aiming is one in which the messages which are intercepted and processed are those which will allow the agency to produce the most important intelligence for the most relevant customers; and the smallest number of signals are intercepted which, because of irrelevance or unprocessability, won’t produce wanted intelligence.

The need for a Collection Management organisation arose because of what was at the time referred to as the ‘Enigma Complex’. The value of Enigma decrypts meant that the GC&CS view, that Enigma cryptanalysis was a single problem that needed to be managed centrally, won out over the service view that their collection facilities were theirs to control.

The basic problem for collection management was that at any time there might be ten or twenty enemy transmitters operating for every interception set, and that there weren’t enough intercept operators to operate each interception set 24 hours a day. Overcoming this issue meant the development of an understanding of how the Germans organised their communications networks, how within that organisational structure the individual networks operated, and an assessment of the value of intercepting some or all of the communications passed on that network.

To achieve this, the collection management system had to look two ways: towards the intercept stations to understand what they could collect given both their location and the limitations of their infrastructure; and towards the analysts to understand what was the minimum of allocation of resources that would provide them with enough (sometimes just enough) information.

Some examples: there was little interest in the content of German weather messages, but sent out at the start of the day, and repeated through various networks which used different crypto key settings, it meant that certain weather messages could be decisive for recovering the daily key for a number of networks, and in order to have the cleanest possible version of the message that had been sent, three or even four intercept stations would have a set and an operator copying it. Conversely, a garrison located in the Baltic where there was little or no operational activity might simply be sampled once a week or month, simply to maintain continuity and confirm that the garrison hadn’t moved or changed its comms procedures. During operations, Direction Finding might be the key contributor in the Sigint system, able to say exactly where a unit or formation was (as happened with the Bismarck on 25 May 1941), but was of limited use, and in fact a waste of time and resource, against permanent locations such as those serving Hitler’s Headquarters in the Wolfsschanze.

These examples are extremes. Collection management normally depended on a group of people who understood and trusted each other to manage allocation of receivers according to current operational requirements, the technical sophistication of the transmission, the need for continuity and the laws of physics. Collection would normally be carried out in military stations, and as a rule each service collected the traffic of its counterpart, but this was not a hard and fast rule. Diplomatic and Commercial Sigint, based, of course, in London rather than at Bletchley Park had dedicated civilian-staffed intercept sites, but its collection management staff were at Bletchley to ensure that the entirety of collection resources could be placed against all of Sigint’s targets – two extra brownie points for knowing that this is what Mary and Valerie Glassberrow, the grandmother and great-aunt of the Princess of Wales, were doing at Bletchley Park.

The key takeaway from this isn’t its size, or its complexity, or its flexibility, or that everybody involved in it had to take on trust the relative importance of a target at any time, or that application of the ‘need-to-know’ principle would have stymied this progress from the start: it is the fact that this process was invisible to everybody outside the Sigint organisation. Any bean counter observing the organisational structure might simply ask if all those posts (perhaps a hundred all told) might be abolished and a system set up in which analysts would simply task stations themselves. Well, they could be abolished, and the analysts’ tasking would be productive for a day or two, but with no-one to adjudicate between competing demands for receiver time (never mind the absence of a sophisticated process which worked across all targets and all stations) the value and quality of intercept, and of the intelligence produced from it, would decline rapidly.

We often describe Sigint production as a chain: stations intercept messages and send them to HQ where the intercept is processed before being presented to cryptanalysts and/or traffic analysts whose output goes (through linguists if the resultant text is in a foreign language) to intelligence analysts who issue a report to customers saying what the originator of the message said, but it is a satisfactory as an analogy only in the broadest sense. Collection management isn’t alone in being an internal function which is invisible to those who simply see the chain. Nigel de Grey includes, in what he calls the ‘Ancillary Sections’: the Machine Tabulating Section, Communications, the Central Signals Registry, the Signals Office, Personnel, Recruitment, Administration, Works and Buildings, and, of course, Security.

I blogged the first part of what I thought would be a two parter on Sigint Communications a while back. I got stuck on Part 2 trying to work out how to summarise it in fewer that five or six thousand words. These ancillary sections are key to the success of Bletchley Park during the war and for GCHQ’s success subsequently, and their story is as complex as those of the well known Huts. Sadly, these stories are seldom told.