Collection Management

Collection Management evolved during the Second World War and was honed to a precision instrument during the Cold War. What I describe belongs to the HF era (and I’m only looking in detail at the Second World War here). The same principles apply today, but the way in which they are exercised is as totally different as are the comms in use in the twenty-first century. 

Even within the bounds of the laws of physics and the permissions a Sigint agency has, it can’t collect everything it would like to or even everything its customers say they would like to see; even so, it will always collect more than can be processed; what filters through the processing won’t all be intelligible to its analysts; and a significant amount of what is intelligible won’t actually be relevant.

The city on the hill towards which the collection management system is aiming is one in which the messages which are intercepted and processed are those which will allow the agency to produce the most important intelligence for the most relevant customers; and the smallest number of signals are intercepted which, because of irrelevance or unprocessability, won’t produce wanted intelligence.

The need for a Collection Management organisation arose because of what was at the time referred to as the ‘Enigma Complex’. The value of Enigma decrypts meant that the GC&CS view, that Enigma cryptanalysis was a single problem that needed to be managed centrally, won out over the service view that their collection facilities were theirs to control.

The basic problem for collection management was that at any time there might be ten or twenty enemy transmitters operating for every interception set, and that there weren’t enough intercept operators to operate each interception set 24 hours a day. Overcoming this issue meant the development of an understanding of how the Germans organised their communications networks, how within that organisational structure the individual networks operated, and an assessment of the value of intercepting some or all of the communications passed on that network.

To achieve this, the collection management system had to look two ways: towards the intercept stations to understand what they could collect given both their location and the limitations of their infrastructure; and towards the analysts to understand what was the minimum of allocation of resources that would provide them with enough (sometimes just enough) information.

Some examples: there was little interest in the content of German weather messages, but sent out at the start of the day, and repeated through various networks which used different crypto key settings, it meant that certain weather messages could be decisive for recovering the daily key for a number of networks, and in order to have the cleanest possible version of the message that had been sent, three or even four intercept stations would have a set and an operator copying it. Conversely, a garrison located in the Baltic where there was little or no operational activity might simply be sampled once a week or month, simply to maintain continuity and confirm that the garrison hadn’t moved or changed its comms procedures. During operations, Direction Finding might be the key contributor in the Sigint system, able to say exactly where a unit or formation was (as happened with the Bismarck on 25 May 1941), but was of limited use, and in fact a waste of time and resource, against permanent locations such as those serving Hitler’s Headquarters in the Wolfsschanze.

These examples are extremes. Collection management normally depended on a group of people who understood and trusted each other to manage allocation of receivers according to current operational requirements, the technical sophistication of the transmission, the need for continuity and the laws of physics. Collection would normally be carried out in military stations, and as a rule each service collected the traffic of its counterpart, but this was not a hard and fast rule. Diplomatic and Commercial Sigint, based, of course, in London rather than at Bletchley Park had dedicated civilian-staffed intercept sites, but its collection management staff were at Bletchley to ensure that the entirety of collection resources could be placed against all of Sigint’s targets – two extra brownie points for knowing that this is what Mary and Valerie Glassberrow, the grandmother and great-aunt of the Princess of Wales, were doing at Bletchley Park.

The key takeaway from this isn’t its size, or its complexity, or its flexibility, or that everybody involved in it had to take on trust the relative importance of a target at any time, or that application of the ‘need-to-know’ principle would have stymied this progress from the start: it is the fact that this process was invisible to everybody outside the Sigint organisation. Any bean counter observing the organisational structure might simply ask if all those posts (perhaps a hundred all told) might be abolished and a system set up in which analysts would simply task stations themselves. Well, they could be abolished, and the analysts’ tasking would be productive for a day or two, but with no-one to adjudicate between competing demands for receiver time (never mind the absence of a sophisticated process which worked across all targets and all stations) the value and quality of intercept, and of the intelligence produced from it, would decline rapidly.

We often describe Sigint production as a chain: stations intercept messages and send them to HQ where the intercept is processed before being presented to cryptanalysts and/or traffic analysts whose output goes (through linguists if the resultant text is in a foreign language) to intelligence analysts who issue a report to customers saying what the originator of the message said, but it is a satisfactory as an analogy only in the broadest sense. Collection management isn’t alone in being an internal function which is invisible to those who simply see the chain. Nigel de Grey includes, in what he calls the ‘Ancillary Sections’: the Machine Tabulating Section, Communications, the Central Signals Registry, the Signals Office, Personnel, Recruitment, Administration, Works and Buildings, and, of course, Security.

I blogged the first part of what I thought would be a two parter on Sigint Communications a while back. I got stuck on Part 2 trying to work out how to summarise it in fewer that five or six thousand words. These ancillary sections are key to the success of Bletchley Park during the war and for GCHQ’s success subsequently, and their story is as complex as those of the well known Huts. Sadly, these stories are seldom told.