Collection Management

Collection Management evolved during the Second World War and was honed to a precision instrument during the Cold War. What I describe belongs to the HF era (and I’m only looking in detail at the Second World War here). The same principles apply today, but the way in which they are exercised is as totally different as are the comms in use in the twenty-first century. 

Even within the bounds of the laws of physics and the permissions a Sigint agency has, it can’t collect everything it would like to or even everything its customers say they would like to see; even so, it will always collect more than can be processed; what filters through the processing won’t all be intelligible to its analysts; and a significant amount of what is intelligible won’t actually be relevant.

The city on the hill towards which the collection management system is aiming is one in which the messages which are intercepted and processed are those which will allow the agency to produce the most important intelligence for the most relevant customers; and the smallest number of signals are intercepted which, because of irrelevance or unprocessability, won’t produce wanted intelligence.

The need for a Collection Management organisation arose because of what was at the time referred to as the ‘Enigma Complex’. The value of Enigma decrypts meant that the GC&CS view, that Enigma cryptanalysis was a single problem that needed to be managed centrally, won out over the service view that their collection facilities were theirs to control.

The basic problem for collection management was that at any time there might be ten or twenty enemy transmitters operating for every interception set, and that there weren’t enough intercept operators to operate each interception set 24 hours a day. Overcoming this issue meant the development of an understanding of how the Germans organised their communications networks, how within that organisational structure the individual networks operated, and an assessment of the value of intercepting some or all of the communications passed on that network.

To achieve this, the collection management system had to look two ways: towards the intercept stations to understand what they could collect given both their location and the limitations of their infrastructure; and towards the analysts to understand what was the minimum of allocation of resources that would provide them with enough (sometimes just enough) information.

Some examples: there was little interest in the content of German weather messages, but sent out at the start of the day, and repeated through various networks which used different crypto key settings, it meant that certain weather messages could be decisive for recovering the daily key for a number of networks, and in order to have the cleanest possible version of the message that had been sent, three or even four intercept stations would have a set and an operator copying it. Conversely, a garrison located in the Baltic where there was little or no operational activity might simply be sampled once a week or month, simply to maintain continuity and confirm that the garrison hadn’t moved or changed its comms procedures. During operations, Direction Finding might be the key contributor in the Sigint system, able to say exactly where a unit or formation was (as happened with the Bismarck on 25 May 1941), but was of limited use, and in fact a waste of time and resource, against permanent locations such as those serving Hitler’s Headquarters in the Wolfsschanze.

These examples are extremes. Collection management normally depended on a group of people who understood and trusted each other to manage allocation of receivers according to current operational requirements, the technical sophistication of the transmission, the need for continuity and the laws of physics. Collection would normally be carried out in military stations, and as a rule each service collected the traffic of its counterpart, but this was not a hard and fast rule. Diplomatic and Commercial Sigint, based, of course, in London rather than at Bletchley Park had dedicated civilian-staffed intercept sites, but its collection management staff were at Bletchley to ensure that the entirety of collection resources could be placed against all of Sigint’s targets – two extra brownie points for knowing that this is what Mary and Valerie Glassberrow, the grandmother and great-aunt of the Princess of Wales, were doing at Bletchley Park.

The key takeaway from this isn’t its size, or its complexity, or its flexibility, or that everybody involved in it had to take on trust the relative importance of a target at any time, or that application of the ‘need-to-know’ principle would have stymied this progress from the start: it is the fact that this process was invisible to everybody outside the Sigint organisation. Any bean counter observing the organisational structure might simply ask if all those posts (perhaps a hundred all told) might be abolished and a system set up in which analysts would simply task stations themselves. Well, they could be abolished, and the analysts’ tasking would be productive for a day or two, but with no-one to adjudicate between competing demands for receiver time (never mind the absence of a sophisticated process which worked across all targets and all stations) the value and quality of intercept, and of the intelligence produced from it, would decline rapidly.

We often describe Sigint production as a chain: stations intercept messages and send them to HQ where the intercept is processed before being presented to cryptanalysts and/or traffic analysts whose output goes (through linguists if the resultant text is in a foreign language) to intelligence analysts who issue a report to customers saying what the originator of the message said, but it is a satisfactory as an analogy only in the broadest sense. Collection management isn’t alone in being an internal function which is invisible to those who simply see the chain. Nigel de Grey includes, in what he calls the ‘Ancillary Sections’: the Machine Tabulating Section, Communications, the Central Signals Registry, the Signals Office, Personnel, Recruitment, Administration, Works and Buildings, and, of course, Security.

I blogged the first part of what I thought would be a two parter on Sigint Communications a while back. I got stuck on Part 2 trying to work out how to summarise it in fewer that five or six thousand words. These ancillary sections are key to the success of Bletchley Park during the war and for GCHQ’s success subsequently, and their story is as complex as those of the well known Huts. Sadly, these stories are seldom told.

The Bletchley Chapter of 'People Against Uncongenial Work'

Much of the success of Bletchley Park during the Second World War is owed to the combination of great talent and an atmosphere in which it could flourish: a combination of the Common Room of a college, the managed anarchy of the tiny pre-war GC&CS, and the respect for – or at least the toleration of – people who didn’t conform to the societal norms and expectations of mid-twentieth century Britain.

But that isn’t the whole story. Nigel de Grey, in his Organisation and Evolution of British Sigint (TNA HW 75-78) refers at times to the unwillingness of some of the people who benefited from BP’s toleration of individualism to take their share of necessary chores, the necessity of which was not obvious to them. Here are two examples.

‘The Tube Central

The Pneumatic Tube System was installed in order to overcome the delay in distribution attendant upon the use of messengers, the only labour available for which were girls from local primary schools on leaving, whose hours of work were strictly controlled by Civil Service regulations and whose enthusiasm and discipline was not noticeably high, and to take the place of the belt conveyors which had been in operation to adjacent sections. The principle adopted was to have four main trunk double lines to the four requisite office blocks terminating at a point where a single line internal system conveyed the traffic to the several requisite points within the office block. The "cross country" tubes were laid underground and the distances were in some cases the maximum possible, hence the necessity for relaying within the blocks. The system suffered from two main troubles, firstly that the bore of the tubes and hence of the containers was not large enough to cope with the documents to be distributed or indeed to carry the overall volume; secondly that owing to indifferent manning of the local stations by the user sections the maximum was never got from the system. Communications Section was forced eventually to man both ends of the trunk lines but the interior terminals were as a rule manned by casual, not to say extremely casual, labour and Communications Section received very little cooperation from the Sections, some of which seemed to find any routine beyond their capabilities or infra dig.

Time stamping Machines

Analogous to this laxity was the difficulty experienced by Communications Section over the Stromberg time-stamping machinery. The regular and rapid flow of papers (decrypts, signals and so forth) was an essential to the main function of GC&CS, namely decrypting and reporting to the combat commands. To achieve this it was essential to time the arrival and departure of documents at each stage of the handling so that any bottle neck should at once be revealed and steps taken to relax it. In order to make this easy electric time-stamps were introduced in May 1943 (when, it will be remembered, GC&CS was "working-up" for the Second Front) and installed at strategic points in the Cypher Office, the Main Teleprinter Room, the Auto Room, the Tube Central, Huts 3, 6 and 8, Naval Section and Air Section. To avoid all discrepancies the clocks in these machines had to be synchronised daily, a routine for which was laid down on the principle of dialling TIM in London. The minutes of the Communications Committee show the struggle that Communications Section had to get the Sections to conform to this very simple routine. All sections were careless but Hut 6, a key point, consistently ignored not only the synchronisation of their clocks for weeks at a time but the use of the stamp at all. Thus when in the autumn of 1943 Communications Section were straining every nerve to increase the rapidity of their service to Hut 6 and asked for the cooperation of Hut 6 in testing the results of their experiments in handling, the tests broke down three times because Hut 6 would not use the time stamps properly. Hut 8, another key point, two months after the institution of the time stamps was found to have locked its Stromberg up in a cupboard and left it there. Another section's clock had stopped and never been used again. Thus the two worst offenders were the two sections which employed the highest grades of labour, both being manned principally by university trained personnel who had no doubt been taught to think for themselves. The instances quoted, perhaps trivial in a sense, were none the less symptomatic of the difficulty of getting any businesslike routine carried out, any suggestion of mechanisation in an organisation manned by untrained labour and with whom no disciplinary action could be taken except that of dismissal should they prove insensible to an ordinary reprimand.’

The voices of the educated – ‘university trained personnel who had no doubt been taught to think for themselves’ – tend to predominate in the story of Second World War British Sigint, and it’s too late to gather the memories of those trying to improve the information flow using Time and Motion methods such as the use of Stromberg time stamping machines; and, anyway, the value of the results of the graduates’ work in cryptanalysis and machine-based solutions on any day probably outweighed any benefit potentially lost. But nevertheless qualified people with skills which might have improved the way that BP worked were being ignored.

And this wasn’t the worst thing that was happening. As de Grey hints in telling the story of 14 year old ‘girl’ messengers, and as is clear from any telling of the stories of (for example) cipher operators, not all jobs at BP were stimulating or interesting; not all needed great minds; and many were as dreary as might have been found in any factory job to which labour had been directed, but with the additional pressure that those working (mainly women) on these tasks could say next to nothing outside work about the dreary, mundane nature of their working lives.

A necessary consequence of the industrialisation of UK Sigint during the Second World War was the creation of production line jobs which, like those in contemporary factories, were not stimulating. Add to that a need-to-know system, in which the people doing the most menial jobs knew the least about the way in which their work contributed to the organisation as a whole, and it is not hard to see that without wartime regulations on directed labour, Bletchley Park might have found it difficult to achieve the success it did.

 

 

 

Enigma and the Poles

Chris Smith (@spy_historian) tweeted a series of tweets about the Polish contribution to Enigma cryptanalysis this morning. I copy them here, adding some comments, and then write a bit more broadly about the issue. 

CS: Polish work on Enigma was truly impressive. They broke it while the British basically ignored it because they deemed it insoluble. A waste of time.

TC: The British had broken the commercial variant of Enigma early in 1927, and an improved solution was developed in time to exploit its use by the Spanish and Italian Navies during the Spanish Civil War and subsequently. I don't think it's fair to say that they thought the military variants insoluble, but up to 1939 they had no idea how to approach the problem. Put crudely, people who broke book-based codes, and simple electromechanical ciphers couldn't break more complex machine-based ciphers.

CS: They recognised the value of machine-based approaches. Though the Bomba was rapidly rendered obsolete by upgrades to Enigma systems, the Bomba was proof of concept. Use machines to break machines. I've argued elsewhere that Enigma and Bombas were an industrial revolution.

TC: Mechanical support to cryptanalysis wasn't new: Hollerith machines had been used in Room 40 during the First World War and other machines were being proposed in the second half of 1939 as GC&CS recruited from a deeper pool. The specific Polish breakthrough was to design, build and deploy a machine that mimicked Enigma. Following Turing's meeting in Parish with Rejewski he adopted the same concept for his (otherwise very different) Bombe.

CS: The contribution of the Poles, who shared their successes with the British, paved the way for upscaling that culminated in the SIGINT phenomenon that was #BletchleyPark. The BP Trust were right to create a Polish memorial, Prince Andrew was right to gift Poland an Enigma.

TC: Although the BP Trust were wrong to give one of the Enigmas GCHQ had loaned to them to Prince Andrew to gift to the Poles … !

CS: However, it has become a trope that this Polish contribution has been largely unrecognised in Britain and the US. My argument is that this is simply untrue. In fact, from the early days of Ultra becoming public knowledge - 1974 - the Poles were recognised.

CS: It is almost impossible to find a book that doesn't recognise their *massive* contribution. Yet these same texts often state that they have been under-recognised while recognising them. It is a weird, self-replicating myth. 

TC: The problem is that the nature, and particularly the source, of the Polish contribution was either ignored or grossly oversimplified (as largely was the case with the UK's own cryptanalytic work): so little was released by GCHQ in the first twenty years after the 1974 revelation that mythology filled the gaps. 

CS: So powerful is the myth, it has caused minor international spats. The Polish ambassador to the UK complained about Polish elision and misrepresentation (arguably rightly) in the 2001 film Enigma. In 2016, the Polish state commissioned a touring exhibition to correct the record.

TC: Both Enigma and The Irritation Game played their part in reinforcing the mythology (just as U-571 ignored the UK) but Hollywood blockbusters aren't documentaries.

CS: But the record didn't actually need correcting. The early British lit on BP/Ultra was clear that Poles did the lion share of early work. See: Lewin (1978), Calvocoressi (1979), Hinsley et al (1979), Collier (1982), etc.

------------------------------

Actually, Hinsley et al were wrong and their record did actually need correcting. Vol 3 pt 2 (1988), Appendix 30 (pp 945-939) was written by Joan Murray (née Clarke) and Henry Dryden, both wartime Siginters who had stayed on to work at GCHQ, and who were retained after retirement to update the version of the Polish contribution recounted in Appendix 1 of Vol 1 which was written on the basis of records and knowledge which was later proved – for example by the publication of Rejewski’s memoirs – to be incomplete. At best, I don't think it's unreasonable to have expected Hinsley et al to have done rather better first time.

CS: So where does this all come from? As ever Group Captain F.W. Winterbotham. His big splash book, The Ultra Secret (1974), which (sort of) revealed Ultra totally got the Polish work wrong. He was writing from memory and, besides, didn't know everything. Loads of that book is wrong.

CS: Yet as soon as it was published, key people in the know, not least Tadeus Lisicki, a wartime Polish intelligence officer and cryptanalyst, wrote to the papers, in 1974, to point out the Polish work. Lisicki compiled a dossier that formed the basis of important books by Poles.

TC: By Poles, in Polish, and while some were later translated, their impact was limited, not least because the authors weren't appearing at book festivals, on Radio 4 or in broadsheet review pages.

CS: Examples include Garlinski (1979), Woytak (1979) and Kozaczuk (1984). All of which have appeared in English. Newspapers, TV shows, radio comedies and even movies have made the point - though poorly in the case of Enigma. So why does this myth of anglo 'chauvinism' persist?

CS: None of this is to dispute the role Poles such as Marian Rejewski, Jerzy Różycki and Henryk Zygalski played. Quite the reverse. What fascinates me is the endurance of a myth that these pioneers have been elided from English language historiography. They haven't.

I remember telling a friend, when I became GCHQ's Historian-designate in 2008, that the two things I didn't see myself getting involved in were Enigma and VENONA. How wrong I was! Enigma remains a live issue for many – perhaps most – people outside the academic Intelligence Studies community who were interested in Intelligence history.

In the British public imagination the Bletchley Park story was the acme of the Boffinry: the British had been successful in the Second World War because they had out-thought their enemies. To the list of back-room boys coming up with Spitfire, Radar and bouncing bombs, was added the super-smart mathematicians who in complete secret helped win the war by breaking all German cryptographic systems. As with the Battle of Britain, Bletchley Park became part of a mythology of plucky little Britain fighting on alone and prevailing against all the odds, in spite of the evidence. The Polish contribution became merely transactional: a Pole handing over to the British an Enigma machine which was reverse engineered.

This popular narrative would not survive unchallenged the changes in Poland after 1989: the return to democracy, Poland joining NATO and the EU, and the move to the UK of younger Poles who had learned at least some of the story of Poland's contribution to the Enigma story. A Polish/UK Historical Commission reported in the first decade of the twenty-first century and described in English for the first time the breadth and depth of the Polish contribution to intelligence across the board, while GCHQ's massive 1994-2004 release of Second World War records had been absorbed and it was possible to begin to approach the question of Anglo-Franco-Polish cooperation on Enigma rather more reasonably than had been the case previously. Pioneering work by Dermot Turing and Marek Grajek also produced new accessible historical information that could be shared in English and Polish.

The problem I found as GCHQ Historian was that while 'the record' was becoming clearer, the Polish sense of their contribution having been slighted for so long had allowed a mythological counter-narrative to develop in which the UK and France would never have come near to solving Enigma without the Polish contribution. It was for that reason that I coined the term 'Enigma Relay' to try to make two points: first, that the credit of solving Enigma belonged to the allied team, the Poles, French, British and Americans each running separate laps and passing the baton; and second, that solving Enigma wasn't something worth 100 brownie points and that each of the four nations should scrabble to establish how many of the 100 they could each claim. International Intelligence cooperation doesn't work like that; solving Sigint problems doesn't work like that. I tried the 'Enigma Relay' out on a few people and then, when Polish, UK and French Sigint representatives agreed to meet in Warsaw in 2014 to celebrate the seventy-fifth anniversary of the tripartite meeting at which the three countries agreed to share all they knew, put the 'Enigma Relay' concept into the speech given there by Iain Lobban, Director GCHQ (https://www.gchq.gov.uk/news/director-gchq-commemorates-crucial-pre-war-enigma-information-sharing-meeting-poland).

Did this resolve all of the issues? No, of course not. Narratives and counter-narratives, and mythologies, have a life of their own and accurate history will always find it hard to compete against what non-historians would like a two dimensional 'truth' to look like. But reinforcing on every possible occasion the fact that success against Enigma took a lot more than a couple of very bright mathematicians thinking great thoughts, however crucial those great thoughts were, is part of the job. And making British audiences realise that most Sigint successes since 1939 are due to GCHQ's partnerships with Sigint agencies in other countries goes beyond mere intelligence history, and, hopefully, leads people to reflect that intelligence doesn't happen in a vacuum.

Sigint Communications (part 1)

This follows on from my previous post, in which I said, referring to a twitter discussion: 

'Part of agreeing with @jock_bruce here is a belief that environmental awareness is an essential precursor for Sigint, and that all is a waste of time without adequate Sigint Comms from intercept site to HQ. Sigint is about the whole of Comms.'

Sigint relies on having a dedicated, high-quality, high speed communications network, linking its national headquarters, intercept stations, intelligence allies and customers. High quality is absolutely essential; high speed will always be necessary in some parts of the network but not in others; a dedicated network is the outcome of two factors: the need for the highest levels of security across the whole of the network, and the need to ensure that prioritisation of traffic flow is something decided on by the Sigint agency and not by the organisation that supplies the comms infrastructure.

The first issue is to ensure that the headquarters can communicate with its intercept stations and with its customers. Preparing for the Second World War, one of the advantages of choosing Bletchley Park as the war station for GC&CS was the fact that the GPO had laid trunk cabling along the LMS railway. When the military elements of GC&CS returned to London after having deployed temporarily to Bletchley during the crisis of autumn 1938, they were able to make reasonable guesses about the comms capacity they would need in wartime, and by August 1939 BP had been well linked by teleprinters and telephones both to intercept stations in the UK and to Whitehall. This network of landlines was expanded and improved during the war under the aegis of the Y Committee. Overseas, however, underpinning the whole question of Sigint communications was the GC&CS policy of centralising UK cryptanalytic effort in the UK, partly from the increasing complexity of cryptanalytical problems, partly from the dearth of trained cryptanalytical staff if a large number of service centres were to be established overseas. (Later experience and the development of cryptanalytical machinery tended to confirm the original policy.) Three overseas cryptanalytical centres were in existence in 1939: the FECB for which GC&CS trained first naval and subsequently military officers; Sarafand, an Army commitment insofar as staffing was concerned, and Simla for which the Government of India found the personnel.

The Mediterranean campaign produced the CBME and the various commands in the field. The FECB was replaced by Delhi and Colombo/Kilindini on the entry of Japan which brought into existence the American organisations in Washington and the SWPA and the inter-allied Combined Bureau in Brisbane. The American policy was likewise to centre cryptanalytical work in Washington. Interception was spread right round the world and intercepted traffic had to be sent to the main centres as rapidly as possible. Long distance air mails were as a rule neither rapid nor certain enough as a principal means of conveyance, so that the whole or nearly the whole burden of carrying the signal correspondence of the Japanese Armed forces round the world fell upon Allied telecommunications. The intelligence produced from the intercept then had to be redisseminated by the same means.

The main source of material from GC&CS between the wars was Diplomatic correspondence which passed over the ordinary commercial WT services, for which interception in the United Kingdom was pure routine, or by cable, the lines of which in very many cases transited British territory. By special arrangements copies of all such Diplomatic correspondence were supplied to GC&CS, coming from overseas where required by sea bag. This was a leisurely proceeding but adequate for the Government' s purposes.

On the military side, and recognising that timeliness was likely to be a bigger issue for military Sigint than for diplomatic, the first meeting to investigate improvements in timeliness was held December 1937 but resulted in little other than an agreement that specified military intercept might be sent by air bag (under diplomatic protection) rather than by sea. The collection of DF bearings, however, needed a more real-time solution. 'C' wrote in April 1938 that 'cable companies claimed that the result of the Derby could be received in the uttermost corners of the earth within 5 seconds of the result being known' and that this must be mirrored by the Signal branches of the Services if DF was to be of any value. This didn't result in any specific action.

In September 1939 the problem of Y communications did not loom very large. The Army interception units that accompanied the BEF only had eleven intercept sets and ten DF sets between them and no plan was in place to site them near the network of telegraph and telephone lines established by the Royal Corps of Signals. The use of wireless for passing DF results was forbidden on security grounds. It was assumed that any Sigint would only be relevant to the BEF Staff so no dedicated link to the UK was planned. The RAF intercept unit sent to France was linked by WT to Cheadle, and thus, if necessary, with GC&CS but was regarded as an offshoot of Cheadle rather than an independent unit, since there was no Air GHQ in France until the end of January 1940. In other words no special system of communications for Y formed a part of the BEF pre-war plan, while in the case of the RAF the deployed unit was part of the home defence system. After a reorganisation of the Army Y system an I(s) staff was formed and sited near GHQ, as was 2 Company GHQ Signals, and a cryptanalytic party formed from the Military Section of GC&CS was sent to the French GQG. Smaller intercept units were moved to Corps HQs at Roubaix and Douai but the only communication with them was by bad telephone lines or by dispatch rider over singularly bad roads. I(s) was, however, in touch with MI8 by unreliable teleprinter though it was recognised that these links would only work while units were static. Material intercepted close to the Front could not be sent to GHQ in a timely manner, so I(s) missed many of the perceived advantages of being in the operational theatre rather than in the United Kingdom. The only means of any military intercepted traffic reaching the United Kingdom was by air bag, though a new DF station erected near Chartres was linked to Chatham by telephone and formed part of the Chatham DF network.

The first RAF unit at Fismes placed two DF units in Bar sur Seine and Amiens in order to form a baseline. These stations were linked by French Post Office landlines and Fismes was eventually linked to AI1(e) through the British controlled Rheims Central. This, it would seem, was led through to Leighton Buzzard Defence Teleprinter Network Switchboard and so, when working, gave them access to Cheadle and GC&CS (though all traffic has to pass en clair). The DF telephone lines connecting Bar sur Seine and Amiens with Fismes constantly broke down and when the organisation was later linked to the Intelligence Staff at AASF and BAFF the same occurred: 'our greatest handicap is landlines' reported the unit. The result was that Fismes seldom got more than a single line bearing, which made DF (and the DF units) essentially useless. A second RAF unit was formed to intercept Italian Air Force traffic from the south of France. This unit had no communications with GC&CS (which produced a lot of good IAF reports) and was handicapped by having few if any trained operators: it was of no real value. It used a courier service, French as far as Paris and British thereafter, which was expected to get the material intercepted through to GC&CS in 24 hours but, in fact, took about a week.

The Admiralty claimed to have a satisfactory teleprinter line to pass naval intercepts from the South of France and told AI1(e) who managed to obtain one, but only in May 1940. In the same way efforts to connect the Bar sur Seine station by British line to Fismes were only successful on the day when the station was handed over to the French during the retreat.

There were, however, arrangements for GC&CS communications with the French: in May 1939 it was agreed that Paris should telephone London at 3 p.m. each day and reports would be exchanged and confirmed later by bag. By August 1939 there was some level of communication between Bletchley Park and the Deuxième Bureau by RT, presumably conducted by Section VIII SIS. A courier service carrying cryptanalytical information continued and was extended to cover meteorological intercepts. The cryptanalytical party from GC&CS sent to the French GQG to collaborate on German Police ciphers and medium echelon Army traffic s needed no dedicated communications with the UK.

The 'Manual of Military Intelligence in the Field' current in 1940 stated that 'Wireless telegraphy is such a reliable and efficient medium of intercommunication that its use in war is indispensable to a modern army. Indeed it is likely that it may prove the only practicable method of signal communication in campaigns involving rapid movement over long distances' but in terms of Sigint nothing had actually been planned to make this a practical proposition.

There are several points to be made about this first phase of Sigint communications. First, nobody had thought about the problems of dealing with the dissemination of high echelon traffic that had to be worked at the centre. Second, there was no solution of German Enigma and no optimism in GC&CS that there might be, so there was no reason to think out a plan for disseminating it. When the Norwegian key was broken in April 1940 there were only normal signals channels and normal signals crypt systems to disseminate the intelligence. When the main GAF key was broken during the Battle of France it was far too late to institute a new communications plan.

The Army view of field Sigint was that it would be wanted in the field and they arranged for interception and for DF on a local scale. They foresaw 'rapid movement' by the enemy but not the corollary: that this would need rapid Sigint communications to cope with it. They had been slow to interest themselves in strategic DF and the development of the Chatham DF network was, at the outbreak of war, still a comparatively recent idea. Chartres DF was an outpost of this network and was given communications in conformity with the scheme. But that the same principle applied to the field units seems to have been overlooked. Very much the same may be said of the RAF field organisation.

Lastly, the Battle of France was over so quickly that there was nothing to inform planning for better communications: either between units; or between units and the deployed Intelligence Staffs; or between any deployed unit and the centre in the United Kingdom. Given the overwhelming defeat of the Allied Armies it would have been hard to show that lack of Sigint communications had in any way affected the issue or that the contribution of Sigint to the general pool of intelligence had been other than comparatively meagre. So when the focus of the war moved to the Mediterranean, Sigint as a whole was still lacking in experience of its real requirements for either long distance communications or with Command Headquarters or for field communications between Y units, and between them and any base organisation formed in the rear.

 

Environmental Knowledge as a Precursor to Sigint

 

This post arises from a brief discussion on Twitter recently of a comment by Jock Bruce that 'If amateurs talk tactics, and professionals talk logistics, then intelligencers talk comms' to which I answered 'Part of agreeing with @jock_bruce here is a belief that environmental awareness is an essential precursor for Sigint, and that all is a waste of time without adequate Sigint Comms from intercept site to HQ. Sigint is about the whole of Comms.' This post addresses environmental knowledge, the first part of my reply.

Everything below refers to Sigint as practised in the Second World and Cold Wars. This is mainly because it's simpler and easier to explain and understand, but also because none of it is in any way classified. The principles are as true today as they always have been, but I don't intend to explain how they have been adapted in the digital era. How does an Analyst Catch a Terrorist? might give you a start if you are interested. Also, and for the record, no Danes were harmed in the production of this blog post, and the memory of regular attendance at a NATO committee whose Chair, a (retired) Danish Army officer, offered a shot of Gammaldansk to all attendees at 8.00 each morning the committee met, has always drawn me to an idea that Denmark's marauding past might be a lot nearer to the surface than people think.

Let us imagine that the UK and Denmark have become bitter enemies and that the threat of armed conflict is no longer a matter of fantasy. GCHQ's Director goes to a series of meetings in Whitehall and is told that GCHQ must start producing intelligence on the Danish military to assess the level of threat posed to the UK. He gets back to Cheltenham and finds that GCHQ has never really targeted Denmark: there are a few diplomatic and Comintern reports, as well as a bit of  Venona, but nothing since the early 1940s, and nothing of significance ever. So what does it do?

We know that the Danish armed forces will be highly structured and that there will be a radio communications system that closely mirrors the command structure, and therefore the order of battle, of the Danish military. We know that there will be formal processes for transmitting orders from higher to lower levels in the hierarchy and for these orders to be acknowledged, as well as for other information to be exchanged. We know that these processes will be highly structured. We know this because military communications are pretty well standardised. There is a best way of using the electromagnetic spectrum to support military activity and this was discovered and developed in the first half of the twentieth century in much the same way by everybody.

So GCHQ's first stop is Defence Intelligence to ask what it knows about the Danish armed forces. Through the Defence Attaché in Copenhagen they will have at least a top level understanding of the structure of the Danish military. It is likely that they will quickly be able to come up with a diagram like this from open source (NB not Osint – more on this later):

(see https://en.wikipedia.org/wiki/Royal_Danish_Army#/media/File:Royal_Danish_Army_-_Organization_2021.png)

I will stick to the Danish Army from now on, but developing intelligence on the other two arms of service will happen in the same way.

This is the first concrete information GCHQ has about the organisation of the Danish Army and is of great importance because we can predict from the order of battle how different elements of the Danish Army will communicate with each other. For example, there will be a top level Command network linking Army Command, 1 Bde and 2 Bde. It may well include each of the regiments listed on the Orbat (plus or minus the Guards units: are they purely ceremonial or do they have war roles?) depending on how the military is structured to transition from peacetime to war. Defence Intelligence will be developing its understanding of how the rectangles on the Orbat diagram translate into working relationships with the Danish Army and there will be an ongoing dialogue between DI and GCHQ over this. There will of course be a host of subordinate units: each regiment is likely to have battalions, and each battalion companies: but how many, and where, and what comms structures support them?

GCHQ's collection sites will have been tasked to look for Danish military comms. This is like looking for needles in a haystack, but search specialists have a range of skills and tools which make this less frightening that it might seem to the layman. Direction Finding (DF), for example, makes it easy to say that various unidentified comms aren't from Denmark, as well as giving more or less confidence that other comms are. Linguists will be preparing working aids to teach operators basic differences between Danish, Norwegian and Swedish, so that operator chat can be made use of.

Typically at each level of the hierarchy, from the top level Command network described above, to brigade or regimental networks, the search specialists will be looking for patterns: a Control station calls up subscriber stations to its network at fixed times (say morning and late afternoon) one by one. They acknowledge the call up. Control then says which stations it has messages for, and the subscribers say whether they have messages for Control. Some of the messages in both directions will be relays: for example Control asks the subscriber to forward the message to a subordinate subscriber on a subordinate network or the message might be travelling upwards.

But who is who is not very clear: each entity will have a callsign by which it is identified, and it is highly likely that the callsign will be encrypted and so will change every day. Networks at each level of the hierarchy will work in the same way, so it won't be immediately obvious whether an operator is listening to a top level or a low level. Operator chat will give clues; DF will help; analysis of the messages being relayed to see how many levels of hierarchy they pass through; information from allies will give other clues. You will notice that the content of messages doesn't need to be decrypted for this activity to be carried out. Decryption is a very-nice-to-have short cut, but most of this traffic analysis needs to be done anyway. Hopefully, after a few months, a reasonably clear idea of how the Danish Army communicates will have been developed.

But this is no more than a snapshot. Periodically, each network will change the frequencies it works on and its callsign systems. It will have different procedures for use in wartime, procedures which may or may not be the same as the procedures it uses when the units with which each subscriber is linked deploy on exercise.

But the Danish Army has been brought to the same position as the military forces of countries which GCHQ has been monitoring to 'maintain continuity'. There is an art in making sure that the minimum amount of collection resource (because nobody has ever been able to collect everything) can be applied to making sure that the maximum amount of information about the comms structures of potential targets will be produced. For example, a fortnightly check on the comms structure of a logistics battalion in northern Jutland may be enough to reassure you that nothing has changed; a monthly DF check that the callsign associated with a Headquarters element in a bunkered command post is still in situ may be enough for reassurance, but the frequency with which these checks take place has to be adequate to give adequate notice if there is any change.

The key aim of 'maintaining continuity' is ensuring that regular patterns are known and understood. No country can afford to keep its military on high alert for long, and certainly can't keep it on any sort of alert level for ever. Units have to be rotated between more or less forward roles, have to be reskilled to take on new responsibilities, have to be adapted to new roles, as well as having to respond to random events like weather, but these are all more or less predictable, and the manner in which units adapt, as seen from their comms, can give the clue to how they might adapt to future conditions: to war.

So after a few months we have a good handle on the Danish Army. We know what normality looks like. We know what it looks like when it changes its alert level. We know how its comms are likely to change when readiness levels change. We are confident that if they are ordered to war stations, we will be aware and will be able to report it, and to follow them as their comms move to wartime modes.

But none of this is intelligence. We have developed all of the sources of information available to us to be able to keep a handle on how the Danish Army is communicating. But it is only when we take that information and extrapolate from it, for example to say that a major comms change is a regular occurrence rather than an indicator that the Army is preparing for its war role, that we are producing intelligence.

I said above that the organogram showing the high level Danish Army Orbat wasn't intelligence: it's information. It assembles readily available data into readily usable information, but it doesn't answer what for a Sigint organisation – as well as the subsequent all source assessment that Sigint feeds into – is the key question: 'so what?' The same question applies to Osint: yes, finding the data and assembling it into information is important, but the 'intelligence' part of Osint should mean that it explains how the information illuminates what the target's intentions and realistic capabilities are.

All of this is environmental knowledge. Until this level of knowledge has been built up, a Sigint organisation can't begin to produce intelligence. It has to expend significant resource to maintain the environmental information that enables it to turn into an intelligence producer when a target turns active, and is no longer just watched on a 'care and maintenance' basis. And if it has been difficult and complicated with a nation state's military forces, how much more difficult is it with an individual with a telephone? And if state interception of HF signals in free space where users had 'no expectation of privacy' was once unconstrained by the law, as the state (in the UK at least) 'owned' the way in which the electromagnetic spectrum was used, how much more complex is it today to navigate all of the necessary oversight and legislative constraints to obtain and retain information which might never be used?

I suppose that what I want you to go away with is that Sigint is a much more complicated business than you might have thought, and certainly isn't about intercepting and listening to every telephone conversation between London and Washington. The art and science of Sigint happens a lot further back than the point at which intelligence reports are produced, much as a military force's achieving significant effect on the battlefield owes almost everything to what had happened in the background, over a long period of time.

One Pip Instead Of Two Toots

 

I promised another extract from Signal. This is another story about the Royal Navy's need for comms discipline, and the lengths to which they would act to protect it. This time, instead of chummy telegraphists sending each other messages on the back of official signals, we have a solitary one, onboard a vessel or at a shore station so far down the hierarchy of importance that although present every time his network controller carried out a scheduled call up of his correspondents, our poor hero never once had a signal to send. So one day …  

A nondescript nonentity, a limb of the oppressed,

I wear no badges on my arm, no medals on my chest,

But though my past is colourless, my future dim and bleak,

I cherish a distinction which is probably unique.

Of all the mass of traffic through the tortured ether hurled,

By all the busy Tels of all the navies of the world,

No morse of mine impinged upon a fellow sparker's ear;

I never sent a signal in the whole of my career.

 

I used to wonder meekly when control would let me in

To add my little quota to the universal din.

Then realised my destiny, surrendered to my fate,

Eternally to sit and serve by being told to wait.

 

But once - and only once - I found my baser self constrained

To break the wireless silence I so rigidly maintained.

My weary watch was over, my relief was overdue,

I gently, briefly, pressed the key to see what it would do.

 

I often sit and wonder where that blameless dot has gone,

If still through endless time and space it hurries bravely on,

Disowned by its creator, and dismissed its parent ship,

Unauthorised, attenuated, lonely little pip.

 

But though beyond our universe its travels may extend,

It still will bear my fingerprints on reaching journey's end

And beings in some unknown world may trace it back to me,

As surely as the Flagship did in 1943.

 

Comms Indiscipline In The Royal Navy

 

In 1993 Captain Barrie Kent RN (Retd) published Signal! A History of Signalling in the Royal Navy. It is one of my favourite books, a serious history interlaced with anecdotes varying from the poignant to the slightly scurrilous. Captain Kent was Head of RN 'Y' during the 1950s, but his book has much more to offer than cryptology in the RN. This is the first of two extracts from the book: they show how seriously the RN took comms discipline. The first is a story from 1931 about Lord Louis Mountbatten when he was Fleet Wireless Officer in the Mediterranean Sea and comes from Arthur McCulloch, a Telegraphist in 1931, but who retired eventually as a Lieutenant Commander.  

'Our transmissions were taped resulting in caustic signals to those not up to his high standard. One 'habit' Mountbatten deplored was the sending of cryptic private messages during the night watches: for example in Coventry we might receive 'INT KYE' from the operator in Bulldog, which meant 'request permission to close down for five minutes to make the cocoa?' If things were quiet the reply would be 'AF'. This procedure would be followed in turn by each ship on the destroyer frequency until all had made their kye. '

Drafted to the Royal Sovereign, McCulloch became a watchkeeper on Fleet Wave, a frequency believed to be monitored by Lord Louis.

At a conference for off-watch telegraphists in the Corradino canteen, Lord Louis said that since he had started monitoring, the efficiency of the fleet's communications had improved dramatically, but the making of private messages was to stop, "The sending of one single 'toot' on the key will constitute a private message and the culprit will be severely dealt with.'

That afternoon things were fairly quiet when McCulloch was handed a message to send to Curacao. Switching on the transmitter, he sent the signal, then reached behind his morse key to break the heavy duty switch; in the process he accidentally caught his elbow on the key, blasting out a loud 'toot' for all to hear!

'That was bad enough but the operator in Curacao forgot the forenoon's instructions and sent back a cheery 'toot toot'. With hardly a pause, out came a signal from the C-in-C

'Report name and rating of operator of the watch'.

'On the Saturday I felt I had better have a final run ashore, all sorts of dire punishments having been forecast by my messmates. While enjoying a few pints one of my pals saw a poster for the Flagship's Farewell Ball that evening; we decided to join in! The dance hall was packed with officers and men from every ship in the fleet, together with their ladies, and a great time was being had by all. Our chance to join in came with the announcement: "Take your partners for the Paul Jones."

'My third partner was obviously an officer's wife, she was a superb dancer and I said so. Just then Lord Louis danced by and gave her a big smile which she acknowledged with a discreet wave. "I see you know my Chief,” I said. "I ought to, "she replied, "I'm married to him!" "If you are one of his telegraphists" she went on, "you'll know he is furious with two of you for making 'toots' or something, In fact we've heard so much of these men that we've christened them 'Toots No. I, and 'Toots No. 2"'. I paused for a moment and then said "I'm afraid' I'm Toots No. I ", She stopped in the middle of the floor and held me at arm’s length, "I don't believe it -I'm actually dancing with 'Toots' himself! You must come and meet everyone."

'Having escorted her to her seat, I set off towards the bar. "McCulloch", Lord Louis' voice boomed behind me, "So, you're the culprit are you? What have you to say for yourself?" The ladies of his party caught up with us and Lord Louis introduced me. "Now, what have you to say for yourself?" I explained what had happened: "Bad luck really Sir, particularly coming on top of your lecture." Lady Edwina spoke up: "I'm sure he is telling the truth Dickie, don't be too hard on him," "Fortunately for you, young man, I'm familiar with the layout of the wireless gear and I can see how it could have happened. Perhaps I should give you the benefit of the doubt. I'll write to your Captain before he sees you, but remember to be more careful in future." I thanked him and shot off to tell my pals what had happened.

'In due course I found myself in front of the Captain. "Ah! I've had a letter about you. I see you had an interview with the FWO ashore and he recommends that I give you another chance. Case dismissed. Don't do it again!" Sad to say my oppo in Curacao was not so fortunate; his Commander had been at Jutland and blamed all telegraphists for allowing the German fleet to escape, so he got a month’s stoppage of leave.'

There is more to this story than meets the eye: the memory of Jutland; Lord Mountbatten's determination that RN signalling would be as good as it could be (I have written about this before (Sigint Historian: Ten avoidable problems which made the Royal Navy's encryption exploitable in 1939)); the sophistication of the RN monitoring process; and the way comms operators can undermine security without even realising that they are doing so.