Environmental Knowledge as a Precursor to Sigint

 

This post arises from a brief discussion on Twitter recently of a comment by Jock Bruce that 'If amateurs talk tactics, and professionals talk logistics, then intelligencers talk comms' to which I answered 'Part of agreeing with @jock_bruce here is a belief that environmental awareness is an essential precursor for Sigint, and that all is a waste of time without adequate Sigint Comms from intercept site to HQ. Sigint is about the whole of Comms.' This post addresses environmental knowledge, the first part of my reply.

Everything below refers to Sigint as practised in the Second World and Cold Wars. This is mainly because it's simpler and easier to explain and understand, but also because none of it is in any way classified. The principles are as true today as they always have been, but I don't intend to explain how they have been adapted in the digital era. How does an Analyst Catch a Terrorist? might give you a start if you are interested. Also, and for the record, no Danes were harmed in the production of this blog post, and the memory of regular attendance at a NATO committee whose Chair, a (retired) Danish Army officer, offered a shot of Gammaldansk to all attendees at 8.00 each morning the committee met, has always drawn me to an idea that Denmark's marauding past might be a lot nearer to the surface than people think.

Let us imagine that the UK and Denmark have become bitter enemies and that the threat of armed conflict is no longer a matter of fantasy. GCHQ's Director goes to a series of meetings in Whitehall and is told that GCHQ must start producing intelligence on the Danish military to assess the level of threat posed to the UK. He gets back to Cheltenham and finds that GCHQ has never really targeted Denmark: there are a few diplomatic and Comintern reports, as well as a bit of  Venona, but nothing since the early 1940s, and nothing of significance ever. So what does it do?

We know that the Danish armed forces will be highly structured and that there will be a radio communications system that closely mirrors the command structure, and therefore the order of battle, of the Danish military. We know that there will be formal processes for transmitting orders from higher to lower levels in the hierarchy and for these orders to be acknowledged, as well as for other information to be exchanged. We know that these processes will be highly structured. We know this because military communications are pretty well standardised. There is a best way of using the electromagnetic spectrum to support military activity and this was discovered and developed in the first half of the twentieth century in much the same way by everybody.

So GCHQ's first stop is Defence Intelligence to ask what it knows about the Danish armed forces. Through the Defence Attaché in Copenhagen they will have at least a top level understanding of the structure of the Danish military. It is likely that they will quickly be able to come up with a diagram like this from open source (NB not Osint – more on this later):

(see https://en.wikipedia.org/wiki/Royal_Danish_Army#/media/File:Royal_Danish_Army_-_Organization_2021.png)

I will stick to the Danish Army from now on, but developing intelligence on the other two arms of service will happen in the same way.

This is the first concrete information GCHQ has about the organisation of the Danish Army and is of great importance because we can predict from the order of battle how different elements of the Danish Army will communicate with each other. For example, there will be a top level Command network linking Army Command, 1 Bde and 2 Bde. It may well include each of the regiments listed on the Orbat (plus or minus the Guards units: are they purely ceremonial or do they have war roles?) depending on how the military is structured to transition from peacetime to war. Defence Intelligence will be developing its understanding of how the rectangles on the Orbat diagram translate into working relationships with the Danish Army and there will be an ongoing dialogue between DI and GCHQ over this. There will of course be a host of subordinate units: each regiment is likely to have battalions, and each battalion companies: but how many, and where, and what comms structures support them?

GCHQ's collection sites will have been tasked to look for Danish military comms. This is like looking for needles in a haystack, but search specialists have a range of skills and tools which make this less frightening that it might seem to the layman. Direction Finding (DF), for example, makes it easy to say that various unidentified comms aren't from Denmark, as well as giving more or less confidence that other comms are. Linguists will be preparing working aids to teach operators basic differences between Danish, Norwegian and Swedish, so that operator chat can be made use of.

Typically at each level of the hierarchy, from the top level Command network described above, to brigade or regimental networks, the search specialists will be looking for patterns: a Control station calls up subscriber stations to its network at fixed times (say morning and late afternoon) one by one. They acknowledge the call up. Control then says which stations it has messages for, and the subscribers say whether they have messages for Control. Some of the messages in both directions will be relays: for example Control asks the subscriber to forward the message to a subordinate subscriber on a subordinate network or the message might be travelling upwards.

But who is who is not very clear: each entity will have a callsign by which it is identified, and it is highly likely that the callsign will be encrypted and so will change every day. Networks at each level of the hierarchy will work in the same way, so it won't be immediately obvious whether an operator is listening to a top level or a low level. Operator chat will give clues; DF will help; analysis of the messages being relayed to see how many levels of hierarchy they pass through; information from allies will give other clues. You will notice that the content of messages doesn't need to be decrypted for this activity to be carried out. Decryption is a very-nice-to-have short cut, but most of this traffic analysis needs to be done anyway. Hopefully, after a few months, a reasonably clear idea of how the Danish Army communicates will have been developed.

But this is no more than a snapshot. Periodically, each network will change the frequencies it works on and its callsign systems. It will have different procedures for use in wartime, procedures which may or may not be the same as the procedures it uses when the units with which each subscriber is linked deploy on exercise.

But the Danish Army has been brought to the same position as the military forces of countries which GCHQ has been monitoring to 'maintain continuity'. There is an art in making sure that the minimum amount of collection resource (because nobody has ever been able to collect everything) can be applied to making sure that the maximum amount of information about the comms structures of potential targets will be produced. For example, a fortnightly check on the comms structure of a logistics battalion in northern Jutland may be enough to reassure you that nothing has changed; a monthly DF check that the callsign associated with a Headquarters element in a bunkered command post is still in situ may be enough for reassurance, but the frequency with which these checks take place has to be adequate to give adequate notice if there is any change.

The key aim of 'maintaining continuity' is ensuring that regular patterns are known and understood. No country can afford to keep its military on high alert for long, and certainly can't keep it on any sort of alert level for ever. Units have to be rotated between more or less forward roles, have to be reskilled to take on new responsibilities, have to be adapted to new roles, as well as having to respond to random events like weather, but these are all more or less predictable, and the manner in which units adapt, as seen from their comms, can give the clue to how they might adapt to future conditions: to war.

So after a few months we have a good handle on the Danish Army. We know what normality looks like. We know what it looks like when it changes its alert level. We know how its comms are likely to change when readiness levels change. We are confident that if they are ordered to war stations, we will be aware and will be able to report it, and to follow them as their comms move to wartime modes.

But none of this is intelligence. We have developed all of the sources of information available to us to be able to keep a handle on how the Danish Army is communicating. But it is only when we take that information and extrapolate from it, for example to say that a major comms change is a regular occurrence rather than an indicator that the Army is preparing for its war role, that we are producing intelligence.

I said above that the organogram showing the high level Danish Army Orbat wasn't intelligence: it's information. It assembles readily available data into readily usable information, but it doesn't answer what for a Sigint organisation – as well as the subsequent all source assessment that Sigint feeds into – is the key question: 'so what?' The same question applies to Osint: yes, finding the data and assembling it into information is important, but the 'intelligence' part of Osint should mean that it explains how the information illuminates what the target's intentions and realistic capabilities are.

All of this is environmental knowledge. Until this level of knowledge has been built up, a Sigint organisation can't begin to produce intelligence. It has to expend significant resource to maintain the environmental information that enables it to turn into an intelligence producer when a target turns active, and is no longer just watched on a 'care and maintenance' basis. And if it has been difficult and complicated with a nation state's military forces, how much more difficult is it with an individual with a telephone? And if state interception of HF signals in free space where users had 'no expectation of privacy' was once unconstrained by the law, as the state (in the UK at least) 'owned' the way in which the electromagnetic spectrum was used, how much more complex is it today to navigate all of the necessary oversight and legislative constraints to obtain and retain information which might never be used?

I suppose that what I want you to go away with is that Sigint is a much more complicated business than you might have thought, and certainly isn't about intercepting and listening to every telephone conversation between London and Washington. The art and science of Sigint happens a lot further back than the point at which intelligence reports are produced, much as a military force's achieving significant effect on the battlefield owes almost everything to what had happened in the background, over a long period of time.

One Pip Instead Of Two Toots

 

I promised another extract from Signal. This is another story about the Royal Navy's need for comms discipline, and the lengths to which they would act to protect it. This time, instead of chummy telegraphists sending each other messages on the back of official signals, we have a solitary one, onboard a vessel or at a shore station so far down the hierarchy of importance that although present every time his network controller carried out a scheduled call up of his correspondents, our poor hero never once had a signal to send. So one day …  

A nondescript nonentity, a limb of the oppressed,

I wear no badges on my arm, no medals on my chest,

But though my past is colourless, my future dim and bleak,

I cherish a distinction which is probably unique.

Of all the mass of traffic through the tortured ether hurled,

By all the busy Tels of all the navies of the world,

No morse of mine impinged upon a fellow sparker's ear;

I never sent a signal in the whole of my career.

 

I used to wonder meekly when control would let me in

To add my little quota to the universal din.

Then realised my destiny, surrendered to my fate,

Eternally to sit and serve by being told to wait.

 

But once - and only once - I found my baser self constrained

To break the wireless silence I so rigidly maintained.

My weary watch was over, my relief was overdue,

I gently, briefly, pressed the key to see what it would do.

 

I often sit and wonder where that blameless dot has gone,

If still through endless time and space it hurries bravely on,

Disowned by its creator, and dismissed its parent ship,

Unauthorised, attenuated, lonely little pip.

 

But though beyond our universe its travels may extend,

It still will bear my fingerprints on reaching journey's end

And beings in some unknown world may trace it back to me,

As surely as the Flagship did in 1943.

 

Comms Indiscipline In The Royal Navy

 

In 1993 Captain Barrie Kent RN (Retd) published Signal! A History of Signalling in the Royal Navy. It is one of my favourite books, a serious history interlaced with anecdotes varying from the poignant to the slightly scurrilous. Captain Kent was Head of RN 'Y' during the 1950s, but his book has much more to offer than cryptology in the RN. This is the first of two extracts from the book: they show how seriously the RN took comms discipline. The first is a story from 1931 about Lord Louis Mountbatten when he was Fleet Wireless Officer in the Mediterranean Sea and comes from Arthur McCulloch, a Telegraphist in 1931, but who retired eventually as a Lieutenant Commander.  

'Our transmissions were taped resulting in caustic signals to those not up to his high standard. One 'habit' Mountbatten deplored was the sending of cryptic private messages during the night watches: for example in Coventry we might receive 'INT KYE' from the operator in Bulldog, which meant 'request permission to close down for five minutes to make the cocoa?' If things were quiet the reply would be 'AF'. This procedure would be followed in turn by each ship on the destroyer frequency until all had made their kye. '

Drafted to the Royal Sovereign, McCulloch became a watchkeeper on Fleet Wave, a frequency believed to be monitored by Lord Louis.

At a conference for off-watch telegraphists in the Corradino canteen, Lord Louis said that since he had started monitoring, the efficiency of the fleet's communications had improved dramatically, but the making of private messages was to stop, "The sending of one single 'toot' on the key will constitute a private message and the culprit will be severely dealt with.'

That afternoon things were fairly quiet when McCulloch was handed a message to send to Curacao. Switching on the transmitter, he sent the signal, then reached behind his morse key to break the heavy duty switch; in the process he accidentally caught his elbow on the key, blasting out a loud 'toot' for all to hear!

'That was bad enough but the operator in Curacao forgot the forenoon's instructions and sent back a cheery 'toot toot'. With hardly a pause, out came a signal from the C-in-C

'Report name and rating of operator of the watch'.

'On the Saturday I felt I had better have a final run ashore, all sorts of dire punishments having been forecast by my messmates. While enjoying a few pints one of my pals saw a poster for the Flagship's Farewell Ball that evening; we decided to join in! The dance hall was packed with officers and men from every ship in the fleet, together with their ladies, and a great time was being had by all. Our chance to join in came with the announcement: "Take your partners for the Paul Jones."

'My third partner was obviously an officer's wife, she was a superb dancer and I said so. Just then Lord Louis danced by and gave her a big smile which she acknowledged with a discreet wave. "I see you know my Chief,” I said. "I ought to, "she replied, "I'm married to him!" "If you are one of his telegraphists" she went on, "you'll know he is furious with two of you for making 'toots' or something, In fact we've heard so much of these men that we've christened them 'Toots No. I, and 'Toots No. 2"'. I paused for a moment and then said "I'm afraid' I'm Toots No. I ", She stopped in the middle of the floor and held me at arm’s length, "I don't believe it -I'm actually dancing with 'Toots' himself! You must come and meet everyone."

'Having escorted her to her seat, I set off towards the bar. "McCulloch", Lord Louis' voice boomed behind me, "So, you're the culprit are you? What have you to say for yourself?" The ladies of his party caught up with us and Lord Louis introduced me. "Now, what have you to say for yourself?" I explained what had happened: "Bad luck really Sir, particularly coming on top of your lecture." Lady Edwina spoke up: "I'm sure he is telling the truth Dickie, don't be too hard on him," "Fortunately for you, young man, I'm familiar with the layout of the wireless gear and I can see how it could have happened. Perhaps I should give you the benefit of the doubt. I'll write to your Captain before he sees you, but remember to be more careful in future." I thanked him and shot off to tell my pals what had happened.

'In due course I found myself in front of the Captain. "Ah! I've had a letter about you. I see you had an interview with the FWO ashore and he recommends that I give you another chance. Case dismissed. Don't do it again!" Sad to say my oppo in Curacao was not so fortunate; his Commander had been at Jutland and blamed all telegraphists for allowing the German fleet to escape, so he got a month’s stoppage of leave.'

There is more to this story than meets the eye: the memory of Jutland; Lord Mountbatten's determination that RN signalling would be as good as it could be (I have written about this before (Sigint Historian: Ten avoidable problems which made the Royal Navy's encryption exploitable in 1939)); the sophistication of the RN monitoring process; and the way comms operators can undermine security without even realising that they are doing so.

Secure Speech and Insecure Speech

The story of the inadequacy of allied communications security (Comsec) at the beginning of the Second World War and its gradual improvement tends to focus on signals carrying textual messages, the transition to machine-based rather than book-based encryption systems, and eventually to on-line encipherment, enabling the 'BRUSA Circuit' which linked the UK and US, Australia, Canada, and communications centres serving major allied commands around the world on secure HF radio. It was always possible, though, to deny adversaries any chance of intercepting a message by not transmitting it on a channel accessible to them. For example, transatlantic cables, while theoretically tappable, weren't vulnerable in practice. That meant that the most sensitive material could be sent between the UK and the US without danger of enemy interception. The material would still be encrypted so that as few as possible of the people handling the traffic would see the content, but the authorities on both sides of the Atlantic could be confident that the material would not be seen. This was incredibly expensive – at busy periods a million dollars a month just to US cable companies – but uninterceptable.

But what about telephone calls? There were no secure speech systems before SIGSALY was first used in July 1943. Instead, scrambler systems were used to invert the voice signal. For calls within the UK this was a weaker analogue of the cables: scrambling would be enough to stop operators at exchanges from overhearing the content of a call even though they would know that a call was taking place. There was no doubt that the Germans would be able to 'deinvert' the signal and hear the clear speech if they had access to the telephone lines, but the UK authorities were confident that they hadn't.

The one problem was international telephony. There were no voice grade cables across the Atlantic until the mid-1950s, so international calls had to be made a) on HF which was interceptable and b) protected only by a scrambler, which under certain conditions was processable by German Sigint. Add to the mix the fact that the Prime Minister and the President valued personal contact, and weren't prepared only to communicate in writing, and the potential for significant breaches of security was very high indeed.

Three days after Mussolini was sacked by the King Victor Emmanuel, and after the Italian Government had begun secretly to negotiate armistice terms with the allies, President and Prime Minister had a conversation which led to such a breach. Here is the German report:

'At 0100 hours a radiotelephone conversation between Churchill and Roosevelt was intercepted. It concerned a proclamation by Gen Eisenhower and an imminent armistice with Italy.

Churchill: "We do not wish any armistice terms to be recommended by us until we are formally requested to do so."

Roosevelt: "That's right."

Then they discussed the matter of British prisoners of war in the hands of the Italians with regard to preventing their (the British POW's) removal to the "land of the Huns". Therefore, Churchill wanted to send a dispatch to the King of Italy. Roosevelt took it upon himself to address a statement of his own to "Emmanuel". "I don't quite know just how I'm going to do it."

This is clear proof that secret negotiations between the Anglo-Saxon powers and Italy have been under way.

The Deputy Chief of the Armed Forces Operations Staff made the following observation on this subject: There are some 60,000 British POW's in Italy. The proposals of the Commander in Chief South that these POW's be evacuated from Sardinia and southern Italy have not been acted upon. The Office of Foreign Affairs has been requested to consider the matter.'

This was almost certainly not the first the Germans knew of Italian plans to request an armistice, and, anyway, they had begun to send more troops to Italy as soon as Mussolini had been deposed, but this was, as the German report of the conversation said, clear proof that Italy was ready to change sides.

When I was GCHQ Historian I liked to show a copy of this report to visitors and ask what they thought. The reaction was always the same: the President and the Prime Minister shouldn't have been allowed to speak on insecure links that the Germans could intercept and process. If I then asked who had the authority to tell them that they weren't allowed to speak to each other there was less certainty. It would be a brave official who would attempt to stop them and few people thought that Churchill would meekly accept such advice. 'The King?' one visitor said.

My reaction – at least my first reaction – is different: why was there no secure means for Churchill and Roosevelt to speak to each other? Why did it take until July 1943 (ironically, a fortnight before this particular conversation took place) to develop and field a workable system and why did the UK have to adopt a US system?

I've written enough about the fact that GC&CS didn't take Comsec seriously enough to explain this in part, but I think it's also the case that securing voice communications, which has to be done on-line, was too difficult, and was therefore left to one side until the UK heard that the US was working on a solution. At just about the same time as Alan Turing visited the Bell laboratories and was briefed on the progress of SIGSALY Tommy Flowers was proposing an entirely new sort of machine, Colossus – what would eventually become the computer – to solve the biggest cryptanalytical problem facing GC&CS, and the GPO effort required to make that work probably precluded similar investment in secure speech as well.

The British answer was to copy the US system, but a project (BANGLE) which began in 1944 and which aimed to build 20 machines, based on SIGSALY but miniaturised sufficiently to be used from a 4-ton truck, was unsuccessful and was eventually abandoned in 1953. PICKWICK at the end of the 1950s was the first entirely British system.

A Valuable New Book on Second World War Communications Security

 

Anybody who reads this blog will know of my interest in cryptography: the opposite side of the coin to cryptanalysis. How poor UK cryptography was until the middle of the Second World War, and why that should be the case is something I have written about several times.

Today, however, I want to introduce a new book about communications security, mainly from the German side. How secure did the Germans think Enigma really was? Did they really believe that the allies were not breaking their encryption?

Dermot Turing's lates book, Enigma Traitors: The Struggle to Lose the Cipher War addresses these questions. He has trawled through the reports of interrogations of German cryptographers in the postwar TICOM series as well as material in German, American and British archives to come up with a wealth of evidence to support the contention that the potential vulnerability of Enigma to a concerted machine-based attack was well known to German cryptographers. 

Why they continued to use the machine is explained by a number of factors: the inability to envisage the amount of technological resource the US and UK would be prepared to put into the attack; how difficult it was to accept that a system in whose security you have invested so much might in fact not be so secure; and the sheer impossibility of replacing over 30,000 Enigma machines in wartime by something better.

Some of this ground has been trodden previously, and R A Ratcliff's Delusions of Intelligence is still the leading work looking at a strategic level at the consequences of allied and German cryptologic policy during the Second World War. This book is very much bottom up, and focuses on the individuals who were involved, their doubts, their blind spots and their successes.

It isn't only about German cryptography either. It looks at the the insecurity of the codes used by the Royal Navy, and examines in more detail than I have seen elsewhere the claims that the Germans may have read Typex. However, a more detailed look at allied - British, at least - will have to wait until more material has been released.

This isn't so much a review as a recommendation. There really hasn't been enough  research into Second World War communications security and this books brings together so much that either has been lying unread in various archives, or which has been cited for a particular purpose, outside of the context of communications security policy, that it would be odd not to recommend it to anybody interested in the subject.

There is a tendency to think that cybersecurity is a completely new discipline, something which has nothing to learn from the past. While that might be true technologically, the way that humans think about security, and the way in which they persuade themselves that things are secure in spite of evidence that they might not be, suggests that research into the history of security might shed as much light on today's circumstances as the history of  intelligence has. This book illuminates the present as well as the past.

Factors affecting the Use of Sigint

Cdr Alexander Mackay Scobie “Mack” Mackenzie RNVR was a senior member of Naval Section at Bletchley Park for most of the Second World War. He led TICOM Team 4 for a few months in Germany in 1945 and returned to GC&CS/GCHQ to be part of the History Section staying on to the early 1950s (at least). More than some other members of the section, he was less keen simply to document what had happened at Bletchley Park and why, as to draw out lessons that might be applicable to the National Sigint Organisation during some future conflict.

What follows is a short extract from the first chapter of his classified volume (HW 43/61) about the use of Sigint by the Army and Air Force (Volume XVIII of Army and Air Force Sigint.) I’ve chosen it as it makes several timeless points: the limitations of Sigint; how Sigint should be assessed (he uses the verb ‘appreciate’ where we would say ‘assess’) and by whom; the need for those receiving it at commands to know that the secret intelligence they were reading came from intercepted communications; and the limitations imposed by the strict security regime in place. It is an antidote to the view that Sigint was (and is) produced by an organisation able to hoover up, process, understand and disseminate all of the communications of an adversary.

CATEGORIES OF SPECIAL INTELLIGENCE

The three main categories of Special Intelligence relating to the enemy armies and air forces that became available have been classified as "background", "strategical" and "tactical" (or "operational"). The term "background intelligence" was used to cover the information about the outline and detail of the whole enemy war-making machinery which accumulated with the inflow of decrypts from day to day, and was not only the basis for informed judgment of what might be expected of the enemy in any given set of circumstances, but also the main source of interpretation of new decrypts as they became available. These new decrypts, when they appeared, took their place in the store of background intelligence and at the same time, as productive of either strategic or tactical information, might fall into one or both of the other categories. Strategic Special Intelligence fed the Ministries and planning staffs with the enemy's long-term plans and capabilities. Tactical Special Intelligence fed the commands in the field with the enemy's intentions and provided current reports of his operations. Any one decrypt might be productive of Special Intelligence in any or all of these categories, but there were few that did not make some part of background intelligence, since, in the construction of the complicated pattern of the enemy's organisation for and behaviour in war, every item of information was of value.

FACTORS AFFECTING PRODUCTION

Special Intelligence has been defined and divided into the categories in which it became available; now, consideration must be given to the factors affecting the production of the basic decrypt material. For any high-echelon decrypts to be produced at all, the enemy had to use his W/T services, and these communications had then to be intercepted by the Allies. This problem of interception was solved to a greater or lesser degree in relation to the positions of the original transmission of the signals and the strength of those transmissions. Owing to the enemy's possession of the interior line in Europe, transmissions of army and air force communications were not particularly suited to interception by the Western Allies holding part of an outer ring, and this produced the situation in which decryption could never provide complete access to the enemy's communications relating to the war on land and in the air, or to any part of it. The problem of interception therefore produced an unevenness, detrimental to intelligence, in the cover of the enemy's communications which was exaggerated by the next factor, the cryptanalytic problem. Here the question was further complicated by the fact that with cryptanalysis it was not a matter of enemy communications being readable or unreadable in the way that they could or could not be intercepted. The success of cryptanalysis was, to some extent, conditional. That is to say that, given time, certain enemy cyphers yielded to cryptanalysis, while others yielded to increase in cryptanalytic effort. This implies, of course, a system of priorities for cryptanalytic effort, and such a system was in force in the Sigint centre throughout the war; but, in a situation where cryptanalytic machinery was normally in short supply, a concentration of cryptanalytic effort to produce materially more information on any given area or subject would be certain to produce a corresponding weakness elsewhere. At all times a balance had to be struck between the needs of Service Ministries, which wanted everything they could get about the enemy armed forces as a whole, and the requirements of commands in the field, who wanted everything they could get about their own areas and matters of strategic importance elsewhere.

The third factor affecting the production of Special Intelligence was one of interpretation, which could only be solved by the application of intelligence processes to such decrypts as became available of such messages as had been intercepted. The object in the application of these processes was to convert the decrypt into a form in which it could be appreciated by an intelligence officer outside the Sigint centre. The decrypted text of the enemy signal in the original language had first to be emended into reasonable German, Italian, Japanese or other language of origin by the removal, so far as was possible, of corruptions, and by the expansion of abbreviations or conventional signalese. When this had been done the original text had then to be translated into English (usually into technical or Service English, involving knowledge of the meanings of a large special vocabulary), and when finally the English text was available there still remained the problems created by the enemy systems for disguising originators, addressees, positions or intentions, or proforma introduced for convenience in signalling.

Thus, the access to the enemy's communications that was provided in Special Intelligence was access only to such of his high-echelon WT communications as could be intercepted, decrypted and interpreted. While the comprehensiveness and sheer bulk of Special Intelligence might give an impression of complete cover, and while its accuracy and reliability might give it a further use in the evaluation, interpretation and direction of intelligence from other sources, the intelligence officer had at all times to bear in mind its actual incompleteness. It was never safe to discount evidence from other sources or overlook possibilities on the ground that, since there was nothing in Special Intelligence relating to a possibility, it was not worth entertaining.

FACTORS AFFECTING USE

Such Special Intelligence as became available, therefore, provided a literal English version – or as literal as was possible in the circumstances – of what the enemy was saying in his communications and such interpretation of his statements as was necessary to make his meaning clear; and at this point, in theory, the responsibility of the Sigint centre in the provision of Special Intelligence ended. The product was available in its finished form, and that form was not likely to be improved, except through a later amplification of the information it contained.

It remains to consider the factors affecting the use of Special Intelligence after it became available. The first of these factors was the need for its appreciation as evidence of the enemy's capacity or his intentions in conjunction with evidence from other sources. Once the Special Intelligence was available, there was no doubt as to what certain enemy authorities were saying to each other, it remained to discover what this implied – that is to say that although, from the point of view of Allied Intelligence, what the enemy said was of great value, the real worth of Special Intelligence lay in what the enemy's signals implied.

This appreciation of Special Intelligence was, in theory, the business of the intelligence staffs in the Ministries and at commands, it was specifically not the business of the Sigint centre, but such was the nature of the material that degrees of appreciation went on at all stages of interpretation, appreciation and use, in that the better the intelligence officers handling Special Intelligence, the more information they wrung from the decrypt while it was in their hands. Decrypts of course possessed an almost infinite variation in the degree of intelligence that might be hidden behind their literal translation, but apart from hidden implication and in spite of reliability, comprehensiveness and currency, Special Intelligence was liable to two main types of error. The intelligence officer had to ask himself two questions. First, was the originator of the signal stating the facts? And, secondly, what was the true significance of the signal as decrypted? In reaching a decision on the first question the intelligence officer had to bear in mind that the originator of the signal might have been ignorant of the facts or distorting them for his own purposes, so that a decrypt might be factually wrong or deliberately misleading. It could be wrong, for example, when an air force liaison officer was reporting on army formations, it could be misleading when an anti-aircraft battery, reporting "no damage” after an air attack, was reporting the condition of the battery rather than the state of the target. As to the significance of the decrypt, an intercepted signal might only be a part of the whole message or it might be confused with jargon or cover-names to the point of being meaningless; or, and this was the case with the greater part of the Special Intelligence that became available, the text in itself might be unimportant or apparently routine, an isolated scrap of information for which an operational or intelligence context had to be built up before it assumed any significance at all. This problem of interpretation was dealt with by the intelligence officer attaching such comment to the signal as he considered necessary for its use in the next stage – its appreciation either at the Ministries or at commands.

Dissemination was the next factor affecting use. Once the best possible sense had been made of the Special Intelligence available, it had to be pushed out, some part of it to the Ministries and some part to commands. Throughout the war the handicap imposed by communications on the use of Special Intelligence was gradually reduced from an absolute prohibition in the Norwegian campaign of 1940 to a situation where, in 1944, the Ministries and certain headquarters in Europe were being fed continuously by teleprinter, and other commands, up to fifty or so in all, through the SLU/SCU organisation. To cover the reduction of this handicap in a few words, it can be said that in the Norwegian campaign the comparatively small amount of Special Intelligence available could get no farther forward than the Ministries, as no means of communication capable of carrying material of such secrecy existed between the Ministries and commands in Norway. In the campaign in France in 1940 Special Intelligence could be got as far forward as British GHQ and AHQ and French GQG, but only in a disguise calculated to give the impression that it was a series of reports from agents. As a result, and in the general confusion of the campaign, it was never used effectively. In the Balkan campaign of 1941 Special Intelligence reached as far forward as the British GOC and AOC in Greece in the form of appreciations, and by now recipients were aware of the nature of the source and had begun to benefit from it. It was considered, in the case of GOC Crete that, in his exposed position, Special Intelligence could only be provided under its Secret Service disguise, and this was done on the Prime Minister's decision. It was not until the campaigns in the Western Desert, however, that Special Intelligence began to reach in an organised manner the commands organised to use it, and not until the Battle of Alamein that, in the words of Brigadier Williams, later BGS(I) 21st Army Group (who subsequently wrote a history of the Army’s use of Ultra), Special Intelligence put Army Intelligence on the map and "henceforward we were going to use it".

By the date of the first of the combined landing operations in the Mediterranean (November 1942), the arrangements for the supply of Special Intelligence to commands in the field were being laid on as part of the normal intelligence requirement; and with successive landings in Sicily and Italy the dissemination of Special Intelligence developed into the organised services provided for the landing in Normandy in 1944.

The limitation on the use of Special Intelligence in the field imposed by the need for security was, of course, a part and a cause of the difficulty in the provision of adequate communications; that is to say, that dissemination to a prospective user of Special Intelligence could be as effectively restricted by the danger to the security of the source as by the lack of adequate communications, and the one, the need for security, might be the cause of the other. By 1944 the regulations governing the security of Special Intelligence covered some eight pages of foolscap, and were the result of several years of trial and error in the handling of the material during which it had been discovered that the value of the source was so great, and access to it could have been terminated so certainly by the enemy, that opportunity for effective action that might however have exposed its true nature had constantly to be denied to commands in the field. As an example, no direct action on Special Intelligence was permitted unless there was a possibility, which the enemy would consider reasonable, that the information might have become available from lower grade Sigint or non-Sigint sources. Of course, specific action could be taken to provide such camouflage, as, for instance, in the case of the development of the Allied sea and air offensive against the merchant shipping that carried military supplies to the Axis forces in North Africa. Very full information on the shipping engaged on this traffic was available from several sources, but the cargoes carried in individual ships and the exact routes they would follow were known only from decrypts; other Special Intelligence at the same time showed what commodities – fuel was usually the chief of these – were in short supply, and as a result the Royal Navy and RAF were able to devote attention primarily to those ships whose loss would do the enemy most harm. In order to overcome the suspicion that might be aroused in the mind of the enemy through the continued success of this policy of selection, it became the practice to fly aerial reconnaissance to spot ships whose course or location at a given time had been revealed in Special Intelligence. Once the enemy had been made aware of this reconnaissance the information provided in Special Intelligence as to which ships would make the most profitable target could be used to the full.

That these precautions were effective is shown by the fact that time and again the right ships were sunk, and at no time did the enemy consider that his cyphering systems were vulnerable; but the necessary restriction on the use of Special Intelligence was severe and caused many practical difficulties. Apart from the actual prohibition of use because the Special Intelligence was not covered by information from a more open source, there was, certainly up to the end of 1942, the danger that the disguise of the source as an agent would have the effect of reducing its reliability in the eyes of the recipient and so make him less inclined to take the information provided at its actual value. Moreover, as Brigadier Williams discovered, "once you began to pretend that it was an agent, not only was the story highly unconvincing to those who bothered to think about it (there were surprisingly few who did) but it entailed a lessening of security in discussing it".

The need for security therefore placed a double restriction on the use of Special Intelligence in operations. Not only could disguise adopted for security purposes lower the worth of the information in the eyes of the prospective user, but the lack of a possible open source of the information might prohibit its use altogether.

 

Postwar Development of New Cipher Machines in the UK

Typex Mark 22

Although there were a few other encryption machines available to the UK at the end of the Second World War (eg Rockex, Morsex and 5 UCO), Typex and the CCM were the only general purpose off-line cypher machines available to the British Services. A new machine was needed, though it was clear that this would take some time – years – to be delivered and so modifications were made to the Mark II Typex (the model in most extensive use) and it was rechristened the Mark 22, or Mark 23 when adapted to take a CCM adaptor. 

At GC&CS Gordon Welchman had thought about a successor to Typex and had devised a rotor scrambler both to replace it and to be field-deployable. A key security feature was a daily change carried out using multi-point plugs, but in January 1945 Alan Turing's investigations showed this to be practically impossible. Welchman therefore decided to dispense with plugboards and found a solution, which he embodied in a paper of 6th March 1945, describing his final specifications for a 26-way scrambler - the RM 26.

A Canadian Army officer, Lt Col B de F "Pat" Bayly had developed Rockex, a one-time tape machine during the war, and thought a small and light machine embodying Welchman's scrambler could be made for use in the field.  In November 1945 he signed a five year contract with the Foreign Office to develop such a machine from his base in Canada. Bayly was told in 1946 that there was no idea of tying him down to a specification, because the whole object of the contract was to enable him to do some free research into the electromechanical application of the RM 26 principles.

In August 1947, a further letter requested a technical write up of progress: the Services were looking on RM 26 with interest, and had asked how Bayly might react to a US development engineer's being seconded to work with him. Bayly replied promptly, saying that he had completed his progress report, and reminded GCHQ that he had been told that this was a long-term project. As there were so many innovations and as considerable improvement in reliability would be necessary before this machine would be useful, Bayly did not believe the machine would be ready to show the Americans for another three or four years. (He also believed it would be a bad thing to attach an American to him, as the result might be the development of a purely American machine incorporating RM 26 ideas.) Further delays were incurred due to a serious illness suffered by Mrs Bayly.

After a couple of visits by British specialists to the Canada, Bayly was asked to send the model he had made to the United Kingdom no later than the end of August 1948, together with a detailed statement setting out what items in the project had been incorporated in the model, what had been found impracticable, and what had not been attempted. British experts concluded that the RM 26 proposals used unnecessarily over-complicated mechanical arrangements and that conventional engineering was preferred; and that the project was not worth while pursuing unless there was a specific operational need for a machine operating on RM 26 principles. The experimental model offered little prospect of providing a successful solution. As a result of this report the project was cancelled.

RM 26 is a good example of how not to organise a project. The aim was changed twice and lost clarity in the process; direction was faulty and supervision inadequate; resources were lacking for proper research and development; and both time and money were wasted through delay in investigating the practicability of the design. It probably didn't help either that the work was being carried out in Canada. In terms of development of a Typex replacement, these years of development had been wasted.

Meanwhile, a member of GCHQ had spoken about the need for a small, secure cipher machine with a tape output to the GPO, which produced a draft specification (codenamed Fruitex Minor) of a battery-operated machine, which the services accepted in December 1947 as a basis for design effort, though with the more prosaic name of DUP 1 (Development Unit Project No 1). This quickly ran into trouble. Its specification proved to be over-fussy: if a reliable machine were ever produced from it, it would be too delicate for use in forward areas, and it would be impossible to mass produce. A new specification was drawn up, and subject to security endorsement which was given in December 1949, was offered as a basis for contract discussions. Early in 1950, the Creed company agreed that if a contract were signed by September 1950, it would deliver 5 units a week starting in September 1951, and that 150 units would be delivered by March 1952. This was subject to the prototype passing service acceptance trials and the specification not changing subsequently. In the event, the trials overran, and the specification was developed into four variants, under the overall codename of Portex, delaying production further.

In exchange for an offer from the US for the UK to inspect a pinwheel machine designed for use by merchant shipping, the US authorities were sent the details of Portex. However, in May 1952, a US security evaluation for Portex was issued, saying that it was too weak for its use to be allowed. It turned out that the security approval given in December 1949 should not have been granted, and more design work was need to correct the errors. In the event it was only in September 1954 that production of Portex began, just under seven years from its being proposed.

The specification for a new Typex replacement specification (DUP 2) was only agreed in January 1949. The basic requirements were:

(a) it must be suitable for UK/US communications, which entailed a 26-way keyboard and scrambler;

(b) since many ships in the Fleet could only carry one machine, it would be required for intra-Naval communications, as well as for communications between the Navy and the other two Services;

(c) for Naval use size and weight should be kept to a minimum, and for ease of maintenance it should be mechanically and electrically simple;

(d) in the Army and RAF it should provide a satisfactory replacement for Typex 22, and it should be suitable for use at higher formation headquarters, where page printing and 32-way operation, needed for the rapid handling of traffic, were more important than limitations on size and weight.

It was clear that the small machine that could satisfy (a), (b), and (c), was a different machine from (d). Work on the former began under the codename Singlet, while a new specification was to be drawn up for the second (codenamed Pendragon). This work was delayed because the US was designing a new CCM and there weren't enough British specialists to work on the two projects simultaneously.

A prototype Singlet was completed in 1951 according to the 1949 specification, but the specification was modified in 1951 and a new prototype had to be developed. A prototype Pendragon had been built, but no subsequent modification was carried out. By October 1952, the Service requirements for Pendragon had changed sufficiently that it was decided that while design work would continue on both Singlet and Pendragon, for delivery as soon as possible, planning for a successor to both of them, with an expected delivery date of some five to seven years, should begin under the codename Copperfield.

In the event Singlet deliveries began in 1959, ten years after its first specification was issued, while the requirements for both Pendragon and Copperfield were subsumed into Alvis (which in its first incarnation had been codenamed Cheapex).

This piece complements other posts here about Sir Edward Bridges' thinking about whether the security mission belonged inside GCHQ. None of the machines being used at the end of the war had been designed by GC&CS (indeed, in the case of Typex, GC&CS input had been sedulously avoided) and the project management of RM 26, Portex and Singlet was so deficient, that it is probably not hard to see members of the Cypher Policy Board dreading the agenda item 'Machine Development' at their meetings.

There was a similar lack of development of machinery to support cryptanalysis at GCHQ during this period, though this was because the machines which were moved from Bletchley to Eastcote, and subsequently to Cheltenham, were believed adequate to the tasks at hand, and it was only the development of General Purpose Computers later in the 1950s that began to change ideas about how computers could be used.

There was a requirement for better cipher security after the Second World War, but, separate from the status of communications security in a primarily signals intelligence organisation, the Services' grip on Comsec was much greater by 1946 than it was on Sigint, and the Cypher Policy Board, which was where GCHQ and the Services met to collaborate on security, operated at too high a level to be able to provide the detailed oversight of development activity.