A Valuable New Book on Second World War Communications Security

 

Anybody who reads this blog will know of my interest in cryptography: the opposite side of the coin to cryptanalysis. How poor UK cryptography was until the middle of the Second World War, and why that should be the case is something I have written about several times.

Today, however, I want to introduce a new book about communications security, mainly from the German side. How secure did the Germans think Enigma really was? Did they really believe that the allies were not breaking their encryption?

Dermot Turing's lates book, Enigma Traitors: The Struggle to Lose the Cipher War addresses these questions. He has trawled through the reports of interrogations of German cryptographers in the postwar TICOM series as well as material in German, American and British archives to come up with a wealth of evidence to support the contention that the potential vulnerability of Enigma to a concerted machine-based attack was well known to German cryptographers. 

Why they continued to use the machine is explained by a number of factors: the inability to envisage the amount of technological resource the US and UK would be prepared to put into the attack; how difficult it was to accept that a system in whose security you have invested so much might in fact not be so secure; and the sheer impossibility of replacing over 30,000 Enigma machines in wartime by something better.

Some of this ground has been trodden previously, and R A Ratcliff's Delusions of Intelligence is still the leading work looking at a strategic level at the consequences of allied and German cryptologic policy during the Second World War. This book is very much bottom up, and focuses on the individuals who were involved, their doubts, their blind spots and their successes.

It isn't only about German cryptography either. It looks at the the insecurity of the codes used by the Royal Navy, and examines in more detail than I have seen elsewhere the claims that the Germans may have read Typex. However, a more detailed look at allied - British, at least - will have to wait until more material has been released.

This isn't so much a review as a recommendation. There really hasn't been enough  research into Second World War communications security and this books brings together so much that either has been lying unread in various archives, or which has been cited for a particular purpose, outside of the context of communications security policy, that it would be odd not to recommend it to anybody interested in the subject.

There is a tendency to think that cybersecurity is a completely new discipline, something which has nothing to learn from the past. While that might be true technologically, the way that humans think about security, and the way in which they persuade themselves that things are secure in spite of evidence that they might not be, suggests that research into the history of security might shed as much light on today's circumstances as the history of  intelligence has. This book illuminates the present as well as the past.

Factors affecting the Use of Sigint

Cdr Alexander Mackay Scobie “Mack” Mackenzie RNVR was a senior member of Naval Section at Bletchley Park for most of the Second World War. He led TICOM Team 4 for a few months in Germany in 1945 and returned to GC&CS/GCHQ to be part of the History Section staying on to the early 1950s (at least). More than some other members of the section, he was less keen simply to document what had happened at Bletchley Park and why, as to draw out lessons that might be applicable to the National Sigint Organisation during some future conflict.

What follows is a short extract from the first chapter of his classified volume (HW 43/61) about the use of Sigint by the Army and Air Force (Volume XVIII of Army and Air Force Sigint.) I’ve chosen it as it makes several timeless points: the limitations of Sigint; how Sigint should be assessed (he uses the verb ‘appreciate’ where we would say ‘assess’) and by whom; the need for those receiving it at commands to know that the secret intelligence they were reading came from intercepted communications; and the limitations imposed by the strict security regime in place. It is an antidote to the view that Sigint was (and is) produced by an organisation able to hoover up, process, understand and disseminate all of the communications of an adversary.

CATEGORIES OF SPECIAL INTELLIGENCE

The three main categories of Special Intelligence relating to the enemy armies and air forces that became available have been classified as "background", "strategical" and "tactical" (or "operational"). The term "background intelligence" was used to cover the information about the outline and detail of the whole enemy war-making machinery which accumulated with the inflow of decrypts from day to day, and was not only the basis for informed judgment of what might be expected of the enemy in any given set of circumstances, but also the main source of interpretation of new decrypts as they became available. These new decrypts, when they appeared, took their place in the store of background intelligence and at the same time, as productive of either strategic or tactical information, might fall into one or both of the other categories. Strategic Special Intelligence fed the Ministries and planning staffs with the enemy's long-term plans and capabilities. Tactical Special Intelligence fed the commands in the field with the enemy's intentions and provided current reports of his operations. Any one decrypt might be productive of Special Intelligence in any or all of these categories, but there were few that did not make some part of background intelligence, since, in the construction of the complicated pattern of the enemy's organisation for and behaviour in war, every item of information was of value.

FACTORS AFFECTING PRODUCTION

Special Intelligence has been defined and divided into the categories in which it became available; now, consideration must be given to the factors affecting the production of the basic decrypt material. For any high-echelon decrypts to be produced at all, the enemy had to use his W/T services, and these communications had then to be intercepted by the Allies. This problem of interception was solved to a greater or lesser degree in relation to the positions of the original transmission of the signals and the strength of those transmissions. Owing to the enemy's possession of the interior line in Europe, transmissions of army and air force communications were not particularly suited to interception by the Western Allies holding part of an outer ring, and this produced the situation in which decryption could never provide complete access to the enemy's communications relating to the war on land and in the air, or to any part of it. The problem of interception therefore produced an unevenness, detrimental to intelligence, in the cover of the enemy's communications which was exaggerated by the next factor, the cryptanalytic problem. Here the question was further complicated by the fact that with cryptanalysis it was not a matter of enemy communications being readable or unreadable in the way that they could or could not be intercepted. The success of cryptanalysis was, to some extent, conditional. That is to say that, given time, certain enemy cyphers yielded to cryptanalysis, while others yielded to increase in cryptanalytic effort. This implies, of course, a system of priorities for cryptanalytic effort, and such a system was in force in the Sigint centre throughout the war; but, in a situation where cryptanalytic machinery was normally in short supply, a concentration of cryptanalytic effort to produce materially more information on any given area or subject would be certain to produce a corresponding weakness elsewhere. At all times a balance had to be struck between the needs of Service Ministries, which wanted everything they could get about the enemy armed forces as a whole, and the requirements of commands in the field, who wanted everything they could get about their own areas and matters of strategic importance elsewhere.

The third factor affecting the production of Special Intelligence was one of interpretation, which could only be solved by the application of intelligence processes to such decrypts as became available of such messages as had been intercepted. The object in the application of these processes was to convert the decrypt into a form in which it could be appreciated by an intelligence officer outside the Sigint centre. The decrypted text of the enemy signal in the original language had first to be emended into reasonable German, Italian, Japanese or other language of origin by the removal, so far as was possible, of corruptions, and by the expansion of abbreviations or conventional signalese. When this had been done the original text had then to be translated into English (usually into technical or Service English, involving knowledge of the meanings of a large special vocabulary), and when finally the English text was available there still remained the problems created by the enemy systems for disguising originators, addressees, positions or intentions, or proforma introduced for convenience in signalling.

Thus, the access to the enemy's communications that was provided in Special Intelligence was access only to such of his high-echelon WT communications as could be intercepted, decrypted and interpreted. While the comprehensiveness and sheer bulk of Special Intelligence might give an impression of complete cover, and while its accuracy and reliability might give it a further use in the evaluation, interpretation and direction of intelligence from other sources, the intelligence officer had at all times to bear in mind its actual incompleteness. It was never safe to discount evidence from other sources or overlook possibilities on the ground that, since there was nothing in Special Intelligence relating to a possibility, it was not worth entertaining.

FACTORS AFFECTING USE

Such Special Intelligence as became available, therefore, provided a literal English version – or as literal as was possible in the circumstances – of what the enemy was saying in his communications and such interpretation of his statements as was necessary to make his meaning clear; and at this point, in theory, the responsibility of the Sigint centre in the provision of Special Intelligence ended. The product was available in its finished form, and that form was not likely to be improved, except through a later amplification of the information it contained.

It remains to consider the factors affecting the use of Special Intelligence after it became available. The first of these factors was the need for its appreciation as evidence of the enemy's capacity or his intentions in conjunction with evidence from other sources. Once the Special Intelligence was available, there was no doubt as to what certain enemy authorities were saying to each other, it remained to discover what this implied – that is to say that although, from the point of view of Allied Intelligence, what the enemy said was of great value, the real worth of Special Intelligence lay in what the enemy's signals implied.

This appreciation of Special Intelligence was, in theory, the business of the intelligence staffs in the Ministries and at commands, it was specifically not the business of the Sigint centre, but such was the nature of the material that degrees of appreciation went on at all stages of interpretation, appreciation and use, in that the better the intelligence officers handling Special Intelligence, the more information they wrung from the decrypt while it was in their hands. Decrypts of course possessed an almost infinite variation in the degree of intelligence that might be hidden behind their literal translation, but apart from hidden implication and in spite of reliability, comprehensiveness and currency, Special Intelligence was liable to two main types of error. The intelligence officer had to ask himself two questions. First, was the originator of the signal stating the facts? And, secondly, what was the true significance of the signal as decrypted? In reaching a decision on the first question the intelligence officer had to bear in mind that the originator of the signal might have been ignorant of the facts or distorting them for his own purposes, so that a decrypt might be factually wrong or deliberately misleading. It could be wrong, for example, when an air force liaison officer was reporting on army formations, it could be misleading when an anti-aircraft battery, reporting "no damage” after an air attack, was reporting the condition of the battery rather than the state of the target. As to the significance of the decrypt, an intercepted signal might only be a part of the whole message or it might be confused with jargon or cover-names to the point of being meaningless; or, and this was the case with the greater part of the Special Intelligence that became available, the text in itself might be unimportant or apparently routine, an isolated scrap of information for which an operational or intelligence context had to be built up before it assumed any significance at all. This problem of interpretation was dealt with by the intelligence officer attaching such comment to the signal as he considered necessary for its use in the next stage – its appreciation either at the Ministries or at commands.

Dissemination was the next factor affecting use. Once the best possible sense had been made of the Special Intelligence available, it had to be pushed out, some part of it to the Ministries and some part to commands. Throughout the war the handicap imposed by communications on the use of Special Intelligence was gradually reduced from an absolute prohibition in the Norwegian campaign of 1940 to a situation where, in 1944, the Ministries and certain headquarters in Europe were being fed continuously by teleprinter, and other commands, up to fifty or so in all, through the SLU/SCU organisation. To cover the reduction of this handicap in a few words, it can be said that in the Norwegian campaign the comparatively small amount of Special Intelligence available could get no farther forward than the Ministries, as no means of communication capable of carrying material of such secrecy existed between the Ministries and commands in Norway. In the campaign in France in 1940 Special Intelligence could be got as far forward as British GHQ and AHQ and French GQG, but only in a disguise calculated to give the impression that it was a series of reports from agents. As a result, and in the general confusion of the campaign, it was never used effectively. In the Balkan campaign of 1941 Special Intelligence reached as far forward as the British GOC and AOC in Greece in the form of appreciations, and by now recipients were aware of the nature of the source and had begun to benefit from it. It was considered, in the case of GOC Crete that, in his exposed position, Special Intelligence could only be provided under its Secret Service disguise, and this was done on the Prime Minister's decision. It was not until the campaigns in the Western Desert, however, that Special Intelligence began to reach in an organised manner the commands organised to use it, and not until the Battle of Alamein that, in the words of Brigadier Williams, later BGS(I) 21st Army Group (who subsequently wrote a history of the Army’s use of Ultra), Special Intelligence put Army Intelligence on the map and "henceforward we were going to use it".

By the date of the first of the combined landing operations in the Mediterranean (November 1942), the arrangements for the supply of Special Intelligence to commands in the field were being laid on as part of the normal intelligence requirement; and with successive landings in Sicily and Italy the dissemination of Special Intelligence developed into the organised services provided for the landing in Normandy in 1944.

The limitation on the use of Special Intelligence in the field imposed by the need for security was, of course, a part and a cause of the difficulty in the provision of adequate communications; that is to say, that dissemination to a prospective user of Special Intelligence could be as effectively restricted by the danger to the security of the source as by the lack of adequate communications, and the one, the need for security, might be the cause of the other. By 1944 the regulations governing the security of Special Intelligence covered some eight pages of foolscap, and were the result of several years of trial and error in the handling of the material during which it had been discovered that the value of the source was so great, and access to it could have been terminated so certainly by the enemy, that opportunity for effective action that might however have exposed its true nature had constantly to be denied to commands in the field. As an example, no direct action on Special Intelligence was permitted unless there was a possibility, which the enemy would consider reasonable, that the information might have become available from lower grade Sigint or non-Sigint sources. Of course, specific action could be taken to provide such camouflage, as, for instance, in the case of the development of the Allied sea and air offensive against the merchant shipping that carried military supplies to the Axis forces in North Africa. Very full information on the shipping engaged on this traffic was available from several sources, but the cargoes carried in individual ships and the exact routes they would follow were known only from decrypts; other Special Intelligence at the same time showed what commodities – fuel was usually the chief of these – were in short supply, and as a result the Royal Navy and RAF were able to devote attention primarily to those ships whose loss would do the enemy most harm. In order to overcome the suspicion that might be aroused in the mind of the enemy through the continued success of this policy of selection, it became the practice to fly aerial reconnaissance to spot ships whose course or location at a given time had been revealed in Special Intelligence. Once the enemy had been made aware of this reconnaissance the information provided in Special Intelligence as to which ships would make the most profitable target could be used to the full.

That these precautions were effective is shown by the fact that time and again the right ships were sunk, and at no time did the enemy consider that his cyphering systems were vulnerable; but the necessary restriction on the use of Special Intelligence was severe and caused many practical difficulties. Apart from the actual prohibition of use because the Special Intelligence was not covered by information from a more open source, there was, certainly up to the end of 1942, the danger that the disguise of the source as an agent would have the effect of reducing its reliability in the eyes of the recipient and so make him less inclined to take the information provided at its actual value. Moreover, as Brigadier Williams discovered, "once you began to pretend that it was an agent, not only was the story highly unconvincing to those who bothered to think about it (there were surprisingly few who did) but it entailed a lessening of security in discussing it".

The need for security therefore placed a double restriction on the use of Special Intelligence in operations. Not only could disguise adopted for security purposes lower the worth of the information in the eyes of the prospective user, but the lack of a possible open source of the information might prohibit its use altogether.