The Yangtze Incident And A Do-It-Yourself One Time Pad

The story of HMS Amethyst and the Yangtze Incident used to be well known, but is less so today. Briefly, during the Chinese Civil War HMS Amethyst was sailing up the Yangtze from Shanghai to Nanking to provide support to the British Embassy to the Chinese Nationalist government which was temporarily based there. The British frigate was fired on by Chinese communist forces and severely damaged, and was forced to anchor for ten weeks, every attempt to move provoking renewed fire from the communist batteries. On the night of 30 July 1949 Amethyst made a daring and successful 100 mile dash down the Yangtze and rejoined the fleet. there is a little known cryptographic footnote to this story. 

After being attacked, HMS Amethyst had prepared to scuttle herself and had destroyed all crypto materials. From comments made by the local communist commander during protracted negotiations, it was clear that his forces were intercepting the plain language traffic between Amethyst and CinC Far Eastern Station, so how could Amethyst and the CinC plan her escape?

The Chief Naval Signal Officer Far East (Cdr C R Williams RN) came up with a plan to create a One Time Pad (OTP) which would then be used to re-encipher message sent using the Government Telegraph Code, a copy of which was held by Amethyst but which was also known to be in the possession of the communists.

The OTP was created as follows: in a first column, the surnames of all RN officers and men on board were listed in alphabetical order.

Column two listed the first name of the next of kin of each of the people in column one.

Column three listed the placename in the address of the people in column two as given by the person in column one.

Column one was now discarded.

Working down column two, letters were converted into four figure groups – so JOAN became 1015 0114. Groups were written across the page, with five four figure groups on each line. Once column two was complete, the same process was used with column three.

This pad was only used for one message. In it, Amethyst was told the times, frequencies and indicator groups of a series of dummy messages which would be addressed to CinC FES on the Fleet broadcast. Amethyst should copy these messages and use the groups transmitted to make 'In' and 'Out' pads. She was also told how to use the pads without a codebook.

During the afternoon of 30 July Amethyst sent a Flash signal: "DTG 300657Z July. Top Secret. CinC info Concord from Amethyst. I am going to try and break out two two zero zero item repeat two two zero zero item tonight three zero July Concord set watch eight two nine zero." (HMS Concord was a destroyer sent up the Yangtze to support Amethyst's escape.)

A lot of people still think that the security that OTPs offer is based on their randomness: in fact it's based on their unpredictability. In this context, the Chinese communists' knowing how the single use OTP was constructed introduced significant weaknesses. Apart from the fact than an agent in London might have been able to access the list of crew and their next of kin, column three was likely to have a disproportionate number of repeats of "Portsmouth", "Plymouth" and "London". But even if they had been able to do break and read this one message, it is likely that there would have been a time lag, and during that period they would have had to have copied and saved every single message on the Fleet Broadcast, to have been able to reconstruct the OTPs built from the dummy traffic being broadcast.

I would like to be able to say that after a carefully carried-out risk assessment an expert panel decided that the risk was worth taking, but there is no record to suggest that this was the case. It is unlikely that we'll ever know, but my guess is that this was a piece of local ingenuity, which happened to turn out well.

Ten avoidable problems which made the Royal Navy's encryption exploitable in 1939

By the time of the outbreak of war in 1939, Nazi Germany had thoroughly penetrated Royal Navy encryption systems, and the Navy's changing its main codebook caused barely a hiccough to German Navy cryptanalysts: in fact it took until 1943 to secure RN and allied naval encryption and during this time the German Navy destroyed millions of tons of allied shipping and came close to preventing the supply of food and military supplies to the UK from North America. Just as Bletchley Park exploited poor security by the German military in their use of encryption, the German Navy's B-Dienst was able to exploit the poor security of the codes used by the Admiralty. How could this have happened? A note written years later by Captain D A "Willie" Wilson RN identifies areas where the Admiralty got it wrong. I think the issues they raise are as relevant today as they were between the wars, because although security today addresses technological challenges that our forebears couldn't have imagined, the foundations of security and the mindset of the security practitioner are fundamentally the same.

The Admiralty got things so badly wrong because there was no single coordinating body managing Comsec in the UK, and no recognisable centre or standard of Comsec expertise. GC&CS had a responsibility to provide advice on Comsec to civil ministries, and could be approached by the service ministries if they so wished. Within GC&CS, just as Comint was the responsibility of Denniston, the Head, Comsec was the responsibility of Travis, the Deputy Head. However, Travis was also responsible for: collection management, for the reporting, indexing, and distribution of intelligence reports, for the GC&CS Registry, for liaison with SIS, and, from 1938, for liaison with the service ministries. All of this was in addition to his responsibility (to 'C', interestingly: not the Head of GC&CS) for the British Codes Section.

The fundamental problem was that nobody questioned the principle that good cryptanalysts would make good communications security advisors, when in fact their advice would be limited by the extent of their cryptanalytic skills. In a non-mechanical environment, this meant cryptanalysts saying "this is the sort of code I can't break", and assuming a) that that meant that nobody else would be able to, granting that code an invulnerability; and b) given that cryptanalysts didn't do traffic analysis, that crypt security was the only security that mattered: that wireless security (secure traffic management) wasn't something that needed to be considered.

1. Don't split responsibilities among different people.

The Director of Naval Intelligence (DNI) was responsible for technical and physical security, but was dependent on technical cypher matters on the advice given by GC&CS. The Director of the Signal Department (DSD) was responsible for providing the means of communication, and for providing the Coding Staff, but he had no responsibility for wireless security, (ie for denying the enemy intelligence from traffic analysis). The Paymaster Director General (PDG) had responsibility for the provision and training of cypher staff, but had no responsibility for the cyphers used.  In the Fleet the Secretary to a Flag Officer was the Squadron or Fleet Cypher Officer, responsibly for seeing the proper use of cyphers afloat. Within the Admiralty M Branch of the Secretariat was responsible for the distribution of Cyphers and Codes through the Navy, and the Secretariat provided the Admiralty cypher office, known as "War Registry".

2. Choose your advisers carefully

GC&CS was required to advise (but not mandate) security to the services between the wars. In fact Travis advised the Admiralty, Tiltman advised the War Office, and Josh Cooper advised the Air Ministry, but they didn't talk to each other about security, and didn't consult any of the rest of the cryptanalytic staff in GC&CS. Their advice was, therefore, strictly limited. Cipher security was the subject of a lecture on the Accountant Officers Technical Course (a prerequisite for a Secretary to a Flag Officer) but by the beginning of the 1930s this was a lecture about cryptanalysis given by Bodsworth and Knox, and was unlikely to have been illuminating. The Long Signal Course for officers specialising in communications had no lectures at all on communications security.

3. Know when to ask for a second opinion

The key piece of advice that the Admiralty needed was about the security of the 'Long Subtractor', the key used to encipher messages once encoded. In effect, this was a one time pad which was reused. How many times could it be reused without compromising security? Theoretically, it should never be reused, but in practice, could it be? Wilson wasn't happy with the response: once gave absolute security, twice was almost guaranteed, three times wasn't really dangerous, but more than three was, and using the same piece of key five or six times was positively dangerous. This was very bad advice, and Wilson knew it, but there was no move to change advisor (nor any obvious other advisor to turn to).

4. Security doesn't come second to intelligence

The Admiralty had two billets in GC&CS for officers: the idea was that this would be a source of officers with practical experience in security. But everyone in GC&CS knew that the department's main job – its important job – was intelligence production, and the officers posted in became, for the most part, cryptanalysts. Intelligence breakthroughs are of tremendous value: if your security is no good, it will be your adversary who makes the intelligence breakthrough by exploiting your communications.

5. Don't do the enemy's job for him

The RN used three different basic codebooks: Tactical Code (3 letter groups, not reencrypted); Administrative Code (five figure groups, reencrypted with a Long Subtractor); Naval Cypher (used for operational traffic: four figure groups, reencrypted with a  Long Subtractor). This meant that anybody intercepting a naval message immediately knew, without decryption, what sort of message had been sent. And although different Long Subtractor tables were used for different regions and commands, the indicator – the explanation of the start point in the relevant table – wasn't disguised at all. The German Navy's cryptanalysts had it easy.

6. Don't assume before you understand

In order to gauge how many of each reciphering table to print and distribute (a job that would always be complex given the Navy's worldwide extent) the Admiralty had to make assumptions. It assumed that the proportion of administrative to operational traffic in wartime would be around 80:20 – they were wrong: it was the other way round. If the Royal Navy officers in GC&CS had been doing the job for which they had been posted in, they would have seen this by looking at traffic levels in Italian Naval material collected for GC&CS during the Abyssinian Crisis and the Spanish Civil War.

7. Even if it ain't broke it might still be worth fixing

According to Wilson, by 1937 when Typex, the UK version of Enigma, came into service, Lord Mountbatten was pressing for it to be used by the Royal Navy but got nowhere because of the lack of a single person able to resolve to bring a new encryption system into use. Lord Mountbatten concluded that it was just too difficult, but even putting Typex on a few ships in the North Atlantic could have made a tremendous difference. I asked the Duke of Edinburgh about this in 2013 and he confirmed it. (That is probably the most outrageous name drop you'll see today.)

8. Who needs what?

As a matter of routine, Commands, shore establishment and major ships were given copies of all reciphering tables, so that anybody could communicate securely with anybody else. But why (for example) would CinC Nore need to communicate securely and directly with the Senior Naval Officer Upper Yangtze? Why couldn't the few (I imagine) messages between most territorial commands not be relayed through the Admiralty. Apart from the difficulty of printing and distributing so much key material, it meant that if a ship was believed to have potentially been captured by enemy forces, all key material on board had to be assumed to be compromised.

9. Understand the vulnerability of your mode of communication as well as of your crypto

After Wilson had been Head of a new section NID10/DSD10 in the Admiralty early in the war, he set up a new section to look at the way the Navy communicated to see whether the Germans might be able to derive intelligence from analysis of the traffic. This had never been done before.

10. Security versus Operability

Security and operability don't always have to be in competition: designers of systems need to understand how users communicate to design practicality alongside security; system administrators need to ensure that the reasons for the rules they impose are understood; and users mustn't try to subvert the rules and compromise security simply to make life easier. In order to make Royal Navy comms practice secure against German traffic analysis, DSD10/NID10 introduced a series of security measures without explaining them to the signalmen who had to implement them. As a result, they were widely ignored and subverted. When this was discovered the signalmen asked for a senior officer to investigate the matter. Admiral Somerville (who had been in charge of wireless comms for the Mediterranean Fleet during the First World War) was asked to do this and he came down firmly on the side of security, but was able to explain why to the signalmen, from whom there was no further opposition to the new measures.

Sir Edward Bridges and the Development of UK Communications Security

I think I'm a bit closer to an explanation of something that has exercised me for a few years: why did Sir Edward Bridges, first as Cabinet Secretary and then as Permanent Undersecretary (PUS) at the Treasury, take such an interest in Communications Security (Comsec), from 1941 until his retirement in 1956? He made sure that in January 1944 the creation of a Cypher Policy Board independent of the JIC was approved by the Chiefs of Staff and the War Cabinet. This meant a better defined Comsec organisation in GC&CS with the authority to mandate security standards across the whole of UK Government. In 1954 he drove through the institutional separation of the LCSA (what would eventually be called CESD, then CESG, then turn into the NCSC) from GCHQ, a separation that lasted until 1969.

His first experience of encryption probably came in March 1929, when, as a Treasury civil servant, he became a member of the Inter-Departmental Committee on Cypher Machines, on which he sat alongside Edward Travis, Deputy Head of GC&CS and its lead for Comsec. The committee's report seems to have been ignored: it recommended the "O'Brien-Gardiner machine" no working model of which was available. Enigma was turned down on the grounds that it was of foreign manufacture.

Eventually, Wing Commander Lywood RAF improved the on the design of Enigma and came up with Typex which was first used operationally in 1937. GC&CS was kept away from the development of Typex by either Lywood, or the RAF, or both: they didn't believe that the advice being provided by GC&CS on security was adequate: he (or the RAF, or both) was right. By the time of Dunkirk in 1940, Typex was the only British military encryption system the Germans couldn't read. (I will write more about this catastrophic failure another time.)

In October 1941 four cryptanalysts from Bletchley Park wrote to the Prime Minister complaining of delays in supplying the manpower needed for GC&CS's mission. Churchill's response was immediate: his famous 'Action This Day' minute in which General Ismay was instructed to "Make sure they have all they want on extreme priority and report to me that this has been done". In fact it looks likely that while Ismay worked with 'C' on the manpower required for Bletchley Park, Bridges interested himself in the security mission, at the time mainly farmed out to Mansfield College Oxford, with a small nucleus at Bletchley. (Certainly, some months later, when GC&CS approached the Treasury to increase its establishment across the board, its letter said "if you need more evidence I can only refer you to Bridges or Ismay" (letter quoted by Nigel de Grey in HW 43/76).)

Bridges understood that the primary reason for the failure of British Comsec was due to a systemic issue: GC&CS's charter did not allow it to mandate security standards for the armed forces, only to provide advice. A separate issue, but which contributed to the overall failure, was that the quality of the advice was poor, and, anyway, wasn't always accepted. Bridges spent fifteen years godfathering a new structure for UK Comsec, which resolved both of these issues.

The Admiralty had sponsored an "Inter-Service Committee on the Security of Codes and Cyphers" early in 1941, and Travis and Tiltman represented GCHQ at its first and only meeting. It set up a "Technical Sub-Committee" chaired by Travis with three members: from the Admiralty and Air Ministry, and Tiltman, for GC&CS and the War Office. It met regularly (23 meetings between February and October 1941) and it changed its title to Cypher Security Committee. The parent body was replaced by a new "Inter-Services Cypher and W/T Security Committee" and it adopted the Cypher Security Committee, widening its membership beyond GC&CS and the three services to include Civil Departments (Foreign Office, Colonial Office, India Office, Dominions Office, Ministry of War Transport, Ministry of Food, Ministry of Supply, Postal and Telegraphic Censorship, Ministry of Aircraft Production and Ministry of Information) though these only attended by invitation when matters relevant to them were to be discussed.

This Committee was still advisory (though I don't think that its conclusions and recommendations were ignored or rejected by anybody), and had no forward planning responsibilities, but Bridges was still keen to see a more formal arrangement. He persuaded the original committee members to come up with a paper outlining the problems faced by the committee and after detailed discussion in November and December 1943 a paper was agreed by the Chiefs of Staff in January 1944 and adopted by the War Cabinet. Its main conclusions were:

1.            A new body called the Cypher Policy Board was created under the authority of the Prime Minister and the War Cabinet. Its membership was: 'C' (Chair) as Director-General GC&CS; the Cabinet Secretary (Bridges); Director General Signals Air Ministry (representing the Chiefs of Staff); Director GC&CS (Travis); Secretary (Communications Security Advisor GC&CS). It was responsible for: (a) deciding questions of policy governing the security of British cyphers, including decisions about new cyphering devices proposed and safeguards necessary for their use; (b) ensuring that the use of cyphers by the Services and by Government Departments was properly supervised; (c) advising on the security of cyphers used by Allies where appropriate.

2.            A new section of GC&CS was established to deal with the security of British cyphers and of Allied cyphers (insofar as British commands were concerned). This section was headed by a GC&CS Assistant Director with the title Communications Security Advisor (CSA) (Captain Wilson RN).

3.            The Cypher Security Committee became responsible to the Cypher Policy Board, and the Secretary of the Cypher Policy Board became its chair.

Bridges became PUS at the Treasury but took his membership of the Board with him. Even if communications could only command a small amount of his attention as PUS and Head of the Home Civil Service, he would nevertheless make time for it.

Some years later in 1952 the Admiralty complained about problems caused by the way in which the Cypher Policy Board had developed a committee structure. The matter was taken up at Board level and the Admiralty said that it felt that many of the tasks being performed by the Board might be better carried out by the Sigint Board. Bridges answered that the Cypher Policy Board had been set up in 1944 to see that the right cryptographic policy was devised, both from the standpoints of security and practicability and to ensure that this policy was carried out. He believed that it would be quite inappropriate for these responsibilities to be handled by the Sigint authorities. An officer from the Signals Division of the Admiralty carried out an investigation in conjunction with a Treasury civil servant nominated by Bridges.

The investigation looked at the structures of the Board but, surprisingly, also recommended the creation of a new agency, the Telecommunications Security Agency, which would subsume the Comsec responsibilities of GCHQ for planning and policy for new cryptosystems and speech security devices, and for their design and engineering development. It would also take on the responsibilities for Comsec which were still in control of the services. GCHQ and the service ministries were persuadable: the military complained that existing structures usually saw GCHQ representatives asking for solutions which gave 100% security but were practically unusable; to which GCHQ countered that no new off-line system had been developed since the war because of the military's insistence on continually changing the specifications for new machines. Bridges persuaded them not only of the need for a single body but also that both GCHQ and the services would accept that the new Agency's Director would sit above them and act as arbiter.

In the event, LCSA – the London Communications Security Agency – was set up in 1954. It shared the central London office block on Palmer Street which GCHQ had acquired in 1953 for a 'front office' and maintained the bulk of its staff at Eastcote when the rest of GCHQ went to Cheltenham. The divorce was never final: LCSA (soon renamed CESD) was too small to be an independent agency, and GCHQ continued to provide administrative and estate services; the mathematicians needed by both organisations continued to form a common pool; and particularly after the retirement of Bridges, CESD had no real voice in Whitehall or ability to face the services down where necessary other beyond GCHQ's ability to speak on its behalf.

Trying to work out why Bridges had interested himself to such an extent is difficult. When the NCSC was being set up, and I had been through this tale with Ciaran Martin, its first Head, he asked Lord Bridges if there were any papers held by the family, or any family folk memory on this subject: there weren't.

I think that Bridges, having decided to look out for Comsec while General Ismay did the same for Comint after the 'Action This Day' minute from Churchill in 1941, realised that the subject needed higher price help than it was likely to get if he didn't look after it himself. His view (in modern language) was that Comint professionals are the wrong people to make Comsec policy, even though the input from Comint is the primary tool at the disposal of Comsec professionals. It is clear that he believed that if Comsec and Comint were housed in a single organisation, Comsec would necessarily suffer if its autonomy could not be maintained, and so arranged for it to separate from GCHQ as completely as possible.

Separation from GCHQ simply didn't work, so CESD came back to Cheltenham as CESG: but its autonomy was part of the deal, even if the degree of autonomy was a moveable feast. The NCSC model – "part of GCHQ" but headquartered outside Cheltenham – was a response to the radically different requirements cybersecurity demanded of both Comint and Comsec, but has finally achieved Bridges' vision.

Intelligence Officers' Small Talk: Tiltman Meets Liddell

 

Guy Liddell, the Director of Counter-Espionage in MI5 during the Second World War (and beyond) kept a diary – in fact he dictated an account of each day's activity which was typed up. The diaries from 1939 to 1953 have been released to The National Archives: they are an important resource for anybody interested in UK intelligence history. The entry for 24 October 1944 (TNA reference KV 4/195) recounts a conversation with John Tiltman, the "Chief Cryptographer" at GC&CS, what today would be called the Head of Cryptanalysis. They had each been appointed CBE in the 1944 New Year's Honours List and attended their investiture on 24 October.

"I attended the investiture at Buckingham Palace where I met Tiltman of GC&CS. Tiltman reminded me of the time when he and I had gone down to some firm in Southampton Row to inspect a holorith [sic] machine. This was a good many years before the war. He said he had no idea at that time that the holorith was going to be such a big factor in the work that he would be doing. He did not know what future there would be for GC&CS but he could not help thinking that the difficulties of the work might become insuperable when foreign govts. realised the mistakes that they had made during the war. Intelligence might well be driven back on the old cloak and dagger lines. Tiltman is now Chief Officer on cryptography and a Brigadier. He said that this does not prevent him from a certain amount of daily exercise in his special line without which he thinks his usefulness to the organisation would disappear in a very short time. He says that methods change so quickly that it is absolutely essential to do a certain amount of hack work, otherwise one's usefulness will entirely disappear."

There were very few contacts between GC&CS and MI5, even on operational matters: all communication was expected to go through SIS, and SIS decided what each needed to know about the other. This is the only record, I believe, of Tiltman dealing with a member of MI5 during the war and the conversation, as recorded by Liddell, looks like the sort of small talk two senior members of different agencies might exchange at such an event. But effectively Tiltman says only three things, and doesn't really tell Liddell anything.

Hollerith machines were extremely important at Bletchley Park, as they had been in Room 40, as a means of sorting and analysing information, but they were not the sharp end of Bletchley's information technology. By this time Bletchley's cryptanalytic work was being supported by some 200 Bombes (as well as sharing time on US Bombes) and by 5 Colossus machines (a sixth would come into service four days later). It is instructive that Liddell obviously got no sense of this.

The statement that cryptanalysis might become impossible once foreign governments became aware of the mistakes they had made and that therefore Humint would become dominant is superficially plausible but it begs the question of how they were going to find out. Sigint security had developed into a sophisticated process during the war, with even the fact of interception of enemy communications not being discussed publicly. The Ultra dissemination system was designed to restrict knowledge of the fact of successful cryptanalysis of enemy communications to an absolute minimum, while an equally sophisticated process allowed action to be taken on Ultra intelligence by finding plausible alternative sources for it. The success of the US attack on Japanese encryption would become known during the Pearl Harbor enquiry, but that was a year after this conversation.

Tiltman's views about the need to keep his hand in if he is to remain useful to GC&CS is uncontentious, though I imagine he really meant that he enjoyed any opportunity to be a cryptanalyst instead of an administrator.

The main surprise to me in this conversation is that the subject of signal security didn't come up. At this time MI5 was trying to establish itself as lead department in this field even though it controlled none of the assets required either to monitor UK service or civilian traffic or to secure communications. A Wireless Telegraphy Security Committee had been in existence since 1941 but MI5 had never been a member, and MI5 raised the subject at the JIC – which GC&CS wasn't a member of. Liddell's diaries show that MI5 was not impressed by the Radio Security Service, whose job was to intercept and process illicit signals (messages sent by unauthorised individuals using radios) and which was controlled by SIS. (It would become part of GCHQ after the war for a short period before its functions became part of GCHQ normal business.) Reading Liddell's diaries, it feels to me that MI5's instinct was that signal security was part of national security, and therefore MI5's business, even if it had no realistic way of securing signals.

This is a really frustrating extract: there was so much that Liddell and Tiltman could have talked about, but they didn't. Two senior members of two secret agencies recognised each other at a formal state occasion and exchanged pleasantries, but nothing else.