When Opposite Sides Share the same Crypto

I imagine that very few people think about the cryptographic consequences of a country breaking up but they can be significant, especially if the break-up is violent. The outbreak of the Spanish Civil War, for example, produced within days two opposing armed forces which used the same encryption systems which had to be modified on the fly by people with little cryptographic experience. The same thing happened in France in 1940 when the Free French military broke away from those who accepted the Armistice. One such break-up during the Second World War, that of the division of Italy into areas controlled by Mussolini’s Italian Social Republic based in Salò and those controlled by the Kingdom of Italy and based in Salerno, were written up after the war by Fred Catty, Head of the Italian Diplomatic Section in Berkeley St. (Commercial and Diplomatic Sigint were hived off from Bletchley Park to London early in 1942.)

Italian Diplomatic ciphers had been read without difficulty by GC&CS until in 1941 the Germans committed to joint operations with the Italians in North Africa. The Germans had been reading Italian systems themselves and simply refused to share classified material with their ally. However, the Germans decided that ensuring that the Italians gave nothing away of German intentions was more important than reading their traffic. They informed the Italians about the weakness of their systems and in July 1942 the old systems were gradually replaced by new ones. Fortunately for Berkeley Street the Germans did not bother to supervise Italian use of the new systems so they were never as secure as they should have been. In fact, by the end of 1942 the Italian Diplomatic Section was as successful as it had been before the new systems were introduced. In January 1943 the Italians appear to have had some misgivings and they added an extra level of security which should have brought GC&CS to a full-stop: but operator error neutralised the precautions the Italians had taken.  

In July 1943 Mussolini was deposed but escaped from captivity and set up his own Neo-Fascist Government in North Italy. In September the Royal Italian Government accepted unconditional surrender. Mussolini had his own Foreign Ministry located at Salò (Lake Garda) with its own WT Station while the Royal Italian Foreign Ministry was based at Salerno.

What follows is Mr Catty’s description (from HW 43/4 which hasn’t been released but which is quoted in HW 43/75-77, de Grey’s draft history, which has) of what happened next. It is worth highlighting that that the UK and US were as lax as the Germans had been in supervising their new allies’ security, and that the organisational and geographical separation of communications intelligence and communications security in the UK wasted cryptanalytic effort at Berkeley St.

‘There was quite an extensive array of posts which would claim to represent this Neo-Fascist Government abroad. In all of Europe occupied by the Germans the former Italian Ministers perforce remained Fascist; in addition, in Madrid, besides the 'Royal" Italian Embassy, Mussolini 'accredited' a Dr. Morreale (believed to have been the brother of Clara Petacci) as his 'Ambassador'. In Tokyo the Ambassador preferred interment as a 'Royalist', but Col. Principini, Military Attaché, took over. All these and isolated posts in various countries would now be trying to report to and take orders from Mussolini.

In the South there was now a 'Royal Italian' Government (at Salerno until Rome was taken nearly a year later). The Italian Diplomatic Service was always royalist by tradition. And now all the major ambassadors and ministers in countries where they could safely do so, declared for the King and put themselves at the disposal of the Allies. In Madrid the Royal Embassy handed over all its cyphering material for inspection to the Allied Representatives.

Meanwhile the Salerno Government, being subject to the control of Allied Force Headquarters for its communications, made a case first that it needed to communicate with Royal Italian posts abroad and subsequently that it should have cyphers (deposited if need be) in which to do so.

All the cyphers hitherto used by the Foreign Ministry in Rome had obviously to be considered as now equally useless to either side. The Germans in Rome had ample opportunity of ransacking the "Ufficio Cifra". We know that they did so. At the same time they must have known that Royal Embassies would deliver up their stock to us and that men who knew all this material were in Allied hands. It was a reasonable conclusion that neither we nor the Germans would ever be concerned with these codes and tables again.

It should be recorded that it was the unreasonable that happened. When cypher traffic was intercepted, as in the French case, both sides were found using the compromised cyphers.

Owing to the urgent needs of Japanese and other sections at Berkeley Street the Italian Section had, as soon as Italy surrendered, been quickly dissolved. Fortunately there remained a small nucleus to examine any cypher traffic which might need investigating, and its problem, for the first six months was (again contrary to all reason) not a Fascist, but a Royal, pro-allied cypher.

For the first seven months the Fascist traffic which was intercepted was almost entirely in plain text. This was probably an order made by the Germans, since, from an intercept in December we knew that they were reducing the strength and status of Fascist posts everywhere and they may well at this juncture not have had time to bother with supervising Fascist cyphers.

Traffic was intercepted between Salò and Berlin, Madrid, Tokyo and Budapest. Between September 1943 and January 1944 there were 800 intercepts, none of them separately important, but together giving a picture of the set-up at Salò the attitude of the Germans, and who were Mussolini's adherents abroad.

But there began to be intercepted in January cypher traffic between the Royal Embassy in Madrid and the Salerno Foreign Office which was entirely different from any known Italian Diplomatic Cypher. It was found to be a transposition, believed double since prolonged efforts failed to break it. We were in fact spending our efforts on a system which Allied Military headquarters had approved (and as we later learned) improved for the Italians on our side.

Eventually the key-book was sent to us. The keys at that time were derived from an English illustrated magazine. Though it was a 'deposited' system, it was not certain that the traffic was being read at Caserta [ie Allied Military HQ], and this was therefore done and the contents, when of interest, translated at Berkeley Street.

From this it became clear firstly that the system, being laborious, was unsuited to the long dispatches which diplomats have to send; secondly that the Italians were ignoring rules which were part of the security of the system, and thirdly that Madrid was being told to retransmit messages received in this system to a post which did not hold it. This last would be a major blunder, of which the Italians were quite capable.

But our interception of Royal Italian showed even worse to be going on, of which Allied Force Headquarters in Italy could not be aware, since they controlled only the use of cypher into and out of Italy. Between Madrid and seven other posts various of the old and completely compromised cyphers were being used.

This gradually led the Foreign Office to the conclusion that to advise the Italian Foreign Office and ensure that it used only secure cyphers and used them correctly a joint Anglo-American Mission staffed by Berkeley Street and ASA, Washington should be attached to the Allied Control Commission in Italy. This mission arrived in Rome on 1st October 1944. It consisted of the head of the Italian Section at Berkeley Street with one secretary and of an officer who had worked in the Italian Section at ASA, Washington. It worked in Rome as a subsection of the Communications Section of the Allied Control Commission until the end of hostilities in Italy in May 1945. Its terms of reference were:

1. To watch Italian diplomatic communications and their security.

2. To provide the Italian Ministry of Foreign Affairs and the Italian missions abroad with suitable means of communication.

3. To keep the Foreign Office, State Department and Allied Control Commission informed of the contents of Italian telegrams.

4. To keep GC&CS and the corresponding US organisation informed regarding Italian cyphers and lines of communication.

The Italian Foreign Office, now reestablished in Rome, had been informed through the Allied Control Commission of items 1 and 2, and no doubt suspected 3. Contact with its cypher bureau was established at once on the amicable footing that our control was a necessity while operations were impending. Each party was well aware that it was dealing with its professional opposite numbers who had obligations which would be veiled throughout the official dealings by a good-humoured reticence.’

 

Austerity in Post-War GCHQ - and Help from Friends

Austerity, belt-tightening, peace dividend: at various points during my career HMG found ways to make my life less simple, either by cutting GCHQ’s budget and therefore the resource we had available to put against a particular issue; or, more simply, by making sure I had a pay rise of less than the rate of inflation (as happened for each of the last ten years before I retired).

But much as I might have grumbled at the time (and since) I have to say that I never faced the conditions that GCHQ staff faced in the first few years after they left Bletchley Park in April 1946 to return to London: in fact to a former Bombe outstation at Eastcote, between Harrow and Ruislip in NE London.

Bletchley Park was too big for the much reduced Sigint organisation, and staff who had been with the organisation before the outbreak of war had been promised a return to London when peace came. However, not only was the London location not civil service premises in Broadway, nor anywhere else in SW1 (as returnees might have expected) for that matter, but the premises they were to occupy in Eastcote were just the same wartime TOBs – Temporary Office Buildings – many of them had spent years working in during the war.

The country was broke: there was no money for infrastructure other than to make do, and mend what could be repaired. Much of the damage caused by the 1940-42 blitz and the V-weapon attacks of 1944-45 remained unrepaired. The housing stock, which was already substandard in the 1930s, was in a much worse state after a decade of neglect. Not only that, but food rationing was still in place – in fact was even more severe than in wartime.

The winter of 1947/48 was exceptionally cold, so cold that transport was sufficiently disrupted to cause the breakdown of the national distribution network. And this caused more problems in the cities than in more rural areas, as food could not be brought from the areas in which it was produced to the areas in which it could not be produced. Rationing of potatoes — something that hadn't happened during the war — was introduced.

So I think of my predecessors in Eastcote in December 1949. It was more than ten years since the beginning of the war, and yet heating was limited, public transport was the only way to and from work, and salaries were low and highly taxed. (Income tax was 45%, while for those on salaries above £2000 (there were very few at GCHQ) the higher level of tax was 55%.) The 56 conditioned weekly working hours of wartime had been moved back to 42.5 (exceptionally: the rest of the Civil Service was on 45), but it had already been announced that these 42.5 hours would rise to 45.5 hours the following year, with Saturday morning working reintroduced. In short, things were bleak.

As I said at the beginning, this was austerity with a vengeance, but in December 1949, as had been the case in December 1947 and December 1948, there was a tiny challenge to the austerity in GCHQ. Not a staff rebellion, not some outrageous act of theft, but the action of a large number of unknown friends. On 22 December 1949 staff received a message from AD(P) (Assistant Director Personnel. responsible for HR, Finance and Personnel Security). This post was held by Eric Jones, who would later become GCHQ Director. He came from a family of textile manufacturers based in Manchester who had been commissioned into the RAF as a Group Captain and had been posted to Bletchley Park specifically to make Hut 3 (German Army and Air Force analysis and reporting) work efficiently and productively. For the time, his approach was singular: he dealt with all of his subordinates with courtesy and tact, treating them all, from the most junior, as individuals. This brought out the best in his staff and as a result Hut 3 tended to be a happy place to work. (Some of his Directorate colleagues referred to him, somewhat sniffily, as 'the Manchester Businessman': it was not meant to be complimentary.)

AD(P)'s message informed staff of the Christmas signal that the Director had sent to the Coordinator, the head of the (US) Armed Forces Security Agency (AFSA), the predecessor of NSA.

The Notice reads as follows

‘Signal to Coordinator

The distribution of gifts so kindly sent by staff of AFSA has just taken place. This wonderful generosity, and the friendship of which it is the token, is warmly appreciated by all, whether lucky or unlucky in the draw. By it, over the past three years, about a thousand of GCHQ staff, covering the lower salary scales, have shared the parcels.

Please be good enough to convey this message to the staff of AFSA, with the thanks and best wishes for Christmas and the New Year of the whole staff at GCHQ.’

What had happened was that the staff at AFSA, who had heard through US liaison staff in London of the hardships facing their UK opposite numbers, had decided in 1947 to send Christmas parcels to GCHQ. At GCHQ, it was agreed that only clerical staff, the people who couldn’t easily afford to supplement their rations on the black market, should enter a draw for the parcels. And the clerical staff quickly worked out that by pooling their tickets and sharing the results of the draw for the parcels, the chances of some supplement to the Christmas ration was more likely. In this context, ‘about a thousand of GCHQ staff, covering the lower salary scales’ meant just about everybody in clerical positions. It's hard to imagine nowadays, but each of those food parcels made a difference between an austerity Christmas and an old-fashioned "eat, drink and be merry" Christmas.

This is also a demonstration of the fact that UKUSA has always been something more than an arrangement of expedience. The relationship with partners has an added dimension because of the continual contact between individuals from each of the agencies. At this level, the relationship isn’t between allies so much as friends, and friends look out for each other. (And it shows the boundless generosity that characterises so many Americans. In November 1948 the organising committee at AFSA was quoted $1000 for the thousand eight ounce packages of tea and the thousand one pound packets of sugar it sent alongside the CARE packages.)

I wonder what the Ghost of Christmas Yet To Come would have had to show all those people: their dreams might not have gone much beyond the basics: enough to eat; a warm house; nice clothes; but imagine how pleased they would be if they could look into the future and see that as the world changed, and the country managed to sort itself out, the need for the food parcels went away.

One or two old lags might recognise this as a reworked version of something I posted internally in GCHQ when I was still ‘inside the wire’. My thanks to my successor for letting me retell the story and for letting me use the photographs.

Sigint in Fiction

I had an articled published last month in the John Buchan Journal (unsurprisingly, the journal of the John Buchan Society). It is about the way that John Buchan drew on his First World War experience as a customer of Sigint to use cryptanalysis in one of his books and in a short story to advance the narrative, and to develop characters.

A key point, I feel, is that it is impossible to describe the process of cryptanalysis in a work of fiction and make it interesting for the general reader. Buchan’s answer was to give a vague idea of an encryption system: Playfair in one, double transposition in the other; make the reader understand that this is something really difficult, and that therefore the practitioners have to be intelligent, and not just lucky; and move on, making the story (or the relevant part of the story) about people who break codes or ciphers, not about the process of breaking them.

Two of the novels of Dorothy L Sayers have cryptanalytic sub-plots. Lord Peter Wimsey certainly shows himself to be intelligent in breaking the messages concerned, but the eight or nine pages in Chapter 28 of Have His Carcase, in which he goes step by step through the process of breaking a message enciphered with Playfair, and takes no prisoners while doing so. (She doesn’t try quite as hard in The Nine Tailors, in which she describes in just two or three pages an encryption method which uses as its key the course of the treble bell in a peal of Kent Treble Bob Major.)

Perhaps the process of cryptanalysis is of limited interest to the reader of a novel because the process of Sigint – interception, analysis, cryptanalysis – is analogous (though less interesting) to having somebody physically steal a copy of the message. The point the author is aiming at is that somebody who shouldn’t have had access to the message now has it, so the story can move on. There are more and less interesting ways of doing this, and pausing the story so that Lord Peter can show off is at the ‘less’ end of the scale.

I’m not aware, and I think the reason why is clear, that there is that much at all about Sigint in literature. I’ve dealt with Buchan; From Russia With Love is the only Bond story which touches on it (the attempt to steal a Soviet encryption device draws from British plans to steal Enigmas during the Second World War that Ian Fleming was closely involved with); John Hale’s The Whistle Blower is about cryptologists, but not cryptology; are there any more?  (I'm speaking of British novels.)

There is one candidate for a novel about GCHQ, though: The Tin Men by Michael Frayn. His National Service was spent being trained in Russian so that he could be a Sigint linguist. The novel is about a research organisation: The William Morris Institute of Automation Research. The institute is at the forefront of automation: computers are being programmed to write newspaper articles, produce popular TV shows and say prayers. But problems begin when programmers begin to program computers to welcome HM The Queen on a royal visit to the Institute. No spoilers, but it is as funny a book as you would expect the author of Noises Off to have written, and could probably claim the title of the first satirical novel about AI.

The novel was published in 1965, at a time when any mention of GCHQ and the work it carried out would have brought an author to charges under the Official Secrets Act. But in the succeeding fifty years the story has been passed down in GCHQ that The Tin Men was Frayn’s attempt to make sense of the UK’s national Sigint organisation.

Do read it!

 

More Detail on Cairncross at Bletchley Park

Last week saw a major release to The National Archives of MI5 files, mainly concerning ‘The Cambridge Five’ or at least the three of them, Philby, Blunt and Cairncross, whom MI5 had the opportunity to investigate (Burgess and Maclean having skipped the jurisdiction).

I’ve obviously not been able to go through this new material in any detail, but I have looked at the parts of the Cairncross files which shed light on his time at Bletchley Park.

I covered some of this three years ago in Sigint Historian: Personnel Security at Bletchley Park - Part Two

There is some interesting new (to me) detail. Cairncross had been encouraged by his controller to join GC&S or SIS. When he first identified to MI5 the battle in which the Ultra material he had passed to his controller had given the Red Army a decisive advantage over the Germans he claimed it was the Battle of Kharkov (Kharkiv, of course, today), not Kursk. Interviewed in 1967, Peter Wright asked him whether he could remember anybody at Bletchley who might have been pro-Russian: his answer was that everybody at BP was pro-Russian because of the events of the war, but he named three people Hugo Gatti, Philip Pounsey and Douglas Parmier, but these are likely simply to be three names plucked out of the air. (None appear on the Bletchley Park Roll of Honour.)

Edit: I stand corrected. Hugh Gatty, Philip Pouncey and Douglas Parmeé are all on the BP Roll of Honour.

More insidiously, he said that his Soviet controller had once asked him why a former FO colleague, Roddy Greiffenhagen, had transferred to GC&CS, and whether this move would hit Greiffenhagen financially. After some investigation, MI5 discovered that Greiffenhagen was eased out of the FO ‘because of his total inability to do the work required’, and concluded that Cairncross was likely to have passed biographical details of other FO colleagues to the Soviets as well. He admitted to having passed a couple of pen pictures of GC&CS colleagues to his controller, but couldn’t remember who they were. This part of the release leaves a question mark over Greiffenhagen’s reputation: it’s a bit unfair, as there was no doubt an investigation subsequently: one, potentially, for a future release.

Cairncross said that his controller was annoyed when he engineered a transfer for himself from BP to SIS Section V at Ryder St to work on counter espionage (a section to which GC&CS deployed several members of staff as it began to resume its work on Soviet targets), the annoyance, he claimed, being because of the quality of the material he was passing from Hut 3. I wonder whether the presence there already of Kim Philby might have been a better reason for the controller’s annoyance.

There is a curious tale of Cairncross at Ryder St pestering somebody (whose name is redacted) at Berkeley St (where GC&CS diplomatic traffic was worked) to get information about a breakthrough made by Berkeley St. The unnamed person said that he had probably been indiscreet at the time, but reflecting on matters in 1967, thought that as Philby was handling this material, his indiscretions were probably not that relevant.

There is an amusing 1955 account by Henry Dryden (a Second World War Siginter who stayed on after the war). In the spring of 1949 Cairncross phoned Dryden at GCHQ, then at Eastcote, inviting him to lunch, and over the soup course asked whether GCHQ was a successful with Soviet encryption as it had been with German. Dryden ‘mumbled some sort of non-committal reply, bringing in the phrase “one-time pad” and tried to give the impression that one could not do anything to make progress’. This story, at least, is already in the public domain, having been told by Dryden in 1993, in a postscript to his contribution to Hinsley and Stripp’s Codebreakers.

There may well be more details in these files, and I’m sure there will certainly be in other files in the release, but three things stand out to me. The first is something I have mentioned before: how little the MI5 investigators knew of (never mind understood) how Sigint worked and their consequently appearing to deal with it as not as key an issue as were some others. The second, is that my judgement of three years ago, that of course it was likely there were other Soviet sympathisers at Bletchley, but that they were unlikely to have been able to pass on many great secrets, hasn’t changed, not least because of the third: that only if somebody had access to Bletchley’s serialised reported intelligence would it be possible to pass intelligible information to a hostile intelligence service.

Preparing the Ground for Room 40

There was nothing recognisable as Sigint in the UK before the First World War started, but we know a fair amount about interception before the war started and can make reasonable assumptions about why there was no permanent organisation created to monitor the wireless traffic of foreign armed forces. The idea that there was no serious thinking about the value of interception, and, its corollary, that there was no thinking about how an enemy might exploit UK communications, is not tenable. What follows is about the Admiralty – I’ll perhaps look at the War Office in another post.

The earliest indication I have seen so far of somebody thinking about a potential downside to the wireless telegraphy (WT) systems first being trialled by RN ships at the start of the last century comes from a 1900 report. Between March and May 1900 four ships on the Cape of Good Hope Station, HMS Forte, HMS Magicienne, HMS Raccoon and HMS Thetis, were equipped with Marconi WT equipment and carried out trials. Their reports, forwarded to the Admiralty by the Senior Naval Officer on the Delagoa Bay and Natal Division, concentrated on the performance of the equipment and its potential, but the report from the Captain of HMS Raccoon added:

‘I am strongly in favour of its suitability for communications between Her Majesty’s ships especially during patrolling operations of a similar character as those existing here. Had the presence of an enemy’s fleet to be considered, it would be a question if the possession of this instrument would not tend to disclose the proximity of your presence to the enemy.’

At its most basic, those thinking about foreign communication usage thought at the message level, rather than at the carrier level, and saw any potential value as deriving from messages transmitted during actions. Thus, in the 1902 Royal Navy fleet exercises, the Admiralty issued an instruction forbidding either fleet from acting on information derived from intercepted messages alone, because ‘in a real action, the signals would probably be unintelligible to the other side’; and in 1904, the British squadron in Chinese waters intercepted messages being passed between a Russian Admiral and his staff. The RN Commander reported to the Admiralty that ‘the character of the messages was not important, but the fact of interception showed possibilities that a belligerent's messages could be and probably would be intercepted’.

Radio procedures of foreign navies (French, German, Japanese, American and Italian) were studied to provide recognition: in 1904 British warships were instructed to transmit to the Admiralty copies of all foreign radio messages which they intercepted and in 1904 and 1905 the NID was informed of Intercept of Russian naval traffic by ships at Suez, the intercept of German naval traffic by ships at Corfu; and of Dutch stations by ships in the Baltic. Furthermore, the RN had acquired the French Naval Wireless Telegraphy Handbook of 1904; and by 1906, procedures used by the American and Japanese Navies had been added to the list.

The accepted view was that though wireless messages might be intercepted, encryption would render any intercepted message unreadable. In 1906 the commander of the Home Fleet (and later First Sea Lord) Arthur Wilson, stated that even if ‘the enemy has our signal books’, all messages in this compromised codebook would remain entirely secret so long as they were superenciphered by means of an elementary transposition cipher. In 1907 the head of the navy’s radio service and the Director of Naval Intelligence (DNI) both doubted that encoded radio traffic could ever be broken, since ‘under most circumstances it will only be necessary to cypher a few words in each message to render it quite unintelligible’.

The belief that encryption would render any message unreadable led to thinking in another direction: how to deny an enemy the possibility of wireless communication. Again, this thinking was not developed: jamming was not a possibility because it would mean jamming of own comms as well as of the enemy's. There was little thinking beyond the destruction from the sea of enemy coastal wireless stations and the cutting of enemy telegraphic cables.

Counter-jamming was studied: already by 1909 the first experiments in directional wireless telegraphy were being carried out by the Admiralty, at both shore stations and in ships at sea.

There had been no cryptanalytic organisation in the UK since 1844 but crucially, the pre-1844 Secret Offices looked only at civil and diplomatic material, not military. The Indian Army, following its experiences in South Africa, had set up a two man cryptanalytic bureau in India, but its function was to look at diplomatic and attaché traffic acquired from telegraph companies, something that, while legal in India, was illegal in the UK, and the product of which was of no interest to the British government.

In 1908 Fleet Paymaster Charles Rotter of the Naval Intelligence Division asked that any German Naval messages intercepted should be logged and forwarded to the NID. This work had certainly begun by 1909 and a file of this material certainly existed in 1973 in GCHQ though it was probably subsequently returned to the MOD and is now waiting to be discovered in an ADM piece in The National Archives.

The file contained raw traffic and work-sheets. Some of the raw traffic was recorded on specially printed forms headed ‘Return of Intercepted Wireless Telegraph Messages for the month of ...’ These returns were prepared at the point of interception, presumably by transcription from the actual logs, and sent to ‘Admiral Commanding Reserves’ by the ‘Chief Officer’.

The worksheets showed successful decryption of one encryption system, and work on callsigns and the collation of versions received from the different interception points. These interception points were: Whitehall (presumably the Admiralty), Aberdeen, Ipswich, Dover, Rosyth, Port Patrick, Pembroke Dock, St. Helier, Essex Hill, Hunstanton, and, at least from 1912, Scarborough. At the Admiralty end, monthly batches of returns as received from the stations were ‘referred for information’ to DID in person. What the latter made of the raw intercepts is not clear.

Callsigns were two-, three-, or four-letter. It was easy to identify many: for example RO, NA, AKO and AHO stood respectively for the ships ROON, NAUEN, KOENIGSBERG and HOHENZOLLERN. Frequencies were reported as ‘Wavelengths in feet’. They included 5000, 6000 and 6500 (the roundness of the figures suggests strongly that the measurements were pretty rough).

The monthly return forms had a column for ‘Remarks, Strength of Signals; If musical, etc.’ It was not much used, but some transmissions were described as ‘musical’ or ‘high musical’; others were described as ‘whistling note’ and ‘quenched spark’. Strengths were given in a scale up to 10 and ‘rates’ – presumably words-per-minute – were of the order of 12 to 15.

The traffic that was read at the time was a 10-letter code; each group consisted of two more or less pronounceable 5-letter ‘words’. There were clear indications that a German codebook had been acquired physically by some means, rather than that it had been reconstructed cryptanalytically. All the messages were originated by ‘Admiralstab’, callsign KMD (probably Kriegsmarinedienststelle – Naval HQ), using the main German naval transmitter at Norddeich. The messages refer to movements of German ships or to the deployment of British and other warships in the Mediterranean and home waters – and on one occasion to an eruption of Mount Etna! The readable sequence started in March 1910 and finished with the following message dated 11 July 1910: ‘This is the last telegram from the Admiralstab’. (Or might this have been a comment?)

There is no evidence that any other cypher was read at the time. Most of the unread traffic is in 5-letter transposition systems.

Conventional wisdom is that the Naval Intelligence Division had arranged for the Secret Service to purchase a copy of a German Naval codebook from an agent. £600 went into this venture but the codebook which eventually turned up was a forgery. I wonder, however, whether it isn’t just as likely that the codebook was used until 11 July 1910 at which point it was superseded.

Though this activity by Rotter was tolerated, he was acting in isolation. At his request, during the naval manoeuvres of 1912 and 1913 a separate division of the Admiralty War Room, known as Section C, had been assigned to work on intercepted wireless traffic, but nevertheless when the War Room was mobilised on 30 July 1914 it was simply decided that such a unit was unnecessary, and Section C was dropped.

Just before the advent of war, Cdr Frederick Loring RN, probably the Navy's expert in wireless telegraphy at the time, summed up why interception would not prove a serious threat to naval wireless in wartime.

‘... the more skilled the organisation, the more difficult it is for a strange operator to take down with the necessary accuracy the groups of a code message: he cannot ask for the repetition of doubtful groups, and he has no intimate and daily familiarity with the methods of his opponent to assist him in his task. And, after all, giving the enemy every advantage, giving him a perfect record of the signals, the key of the code to his hand and equal facility of skill and language to translate it for use—a most improbable combination, it must be admitted—he has still failed to prevent the all-important information reaching its destination.’

That Loring was wrong in 1914, just as the FSL and the DNI had been wrong in 1906 and 1907, shows how different Sigint is from any other sort of intelligence that existed in 1914 and how difficult it was to imagine it. A combination of the traffic analysis carried out in the War Office by Room 40 staff in September and October 1914, the capture of the Magdeburg Codebook and its arrival in London at the end of October, Rotter’s working out the superencryption system used by the German High Seas Fleet, and Winston Churchill’s imaginative insight into this totally new source meant that by 8 November Room 40 had been established as the first cryptanalytic bureau in the UK since 1844 and was ready to become the UK’s first Sigint organisation.

Collection Management

Collection Management evolved during the Second World War and was honed to a precision instrument during the Cold War. What I describe belongs to the HF era (and I’m only looking in detail at the Second World War here). The same principles apply today, but the way in which they are exercised is as totally different as are the comms in use in the twenty-first century. 

Even within the bounds of the laws of physics and the permissions a Sigint agency has, it can’t collect everything it would like to or even everything its customers say they would like to see; even so, it will always collect more than can be processed; what filters through the processing won’t all be intelligible to its analysts; and a significant amount of what is intelligible won’t actually be relevant.

The city on the hill towards which the collection management system is aiming is one in which the messages which are intercepted and processed are those which will allow the agency to produce the most important intelligence for the most relevant customers; and the smallest number of signals are intercepted which, because of irrelevance or unprocessability, won’t produce wanted intelligence.

The need for a Collection Management organisation arose because of what was at the time referred to as the ‘Enigma Complex’. The value of Enigma decrypts meant that the GC&CS view, that Enigma cryptanalysis was a single problem that needed to be managed centrally, won out over the service view that their collection facilities were theirs to control.

The basic problem for collection management was that at any time there might be ten or twenty enemy transmitters operating for every interception set, and that there weren’t enough intercept operators to operate each interception set 24 hours a day. Overcoming this issue meant the development of an understanding of how the Germans organised their communications networks, how within that organisational structure the individual networks operated, and an assessment of the value of intercepting some or all of the communications passed on that network.

To achieve this, the collection management system had to look two ways: towards the intercept stations to understand what they could collect given both their location and the limitations of their infrastructure; and towards the analysts to understand what was the minimum of allocation of resources that would provide them with enough (sometimes just enough) information.

Some examples: there was little interest in the content of German weather messages, but sent out at the start of the day, and repeated through various networks which used different crypto key settings, it meant that certain weather messages could be decisive for recovering the daily key for a number of networks, and in order to have the cleanest possible version of the message that had been sent, three or even four intercept stations would have a set and an operator copying it. Conversely, a garrison located in the Baltic where there was little or no operational activity might simply be sampled once a week or month, simply to maintain continuity and confirm that the garrison hadn’t moved or changed its comms procedures. During operations, Direction Finding might be the key contributor in the Sigint system, able to say exactly where a unit or formation was (as happened with the Bismarck on 25 May 1941), but was of limited use, and in fact a waste of time and resource, against permanent locations such as those serving Hitler’s Headquarters in the Wolfsschanze.

These examples are extremes. Collection management normally depended on a group of people who understood and trusted each other to manage allocation of receivers according to current operational requirements, the technical sophistication of the transmission, the need for continuity and the laws of physics. Collection would normally be carried out in military stations, and as a rule each service collected the traffic of its counterpart, but this was not a hard and fast rule. Diplomatic and Commercial Sigint, based, of course, in London rather than at Bletchley Park had dedicated civilian-staffed intercept sites, but its collection management staff were at Bletchley to ensure that the entirety of collection resources could be placed against all of Sigint’s targets – two extra brownie points for knowing that this is what Mary and Valerie Glassberrow, the grandmother and great-aunt of the Princess of Wales, were doing at Bletchley Park.

The key takeaway from this isn’t its size, or its complexity, or its flexibility, or that everybody involved in it had to take on trust the relative importance of a target at any time, or that application of the ‘need-to-know’ principle would have stymied this progress from the start: it is the fact that this process was invisible to everybody outside the Sigint organisation. Any bean counter observing the organisational structure might simply ask if all those posts (perhaps a hundred all told) might be abolished and a system set up in which analysts would simply task stations themselves. Well, they could be abolished, and the analysts’ tasking would be productive for a day or two, but with no-one to adjudicate between competing demands for receiver time (never mind the absence of a sophisticated process which worked across all targets and all stations) the value and quality of intercept, and of the intelligence produced from it, would decline rapidly.

We often describe Sigint production as a chain: stations intercept messages and send them to HQ where the intercept is processed before being presented to cryptanalysts and/or traffic analysts whose output goes (through linguists if the resultant text is in a foreign language) to intelligence analysts who issue a report to customers saying what the originator of the message said, but it is a satisfactory as an analogy only in the broadest sense. Collection management isn’t alone in being an internal function which is invisible to those who simply see the chain. Nigel de Grey includes, in what he calls the ‘Ancillary Sections’: the Machine Tabulating Section, Communications, the Central Signals Registry, the Signals Office, Personnel, Recruitment, Administration, Works and Buildings, and, of course, Security.

I blogged the first part of what I thought would be a two parter on Sigint Communications a while back. I got stuck on Part 2 trying to work out how to summarise it in fewer that five or six thousand words. These ancillary sections are key to the success of Bletchley Park during the war and for GCHQ’s success subsequently, and their story is as complex as those of the well known Huts. Sadly, these stories are seldom told.

The Bletchley Chapter of 'People Against Uncongenial Work'

Much of the success of Bletchley Park during the Second World War is owed to the combination of great talent and an atmosphere in which it could flourish: a combination of the Common Room of a college, the managed anarchy of the tiny pre-war GC&CS, and the respect for – or at least the toleration of – people who didn’t conform to the societal norms and expectations of mid-twentieth century Britain.

But that isn’t the whole story. Nigel de Grey, in his Organisation and Evolution of British Sigint (TNA HW 75-78) refers at times to the unwillingness of some of the people who benefited from BP’s toleration of individualism to take their share of necessary chores, the necessity of which was not obvious to them. Here are two examples.

‘The Tube Central

The Pneumatic Tube System was installed in order to overcome the delay in distribution attendant upon the use of messengers, the only labour available for which were girls from local primary schools on leaving, whose hours of work were strictly controlled by Civil Service regulations and whose enthusiasm and discipline was not noticeably high, and to take the place of the belt conveyors which had been in operation to adjacent sections. The principle adopted was to have four main trunk double lines to the four requisite office blocks terminating at a point where a single line internal system conveyed the traffic to the several requisite points within the office block. The "cross country" tubes were laid underground and the distances were in some cases the maximum possible, hence the necessity for relaying within the blocks. The system suffered from two main troubles, firstly that the bore of the tubes and hence of the containers was not large enough to cope with the documents to be distributed or indeed to carry the overall volume; secondly that owing to indifferent manning of the local stations by the user sections the maximum was never got from the system. Communications Section was forced eventually to man both ends of the trunk lines but the interior terminals were as a rule manned by casual, not to say extremely casual, labour and Communications Section received very little cooperation from the Sections, some of which seemed to find any routine beyond their capabilities or infra dig.

Time stamping Machines

Analogous to this laxity was the difficulty experienced by Communications Section over the Stromberg time-stamping machinery. The regular and rapid flow of papers (decrypts, signals and so forth) was an essential to the main function of GC&CS, namely decrypting and reporting to the combat commands. To achieve this it was essential to time the arrival and departure of documents at each stage of the handling so that any bottle neck should at once be revealed and steps taken to relax it. In order to make this easy electric time-stamps were introduced in May 1943 (when, it will be remembered, GC&CS was "working-up" for the Second Front) and installed at strategic points in the Cypher Office, the Main Teleprinter Room, the Auto Room, the Tube Central, Huts 3, 6 and 8, Naval Section and Air Section. To avoid all discrepancies the clocks in these machines had to be synchronised daily, a routine for which was laid down on the principle of dialling TIM in London. The minutes of the Communications Committee show the struggle that Communications Section had to get the Sections to conform to this very simple routine. All sections were careless but Hut 6, a key point, consistently ignored not only the synchronisation of their clocks for weeks at a time but the use of the stamp at all. Thus when in the autumn of 1943 Communications Section were straining every nerve to increase the rapidity of their service to Hut 6 and asked for the cooperation of Hut 6 in testing the results of their experiments in handling, the tests broke down three times because Hut 6 would not use the time stamps properly. Hut 8, another key point, two months after the institution of the time stamps was found to have locked its Stromberg up in a cupboard and left it there. Another section's clock had stopped and never been used again. Thus the two worst offenders were the two sections which employed the highest grades of labour, both being manned principally by university trained personnel who had no doubt been taught to think for themselves. The instances quoted, perhaps trivial in a sense, were none the less symptomatic of the difficulty of getting any businesslike routine carried out, any suggestion of mechanisation in an organisation manned by untrained labour and with whom no disciplinary action could be taken except that of dismissal should they prove insensible to an ordinary reprimand.’

The voices of the educated – ‘university trained personnel who had no doubt been taught to think for themselves’ – tend to predominate in the story of Second World War British Sigint, and it’s too late to gather the memories of those trying to improve the information flow using Time and Motion methods such as the use of Stromberg time stamping machines; and, anyway, the value of the results of the graduates’ work in cryptanalysis and machine-based solutions on any day probably outweighed any benefit potentially lost. But nevertheless qualified people with skills which might have improved the way that BP worked were being ignored.

And this wasn’t the worst thing that was happening. As de Grey hints in telling the story of 14 year old ‘girl’ messengers, and as is clear from any telling of the stories of (for example) cipher operators, not all jobs at BP were stimulating or interesting; not all needed great minds; and many were as dreary as might have been found in any factory job to which labour had been directed, but with the additional pressure that those working (mainly women) on these tasks could say next to nothing outside work about the dreary, mundane nature of their working lives.

A necessary consequence of the industrialisation of UK Sigint during the Second World War was the creation of production line jobs which, like those in contemporary factories, were not stimulating. Add to that a need-to-know system, in which the people doing the most menial jobs knew the least about the way in which their work contributed to the organisation as a whole, and it is not hard to see that without wartime regulations on directed labour, Bletchley Park might have found it difficult to achieve the success it did.