Portrait of the NSA: no detail too small in quest for total surveillance

The NSA gathers intelligence to keep America safe. But leaked documents reveal the NSA's dark side – and show an agency intent on exploiting the digital revolution to the full

Ewen MacAskill and James Ball

Barack Obama hailed United Nations secretary general Ban Ki-moon as a "good friend" after the two had sat down in the White House in April to discuss the issues of the day: Syria and alleged chemical weapons attacks, North Korea, Israel-Palestine, and climate change.

But long before Ban's limousine had even passed through the White House gates for the meeting, the US government knew what the secretary general was going to talk about, courtesy of the world's biggest eavesdropping organisation, the National Security Agency.

One NSA document – leaked to the Guardian by whistleblower Edward Snowden just a month after the meeting and reported in partnership with the New York Times - boasts how the spy agency had gained "access to UN secretary general talking points prior to meeting with Potus" (president of the United States). The White House declined to comment on whether Obama had read the talking points in advance of the meeting.

Spying on Ban and others at the UN is in contravention of international law, and the US, forced on the defensive this week over the Snowden leaks about worldwide snooping, ordered an end to surveillance of the organization, according to Reuters.

That the US spied on Ban is no great surprise. What is a revealing is that the disclosure is listed in the NSA's 'top-secret' weekly report from around the world as an "operational highlight".

It sits incongruously alongside other "operational highlights" from that week: details of an alleged Iranian chemical weapons program; communications relating to an alleged chemical weapons attack in Syria and a report about the Mexican drug cartel Los Zetas.

Bracketing the benign, US-friendly Ban alongside drug traffickers and weapons in the Middle East and Central Asia points to a spy agency that has lost its sense of proportion.

The incident is consistent with the portrait of the NSA that emerges from the tens of thousands of documents leaked by Snowden. Page after page shows the NSA engaged in the kind of intelligence-gathering it would be expected to carry out: eavesdropping on Taliban insurgents planning attacks in remote Afghanistan valleys, or listening in on hostage-takers in Colombia.

But the documents reveal, too, the darker side of the NSA. It is indiscriminate in the information it is collecting. Nothing appears to be too small for the NSA. Nothing too trivial. Rivals, enemies, allies and friends – US citizens and 'non-Americans' – are all scooped up.

The documents show the NSA, intent on exploiting the communications revolution to the full, developing ever more intrusive programmes in pursuit of its ambition to have surveillance cover of the whole planet: total command of what the NSA refers to as the 'digital battlefield'.

'Graying and shrinking'

When the NSA was founded in 1952, its task was primarily to target the Soviet Union.

And so it did, decade after decade, until the fall of the Berlin Wall in 1989 and the end of the cold war soon afterwards.

With the collape of the Soviet Union, the NSA entered a decade of uncertainty. Morale slumped. The mood is caught in a document dated February 2001, only a few months before 9/11. In it, the agency admitted its capacity for intercepting electronic communications had been eroded during the 90s.

"NSA's workforce has been graying and shrinking. The operational tools have become antiquated and unable to handle the emerging signal structure," it says.

"Ten years ago we had a highly skilled workforce with intimate knowledge of the target and the tools to analyse the data.

"We have now reached the point of having a workforce where the majority of analysts have little-to-no experience."

Tellingly, in the light of the attacks on New York and Washington six months later, the document complained about a lack of linguists and analysts covering Afghanistan. The same pool of experts covering Afghanistan as a whole were the same that "assist NSA's Office of Counter-terrorism in following the Taleban-Usama bin Laden relationship", it said.

'Sanitize personal effects'

The NSA refers to the people it serves as 'external customers'. Photograph: Felix Clay

The attacks on New York and Washington ended the NSA's decade of torpor. Suddenly, it found funding, and staff recruitment was no longer a problem. Since 9/11, expansion has been rapid. The NSA was one of the main beneficiaries of the doubling of the intelligence budget since 9/11.

Its proposed budget allocation for 2013 is $10.8bn, with 35,000 staff and bases in Georgia, Texas, Colorado, Hawaii and Utah adding to its headquarters at Fort Meade, Maryland. Its antennae can be found on the rooftops of 80 American embassies around the world.

It has large posts in the UK, Australia and Japan, but also operates elsewhere, sometimes covertly. In one country, Americans are secretly present at a base where exposure of their presence would provoke a major diplomatic incident, as it is in breach of an international treaty signed by the NSA's host nation. Agency staff visiting the base have to hide their real identities, posing as contractors working on communications equipment and carrying fake business cards to back up their story.

A PowerPoint briefing warns staff heading to this secret base: "Know your cover legend". It urges them to "sanitize personal effects" and to send no postcards home. Nor should they take souvenirs home with them. The NSAbriefing makes an exception for jewellery, because "most jewellery does not have markings identifying it" as coming from that country.

The NSA refers to the people it serves as "external customers": the White House, the State Department, the CIA, the US mission to the UN, the Defense Intelligence Agency and others.

Its remit for those customers has become ever more complex. During the cold war, the NSA mainly targeted state institutions: the political, military and intelligence structures of Russia and Eastern Europe. Today, the main targets – al-Qaida and its related groups – are much more diffuse and elusive.

The NSA sets out its mission statement in its current five-year plan. In it, the agency insists Sigint (signals intelligence, or the interception of communications) will adhere to the highest standards. "Sigint professionals must hold the moral high ground, even as terrorists or dictators seek to exploit our freedoms. Some of our adversaries will say or do anything to advance their cause; we will not."

Summing up the reason for its existence, it says: "Our mission is to answer questions about threatening activities that others mean to keep hidden."

But its actual scope goes well beyond that. It is hard to see where surveilling Ban Ki-moon or German chancellor Angela Merkel fits into answering questions about "threatening activities".

Mission creep

At a press conference in August, Obama defended the NSA and defined its role in narrow terms. He described the agency's remit purely as counter-terrorism. "We do not have an interest in doing anything other than that," he said.

The remark was striking. Counter-terrorism has been the justification for huge budget increases, but the agency is involved in much more than that. The NSAdiscloses in one leaked document that only 35% of available resources are dedicated to the 'global war on terrorism'.

Obama later amended his statement. The NSA was not only engaged in counter-terrorism, he said, but also cyber-security and combating weapons of mass destruction. Even this does not begin to capture the sheer variety and reach of NSA operations.

Its own list of strategic targets includes: support for US military in the field; gathering information about military technology; anticipating state instability; monitoring regional tensions; countering drug trafficking; gathering economic, political and diplomatic information; ensuring a steady and reliable energy supply for the US; and ensuring US economic advantage. It boasts it can collect information from "virtually every country".

Hundreds of the documents show the NSA engaged in activities that would generally be applauded. One credits the NSA's Texas base as intercepting 478 emails that helped to foil the Jihad Jane plot to kill Swedish artist Lars Vilks over his depiction of the prophet Muhammad.

Another shows the NSA, during a deadly takeover of the Intercontinental Hotel in Kabul by the insurgent Haqqani group, able to listen in, minute-by-minute, to what the gunmen were saying.

There is an account, too, of the NSA's part in disrupting a human trafficking racket based in Fuzhou, China. It led to two arrests at New York's JFK airport. One of those lifted allegedly carried details of the smuggling routes in his pocket.

Remote surroundings might fool some into thinking they are beyond snooping. An alleged cocaine smuggler might have thought he was relatively safe aboard a yacht in the Caribbean. But he failed to take account of the fact that his partner, also on board, was chatting on Facebook, providing valuable information about the boat's location and planned landfall; information intercepted by one of the NSA's intelligence partners.

Nor is the Iranian leadership beyond reach. In 2009, the NSA was was able to track almost every move made by Iran's supreme leader, the Ayatollah Ali Khamenei, on a rare visit outside Tehran to the mountainous Kordestan province.

The most valuable service the NSA has provided for America and its allies since 9/11 is in support of the military in Iraq and Afghanistan. A 2007 NSA file, called 'State of the Enterprise', is typical of many of the spy agency's documents which list wartime successes.

"Specific results included the identification and location of a sniper targeting personnel inside the Baghdad Green Zone; the confirmation that a CIA asset was operating as a potential 'bad actor'."

Other intelligence agencies such as the CIA complain privately about the degree of co-operation from the NSA in sharing intelligence, but in the end, like most other intelligence agencies, it is generally thankful for it. There are complaints, too, from soldiers in the field that live information is not always transferred to them fast enough, but they, too, express gratitude for snippets passed on about potential Taliban attacks.

The NSA, according to one document, overheard a Taliban figure, Mullah Rahimullah Akhund, known on the US military's kill-or-capture list by the codename Objective Squiz Incinerator, instructing an associate to buy and organize components for a roadside bomb, suicide vests and a Japanese motorbike.

The appreciation of Americans and their allies in Afghanistan for such information is summed up in this letter back to headquarters: "You guys/gals probably have no idea how much we rely on your tool for enabling our CT (counter-terrorism) capture operations in Afghanistan. It really does help us get our enemies off the playing field, so to speak."

Technological spread

When the NSA, the CIA and other parts of the intelligence community spied illegally on American anti-war protesters, civil rights leaders and trade unionists in the 1970s, there was at least a technical limitation of their actions. The difference today is that technological revolution allows them to spy on almost everyone.

The expansion in surveillance that accelerated under George W Bush has continued under Barack Obama. And this growth has not been matched by any corresponding reform of the legal framework or political oversight.

While there are frequent warnings in the documents reminding NSA staff of rules for protecting the privacy of Americans, other documents show repeated violations. Such violations are almost inevitable given the way the NSA collects so much, the technology and analysts unable to distinguish between data on foreigners and American citizens.

The NSA says in public it only collects a tiny percentage of internet traffic, smaller than "a dime on a basketball court". But there is a gulf between what the NSA says in public and what it says in documents, in which technicians and analysts express their glee at finding novel ways of cracking into electronic communications and expanding their reach in ever more imaginative ways.

The question critics of the NSA raise is: just because it has the technical ability to do these things, should it?

One document shows the NSA engaged in a massive snooping operation targeting a United Nations climate change conference in Bali in 2007.

Ban, speaking at the conference, which attracted thousands from around the world, described combating climate change as "the moral challenge of our generation".

However, the NSA's Australian base at Pine Gap was less interested in combating climate change than collecting the numbers of Indonesian security officials in case of a future emergency.

"Highlights include the compromise of the mobile phone number" for one senior Balinese official, an NSA report boasted. "Site efforts revealed previously unknown Indonesian communications networks and postured us to increase collection in the event of a crisis."

This effort-filled collection of the cell phone number falls under the category of information that spies have always gathered. The rationale is: should there be an attack at the conference or some future outrage, such numbers could be valuable. The counter-argument is that Indonesia is a friend of the US and might be expected to share information in the event of an attack, so why does the NSA devote grand resources to harvesting such numbers?

One of the biggest criticisms of bulk data collection is that the agency cannot look at, let alone analyse, all the data it is collecting. One document echoed the problems the agency faced in 2001 when it lamented the lack of linguists pre-9/11. An officer, after checking some messages that might have been from a terrorist group, admitted: "Most of it is in Arabic or Farsi, so I can't make much of it."

The 5-Eyes

The NSA operates in close co-operation with four other English-speaking countries - the UK, Canada, Australia and New Zealand - sharing raw intelligence, funding, technical systems and personnel. Their top level collective is known as the '5-Eyes'.

Beyond that, the NSA has other coalitions, although intelligence-sharing is more restricted for the additional partners: the 9-Eyes, which adds Denmark, France, the Netherlands and Norway; the 14-Eyes, including Germany, Belgium, Italy, Spain and Sweden; and 41-Eyes, adding in others in the allied coalition in Afghanistan.

The exclusivity of the various coalitions grates with some, such as Germany, which is using the present controversy to seek an upgrade. Germany has long protested at its exclusion, not just from the elite 5-Eyes but even from 9-Eyes. Minutes from the UK intelligence agency GCHQ note: "The NSA's relationship with the French was not as advanced as GCHQ's … the Germans were a little grumpy at not being invited to join the 9-Eyes group".

Significantly, amid the German protestations of outrage over US eavesdropping on Merkel and other Germans, Berlin is using the controversy as leverage for an upgrade to 5-Eyes.

The NSA's closest ties are with the GCHQ. Documents suggest the British contribution is significant. In a random selection of NSA documents monitoring weekly reports, the British agency is frequently listed alongside the US agency's biggest regional bases such as Texas and Georgia.

GCHQ operates a vast internet tapping operation based on partnerships between the UK government and telecoms companies based in the UK and overseas. This allows the NSA to "touch" about 90% of the traffic crossing the UK.

Given the UK's location, this is a huge proportion of the internet: the UK hosts one of the major transatlantic internet cables, as well as numerous cables connecting Europe and the Middle East. Each day, a quarter of all internet traffic traverses the UK.

The information collected and stored by the programme, codenamedTempora, is stored by GCHQ for up to a month, with NSA analysts granted direct access to the intelligence.

The NSA – in theory at least – operates inside a legal framework that requires warrants to target Americans. But the Fisa court turns down few such requests. GCHQ operates in an even looser environment. One GCHQ document, referring to UK oversight, says: So far they have always found in our favour."

A GCHQ legal briefing suggests some of the distinctions stressed in policy documents and public statements by staff of both agencies may not be so rigorously enforced in practice. A lengthy legal training slideshow includes several slides explaining the often-complex differences between content andmetadata, which requires substantially different handling, especially under US law.

However, the notes for the presentation say: "GCHQ policy is to treat it pretty much all the same, whether it's content or metadata."

The blurred boundaries are acknowledged, too, in NSA documents, one of which states: "It is often unclear whether individual communication elements, particularly content-related metadata (CRI) – information derived from the message body – is content or metadata? For example, are email subject lines metadata or content? What about an email's signature block or telephone numbers within a message? Questions like these are not necessarily clear-cut."

Gaining access to the huge classified data banks appears to be relatively easy. Legal training sessions – which may also be required for access to information from Australian, Canadian, or New Zealand agencies – suggest that gaining credentials for data is relatively easy. The sessions are often done as self-learning and self-assessment, with "multiple choice, open-book" tests done at the agent's own desk on its "iLearn" system. Agents then copy and paste their passing result in order to gain access to the huge databases of communications.

Conclusion

The NSA, once the most secretive of the 16 US intelligence agencies but now embarrassingly penetrated as a result of Snowden, is facing more scrutiny that at any time since its founding, even more than during the domestic spying scandals of the 1970s.

It is being challenged in Congress. It is being challenged in the courts by an unholy alliance of the liberal American Civil Liberties Union and the right-leaning National Rifle Association. It is coming under pressure from the internet companies to be more transparent. And there is review panel announced by Obama in August. There is also pressure from Germany and France, Mexico and Brazil.

In spite of the furore, reforms may prove modest. The agency is hardly likely to easily relinquish its new-found capability of snooping almost everywhere.

In one of the leaked 'State of the Enterprise' documents from 2007, an NSAstaff member says: "The constant change in the world provides fertile ground for discovering new targets, technologies and networks that enable production of Sigint."

The official happily embraces this: "It's becoming a cliché that a permanent state of change is the new standard. It is the world we live in – navigating through continuous whitewater."

It's an environment in which the NSA thrives, the official says. And adds: "Lucky for us."

http://www.theguardian.com/world/2013/nov/02/nsa-portrait-total-surveillance

•