During his seven years as head of MI5, Andrew Parker has dealt with Islamist terror plots, Russian and Chinese espionage and the rise of far-right white extremism. 
But the biggest challenge, he said, is technology and how it is transforming how the UK Security Service operates. This will require a top-to-bottom examination of MI5’s structures, its relationship with the private sector and a shift to targeted monitoring of terror suspects “in real time”. 
In an interview with the Financial Times, Sir Andrew — who is stepping down as director-general of MI5 in April — suggested his agency would have to form partnerships with retailers, banks and internet companies to access information about the 3,000 people it has under close investigation at any one time. 
“The private sector has way more information and data about any of us than MI5 would ever have about the individuals we’re interested in,” he said. “I’m talking about our focused interest in the few thousand people that . . . we need to know more about.” 
The next step by MI5 is to use machine learning to extract useful intelligence from this vast quantity of information held by the private sector. The Treasury has established a security investment fund so that the UK intelligence agencies can nurture start-ups developing the tools they need. 
Sir Andrew said he is particularly interested in artificial intelligence “because of our need to be able to make sense of the data lives of thousands of people in as near to real time as we can get to”.
But developments in this area will have to be carefully managed: last year, the UK surveillance watchdog took MI5 to task over its “unlawful” handling of personal data, saying the agency had “inadequate control” over its computer databases. 
In a calibrated nod to transparency, Sir Andrew has invited the FT to what he described as the “panelled, cosseted splendour” of his office high up in MI5’s London headquarters to discuss his tenure as head of the agency.
Only two months after he took charge in 2013, the intelligence world was shaken by revelations from former US National Security Agency contractor Edward Snowden, whose leaks exposed mass surveillance of internet communications by spy agencies. 
Arrangements are [now] exactly the same for dealing with Islamist terrorism and extreme, far-right terrorism and in some ways it’s not an encouraging place for society to have reached
The result was a period of intense scrutiny for Britain’s three intelligence agencies. In his first year as MI5 director-general, Sir Andrew appeared alongside his counterparts at MI6 and GCHQ for the first ever parliamentary grilling of security chiefs in public.
MI5 came under further pressure after the five UK terror attacks of 2017 — four in London, one in Manchester — in which 36 people were killed.
A review into the atrocities by a former terrorism watchdog suggested that MI5 had lessons to learn, saying the Manchester Arena attack in particular “might have been averted had the cards fallen differently”. Sir Andrew said at the time that the Islamist threat was at the “highest tempo” of any point in his career. 
Emerging from this turbulent period, Sir Andrew seems to have cautiously embraced greater openness, notably as he seeks a more diverse set of employees and aims for increased partnership with the private sector.
The MI5 head — who attended a comprehensive school in Newcastle before studying natural sciences at Cambridge university — seems genuinely enthusiastic about broadening the skills and talents found in his 4,000 strong workforce.
New initiatives include internships aimed at bringing in more recruits from socially-disadvantaged families and ethnic backgrounds.
But Sir Andrew’s views on the private sector are more nuanced. On the one hand, he knows MI5 will have to form partnerships with companies to develop the technology it requires. But he is outspoken on the need for social media companies to share data relating to terror suspects. 
These groups have an “ethical responsibility” to provide the means for intelligence agencies to view encrypted messages in exceptional cases, he said.
“If the barrier we hit . . . is in the commercial space, where commercial decisions taken by commercial companies prevents us being able to get life-saving intelligence, I think that is unacceptable,” added Sir Andrew. 
While technology has been the major driver of change during his 36 years at MI5, the threats have altered too.
Sir Andrew has worked across the broad range of the agency’s operations, covering international terrorism and the IRA, counter-espionage, and organised crime. His teams were instrumental in disrupting an al-Qaeda plot in 2006 to bomb aircraft with explosives hidden in drinks bottles. 
[Technology] will never replace our need to also have human insight, because technology and data will never tell us what is going on in people’s heads
The Islamist threat is still potent, but resources are now being diverted to a different group of extremists: in the past year MI5 has taken over responsibility from the police for managing the risk posed by rightwing terrorists. Of the six terror plots foiled by the UK intelligence agencies in 2019, half involved those on the far-right. 
“It’s fed by a shared sense of hatred of anything that isn’t white and British in many cases,” said Sir Andrew, referring to rightwing extremism. “It’s completely an intolerable thing in our society . . . arrangements are [now] exactly the same for dealing with Islamist terrorism and extreme, far-right terrorism and in some ways, it’s not an encouraging place for society to have reached.” 
Another significant change is that China has emerged alongside Russia as a perpetrator of espionage against British academia and industry.
It is a particularly sensitive subject for the UK government, because, as Sir Andrew noted, China is not positioning itself as “an enemy of Britain. It is a massive economic partner for Britain.”
But he said China has proved “willing to do things which can harm the security” of the UK and other countries, adding: “It’s not setting out to do that as an aim . . . but it’s willing for that to happen for the sake of achieving its larger goals, which are economic.”
This action by China took the form of cyber espionage in pursuit of intellectual property, particularly in defence, as well as trying to recruit British citizens as agents. “It’s that part of the functioning of the Chinese state that it’s our job to get in the way of,” said Sir Andrew.
Looking ahead, he clearly has high hopes for AI to revolutionise spycraft, but he is emphatic that MI5 officers will always be needed to make the final interpretation of intelligence.
The attack at London Bridge last year, in which two people died, was carried out by a convicted terrorist, and exposed the difficulties in judging the risks posed by known offenders.
“[Technology] will never replace our need to also have human insight, because technology and data will never tell us what is going on in people’s heads,” said Sir Andrew. “What people’s real intent and thoughts are, that they might share with another human being.”