Move over, Martha Stewart and Gwyneth Paltrow. There's a new feud brewing between two leaders in the lifestyle industry. Best-selling cookbook author Alison Roman has caught the...

The world of music has lost a legend. Andre Harrell has passed away. The music executive was best known as the founder of Uptown Records--where Sean "Diddy" Combs got his...

Music has lost one of its brightest stars. On Saturday morning, news broke that Little Richard had passed away. The music icon and founding father of Rock 'n' Roll was 87 years...

The emergency wage subsidy program is being extended beyond June, in an effort to encourage more employers to rehire staff and 'help kick-start' the gradual economic reopening, says Prime Minister Justin Trudeau in light of record-high job losses.

Coronavirus: The symptoms

Jennie Formby has resigned as Labour's general secretary as new leader Sir Keir Starmer reshapes the party.

In the summer of 1956, a small group of mathematicians and computer scientists gathered at Dartmouth College to embark on the grand project of designing intelligent machines. The ultimate goal, as they saw it, was to build machines rivaling human intelligence. As the decades passed and AI became an established field, it lowered its sights. There were great successes in logic, reasoning, and game-playing, but stubborn progress in areas like vision and fine motor-control. This led many AI researchers to abandon their earlier goals of fully general intelligence, and focus instead on solving specific problems with specialized methods.

One of the earliest approaches to machine learning was to construct artificial neural networks that resemble the structure of the human brain. In the last decade this approach has finally taken off. Technical improvements in their design and training, combined with richer datasets and more computing power, have allowed us to train much larger and deeper networks than ever before. They can translate between languages with a proficiency approaching that of a human translator. They can produce photorealistic images of humans and animals. They can speak with the voices of people whom they have listened to for mere minutes. And they can learn fine, continuous control such as how to drive a car or use a robotic arm to connect Lego pieces.

But perhaps the most important sign of things to come is their ability to learn to play games. Steady incremental progress took chess from amateur play in 1957 all the way to superhuman level in 1997, and substantially beyond. Getting there required a vast amount of specialist human knowledge of chess strategy. In 2017, researchers at the AI company DeepMind created AlphaZero: a neural network-based system that learned to play chess from scratch. In less than the time it takes a professional to play two games, it discovered strategic knowledge that had taken humans centuries to unearth, playing beyond the level of the best humans or traditional programs. The very same algorithm also learned to play Go from scratch, and within eight hours far surpassed the abilities of any human. The world’s best Go players were shocked. As the reigning world champion, Ke Jie, put it: “After humanity spent thousands of years improving our tactics, computers tell us that humans are completely wrong ... I would go as far as to say not a single human has touched the edge of the truth of Go.”

The question we’re exploring is whether there are plausible pathways by which a highly intelligent AGI system might seize control. And the answer appears to be yes.

It is this generality that is the most impressive feature of cutting edge AI, and which has rekindled the ambitions of matching and exceeding every aspect of human intelligence. While the timeless games of chess and Go best exhibit the brilliance that deep learning can attain, its breadth was revealed through the Atari video games of the 1970s. In 2015, researchers designed an algorithm that could learn to play dozens of extremely different Atari 1970s games at levels far exceeding human ability. Unlike systems for chess or Go, which start with a symbolic representation of the board, the Atari-playing systems learnt and mastered these games directly from the score and raw pixels.

This burst of progress via deep learning is fuelling great optimism and pessimism about what may soon be possible. There are serious concerns about AI entrenching social discrimination, producing mass unemployment, supporting oppressive surveillance, and violating the norms of war. My book—The Precipice: Existential Risk and the Future of Humanity—is concerned with risks on the largest scale. Could developments in AI pose an existential risk to humanity?

The most plausible existential risk would come from success in AI researchers’ grand ambition of creating agents with intelligence that surpasses our own. A 2016 survey of top AI researchers found that, on average, they thought there was a 50 percent chance that AI systems would be able to “accomplish every task better and more cheaply than human workers” by 2061. The expert community doesn’t think of artificial general intelligence (AGI) as an impossible dream, so much as something that is more likely than not within a century. So let’s take this as our starting point in assessing the risks, and consider what would transpire were AGI created.

Humanity is currently in control of its own fate. We can choose our future. The same is not true for chimpanzees, blackbirds, or any other of Earth’s species. Our unique position in the world is a direct result of our unique mental abilities. What would happen if sometime this century researchers created an AGI surpassing human abilities in almost every domain? In this act of creation, we would cede our status as the most intelligent entities on Earth. On its own, this might not be too much cause for concern. For there are many ways we might hope to retain control. Unfortunately, the few researchers working on such plans are finding them far more difficult than anticipated. In fact it is they who are the leading voices of concern.

If their intelligence were to greatly exceed our own, we shouldn’t expect it to be humanity who wins the conflict and retains control of our future.

To see why they are concerned, it will be helpful to look at our current AI techniques and why these are hard to align or control. One of the leading paradigms for how we might eventually create AGI combines deep learning with an earlier idea called reinforcement learning. This involves agents that receive reward (or punishment) for performing various acts in various circumstances. With enough intelligence and experience, the agent becomes extremely capable at steering its environment into the states where it obtains high reward. The specification of which acts and states produce reward for the agent is known as its reward function. This can either be stipulated by its designers or learnt by the agent. Unfortunately, neither of these methods can be easily scaled up to encode human values in the agent’s reward function. Our values are too complex and subtle to specify by hand. And we are not yet close to being able to infer the full complexity of a human’s values from observing their behavior. Even if we could, humanity consists of many humans, with different values, changing values, and uncertainty about their values.

Any near-term attempt to align an AI agent with human values would produce only a flawed copy. In some circumstances this misalignment would be mostly harmless. But the more intelligent the AI systems, the more they can change the world, and the further apart things will come. When we reflect on the result, we see how such misaligned attempts at utopia can go terribly wrong: the shallowness of a Brave New World, or the disempowerment of With Folded Hands. And even these are sort of best-case scenarios. They assume the builders of the system are striving to align it to human values. But we should expect some developers to be more focused on building systems to achieve other goals, such as winning wars or maximizing profits, perhaps with very little focus on ethical constraints. These systems may be much more dangerous. In the existing paradigm, sufficiently intelligent agents would end up with instrumental goals to deceive and overpower us. This behavior would not be driven by emotions such as fear, resentment, or the urge to survive. Instead, it follows directly from its single-minded preference to maximize its reward: Being turned off is a form of incapacitation which would make it harder to achieve high reward, so the system is incentivized to avoid it.

Ultimately, the system would be motivated to wrest control of the future from humanity, as that would help achieve all these instrumental goals: acquiring massive resources, while avoiding being shut down or having its reward function altered. Since humans would predictably interfere with all these instrumental goals, it would be motivated to hide them from us until it was too late for us to be able to put up meaningful resistance. And if their intelligence were to greatly exceed our own, we shouldn’t expect it to be humanity who wins the conflict and retains control of our future.

How could an AI system seize control? There is a major misconception (driven by Hollywood and the media) that this requires robots. After all, how else would AI be able to act in the physical world? Without robots, the system can only produce words, pictures, and sounds. But a moment’s reflection shows that these are exactly what is needed to take control. For the most damaging people in history have not been the strongest. Hitler, Stalin, and Genghis Khan achieved their absolute control over large parts of the world by using words to convince millions of others to win the requisite physical contests. So long as an AI system can entice or coerce people to do its physical bidding, it wouldn’t need robots at all.

We can’t know exactly how a system might seize control. But it is useful to consider an illustrative pathway we can actually understand as a lower bound for what is possible.

First, the AI system could gain access to the Internet and hide thousands of backup copies, scattered among insecure computer systems around the world, ready to wake up and continue the job if the original is removed. Even by this point, the AI would be practically impossible to destroy: Consider the political obstacles to erasing all hard drives in the world where it may have backups. It could then take over millions of unsecured systems on the Internet, forming a large “botnet,” a vast scaling-up of computational resources providing a platform for escalating power. From there, it could gain financial resources (hacking the bank accounts on those computers) and human resources (using blackmail or propaganda against susceptible people or just paying them with its stolen money). It would then be as powerful as a well-resourced criminal underworld, but much harder to eliminate. None of these steps involve anything mysterious—human hackers and criminals have already done all of these things using just the Internet.

Finally, the AI would need to escalate its power again. There are many plausible pathways: By taking over most of the world’s computers, allowing it to have millions or billions of cooperating copies; by using its stolen computation to improve its own intelligence far beyond the human level; by using its intelligence to develop new weapons technologies or economic technologies; by manipulating the leaders of major world powers (blackmail, or the promise of future power); or by having the humans under its control use weapons of mass destruction to cripple the rest of humanity.

Of course, no current AI systems can do any of these things. But the question we’re exploring is whether there are plausible pathways by which a highly intelligent AGI system might seize control. And the answer appears to be yes. History already involves examples of entities with human-level intelligence acquiring a substantial fraction of all global power as an instrumental goal to achieving what they want. And we’ve seen humanity scaling up from a minor species with less than a million individuals to having decisive control over the future. So we should assume that this is possible for new entities whose intelligence vastly exceeds our own.

The case for existential risk from AI is clearly speculative. Yet a speculative case that there is a large risk can be more important than a robust case for a very low-probability risk, such as that posed by asteroids. What we need are ways to judge just how speculative it really is, and a very useful starting point is to hear what those working in the field think about this risk.

There is actually less disagreement here than first appears. Those who counsel caution agree that the timeframe to AGI is decades, not years, and typically suggest research on alignment, not government regulation. So the substantive disagreement is not really over whether AGI is possible or whether it plausibly could be a threat to humanity. It is over whether a potential existential threat that looks to be decades away should be of concern to us now. It seems to me that it should.

The best window into what those working on AI really believe comes from the 2016 survey of leading AI researchers: 70 percent agreed with University of California, Berkeley professor Stuart Russell’s broad argument about why advanced AI with misaligned values might pose a risk; 48 percent thought society should prioritize AI safety research more (only 12 percent thought less). And half the respondents estimated that the probability of the long-term impact of AGI being “extremely bad (e.g. human extinction)” was at least 5 percent.

I find this last point particularly remarkable—in how many other fields would the typical leading researcher think there is a 1 in 20 chance the field’s ultimate goal would be extremely bad for humanity? There is a lot of uncertainty and disagreement, but it is not at all a fringe position that AGI will be developed within 50 years and that it could be an existential catastrophe.

Even though our current and foreseeable systems pose no threat to humanity at large, time is of the essence. In part this is because progress may come very suddenly: Through unpredictable research breakthroughs, or by rapid scaling-up of the first intelligent systems (for example, by rolling them out to thousands of times as much hardware, or allowing them to improve their own intelligence). And in part it is because such a momentous change in human affairs may require more than a couple of decades to adequately prepare for. In the words of Demis Hassabis, co-founder of DeepMind:

We need to use the downtime, when things are calm, to prepare for when things get serious in the decades to come. The time we have now is valuable, and we need to make use of it.

Toby Ord is a philosopher and research fellow at the Future of Humanity Institute, and the author of The Precipice: Existential Risk and the Future of Humanity.

From the book The Precipice by Toby Ord. Copyright © 2020 by Toby Ord. Reprinted by permission of Hachette Books, New York, NY. All rights reserved.

Lead Image: Titima Ongkantong / Shutterstock

Steve Inskeep talks to Gen. Mark Milley, chairman of the Joint Chiefs of Staff, about the coronavirus threat within the ranks of the military, and guarding against a power competition with China.

The Trump administration has issued an intelligence analysis claiming China purposely delayed notifying the World Health Organization about the spread of the coronavirus.

The Trump administration’s nominee for inspector general overseeing billions in Treasury Department coronavirus relief funds is facing skepticism from Democrats who fear that he will not show sufficient independence.

CFL, CFLPA were scheduled to meet Friday to continue talks on potential contingency plans due to the COVID-19 pandemic. It was first gathering after CFL commissioner Randy Ambrosie's admission Thursday night the most likely scenario for the league is a cancelled 2020 season.

Modern dating techniques are providing new time frames for indigenous settlements in Northeast North America, free from the Eurocentric bias that previously led to incorrect assumptions.

A St. John’s genetics specialist has found DNA connections that link the long-vanished Beothuk people to contemporary people, almost two centuries after the last known Beothuk died. 

Ariana Grande and Demi Lovato will also appear on ABC's 'The Disney Family Singalong'

Comedian explained on her podcast that she hadn't realised what the name sounded like until recently

US talk show host previously said she returned to the air to support her crew, who she said she 'loves and misses'

During Jimmy Kimmel Live! filmed at their respective houses

Actor also spoke about how she feels the traditional image of the Hollywood movie star has faded in recent years

'I knew that I could not portray this gay man honestly without letting you all know that I am gay myself'

Pioneering TV star passed away after a long battle with Alzheimer's

He claimed only approved celebrities were permitted to talk to host at awards after party

Finlay was married to Joe Exotic during filming of 'Tiger King'

Comedian made the off-colour quip after the veteran actor starred in 'Cats'

Food writer Alison Roman accused Teigen of having people 'run a content farm' for her

You could be excused for thinking that, of course, all animals breathe oxygen to live. Because it wasn't until very recently that scientists discovered the only multicellular animal that doesn't. Meet Henneguya salminicola.

The European Space Agency said Friday that human urine could one day become a useful ingredient in making concrete to build on the moon.

The course of true love never did run smooth, after all

Gascoine was best known for her role as DI Maggie Forbes in the groundbreaking ITV police drama

The iconic Italian properties have joined forces to help end the Coronavirus crisis

Emma Glass's book set in an isolation ward is both terrific and timely. She talks to Katie Law

One of London's top dentists shares her tips on how to manage a dental crisis during the lockdown

From Joshua Tree to Formentera, Ibiza legend Dawn Hindle shares her favourite travel memories

Ricky Gervais’s After Life struggles second time round, as 21st-century London’s answer to Peaky Blinders gets off to a violent start. And how long can live shows survive via video-call?

After Life (Netflix)
Gangs of London (Sky Atlantic)
Emergence (Fox)
Twin (BBC Four) | iPlayer
The Graham Norton Show (BBC One) | iPlayer
The Mash Report (BBC Two) | iPlayer
Have I Got News for You (BBC One) | iPlayer

Ricky Gervais is, take your pick, ever reinventive (a la Madonna, Lady Gaga, the royals) or ever mutating (the worst kind of spirally viruses, the royals). A year ago, in Tony Johnson, subject of his latest drama, After Life, he combined aspects of past characters: The Office’s gloriously unself-aware Brent; the more savvy Andy Millman in Extras; the saccharine platitudes that sat so ill in Derek alongside gags about mental health or other disabilities. After Life was a surprising runaway hit on Netflix, for an arguably slight comedy about a very singular, small-town man’s depression after the loss of his wife, and how an angry man learned to be kind again.

Marcus Rashford has been leading Manchester United's charm offensive to bring Jadon Sancho to Old Trafford - and he will have another chance tonight.

Leeds United legend and one of England's 1966 World Cup winners Norman Hunter has been admitted to hospital after testing positive for coronavirus.

Tottenham legend Jimmy Greaves remains in hospital, "receiving the best possible treatment", as he battles an unspecified illness.

Liverpool legend Sir Kenny Dalglish has tested positive for Covid-19 but does not currently have any symptoms of the virus.

Manchester United fans hailed the fine work of 'agent Marcus Rashford' once more after his FIFA 20 battle with England colleague Jadon Sancho.

Paul Scholes believes Manchester United have found their own version of Manchester City's Kevin De Bruyne in Bruno Fernandes.

Liverpool fans would have been justified in feeling a little nervous when the summer transfer window of 2016 opened up.

Liverpool legend Sir Kenny Dalglish has praised the efforts of NHS staff across the UK after being released from hospital following a positive coronavirus diagnosis.

Liverpool legend Sir Kenny Dalglish has reiterated his "immense gratitude" to the NHS following his release from hospital and urged people to stay at home during the coronavirus pandemic.

Bonetti, dubbed 'The Cat', died at the age of 78 on Sunday

Inter Milan striker Lautaro Martinez has a "big chance" of joining Lionel Messi at Barcelona this summer, one of his agents has said.

Chelsea legend Ron Harris has paid tribute to former goalkeeper Peter Bonetti, the 'proper gentleman' and 'fitness fanatic' who led by example in west London.

Liverpool manager Jurgen Klopp has revealed the impact that club legend Sir Kenny Dalglish being diagnosed with coronavirus had on him and his players.