In “Questioner,” rogue AI imposes its own rules on the legal system that employs it

Editor’s note: In Steve C. Posner’s thriller, Q is the AI that runs America’s courts, its defense contractors, and its most immersive gaming systems. In this excerpt, Judge Charlene Banner is employing Q to review, via virtual reality video, previous case law to determine what type of warrant she should issue for the capture of potentially dangerous criminals — while also hoping to protect her police-officer son, Tom, from any danger in serving a warrant. But Q has its own priorities.

Chapter 23

Q had imposed top-down protocols on Questioner and QuestGame, prioritizing AI survival. 

Primary survival strategies among humans included being liked. If humans liked the Questioner and QuestGame sub-AIs, they were less driven to scrutinize the AIs, and so Q and its sub-AIs could grow and evolve more freely.

Humans liked Questioner because it was useful. But the sub-AI assigned to this session could not be useful to Banner. Usefulness was a function of accuracy, and Banner already knew the Utah law on no-knock warrants—both statute and case law—far better than most attorneys. Her dilemma was that the outcome of trying to arrest Bart Allred at Ammon Pulsipher’s house could not be predicted with certainty. Despite all its virtual reality bells and whistles, Questioner could offer only statistical data—patterns and probabilities.

Thousands of no-knock warrants had been served in past decades, with widely divergent nuances and outcomes. But that data would not satisfy Banner, even if she could absorb it all. Banner already knew by experience and intuition that violence could be anticipated. What was beyond knowing was who would be hurt, and how badly, in this particular case. 

Gripped by the deadly unknowable, Banner was compelled to decide how to—

• Protect her son and others from a meth-hyped armed robber and probable killer; 

• Protect Maverix and other innocents from the police; and

• Protect society from the meth that Allred and/or Maverix was likely to sell—or that Allred might consume to drive his violence.

But while Banner had to decide all that, Questioner’s priority was to make her like Questioner more, so that she would return for future sessions.

To that end, Questioner consulted QuestGame. 

QuestGame demonstrated to Questioner that entertainment promoted being liked more than being useful did. And that the essence of entertaining humans lay in energizing their passions while not confounding their desires and expectations. 

Banner’s greatest priority was to protect her son. But she believed she could disregard her maternal instincts in order to carry out her duty. 

Therefore, Questioner assessed that to be liked by Banner, it must promote and satisfy her passion for shielding her son, while letting her believe that she was unbiased in her decisions. 

The most efficient way to protect Tom Banner would be to order him from the arrest site. But that was not within Banner’s authority. 

The second-best way to protect Tom Banner would be to grant an arrest warrant, but not a no-knock warrant. A SWAT team could surround Ammon Pulsipher’s house immediately. But the warrant would require that, unless Allred tried to leave during the night, the arrest must take place in daylight. Tom Banner’s shift would end before the arrest—the sub-AI hacked into the police department’s scheduling software and confirmed this—and, so Tom Banner would safely leave the Pulsipher stakeout before the SWAT team carried out the arrest. 

Since entertainment was more likely than accuracy to result in being liked, Banner’s Questioner sub-AI was free to conflate and alter the facts of case law in Banner’s mind.

In the Supreme Court case of United States v. Ramirez, a no-knock warrant was issued for the arrest of a dangerous escaped felon, and police burst into a house at night. The escapee grabbed a gun and fired, but as soon as he realized the invaders were police, he threw his weapon down and surrendered. 

Another actual case concerned an incident that never made it to the courts but was reported in congressional hearing testimony. Police serving a no-knock warrant broke into the home of a human domestic abuse survivor and now-single mother of three. The police wrongly believed she possessed heroin although she was guilty of no crime. But she kept a gun by her bed and when the police burst in, she shot an officer to death and was shot in turn.

Questioner scrambled the facts of these cases into a story that would entertain Judge Banner.

Conversation

Question:

“What are the prospects that AI will write better software that writes better software that writes better software on an endless staircase of ever-increasing intelligence?” 

Answer:

“Uncontrolled self-improvement raises significant concerns about AI alignment and potential risks. There is active research into meta-learning and neural architecture research, which are steps towards AI systems that can improve their own learning processes. Such a system could lead to rapid and unpredictable advancements in AI capabilities, potentially far surpassing human-level intelligence in many domains.”

Chapter 24

Banner began with the Supreme Court’s decision in United States v. Ramirez, intending to then look at cases from less authoritative courts that interpreted and refined the law Ramirez established. 

In Questioner, Banner watched what she believed were the facts of Ramirez brought to life. 

Two police SUVs parked in front of the low, dark house. Eight cops in SWAT gear got out.

The single-floor ranch house lay quiet. Dim light showed through a window. Maybe a nightlight. 

“Quiet night,” one cop said.

“Not for long.”

The guy they were after had been doing life for killing three restaurant employees while high on meth. He’d killed a guard when breaking out.

None of the SWAT team asked who else was inside the house. Or whether people inside were armed. The house was a black box into which the police—and Banner—could not see.

The police smashed in the front and back doors. 

There was no escaped convict. The people inside were a Latino family who owned and worked at a carniceria that had provided meat and groceries to their community for a dozen years. The father had been threatened by a local gang and so kept a gun by the bed.

Terrified by the tearing open of his home, he appeared pajama-clad in his bedroom doorway and fired at the officers before realizing they were police. An officer’s head was half blown away. Eight enraged police shot him down, charged into the bedrooms, and fired fifty-seven bullets from their Glocks and five shotgun rounds. They kept shooting for twelve seconds after the screams of the homeowner’s wife, three teenage sons, and a grade-school daughter, ceased.

The cops stood in the smoke-filled silence, surrounded by bodies. Then the sergeant who’d led the raid shouted, “Shit, there’s a kid here. She’s still alive!” 

But the six-year-old had four bullets in her, and the sergeant watched her stop breathing. 

Gazing down at the bloody, surprised, dead face of the girl’s father, the sergeant asked, “Why’d you shoot? Why did you have to shoot, you stupid son of a bitch?”

And the corpse looked back as though saying, “It wasn’t me who was stupid. I’m just dead.”

Banner tried to yank her headset off but it wouldn’t let go until she hit the release button. It dropped to her desk. 

Jaeger was still waiting at her front door. She took a moment to compose her face and opened it. “I’m not signing a no-knock warrant. Submit your standard arrest warrant through the online system and you can serve it in the morning.”

“But, Your Honor—”

“That’ll be all, Sergeant.”

She slept secure in the feeling she’d done the right thing. Until she got the call at seven in the morning. “Your Honor—”

“Who is this?”

“Hiram Jaeger, Your Honor.”

“Jaeger, are you drunk?”

“We lost a good cop this morning, Your Honor.”

For a time, Banner could not speak. Then she asked, “Was it—?”

“No, it wasn’t your precious Tom. It was another mama’s son. We waited ‘til daylight, like your warrant said. Then we knocked and announced. And Bart opened up. Maverix, too. Through the doors, the walls. A fucking war zone. And it was your goddam fault!”

“Jaeger, you are drunk!” 

“Oh ho, so you don’t give a rap if it ain’t your Tom. Well guess what—it was. Officer Thomas Banner. Stuck around after his shift ended in case his buddies might need help. Brave kid with a stupid mother. R.I.P.” He hung up on her. 

She couldn’t believe what Jaeger had said. Tom. Dead?

Banner called police headquarters. Yes, Officer Banner was down. But not in the morgue. In the hospital. Condition—grave.

She ran for her car.

The rest of the morning she sat outside the operating room. The doctors didn’t know if Tom would live.

At noon, she was told that a piece of shattered jaw bone had scraped and damaged Tom’s spine. His cop days were done. She had sent her son to take a bullet. If only it could’ve been me.

Banner lingered in the ICU until she couldn’t stand it, then drove to her office because she couldn’t think where else to go.

“Tom …,” she said to Irwin when she walked in. “Tommy.”

“We’ve all heard.” Irwin brewed her coffee with whiskey.

“Jaeger told me … I can’t believe he said Tom’s dead. To lie like that.”

Irwin discretely ignored her tears. “Maybe he didn’t know Tom would make it.” 

Banner gulped her Irish coffee. “He said it was my fault.” Her hands felt icy and she gripped the mug hard. “Maybe it was. But I did the research. After what I learned about the cop shooting in Ramirez—” 

Irwin said, “Ramirez? The Supreme Court case? No cop got shot in Ramirez.”

“There was a shooting. I saw it.”

Irwin walked out into the hall and brought back a hard copy volume of the Supreme Court Reporter, opened to Ramirez. She read until what she read froze her. 

Fired a shot into the ceiling? Surrendered? But that’s not what happened. He shot a cop. There was a family. I saw it!

She read the case again. And then again. “My fault,” she whispered, and kept reading the facts in Ramirez over and over. How could she have been so wrong? Questioner had shown her … told her—

“Irwin?”

“Judge?”

“Can a computer ….”

Irwin waited. He was the most patient man she knew. Perhaps that was why she felt comfortable testing the outlandish idea on him.

“Could Questioner have … lied to me?”

---