A US teenager was handcuffed by armed police after an artificial intelligence (AI) system mistakenly said he was carrying a gun – when really he was holding a packet of crisps.
“Police showed up, like eight cop cars, and then they all came out with guns pointed at me talking about getting on the ground,” 16-year-old Baltimore pupil Taki Allen told local outlet WMAR-2 News.
Baltimore County Police Department said their officers “responded appropriately and proportionally based on the information provided at the time”.
It said the AI alert was sent to human reviewers who found no threat – but the principal missed this and contacted the school’s safety team, who ultimately called the police.
But the incident has prompted calls by some for the schools’ procedures around the use of such technology to be reviewed.
Mr Allen told local news he had finished a bag of Doritos after football practice, and put the empty packet in his pocket.
He said 20 minutes later, armed police arrived.
“He told me to get on my knees, arrested me and put me in cuffs,” he said.
Baltimore County Police Department told BBC News Mr Allen was handcuffed but not arrested.
“The incident was safely resolved after it was determined there was no threat,” they said in a statement.
Mr Allen said he now waits inside after football practice, as he does not think it is “safe enough to go outside, especially eating a bag of chips or drinking something”.
In a letter to parents, school principal Kate Smith said the school’s safety team “quickly reviewed and cancelled the initial alert after confirming there was no weapon”.
“I contacted our school resource officer (SRO) and reported the matter to him, and he contacted the local precinct for additional support,” she said.
“Police officers responded to the school, searched the individual and quickly confirmed that they were not in possession of any weapons.”
However, local politicians have called for further investigation into the incident.
“I am calling on Baltimore County Public Schools to review procedures around its AI-powered weapon detection system,” Baltimore County local councilman Izzy Pakota wrote on Facebook.
Omnilert, the provider of the AI tool, told BBC News: “We regret this incident occurred and wish to convey our concern to the student and the wider community affected by the events that followed.”
It said its system initially detected what appeared to be a firearm and an image of it was subsequently verified by its review team.
This, Omnilert said, was then passed to the Baltimore County Public Schools (BCPS) safety team along with further information “within seconds” for their assessment.
The security firm said its involvement with the incident ended once it was marked as resolved in its system – adding it had “operated as designed” on the whole.
“While the object was later determined not to be a firearm, the process functioned as intended: to prioritise safety and awareness through rapid human verification,” it said.
Omnilert says it is a “leading provider” of AI gun detection – citing a number of US schools among its case studies on its website.
“Real-world gun detection is messy,” it states.
But Mr Allen said: “I don’t think no chip bag should be mistaken for a gun at all.”
The adequacy of AI to accurately identify weapons has been subject to scrutiny.
Last year, a US weapons scanning company Evolv Technology was banned from making unsupported claims about its products after saying its AI scanner, used in thousands of US schools, hospitals and stadiums entrances, could detect all weapons.
BBC News investigations showed these claims to be false.
