Why Apple’s child safety updates are so controversial

Last week, Apple previewed updates meant to beef up child safety features on its devices. Among them is a new technology that can scan the photos on users’ devices to detect child sexual abuse material (CSAM). Though some lawmakers and child safety advocates widely praised the change, it prompted immediate pushback from many security and privacy experts, who say the update amounts to Apple walking back its commitment to putting user privacy above all else.

Apple has disputed that characterization, saying that its approach balances both privacy and the need to do more to protect children by preventing some of the most offensive content from spreading more widely.

What did Apple announce?

Apple announced three separate updates, all of which fall under “child safety.” The most significant feature that’s gotten the bulk of the attention is a feature that will scan iCloud Photos for known CSAM. The quality, built into iCloud Photos, compares a user’s photos against a database of previously identified material. If a certain number of those images is detected, it triggers a review process. If human reviewers verify the photos, Apple will suspend that iCloud account and report it to the National Center for Missing and Exploited Children (NCMEC).

Apple also previewed new “communication safety” features for the Messages app. That update enables the Messages app to detect when sexually explicit photos are sent or received by children. Importantly, this feature is only available for children who are part of a family account, and it’s up to parents to opt in.

Parents who opt into the feature will be alerted if a child under 13 views one of these photos. For children older than 13, the Messages app will show a warning upon receiving an explicit image but won’t alert their parents. Though the feature is part of the Messages app and separates from the CSAM detection, Apple has noted that the quality could still play a role in stopping child exploitation, as it could disrupt predatory messages.

Finally, Apple is updating Siri and its search capabilities to “intervene” in queries about CSAM. For example, if someone asks how to report abusive material, Siri will provide links to resources. If it detects that someone might be searching for CSAM, it will display a warning and surface resources to offer help.

When is this happening, and can you opt out?

The changes will be part of iOS 15, which will roll out later this year. Users can effectively opt out by disabling iCloud Photos (instructions for doing so can be found ). However, anyone undermining iCloud Photos should remember that it could affect your ability to access photos across multiple devices.

So how does this image scanning work?

Apple is far from the only company that scans photos to look for CSAM. Apple’s approach to doing so, however, is unique. The CSAM detection relies on a database of the known material maintained by NCMEC and other safety organizations. These images are “hashed” (Apple’s official name for this is NeuralHash) — a process that converts images to a numerical code that allows them to be identified, even if they are modified in some way, such as cropping or making other visual edits. As previously mentioned, CSAM detection only functions if iCloud Photos is enabled. What’s notable about Apple’s approach is that rather than matching the images once they’ve been sent to the cloud — as most cloud platforms do — Apple has moved that process to users’ devices.

Here’s how it works: Hashes of the known CSAM is stored on the device, and on-device photos are compared to those hashes. The iOS device generates an encrypted “safety voucher” sent to iCloud and the image. If an instrument reaches a certain threshold of CSAM, Apple can decrypt the safety vouchers and conduct a manual review of those images. Apple isn’t saying what the point is but has made clear a single image wouldn’t result in any action. Apple also published a detailed technical explanation of the process.

Why is this so controversial?

Privacy advocates and security researchers have raised several concerns. One of these is that this feels like a significant reversal for Apple, which five years ago refused the FBI’s request to unlock a phone and has put up stating “what happens on your iPhone stays on your iPhone.” To many, Apple created a system that can proactively check your images for illegal material and refer them to law enforcement feels like a betrayal of that promise.

In a statement, the Electronic Frontier Foundation “a shocking about-face for users who have relied on the company’s leadership in privacy and security.” Likewise, Facebook — which has spent years taking heat from Apple over its privacy missteps — has taken issue with the iPhone maker’s approach to CSAM. WhatsApp chief Will Cathcart, as “an Apple built and operated surveillance system.”

While CSAM detection will only be in the US to start, Apple has suggested it could eventually expand to other countries and work with other organizations. More specifically, there are real concerns that Apple could be pressured — either by law enforcement or governments — to look for other types of material once such a system is created. It’s not difficult to imagine scenarios where Apple could be pressured to start looking for different types of content that are illegal in some countries. The company’s concessions in China — where Apple reportedly of its data centers to the Chinese government — prove that it isn’t immune to the demands of less-democratic governments.

There are other questions too. Like whether someone can abuse this process by maliciously getting CSAM onto someone’s device to trigger them to lose access to their iCloud account. Or whether there could be a false positive or another scenario that results in someone incorrectly flagged by the company’s algorithms.

What does Apple say about this?

Apple has vehemently denied that it’s degrading privacy or walking back its previous commitments. The company published a second document that made many of these claims.

On the issue of false positives, Apple has repeatedly emphasized that it is only comparing users’ photos against a collection of known child exploitation material, so images of, say, your children won’t trigger a report. Additionally, Apple has said that the odds of a false positive is around one in a trillion when you factor in the fact that a certain number of images must be detected to even trigger a review. However, Apple is saying we have to take their word on that. As Facebook’s former security chief Alex Stamos and security researcher Matthew Green wrote in a joint New York Times op-ed, Apple hasn’t provided outside researchers with much visibility into all this.

Apple says its manual review, which relies on human reviewers, would detect if CSAM was on a device due to some malicious attack.

Although, once again, we have to take Apple at its word here. Regarding pressure from governments or law enforcement agencies, the company has said it would refuse to cooperate with such requests. “We have faced demands with building and deploying government-mandated changes that degrade the privacy of users before and have steadfastly refused those demands,” it writes. “We will continue to refuse them in the future. Let us be clear, this technology is limited to detecting CSAM stored in iCloud, and we will not accede to any government’s request to expand it.”

If it’s so controversial, why is Apple doing it?

The short answer is that the company thinks this is finding the right balance between increasing child safety and protecting privacy. CSAM is illegal, and companies must report it in the US when they see it. As a result, CSAM detection features have been baked into popular services for years. But unlike other companies, Apple hasn’t checked for CSAM in users’ photos, mainly due to its stance on privacy. Unsurprisingly, this has been a significant source of frustration for child safety organizations and law enforcement.

To put this in perspective, in 2019, Facebook reported 65 million instances of CSAM on its platform, The New York Times. Google reported 3.5 million photos and videos, while Twitter and Snap reported “more than 100,000,” Apple, on the other hand, reported 3,000 photos.

That’s not because child predators don’t use Apple services but because Apple hasn’t been nearly as aggressive as some other platforms in looking for this material, and its privacy features have made it difficult to do so. What’s changed is that Apple has developed a technical means of detecting collections of known CSAM in iCloud Photos libraries that still respects users’ privacy. There’s a lot of disagreement over the details and whether any kind of detection system can truly be “private.” But Apple has calculated that the tradeoff is worth it. “If you’re storing a collection of CSAM material, yes, this is bad for you,” Apple’s head of privacy, The New York Times. “But for the rest of you, this is no different.”