#7 Explained: Apple’s New Child Safety Features
Hello friends,
I’ve been seeing a few outraged Social Media posts about a particular piece of news (not COVID-19 related) that I found interesting so I did a little deep dive into it. The clickbait tagline for it is Apple spies on you and snitches to authorities. The reality is a little more nuanced.
What is Apple doing?
As part of their iOS 15 update, Apple is rolling out three new features aimed at Child Safety. They released a bundle of information that you can find here. If you can’t be bothered, you’ve come to the right place because I’ve read it for you. They are described below in the order from least to most controversial. It is important to point out that the features will initially be U.S. only and presumably be rolled out to other countries on a case by case basis.
New Siri and Search prompts
This part is fairly straightforward. Essentially, if someone asks their Apple device how they can report child exploitation, then Siri and Search will direct them to more specific resources. If Siri or Search detects someone actively looking for child exploitation material, then they will be presented with prompts and warnings to rethink their decision.
Communication safety in Messages
Then Apple steps it up a notch by introducing a function that detects sexually explicit photos when sent through the native Messages app. This feature is only active on child accounts (under 18) in Family Sharing (part of iCloud). It is also important to note that the detection of sexually explicit photos occurs on-device, part of the reason why this update has raised some criticism.
Teenagers (12 to 17)
When a teenager receives content their device considers sexually explicit (i.e. nudity), the device will blur out that content and provide the receiver with a warning. According to Apple, the warning will tell the receiver the nature of the content and let them know that it is ok not to view it if they don’t want to and direct them to some helpful resources. If the receiver then chooses to proceed, the content will then be revealed. The sender of such content will also be warned prior to sending.
Children 12 and under
For this age group, there is an additional function of parental notifications. The receiver of the content will be warned that if they choose to view the content, their parents will receive a notification. Apple makes a point to state that notification received by parents will not include any details of the content itself. This additional element functions similarly for senders of this age group. If they choose to send, then the parents will receive a notification that content of such nature has been sent.
Some people have taken issue with the fact that the actual detection of the photos is done on-device. Apple achieves this through machine learning algorithms. It is clear that Apple is very aware of this objection. They create an entire FAQ, most of which is dedicated to reassuring people’s expected privacy concerns. The gist is that Apple claims it does not have access to the images, evaluations, notifications or interventions. One of the FAQ questions is whether this feature will prevent children in abusive homes from seeking help. I personally found this interesting as I wouldn't have imagined kids needing to send nude photos for that purpose but I guess it may need to be done as a proof-of-evidence process to peers or would-be-helpers on certain occasions. Another angle I didn’t personally think of was mentioned by Kendra Albert, a lawyer at Harvard Law School’s Cyberlaw Clinic. He tweets on August 6,
“these "child protection" features are going to get queer kids kicked out of their homes, beaten, or worse”
“I just know (calling it now) that these machine learning algorithms are going to flag transition photos. Good luck texting your friends a picture of you if you have "female presenting nipples.”
Personally, I think Apple has done their homework and probably came to the conclusion that the pros of this feature outweigh the cons. I do agree that there will be some unforeseeable consequences that will negatively impact the kids involved in ways that Apple has not considered. More importantly, I personally feel this will simply encourage users to migrate to an alternative messaging platform.
Child Sexual Abuse Material (CSAM) Detection
This is by far the most contentious component of the features. The way it functions is also highly technical. So technical that Apple includes several detailed summaries to explain the various technologies involved. The three technologies involved are NeuralHash, Private Set Intersection and Threshold Secret Sharing. Before I attempt to break down these technologies, the grossly oversimplified version is:
Apple will create a digital thumbprint for all photo and video content users choose to sync with iCloud. Those thumbprints will be matched against a list of existing thumbprints, generated from a database of known CSAM material from the National Center for Missing and Exploited Children (NCMEC). This matching process is done on-device. Once a certain number of matches is met, the matching content is then revealed and forwarded for manual review by Apple. If the report is validated by the human reviewer, the user’s account will be disabled and the report will be forwarded to NCMEC and their law enforcement partners.
Apple goes to great lengths to state that they will learn absolutely nothing about the user’s content or other meaningful data generated from this process until that threshold of matches is crossed and a report is sent to them for manual review. They further reaffirm that the only matches this feature applies to are with known material from the NCMEC and nothing else. I have read all of their technical summaries and below is my best attempt at explaining the technologies in ways that I can understand myself.
NeuralHash
The actual process through which hashes are generated is actually quite complex so I will not attempt to explain it. Essentially, images are given a digital thumbprint, known as a hash and that hash is then put through a compression process so the final thumbprint is small enough for storage and transmission purposes.
Similar images are given the same hash, even if they differ by scale, colour, size or other superficial changes while images deemed different are given separate hashes. Apple gives the example below.
The NCMEC maintains a database of known CSAM material. Each piece of material is given a unique hash. This set of hashes from the NCMEC goes through a ‘blinding’ process and is stored on user devices. This process ensures that no one can infer anything about the underlying CSAM content from looking at the hash database.
Private Set Intersection (PSI)
The Apple device will scan all content synced with iCloud and assign a hash to all of that material. It will then match those hashes with the NCMEC hash database securely stored on those devices. If a match is found on-device, it will generate what Apple calls a Safety Voucher and uploaded to the iCloud server. This next part is where it becomes exceptionally technical. What I surmised is that PSI is broken down into two steps. First step is on-device and the second step is server side. The reason why Apple has created this further step is actually to strengthen security and privacy. At the first step, even if someone can decrypt and extract the hash matches on-device, those matching results are only arbitrary numbers. It will just look like a random set of 1s and 0s. The second step is done on the server side. The matching safety vouchers are combined with the server side blinding secret to decrypt and reveal the content. The way I understand this as a total layman is that the NCMEC hash database stored on Apple devices are pre-scrambled by Apple so no one can infer anything about the original content from those matching hashes. Once the matching hashes are uploaded onto iCloud, Apple is able to decrypt them and unscramble them because only the server side knows how they were scrambled in the first place. Without additional steps or technologies, it would mean that Apple would be able to decrypt and view the contents each time there was a match but they actually combine it with something called Threshold Secret Sharing to further improve privacy.
Threshold Secret Sharing (TSS)
Essentially, instead of being able to decrypt the Safety Voucher each time there was a match, Apple adds TSS to ensure that it takes a certain number of matches before Apple is able to decrypt and view the image itself. Supposedly the addition of this step means that the chances of an incorrect flagging will be 1 in a trillion. Apple explains it using the following Example
if a secret is split into one-thousand shares, and the threshold is ten, the secret can be reconstructed from any eleven of the one-thousand shares. However, if only ten shares are available, then nothing is revealed about the secret.
Before that threshold is reached, the server only learns the number of matches for a particular user. What is fascinating is that Apple introduces a further step here to obfuscate that number. They do this by having the devices send out a random number of dummy vouchers to look like encrypted CSAM data. Apple servers initially only learns the combined number of matches, fake and real. Only if the threshold is reached will the server become able to identify the actual matches.
Once Apple’s threshold is hit, Apple will be able to decrypt and view the images and that information is forwarded to a manual reviewer. If that manual reviewer validates CSAM material, then the user’s account will be disabled and their information sent to the NCMEC and presumably law enforcement. Apple does state that there is the ability for users to request an appeal process if they feel their account was wrongly disabled.
Is Apple Aware of Privacy Concerns?
Most definitely. It is pretty obvious reading Apple’s materials and looking at the design of these features that Apple anticipated criticism. They devote a significant amount of energy to reassure users that no meaningful data can be extracted about the user either on-device or server side until the threshold is met and the results revealed to Apple. They include several analyses from industry experts as testimonials for the strength of the system. So what are people really worried about?
What Are The Critics Saying?
I reviewed a bunch of different criticisms. There is an open letter that outlines the main public and industry issues with these new features. Overall, the criticisms can be summarised in the following categories.
Apple is spying on us through a ‘backdoor’ and telling our secrets to the government.
I don’t personally think this criticism holds any water for what the features currently do. You can certainly infer a lot of things and get very mad if you were simply reading headlines but in its current form, I think a lot of people are just misunderstanding how these features function.These features will have unintended consequences and victims.
It’s pretty hard to debate this one. I’ve already touched on this above. I am certain that there will be issues with these features that will hurt kids, families and relationships in ways that Apple hasn’t anticipated.These features may not be a problem now but they open a door we cannot close.
I think this is by far the strongest argument. Apple reiterates several times through their materials that the CSAM detection is only matching known materials from the NCMEC. They also explicitly state that they will never give in to government demands to change the matching targets for other purposes. I personally don’t doubt this pledge from Apple as it stands today but people and organisations are prone to change and there is little guarantee that the target won’t be moved in the future. The on-device scanning component of these new features are controversial because there is no real ‘opting out’. Once these features get rolled out into other countries, you simply cannot rule out other types of materials becoming targets for matching. Even if this only remains in the US, the possibility of scope creep is ever present and Apple cannot assure that the target will always be CSAM material.Bad actors using this feature to frame innocent people
I found this criticism while looking at some random threads. I am not versed enough in the technical aspects to understand whether this criticism is possible or not but I’ll leave it here in full so you can make your own decision.
An attacker can easily produce natural images that have a specified perceptual Hash. An attacker can generate a meme image which looks normal but has a hash which is inside the CSAM database. (he can easily get one of these bad hash values by computing the neural hash of a known offensive image) Then he send you a mail, then you save the image to your cloud because you find it funny (or because your phone automatically back-up your mail to your cloud) , and a collision is registered and you get arrested (when the manual review fails, for example if the attacker has steganographically hidden offensive content in those images).
Personally, I find the scope-creep argument the most convincing. Technology is akin to Pandora’s box. Once you put it out there into the world, no amount of good intention will prevent it from being used in ways you did not first intend. It is just the nature of things and I think Apple took a step that it won’t be able to walk back on.
What Are The Other Tech Giants Doing?
I think one of Apple’s most convincing counter arguments is that there are already other significantly less privacy conscious methods being used to scan photos on other non-Apple devices. Another method to yield the same result would be to scan the images on a cloud server in the clear without any process to encrypt or obscure user data. If you look at the data published by NCMEC for the year 2020 on the number of reports submitted by other tech and social platforms, Apple has barely made the list.
Facebook: 20,307,216
Google: 546,704
Snapchat: 144,095
Microsoft: 96,776
TikTok: 65,062
Dropbox: 20,928
Apple: 265
I would imagine such low numbers are a reflection of Apple’s long held value of maintaining user privacy over other goals, some of them perfectly legitimate such as curbing child exploitation. If the only goal is privacy, then Apple’s new Child Safety features are moving away from that but I would argue this new development essentially puts Apple on par with the practices of most of their industry peers, perhaps with greater privacy considerations. Personally, I don’t believe these functions as they stand pose any real threat to user privacy. However, I do share certain concerns such as scope creep. The capability is there now and there will always be the temptation in the future to use it in ways that won’t align with their purpose today.
Other interesting things I learnt
Facebook’s numbers from the NCMEC are extremely high. However, it turns out that over 90% of those reports refer to the same or previously reported content. Facebook also reports that copies of six videos are responsible for over 50% of these reports. Even one report is too many but this gives me some comfort that it isn’t 20 million unique cases of child exploitation.
Those numbers reported above also include subsidiaries such as WhatsApp and Instagram under Facebook.
WhatsApp uses an entirely different way to detect CSAM material. As the platform boasts end-to-end encryption, they do not do any type of scanning of content shared between users. However, there are certain data that is considered unencrypted such as group names or profile photos. With the assistance of artificial intelligence algorithms, WhatsApp files reports (combined with Facebook’s numbers) to NCMEC based on these types of unencrypted data.
I hope you found this week’s content interesting. If you made it this far, I presume you didn’t hate it. I hope to be putting out more research based pieces in the future.
