Transport for London’s (TfL) reliance on information from Uber’s facial verification-based driver ID system is under increased scrutiny following multiple instances of misidentification, leading to unionized drivers losing their private hire licenses. The drivers and their union say that, despite the technology’s questionable veracity, TfL has been overly reliant on it when making license revocation decisions, severely impacting drivers’ livelihoods if they cannot legally operate their vehicles.
TfL’s unchallenging acceptance of the evidence provided by Uber’s Microsoft-based verification system has raised numerous legal challenges by drivers who say they have had their licenses unfairly revoked.
Earlier this month (5 July 2021), for example, a Magistrate’s Court ordered TfL to reinstate the license of an Uber driver, which it found was revoked without notice or appeal following a failed facial verification check.
The City of London Magistrates Court found TfL moved straight to revoke the driver’s private hire license on identity fraud grounds and relied solely on evidence from Uber’s facial verification software without conducting its investigation, as a licensing authority, into the incident.
“The Court noted that at no stage had there been any risk to the public and that TfL had proceeded straight to revocation, despite other sanctions being available to them,” wrote the driver’s solicitor, Abbas Nawrozzadeh of Eldwick Law, in his notes of the judgment.
“They had relied on the actions of a body that fell outside of their licensing power and revoked the license without any investigation into actual events on that day. The Court described the decision as perverse and unreasonable, stating that as a licensing body, TfL should be ashamed of bringing this case to Court as it represented a waste of public funds.”
Facial verification
Speaking to Computer Weekly, Nawrozzadeh added that he had been “dealing with a raft of these cases” since Uber introduced its Real-Time ID Check facial verification system.
In the case of 5 July, the misidentification occurred while the driver, a member of the App Drivers & Couriers Union (ADCU), worked for the ride-hailing app’s food delivery arm, Uber Eats. The error, however, resulted in the company deactivating his Uber Eats and driver’s accounts.
Under London’s transport regulations, Uber must inform TfL when a driver is dismissed via a specific form, which Computer Weekly understands should trigger a “fitness assessment” by the regulator to decide whether the driver should continue to hold a private hire license.
This process would also typically entail TfL notifying the driver in writing that it had received an adverse report about them so they can respond and appeal if TfL decides to revoke their license.
According to the general secretary of the ADCU, James Farrar, drivers can usually continue working. At the same time, a decision is appealed, except in more severe cases with a perceived safety risk or fraud. It should be noted that Uber was allowed to continue operating while it appealed its unfavorable licensing decision.
“Then they can immediately revoke,” he told Computer Weekly. “You can continue to appeal to the Magistrate court, but you can’t work.” Following the driver’s misidentification by Uber’s Facial-verification system and the subsequent deactivation of his accounts, TfL was notified and immediately revoked his license, Farrar added. This is a course of action it usually takes in response to serious allegations, such as sexual or physical violence.
He further claimed the TfL decision was based on little data and that the regulator only asked Uber to substantiate the allegation when the driver challenged TfL’s decision. “Even if he had engaged in any kind of identity fraud, which he didn’t [as the court decision found], he would only have done it as a delivery driver, but TfL revoked the [private hire] license anyway,” he said. “There was no question he did anything wrong while acting as a licensed driver.” Computer Weekly asked Uber if it would like to comment on why the Eats and driver’s accounts were deactivated, but it received no response.
As a result of the Court’s findings, the driver was also awarded his legal costs against TfL, which Farrar described as “extraordinary”. “Usually they’re immune to cost claims because, if you’re a licensing authority in the interest of public safety, only under exceptional circumstances will a cost be awarded against you for an adverse licensing decision, even if it’s overturned,” he said.
Issuing a general response to Computer Weekly’s questions, a TfL spokesperson said: “The safety of the traveling public is our top priority, and where we are notified of cases of driver identity fraud, we take immediate action to revoke a driver’s license so that passenger safety is not compromised.” Uber was also asked if it would like to respond to the ruling, but Computer Weekly received no response.
Not an isolated incident
The plight of the ADCU driver, the Magistrates Court, heard about this month is far from an isolated incident concerning Uber drivers losing their licenses due to facial recognition errors.
In March 2021, the ADCU and its associated data trust, Workers Info Exchange (WIE), said it had identified a further seven cases of Uber drivers losing their jobs and having their licenses revoked by TfL because the company’s Real-Time ID Check system failed to recognize their faces.
Incidentally, when TfL was asked to comment on the July 2021 ruling, it made an abortive request for the driver’s name to substantiate it was commenting on the correct case, which indicates the number of cases it is involved with.
The ADCU did not wish to disclose the driver’s name to Computer Weekly out of a concern they could face repercussions from TfL. As to why Uber’s facial-verification tool may be considered error-prone, research into Microsoft’s systems, as well as actions of Microsoft itself, suggest there is bias against particular groups, particularly people of color and women.
In 2018, research from MIT indicated that Microsoft’s facial recognition and detection systems – specifically the Face API being used by Uber – had gender and racial biases, finding it had much higher error rates when identifying women or people with darker skin.
“The substantial disparities in the accuracy of classifying darker females, lighter females, darker males, and lighter males in gender classification systems require urgent attention if commercial companies are to build genuinely fair, transparent, and accountable facial analysis algorithms,” said authors Joy Buolamwini and Timnit Gebru.
Sales suspended
In June 2020, Microsoft – alongside Amazon and IBM – suspended sales of its facial-recognition technologies to US law enforcement agencies in response to several weeks of mass protests against the police murder of George Floyd on 25 May.
Microsoft President Brad Smith previously told ITV in January 2019 that one of the challenges with the technology in its current form was that “it doesn’t work as well for women as it does for men, it doesn’t work as well for people of color”, adding that it was more likely to find errors, mismatch and generally “fail to identify” people from these groups.
A Microsoft spokesperson commented on the union’s allegations regarding Uber’s Real-Time ID Check system when they were first made in March, said the company was “committed to testing and improving Face API, paying special attention to fairness and its accuracy across demographic groups”.
“We also provide our customers with detailed guidance for getting the best results and tools that help them to assess fairness in their system.” An Uber spokesperson added that the system was designed to protect the safety and security of passengers by ensuring the correct driver or courier is using the account.
“While no tech or process is perfect and there is always room for improvement, we believe the technology, combined with the thorough process in place to ensure a minimum of two manual human reviews before any decision to remove a driver, is fair and important for the safety of our platform,” they said.
In an April 2021 letter sent directly to the ADCU – which has also been shared with Computer Weekly – Microsoft told the union, “Errors in the human review process associated with Uber’s implementation of our facial recognition technology do not provide a basis for Microsoft to terminate its license to use the technology, especially when Uber has acknowledged the failure and is committed to improvement”.
As part of an investigation for Wired from early March 2021, 14 Uber Eats couriers shared evidence with journalist Andrew Kersley that showed how the technology failed to recognize their faces, leading to threats of termination and account closure.
In terms of the impact on drivers, Farrar said, “They’ve either had to fall back on Universal Credit or try to find some alternative means of work and haven’t been able to get it – it’s had a devastating effect, and a lot of these guys are main breadwinners in their families as well”.
Computer Weekly asked Uber if it would like to comment on the claim that its facial-verification software has led to the wrongful deactivation of multiple driver’s accounts but received no response.
Other automated processes fire driver.
The ADCU is also appealing a series of other license revocation decisions by TfL at the Magistrate’s Court, which is similarly based on false information that led to accusations of fraud from Uber.
In April 2021, for example, Uber was ordered in a default judgment by the district court of Amsterdam (where Uber’s European headquarters is located) to reinstate six drivers because the decision to deactivate their accounts and terminate their employment was “based solely on automated processing, including profiling”, which goes against Article 22 of the General Data Protection Regulation (GDPR).
London-based driver Abdifatah Abdalla, for example, claimed he was accused, without providing evidence, of sharing his account details with a third party when the app detected two sign-in attempts from different locations resulting in his deactivation.
TfL revoked his private hire license a month later, leaving him unable to drive for alternative ride-hailing apps like Kapten and Bolt. In the same month, the City of London Magistrates’ Court separately ordered TfL to reinstate Abdalla’s private hire license, concluding that “no investigation has taken place, ” further criticizing the regulator’s “willingness to accept” the evidence provided by Uber.
Appeals
The drivers involved in the Amsterdam case have to separately appeal TfL’s licensing decisions in the UK Magistrate’s Court as the immediate revocation of a license means it cannot be appealed at the TfL level Farrar.
“As soon as TfL got these dismissal notices [from Uber], it took a very harsh view that these were safety threats and immediately revoked those licenses,” said Farrar. “Normally, immediate revocation would be for something like physical or sexual violence, but in these cases, TfL said they were immediate and those drivers and no notice – just ‘boom’, Kafkaesque, you’re deactivated from Uber, and you’ve lost your license.
“We [now] have to separately turn around and fight for the restoration of the licenses … appealing to the Magistrate’s to review the TfL decisions, but to do that; the bar is pretty high because you’ve got to prove the decision is wrong … it’s not just about presenting an alternative view, you have to reach a standard.”
Computer Weekly asked TfL if it would like to comment that multiple magistrates’ rulings have found it relied solely on evidence from Uber without conducting its investigation as a licensing authority into the actual events but received no response.
Rushed facial-verification implementation
The ADCU has previously claimed and maintains that Uber rushed to implement its identification system to win back its London operating license after TfL decided in late 2019 that it would not be renewed over problems with unauthorized drivers using the platform.
Evidence given to TfL by Uber during its licensing appeal in September 2020 showed that it had started to roll out the system from April 2020 onwards because of its manual identification process failures.
“TfL has taken an active interest in [Uber’s] proposals about this product. There are clear benefits to the product, and TfL supports any technology which increases passenger safety by ensuring the driver is licensed by TfL and permitted to use the Uber app,” wrote TfL’s director of licensing, regulation, and charging, Helen Chapman, in her witness statement.
“I consider the use of this product a step in the right direction, although clearly, its implementation is still at a very early stage,” said Chapman. “I, therefore, cannot meaningfully comment on the effectiveness of it at this stage.”
She added that TfL had received a Data Protection Impact Assessment (DPIA) for the system from Uber in March 2020. DPIAs are of note because they are designed to increase awareness around privacy and data protection issues within organizations and allow them to comply with relevant laws and identify and fix any problems before harm is caused.
“A DPIA is not simply a rubber stamp or a technicality as part of a sign-off process. It’s vital to integrate the outcomes of your DPIA back into your project plan,” says the Information Commissioner’s Office (ICO) in its guidance on conducting DPIAs. “You should not view a DPIA as a one-off exercise to file away. A DPIA is a ‘living’ process to help you manage and review the risks of the processing and the measures you’ve put in place on an ongoing basis.”
Letter to the mayor
In March 2021, the ADCU, WEI, and digital rights group Big Brother Watch co-signed a letter to London Mayor Sadiq Khan about the case of another driver misidentified by Real-Time ID Check.
In it, they claimed TfL had “placed significant pressure on Uber, under threat of losing their license, to rapidly introduce facial recognition identity technology in London”, adding that it has also refused to share the DPIA with the DCU.
“It is our view that the mandatory use of such technology cannot be justified and is entirely disproportionate to the risk. In the evidence TfL presented at Uber’s licensing appeal, only 21 drivers were found to share access to their driver app out of 90,000 analyzed over several years. This was only possible at the time due to a security problem in Uber’s app, which has since been rectified,” said the letter, seen by Computer Weekly.
“The use now of such intrusive, inaccurate, and dangerous technology is disproportionate and unnecessary since there are other more proportionate means to achieving the same end. TfL’s promotion of Orwellian levels of surveillance of 100,000 Londoners working in the gig economy is not a reasonable or proportionate response for the failure of the gig employer to audit and test the security of their systems.”
It added: “Checks, balances, and due diligence has failed at every level. TfL has bypassed all due process in its political rush to be seen as an aggressive enforcement authority.”
Given multiple reports of misidentification leading, in turn, to Uber’s deactivations and TfL license revocations, Computer Weekly asked the regulator if it still supported the roll-out of the technology but received no response on this point.
Computer Weekly also asked TfL whether it would like to comment on the ADCU’s claim that it had placed significant pressure on Uber to introduce facial-recognition technology but received no response.
