3 hours ago
7
AI tools are making potentially harmful errors in social work records, from bogus warnings of suicidal ideation to simple “gibberish”, frontline workers have said.
Keir Starmer last year championed what he called “incredible” time-saving social work transcription technology. But research across 17 English and Scottish councils shared with the Guardian has now found AI-generated hallucinations are slipping in.
As scores of local authorities begin to use AI note-takers to accelerate recording and summarisation of meetings with adult and child service users, an eight-month study by the Ada Lovelace Institute found “some potentially harmful misrepresentations of people’s experiences are occurring in official care records”.
The independent thinktank found that one social worker who had used an AI transcription tool to create a summary said the technology had incorrectly “indicated that there was suicidal ideation”, but “at no point did the client actually … talk about suicidal ideation or planning, or anything”.
Another said that the AI’s notes might refer to “fishfingers or flies or trees” when in fact a child was talking about their parents fighting. Social work experts said such glitches were particularly worrying as it could cause a risky pattern of behaviour to be missed.
Other social workers raised concerns about inaccuracies in transcribed conversations with people with regional accents. One described how their AI-generated transcriptions often included “gibberish”. Another said: “It’s become a bit of a joke in the office.”
Dozens of councils from Croydon to Redcar and Cleveland have given social workers access to AI transcription tools, which record and summarise case conversations. The potential time savings are appealing to town halls with chronic staff shortages.
One popular system, called Magic Notes, is sold to councils at a cost of between £1.50 and £5 for each hour of transcription. Most of the social workers interviewed used either the specialist Magic Notes AI or the general purpose Microsoft Copilot AI.
The research also found AI transcription delivered observable time savings and liberated social workers to focus more on relations with service users.
“Our evidence shows these tools can also improve relational aspects of care work and the quality of information recorded by social workers,” it said after interviews with 39 unnamed social workers.
But when one social worker used an AI tool to redraft care documents in a more “person-centred” tone, the system inserted “all these words that have not been said”. Another social worker reported the technology had “crossed the line between it being your assessment and being AI’s assessment”.
“AI-produced inaccuracies that enter these records may have far-reaching impacts, such as a social worker making an incorrect decision about a child’s care, which could lead to harm for the child and professional consequences for the social worker,” the report concluded.
The impact of AI errors is already being felt in the profession, with reports of disciplinary action for failing to properly check the outputs of AI note-takers and missing obvious errors, according to the British Association of Social Workers (BASW). It is calling for social work regulators to issue clear guidance on how and when AI tools should be used.
While there is “real excitement” among some social workers about their potential, “these tools also introduce new risks to social work and society, from potential bias in report summaries to inaccurate ‘hallucinations’ in transcripts,” said Imogen Parker, associate director at Ada Lovelace. “These risks aren’t being fully assessed or mitigated, leaving frontline workers to navigate these challenges on their own.”
Social workers often receive little AI training – just an hour in one case. While some social workers said they spent up to an hour checking AI transcripts, others said they spent just two minutes. One said they took “five minutes to literally just quickly screen it […] and then cut and paste it on to the system”. Another said AI-generated cut-and-paste care plans could read “horrifically”.
Others said some colleagues were too lazy or busy to check the transcripts.
“The risk here is that people aren’t checking what’s been written for them,” said Andrew Reece, BASW strategic lead for England and Wales. “The time you spend writing helps you make sense of what you have heard. If the computer is doing that for you, you’re missing out on the important parts of reflective practice.”
Beam, which operates Magic Notes, stressed its outputs were a first draft, not a final record. “AI tools are being embraced by social workers for good reason,” said Seb Barker, its co-founder. “Services are overwhelmed and hard to access; a generation of social workers are at risk of burnout, and needs for accurate, compliant documentation are growing.”
He said an evaluation of bias had found Magic Notes performed “consistently and equitably” and highlighted its specialist features for social work, including automated hallucination risk checks. “Not all AI tools are the same, with unspecialised, low-quality or generic tools failing the specific needs of the sector,” he said.
The UK government and Microsoft were approached for comment.
