AI destroyed Paula’s life: – Was deprived of the opportunity to be a functioning human being

– I was deprived of the opportunity to be a functioning human being, says Paula Bouwer to NRK.

She was forced to pay the Dutch authorities several hundred thousand kroner after she was falsely accused of fraud.

– I worked in a bank during the day, and in a restaurant in the evening, to be able to pay the money to the authorities, she says.

She no longer got a loan from the bank. In order to survive, she had to sell off much of what she owned.

This happened due to an error in a computer system. Dutch authorities used artificial intelligence with machine learning to find tax cheats.

But the tool also picked out tens of thousands of others who had done nothing wrong.

– It was consistently people with a minority background who were flagged as social security fraudsters, says Bouwer.

Based on residential addresses, income, who one was married to and many other factors, the system picked out people who could be at risk of engaging in fraud.

– We did not find out why the KI had chosen to flag us as frauds.

Bouwer learned in 2021 that she was found not guilty. In the same year, the Dutch government had to go because of the scandal.

Afraid it could happen in Norway

There is a danger that algorithms will also discriminate in Norway, if the legislation is not changed, believes the Equality and Discrimination Ombudsman, Bjørn Erik Thon.

– The Discrimination Act should be changed now. These regulations were created before we almost even knew the word artificial intelligence, says Thon.

The government wants 80 per cent of the public sector to adopt artificial intelligence by next year.

Digitization Minister Karianne Tung believes that each individual business must find a version of AI that suits them.

The municipalities do not have the prerequisites for that, Thon believes.

– That is the recipe for something to go really wrong. I think giving a kind of marching order that everyone should now go out and use AI is probably not very smart, says Thon.

Digitization Minister Tung believes it is important to think anew in order to renew the public sector. She says that everyone who uses KI has a responsibility to do so responsibly, and refers to supervisors from the Equality and Discrimination Ombudsman (LDO)the Norwegian Data Protection Authority and the Directorate of Digitalisation.

– I expect everyone who uses AI to familiarize themselves with this to avoid possible pitfalls, she writes in an e-mail to NRK.

– What is not legal otherwise, such as discrimination, is also not legal when using AI, she emphasizes.

New report: The law should be changed

To secure that algorithmic discrimination is caught in a better way than today, the law against discrimination should be changed. This is confirmed by a recent report.

– The law is structured with the view that discrimination occurs in interpersonal relationships. But KI will enter across all areas of society, says pprofessor of jurisprudence at the University of Oslo, Vibeke Blaker Strand.

On assignment for LDO, she has mapped gaps in the legislation using artificial intelligence.

Artificial intelligence can easily categorize people into different groups. It can be a challenge if it is not regulated correctly.

– For example, there is nothing in the current law stating that it is not permissible to discriminate on the basis of address or educational background, says Strand.

In the Netherlands, people were sorted into such categories, based on this type of data.

In Norway, we are well advanced in adopting new technology. We should also be early on in securing robust legislation, which ensures that people are not discriminated against in the face of algorithms, Strand believes.

– The review showed that there are several points in the law where adjustments should be considered.

Huge consequences

In the Netherlands, the clean-up after the social security scandal is still ongoing.

The error in the public computer system was first discovered in 2018. In the previous years, many people were falsely accused of fraud.

Not everyone has received compensation. But according to Dutch media, 33,000 families have so far been compensated.

– Artificial intelligence flagged us as fraudsters. But if you forget the ethics of such tools, it can ruin people’s lives, says Paula Bouwer.

If the authorities are to use AI tools, people must know how the systems work, believes Dutch Bouwer.

– People must also be involved. You have to ensure that you actually do what the algorithm aims to do, she says.