Fake citations causing real-world damage

The world's most cited cat had a brief, successful run. Then Google Scholar brought his career to an abrupt halt.

For a brief moment, Larry Richardson seemed like he was on a trajectory to success as a prolific academic.

His 12 papers were cited nearly 150 times, within just a couple of hours.

But his success was short-lived. After only a week, Google Scholar recognised that the papers and citations were fake - everything had been generated by AI.

And Larry it turns out, was fake too. He was not the man who appeared in the profile photo, but actually that man's grandmother's cat.

It was all part of an experiment conducted by two researchers in the UK. Their goal was to prove that the way researchers' productivity is measured is flawed, and easy to manipulate, and it showed how easily fake citations are making their way into academic research.

In July, a non-profit journalism institute discovered the Make America Healthy Again report released by the Trump administration was riddled with errors and cited studies that did not exist.

And it's not just happening in the science field. There have been a slew of incidents where both lawyers and plaintiffs representing themselves have been caught referencing fake case studies.

Associate law professor at Otago University Simon Connell, said when it comes to fake citations in the legal field, AI is the problem.

"There's a legal researcher who has been tracking AI hallucination cases and so far has got a list of 380 legal cases from a range of different jurisdictions where hallucinated cases have been used," he said.

AI hallucinations are when an artificial intelligence model, like Chat GPT, generates misleading information which is plausible, but isn't factual.

Connell said these hallucinations are a huge problem in the legal field.

"If you say, 'well the law is like this because this case says it is,' and that case doesn't exist then your argument completely falls flat," he said.

In New Zealand there have been a couple of incidents where fake cases have been used to make a legal argument, but so far it seems these have come from citizens representing themselves, and not from lawyers.

But it's enough to prompt concern among experts. Earlier this month the New Zealand Law Society published an article warning lawyers to beware of fake cases.

For Peter Gluckman, president of the International Science Council, this is just part of a much larger problem, which is the way that academic quality is measured, which he said is flawed, easy to manipulate and values quantity over quality.

"What happened was until about 1980 the global scientific community was relatively small ... and in your field you could be aware of all of the live scientist journals ... by referring to a little booklet that was published every week.

"[But] then it got computerised into larger systems and people decided they could put metrics around it," he said.

Those metrics considered how many papers a researcher had published and how many times those papers had been cited in other sources, a system that's become known as the H-index.

"And that became a proxy for quality used by employers, universities, by funders, even by governments as a pseudo-measure of impact."

A higher H-index is seen as desirable. But it's a flawed metric, Gluckman said, and it's easily manipulated. To raise their H-index, researchers can cite their own work in new papers, or like the scientists behind Larry the cat proved, they can use a citation-boosting AI model.

Gluckman said science tends to correct itself overtime, but that AI makes it easier for people to falsify information - part of a concerning trend he's seen over the last decade.

"Whether it's in law or in science or in any other area, the second that reality is compromised and we're living in a world that is made up, then societies ultimately can't function well."

Check out how to listen to and follow The Detail here.

You can also stay up-to-date by liking us on Facebook or following us on Twitter.

Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.