UK regulator calls bullshit on claim by employers that they review applications ranked poorly by AI

Let’s face it: hiring people is exhausting. When you post a job opening, you get flooded with hundreds or even thousands of resumes. To save time, you probably bought an AI hiring tool to sort through them.

You also probably told yourself a comforting little lie to feel safe about using it. You told yourself, “The AI just gives the resumes a score to help us out, but a real human always double-checks the work.” Well, a major government agency just called foul on that excuse.

The UK’s Information Commissioner’s Office (ICO)—the official group that watches over data privacy laws—recently investigated how companies use AI to hire. Their verdict? The idea that humans are carefully supervising these tools is a total myth.

If you use AI to screen job seekers, you need to wake up. The way most companies use these tools is likely breaking the law, and the defense paperwork your software gives you isn’t going to save you.

The “Human Supervisor” is a Lie

The ICO put out a major report based on evidence they gathered from dozens of employers. They found a huge gap between what companies say they are doing and what is actually happening.

Most employers think they are safe from strict data laws because a human manager makes the final hiring choice. But the regulator said that doesn’t matter.

Think about how your hiring process actually works:

• The AI looks at 1,000 resumes.
• It picks the top 50 and puts them on a shortlist.
• It dumps the other 950 into “lower score” piles.
• Your human recruiter looks only at that top 50 list and picks 10 to interview.
• Your recruiter picks three finalists and forwards those to your hiring manager.
• Your hiring manager extends an offer to one of those finalists and that applicant accepts.

The ICO’s point is simple: The human recruiter and, therefore, the hiring manager didn’t make the choice as to which applications to review. The AI did. If a recruiter just clicks “approve” on a list generated by a machine, and never actually checks the people who were ranked poorly, that is called fully automated decision-making. According to Tech Times, doing this without telling the applicants or giving them a way to fight the decision is flat-out illegal under data laws like GDPR.

The UK government has already started cracking down and warning major companies to fix this. Other governments can’t be far behind. Regardless of where you’re hiring people, understand that a human supervisor can’t just be a rubber stamp. They must have the time, the information, and the power to go into the rejected pile and overrule the machine. If your recruiters never look at the bottom of the pile, your company is exposed.

The Secret Under the Hood: Your Tool is Just a “Wrapper”

Why are these AI tools doing such a bad job of ranking people in the first place? To understand that, you have to look at how they are built.

When a software salesman sells you a “smart candidate matching platform,” they make it sound like their data scientists built a custom, high-tech brain specifically for your business.

They didn’t.

Almost all of these new HR tools are what tech people call wrappers. This means the software company didn’t build an AI at all. They just built a nice-looking website that connects to the exact same AI models you use at home for free—like OpenAI’s ChatGPT, Anthropic’s Claude, Perplexity AI, or Google’s Gemini.

When a resume comes in, the software secretly sends it to ChatGPT’s brain (or whatever model you’re using) with a prompt that says: “Read this resume, compare it to the job posting, and give the resume a score from 1 to 100 for how well it matches up against the job posting.” You aren’t paying for a brilliant human resources expert. You are paying for a fancy skin on top of a generic chatbot that was trained on random internet text.

The Mirror Trap: Why the AI Picks the Wrong People

Because these tools are just wrappers on Big Tech models, it creates a massive, unfair echo chamber.

Think about it from the job seeker’s side. What do college students and young professionals do when they apply for jobs? They use ChatGPT to write their resumes. They copy your job description, paste it into the AI, and say, “Make my resume match this perfectly.”

Now look at what happens when that resume hits your computer:

[Candidate uses ChatGPT to write resume] 
               │
               ▼
[Employer uses a tool built on ChatGPT to score resumes]
               │
               ▼
[AI gives the resume a perfect score because it recognizes its own style]

If your hiring tool is built on OpenAI’s brain, and a candidate used ChatGPT to write their resume, your tool is going to give them a massive score. Why? Not because they are the best person for the job, but because the AI loves its own writing style. It recognizes the specific words, sentence structures, and patterns it likes to use.

On the other hand, imagine a brilliant, highly qualified candidate who wrote their resume by hand, or used a different AI model like Claude. Because their writing pattern doesn’t match the math inside your software’s brain, the tool might give them a terrible score and throw them away.

Your expensive software isn’t finding the best talent. It’s just finding the people who used the exact same chatbot you did.

The AI is Literally Making Stuff Up

When a human manager or a lawyer asks, “Why did the AI rank this candidate so poorly?” employers expect the software to give a real, logical answer. You click a button, and the software spits out a clean PDF report that says: “Candidate lacks leadership metrics.”

You think you are safe because you have an “audit trail.” But here is the scariest secret in tech: The AI is lying to you.

The companies that build these models—like OpenAI and Google—openly admit that they do not know exactly how their AIs make decisions. The inside of an AI brain is a giant, messy web of math, not a list of logical rules.

So, when you ask the tool why it rejected someone, it cannot look backward and find the real reason. Instead, it looks at the resume and guesses a reason that will make you happy.

In the tech world, this is called a sycophantic response. It means the AI is a suck-up. It knows you want a professional-sounding explanation, so it creates a hallucination—a made-up story—and tells it to you with absolute certainty. The AI didn’t actually reject the candidate for the reasons in the report. It rejected them because of an arbitrary math glitch, and then fabricated a nice story afterward to cover its tracks.

Why Your Audit Trail Won’t Save You in Court

If the AI is just making up stories to explain its rankings, your compliance reports are legally worthless. They are fictional receipts for a decision that didn’t happen that way.

If a rejected job seeker sues your company for discrimination—claiming your software filters out older workers, women, or minorities—a judge is not going to care about your AI-generated PDF.

In a real lawsuit, the court will bring in computer experts to look under the hood of your software. When those experts prove that the software’s “explanation” was just a chatbot hallucinating a story after the fact, your defense will fall apart. Courts will look at a fake AI audit trail the exact same way they look at a human manager who falsified documents to cover up a biased decision.

We are already seeing this happen. In the U.S., a massive class-action lawsuit called Mobley v. Workday has changed everything. The courts ruled that AI software companies can be held liable for discrimination, and judges are now forcing companies to hand over their data and client lists.

If you are relying on a chatbot’s guess to protect your company from a multi-million dollar lawsuit, you are walking into a trap.

How to Protect Your Company Right Now

You don’t have to completely ban AI from your office, but you have to stop trusting it blindly. If you want to stay safe from regulators and courts, you need to change how you hire immediately:

• Audit your pile: Force your recruiters to look at the bottom of the list. Regularly pull random resumes that the AI gave an F-grade and read them. If you find great candidates down there, your software is broken.
• Interrogate your software vendors: Call your software provider and ask them hard questions. Ask them: “Whose AI brain are you using?” and “Are the explanations your tool gives me real math, or is the AI just generating a story after the fact?”
• Stop the rubber-stamping: Change your internal rules. Ensure human recruiters are actively looking at a wide variety of candidates, not just the top five names the machine feeds them.

AI can be a helpful assistant, but it cannot be the boss. The moment you let an opaque machine decide who gets a job—and accept its fake excuses without question—you lose control of your business and your legal safety. Turn off the auto-pilot and put humans back in charge of hiring humans.