Your AI will believe whatever your data says

In May 2022, Unity Software told investors to expect about $110 million less revenue that year than planned. Its shares fell by around a third in a day.

Two things had gone wrong in its advertising business. A fault in its platform had reduced the accuracy of a tool called Audience Pinpointer, which decided which players to show which ads. And, in the company’s words on the earnings call, it had lost the value of part of its training data, “due in part to us ingesting bad data from a large customer”.

What happened

Machine learning models learn from whatever they are given. Unity’s models were trained on a stream of data about players and how they behaved. When bad data from one large source went into that stream, the models learned from it as faithfully as they learned from everything else. Advertisers saw the tool get worse, spent less on it, and the revenue followed.

Unity had to rebuild and retrain, and said the effect would run through most of the year. A model is only ever as good as what it was taught, and once it has been taught something wrong, unlearning it is expensive.

What was actually wrong

Unity is a sophisticated engineering company. This wasn’t a failure of skill. It was a failure of a step that is easy to skip because it feels like housekeeping: checking what goes into the model before it goes in.

A data source that changes its shape, its volume, or its typical values has stopped being the source the model was built on. That change is measurable before any training happens, and is cheap to catch at the door and expensive to find in the revenue figures.

The CRM version

Every major CRM is now adding AI that reads your records and acts on them: summarising accounts, drafting emails, deciding which leads matter, booking jobs. That AI will treat what is in your records as true.

In one field service company’s Salesforce, I found the due date on 1,999 of the last 2,000 appointments was exactly seven days after the earliest start. It wasn’t anyone’s deadline. It was a default the system filled in. A person who works there knows to ignore it. An AI agent asked “which jobs are overdue?” will not.

The same org had two in five job items pinned on the map in Ohio, when the addresses said London. A person sees the address. A routing model sees the coordinates.

None of this is exotic. Every org that has been running for a few years carries defaults, old loads, and fields people learned to ignore. People work around them without thinking. AI doesn’t know there is anything to work around.

Before your own rollout

Before you switch on an AI feature over your records:

• Find the defaults. Which dates, statuses and values are set by the system rather than by a person? If most records share one value, that value probably means nothing.
• Find what people ignore. Ask the team which fields they never trust. Those are the fields the AI will trust completely.
• Check the inputs, not just the outputs. Testing an AI’s answers tells you it is wrong. Testing what it reads tells you why, and before it costs anything.

Sources: Unity Software Inc., Q1 2022 earnings call, 10 May 2022; market coverage of the share price reaction, May 2022.