Overfitting, Evaluation And The Basics Of Trustworthy AI

A vendor tells you their model is 99 percent accurate. That number means very little until you know what it was measured on. A model can score perfectly on the data it learned from and still make poor decisions on the cases it meets in use.

This article explains why that happens, how careful evaluation exposes it, and how the US National Institute of Standards and Technology (NIST) places accuracy within a wider picture of what makes an AI system trustworthy.

What Overfitting Is

Overfitting is what happens when a model learns its training examples too well. It latches on to details that only exist in those examples, so it looks impressive during development and then gets new cases wrong, which is how Google’s machine learning course frames the problem.¹ The opposite quality, performing well on data the model has never seen, is called generalisation, and it is the whole point of machine learning.

An analogy helps. A student who memorises last year’s exam answers will score perfectly on last year’s paper and badly on this year’s. The student has learned the paper, not the subject. A model with enough parameters can do the same thing: it can memorise quirks and noise in its training examples rather than the underlying pattern.

Suppose a model learns to spot damaged parcels from warehouse photos. If every damaged parcel in the training set happened to be photographed at one loading bay, the model may learn to recognise that bay’s lighting rather than the damage. It will look excellent in training and fail at every other site.

How Held-Out Data Exposes It

The defence is to judge a model on data it did not learn from. That is why data is split into training, validation and test sets before training begins, as explained in How A Machine Learning Model Is Built.

During training, developers track the error, or loss, on both the training set and the validation set. Early on, both usually fall together. In an overfitting model, the training loss keeps falling while the validation loss levels off and may start to rise. Google’s course treats that divergence as a sign of overfitting.¹ It is a warning, not proof of exactly when memorising began. Common responses include early stopping, which Google’s course describes as ending training once validation loss starts to climb, along with simplifying the model or collecting more varied data.²

The test set has a special role. It is meant to be kept apart from every decision about the model and used at the end, so that it gives an independent estimate of performance. If a team keeps checking the test score and adjusting the model in response, the model gradually becomes tuned to that test set, and the score says less about new data.³

Accuracy Is Only One Number

A single accuracy figure can hide a lot. If one in a hundred transactions is fraudulent, a model that always says “not fraud” is 99 percent accurate and useless. For problems like this, teams track false positives (good cases wrongly flagged) and false negatives (bad cases missed) separately, because each has a different cost.

NIST’s AI Risk Management Framework makes a similar point. It names false positive and false negative rates among the measures to consider, and says any accuracy figure should come with a description of the test data, which should look like the situations the system will really face, and of the method used.⁴ It adds that accuracy results may be broken down by data segment. That breakdown is how a model that works well overall but poorly for one group of customers gets caught.

What Trustworthy AI Means In NIST’s Framework

NIST’s framework, published in January 2023 as voluntary guidance, lists seven characteristics of trustworthy AI systems.⁴ It describes “valid and reliable” as a necessary condition for the others, which is why evaluation sits at the base of the stack below. It treats accountability and transparency as relating to all of the other characteristics.

NIST stresses that these qualities have to be balanced against each other for the specific context of use; for example, an organisation may have to trade some predictive accuracy for a model that people can interpret. It also says AI systems should be tested before deployment and regularly while they are in operation, because a model that was valid at launch may not stay valid as conditions change.⁴ How organisations turn these principles into policy and controls is covered in AI Governance And Regulation, and the security characteristics in AI Security.

Questions To Ask Of Any Model

When you are buying or approving a model, a few questions go a long way. What data was it tested on, and does that data look like your own? Were results broken down by customer group, region or product line? How does it perform on the errors that matter most to you? Who monitors it after launch, and what triggers retraining?