A detection model that passed acceptance at 96 percent in March can be missing a quarter of its detections by September with no code change, no model change, and no alert. The camera drifted two degrees on its mount, the warehouse relamped to cooler LEDs, and a contractor arrived in a different shade of vest. Accuracy in computer vision is a property of the whole installation, and installations move.
THE PROBLEM
Why A Model That Passed Acceptance Stops Performing
A deployed model loses accuracy because the scene it watches keeps changing while the model stays frozen at the state it was accepted in. The gap opens slowly, it opens in one zone before it opens across a site, and it is invisible on a monitor because every one of these changes still looks fine to a human reviewing the same footage.
This is the failure mode that industry documentation calls data drift, and monitoring guidance for production vision systems treats continuous performance tracking as a standing operational requirement rather than a post-launch extra (Ultralytics model monitoring guide). Research on continuous training for deep-learning inspection systems makes the same point from the other side: retraining pipelines that run without a measured trigger can degrade a working model as easily as they repair one (arXiv 2409.09108).
The practical consequence for a buyer is narrow. A pilot proves that accuracy is achievable at your site. It proves nothing about whether accuracy will still be there in month fourteen, because that depends entirely on who is measuring and who is authorised to act on what they find.
CAUSES
Three Ways A Working Site Drifts
Most accuracy loss traces to one of three sources, and they call for different responses. Separating them matters because retraining a model will fix exactly one of the three.
The Image Changed
A pole camera settles. A cleaner nudges a housing. A firmware update resets minimum shutter to automatic, so the encoder lengthens exposure in low light and smears anyone walking. Smart codec gets re-enabled on the analytic stream and compresses the static background where small distant targets live. A varifocal lens creeps. None of these appear as a fault, and all of them remove information the model needs before the model ever sees a frame. Our camera and site requirements standard lists the settings that degrade analytics while looking identical to a human viewer.
The Scene Changed
New racking obstructs the far half of a zone. A loading door is bricked up and traffic reroutes through a corner the camera covers at 55 degrees. Site lighting is replaced and the colour temperature shifts, which matters because any colour-dependent rule, hi-vis vest detection included, needs white light at the target plane and stops working under infrared. Seasonal light does the same thing on a slower cycle, which is why a system commissioned in one season often shows its first real accuracy complaint in another.
The Rule Changed
The site adopts bump caps in the pick aisle, so the system now flags compliant workers. A second contractor arrives in orange where the standing rule assumed yellow. Operations tighten the tolerance on how long a violation must persist before it is worth an alert. Model accuracy is unchanged here. The definition of a correct answer moved, and only a label review catches it.
MEASUREMENT
How Drift Gets Measured Rather Than Argued About
Drift is measured by sampling frames from live zones, having a human label them, and comparing that label set against what the system reported for the same frames. Everything else is inference.
A workable cadence is small. Two hundred sampled frames per zone per quarter, drawn across shifts rather than from one convenient afternoon, is enough to move a conversation from impression to number. Standard divergence measures such as population stability index will flag that the distribution of inputs has shifted, and they are useful early warning, though they cannot tell you whether the shift hurt anything (Label Your Data, data drift detection). Only labelled ground truth does that.
One detail decides whether any of this is possible at all. ARSA processes video on site and video leaves your premises only where you explicitly configure it to. What crosses the network is analytic metadata: counts, events, and, where you enable it, small cropped images. That architecture is the reason customers choose on-premise, and it means ARSA cannot watch your accuracy from a remote dashboard by default. The sampling loop has to be designed and agreed, or it does not exist.
MAINTENANCE SCHEDULE
What To Re-Check, And On What Interval
The checks below re-verify the physical conditions that the accuracy figures depend on. The reference column gives the governing figure from ARSA-CVS-001.
| Check | Reference Figure | Suggested Interval | What Failure Looks Like |
|---|---|---|---|
| Pixel density at the far edge of each zone | 262 PPM for helmets, 153 PPM for vests | Annually, and after any camera or lens change | Detections fall off at range while the near field stays clean |
| Camera tilt below horizontal | Optimal 10 to 25 degrees, maximum 35 for helmets and 40 for vests, prohibited above 45 | Quarterly, visual, plus after any mount work | Helmets read from above, faces and vest fronts occluded by the wearer |
| Illumination at the target plane | Minimum 50 lux for helmet and vest detection | Quarterly, metered at the target, seasonally in daylit areas | Night and early shift accuracy diverges from day shift |
| Analytic stream settings | Smart codec off, constant bit rate, manual minimum shutter | After every camera firmware update | Sudden step change in accuracy with no site change to explain it |
| Frame rate and resolution | 15 fps minimum, 2 MP minimum and 4 MP recommended | Annually | Fast movement through a zone goes uncounted |
| Outdoor enclosure condition | IP66, with IK08 recommended | Twice yearly | Water ingress haze, spider webs, and IR wash on the housing glass |
| Label set and rule definitions | Site PPE policy of record | On every policy change | Compliant workers flagged, or a new violation class silently uncovered |
Zone geometry can be re-checked before anyone climbs a ladder. The lens and field of view visualiser will tell you whether a proposed camera position still reaches the required pixel density at the far gate line.
CONTRACT
The Accuracy Figure In Your Proposal Attaches To Accepted Zones
The accuracy numbers in an ARSA proposal apply to detection zones recorded as ACCEPTED in the Site Acceptance Record, and to no others. If a camera is moved, lighting is altered, or racking obstructs a view, the affected zones revert to unaccepted status until they are re-surveyed.
Buyers sometimes read that as a vendor protecting itself. The more useful reading is that it converts a vague argument about whether the system still works into a specific question about which zones changed and when. A site that keeps its acceptance record current knows exactly which of its zones carry a commitment. A site that does not keep it current will end up in month eighteen with an operations team that has quietly stopped trusting the alerts.
OWNERSHIP
Somebody Has To Own Accuracy After Handover
The single strongest predictor of whether a vision system still works in year two is whether one named person is accountable for its accuracy number. Most programmes that fail after a successful pilot fail here, because handover transferred the hardware and the login while leaving accuracy ownerless.
What that ownership contains is short: a sampling and labelling cadence, a re-survey trigger tied to physical site change, a threshold that decides when retraining is justified, and an agreed turnaround for pushing an updated model. ARSA contracts this as the support and update cadence, and the cost of it belongs in the payback model alongside the build, because it is the line item that determines whether the build keeps paying. Evaluators comparing custom computer vision development services should ask for that cadence in writing before signing, since a quotation that omits it has quietly moved the cost to your team.
Where the drift is physical rather than statistical, a review is cheaper than a retrain. A Remote Camera Design Review is $1,500 and takes one week, producing a marked-up camera schedule with a pass or fail per position against ARSA-CVS-001 and a priced remediation. An On-Site Camera Survey is $4,500 plus travel and covers up to 50 cameras. Either fee is deducted from the project fee if you contract within 90 days.
Frequently Asked Questions
How Often Does A Computer Vision Model Need Retraining?
There is no calendar answer, and any vendor giving you one is guessing. Retraining is triggered by measurement: a labelled sample showing accuracy below the agreed threshold in a specific zone. In practice, sites with stable conditions and a fixed PPE policy can run for well over a year without a model change, while a site adding new uniforms, new equipment, or new zones will need one within months.
Can Retraining Fix Drift On Its Own?
No, and attempting it is the most common wasted spend in vision maintenance. Retraining fixes a changed scene and a changed label definition. It cannot recover information that the camera no longer captures. A camera at 4 mm and 14 metres delivers 96 PPM against a 262 PPM helmet requirement, and no amount of additional training data puts the missing pixels back.
How Do We Measure Accuracy Without Labelling Everything?
Sample. A few hundred frames per zone per quarter, stratified across shifts and weather, gives a defensible accuracy estimate at a labelling cost of a few hours. Full labelling of a live stream is neither affordable nor necessary, since the purpose of the sample is to detect a change in a number you are already tracking.
Does Moving A Camera Void The Accuracy Commitment For That Zone?
Yes, until the zone is re-surveyed. A camera move changes distance, angle, and framing simultaneously, which is to say it changes all three of the variables the original acceptance was measured against. Re-survey of a single moved position is a small piece of work, and the cost of skipping it is discovering the problem through a missed incident.
Who Owns A Model Retrained On Our Footage?
Ownership is agreed before work starts, and your data remains yours in every case. A model trained exclusively on your data for your use case is typically yours to use, while the base architecture and pretrained weights remain ARSA’s. Data volume, labelling responsibility, and confidentiality terms are settled at the feasibility stage rather than at handover.
What If Our Existing Cameras Turn Out To Be The Problem?
That is a common finding and a cheap one to establish. A design review states the specific remediation per position and prices it separately: a different lens, a different position, a different camera, additional lighting, or an additional camera. You decide what to remediate, and zones left unremediated are simply excluded from the accuracy commitment rather than quietly counted in it.
Hold The Accuracy You Paid For
A pilot that degraded is a maintenance failure more often than a modelling failure, and the fix usually starts with a camera schedule rather than a training run. ARSA scopes the support and update cadence as part of the build, so accuracy has an owner from the day the system goes live.
Read what ARSA includes in Custom Computer Vision Development, review the conditions accuracy is measured against in the camera and site requirements standard, or contact ARSA with your current camera schedule and a description of where accuracy has slipped.
Sources: Ultralytics CV model monitoring and maintenance, Trimming the Risk: Towards Reliable Continuous Training for Deep Learning Inspection Systems, Label Your Data, data drift detection and monitoring