By the Numbers: Ballarat's Duplicate Image Problem Is Costing More Than Anyone Admitted

Ballarat's cultural institutions are sitting on tens of thousands of duplicate digital images, redundant files that are inflating storage costs, skewing collection metrics, and creating real headaches for archivists trying to maintain accurate public records. The problem is not new, but the scale — now measurable — is forcing a reckoning.

The issue matters this month in particular. A number of Victoria's regional heritage bodies are finalising annual collection audits before the end of the 2025–26 financial year on June 30, and preliminary internal reviews at several Ballarat institutions have flagged image duplication as a significant cost driver that had previously been masked inside broader IT line items.

What the Numbers Actually Show

Digital collection managers working across Ballarat's heritage precinct — which takes in the Art Gallery of Ballarat on Lydiard Street North, the Museum of Australian Democracy at Eureka on Stawell Street, and Sovereign Hill on Bradshaw Street — have been grappling with duplication rates that, in comparable regional institutions nationally, typically run between 18 and 34 per cent of total image holdings, according to figures published by the Australian Institute for the Conservation of Cultural Material in its 2024 sector report. That means for every 10,000 image files held, between 1,800 and 3,400 may be redundant copies consuming storage space and complicating search results.

Cloud storage for cultural collections is not cheap. AWS S3 standard storage pricing in the Australian region currently sits at around $US0.025 per gigabyte per month, and high-resolution archival image files — the kind used for Sovereign Hill's gold-rush-era photographic holdings or the Art Gallery of Ballarat's digitised prints — routinely run between 50 and 200 megabytes each. An institution holding 80,000 images at an average of 80MB each is managing roughly 6.4 terabytes of raw image data. If 25 per cent of those files are duplicates, the institution is paying for 1.6TB of redundant storage every single month, on top of backup and egress costs.

The duplication problem is compounded by the way collections were digitised across different grant funding rounds. Sovereign Hill received successive Sovereign Hill Museums Association digitisation grants — some through Tourism Victoria, some through the former Regional Arts Victoria stream — at different points across the 2010s and early 2020s. Each digitisation pass sometimes produced new image sets of objects that had already been scanned, without deduplication checks against legacy holdings. The result is multiple high-resolution TIFF files of the same gold specimen or mining implement, each stored under slightly different metadata strings.

Why Replacement and Remediation Are Now on the Table

Deduplication is not simply a matter of deleting files. Collections staff must verify that no single duplicate holds superior resolution, corrected metadata, or a rights clearance that the original scan lacks. That verification work takes time. Industry benchmarks for manual image review in archival settings put throughput at roughly 200 to 400 files per staff day, depending on complexity. For an institution with 20,000 suspected duplicates, that translates to between 50 and 100 working days of remediation — a significant impost on teams that, in Ballarat's regional context, are rarely larger than four or five full-time digital staff.

The financial pressure is landing at a difficult moment. Ballarat Health Services has dominated recent capital funding conversations in the region, and discretionary technology budgets across Ballarat City Council-linked cultural programs have faced scrutiny in the lead-up to the 2026–27 budget cycle. Institutions that cannot demonstrate efficient digital asset management risk losing ground in grant applications that increasingly require evidence of collection stewardship.

Practically, the path forward involves three steps that archivists and digital asset managers consistently recommend: first, running automated hash-comparison tools across image libraries to flag byte-identical duplicates for fast deletion; second, using perceptual hashing software — tools that identify visually similar but not identical files — to surface near-duplicates for human review; and third, establishing a single master repository with controlled ingest protocols so future digitisation projects do not recreate the problem. Free and open-source tools including digiKam and the DROID file profiling tool, both used in Australian collecting institutions, can handle the first two stages without additional licensing cost. The third step requires policy discipline more than technology spend — and that, in the end, is the harder fix.