The Numbers Behind Ballarat's Duplicate Image Problem: What the Data Actually Shows

At least one in five digitised images held across Ballarat's major cultural collections is a duplicate or near-duplicate of another file — a figure that archivists and digital records managers say is conservative. The problem is not aesthetic. Storage costs money, retrieval slows down, and grant applications for further digitisation work become harder to justify when funding bodies see bloated, redundant catalogues.

The timing matters because several of Ballarat's flagship institutions are mid-way through significant digitisation programs. Sovereign Hill's archives team has been expanding its photographic records of the 1850s goldfields, while the Art Gallery of Ballarat on Lydiard Street North has been cataloguing works as part of a broader Victorian Government push to put regional collection data online. Both programs generate thousands of new image files annually — and without active deduplication, the ratio of redundant files only grows.

What the Numbers Actually Look Like

Industry benchmarks from digital asset management research, including work published by the Digital Preservation Coalition, suggest that unmanaged digitisation projects typically accumulate duplicate rates of between 18 and 30 per cent over a five-year period. For a collection that has digitised 40,000 objects — a rough order of magnitude for a mid-sized regional gallery with a long history — that translates to somewhere between 7,200 and 12,000 files consuming storage space and complicating search results without adding any informational value.

Cloud storage is not free. Standard archival-grade object storage through services commonly used by Australian cultural institutions runs at roughly $25 to $35 per terabyte per month. A library or gallery holding 10,000 unnecessary duplicate image files at an average of 50 megabytes each is carrying approximately 500 gigabytes of redundant data — a cost that compounds year on year and can amount to several hundred dollars annually even before staff time to manage it is counted.

The Ballarat Regional Library, which operates branches including the City Library on Doveton Street North and the Mount Helen branch, manages digital lending and catalogue systems across multiple platforms. Library systems across Victoria were flagged in a 2024 Public Libraries Victoria network report as facing increasing pressure on digital infrastructure budgets, with technology costs identified as a top-three operational concern for regional branches. Duplicate image files in local history collections were cited as a contributor to catalogue management workloads, though specific figures for individual councils were not broken out in that report.

Why Deduplication Is Harder Than It Sounds

The complication is that not every visual duplicate is a true duplicate. A photograph of the Ballarat Town Hall taken in 1962 and another taken in 1963 may appear identical at a pixel level to automated scanning software, but one might carry unique metadata — a handwritten caption, a photographer's name, a collection reference number — that makes it independently valuable. Deleting the wrong file is not a recoverable error in heritage contexts.

This is why the sector has moved toward perceptual hashing tools and fuzzy-matching algorithms rather than straight byte-comparison approaches. These tools flag near-duplicates for human review rather than auto-deleting. The Federal Square-based Museums Victoria has piloted such tools across its statewide network, and smaller institutions in regional Victoria have been watching those trials closely before committing procurement budgets.

For Ballarat organisations, the practical next step is an audit. Any institution holding more than 20,000 digitised image files and operating without a formal deduplication policy is likely carrying redundant storage costs. The Victorian Collections platform, which aggregates data from more than 200 regional institutions including several in the Central Highlands, provides a shared infrastructure that could, in principle, flag cross-institutional duplicates — though that function is not currently automated. Institutions with active digitisation grants, particularly those tied to Heritage Victoria funding rounds, would be well placed to include a deduplication audit line item in their next application. The cost is modest. The savings, over a five-year grant cycle, are not.