A confused mass of images needs sorting out, any recommendations?

[aquatarkus]

My wife’s laptop failed and I replaced it, using what I thought was a full backup of her previous laptop to recover her data. While working through her photo archives she found many missing images so I have gone through several older backups retrieving folders of photos. In the process I learned that (a) she is an inveterate tinkerer with her file names and folder names and structures, and (b) import of images from phone to device isn’t always performed consistently. The net result is we have a lot of duplicated images in different locations and need to go through them identifying and eliminating duplicates.

I’m guessing this is not a problem that is unique to us and some members here will have had some experience in this field; can you recommend anything that would make this task more manageable?

At the moment there are approx 40,000 images and short videos in her Pictures library and I’d estimate that up to 25% of these are duplicates.

Thanks for any suggestions

Cheers
T

[Kirsty]

aquatarkus wrote:

The net result is we have a lot of duplicated images in different locations and need to go through them identifying and eliminating duplicates.

@Oldergeeks-com have a freeware program that can highlight duplicated files, that might be worth having a look at – usual proviso, do a backup before making such wholesale changes 🙂
AllDup v4.4.10 (last updated Nov. 20, 2019)

3 users thanked author for this post.

Elly, Bill C., aquatarkus

[chaloots]

More often than not I’ve downloaded/installed apps for a specific purpose, all the while being oblivious to or forgetting that an already existing installed app was capable of performing that very same function.

While I’m also aware of the dubious reputation Ccleaner has created for themselves over the past year or so, I use an older portable version, 5.46, which has a built-in duplicate finder, with options to fine tune the search. Point being: you too may already have a dupe finder feature tucked away in an app you may typically use for a different purpose.

The duration of a minute depends on which side of the bathroom door you're on.

[EP]

Speaking of CCleaner, avast just released version 5.64.7613 of CCleaner recently, which they declare this new version the last one to support Windows XP & Vista:

https://www.ccleaner.com/news/release-announcements/2020/03/03/ccleaner-v5647613

– Version 5.64.7577 is the final planned build for Windows XP and Vista

• This reply was modified 3 weeks, 3 days ago by EP.

[JohnW]

That XP & Vista final version of CCleaner should be fine for a good long while, considering that they no longer have to keep pace with changes in those old operating systems…  🙂

But the worst case is that you might miss out on some new CCleaner features developed down the road, but as a mature application that’s probably not likely to be a deal breaker, and it should remain quite useful.

[Microfix]

I was initially thinking of using some sort of exif (metadata) duplicate finder but to no avail 🙁
Upon this search I came across https://www.auslogics.com/en/software/duplicate-file-finder/
which may be of some use in sorting the image duplication out.

Win7 Pro x86/x64 | Win8.1 Pro x64 | Linux Hybrids x86/x64 |

1 user thanked author for this post.

aquatarkus

[Alex5723]

aquatarkus wrote:

At the moment there are approx 40,000 images and short videos in her Pictures library and I’d estimate that up to 25% of these are duplicates.

Visual Similarity Duplicate Image Finder :

“will find all similar and duplicate images, pictures and photos in a folder and its subfolders, computer or network in order to organize photos.
The duplicate image finder uses advanced algorithms to find duplicate photos in a way that a human does. Due to that fact it can find duplicate images even if they are in different image formats, different bit depths, and image sizes.”..

https://www.mindgems.com/products/VS-Duplicate-Image-Finder/VSDIF-About.htm

[bbearren]

I use NameEXIF for all my digital photos.  The only renaming options are by date taken, which is taken from the file metadata.  I use yyyy-mm-dd-time, then use the date to sort into folders named by year/sub-folders named by month.

It will rename large batches in a single run.  For duplicates it adds a number.

Create a fresh drive image before making system changes/Windows updates, in case you need to start over!

"When you're troubleshooting, start with the simple and proceed to the complex."—M.O. Johns

"Experience is what you get when you're looking for something else."—Sir Thomas Robert Deware

Computer Specs (second paragraph)

• This reply was modified 1 month, 3 weeks ago by bbearren.

[aquatarkus]

Thanks for the suggestions.

I have tested AllDup and the AusLogics duplicate finder. They agree 🙂

Looks like my 25% estimate was almost exactly correct 🙁

Cheers

1 user thanked author for this post.

Microfix

[hhodges]

It’s remarkable how such a seeming easy task it ought to be to identify duplicates, but it’s not.  Various schemes are used and the tools rarely tell you how they work.  Each can use a different method, CRC, Hash Values, etc to compare but the actual calculation can be complex.  I’ve learned to rely on CloneMaster. It’s quite reliable at finding exact duplicates and after checking often to be sure, I allow the tool to be automatic.   But it’s somewhat slow on a large collection because it’s single threaded.  For a muti-threaded, faster tool I use DupDetectorPro which is multi-threaded and can even work across a network.  For the average user, CloneMaster would be fine.  If you have a bunch of de-duping work particularly across a network DupDetector is solid.

http://softbytelabs.com/wp/clonemaster/

https://www.dupscout.com/

[Paul T]

Clonemaster =  shareware, $40.
DupDetector = commercial, $50 – $625.

cheers, Paul

[DaxPagan]

Recently, there was a discussion on software for finding near-duplicate images.  (https://www.askwoody.com/forums/topic/a-confused-mass-of-images-needs-sorting-out-any-recommendations/)  I’m interested in finding near-duplicate video files.

I have a lot of videos downloaded from YouTube, Vimeo, or acquired from others over time.  I know there are “near duplicates” – videos that differ by resolution or size differences because they were downloaded at different times, downloaded by different people, uploaded by different people, trimmed/edited slightly differently, etc.

General duplicate-finding software can find exact matches for size or title of video files.  Is there software that can search the content of the file to find “near duplicates” of video files like there is for image files?

[Alex5723]

Try Video Duplicates Finder : https://www.visioforge.com/video-duplicates-finder-sample

The SDK can detect a video even if it is integrated into another clip, or if only a piece of it is available…

[Zig]

Darned if I can get this SW to work, even running as administrator and in Win 7 Compatibility Mode. Just goes away if I use LAV Decoding Engine. Unable to load FFMPEG and VLC (both installed on my computer).

[Gingermum]

For many years I have used “Duplicate File Finder” by Rashid Hoda and I admit I am still using an old version. It’s free and allows me to select which folders to search and what type of file to search for or if I want to find any type. All duplicate files are shown with their path and by default nothing is ticked for deletion. I can then select one by one which to delete. (I do find this useful) or just opt to delete all duplicates. I can also see a preview of the file I have currently selected. Really happy with it. My old version doesn’t work across a network but possibly the newer one does. As it’s free, give it a try.

[Author]

Posts