News, tips, advice, support for Windows, Office, PCs & more. Tech help. No bull. We're community supported by donations from our Plus Members, and proud of it
Home icon Home icon Home icon Email icon RSS icon
  • A confused mass of images needs sorting out, any recommendations?

    Posted on aquatarkus Comment on the AskWoody Lounge

    Home Forums AskWoody support Windows Windows 10 Questions: Win10 A confused mass of images needs sorting out, any recommendations?

    Topic Resolution: Resolved

    Tagged: 

    Viewing 8 reply threads
    • Author
      Posts
      • #2124625 Reply
        aquatarkus
        AskWoody Plus

        My wife’s laptop failed and I replaced it, using what I thought was a full backup of her previous laptop to recover her data. While working through her photo archives she found many missing images so I have gone through several older backups retrieving folders of photos. In the process I learned that (a) she is an inveterate tinkerer with her file names and folder names and structures, and (b) import of images from phone to device isn’t always performed consistently. The net result is we have a lot of duplicated images in different locations and need to go through them identifying and eliminating duplicates.

        I’m guessing this is not a problem that is unique to us and some members here will have had some experience in this field; can you recommend anything that would make this task more manageable?

        At the moment there are approx 40,000 images and short videos in her Pictures library and I’d estimate that up to 25% of these are duplicates.

        Thanks for any suggestions

        Cheers
        T

      • #2124650 Reply
        Kirsty
        Da Boss

        The net result is we have a lot of duplicated images in different locations and need to go through them identifying and eliminating duplicates.

        @Oldergeeks-com have a freeware program that can highlight duplicated files, that might be worth having a look at – usual proviso, do a backup before making such wholesale changes 🙂
        AllDup v4.4.10 (last updated Nov. 20, 2019)

        3 users thanked author for this post.
        • #2174905 Reply
          chaloots
          AskWoody Plus

          More often than not I’ve downloaded/installed apps for a specific purpose, all the while being oblivious to or forgetting that an already existing installed app was capable of performing that very same function.

          While I’m also aware of the dubious reputation Ccleaner has created for themselves over the past year or so, I use an older portable version, 5.46, which has a built-in duplicate finder, with options to fine tune the search. Point being: you too may already have a dupe finder feature tucked away in an app you may typically use for a different purpose.

          The duration of a minute depends on which side of the bathroom door you're on.

          • #2176351 Reply
            EP
            AskWoody_MVP

            Speaking of CCleaner, avast just released version 5.64.7613 of CCleaner recently, which they declare this new version the last one to support Windows XP & Vista:

            https://www.ccleaner.com/news/release-announcements/2020/03/03/ccleaner-v5647613

            – Version 5.64.7577 is the final planned build for Windows XP and Vista

            • This reply was modified 3 weeks, 3 days ago by EP.
            • #2176489 Reply
              JohnW
              AskWoody Plus

              That XP & Vista final version of CCleaner should be fine for a good long while, considering that they no longer have to keep pace with changes in those old operating systems…  🙂

              But the worst case is that you might miss out on some new CCleaner features developed down the road, but as a mature application that’s probably not likely to be a deal breaker, and it should remain quite useful.

      • #2124651 Reply
        Microfix
        Da Boss

        I was initially thinking of using some sort of exif (metadata) duplicate finder but to no avail 🙁
        Upon this search I came across https://www.auslogics.com/en/software/duplicate-file-finder/
        which may be of some use in sorting the image duplication out.

        Win7 Pro x86/x64 | Win8.1 Pro x64 | Linux Hybrids x86/x64 |
        1 user thanked author for this post.
      • #2124669 Reply
        Alex5723
        AskWoody Plus

        At the moment there are approx 40,000 images and short videos in her Pictures library and I’d estimate that up to 25% of these are duplicates.

        Visual Similarity Duplicate Image Finder :

        “will find all similar and duplicate images, pictures and photos in a folder and its subfolders, computer or network in order to organize photos.
        The duplicate image finder uses advanced algorithms to find duplicate photos in a way that a human does. Due to that fact it can find duplicate images even if they are in different image formats, different bit depths, and image sizes.”..

        https://www.mindgems.com/products/VS-Duplicate-Image-Finder/VSDIF-About.htm

      • #2124700 Reply
        bbearren
        AskWoody MVP

        I use NameEXIF for all my digital photos.  The only renaming options are by date taken, which is taken from the file metadata.  I use yyyy-mm-dd-time, then use the date to sort into folders named by year/sub-folders named by month.

        It will rename large batches in a single run.  For duplicates it adds a number.

        Create a fresh drive image before making system changes/Windows updates, in case you need to start over!
        "When you're troubleshooting, start with the simple and proceed to the complex."—M.O. Johns
        "Experience is what you get when you're looking for something else."—Sir Thomas Robert Deware

        • This reply was modified 1 month, 3 weeks ago by bbearren.
      • #2124926 Reply
        aquatarkus
        AskWoody Plus

        Thanks for the suggestions.

        I have tested AllDup and the AusLogics duplicate finder. They agree 🙂

        Looks like my 25% estimate was almost exactly correct 🙁

        Cheers

        1 user thanked author for this post.
      • #2174801 Reply
        hhodges
        AskWoody Plus

        It’s remarkable how such a seeming easy task it ought to be to identify duplicates, but it’s not.  Various schemes are used and the tools rarely tell you how they work.  Each can use a different method, CRC, Hash Values, etc to compare but the actual calculation can be complex.  I’ve learned to rely on CloneMaster. It’s quite reliable at finding exact duplicates and after checking often to be sure, I allow the tool to be automatic.   But it’s somewhat slow on a large collection because it’s single threaded.  For a muti-threaded, faster tool I use DupDetectorPro which is multi-threaded and can even work across a network.  For the average user, CloneMaster would be fine.  If you have a bunch of de-duping work particularly across a network DupDetector is solid.

        http://softbytelabs.com/wp/clonemaster/

        https://www.dupscout.com/

        • #2174908 Reply
          Paul T
          AskWoody MVP

          Clonemaster =  shareware, $40.
          DupDetector = commercial, $50 – $625.

          cheers, Paul

      • #2175050 Reply
        DaxPagan
        AskWoody Plus

        Recently, there was a discussion on software for finding near-duplicate images.  (https://www.askwoody.com/forums/topic/a-confused-mass-of-images-needs-sorting-out-any-recommendations/)  I’m interested in finding near-duplicate video files.

        I have a lot of videos downloaded from YouTube, Vimeo, or acquired from others over time.  I know there are “near duplicates” – videos that differ by resolution or size differences because they were downloaded at different times, downloaded by different people, uploaded by different people, trimmed/edited slightly differently, etc.

        General duplicate-finding software can find exact matches for size or title of video files.  Is there software that can search the content of the file to find “near duplicates” of video files like there is for image files?

        • #2175136 Reply
          Alex5723
          AskWoody Plus

          Try Video Duplicates Finder : https://www.visioforge.com/video-duplicates-finder-sample

          The SDK can detect a video even if it is integrated into another clip, or if only a piece of it is available…

          • #2176469 Reply
            Zig
            AskWoody Plus

            Darned if I can get this SW to work, even running as administrator and in Win 7 Compatibility Mode. Just goes away if I use LAV Decoding Engine. Unable to load FFMPEG and VLC (both installed on my computer).

      • #2175088 Reply
        Gingermum
        AskWoody Lounger

        For many years I have used “Duplicate File Finder” by Rashid Hoda and I admit I am still using an old version. It’s free and allows me to select which folders to search and what type of file to search for or if I want to find any type. All duplicate files are shown with their path and by default nothing is ticked for deletion. I can then select one by one which to delete. (I do find this useful) or just opt to delete all duplicates. I can also see a preview of the file I have currently selected. Really happy with it. My old version doesn’t work across a network but possibly the newer one does. As it’s free, give it a try.

    Viewing 8 reply threads

    Please follow the -Lounge Rules- no personal attacks, no swearing, and politics/religion are relegated to the Rants forum.

    Reply To: A confused mass of images needs sorting out, any recommendations?

    You can use BBCodes to format your content.
    Your account can't use Advanced BBCodes, they will be stripped before saving.