News, tips, advice, support for Windows, Office, PCs & more. Tech help. No bull. We're community supported by donations from our Plus Members, and proud of it
Home icon Home icon Home icon Email icon RSS icon
  • Copying multiple files but want to avoid duplicates

    Posted on Robert Ball Comment on the AskWoody Lounge

    Home Forums AskWoody support Productivity software by function Copying multiple files but want to avoid duplicates

    This topic contains 12 replies, has 7 voices, and was last updated by  Paul T 2 years, 9 months ago.

    • Author
      Posts
    • #507728 Reply

      Robert Ball
      AskWoody Plus

      I have an old music server (about 1 TB) with FLAC files. The company (Olive) is now out of business. I would like to copy all the files which I can access over my LAN to my NAS . However, errors can cause the transfer (copy) to crash or interrupt. Next copy attempt ends up with additional copies.

      Any suggestions for copying files to my NAS without making duplicates? Have tried searching for solutions (Google, etc) without finding good alternative. Inexpensive alternative acceptable, don’t need freeware although nice.
      Thanks

    • #1586423 Reply

      WSRolandJS
      AskWoody Plus

      I’ve used TeraCopy, in More mode, Overwrite All, has served me well. Or, you can tell it to skip any file with same name; although for the copying you want – I recommend Overwrite All and More mode.

      "Take care of thy backups and thy restores shall take care of thee." Ben Franklin, revisted

      • #1586430 Reply

        Robert Ball
        AskWoody Plus

        I’ve used TeraCopy, in More mode, Overwrite All, has served me well. Or, you can tell it to skip any file with same name; although for the copying you want – I recommend Overwrite All and More mode.

        How do I enter the whole source folder which is on a different disc into Teracopy? I can search for and find the target folder on the NAS to copy to.
        The instruction are fairly sketchy.
        Thanks

    • #1586473 Reply

      access-mdb
      AskWoody MVP

      I use Robocopy (as do others on the Lounge). I use the following code to copy files to a backup destination. It won’t copy files that haven’t changed. The /R and /W parameters will allow retries and waits. See this website for all the parameters.

      Code:
      robocopy E:source \nasfolder /V /TEE /MIR /NP /XJD /R:3 /W:1 /log+:E:logsPictures.txt /XD E:folderssubfolder

      Options:
      /V – Produce Verbose output, showing skipped files.
      /TEE – Output to console window, as well as the log file.
      /E – Copy Subdirectories, including empty ones.
      /MIR – MIRror a directory tree (equivalent to /E plus /PURGE).
      /XD – eXclude Directories matching given names/paths.

      • #1586474 Reply

        Robert Ball
        AskWoody Plus

        I was able to get to get Teracopy to work perfectly after setting as default copy and rebooting computer. I will look into Robocopy as well. It is great to have this as a resource of ideas, so many out there if you know where to look.
        Thanks.

        I use Robocopy (as do others on the Lounge). I use the following code to copy files to a backup destination. It won’t copy files that haven’t changed. The /R and /W parameters will allow retries and waits. See this website for all the parameters.

        Code:
        robocopy E:source \nasfolder /V /TEE /MIR /NP /XJD /R:3 /W:1 /log+:E:logsPictures.txt /XD E:folderssubfolder

        Options:
        /V – Produce Verbose output, showing skipped files.
        /TEE – Output to console window, as well as the log file.
        /E – Copy Subdirectories, including empty ones.
        /MIR – MIRror a directory tree (equivalent to /E plus /PURGE).
        /XD – eXclude Directories matching given names/paths.

      • #1586514 Reply

        bharder
        AskWoody Lounger

        The RoboCopy suggestion is a good one. I’d make one small alteration though, the suggested /XJD switch only excludes Junction Directories. This is a good idea (to prevent recursive copies) but incomplete.

        There is such a thing as a Junction File as well, and the switch to exclude those is /XJF. However there’s a better option yet again. Just use /XJ and you exclude both Junction Files and Junction Directories.

    • #1586484 Reply

      WSRolandJS
      AskWoody Plus

      “Can enter the target file but can’t get it to recognize the source folder on another disc”
      I don’t know the answer to that; maybe RoboCopy might be better in this case.

      "Take care of thy backups and thy restores shall take care of thee." Ben Franklin, revisted

    • #1586491 Reply

      Paul T
      AskWoody MVP

      +1 for Robocopy, especially the Wait and Retry options.

      cheers, Paul

    • #1586548 Reply

      access-mdb
      AskWoody MVP

      Thanks Mr Harder. While I was looking up the parameters (to let the OP know what they meant), I realised my original list had some which were contradictory or redundant, so that helped me clean the list up a bit. There are just so many!

      • #1586591 Reply

        bharder
        AskWoody Lounger

        RoboCopy is one of those rare programs with almost too much power, too many options!

        🙂

        There’s another neat switch that I like a lot, it’s /XO. This is wonderful when you expect the destination to already have a lot of your data, but you aren’t completely sure about the versioning of all your files. /XO will decline to copy an older file from the source location, over a newer file in the destination.

    • #1587570 Reply

      WSdogberry
      AskWoody Lounger

      I’m a sucker for GUI and transparency and being able to see what’s there and what’s going on, complete with filters and options, and I’m willing to pay for it. At the command line, I’m flying blind. I suggest you give GoodSync a try.

    • #1587625 Reply

      macropod
      AskWoody_MVP

      Of course, you could just use Windows Explorer and, when it eventually tells you it’s found duplicates, you could simply tell it to skip them all.

      Or you could use Windows’ own xcopy utility…

      Neither of these entails installing any 3rd-party software.

      Cheers,
      Paul Edstein
      [Fmr MS MVP - Word]

    • #1587628 Reply

      Paul T
      AskWoody MVP

      Neither does using Robocopy – xcopy on steroids.

      cheers, Paul

    • #1587672 Reply

      Berton
      AskWoody_MVP

      Something one could try before copying is to use CCleaner’s duplicate file finder to clean the source Folder. I usually create a new Folder or subFolder on an External USB drive or a NAS drive to copy the source files to then sort them out later, tends to be a lot quicker than sorting while copying since the copied files are then on the same drive as the destination Folder.

      Before you wonder "Am I doing things right," ask "Am I doing the right things?"
    • #1587675 Reply

      WSdogberry
      AskWoody Lounger

      I suggested you give GoodSync a try. I didn’t say ‘buy’.

      The ‘trial’ appears to be next to nothing until you realize that you can use it to monitor the state of your source and destination files and folders, complete with date and time and the direction you may want transfers to take place. Just perform the scan. Once you have examined it, you can go back to your command line utilities and perform the operations themselves. There is no time limitation on the trial itself, so you can do this as many times as you like.

      You can also tell Microsoft that Windows was a huge mistake and a terrible investment, because doing everything at the command line is so much better.

    • #1587679 Reply

      Berton
      AskWoody_MVP

      One caveat with the latest Win10 Insider Preview, the Command Prompt and Command Prompt (Admin) on the right-click of the Start button have been replaced with the equivalent Windows PowerShell and Windows PowerShell (Admin). [Omen of the future?]

      Before you wonder "Am I doing things right," ask "Am I doing the right things?"
      • #1587685 Reply

        WSjwitalka
        AskWoody Lounger

        One caveat with the latest Win10 Insider Preview, the Command Prompt and Command Prompt (Admin) on the right-click of the Start button have been replaced with the equivalent Windows PowerShell and Windows PowerShell (Admin). [Omen of the future?]

        Note that you can get the Command Prompt back on the right click menu via
        Settings > Personalization > Taskbar > Turn off “Replace Command Prompt with Windows Powershell in the Menu when I right Click the Start Button or press Windows Key + X”

        Jerry

    • #1587733 Reply

      WSmrgregfox
      AskWoody Lounger

      I’ve been using FreeFileSync for several years to synchronize similar but not identical directories. It has an easy to use interface and lets you sych based on filename and date/time, or full content of files (slower that way) and doesn’t replace those that match. There is a checkbox during the install to avoid including an added junkware program.

    • #1593226 Reply

      Erik_S47
      AskWoody Plus

      I have an old music server (about 1 TB) with FLAC files. The company (Olive) is now out of business. I would like to copy all the files which I can access over my LAN to my NAS . However, errors can cause the transfer (copy) to crash or interrupt. Next copy attempt ends up with additional copies.

      Any suggestions for copying files to my NAS without making duplicates? Have tried searching for solutions (Google, etc) without finding good alternative. Inexpensive alternative acceptable, don’t need freeware although nice.
      Thanks

      Open a command window and type

      XCOPY /?

      To see the options. I’ve never had any trouble with XCOPY.

    • #1593249 Reply

      Paul T
      AskWoody MVP

      Robocopy is better than xcopy. 🙂

      cheers, Paul

    Please follow the -Lounge Rules- no personal attacks, no swearing, and politics/religion are relegated to the Rants forum.

    Reply To: Copying multiple files but want to avoid duplicates

    You can use BBCodes to format your content.
    Your account can't use Advanced BBCodes, they will be stripped before saving.