Slight problem with Robocopy

Topic

New Reply

I thought I would use Robocopy to back up my new PC to my old one (I do regular images and use File History as well). This is to learn about what options there are and also to try and write a Powershell script as well.

But I have a slight problem and I have just run the command on a CMD window to see what’s going on. Here’s the command

Code:

robocopy E:MyNameDocuments \ENTERPRISEDocumentsRobocopy /V /TEE /S /E   /PURGE /MIR /NP /XJD /R:3 /W:1 /log:E:MyNameDocumentslogs2016-01-04_Documents.txt /XD \ENTERPRISEDocumentsRobocopy E:MyNameDocumentsDownloads 

It all works fine but when I look at the old PC there’s no Robocopy folder but there is an Documents folder in the documents folder (but it doesn’t have the normal folder icon). If I try and create a new Robocopy folder it wants to merge it with the new document folder. I put the robocopy folder as one of the parameters for the XD option (that works as the downloads folder is quite large so I removed it from being backed up for quickness).

I’ve just noticed that when I selected the address in file manager, it came up with \ENTERPRISEDocumentsRobocopy – but in the list it’s effectively \ENTERPRISEDocumentsDocuments

I’ve Googled the problem (it’s why I excluded the Robocopy folder thinking it was being deleted first) but without any success.

Note that the data is being backed up OK – it’s just the name which is slightly confusing.

I hope this is clear

Eliminate spare time: start programming PowerShell

Reply | Quote

Replies

Access,

Here’s the RC command I use for my Scheduled Documents folder backup every day at 18:30
[noparse]
Robocopy “G:BEKDocs” “\MYBOOKLIVECMSharedBEKDocs” /MIR /XJD /R:5 /W:15 /MT:32 /V /NP /LOG+:G:BEKDocsBatchBackupTasksRoboCopyNAS.log
[/noparse]

Works flawlessly.

HTH :cheers:

May the Forces of good computing be with you!

RG

PowerShell & VBA Rule!
Computer Specs

Reply | Quote

Does the folder \ENTERPRISEDocumentsRobocopy exist? RC doesn’t create the target folder, only sub-folders.

cheers, Paul

Reply | Quote

Paul, yes, but it would seem that it’s changed to documents.

RG, thanks, I’ll give that a whirl sometime today.

Eliminate spare time: start programming PowerShell

Reply | Quote

RG, looked at your command and it’s essentially the same as mine. It would appear that copying from the documents folder on my new PC (which is a library) effectively renames the robocopy folder on my old PC. It appears to be a feature! As it works fine otherwise, I just have to remember that my new PC’s backup is in the folder \ENTERPRISEDocumentsDocuments.

That’s sorted my documents, now I have to think about my pictures! A trial with just a few as I have 85Gb worth!:wacko:

Eliminate spare time: start programming PowerShell

Reply | Quote

Could it be something to do with Microsoft’s fatal dalliance with the folder name “My Documents”, where nobody is ever sure whether it or “Documents” is the correct terminology?

BTW I have never observed that the target folder for Robocopy has to pre-exist before it is run.
In fact I have just done

Code:

cd /d d:
:: folder bat exists, folder newbat doesn't
robocopy bat newbat /copyall
:: now folder newbat exists, too

and happiness occurred!

BATcher

Plethora means a lot to me.

Reply | Quote

BTW I have never observed that the target folder for Robocopy has to pre-exist before it is run

MS may have changed it, but it was an issue a long time ago.

cheers, Paul

Reply | Quote

What’s the theory behind the /XJD switch? Why is it valuable to exclude Junction Points?

Reply | Quote

What’s the theory behind the /XJD switch? Why is it valuable to exclude Junction Points?

There is (was?) a problem/bug/feature when, for example, you are copying user profiles, which contain junctions. The output folders can become an almost endless repeat of the same folder (AppData, if memory serves).

So if you don’t specify /XJ (which I use rather than /XJD and/or /XJF) one can get as output
X:UsersBATcherAppDataAppDataAppDataAppDataAppDataAppDataAppData…..

A real pig of a situation to retrieve.

Later…

Just found this in one of my documents from 2010:

Windows 7 infinite loop while using Robocopy
(or: the […]Application DataApplication DataApplication Data[…] problem)

Problem

I recently was migrating data from my laptop (Win7-x64) to a new all-in-one (Win7-x86), and discovered something odd. When Robo got to “Application Data” – which is just a redirection – it started copying the files, and then got stuck in a loop. Inside “Application Data” was another “Application Data” with all of the pertinent files and folders. Inside that, another “Application Data” with everything, on and on until the file system couldn’t handle the depth level.

Answers
Monday, July 26, 2010 1:13 PM

Short answer: If you’re using Robocopy, simply using the /XJ switch will avoid this problem.

Long answer:
This recursive file structure is indeed caused by the implementation of junction points in the NTFS file structure by Microsoft. Essentially, some “folders” are not actual folders at all, but junction or reparse points, which essentially reroute a directory name to another directory. This is much like mapping a drive letter to a directory or another drive, and the Windows system is designed to handle this NTFS structure situation behind the scenes.

In Vista and 7, Microsoft decided to standardize some of the typical Windows settings for users, documents, etc., which had varied over time. To maintain backward compatibility, the old standards were also retained BUT WITH JUNCTIONS REPARSING THEM TO REDIRECT INQUIRIES INTO THE NEW ACTUAL DIRECTORIES. Most programs work right through these junctions without even noticing. For example, C:Documents and Settings points to C:Users. Even though the former directory does not actually exist, it APPEARS to have the files and directories under the latter, and a change in the contents of either directory will be reflected in the other, since they are in fact, a single directory.

The difficulty comes from certain Reparse Points which essentially redirect to a parent directory. This causes a recursive cascade for unwary programs or users who are unaware of the proper handling of the junctions (even though the junctions are designed to be transparent to software looking for either directory). The usual “subdirectory explosion” situation comes from the fact that the Documents and Settings$USER$Application Data folder (consistent with XP convention) is a junction point in Vista to Users$USER$AppDataRoaming, which in turn can contain the Application Data junction as a subdirectory…

Under normal Windows permissions, the junctions and their directories are inaccessible and hidden. If permissions are changed or programs working outside Windows are not prepared to encounter these junctions, duplication and recursion is destined to occur. For those using powerful low-level command interfaces such as Repair Console, it is best to check the capabilities and switches on the commands by using the /? help switch (for example, Robocopy /?, xcopy /?, rd /?, etc.). If you are not sure what you are doing, be sure to backup ahead of time with a reliable Windows program, with proper use of Restore Points.

Ronnie Vernon MVP

BATcher

Plethora means a lot to me.

Reply | Quote

I had an idea that Junction Points could be problematic as pointers, but I was unable to materialize the concern beyond that. Your post clarified the matter very well.

My use of Robocopy is perhaps simpler. I use the /XO switch rather than /MIR. It works reasonably well but I do have to be careful when I add more structure to my data folders. My use case is replicating data between 2 PC’s, one a laptop. My habit is to use one system or the other but rarely both at the same time.

Reply | Quote

I’ve got round the renaming problem by not copying my files into the library item ‘Document’. I’ve created a new folder on the target PC at root level. I then had to share it so it could be seen by the newer PC and just specified the the source and destination as E:MyNamePictures and \ENTERPRISERobocopy respectively. This explains why I have to have a pre-existing folder as it’s to a PC not a another disk.

I now have on Enterprise a folder called F:Robocopy, which in the address bar of File Manager is called Pictures. This means it won’t get mixed up with the original Pictures library. Similarly with Documents which goes to a folder called Documents2.

I understand it even if you don’t. This comes of using a PC as a source for Robocopy rather than an external disk or NAS.

In all, a very instructive procedure (well to me anyway).

I was going to try and use PowerShell to run teh copy but I think I’ll run it in a batch file.

Eliminate spare time: start programming PowerShell

Reply | Quote

it’s just the name which is slightly confusing

I strongly suspect that the problem is due to a (possibly hidden) DESKTOP.INI file in the source Documents folder that’s being copied to the target. Delete the file in the target and the folder names should align.

Reply | Quote

I used RG’s command line edited as follows:

Robocopy “V:UsersbbearrenDocuments” “A:OneDriveDocuments” /MIR /XJD /R:1 /W:1 /MT:64 /V /NP /LOG+:V:UsersbbearrenDesktopRoboCopy.log

and saved it as a .cmd file (to use in Task Scheduler later). The target is my OneDrive folder located on a separate HDD on this PC. Once the command has run, OneDrive kicks in on its own to update my online OneDrive folder, which then triggers an update of the OneDrive folder on my NAS, which is located on my RAID 10 Array. I have Office 365 which adds 1TB to my OneDrive capacity.

At the moment, OneDrive is uploading changes from the local OneDrive folder on this PC to the online OneDrive folder. I RD’d into my NAS to find that its OneDrive is busily downloading changes to the OneDrive folder. Hopefully they will all sync correctly. Once they finish I’ll check that end of it.

I’m still feeling my way around with OneDrive, and I’m not committed to saving any files directly to it, instead copying files from my habitual file structure to the OneDrive folder. This Robocopy command file makes that simpler and easier than drag-and-drop. If OneDrive sync’s as I think it should, I’ll setup a scheduled task to run the command in the wee hours of the morning. My Outlook.pst files are in a folder in my Documents folder, so they can’t be copied when Outlook is open. Hence the use of task scheduler when I’m signed off the PC and those files won’t be in use.

Always create a fresh drive image before making system changes/Windows updates; you may need to start over!

We all have our own reasons for doing the things that we do with our systems; we don't need anyone's approval, and we don't all have to do the same things.

We were all once "Average Users".

Computer Specs

Reply | Quote

My Outlook.pst files are in a folder in my Documents folder, so they can’t be copied when Outlook is open.

Well, yes and no.
If you were to use a program which supports Volume Shadow Services (VSS) (example: SyncBack) then you could copy open files.

BATcher

Plethora means a lot to me.

Reply | Quote

Well, yes and no.
If you were to use a program which supports Volume Shadow Services (VSS) (example: SyncBack) then you could copy open files.

My PC and my NAS stay powered 24/7, but I sign off each when not in use. I use Task Scheduler for all of my routine maintenance on both, and this is just one more piece of routine maintenance. Robocopy won’t have a problem with the .pst files when it runs via Task Scheduler, because the files won’t be in use, so I don’t really see a need for another program.

The duplication of my data to other locations is just another part of my backup regimen, along with drive images. RG’s command line was an improvement on my data duplication, so I’m incorporating it into my routine.

Always create a fresh drive image before making system changes/Windows updates; you may need to start over!

We all have our own reasons for doing the things that we do with our systems; we don't need anyone's approval, and we don't all have to do the same things.

We were all once "Average Users".

Computer Specs

Reply | Quote

I have the Robocopy command line setup (and tested) in Task Scheduler to run at 3:00 AM. The Task Scheduler Action tab looks like this:

43289-Robocopy-Task

The arguments are

/c Robocopy “V:UsersbbearrenDocuments” “A:OneDriveDocuments” /MIR /XJD /R:1 /W:1 /MT:64 /V /NP /LOG+:V:UsersbbearrenDesktopRoboCopy.log

The /c switch is used to close the Command Console when the task is completed.

Always create a fresh drive image before making system changes/Windows updates; you may need to start over!

We all have our own reasons for doing the things that we do with our systems; we don't need anyone's approval, and we don't all have to do the same things.

We were all once "Average Users".

Computer Specs

Reply | Quote