How to Synchronize Files between a Source and Destination Directory

Keywords:  synchronize files

FILESYNC.WBT: File synchronization

PURPOSE: to synchronize files between a provided source and destination (including all subdirectories but specified exclusions) using mod-dates.

Author's Note:

This uses Binary search and comparison routines. It's built to accommodate for large directory structures (long filenames, long directories, etc). I began using lists, however, I quickly discovered that lists can run out of memory when storing really long directory paths. Binary use doesn't run in to this problem as readily, and it is VERY fast!

This is built to go through the entire tree starting with InitSourceDir and 'mirror' it with some other destination (we use a personal share on the network). There is a companion *.wbc file (syncmess.wbc/wbt) which is the messaging component for the file-checking section. I did this simply because I wanted the user to be notified that something is indeed going on (had too many people running the program multiple times) and I didn't know of a simple way to put a message up and keep one up that didn't take at least a second out of each loop -- and if you're going through 1000+ files, that's a lot of wasted time! Instead, I 'spawn' off the second wbc file that keeps a message up and doesn't take any extra time. Also note that I usually distribute these as winbatch first then as exe's. This is why I have a button rename for both types.

This took me a little while in developing, plus I used this instance to learn how to use binary searching/indexing (now I use it in almost every case...it's sped up many of the other routines I've written). It has turned out to be extremely useful in our situation (laptop users w/ no local backup routines) and I hope parts or all of this code is useful to someone else.

Matt Brownell
PC Admin
Semiconductor Research Corp
brownell@src.org

goto Begin

;Created: 10/30/96 MJB
;Last Modified: 4/9/97 MJB

:Begin
;Debug(@ON)
ErrorMode(@OFF)
WinBatchProg = "WBT - C:\ADMIN\BAT\File"
WinBatchExe = "WBT - FILE"
ButtonName = "FileSync"
InitSourceDir = "C:\DATA"       ;the initial source directory to start comparison
InitDestDir = "M:\DATA"         ;the initial destination directory to compare to
SrcDrive = StrSub(InitSourceDir,1,2) ;pull the drive only from the InitSourceDir var
DestDrive = StrSub(InitDestDir,1,2)  ;pull the drive only from the InitDestDir var
WinBatch = "c:\admin\winbatch\system\winbatch.exe"
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Messaging = "C:\ADMIN\BAT\SYNCMESS.WBC" ;set the messaging program to call
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
TotalFilesTly = 0 ;var used to tally total files checked regardless if copied
CopiedFilesTly = 0 ;var used to tally the number of files to copy

:ChangeButtonName
;change the name of the wbt program button to be more legible
If WinExist(WinBatchProg) then WinTitle(WinBatchProg,ButtonName)
If WinExist(WinBatchExe) then WinTitle(WinBatchExe,ButtonName)

:NetCheck
;if the dest drive doesn't exist, error w/ message
If !DiskExist(DestDrive)
	Message("No Network Drives!","You do not have the necessary network drive attached%@CRLF%Please connect to network services before synchronizing.")
	Goto Cancel
endIf
;check for Modem connection and warn
If WinExist("Connected") || WinExist("SRC-PPP")
	Pause("Warning: Slow Connection","You are attempting to synchronize over a slow network connection.%@CRLF%This will take much longer (potentially multiple hours) to complete.%@CRLF%%@CRLF%Continue?")
endIf

;;GenerateDirTree Desc: GenDirTree's purpose is to produce a complete list (CompleteDirList)
;;;of all directories(w/ full path-names) in the source tree(InitSourceDir and down).  It does
;;;this by first generating a list of all dirs in the sourcedir.  (Note I remove the Template dir
;;;from this list right away so as to not create unnecessary work).  With the sourcedir directory
;;;list, it starts into the CompleteDirLoop that uses the total number of entries in CompDirList
;;;as its stopping point (CompleteDirCount).  The first step in the for-loop is to extract an
;;;entry (which is one of the subdir's found in the InitSourceDir).  With this CurrentDir
;;;it first makes sure it's not the topmost level (so as not to get double \\'s) and then gets
;;;a list of directories found in it (all of its subdirs).  With this second list, the second
;;;for-loop then cycles through each entry (each of the sub-subdirs), generates it's full path-name,
;;;as referenced from the InitSourceDir level, and Inserts these into the CompleteDirList (the
;;;master list) -- appending to the end.  Once the sub-subdir list has been entered, I do
;;;another count of the CompleteDirList, which now contains all the sub-dirs of the first directory
;;;and finish the CompleteDirLoop.  This means that CompleteDirCount (the stopping point for
;;;the master list) will keep getting larger and the loop will continue to cycle through the
;;;constantly-growing CompleteDirList until all subdirs are entered

:GenerateDirTree
CompleteDirList=DirItemize("%InitSourceDir%\*.*")       ;get a list of InitSourceDir first to populate CompDirList
;Remove the system-known dirs from the search so system templates don't 'clog' the update proc
TempIndex = ItemLocate("template",StrLower(CompleteDirList),@TAB)
CompleteDirList = ItemRemove(TempIndex,CompleteDirList,@TAB)
TempIndex = ItemLocate("ontime",StrLower(CompleteDirList),@TAB)
CompleteDirList = ItemRemove(TempIndex,CompleteDirList,@TAB)
TempIndex = ItemLocate("backup",StrLower(CompleteDirList),@TAB)
CompleteDirList = ItemRemove(TempIndex,CompleteDirList,@TAB)
;
CompleteDirCount = ItemCount(CompleteDirList,@TAB)	;get count of items in CompList to start For-loop
;Flesh out dirlist with entire subtree
For CompleteDirLoop = 1 to CompleteDirCount	;CompDirCount will change (see further down) to accommodate for subdirectories
	CurrentDir = ItemExtract(CompleteDirLoop,CompleteDirList,@TAB)	; the current dir. is pulled from the CompDir List
	If CurrentDir <> "" then CurrentDir = StrCat(CurrentDir,"\")	;accommodates for when at top-most level (InitSourceDir)
	DirList = DirItemize("%InitSourceDir%\%CurrentDir%*.*")	  ;get list of directories in CurrentDir
	DirCount = ItemCount(DirList,@TAB)	;count the number of entries for for-loop
	For DirLoop = 1 to DirCount	
		CurrentListItem = ItemExtract(DirLoop,DirList,@TAB)	;pull the first subdir from the CurrentDir
		If CurrentDir <> "" then FullDirName = StrCat(CurrentDir,CurrentListItem)	;if the currentDir is not "" (meaning at the InitSourceDir level) concat the CurrentDir to the current list item to complete the full pathname
				    else FullDirName = CurrentListItem
		;include the full-path-name of the current subdir into the master CompleteDirList
		CompleteDirList = ItemInsert(FullDirName,-1,CompleteDirList,@TAB)
	endfor
	CompleteDirCount = ItemCount(CompleteDirList,@TAB)	;get a new count of the master CompleteDirList so the master For-loop will keep going until there are no more items in CompleteDirList (therefore no more subdirs)
endfor

;;DirectoryLoop Desc: DirectoryLoop's purpose is to go through each dir-entry in CompleteDirList,
;;;as generated by GenDirTree section, and get a list of the files.  With that list, it will
;;;go through a CheckFiles sub-routine to compare the file-states 
:DirectoryLoop
BCpyFileLst = BinaryAlloc(50000)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
RunShell(Winbatch,"%Messaging% 1","C:\",@NORMAL,@NOWAIT)     ; this is the secondary WBT file
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
For DirLoop = 0 to CompleteDirCount     ;loop through the entire CompleteDirList
        CurrentListItem=ItemExtract(DirLoop,CompleteDirList,@TAB)       ;pull out the full-path-name of a directory to compare files from
        If DirLoop == 0 ;DirLoop=0 means it's at the topmost level (InitSourceDir)
		CurrentSourceDir=InitSourceDir
		CurrentDestDir=InitDestDir
	else
		;The sourceDir will be InitSourceDir\currentdir pulled from above
		CurrentSourceDir=StrCat(InitSourceDir,"\",CurrentListItem)
		;same with Destdir except InitDestDir
		CurrentDestDir=StrCat(InitDestDir,"\",CurrentListItem)
	endif
	Gosub CheckFiles	;go to the check-files routine to check all files in current source/dest directories
endfor
BinaryWrite(BCpyFileLst,"c:\admin\filesync.txt") ;check to test mem-req's for binary file list, may not need 50000bytes
If TotalCopiedFilesTly == 0 then GoTo End
ContinueCopy = AskYesNo("Synchronize?","%TotalCopiedFilesTly% out of %TotalFilesTly% files need to be%@CRLF%copied.  Do you wish to continue?")
If ContinueCopy == @NO then GoTo Cancel
GoSub CopyFiles
GoTo End

:End
BinaryFree(BCpyFileLst)
If TotalCopiedFilesTly == 0 then Message("Files Synchronized!","Your files are already synchronized.%@CRLF%There is no need to copy any files.")
                                else Message("Files Synchronized!","Your files have been succesfully synchronized.")
Exit

:Cancel
Message("Synchronization Cancelled!","You have cancelled the synchronization routine.  Please%@CRLF%run again if you wish to synchronize your files with the network.")
Exit

;**** GoSub Subroutines ****

;;1) Check for dir existence on m: drive.  If not, make it
;;2) changedir to current dest and current src -- taking advantage of relative drive checking
;;   by c:filename vs. c:\data\filename.  Important because nested dirs can get way too big
;;   for Winbatch to be able to put on a command line (256chr limit).  This way, only limited
;;   by filename length
;;3) fileItemize src dir for processing
;;4) get count of files for Forloop
;;5) Loop through all files in current dir:
;;      - get the location in the list of it's directory (for use in retrieval);
;;	  locking it to a known 4 positions (assuming no one will have more than
;;	  9999 directories!)
;;	- based on existence and filetimecode, determine if src or dest file is newer
;;	- put the info into binary for fast processing (and eliminate memory probs) in
;;	  the form: ,<#### = dirlist locator>,
;;6) get count of total files to be copied (based on number of @CRLF's in binary list)
;;7) get count of total number of files checked (good way to 'scare' the user into cleanup ;-)

:CheckFiles
;Debug(@ON)
;generate a list of all files in current directory as passed from directory loop
If !DirExist(CurrentDestDir) then DirMake(CurrentDestDir)
If LastError() == 1028
        Message("Little Bug","%CurrentDestDir% does not exist.  For some unknown reason,%@CRLF%this program cannot create it.  Please manually create%@CRLF%this directory and re-synchronize.")
        GoTo Cancel
endIf
DirChange(CurrentDestDir)
DirChange(CurrentSourceDir)
FileList = FileItemize("*.*") ;get list of files in currentdir -- which is CurrentSourceDir from above stmt
FileCount = ItemCount(FileList,@TAB)    ;get number of list entries in FileList for while-loop
For CurrentFile = 1 to FileCount       ;while there are still entries, keep looping
        ;pull out a file from the FileList
        CurrentListItem=ItemExtract(CurrentFile,FileList,@TAB)
        DirListLocation = StrFixLeft(DirLoop,"",4)
        If !FileExist("%DestDrive%%CurrentListItem%")
                CopyFileInfo = "%SrcDrive%,%DirListLocation%,%CurrentListItem%"                          
        else 
                If FileTimeCode("%SrcDrive%%CurrentListItem%") > FileTimeCode("%DestDrive%%CurrentListItem%")
                        CopyFileInfo = "%SrcDrive%,%DirListLocation%,%CurrentListItem%"
                else 
                        If FileTimeCode("%SrcDrive%%CurrentListItem%") == FileTimeCode("%DestDrive%%CurrentListItem%")
                                Continue
                        else 
                                CopyFileInfo = "%DestDrive%,%DirListLocation%,%CurrentListItem%"
                        endIf
                endIf
        endIf
        BinaryPokeStr(BCpyFileLst,BinaryEODGet(BCpyFileLst),"%CopyFileInfo%%@CRLF%")
endFor
TotalCopiedFilesTly = BinaryStrCnt(BCpyFileLst,0,BinaryEODGet(BCpyFileLst)-1,@CRLF)
TotalFilesTly = TotalFilesTly + FileCount
Return

:CopyFiles
LastFile = BinaryEODGet(BCpyFileLst) ;get the binary offset of the end of the mem-space for use in for-loop
;For loop will loop through until gets to end of binary space.
;;1) get first two values and see if they equal either SrcDrive or DestDrive, if not, continue loop until they do
;;2) once BValue is either src or destdrive, the filedir list locater is read
;;3) next get the filename -- first need to determine at what offset it ends (know begin offset)
;;4) based on BValue, either copy file from c: to m: or vice-versa
;;5) increment BPosition to the offset of the Filename End -- cuts down time on binary
;;   retrieval since already know stuff in between is not correct
For BPosition = 0 to LastFile
;Debug(@ON)
        BValue = BinaryPeekStr(BCpyFileLst,BPosition,2)
        FileDirLocation = StrTrim(BinaryPeekStr(BCpyFileLst,BPosition + 3, 4))
        If FileDirLocation == "" then Continue
        FileDir = ItemExtract(FileDirLocation,CompleteDirList,@TAB)
        FileNameLocBegin = BPosition + 8
        FileNameLocEnd = BinaryIndex(BCpyFileLst,BPosition,@CRLF,@FWDSCAN)
        FileName = BinaryPeekStr(BCpyFileLst,FileNameLocBegin,FileNameLocEnd - FileNameLocBegin)
        If BValue == SrcDrive then CaseValue = 1
                                else If BValue == DestDrive then CaseValue = 2
                                                                else CaseValue = 3
        Switch CaseValue
                Case 1
                        DirChange("%InitSourceDir%\%FileDir%")
                        DirChange("%InitDestDir%\%FileDir%")
                        FromFile = "%SrcDrive%%FileName%"
                        ToFile = "%DestDrive%%FileName%"
                        FileCopy(FromFile,ToFile,@FALSE)
                        CopiedFilesTly = CopiedFilesTly + 1
                        WinTitle("","%CopiedFilesTly% of %TotalCopiedFilesTly% Sync'd")
                        Break
                Case 2
                        DirChange("%InitSourceDir%\%FileDir%")
                        DirChange("%InitDestDir%\%FileDir%")
                        FromFile = "%DestDrive%%FileName%"
                        ToFile = "%SrcDrive%%FileName%"
                        FileCopy(FromFile,ToFile,@FALSE)
                        CopiedFilesTly = CopiedFilesTly + 1
                        WinTitle("","%CopiedFilesTly% of %TotalCopiedFilesTly% Sync'd")
                        Break
                Case 3   ;default case
                        Break
        endSwitch
        BPosition = FileNameLocEnd + 1
endFor
Return

Here is the secondary WBT file, SYNCMESS.WBT, File synchronization processing message WBT:

goto Begin

;Created: 10/31/96 MJB
;Last Modified: 4/9/97 MJB

:Begin
;Debug(@ON)
MessageType = param1
WinBatchProg = "WBT - C:\ADMIN\BAT\SYNC"
WinBatchExe = "WBT - SYNC"
ButtonName = "Progress..."

:ChangeButtonName
;change the name of the wbt program button to be more legible
If WinExist(WinBatchProg) then WinTitle(WinBatchProg,ButtonName)
If WinExist(WinBatchExe) then WinTitle(WinBatchExe,ButtonName)

While MessageType == 1
        Display(2,"Synchronizing Files","Program is checking your files.  Please Wait.")
        If WinExist("Synchronize?") || WinExist("Files Sync") || WinExist("Little") || !WinExist("FileSync") then Break
endwhile
Break

:EndProcedure
Return

Article ID:   W13801

Filename:   Synchronize Files between Source and Destination Example 1.txt

File Created: 2001:03:06:11:49:24

Last Updated: 2001:03:06:11:49:24

Database Search

TechHome

How to Synchronize Files between a Source and Destination Directory

FILESYNC.WBT: File synchronization

Author's Note: