WinBatch Tech Support Home

Database Search

If you can't find the information using the categories below, post a question over in our WinBatch Tech Support Forum.

TechHome

Samples from Users
plus
plus
plus
plus
plus
plus
plus
plus
plus
plus
plus
plus
plus
plus
plus
plus

Can't find the information you are looking for here? Then leave a message over on our WinBatch Tech Support Forum.

Split Up Large Text File


Question:

I have "big" (approx. 400 MB) text file with more than 2800000 lines. I need split this file to smaller files with 64000 lines. I need to do this many times - I have database export and I need work with these data in Excel

Answer:

Try this:

;------------------------------------------------------------------------------------------------------------------------------------------
; Detlev Dalitz.20040331
;------------------------------------------------------------------------------------------------------------------------------------------
;   Wednesday, March 31, 2004 12:15 AM
;------------------------------------------------------------------------------------------------------------------------------------------

sFilename = IntControl(1004,0,0,0,0) ; We use this script as test input file.
;sFilename = "bigfile.txt"

iFilesize = FileSize(sFilename)
Terminate(!iFilesize,"Error","Filesize is zero.")

; --- Pass 1 ---

sSearch = @CRLF
iSearch = StrLen(sSearch)

iMaxSearch = 20
;iMaxSearch = 64000

iChunkSize  = iFilesize/100
iChunkCount = 1+(iFilesize/iChunkSize)

sListOffsetCut = ""
iOffsetFile = 0
iCountSearch = 0

hBB = BinaryAlloc(iChunkSize)

While iChunkCount
   iOffsetBB = 0
   iResult = BinaryReadEx(hBB,iOffsetBB,sFilename,iOffsetFile,iChunkSize)

   While (iOffsetBB < iChunkSize)
      iOffsetBB = BinaryIndexEx(hBB,iOffsetBB,sSearch,@FWDSCAN,@TRUE)
      If (iOffsetBB < 0) Then Break
      iOffsetBB = iOffsetBB + iSearch
      iCountSearch = iCountSearch + 1
      If !(iCountSearch mod iMaxSearch)
         iOffsetCut = iOffsetFile + iOffsetBB
         sListOffsetCut = ItemInsert(iOffsetCut,-1,sListOffsetCut,@TAB)
      EndIf
   EndWhile

   iChunkCount = iChunkCount - 1
   iOffsetFile = iOffsetFile + iChunkSize
EndWhile
If (iOffsetCut < iFileSize) Then sListOffsetCut = ItemInsert(iFileSize,-1,sListOffsetCut,@TAB)

BinaryFree(hBB)


; --- Pass 2 ---

iCount = ItemCount(sListOffsetCut,@TAB)

@P1 = "{1}"
sFileOutMask = StrCat(sFilename,".part.",iCount,".",@P1,".txt")

iCutBegin = 0
iCutEnd = 0
For ii=1 To iCount
   iCutEnd = ItemExtract(ii,sListOffsetCut,@TAB)
   iBBSize = iCutEnd-iCutBegin
   hBB = BinaryAlloc(iBBSize)
   iResult = BinaryReadEx(hBB,0,sFilename,iCutBegin,iBBSize)
   sFilenameOut = StrReplace(sFileOutMask,@P1,ii)
   BinaryWrite(hBB,sFilenameOut)
   BinaryFree(hBB)
   iCutBegin = iCutEnd
Next

Run("explorer.exe",StrCat("/select, ",sFileName,"*.txt"))
Exit
;------------------------------------------------------------------------------------------------------------------------------------------
;*EOF*

Article ID:   W16705
File Created: 2005:02:18:12:21:54
Last Updated: 2005:02:18:12:21:54