Compiler 1126 BinaryAlloc Error

 KEYWORDS: WBC Compiler WinBatch+Compiler Limit Limitations Other Files OtherFiles Attachments Extenders Size

Question:

I need to have the files compiled into the executable. My largest "other file" is 500 MB. Is it possible to get this done? the largest one i've had success with so far is about 300 MB.

Answer:

The largest binary buffer that can be allocated, and therefore the Compiler can support would be about 800MB. Being able to get that much depends on how much physical RAM you have and your paging file size. On my system, 512MB RAM and pagefile size of 2299, I can only allocate about 403MB.

In the case of WinBatch+Compiler, inorder for the files to be attached to a EXE file they are being read, in their entirety, into a binary buffer before they can be written to the EXE file. Since there are limits to how large of a binary buffer you can have, even if you have a *LOT* of physical memory and a large paging file, it is to be expected that you can't just attach an arbitrarily large file to the EXE in this manner.

Assuming you are running your code in the Win32 Subsystem on the Windows NT Kernel [e.g. WinNT, Win2K, WinXP, Win2K3, Vista], and are using a 32-bit x86 CPU, the limit is 4GB of addressable virtual memory, of which 2GB, or, alternatively, 3GB, can be addressed within process-private virtual memory. Now, within that range, you have the program's main EXE file and *EVERY* DLL that it has loaded, all consuming some of that virtual memory address space. On top of that, you have other heap-based memory allocations that the EXE and DLLs may make as their code is executed, and all that heap memory occupies portions of the process-private virtual address space. Then, all of the pages of virtual memory that the process has mapped have to either reside in physical memory, or have to be paged out to the paging file. The memory manager in the kernel takes care of these paging activities automatically.

Given those assumptions, the single largest contiguous block of virtual address space that can be allocated will be limited, in practice to approximately 1.2GB to 1.5GB in size, and that's provided that you have enough page file space and enough physical memory. When you count all of the other "overhead" that fragments your virtual memory address space, that upper limit can drop down significantly lower, which is where you see most binary buffer allocations in WinBatch having a maximum size of approximately 800MB.

This type of limitation isn't unique to WinBatch and its binary buffers. This limitation applies to *ANY* program that attempts to make such huge memory allocation requests. Any C/C++ usage of the "new" operator or the malloc() family of RTL functions is limited in the same way. Applications built for x86-64 architectures still have the same types of limits, but the upper bounds of those limits are significantly higher.

If you need to have an arbitrary # of arbitrarily large files all lumped together into a single file for distribution purposes, you might find that putting them, along with the compiled script, into a ZIP file and then creating a EXE using the WinZIP Self-Extractor tool may alleviate the problems that you're encountering right now. Ultimiately, it's the underlying implementation technique [e.g. whole file in memory vs. streaming] that sets the primary limitation on the largest size file that you can process in this manner.

Article ID:   W17384

File Created: 2008:09:16:11:33:20

Last Updated: 2008:09:16:11:33:20

Database Search

TechHome