Memory Allocation StringSpace and Arrays

Wilson WindowWare Tech Support

WinBatch WinBatch+Compiler WebBatch
Home | Tech Database | Tech BBS | White Papers | Purchase

Memory Allocation, StringSpace, and Arrays
Keywords:  Arrays Memory Allocation Multidimensional Arrays  string space stringspace=  maxvars=
Question:
I'm encountering the dreaded "out of stringspace memory". My question is, if I don't drop variable inside of a called .wbc and then return to the caller, am I using up memory space that I can't drop? The help on CALL states they share the same memory space, so I'm assuming that if I keep calling a .wbc and not dropping the variables, I'm eating up stringspace.
I tried using the .INI and registry edit to increase the string space to no avail under Windows 95. Why does the editing of the [stringspace] within the registry under W95 not work? I've also tried adding it to the www?.ini (forgot the specific name) but we still get the error at the EXACT same location! We cranked it up from 32k to 65k but it still bombed.
We are compiling all our .wbts into .wbc, so do we use the www-prod/[stringspace] key then at compile time for the space to increase when we run the .wbcs?
The funny thing is that at one point winbatch was giving us an "unable to allocate memory" at a line in the middle of a
'if (isdefined(var)) then drop(var)'
section! We've tracked down just about anything that could take LARGE amounts of memory, but it's not easy without some sort of FreeStringspace function.
With the memory allocation failure: out of memory for strings: Is there a way to avoid this problem on a large script, say by using a call function?
What sort of debugging do you suggest?
Answer:
The CALL function will not affect the string space.
If the Call'ed WBT files are COMPLETELY independent (as in no variables in common) AND you are NEVER NEVER going to compile the script, you can try the CallExt function. If you are going to compile them, use RunWait of the other scripts instead.
Or you can try the Drop function on unneeded large string variables.
In a pinch, you can allocate a large binary buffer and store strings into the binary buffer, which can store different strings at different offsets and use it as sort of an array.
More details on memory allocation debugging below:

3097 Memory Allocation Failure - Out of memory for variables
Older versions of Winbatch (prior to WB 99) can only support around 400 variables. (Winbatch 99 can handle up to almost 1000 variables).
Note that if you are using the Searcher extender, and if you are recursing through numerous directories and saving those as variable names (e.g., dir%num%, where num=980+ dirs), you can use up the last available variable and crash. In this scenario, you might be able to eliminate a lot of code with the new "Shell Operations" extender. It's in the download area.
All variables are one big happy family. Be aware that EACH faked array element (array%xx%) counts as one variable.
A variable allocated in a WBC file can be dropped at the end of the wbc file or anywhere. It's a good idea to use the Drop command to drop variables whenever possible, to conserve memory usage. However that is usually not the problem. The problem usually is that one or two variables are eating up ALL the space. These must be watched carefully.
The following bit of code will also create the Memory Allocation Failure error:
        num=0

        :loop
        num=num+1
        a%num%=num

        goto loop  
One technique - other than re-writing code, if you have the WinBatch script launch another copy of itself (and then the orignal exit). The new copy starts off with a clean slate.

3096 Memory Allocation Failure - Out of memory for strings

Winbatch 98F has unlimited string space (only limited by your memory), so upgrade to the latest version.

In older versions, WinBatch has a limited amount of space to store string variables. WinBatch uses a 64K (recently raised (in WB97 through WB98F) from 32K) memory buffer to hold the string variables. The error occurs when WinBatch is unable to allocate a sufficiently large chunk of data needed to store a string. Getting rid (dropping) of large strings helps. However "memory fragmentation" does occur and may cause the error.
As you use strings, the string storage area becomes "fragmented" (holes or wasted spaces start appearing). This can impact the ability to store large strings.

In addition, certain operations like ItemSort need three copies of the string to work with. So if you have maybe a 10K string (500 items of 20 characters each) and do an ItemSort you've just used 30K of string space. If a little fragmentation occurred previously, it will die with the error you got.
There are various workarounds available:

Do horrible sorting operations first, before string space becomes fragmented.

(High-Tech one here) If you need to hold LARGE variables or entire files, consider using the BinaryBuffers. The Binary commands are able to utilize the string space more efficiently. BinaryBuffers hold large amounts of data. Allocate a LARGE binary buffer at the beginning of your program. When you need to de-fragment memory, BinaryPokeStr all needed variables into reserved spots in BinaryBuffer, drop all variables except binarybuffer handle, then reload variables with BinaryPeekStr.

Why do you need a sorted list? If list is a result of FileItemize and you don't need it sorted, and if you are running 32 bit WinBatch, try the Searcher extender.
So give up trying to FileItemize a massive directory and get the "SEARCHER" extender from our download area and modify your code to use the SEARCHER extender instead.
The problem is that the directory list returned by FileItemize is stored in a WinBatch varaible and it exceeds WinBatch's capabilities in that case.
Searcher is a different way of doing the same function, but only holding one file name at a time in memory.

With work, a BinarySort can be used to sort instead of ItemSort.

Update to the 97 series (free from 95/96 series) - More string space is available by default.

If you are using versions prior to WinBatch 97, you can double the amount of string space available, from 32K to 64K. This rarely helps the problem, as a program that chews through the available memory often has no trouble chewing thru twice that.

The editing of the string space (discussed later in this article) in the registry usually works. There are many considerations though: Most programs that bomb at 32K also bomb at 64K. Usually some program bug or technique is eating up gobs of memory and merely doubling the available memory does not affect it. For this reason, the StringSpace patch is not highly publicized. It rarely really works.

The stringspace item in the registry works with 32 bit Winbatch only. 16 bit uses an INI file. If you don't get the keys exactly correct, it does not work. Interpreted and compiled winbatch scripts use different keys.

With the default 32k of stringspace, the largest reasonable variable is maybe 6K. With 64K stringspace, the largest reasonable variable is maybe 12K.

How to debug:

It's tricky to debug. Usually you KNOW what your fat variables are. You could write a subroutine that looks like...
:sizechk
Message("varname",strlen(varname))
Message("varname2",strlen(varname2))
etc etc etc
return
and then sprinkle in:
GOSUB SIZECHK
in the code till you spot the big one.
Another possible problem is fragmentation. The string space could get so broken up that you can't allocate any more space.
Question (tread continued):
You mention string space fragmentation. How do we clean it up? We allocate alot of variables (and dialogs), but we drop them as well.
How does one utilize the registry edits to increase stringspace? Do we have to do this on the machine we are going to compile .wbcs on, or do it to each machine we are going to run the .wbc on?
Answer:
It has to be done on each machine you run it on. Although here is some code that can do it automagically for you...
It checks the entries, and it does not like them, changes them and then re-runs the script.
The code is a tad messy as it is designed to work with both 16 and 32 bit interpreted and compiled versions...
And without further ado...
if WinMetrics(-2) == 0 ;16 bit version
val=IniReadpvt("WB16I","StringSpace",32000,"www-prod.ini")
else ;32 bit version
errormode(@OFF)
val=RegQueryValue(@REGMACHINE,"SOFTWARE\Wilson
WindowWare\Settings\WWW-PROD\WB32I[StringSpace]")
ErrorMode(@CANCEL)
endif
if val!=65500
if WinMetrics(-2)==0 ; 16
   IniWritePvt("WB16I","StringSpace",65500,"www-prod.ini")
   IniWritePvt("WBC16I","StringSpace",65500,"www-prod.ini")
else ; 32
   RegSetValue(@REGMACHINE,"SOFTWARE\Wilson WindowWare\Settings\WWW-PROD\WB32I[StringSpace]",65500)
   RegSetValue(@REGMACHINE,"SOFTWARE\Wilson WindowWare\Settings\WWW-PROD\WBC32I[StringSpace]",65500)
endif

moi=strlower(IntControl(1004,0,0,0,0))

if FileExtension(moi)=="exe"
   Run(moi,"")
else
   Run(WinExeName(""),moi)
endif
exit
endif
Question:
We're developing a Winbatch program that reads in a fairly long list of modems and puts them into a tab delimited list. The list is about 15 to 20K. This tab delimited list is them passed as an Itembox to dialog() which enables the user to select a COM port and a modem from the list.
All is fine so far. However, given that NT can support a large number of COM ports, the user may go back into this dialog box several times. Since dialog() overwrites the original variable passed by Itembox with the item(s) the user selected, the variable must be reloaded each time dialog() is called.
The combination of the small stringspace, large list and memory fragmentation causes the "Out of stringspace" error after about 12 calls to this routine.
I've done everything I can think of including bumping stringspace up to 64K, dropping variables, poking the original list of modems into a binary buffer, refreshing the Itembox "var" from the binary buffer right after the call to dialog(), etc. The only thing I haven't done so far is to break the list up into smaller lists and force the user to select from sublists. If anyone has any other ideas, I would be v-e-r-y greatful.
Answer:
You are running into fragmentation of WinBatch string space problems with your extremely large variables.
You already figured out one way to fix it - break up the list into smaller chunks.
Or once thru the script, you can have the script completely re-launch itself for the next one - or if the user goes back to the dialog box.

StringSpace and FileItemize
A common problem is a FileItemize of large network volumes. Our Searcher extender (available from the download area of our web site, under WinBatch Add-Ons) covers this fairly well. If you're running into out of stringspace errors on a FileItemize, then do a FileItemize of a*.*, then b*.* etc. and process the list in 26 chunks.
Or...
Use a DOS command to grab a list of files and then redirect the output to a file and then use the file commands to go through the file line by line.
RunWait("command.com", "/c dir /b > newfilename")
How to Increase StringSpace
In WinBatch version 5.1a, or newer versions of WinBatch:
To change the string space in Win3.1/3.11:
There is an option to set number of variables and amount of string space allocated, in the WWW-PROD.INI in C:\WINDOWS:
  [WBxxx]
  MaxVars = 400
  StringSpace = 32000
The defaults are shown above. The maximum allowable value for 'StringSpace' is 65535.
To change the string space in Windows 95 and NT:
A change can be made to the registry, under the WinBatch/WWW-PROD section, as in the following:
RegSetValue(@REGMACHINE, "SOFTWARE\Wilson WindowWare\Settings\WWW-PROD\WB32I[stringspace]", "65000")
a=RegQueryValue(@REGMACHINE, "SOFTWARE\Wilson WindowWare\Settings\WWW-PROD\WB32I[stringspace]")
message("",a)  
How to Parse Two-Dimensional Arrays:
WinBatch does not support arrays. Arrays are not currently a feature of Winbatch because of the limitations of the parser we're using. Implementing arrays is on the list, but for obscure reasons are technically difficult to implement.
WinBatch cheats when it does arrays and whether or not Winbatch can accomplish what you need in this capacity depends on what you want to do. There are no functions per se to parse two-dimensional arrays in WinBatch.
You need to use variable substitution, in combination with FileRead, FileItemize, ItemExtract, ParseData, etc. functions. ItemExtract will allow you to extract an item from a string in a list. ParseData breaks a string constant or string variable into new sub-string variables, with a maximum of nine parameters.
Winbatch can also load small (under 4K) files into a single variable. We can load HUGE files into Binary Buffers and then extract parts of them piecemeal. We can read a text file a line at a time and process each line as it goes by.
You could also do something like:
        ;Parse ascii delimited strings

        ;Comma delimited
        xstring="joe,bob,fred,julie,jennifer,jan"

        count=itemcount(xstring,",")
        
        For x=1 to count
           field%x%=Itemextract(x,xstring,",")
           display(2,"field %x%",field%x%)
        Next

        ;Tab delimited
        xstring=strcat("joe",@TAB,"bob",@TAB,"fred",@TAB,"julie",@TAB,"jennifer",@TAB,"jan")

        count=itemcount(xstring,@TAB)
        
        For x=1 to count
           field%x%=Itemextract(x,xstring,@TAB)
           display(2,"field %x%",field%x%)
        Next
Article ID:   W13435
Filename:   Memory Allocation StringSpace and Arrays.txt