WinBatch Tech Support Home

Database Search

If you can't find the information using the categories below, post a question over in our WinBatch Tech Support Forum.

TechHome

Tutorials
plus
OLE TUTORIAL

Can't find the information you are looking for here? Then leave a message over on our WinBatch Tech Support Forum.

The WinInet extenders and Web Servers -- An Opus

Interactions between a user at a Web Browser and a Web Server at a Web Site tend to fall into two categories:

The WinInet extender appears to the Web Server to be just another Web Browser, and it allows interactions with a Web Server to be automated. Doing the equivalent of clicking on links is a trivial operation whileas emulating the user filling out forms can get to be a tricky business. We will attempt to provide some clues to how to figure out what to do in this process to automate these processes.

Grabbing HTML pages off a Web Server

In general, when a user clicks on a link, or types in a URL on the Browser address/location line, the Web Browser runs out, grabs a file (usually an HTML file) from a Web Server. Depending on the type of file the browser can perform various operations. For the standard HTML files, the Browser displays it, and often fetches any images contained within it. For other types of files, ZIP files for instance, the Browser may prompt the user to save the file to disk.

Assuming you know URL of the file you wish to grab, the WinInet extender makes it a (usually) painless process. Also assuming that you can access the Web Server directly and do not have to work via a proxy server the follow discussions hold true. If you have to use a proxy server or somehow negotiate via a corporate firewall, you will need to modify the iBegin statements in all the examples.

Here is the basic Web page grab example. You should look all the functions up in the WinInet help file and become very familiar with this chunk of code before proceeding. All the rest of this write-up assumes great familiarity with this particular chunk of code. Copy and paste it. Make it work. Download other web pages. You MUST be comfortable with this before proceeding. 


;This example downloads the http://www.winbatch.com/download.html page

;Load the WinInet extender
AddExtender("WWINT34I.DLL")

; The host name is the first part of the URL, minus the https://
host="www.winbatch.com"

;And the part of the URL we wish to GET is *everything* after the host name
geturl="winware/download.html"

;Define output file name
outfile=strcat(DirGet(),"result.html")

; Basic web page fetch script
tophandle=iBegin(0,"","")

connecthandle=iHostConnect(tophandle, host, @HTTP, "", "")
datahandle=iHttpInit(connecthandle,"GET",geturl,"", 0)
rslt = iHttpOpen(datahandle, "", 0, 0);

if rslt!=200
      headers=iHttpHeaders(datahandle)
      AskItemList("Rslt=%rslt%",headers,@tab,@unsorted,@single)
endif

xx=iReadData(datahandle,outfile)

iClose(datahandle)
iClose(connecthandle)
iClose(tophandle)

;We now have the returned web page.  Since it is just an
;HTML page, we'll just use the users Web Browser to display it.
ShellExecute(outfile,"","",@NORMAL,"OPEN")
exit

Web Forms

For starters, it is necessary to get a copy of the HTML that made the form. Simply use the Web Browser to go to the web page that has the form on it, then "View Source". Save the source to a file. Maybe even print it out. Scan through the HTML source looking for the <FORM ...> and </FORM> HTML tags. Almost all the information required will be in or between the FORM tags.

There are three basic types of forms, as detailed below. In addition to the information we need, there is often a ton of other irrelevant formatting information we can ignore.

Types of Web Forms

Type 1: "URL-Encoded GET"

A vaguely obsolete type, but still used for an incredible number of simple forms.

How to recognize

Once extraneous formatting information is stripped away, it looks a bit like... 


<FORM METHOD="get" ACTION="/webcgi/webbatch.exe?dumpinputdata">
<INPUT TYPE="checkbox" NAME="myitem"   VALUE="YooHoo">
<input type=submit value="Do it now!">
</FORM>

Type 2: "URL-Encoded POST"

This is the currently recommended form, used *almost* everyplace that the URL-Encoded GET is not.

How to recognise

Once extraneous formatting information is stripped away, it looks a bit like... 

         
<FORM METHOD="post" ACTION="/webcgi/webbatch.exe?dumpinputdata">
<INPUT TYPE="checkbox" NAME="myitem"   VALUE="YooHoo">
<input type=submit value="Do it now!">
</FORM>

Type 3: "multipart/form-data POST"

This type is very rare and is usually only used when an entire file is to be uploaded to a web server - optionally with other information.

How to recognise

Once extraneous formatting information is stripped away, it looks a bit like... 

         
<FORM enctype="multipart/form-data" method="post" action="/webcgi/webbatch.exe?dumpinputdata">
<input type=file name="upl-file" size=40>
<input type=text size=40 maxlength=40 name="desc">
<input type="radio" name="another" value="yes">
<input type="radio" name="another" value="no" checked>
<input type=submit value="Upload Now">
</FORM>

Note the enctype parameter in the FORM line and the special <input type=file ...> data tag.

The URL-Encoded GET, POST and the multipart forms require different script processing to sort out, even though there are many common elements. We'll be going over a script to communicate with the web servers, enter the form information and get back the response without actually using a Web Browser. I won't even mention what day traders can do with this.

URL-Encoded GET example.

For the first example we'll do something useful.

On the UPS (United Parcel Service) site is a small form which you can track your UPS packages with. Currently the link is....

http://www.ups.com/tracking/tracking.html

In viewing the page you can extract all the items between the <FORM... > and </FORM> tags.

There is a LOT of formatting information in there. Keeping the FORM tags and the <INPUT...> tags and deleting the rest leave this raw form.... 



<FORM ACTION="https://wwwapps.ups.com/etracking/tracking.cgi" METHOD="GET">

   <INPUT TYPE="SUBMIT" NAME="submit" VALUE="Track">

   <INPUT TYPE="RESET" VALUE="Clear">

   <INPUT TYPE="TEXT" NAME="InquiryNumber1" SIZE="24" MAXLENGTH="24">

   <INPUT TYPE="TEXT" NAME="InquiryNumber2" SIZE="24" MAXLENGTH="24">

   <INPUT TYPE="TEXT" NAME="InquiryNumber3" SIZE="24" MAXLENGTH="24">

   <INPUT TYPE="TEXT" NAME="InquiryNumber4" SIZE="24" MAXLENGTH="24">

   <INPUT TYPE="TEXT" NAME="InquiryNumber5" SIZE="24" MAXLENGTH="24">

   <INPUT TYPE="HIDDEN" NAME="TypeOfInquiryNumber" VALUE="T">

</FORM>

As mostly we are interested in only one package and don't intend to hit the "CLEAR" button we can further logically simplify the form to: 


<FORM ACTION="https://wwwapps.ups.com/etracking/tracking.cgi" METHOD="GET">

   <INPUT TYPE="SUBMIT" NAME="submit" VALUE="Track">

   <INPUT TYPE="TEXT" NAME="InquiryNumber1" SIZE="24" MAXLENGTH="24">

   <INPUT TYPE="HIDDEN" NAME="TypeOfInquiryNumber" VALUE="T">

</FORM>

It looks like...

So now to interpret the form and extract required information.

From the METHOD=GET in the form line we can see that it is an URL-Encoded GET type form. 


host="wwwapps.ups.com" ; Its right there in the FORM line ACTION parameter
mailurl="/etracking/tracking.cgi"   ; Also in the same place.

Then we can see there are three input controls. There is the submit button, the Inquiry number, and some kind of hidden (from a web browser user) variable.

The TYPE="SUBMIT" button control is called "submit" (from NAME="submit") and when clicked it is set to a value of "Track" (from VALUE="Track")

The TYPE="TEXT" edit box control is called "InquiryNumber1" and it will hold the value the user types in.

The TYPE="HIDDEN" control is called "TypeOfInquiryNumber" and it holds a value of "T".

This information needs to be returned to the web server to get the desired information.

The information must be "Url-Encoded" then appended to the URL to ship out to the Web Server via the iHttpInit function in the WinInet extender. The iContentUrl function was designed to allow you to build the required Url-Encoded string that has to be glued on to the back of the url.

It looks like... 

;Format form data.  First define the formdata variable as a null string
formdata=""

;Then add the required form fields one at a time.
formdata=iContentUrl(formdata,"InquiryNumber1",UPSNumber) ;UPSNumber defined elsewhere
formdata=iContentUrl(formdata,"TypeOfInquiryNumber","T")
formdata=iContentUrl(formdata,"submit","Track")

Note how the formdata variable is first set to a null string, then added to with each of the iContentURL functions, properly formatting the URL-Encoded string and adding it to the formdata variable.

Then the formdata is glued to the end of the url, separated by a question mark.... 

;Build GETURL  (unique to the URL-Encoded "GET" type of form)
geturl=strcat(mainurl,"?",formdata)

From this point on it is a standard web page fetch operation. The entire example is... 


;Actually useful URL-Encoded GET example
;This example asks the UPS.COM web site for information about a package


;The form on the UPS page to request tracking info, boiled down to the bare
;minimum, looks like this...


;<FORM ACTION="https://wwwapps.ups.com/etracking/tracking.cgi" METHOD="GET">
;   <INPUT TYPE="SUBMIT" NAME="submit" VALUE="Track">
;   <INPUT TYPE="TEXT" NAME="InquiryNumber1" SIZE="24" MAXLENGTH="24">
;   <INPUT TYPE="HIDDEN" NAME="TypeOfInquiryNumber" VALUE="T">
;</FORM>


;Load the WinInet extender

AddExtender("WWINT34I.DLL")

;Query user for desired package number to track
UPSNUmber=AskLine("UPS Tracker","Enter tracking number","Enter UPS Tracking number here.")

;Clean the number up by removing hyphens and spaces if they exist
UPSNUmber=StrReplace(UPSNUmber," ","")
UPSNUmber=StrReplace(UPSNUmber,"-","")

;Define key values
;Get host name from ACTION= parameter or induce it from Web page viewed
host="wwwapps.ups.com"

;Get mainurl also from action line
mainurl="/etracking/tracking.cgi"

;Define output file name
outfile=strcat(DirGet(),"result.html")

;Format form data.  First define the formdata variable as a null string
formdata=""

;Then add the required form fields one at a time.
formdata=iContentUrl(formdata,"InquiryNumber1",UPSNumber)
formdata=iContentUrl(formdata,"TypeOfInquiryNumber","T")
formdata=iContentUrl(formdata,"submit","Track")

;Build GETURL  (unique to the URL-Encoded "GET" type of form)
geturl=strcat(mainurl,"?",formdata)

; Basic web page fetch script
tophandle=iBegin(0,"","")
connecthandle=iHostConnect(tophandle, host, @HTTP, "", "")
datahandle=iHttpInit(connecthandle,"GET",geturl,"", 0)
rslt = iHttpOpen(datahandle, "", 0, 0);

if rslt!=200
      headers=iHttpHeaders(datahandle)
      AskItemList("Rslt=%rslt%",headers,@tab,@unsorted,@single)
endif

xx=iReadData(datahandle,outfile)
iClose(datahandle)
iClose(connecthandle)
iClose(tophandle)

;We now have the returned web page.  Since it is just an
;HTML page, we'll just use the users Web Browser to display it.
ShellExecute(outfile,"","",@NORMAL,"OPEN")
exit

 

 

With any luck you can cut and paste this example and run it. Note that from time to time UPS changes their forms, so if it does not work, then it is an exercise for the reader to debug it again.

URL-Encoded POST example

The AMAZON website at http://www.amazon.com has a nice little generic search box where you can type in a keyword and it will produce a list of products that it sells that you may be interested in. To see if AMAZON carries the item you are interested in, the usual procedure is to fire up the Web Browser, type in the www.amazon.com URL, poke around to see where the search box is this week, type in the keyword and hit the Go button.

This script makes it a lot easier to see if AMAZON has anything like the desired item at all. To start, we hoof it over to http://www.amazon.com and locate the search box, View source, and capture the desired form. 


<form method="post" action="/exec/obidos/search-handle-form" name="searchform">
<select name=index>
<option value=aps selected>All Products
<option value=books>Books
<option value=music>Popular Music
<option value=music-dd>Music Downloads
<option value=classical>Classical Music
<option value="dvd">DVD
<option value="vhs">VHS
<option value=restaurants>Restaurants (Beta)
<option value=theatrical>Movie Showtimes
<option value=toys>Toys
<option value=baby>Baby
<option value=pc-hardware>Computers
<option value=videogames>Video Games
<option value=electronics>Electronics
<option value=photo>Camera & Photo
<option value=software>Software
<option value=tools>Tools & Hardware
<option value=magazines>Magazines
<option value=garden>Outdoor Living
<option value=kitchen>Kitchen
<option value=travel>Travel
<option value=wireless-phones>Cell Phones & Service
<option value=outlet>Outlet
<option value=auction-redirect>Auctions
<option value=fixed-price-redirect>zShops
<option value=moc-scientific-supplies>Sci. Supplies (Beta)
<option value=moc-medical-supplies>Med. Supplies (Beta)
<option value=moc-industrial-supplies>Indust. Suppl. (Beta)
<option value=moc-cars>Car Parts (Beta)
<option value=moc-home-furnishings>Home Furnish. (Beta)
<option value=moc-lifestyle>Lifestyle (Beta)
<option value=moc-pet-toys>Pet Toys (Beta)
<option value=moc-arts-hobby>Arts & Hobbies (Beta)
</select>
<input type="text" name="field-keywords" size="15">
<input type="image" height="21" width="21" border=0 value="Go" name="Go" src="https://g-images.amazon.com/images/G/01/v9/search-browse/go-button-gateway.gif" align=absmiddle>
</TD> </TR> </TABLE> </TD> </TR> </TABLE> </TD> </form>

As we are interested in the bare-bones approach, we can logically simplify the form to 


<form method="post" action="/exec/obidos/search-handle-form" name="searchform">
<select name=index>
<option value=aps selected>All Products
</select>
<input type="text" name="field-keywords" size="15">
<input type="image" height="21" width="21" border=0 value="Go" name="Go" src="https://g-images.amazon.com/images/G/01/v9/search-browse/go-button-gateway.gif" align=absmiddle>
</TD> </TR> </TABLE> </TD> </TR> </TABLE> </TD> </form>

So now to interpret the form and extract required information.

From the METHOD=POST in the from line and the absence of any multipart/form-data encoding type specification we can see that it is a standard URL-Encoded POST type form. The recommended type for most normal Internet transactions.

The required HOST information is missing from the FORM statement, but can be easily deduced as it would be related to the name of the web site you are browsing. 


   host="www.amazon.com"  ;Deduced from form source

   mainurl="/exec/obidos/search-handle-form"   ;From FORM ACTION parameter

There are basically three input controls. The submit button, the keyword to search for, and a option variable to control how far across the various products that Amazon sells the net should be cast. From the simplified form...

The SELECT control named "index" can hold any of the option values. We'll search all products in the this example.

The TYPE="TEXT" edit box control is called "field-keywords" and it will hold the value the user types in.

The TYPE="SUBMIT" button control is called "Go" and when clicked it is set to a value of "Go".

This information needs to be returned to the web server to get the desired information.

The information must be "Url-Encoded" then added as a parameter to the iHttpOpen function in the WinInet extender. The iContentUrl function was designed to allow you to build the required Url-Encoded string that has to be used in the iHttpOpen formdata parameter. It looks like... 


	;Format form data.  First define the formdata variable as a null string
	formdata=""

	;Then add the required form fields one value=pair at a time.
	formdata=iContentUrl(formdata,"index","aps selected>All Products")
	formdata=iContentUrl(formdata,"field-keywords",Keyword)
	formdata=iContentUrl(formdata,"Go","Go")

From this point on it is again almost a standard web page fetch operation. The entire example is... 


;Actually useful URL-Encoded POST example
;This example searches the AMAZON.COM web siterfor something you might 
;want to buy without having to go there.  Quick Easy.  See if they have it

;The form on the AMAZON page for a generic seatch, boiled down to the bare
;minimum, looks like this...

;<form method="post" action="/exec/obidos/search-handle-form" name="searchform">
;<select name=index>
;<option value=aps selected>All Products
;</select>
;<input type="text" name="field-keywords" size="15">
;<input type="image" height="21" width="21" border=0 value="Go" name="Go" src="http://g-images.amazon.com/images/G/01/v9/search-browse/go-button-gateway.gif" align=absmiddle>
;</TD> </TR> </TABLE> </TD> </TR> </TABLE> </TD> </form>

;Load the WinInet extender
AddExtender("WWINT34I.DLL")
origdir=DirGet()

;Query user for keyword to search for
Keyword=AskLine("Amazon Searcher","Enter keyword (15 chars max)","")

;Limit keyword to 15 chars
if StrLen(Keyword)>15 then Keyword=strsub(Keyword,1,15)

;Define key values
;Get host name from ACTION= parameter or induce it from Web page viewed
host="www.amazon.com"

;Get mainurl also from action line
mainurl="/exec/obidos/search-handle-form"

;Define output file name
outfile=strcat(origdir,"result.html")

;Format form data.  First define the formdata variable as a null string
formdata=""

;Then add the required form fields one value=pair at a time.
formdata=iContentUrl(formdata,"index","aps selected>All Products")
formdata=iContentUrl(formdata,"field-keywords",Keyword)
formdata=iContentUrl(formdata,"Go","Go")

; Basic web page fetch script
tophandle=iBegin(0,"","")
connecthandle=iHostConnect(tophandle, host, @HTTP, "", "")
datahandle=iHttpInit(connecthandle,"POST",mainurl,"", 0)
rslt = iHttpOpen(datahandle, 0, formdata, -1);

if rslt!=200
      headers=iHttpHeaders(datahandle)
      AskItemList("Rslt=%rslt%",headers,@tab,@unsorted,@single)
endif

xx=iReadData(datahandle,outfile)
iClose(datahandle)
iClose(connecthandle)
iClose(tophandle)

;We now have the returned web page.  Since it is just an
;HTML page, we'll just use the users Web Browser to display it.
ShellExecute(outfile,"","",@NORMAL,"OPEN")
exit

 

Again, with any luck it will run for you too. Do note that Amazon continually improves their website and makes changes willy nilly to accommodate their sentient customers, thus indubitably this script will one day fail and need to be modified. Consider it yet another exercise for the reader.

Multipart/form data POST example

Well, its not easy finding a website that likes people to upload files to the web. So I've written a file upload utility at the winbatcj web site. It will tell you how big the file is. So if you ever have a file and you cannot figure out how big it is for some reason, you can try uploading it to the file upload demo utility and it will tell you how big it is. Please no humongous bandwidth busting files. The upload web page looks like... 



   <html><head><title>Upload Test</title></head>

   <body>

   <h1> Upload Test</h1>

   <form method="POST" enctype="multipart/form-data" 

   action="https://techsupt.winbatch.com/webcgi/webbatch.exe?How2Forms/upltest.web">

   <p><input type="file" name="TheFile" size="45"></p>

   <input type="submit" value="Submit File" name="submit">

   </form>

   </body>

   </html>

Extracting the important stuff reveals a form that looks like.... 



   <form method="POST" enctype="multipart/form-data" 

   action="https://techsupt.winbatch.com/webcgi/webbatch.exe?techsupt/How2Forms/upltest.web">

   <p><input type="file" name="TheFile" size="45"></p>

   <input type="submit" value="Submit File" name="submit">

   </form>

It looks like

You may have noticed that a good working knowledge of HTML and especially FORM HTML is handy. If so, then you notice well.

So now to interpret the form and extract required information.

From the METHOD=POST in the form line and the enctype="multipart/form-data" encoding type specification we can see that it is a rare Multipart/form-data POST type form. Basically the only way to use http to upload files to websites. I hope it goes without mentioning that you cannot upload forms to any website, the web-server via some sort of scripting (WebBatch/Perl/What-have-you) must be coded to expect file uploads. And that is an entirely separate topic. 



   host="techsupt.winbatch.com"                       ;From FORM ACTION parameter

   mainurl="/webcgi/webbatch.exe?How2Forms/upltest.web"   ;From FORM ACTION parameter

There are just two input controls. The Submit button, and the BROWSE control. Instead of the iContentURL used in the previous two examples, there are two new functions to use, "iContentData" and "iContentFile". Instead of using the iContentURL function to encode the data, we use the iContentData function to help build a BinaryBuffer to contain the normal control data and the iContentFile function to place the file into the Binary Buffer.

This process looks like... 


fs=FileSize(thefile) ; Figure out size of the file to help in estimating
                     ; required BinaryBuffer size

bbsize=fs+4000      ; Make binary buffer big enough  (Educated Wild-eyed Guess method)
                    ; It must hold the file(s) and all other form data
                    ; Be generous.
								  
bb=BinaryAlloc(bbsize)  ; Allocate the Binary Buffer

BinaryEODSet(bb,bbsize)  ; Since we will not be using the standard Binary
                         ; functions, mark buffer as 100% used.

bbaddr=IntControl(42,bb,0,0,0) ; Get the real memory address of the BinaryBuffer

endofcontent=0      ; Now we start building content.  Set the
                    ; endofcontent marker to zero

; Use iConntentFile to store the File information 
; into the BinaryBuffer
endofcontent = iContentFile(bbaddr, endofcontent, "TheFile",  thefile  ,"text/plain")

; Use iContentData to store all other non-file information
endofcontent = iContentData(bbaddr, endofcontent, "submit" ,  "Submit File" )

; Add a special form of iContentData to write the "end of buffer" markers.
endofcontent = iContentData(bbaddr, endofcontent, ""       ,  ""            )

From this point on it is almost a standard web page fetch operation. The entire example is... 





  AddExtender("WWINT44I.DLL")
  origDir=FilePath(IntControl(1004,0,0,0,0))
  outputfile=StrCat(origdir,"result.html")
  
  thefile=AskFilename("FileSize Checker",origdir, "All files|*.*", "*.*", 1 )
  
  host="techsupt.winbatch.com"
  mainurl="/webcgi/webbatch.exe?techsupt/How2Forms/upltest.web"
  
  fs=FileSize(thefile)  ; Figure out size of the file to help in estimating
                        ; required BinaryBuffer size
  bbsize=fs+4000        ; Make binary buffer big enough  (Educated WEG method)
                        ; It must hold the file(s) and all other form data
                        ; Be generous.
  
  bb=BinaryAlloc(bbsize) ; Allocate the Binary Buffer
  
  BinaryEodSet(bb,bbsize) ; Since we will not be using the standard Binary
                          ; functions, mark buffer as 100% used.
  
  bbaddr=IntControl(42,bb,0,0,0); Get the real memory address of the BinaryBuffer
  
  endofcontent=0     ; Now we start building content.  Set the
                     ; endofcontent marker to zero
  
  ; Use iConntentFile to store the File information into the BinaryBuffer
  endofcontent = iContentFile(bbaddr, endofcontent, "TheFile",  thefile  , "text/plain")
  
  ; Use iContentData to store all other non-file information
  endofcontent = iContentData(bbaddr, endofcontent, "submit" ,  "Submit File" )
  
  ; Add a special form of iContentData to write the "end of buffer" markers.
  endofcontent = iContentData(bbaddr, endofcontent, ""       ,  ""            )
  
  tophandle=iBegin(0,"","")
  connecthandle=iHostConnect(tophandle,host,@HTTPS,"","")
  datahandle=iHttpInit(connecthandle,"POST",mainurl,"",4)
  
  rslt=iHttpOpen(datahandle,"",bbaddr,endofcontent);
  
  If rslt!=200
        headers=iHttpHeaders(datahandle)
        AskItemlist("Rslt=%rslt%",headers,@tab,@unsorted,@single)
  EndIf
  
  xx=iReadData(datahandle,outputfile)
  iClose(datahandle)
  iClose(tophandle)
  ShellExecute(outputfile,"","",@normal,"Open")

The Finicky Example

In order to help demonstrate the problems you might run into, I've developed a simple form that can be found at...
https://techsupt.winbatch.com/techsupt/finickytest.html

You press the "Try It" button and the Web Server indicates that is is happy with the form.

The form in the HTML looks like... 


<form method="POST" 

action="https://techsupt.winbatch.com/webcgi/webbatch.exe?How2Forms/finicky.web+bosco+Tom Smith+Fr%21ed&a=v"> 

<p><input type="hidden" name="MushRoom" value="fungusamungus"></p> 

<input type="submit" value="Try It" name="submit"> 

</form>

It looks like this.... But you will notice it does not run here. This will be the the first of many problems with this form. To try it, just push the "Try It" button.

Finicky

When run from this web page, it gets some kind of error message.

So we start. The first attempt at this script is pretty much just a straight-forward implementation of the Url-Encoded POST example above.

Finicky Try #1


;Load the WinInet extender
AddExtender("WWINT34I.DLL")

origdir=DirGet()

;Define key values
;Get host name from ACTION= parameter or induce it from Web page viewed
host="techsupt.winbatch.com"

;Get mainurl also from action line
mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web"

;Define output file name
outfile=strcat(origdir,"result.html")

;Format form data.  First define the formdata variable as a null string
formdata=""

;Then add the required form fields one value=pair at a time.
formdata=iContentUrl(formdata,"Mushroom","Fungusamungus")
formdata=iContentUrl(formdata,"submit","Try it")

; Basic web page fetch script
tophandle=iBegin(0,"","")
connecthandle=iHostConnect(tophandle, host, @HTTP, "", "")
datahandle=iHttpInit(connecthandle,"POST",mainurl,"", 0)

rslt = iHttpOpen(datahandle, 0, formdata, -1);

if rslt!=200
      headers=iHttpHeaders(datahandle)
      AskItemList("Rslt=%rslt%",headers,@tab,@unsorted,@single)
endif

xx=iReadData(datahandle,outfile)
iClose(datahandle)
iClose(connecthandle)
iClose(tophandle)

;We now have the returned web page.  Since it is just an
;HTML page, we'll just use the users Web Browser to display it.
ShellExecute(outfile,"","",@NORMAL,"OPEN")
exit

It flat out does not work. What to do? What to do?

For starters we have to make sure that what we are sending the Web server is the same thing the web server gets when it is done from your Web Browser. And we do know that when we copy the form (as above on this page) and run it, the Web Server does not like it either.

The MAIN Trick

Various servers on the Internet support a "data echo" type of affair. The Island Lake Consulting LLC tech support servers also support this in the form of

http://techsupt.winbatch.com/webcgi/webbatch.exe?dumpinputdata

If you click the link, you get back a page of all kinds of interesting information. It is a mix of information from your computer and the computer that the Web Server runs on. We can use this link to compare what the form via a Web Browser is sending the Web Server, and what your script is sending the web server. The first step is to alter the form to use the data echo server…




<form method="POST" 


action="https://techsupt.winbatch.com/webcgi/webbatch.exe?dumpinputdata+bosco+Tom Smith+Fr%21ed&a=v"> 


<p><input type="hidden" name="MushRoom" value="fungusamungus"></p> 


<input type="submit" value="Try It" name="submit"> 


</form>

The dumpinputdata version of the form:

Finicky

So you get back a dump of the form data. Save it. Then we alter the script slightly. Change the host and mainurl lines from 



      host="techsupt.winbatch.com" 

      mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web"

to 



      host="techsupt.winbatch.com" 

      mainurl="/webcgi/webbatch.exe?dumpinputdata"

Note that the HOST= variable did not change in this specific case, but you would often have to change it to redirect your script to a server that supports the data echo.

Run the script. Now compare the output with the output you received when you clicked the button above. A fine toothed comb is very handy here.

Buried in those listings will be some key fields that need to be modified to make this thing work. There will also be a great number of other differences that will have no effect on the proceedings. You have to winnow out the salient differences and modify your script to take them into account. Good Luck. In order to assist you in learning how to do this, a special "trap door" was installed on the finicky CGI script.

To activate the trap door change the host= and mainurl lines to 



      host="techsupt.winbatch.com" 

      mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web+helpme"

So now the script looks like 


;Load the WinInet extender
AddExtender("WWINT34I.DLL")
origdir=DirGet()

;Define key values
;Get host name from ACTION= parameter or induce it from Web page viewed
host="techsupt.winbatch.com"

;Get mainurl also from action line
mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web+helpme"

;Define output file name
outfile=strcat(origdir,"result.html")

;Format form data.  First define the formdata variable as a null string
formdata=""

;Then add the required form fields one value=pair at a time.
formdata=iContentUrl(formdata,"Mushroom","Fungusamungus")
formdata=iContentUrl(formdata,"submit","Try it")

; Basic web page fetch script
tophandle=iBegin(0,"","")
connecthandle=iHostConnect(tophandle, host, @HTTP, "", "")
datahandle=iHttpInit(connecthandle,"POST",mainurl,"", 0)

rslt = iHttpOpen(datahandle, 0, formdata, -1);

if rslt!=200
      headers=iHttpHeaders(datahandle)
      AskItemList("Rslt=%rslt%",headers,@tab,@unsorted,@single)
endif

xx=iReadData(datahandle,outfile)
iClose(datahandle)
iClose(connecthandle)
iClose(tophandle)

;We now have the returned web page.  Since it is just an
;HTML page, we'll just use the users Web Browser to display it.
ShellExecute(outfile,"","",@NORMAL,"OPEN")
exit

With this new script with the trap door in it, it provides a number of hints about what it does not like. This script was especially bred to be particularly finicky, much more so than any scripts you would encounter in the wild.

So the list of errors, once the trapdoor is activated, looks like... 


Error 1 Command line bad
Error 2A MushRoom not found
Error 2B Mushroom value bad
Error 3B submit value bad
Error 4 Accept language not en-us
Error 5 User Agent does not have Mozilla in it
Error 6 Referrer bad
Error 7 Encoding info bad

Amazingly we seemed to have been able to avoid error 3A. In summary.

Error 1 Command line bad

When I copied the main URL line from the form, I cheated and did not copy the entire line. Try copying the entire line. As in 


mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web+bosco+Tom Smith+Fr%21ed&a=v"

However this STILL does not work. The WinBatch script will cry foul due to the % on the line. A % is a special character in WinBatch. Basically if you want one of them, you have to code two of them into your script. It should read. 


mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web+bosco+Tom Smith+Fr%%21ed&a=v"

And if we want to keep the secret trapdoor in there you can make it... 


mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web+helpme+bosco+Tom Smith+Fr%%21ed&a=v"

Error 2A MushRoom not found
Error 2B Mushroom value bad
Error 3B submit value bad

If you examine the form data... 



<input type="hidden" name="MushRoom" value="fungusamungus"> 

<input type="submit" value="Try It" name="submit">

and compare it with out iContentURL lines 


formdata=iContentUrl(formdata,"Mushroom","Fungusamungus")
formdata=iContentUrl(formdata,"submit","Try it")

You will notice that there are some capitalization differences. The iContentUrl lines should read 


formdata=iContentUrl(formdata,"MushRoom","fungusamungus")
formdata=iContentUrl(formdata,"submit","Try It")

You might note that the name="submit" is the only name/value word that was capitalized correctly. Otherwise that would have been the 3A error.

Error 5 User Agent does not have Mozilla in it

Browsers often send a "User Agent" field to the Web Server so that the Web server can guess at the capabilities of the Browser being used, or even refuse to communicate with "unauthorized" Browsers. This field shows up in the dumpinputdata log as USER_AGENT or sometimes HTTP_USER_AGENT. Mozilla was the name used by Netscape to indicate you had a Netscape Browser. MSIE used it to indicate that the MSIE Browser was Netscape compatible. The script is checking to be sure a Mozilla compatible browser is being used.

It is possible to set a user agent field with the WinInet extender. It is done via an iOptionSet function IMMEDIATELY after the iBegin statement for the session, before other handles are derived from the tophandle returned by iBegin. It should look kind of like... 


; Basic web page fetch script
tophandle=iBegin(0,"","")
iOptionSet(tophandle,"user_agent","Mozilla compatible WinBatch")

Error 4 Accept language not en-us
Error 6 Referer bad
Error 7 Encoding info bad

These errors indicate that additional header information is required. There was a clue that "Error 6 Referer Bad" problem existed when the form did not work when copied to a different html file. The CGI script was checking that the form was filled out on an authorized page, usually to help prevent unauthorized access to some service. In any case all three of these items are simply "Headers" and have to be added into the script. Something like 


moreheaders="Accept Language: en-us"
moreheaders=StrCat(moreheaders,@crlf,"Referer: https://techsupt.winbatch.com/techsupt/finickytest.html")
moreheaders=StrCat(moreheaders,@crlf,"Accept encoding: gzip, deflate")

rslt = iHttpOpen(datahandle, moreheaders, formdata, -1);

So the final working Finicky Script looks like


; An example of a hard form to post due to the excessive
; finickyness of the cgi script receiveing the form

;<form method="POST"
;   action="https://techsupt.winbatch.com/webcgi/webbatch.exe?How2Forms/finicky.web+bosco+Tom Smith+Fr%21ed&a=v">
;<p><input type="hidden" name="MushRoom" value="fungusamungus"></p>
;   <input type="submit" value="Try It" name="submit">
;</form>

;Load the WinInet extender
AddExtender("WWINT34I.DLL")
origdir=DirGet()

;Define key values
;Get host name from ACTION= parameter or induce it from Web page viewed
host="techsupt.winbatch.com"

;Get mainurl also from action line
mainurl="/webcgi/webbatch.exe?How2Forms/finicky.web+bosco+Tom Smith+Fr%%21ed&a=v"

;Define output file name
outfile=strcat(origdir,"result.html")

;Format form data.  First define the formdata variable as a null string
formdata=""

;Then add the required form fields one value=pair at a time.
formdata=iContentUrl(formdata,"MushRoom","fungusamungus")
formdata=iContentUrl(formdata,"submit","Try It")

; Basic web page fetch script
tophandle=iBegin(0,"","")
iOptionSet(tophandle,"user_agent","Mozilla compatible WinBatch")
connecthandle=iHostConnect(tophandle, host, @HTTP, "", "")
datahandle=iHttpInit(connecthandle,"POST",mainurl,"", 0)

;Define the headers
moreheaders="Accept Language: en-us"
moreheaders=StrCat(moreheaders,@crlf,"Referer: https://techsupt.winbatch.com/techsupt/finickytest.html")
moreheaders=StrCat(moreheaders,@crlf,"Accept encoding: gzip, deflate")

rslt = iHttpOpen(datahandle, moreheaders, formdata, -1);

if rslt!=200
      headers=iHttpHeaders(datahandle)
      AskItemList("Rslt=%rslt%",headers,@tab,@unsorted,@single)
endif

xx=iReadData(datahandle,outfile)
iClose(datahandle)
iClose(connecthandle)
iClose(tophandle)



;We now have the returned web page.  Since it is just an
;HTML page, we'll just use the users Web Browser to display it.
ShellExecute(outfile,"","",@NORMAL,"OPEN")
Exit

Good Luck!

Article ID:   W14203

Filename:   HTTP and WinInet - An Opus.txt
File Created: 2021:04:03:09:14:38
Last Updated: 2021:04:03:09:14:38