Home |  White Papers |  Message Board |  Search |  Products |  Purchase |  News | 
         

Building distributed Web Applications with Visual FoxPro

by Rick Strahl, West Wind Technologies
http://www.west-wind.com/

Download wwIPStuff from:
http://www.west-wind.com/wwipstuff.asp

ActiveDoc Sample (requires VFP 7.0)
http://www.west-wind.com/wconnect/wwHTTPDemo.app

last updated: September 7, 2000

The Web has opened a whole new area of development for building truly distributed applications that can run over widely distributed networks and has made it possible to build public access applications at relatively minor cost compared to the infrastructure that was previously required to build this type of distributed application.

Unfortunately, this new medium requires an entirely new approach to application development that focuses heavily on a very limited user interface that is presented in HTML. HTML is a text based markup language that is essentially line based producing output much in the way that ancient word processors like WordStar and WordPerfect of the DOS days produced. Compared to typical rich Windows UI applications even the new 4.0 versions of browsers have a lot of catching up to do in ease of use and usability of the form user interface used for data entry, which is typical for database applications. A lot of things can be done with HTML if you're imaginative, but the end result still leaves a lot to be desired in usability.

But what if you could build an application using Visual FoxPro on the client side talking to a Web server on the other end of the connection that is also a Visual FoxPro application? Rather than using clumsy HTML you could take advantage of the power of VFP's User Interface and ease of data access to build a truly user friendly application and still get the distributability that HTML based Web applications promise.

In this document I'll discuss how to do just by demonstrating some free tools that are available on this site and outlining an example application that uses this technology. You can take a look at the results on the message board of this site - the message board also features an offline reader, which uses a Visual FoxPro client application to communicate with the Web server to retrieve and post messages to and from the server.

HTTP -  More than a protocol for transmitting HTML

The good news is that you don't have to build distributed Web applications with HTML. It is absolutely possible to build an application using a rich UI and use the Web simply as a database interface to communicate. The key to make this work is HTTP - the HyperText Transfer Protocol.

The HyperText Transfer protocol is the underlying messaging protocol that is used for all transactions on the World Wide Web. Although the primary use of the protocol is to power the World Wide Web and HTML based applications, it really can do a lot more than plain HTML. Essentially, you can use HTTP to transport any kind of data over the Web including binary data and even database files!

HTTP is based on a client/server model. Typically, the Web browser is the client that requests data from the Web server. The browser is the display mechanism that shows the content that was served up by the Web server. The server is nothing more than a way station that figures out what type of content to provide to the client. To think of a Web server and HTTP as only providing HTML is a mistake – any kind of data, including files can be passed back and forth over this protocol as long as you follow the rules of the protocol. I’ll show an example of this shortly.

In the latest versions of Windows (Windows 95, Windows NT 4.0), Microsoft has provided high-level support for various Internet protocols in a system library called WinInet. This library supports relatively simple API interfaces for accessing FTP, HTTP and WinSock. Microsoft endowed it with a familiar file based architecture where you can open a connection and then read and write to it directly. This wrapper on top of the WinSock API makes it possible for high level languages such as Visual FoxPro to access the functionality in this system interface directly.

Before I jump in and show you how to create a class that can access the HTTP functionality in WinInet, let’s take a look at the implications of direct access to HTTP. The ability to send and receive data in any format you choose gives you the capability to implement your own client/server architecture that can communicate over any Internet connection.

Here are some useful applications that jump out:

  • Any real time data connection that updates a form from data found on the Web – stock charts or weather information for example – can simply access an HTTP link and download the data. If you provide timely data to your clients (whether it's financial data or an image captured from your backyard), you can make the data available directly from within a VFP application. The data can either be HTML (which you’d have to parse or display via the Web browser control) or data that was formatted a specific way so that it’s easy to get into a VFP table (a comma delimited string for example) and then into a listbox or grid or so forth.
  • Software or data updates lend themselves immediately to this technology. Subscription based services might provide dataupdates over the Internet using HTTP to download files. The same mechanism can be used to update your actual application files, downloading the update and then running an update program.
  • Another good example is an offline reader for an online message board. You might have seen various versions of HTML based newsgroup type applications, such as the Visual FoxPro Universal Thread (www.universalthread.com) or my own message board at www.west-wind.com/wwthreads/. With the ability to read and write data to and from an HTTP link you can use the same existing Web server interface to send down all the data since your last visit, update an existing FoxPro table with the data, then use the Visual FoxPro GUI application to browse the messages. When you need to post a message, all messages are stored in a temporary table, which can then be posted to the Web site the next time you are online or you decide to click upload messages.

One thing to keep in mind, though: Although you are not using HTML to display the output, you still need a backend Web application as discussed in previous Web articles. You still need to run a Web server and some backend software such as Web Connection, Active Server Pages, FoxISAPI or FoxWeb to provide the server portion of your application. The difference is that I won't send back HTML in any of the examples here, but rather data that is shared by the client and server.

WinInet with Visual FoxPro

In order to make life easier in using the WinInet functions I built a class called wwIPStuff which you can download from http://www.west-wind.com/webtools.htm. In addition to the actual HTTP samples I’ll describe below, the class also contains support for sending SMTP mail sending FTP file uploads and downloads, as well as some utility functions to dial and hang up Remote Access (RAS) connections, resolve IP Addresses and Domains using an external DLL. The HTTP functions I’ll discuss here are all implemented in pure Visual FoxPro code using DECLARE –API calls while most of the other methods of the class are wrappers around the calls in the wwIPStuff DLL that makes the more WinAPI and Winsock calls in C++ code.

In order to use WinInet you need the WinInet.dll in your Windows System directory. This DLL installs with most of Microsoft’s latest Internet tools – if you have Internet Explorer 3.0 or later installed you will have it. If you don’t have WinInet on your system you can download it from http://www.west-wind.com/files/wininet.zip. If the examples fail with 'Cannot find entry point to <DLLfunction>' you're likely missing WinInet.dll.

All of the HTTP functions in wwIPStuff are implemented with pure Visual FoxPro code that accesses the WinInet functions directly. You can review the code in the class library on your own to see how it's done - we'll just look at some examples of how to put these tools to work.

Let's start with the simplest way to get data of the Web: The wwIPStuff::HTTPGet method. This method simply retrieves a URL from the Web and stores is to a string. Figure 1.1 shows an example of a simple form that uses this class method. Add an edit box to show the result, a textbox for the URL to retrieve and a button that will perform the actual task of retrieving the data.

 

Figure 1.1 – The HTTPGet method allows retrieval of HTTP requests into a string which is displayed in the edit box. Content can be HTML code as shown here or any type of data you choose to send.

In the button code add the following code:

SET PROCEDURE TO wwIPStuff ADDITIVE
o = CREATE("wwIPStuff")
THISFORM.edtHTML.Value = o.HTTPGet(TRIM(THISFORM.txtUrl.Value),100000)

Note that HTTPGet() expects a full URL that contains the protocol, server and optionally a file to display. The leading http:// is required!

The previous links will retrieve an HTML string as text into the editbox of the form. Retrieving HTML in this manner can be very useful for things such as monitoring sites, or verifying links in directory, or even for building a Web spider that crawls the Web looking for links in pages.

However, if the server were to return plain data as text in result you could do something more useful with the data. Take a look at Figure 1.3, which retrieves data in a comma delimited string over the Web and fills a listbox from the returned result.

 

 Figure 1.3 – Retrieving data over the Web. In this example a comma delimited string is pulled down over the wire and turned into a cursor that can be displayed in a listbox.

The relevant code that goes into the Reload button of the form looks like this:

o=CREATE("wwIPStuff")

*** Retrieve all companies starting with "A"
lcText = o.HTTPGet("http://www.west-wind.com/wconnect/wc.dll?http~CustList1~"+;
TRIM(THISFORM.txtQueryCompany.value) )

if EMPTY(lcText) OR lcText="FAILED"
wait window "Invalid HTTP Response..." nowait
RETURN
endif 

tcFilename=SYS(3)+".txt"
File2Var(tcFileName,lcText)

CREATE CURSOR TCustList ;
( CUSTNO C (8),;
COMPANY C (30),;
CAREOF C (30) )

APPEND FROM (tcFileName) DELIMITED
ERASE (tcFileName)

THISFORM.lstCustList.RowSourceType = 2
THISFORM.lstCustList.RowSource = "tCustList.company, careof"
THISFORM.lstCustList.Requery

You have just run a query over the Web and displayed the data in a listbox with 15 lines of code! The server request sends down a comma delimited string that you can import into a table and use as you see as you see fit in your code. I pass the query parameter specified in the listbox as part of the URL Query string (this would be the Address/Location line if run from a browser). This works well for simple parameters, but is rather limiting if you need to pass more information. We'll look at another approach later.

As previously stated you need to have a server in place that can actually service the request made. In this case the Visual FoxPro Web Connection server code is very short as well:

********************************************************
* HTTPDemo :: CustList1
*********************************
***  Function: Returns a customer list based on the URL 
***            'parameter' passed. Delimited returned.
***    Assume: wc.dll?http~CustList1~CompanySearchString
********************************************************
FUNCTION CustList1

lcCustToFind = THIS.oCGI.GetCGIParameter(3)

lcFile=SYS(3)+".TXT"

SELECT custno,Company, Careof ; 
   FROM (DATAPATH + "TT_Cust") ;
   WHERE Company = lcCustToFind ;
   ORDER BY Company ;
   INTO CURSOR TQuery

COPY TO (lcFile) TYPE DELIMITED 

*** Send the Delimited string over the wire
THIS.oHTML.Send(File2Var(lcFile))

ERASE (lcFile)

USE IN TQuery
USE IN TT_Cust

ENDFUNC
* CustList1
************************************************************************
FUNCTION File2Var
******************
***    Author: Rick Strahl
***            (c) West Wind Technologies, 1995
***  Modified: 01/28/95
***  Function: Takes a file and returns the contents as a string or
***            Takes a string and stores it in a file if a second
***            string parameter is specified.
***      Pass: tcFilename  -  Name of the file
***            tcString    -  If specified the string is stored
***                           in the file specified in tcFileName
***    Return: file contents as a string
************************************************************************
LPARAMETERS tcFileName, tcString
LOCAL lcRetVal, lcOldAlias, lnHandle, lcOldSafety

tcFileName=IIF(type("tcFileName")="C",tcFileName,"")
tcString=IIF(type("tcString")="C",tcString,"")

lcRetVal=""
lcOldAlias=ALIAS()

IF EMPTY(tcString)
   *** File to Text
   
   *** Make sure file exists and can be opened for READ operation
   lnHandle=FOPEN(tcFileName,0)
   IF lnHandle#-1
     DO WHILE !FEOF(lnHandle)
        lcRetVal=lcRetVal+FREAD(lnHandle,16484)
     ENDDO

     =FCLOSE(lnHandle)   && Close the file
   ENDIF
ELSE
   *** Text to File
   lnHandle=FCREATE(tcFileName)
   IF lnHandle=-1
      RETURN .F.
   ENDIF
   =FWRITE(lnHandle,tcString)
   =FCLOSE(lnHandle)
   RETURN .T.
ENDIF

RETURN lcRetVal
*EOP File2Var

Note that the server is sending the text down without any decorations: No HTTP Header, which is required only for display in a browser. This is raw data containing only the comma delimited list of records downloaded.

 Let's take this a step further. A comma delimited list is cool, but it won't work for complex data or data containing memos. Wouldn't it be nice to have an automated mechanism for sending database files? To do so I created a couple of high-level methods that wrap encoding and decoding of DBF files sent over HTTP into single method calls that encode a DBF file and possible memo. There are two reasons for this: The header identifies the result as a file and provides a minimal check to make sure the file(s) made it, and second it automates the process of reading the file into a string in a single method call. wwIPStuff implements EncodeDBF() and DecodeDBF() methods to do just this.

With these methods in place you can now send files even more easily – the logistics of converting the file are handled by the single method calls to DecodeDBF and EncodeDBF. In the following example, the client code uses DecodeDBF() to decode the file that was sent down from the Web server:

*** Retrieve all companies starting with "A"
lcText = o.HTTPGet("http://localhost/wconnect/wc.dll?http~CustList2~A")
*** Creates the file including Memo
IF !o.DecodeDBF(lcText,"TCustList.dbf")
   RETURN && DecodeDBF will display nowait WAIT win
ENDIF
USE TCustList
BROWSE
USE
ERASE TCustList.dbf
ERASE TCustList.FPT
RETURN

On the server the code uses EncodeDBF() to encode a query that was run and sends it out through the Web server:

********************************************************
* HTTPDemo :: CustList2
*********************************
***  Function: Returns a customer list based on the URL 
***            'parameter' passed. This time as file!
***    Assume: wc.dll?http~CustList1~CompanySearchString
********************************************************
FUNCTION CustList2

lcCustToFind = THIS.oCGI.GetCGIParameter(3)

lcFile=SYS(3)+".DBF"

*** This query includes Memos
SELECT custno,Company, Careof, Address, phone ; 
   FROM (DATAPATH + "TT_Cust") ;
   WHERE UPPER(Company) = UPPER(lcCustToFind) ;
   ORDER BY Company ;
   INTO DBF (lcFile)
USE

o=CREATE("wwIPStuff")

*** Encode with Memo File
lcText=o.EncodeDBF(lcFile,.T.)

*** Send the Delimited string over the wire
THIS.oHTML.Send(lcText)

ERASE (lcFile)

USE IN TT_Cust

ENDFUNC
* CustList1

Cool, isn't it? Less than 20 lines of code for both the client and server! With this basic technology you can very easily update data over the Web. To make this even more efficient you could add third party ZIP control to the Encode and Decode functions and ZIP the data on the fly once it gets over a certain size cutting down on the size of the text travelling over the wire.

You can use a standard GUI VFP application to access information that is retrieved on a remote server. This can be a polling type link that might use a timer to occasionally update data you see in a form, or a by-request operation where the user requests the update from a button click.

But wait, there's more – Sending data to the server

Up to now we've only requested data from the server, but your application might need to update data directly on the server. For example, you might have a salesperson log on to the Web and dump the sales data she collected over the course of the day back to the homebase. In order to do this you'll need a mechanism to send data to the Web server from your VFP application.

HTTP has a built-in mechanism for sending data to the server called a POST request (actually there are several ways, but the POST is the one most commonly used). The most common use of POST requests occur when you submit an HTML input form on a Web page. The data is encoded (URLEncoded format and sent up to the server via the HTTP request initiated by the browser. A typical Web backend application can then query the posted data to retrieve the form variables as part of the server data sent to the backend. You've seen examples of this with Web Connection (loCGI.GetFormVar()), Active Server (Request.Form()) or FoxWeb (FormVar()). Typically, only small amounts of data are sent via POST fields, but here again you can really send any kind and size of data.

When I showed the HTTPGet() method I used the simplified WinInet function InternetOpenUrl() which handles most of the transaction of opening, loading and retrieving the result. When using POST you need to use the low level functions to add the additional settings required to send a buffer. This means calling three separate methods: HTTPConnect() to connect to a server, HTTPGetEx() to actually retrieve or send the data and HTTPClose() to shut down the connection. The actual code also has to take a few extra steps, making a few additional low level connections through WinInet – in all it's quite a jumble of DECLARE – API definitions. But the low level mechanism pays off by providing many other options such as allowing access to Authentication of username and password, secure connections via SSL, the HTTP headers and most importantly the POST buffer.

In order to send data to the server you need to POST the data. The wwIPStuff class implements this functionality via a cursor with a memo field and the AddPostKey() method shown below. Why a cursor? Originally I intended to use a string property of the class, but unfortunately the code would blow up when I tried to assign a huge string (1 meg and over) containing an URLencoded file to the property. Apparently, string properties cannot be of unlimited size. The memo field solved this problem, but required some logic to create and clean up the file (the cleanup occurs in the form's Destroy which is not shown here).

AddPostKey() properly encodes the variable data and formats it to simulate an HTML form submission. POSTed data must be in URLEncoded format, which strips out all 'unsafe' characters and converts the input into a plain ASCII string that. Safe in this context are only A-Z, a-z and 0-9 – all other characters are converted into their hex ASCII codes: %0D for a carriage return (CHR(13)) for example. This process can be very slow, especially if you do it with Visual FoxPro code on a very large binary file(remember our goal is to send binary file images of DBF files). The URLEncode function handles the conversion of the string. To make this slow process a lot faster on large files, the wwIPStuff.dll file contains a routine that does it with C code when the buffer is greater than a couple of thousand bytes.

The example in Figure 1.3 is a form that contains a grid into which you can type a company, name and message. The idea is that you can dynamically create the table that is to be sent to the server. Type some data into the grid then click the 'Send File to Server' button to actually POST the data by sending the file as a POST variable called CustFile.

The server receives the data and decodes the file back into its DBF form. The server then inserts a new record into the table ("Hey there from the server") re-encodes the file and sends it back to your VFP form on the client side, which then displays the result in the lower grid.

Figure 1.3  – This form allows entering of data into a grid and then sends the information to the Web server.
In this example, the server responds by appending a record to the data and then retunring it back to the client which is then displayed in the lower grid.

Here's the relevant code that goes into the 'Send' button Click event:

o=CREATE("wwIPStuff")

wait window nowait "Selecting data to send..." 

*** Clear the result file and the result grid
SELE TGetDownload
ZAP
THISFORM.Grid2.refresh

*** Select all items from the input cursor
SELECT Company, Name, Message FROM TPostTest ;
    ORDER BY Company, Name ;
    INTO DBF TEMPFILE 
USE  && Must close before reading

wait window nowait "Encoding data..."

*** Encode the file and memo
lcFileText=o.EncodeDBF("TempFile.dbf",.T.) 

*** Create the Post Buffer
o.AddPostKey("CustFile",lcFileText)

*** Init vars that need to be passed by reference
lcBuffer=""
lnSize=100000

wait window nowait "Connecting to site..."

lnResult = o.HTTPConnect("www.west-wind.com")  
IF lnResult # 0
   wait window "HTTPConnect error: "+o.cErrorMsg
   RETURN
ENDIF   

wait window nowait "Sending data and retrieving result file..."
lnResult=o.HTTPGetEx("/wconnect/wc.dll?http~SendCustList3",;
            @lcBuffer,@lnSize)
IF lnResult # 0
   wait window "HTTPGetEx error: "+o.cErrorMsg
   RETURN
ENDIF   

file2var("temp.txt",lcBuffer)

*** Decoding result file from server
lcFileText = o.DecodeDBF(lcBuffer,"TempFile.dbf")

file2var("temp.txt",lcFileText)

SELE TGetDownload
ZAP
APPEND FROM TempFile
GO BOTTOM
THISFORM.Grid2.refresh

ERASE TEMPFILE.DBF
ERASE TEMPFILE.FPT

wait clear

o.HTTPClose()
RETURN

On the Web Connection backend server, only a few more lines of code are required. The server code retrieves the "CustFile" form variable and decodes the string into tTempfile.dbf. At this point your code could do whatever it needs with the file retrieved from the client. In this sample I'll just add another record to this file to indicate that this file actually reached the server by embedding the server name, my name and a message that contains the date and time so you can see it change when you run this request multiple times. The updated file is then sent back to the client side.

********************************************************
* HTTPDemo :: SendCustList3
*********************************
***  Function: Sends a customer list that's encoded 
***            in CustFile and displays result as HTML
********************************************************
FUNCTION SendCustList3

loHTML=THIS.oHTML
lcFileBuffer = THIS.oCGI.GetFormVar("CustFile")

o=CREATE("wwIPStuff")
IF !o.DecodeDBF(lcFileBuffer,"TTempFile.dbf") 
   THIS.ErrorMsg("Invalid File info")
   RETURN
ENDIF

USE TTempFile

INSERT INTO TTempFile(Company, Name, Message) ;
    VALUES (THIS.oCGI.GetServerName(),"Rick Strahl",;
     "Hey there from the server at: "+TIME())

*** Close the file and delete it!
USE In TTempFile

lcFileText=o.EncodeDBF("TTempFile.dbf",.T.)

loHTML.Send(lcFileText)

ERASE TTempFile.DBF
ERASE TTempFile.FPT

RETURN
ENDFUNC

Putting it all together

You now have all the pieces to build an application that can run over the Internet as a Client Server application using a totally open, non-propriatary protocol that can access your remote server from anywhere there's an Internet connection available. Whether you want to build offline applications like a message reader, or whether you just want to build applications that have a more sophisticated front end than what HTML can provide, this simple client/server architecture makes it possible to build distributed applications with the tools you already know.

In review a typical application that runs over HTTP should take advantage of the HTTP connections in the following ways:

  • Sending 'Command' via the URL's command line. For example, wc.dll?http~ShowCustData~DA1111.
  • Retrieving data from an HTTP link via HTTPGet
  • Sending data to the server via POSTing data using HTTPGetEx

Let's come back to the message board example on my Web site I mentioned in the first article. My site hosts a message board that is used to post messages for support of Web Connection and general Web programming issues. People access the Web site to view information online. I now want to build an offline reader that allows these same visitors to use a VFP application to view that same data. Rather than browsing the Web site the users will download messages via HTTP and merge them into an existing file. At the same time any messages that users want to post can be sent up to the server and merged into the online file for updating. The motivation for this operation is pretty clear: You can build a vastly more efficient UI with a VFP application than on the Web and the access to the data is much faster. A one time download which usually takes less than a minute can bring down the data for the day immediately.The reader application is available for download at http://www.west-wind.com/wwReader.asp.

http4.gif (40260 bytes)

Figure 1.4 - Which application do you think is faster and easier to use? The VFP application (top) has a rich UI using a Treeview to display data that is running against local data downloaded from the Web. The Web application doesn't allow the same type of flexibility like determining unread messages (the red marks top), since that information cannot be stored on the server.

Let's take a look at some of the key code elements of this application. The following code demonstrates a real world example including error checking on how you can put this technology to work:

* Form method that handles file downloads
* Pass qualified WHERE clause for the SELECT statement
* Filter must be timezone adjusted (in Download form)
Function DownLoadMessages
LPARAMETER lcDownLoadFilter
loIP=CREATE("wwIPStuff")
*** Add the Download Filter as a POST key
loIP.AddPostKey("Filter",lcDownLoadFilter)
THISFORM.StatusMessage("Downloading Messages...",,1)
lnResult = loIP.HTTPConnect(wwt_cfg.server)
IF lnResult # 0
   THISFORM.StatusMessageWAIT WINDOW NOWAIT ("Error: "+loIP.cErrorMsg)
   RETURN -1
ENDIF
*** Presize the result buffer
lcBUffer = SPACE(500000)
lnSize = LEN(lcBUffer)
lnStartTime = SECONDS()
lnResult = loIP.HTTPGetEx(;
      "/wconnect/wc.dll?wwthreads~Downloadmessages",;
      @lcBUffer,@lnSize)
IF lnResult # 0
   THISFORM.StatusMessageWAIT WINDOW NOWAIT ("Error: "+loIP.cErrorMsg)
   RETURN -1
ENDIF
IF lcBUffer = "ERROR - No Records"
   THISFORM.StatusMessage("No messages to download")
   RETURN -1
ENDIF
IF EMPTY(lcBUffer)
   THISFORM.StatusMessage(;
     "Error: No data was returned by the server...")
   RETURN -1
ENDIF
IF !loIP.DecodeDbf(lcBUffer, "TImport.dbf")
   THIS.StatusMessage("Error: File Import failed. "+;
                      "Too many messages!")
   RETURN -1
ENDIF
*** All went well - Now import the messages
lnReccount = 0
SELE 0
USE TImport
SCAN
   SELE wwThreads
   LOCATE FOR Msgid = TImport.Msgid
   IF !FOUND()
      SELE TImport
      SCATTER MEMO MEMVAR
      SELE wwThreads
      APPEND BLANK
      GATHER MEMO MEMVAR
      lnReccount = lnReccount + 1
   ENDIF
ENDSCAN
USE IN TImport
ERASE TImport.* DBF
ERASE TImport.fpt
*** Refresh the form with the new data
THISFORM.BuildTree()
THISFORM.StatusMessage("Downloaded "+LTRIM(STR(lnReccount))+" message(s) in " +;
                               STR(SECONDS() - lnStartTime,2) + " seconds")
RETURN lnReccount

There are a couple of notable issues here. This code works by sending a request to the server with a POST variable called FILTER, which is a complete WHERE clause to a SELECT statement. Typically the filter contains a date range that is adjusted for timezones, but this code can also be used from a search dialog simply by passing the appropriate search parameters in the filter expression to provide a Web based search. This makes this method reusable for all download operations that are required by the application.

Note that the HTTP download buffer is allocated to a whopping 500k bytes in order to make sure that I can capture a reasonably large file. Even so, 500k will only allow retrieval of approximately 3 weeks' worth of data – anything more and the download will fail. Since I have a rough idea of traffic I give the user a warning dialog if he's trying to download more than 3 weeks data at once.

Next notice all the error handling. With these file downloads it's extremely important to check every call that is returned from Wininet. If an error occurs, back out gracefully with a message to the user. One of the things you need to decide on the server end is what to return in case of an error. I tend to use a simple command language to describe operations. For example, if I run a request that asks the server to perform a task and return no data, I return 'OK' for success and 'ERROR – Error Message' for any error. In this case the request is returning a file so a check for an empty string and then for proper size as specified by the encoding header (this happens in DecodeDBF()) is performed on the result.

On the Web Connection server end the code looks like this:

* HTTPProcess :: DownloadMessages
FUNCTION DownLoadMessages
LOCAL lcFilter, lcToDate, lcFromDate, lcForum
lcFilter = THIS.oCGI.GetFormVar("Filter")
IF EMPTY(lcFilter)
   lcForum = THIS.oCGI.GetFormVar("Forum")
   lcFromDate = THIS.oCGI.GetFormVar("FromDate")
   lcToDate = THIS.oCGI.GetFormVar("ToDate")
   
   lcFilter="Forum='"+PADR(lcForum,30) +"' AND " +;
            "timestamp >= {" + lcFromDate +"} AND "+;
            "timestamp <= {" + lcToDate + "} + 1"         
ENDIF         
lcFile = SYS(3)
SELECT ThreadId, Msgid, Subject, Message, FromName, FromEmail, To,Forum,;
       TimeStamp ;
       FROM (DATAPATH + "wwThreads") ;
       WHERE &lcFilter ;
       INTO DBF (lcFile)
IF _TALLY < 1 
   THIS.oHTML.Send("ERROR - No Records")
   USE
   ERASE (lcFile + ".*")
   RETURN
ENDIF
USE
loIP=CREATE("wwIPStuff")
lcFileText=loIP.EncodeDBF(lcFile+".dbf",.T.)
IF EMPTY(lcFileText)
   THIS.oHTML.Send("ERROR - File not encoded.")
   ERASE (lcFile + ".*")
   RETURN
ENDIF   
IF LEN(lcFileText) >= 500000
   THIS.oHTML.Send("ERROR - File is too large to send.")
   ERASE (lcFile + ".*")
   RETURN
ENDIF
   
*** This is the actual FILE Send operation!
THIS.oHTML.Send(lcFileText)
   
ERASE (lcFile + ".*")
ENDFUNC

Note the sending of the error messages in the appropriate places. The error messages can be retrieved on the client side and can provide meaningful display on in the status bar of the reader. Note in particular the size check for the 500k response size. If the encoded file is greater than 500k a message is sent back rather than attempting (and failing) to send a file that's too large to send to the client. The client and server sides obviously need to agree on these sizes for this to work properly.

For good measure here's the client code for uploading messages to the server:

FUNCTION UploadMessages
loIP=CREATE("wwIPStuff")
THISFORM.StatusMessage("Uploading Messages...",,1)
SELECT * FROM wwThreads ;
   WHERE Post AND !DELETED() ;
   INTO DBF TExport
   
IF _TALLY < 1
   USE
   ERASE TEXPORT.DBF
   ERASE TEXPORT.FPT
   THISFORM.StatusMessage("No messages to upload...")
   loAPI=CREATE("wwAPI")
   loAPI.Sleep(2000)
ENDIF   
THISFORM.StatusMessage("Uploading "+LTRIM(STR(_Tally))+" Messages...",,1)
USE
lcFileText=loIP.EncodeDBF("TExport.dbf",.T.)
ERASE TEXPORT.DBF
ERASE TEXPORT.FPT
IF EMPTY(lcFileText)
   THISFORM.StatusMessage("Invalid File Info - not uploaded")
   RETURN
ENDIF

loIP.AddPostKey("FileText",lcFileText)

lnResult = loIP.HTTPConnect(wwt_cfg.server)
IF lnResult # 0
   THISFORM.StatusMessage("Error: "+loIP.cErrorMsg)
   RETURN
ENDIF   
lcBUffer = SPACE(500000)
lnSize = LEN(lcBuffer)
lnResult = loIP.HTTPGetEx("/wconnect/wc.dll?wwthreads~UploadMessages",;
                          @lcBuffer,@lnSize)
IF lnResult # 0
   THISFORM.StatusMessage("Error: "+loIP.cErrorMsg)
   RETURN
ENDIF   
*** Must check if the Upload went Ok
if lcBuffer # "OK"
   *** No - don't delete messages to post
   THISFORM.StatusMessage("File Upload Failed")
   RETURN
ENDIF
THISFORM.StatusMessage("Deleting Posted Messages...",,1)
DELETE FROM wwThreads WHERE Post 
THISFORM.StatusMessage()
RETURN

Here the result from the POST operation simply returns OK or an error code which is irrelevant – it either worked or didn't.

Don't Forget about Security!

Sending data over the wire, especially over an open Internet connection, can be dangerous of course. Remember that it's possible to intercept the data travelling over the net with a packet analyzer and potentially highjack sensitive information. Your first line of defense is to use HTTPS (Secure HTTP or SSL) to transmit your data to and from the Web server. HTTPS requires a secure certificate on the Web server (see your Web server documentation on how to obtain and install a secure certificate or go to www.verisign.com for signup). Once installed all communication that occurs over HTTPS is encrypted. Unfortunately, the encryption process also slows down communications noticeably. The wwIPStuff class supports HTTPS using the low level HTTP functions by specifying the fourth parameter of .T. for HTTPConnect() to establish a secure link. 

FUNCTION HTTPConnect
LPARAMETER lcServer, lcUsername, lcPassword, llSecure

Another simple way to protect yourself from unauthorized access is to use passwords. WinInet supports security via standard Web-based Basic Authentication or via NT Challenge Response that uses NT domain security for permissions. You can connect to the server using a specific security context which exists only while connected to the server over the HTTP connection. wwIPStuff supports this with the second and third parameter to the call to HTTPConnect(). In Web Connection, you can force a request to password validate a user with the following code:

*** See whether user is already Authenticated
lcUser = THIS.oCGI.GetAuthenticatedUser()
IF EMPTY(lcUser)
   *** Nope – force Login dialog
   THIS.oHTML.HTMLAuthenticate()
   RETURN
ENDIF

HTMLAuthenticate() is a method in the Web Connection wwHTML class that requests Authentication from the Web server via an HTTP header that is returned as a result. The HTTP result looks like this:

HTTP/1.0 401 Not Authorized
WWW-Authenticate: basic realm="localhost"
<HTML><h2>Gotta enter your password to get in!</h2></HTML>

WinInet supports navigating this request and sending the username/password to the server and essentially logging the user in. The server request is re-run and this time the Authenticated user is authenticated and the request can proceed. The client end is automatic with the call to HTTPConnect(). On the server you have to implement the code above to enforce the authentication check.

Another security issue to keep in mind is that links that download data typically can also be accessed by a Browser directly. While users will likely never see the actual link that is called, it's possible to access the same data link you might use to retrieve data directly via the Location URL line in the browser if a user can guess the URL. Nothing like somebody figuring out your file upload link, and sending you continuous uploads that will fill up and crash your server's hard disk. Consider your URL names carefully and use Authentication where applicable to avoid these problems.

Note that both Secure HTTP and Authentication don't work with the plain HTTPGet. If you only want to retrieve data securely you have to use HTTPConnect() and HTTPGetEx().

What about other Server Tools

I've used Web Connection as the Web Server backend in the examples above, since that's what I use for most of my development. If you use another tools such as FoxISAPI or Active Server Pages you need to make a few adjustments. I haven't checked out tools like FoxWeb or X-Works – you need to check with the authors whether binary output is supported.

The process for FoxISAPI is identical to the one used by Web Connection and you use the same exact classes to generate your binary data to send over the wire. To send a request back you simply return the string as a result from your FoxISAPI Automation server method. However, the version of FoxISAPI that ships with VFP does not support binary data that contains NULLs (chr(0)) as the Variant conversion function that sends your output back to the Web server truncates any string at the NULL. I've fixed FoxISAPI.dll to work around this problem and the updated, compiled version is included in the wwIPStuff ZIP file.

With Active Server Pages the problem is more complicated. VBScript uses double byte Variants internally which can cause several problems with binary data. Variant strings are null terminated so when you send a binary string to output they terminate at the NULL. You can get around this problem by using the Response object's BinaryWrite method instead of the basic Write. This method directly outputs any text string (or Byte Array as the docs call it) as is without first converting it to VB's double byte wide character formatting.

It's not quite as easy as simply calling BinaryWrite from within a VFP COM server however, because BinaryWrite cannot deal with NULL string ( CHR(0) ) values passed to it as a string over COM interfaces. BinaryWrite can only deal effectively binary data in byte arrays. In VFP you can use the little know  CreateBinary() function to create a byte array from a string. You'd use this with a binary value like an encoded DBF file like this:

Response.BinaryWrite( CreateBinary(lcBinaryData) )

This will send the binary string into the ASP HTTP output stream that allows you to send any type of binary data to the client as shown in the examples above. Most of wwIPStuff's functions can still be utilized in this fashion with ASP.

WinInet Issues

There are a couple of issues related to using WinInet's HTTP functions that you should be aware of in the context of creating HTTP transfer operations.

Although the wwIPStuff class contains support for connection, receive and send timeouts (WinInetSetTimeout()), a request that hangs in the middle of a connection will not timeout according to these values. Instead a Windows system default (typically 30-40) seconds is applied by WinInet. This may be a problem because the WinInet functions don't provide any feedback while in process and the user may think the request hung. Microsoft calls this by design, because that's the way Web servers typically handle timeouts – based on the Web server specified timeout value to actually allow large requests to complete. I suggest you carefully test your requests under various connection environments to see exactly how your application may be affected.

Sending huge files in either direction is probably not a good idea, unless you have a speedy Internet connection. Bandwidth is always critical, but you also need to think of some of the system limitations. The process of URLEncoding a string to be sent to the server is slow especially when using FoxPro code to do the Encoding. This is why the online version switches to .DLL functions for faster encoding operation.

WinInet also requires that all info is passed in character buffers. For example, on my message board I have set up a buffer of 500k bytes to receive messages in. While that seems like a lot, if somebody tries to download all messages since the dawn of time it's not going to fit into 500k! What's worse is that you really have no way of knowing up front how much data comes down unless you run two requests – one to figure out how much data will be returned and the other to actually get it. Once you start downloading there's no feedback on progress and if there's a failure you have to start over. In addition to data size, the URLEncoding process also makes your data much larger - up to three times as big as the original, as a single character can be replaced by three (%OD for example for a Carriage Return (Chr(13)). You can get around the size limitations and feedback issues by 'chunking' your downloads – break them into smaller files and then request the remaining files successively. It requires some logic and statekeeping to make this work efficiently.

Another size saver is to use multi-part form encoding instead of URLEncoding which is standard for Web browsers. Multi-part forms can send data in raw form with only an added text header but no encoding of the actual data. This saves both time and memory since no encoding or bloating of the data occurs. wwIPStuff supports posting data with its nHTTPPostMode property with a value of 2. Support on the server may be another story however - it's directly supported in Web Connection ( using GetMultiPartFormVar() ), but not in FoxISAPI and only in limited form for Active Server Pages

You can also get some size relief by using a third party Zip control like DynaZip. .DBF data is a prime candidate for compression yielding 80% compression or better for typical tables. The message board actually uses DynaZip to perform compression and this simple step has brought tremendous improvements in transfer times bringing down 200+ messages over 28.8k in under a minute.

Unrelated to file transfers, I've had a few problems with WinInet when accessing a lot of different sites in quick succession. For example, using wwIPStuff to build a Web Crawler or to verify site links, you might fire off various requests in quick succession. WinInet handles some operations in the background on separate execution threads and sometimes when quickly connecting and disconnecting from different sites, some of the threads don't clean up properly, leaking handles. Furthermore, accessing a link that redirects to another page causes WinInet to leak 3 handles as the connection is never properly closed regardless of your releasing the IP handles. Make sure you check typical operation with WinInet to see whether your code is affected by these problems. I expect these to get fixed by Microsoft as WinInet use becomes more prevalent. Neither of these issues should be a problem if you're implementing applications that connect to the same server all the time.

Your Turn

Realize that this architecture is not meant to be a replacement for HTML front end applications. By using this approach you are requiring all clients to have Win95/NT and the Visual FoxPro runtime, so all the cross-platform and thin client advantages that HTML brings to the party don't apply. But you do gain the ability to take advantage of the Web's distributed environment to make your application reach out and communicate with users from anywhere.

If you're building corporate or shrink-wrapped applications that need to communicate from widely spread out locations, this is an easy way to link applications to a home site. Whether you're pulling occasional data updates or querying real time data over the Web, you get the distributed aspect of Web applications without having to bite the HTML bullet.

The big benefit is that you can take advantage of VFP's rich UI features to provide a productive work environment and still provide the distributed, plug-in-from-anywhere connectivity. I use this front end for a number of maintainence operations, from downloading orders from my Web site directly into my Point of Sale application to checking my error logs on various sites by downloading them to the local machine. In all of my uses, this VFP front end is only an extension to existing Web applications that are running a full HTML interface. The possibilities are endless and many companies are already taking advantage of this type of interface.
   Home |  White Papers |  Message Board |  Search |  Products |  Purchase |  News |