| Home | White Papers | Message Board | Search | Products | Purchase | News | | |
Building distributed applications
with XML messaging
By Rick Strahl
http://www.west-wind.com/
Last Updated: 02/11/2001
Related Links:
General XML Examples:
http://www.west-wind.com/wconnect/enterprise.htm
wwXML XML Conversion class
http://www.west-wind.com/wwxml.asp
Using SOAP to call Remote Objects
http://www.west-wind.com/presentations/soap/
XML is one of the key technologies that is driving Enterprise Web development today. XML promises a standard data format that can be shared easily across applications and especially by different organizations that need to share data. In this article Rick reviews XML's key features and problems as a data representation format for relational data and objects. He'll also introduce some free tools to provide easy translation between XML and traditional data structures and shows how to use them with several real-world examples.
XML is shaping up as a key technology in distributed applications. You couldn't have missed it in the hype happening in the trade papers. But unlike other over-hyped technologies XML has been rapidly accepted because it solves very specific technical problems by using a standard protocol/data representation. The simple act of agreeing to a common data format for data is making data exchange drastically easier than it ever was before.
In this article I want to discuss XML from the perspective of a messaging and data representation mechanism in distributed applications. For these applications XML is the ideal transport mechanism as it can open up applications to all sorts of clients including Fat Client, Thin Client, Browser based and even Palm Top applications. I'll introduce some concepts of persisting data and objects to XML generically that make it possible to build flexible solutions that can work well in standalone applications as well as distributed applications sharing data over the Web by building smart components and objects and using XML to represent that data as it travels over the Web.
In the process I'll also introduce some free Visual FoxPro and COM tools that help you easily persist your objects and data into XML quickly and efficiently with a few lines of code as well as introducing you to the client side tools you can use to use the XML data in a browser and in a Visual FoxPro Fat Client application transparently. But before we dig into code, let's review the premise of XML as it relates to a message mechanism used to represent and transfer data.
XML as a Data Representation and Messaging Standard
The full XML spec is rather complex because there are so many related technologies such as XSL, schemas and data definitions that are making XML complex for the first time user. Those offshoots provide powerful functionality to XML, but they don't exactly make learning it easy. However, at the core XML is a straightforward standard that's easy to understand and work with thanks to a standard XML parser object model which is consistent on different development platforms (client/server, Thin Client/Fat Client, Windows/Unix/Mac).
Even in its simplest form XML as a standard has lots of potential as a data representation and messaging mechanism. Data representation typically involves translating the data from a native format into XML, and then back into the same format or even a completely different one on the other end of a connection. A good example may be a Visual FoxPro/VB application publishing some of its data via XML and another application, possibly running a Java applet on Unix, picking up that XML data and using it internally. For a second consider that this data is dynamically generated on a Web server with the Java applet asking for specific data and receiving the result back as XML.
XML as a standard data exchange format
As you might expect, this is particularily useful in Web applications where XML can provide a standard, agreed upon format to transfer data over the Internet. In this respect XML is similar to a data format such as Comma Delimited or SDF file in the past. However, as a data representation format XML is also much more flexible than these old formats, because it can carry just about any kind of data. Most other text formats have been hampered by their limited ability to transport complex data like memo fields or binary data. XML is very flexible in what kind data it can carry because of its tag based language definition where every XML data element is marked with a start and end tag (<tag>value</tag>).
Support for multiple sub-documents
Another huge benefit of XML as a data representation mechanism is that XML can combine multiple pieces of data into a single document. The markup language has support for stacked and hierarchical data representation. XML documents can combine several separate entities (be it tables, objects, messages or metadata) into a single XML document. For example, you can send the actual data of say a table, as well as a message header that describes the data or maybe contains any error conditions that might have occurred in obtaining that data. You could also combine multiple tables (as an example) into a single document. Or a table and an object both parsed into XML.
A stacked document may look like this:
<?xml version="1.0"?>
<docroot>
<errors>
<error errorno="1">No PO data to retrieve</error>
<error errorno="2">Invoice rebalancing failed</error>
</errors>
<jobinfo>
<action>processcustomers</action>
<action>processinbox</action>
</jobinfo>
<customers>
<customer>
<lastname>Strahl</lastname>
<firstname>Rick</firstname>
</customer>
<customer>
<lastname>Shizm</lastname>
<firstname>Frank</firstname>
</customer>
</customers>
<invoices>
... Invoice data here
</invoices>
</docroot>
There are multiple data representations in this single document such as customer, invoices and even the jobinfo and errors XML fragments. Note that you can only have a single root element (<docroot> in this case), but you can nest multiple items on the second level and down. The XML fragments may be totally unrelated to each other or they may all be related – it's entirely up to your implementation.
XML can also represent hierarchical data. Hierarchical data is extremely useful for packaging related data in a logical fashion that is easy to read and group without relational concepts. Instead of representing say an invoice as a set of related tables, you can actually have an invoice XML fragment, which nests inside of it the invoice header, the customer information, and a set of lineitems:
<?xml version="1.0"?>
<invoices>
<invoice>
<invno>IN_122121</invno>
<date>01/01/99</date>
<terms>Net 30</terms>
<total>123.12</total>
<customer>
<lastname>Strahl</lastname>
<firstname>Rick</firstname>
<address>32 Kaiea</address>
...
</customer>
<lineitems>
<item>
<sku>LABOR_PRG</sku>
<descript>Programming Labor</descript>
<qty>5.5</qty>
<price>150.00</price>
</item>
<item>
<sku>LABOR_PRG</sku>
<descript>Programming Labor</descript>
<qty>3.5</qty>
<price>200.00</price>
</item>
</lineitems>
</invoice>
<invoice>
<invno>IN_122122</invno>
...
</invoice>
</invoices>
XML provides flexibility and the ability to do things in a single pass that may otherwise require multiple passes. With XML I'm able to send multiple files at once where with an encoded file I'd have to make multiple requests to the server. Also, what happens if there's a problem? With file encoding there's no standard form to report errors. With XML error handling will be immediately obvious with an error header part of the document. You wouldn't have to explain the error format to anybody – one look at the result XML would tell the story.
All of this provides for a lot of flexibility in how the data is packaged for using XML in messaging between
applications either locally or remotely over the Internet. The cool thing is that you as the developer can determine the level of complexity you want to implement. You can go the simple route and simply dump data into XML or you can build a whole framework that deals with error handling and processing instructions implemented through special XML fragments that are part of the complete XML document as I hinted at in the first XML example. You can use the parser and build complex hierarchical objects or you can use plain application development code to generate the XML as a string yourself or the wwXML generating class I'll describe shortly.
Version Independence
If you use XML you're also not relying on a specific binary mechanism like COM, which requires a binary contract between the client and server. You're not bound to a property interface with XML – the data structures themselves can change by way of the XML structure without affecting the binary data representation. The XML may map to a binary object eventually, but there's an intermediary layer that pulls the data that knows what to do with it and can choose to ignore any enhancements a newer version might have added to the data.
This makes for easier maintenance as you're not relying on binary binding and recompiling your code to make changes to data structures. It also makes it a snap to add new functionality without breaking compatibility with existing clients. Old and new functionality can coexist with the same data structures without breaking either version of the app. Older clients simply don't include the new data and your application can handle this accordingly while newer client can use the additional data as needed.
Object Persistance
XML can also provide a good way to present persistent data from an abstract element such as an object that exists in memory. XML's hierarchical structure actually makes a very good fit for mapping complex nested objects into XML structures. Typical real world objects tend to be hierarchical and a good programmatic implementation of these objects can closely map those relationships. The Invoice example above is a simple example. You'd have an oInvoice object with sub objects for oCustomer, oLineItems and so on.
These types of objects are great for passing data around locally, but when the times comes to persist this data things get more tricky. XML can provide a great way to take a snapshot of the object and store the contents into persistable form – a file or a database typically. Or you can send the result and send it over the Web essentially marshalling an object from a client to the server.
This has a large number of applications: Transferring data over the Web is an obvious choice. For example, a client application can build a purchase order object and send that object persisted as XML to the server. The Server validates the PO and then either saves it into the database (unwinding the hierarchical relationship into a relational DBMS structure) or sends back a note that says the PO could not be accepted.
Another extremely useful use of persisted objects in XML is Server Session state. Servers, especially Web servers should be stateless, but most Web applications need to track state for users using a site. Persisting real binary (COM) objects across multiple requests is problematic in terms of resources and scalability and multiple machines. But persisting state as strings in a database table via XML is fairly lightweight and flexible. As long as the data can be written somewhere persistent it can be easily retrieved
using XML.
For example you could have a COM object that can accept an XML input parameter, and generate XML output. By doing so, you can use that same object on a Web backend to process incoming Web requests directly over the Web using exactly the same code base. This is one of the biggest points for XML in my experience – the ability to apply logic directly without change in multiple layers of applications, just by using a common XML interface to pass messages around. Once such a mechanism is in place it doesn't matter much whether you use COM, HTTP or any other client to access your code, either directly or indirectly over the Internet.
Some problems with XML
But XML also has a few downsides that you need to be aware of:
· XML conversion can be very resource intensive especially with large data sets
· The current XML parsers are fairly slow especially COM based access in high level languages such as Visual FoxPro, VB and VBScript.
· XML documents can bloat data considerably with all of the markup tags. When sending data over the wire this can be a problem if data is not compressed
· XML has issues with binary data
The downsides are not so frequently discussed and as well known as the pros of XML. In short, XML conversion is very resource intensive both in terms of CPU resources to create and unpack XML into native data as well the actual data size. In high performance applications – especially those that need to provide lots of data in single transactions - this can be a major hurdle for XML implementation.
Even if you do build Internet applications XML may not always be the right choice. For example, if you have a client/server application that uses the Web as the network XML may seem obviously very useful. But if the app is fully under your control on both client and server and you use say Visual FoxPro on both sides, there's little to gain by passing messages back and forth via XML. Why convert a table to XML on the server, only to convert the XML back to a cursor – it may be much more efficient to package up the table in a more native format like in the Distributed HTTP applications article (http://www.west-wind.com/articles.asp) using EncodeDBF/DecodeDBF. The original code for exporting is about twice as fast than XML conversion and a whopping 5 times faster for importing. It's also much faster than marshalling ADO recordsets using RDS. There's a fine line where this rule applies – efficiency over possible functionality.
Size is also an issue for XML. The start and end tags cause significant bloat of the data being sent over the wire. Additionally binary formats like numerics must be converted to human readable string to be embedded in an XML document. In many cases this string is larger than the original data format. On the other hand XML can strip out trailing spaces that are embedded in string fields, thus saving some bytes. XML data size can be significantly reduced by using compression of some sort, since much of the data is highly repetitive (start and end tags).
But remember, performance isn't everything! XML is a great tool when data needs to be shared with disparate clients, especially if those clients are Internet based. By exposing data outputs and even inputs via XML you can possibly open up an application to other platforms since XML can be used with every one of them.
The point I'm making here is simple: Don't use XML unless you have a good reason to use it. In many full-on, high performance environments you may find that XML will only slow you down without providing any real benefits. If you're in full control of your application and control data flow on both ends, XML may be overkill. At the same time think about the benefits that XML can provide especially when you want to build open applications that can be accessed from various platforms and dev tools. XML is the best way to do this in a consistent fashion and I'll show you some examples that demonstrate how a framework can take advantage of this using standard functionality that you'll want to implement into any object architecture you might want to expose to this approach.
As with most technologies there are serious pros and cons when using XML in your applications. I think the pros heavily outweigh the cons, especially if applications are designed to bring back small amounts of data at a time so that the XML conversion doesn't impact the process significantly. Take the time to make sure you understand what impact use of XML has on your application, both in terms of functionality that it provides your app and the performance implications it imposes on it.
Now I want to show a few practical examples of how you can take advantage of XML with minimal effort in your own applications today. In the process, I introduce a free tool for Visual FoxPro that you can use to create XML from your data and objects, and pass that XML data between client and server applications over the Internet — with literally just a few lines of code.
Building tools to work with XML
The true power of XML really doesn’t shine until you put it to work. To take advantage of XML, you can handcraft XML from a database table or data contained in objects, simply by building XML strings on the fly via code or by using an XMLDOM parser to generate a full XML document object.
If you’re running in a COM-based environment, the parser of choice is the Microsoft XMLDOM parser, which ships with Internet Explorer 5.0 and later. The XMLDOM parser is XML 1.0 compliant and relatively easy to work with. XML parsers are meant to have a standard feature set, so once you get familiar with a parser you should be able to use it in different languages and environments. The XML parser you use inside VFP has the same syntax as the one you use in Internet Explorer (it’s the same object after all). Within a Java application, you might use the same syntax in an XMLDOM implementation from a non-Microsoft source (such as IBM, Oracle or Sun). As all these parsers are XML 1.0 compliant the object syntax for all of them is, for the most part (except for vendor specific extensions), the same.
I'm not going to discuss in detail how the parser syntax works. If you haven't done so already, I suggest you pick up a text on XML (XML in Action by William Pardi, Microsoft Press, is a good one). If you’re interested in how to parse XML via Visual FoxPro code, you can look at the source code accompanying this article in the wwXML class.
Parsing XML with the parser is not terribly tricky, but it's definitely tedious. Even worse, it's a repetitive process. You're going to parse data to and from data tables and objects, over and over again. To help you with this process in Visual FoxPro, I’ve provided a free class called wwXML. wwXML handles the following:
- Converting VFP cursors to XML
- Converting any ODBC/OleDB datasource resultset to XML, including stored procedures and multiple result sets
- Converting cursors into ADO-compatible XML that can be loaded directly by ADO recordsets
- Converting XML to cursors Converting objects to XML (including subobjects)
- Converting XML to objects (including subobjects)
- Downloading XML from, and POSTing XML to a Web server
The wwXML class provides simple, single-call methods to perform these tasks, as well as low-level methods. The low-level methods can work on partial XML documents or XML fragments, to allow accessing XML that doesn’t match wwXML’s specific XML document structures.
Generic tools let you avoid having to write repetitive XML parsing code. wwXML provides imports and exports for Visual FoxPro tables and objects without writing any XML code.
For example, CursorToXML exports a cursor to a fully self-contained XML document with XML header and root node, but CreateCursorXML (a 'low level' method) creates only the inner XML fragment representing the cursor. The result of the latter can be strung together with additional XML fragments, or used as a subgroup in a hierarchical layout to build complex XML documents. Likewise, the parsing routines can be pointed at particular nodes in an XMLDOM document, to start parsing from a certain node rather than dealing with the entire XML document.
Here are a couple of simple examples that demonstrate how wwXML works. The first is a Web server-side piece of code that generates XML from a SQL SELECT statement, and then sends this XML back to a client over the Web. I'm using West Wind Web Connection in this scenario, but you can easily use a VFP component with ASP or FoxISAPI, with minor adjustments. Figure 1 shows what the result looks like viewed in raw form in Internet Explorer 5 (other browsers may not display the XML as XML text — you may have to View Source to see the XML).
FUNCTION XMLCursor
SET CLASSLIB TO wwXML ADDITIVE
SET PROCEDURE TO wwUtils ADDITIVE
lcCompany = Request.QueryString("Company")
oXML = CreateObject("wwXML")
oXML.cDocRootName = "WebConnection"
oXML.lCreateDataStructure = .T. && Add a DTD
*** Run a query to render
SELECT * FROM TT_CUST ;
WHERE Company = lcCompany ;
INTO CURSOR TQuery
*** Generate the XML to a string
lcXML = oXML.CursorToXML("customers","customer")
IF oXML.lError
THIS.ErrorMsg("XML Conversion Error",oXML.cErrorMsg)
RETURN
ENDIF
*** Must create an XML header for browser to know
*** to treat this document as XML
Response.ContentType = "text/xml"
Response.Write(lcXML)
To access this document, I can type the following URL into the browser (this is a live link, so you can try it yourself):
http://www.west-wind.com/wconnect/xmlcursor.wwd?Company=A
I can provide a single parameter to my query by adding a Company= key on the query string to limit the companies that are displayed as a query parameter in this case all that start with an A which is picked up on the server using the Request.QueryString(“Company”) code.
When XML output is generated, note that you should set the content type of the document to "text/xml" as shown in figure 1. This triggers Internet Explorer 5's default XSL stylesheet that lets you see the XML as a dynamic document, in which you can expand/collapse the various hierarchical levels of the XML document. IE5 is currently the only browser that supports native viewing of XML content.
Figure 1 – XML content generated on the server with wwXML. Note that the XML contains attributes that describe the structure of the data, which is defined in the DTD of the document.
The generated XML document contains structure information about the data, in the form of XML attributes, because I set the lCreateDataStructure property. When that property is set to .T., wwXML automatically generates a DTD (Document Type Definition) for the table that is exported. You can view the DTD by clicking View Source in the browser. At the top of the XML document you see:
<?xml version="1.0"?>
<!DOCTYPE WebConnection [
<!ELEMENT WebConnection (customers)>
<!ELEMENT customers (customer)*>
<!ELEMENT customer (custno,company,careof,address,shipaddr,email,phone,billrate,btype)>
<!ELEMENT custno (#PCDATA)>
<!ATTLIST custno
type CDATA #FIXED "string"
size CDATA #FIXED "8"
>
... content left out for brevity
<!ATTLIST address
type CDATA #FIXED "bin.hex"
size CDATA #FIXED "4"
>
... content left out for brevity
<!ATTLIST billrate
type CDATA #FIXED "number"
size CDATA #FIXED "7"
precision CDATA #FIXED "2"
>
]>
The DTD describes the data format and thus allows the client application to dynamically recreate the data source by querying the type, size and precision attributes (note: although these are XML compliant types, they are not part of an XML data standard). The client application, whether or not it's a VFP app, can look at the first record of the data to retrieve the format for all of the fields in the table, then actually create that table on the fly.
If you're using wwXML, the DTD is used to create the cursor for you automatically. However, the client could also be a Java application that reads the XML, creates a new table via JDBC, and updates the new table with the data received. Or, it could be a VB app, using ADO to do the same. In these cases, the client applications would be responsible for parsing the XML above into the tables (or other structures) manually, or using a local implementation of the functionality that wwXML provides to Visual FoxPro.
The current version of wwXML uses a custom DTD that is based on an early published third-party XML data specification. However, DTDs are on the way out, and a future version of wwXML will include support for creating an XML-DATA compliant schema. Schema output is not implemented yet, due to the inability to embed a schema directly into an XML document — this feature is slated for the next rev of the MSXML parser.
Schemas versus DTDs
The purpose of DTDs and schemas both is to provide structure information about the XML document and a quick way to check document validity without having to parse into the document first with manual checks. They also provide the ability to add custom attributes with default values and data typing, which is crucial to provide information about the data types in an XML document to client applications.
DTDs are part of the XML 1.0 specification and widely supported by XML parsers. Unfortunately DTDs are not XML based, don't support namespaces (required to avoid naming conflicts) and they are not extensible. Schemas address all of these issues with an XML based implementation that provides all the features of DTDs with all the extensibility of XML, but they are not officially part of the XML standard yet. However, MSXML 1.0 has basic support for external schemas. The latest version (version 3 is in late beta) also adds support for embedded schemas, which are vital to dynamic XML data generated on the fly as in this article.
wwXML can also output the same data to ADO-compatible XML. ADO XML is useful if you plan to share data with another Windows application that uses ADO for its internal data access. To grab the same data in ADO format:
http://www.west-wind.com/XMLCursor.wwd?display=ADO
On the server, only a couple of lines of code are changed to accommodate this behavior. First, we check for the Display QueryString variable, then we call the appropriate wwXML method to convert the data:
llADO = (Request.QueryString("Display") = "ADO")
and
IF !llADO
lcXML = loXML.CursorToXML("customers","customer")
ELSE
lcXML = loXML.CursorToADOXML()
ENDIF
ADO-compatible XML is entirely attribute based, which is rather messy to look at and causes extra parsing overhead. However, because ADO recordsets can directly load this XML, it's ideal for ADO clients, since using rs.Open() to read in the XML is very fast. For generic XML applications that support clients other than ADO, I would recommend that you stay away from this non-standard ADO XML implementation.
Pulling XML from the Web
Once you’ve generated XML on the server, the next step is to retrieve the XML in a client-side application. Pulling and posting data from the Internet is a very powerful concept, and is the cornerstone for building distributed applications, whether you run a Visual FoxPro application as the client or use a browser like Internet Explorer. The good news is that this task is easy, with a number of tools available to perform the HTTP access.
For Visual FoxPro client applications, the wwXML class includes a LoadURL method, which can be used to return any HTTP content directly into a string with a single line of code. Once you have the string, you can then use the XMLToCursor method to convert the string into a VFP cursor:
owwXML = CREATE("wwXML")
lcXML = owwXML.LoadUrl(;
[https://www.west-wind.com/XMLCursor.wwd?Company=A])
owwXML.XMLToCursor(lcXML,"TCursor") && Create Cursor TCursor
Pretty easy, huh? LoadUrl can download any HTTP data, not just XML. It also allows you to post data to the server, as we’ll see a little later. XMLToCursor() then goes to work on the XML document object, figuring out the document structure and creating a cursor TCursor (the second parameter) on the fly. If the cursor doesn’t exist, as is the case here, the cursor is created based on the DTD that accompanies the XML document. If a cursor named TCursor exists already, the data is appended instead. In order for a cursor to be created on the fly, a DTD is required, and it must follow the format described earlier. If no such DTD exists, you have to provide the cursor yourself, either by creating it in your code, or providing a table from your application.
The download in wwXML uses WinInet system API functions from within Visual Foxpro code, which is lightweight and allows good control over all aspects of the HTTP process. It’s based on the wwHTTP class, which is provided as part of the wwXML class. wwHTTP provides access to all of HTTP’s functionality, including security, posting, and custom content types.
If you’d rather use a COM component, or you want to use the parser manually to process the XML from the site, you also can use the XMLDOM parser to download an XML document object:
oxml = CREATEOBJECT('Microsoft.XMLDOM')
oXML.async = .f.
oXML.load("http://www.west-wind.com/XMLCursor.wwd?Company=A")
*** Must check document for validity
IF !EMPTY(oXML.ParseError.Reason)
RETURN .F.
ENDIF
owwXML = CREATEOBJECT("wwXML")
owwXML.XMLToCursor(oXML,"TEMPFILE")
Here, the XMLDOM’s Load method is used to download the data, yielding an XMLDOM object, which can also be passed to the XMLToCursor method. The Load method downloads the XML, and immediately tries to load it into the MSXML parser as an XML document. Load works well for simple things, but it doesn’t support POSTing data to the server, doesn't refresh if the URL is requeried (chaching). Nor can it deal with Basic Authentication security to login users where security is required.
Microsoft also provides the XMLHTTP component as part of the XMLDOM parser COM object. It allows you to download and POST data as well as providing hooks for password values.
oHTTP = CREATEOBJECT('Microsoft.XMLHTTP')
oHTTP.Open("POST","http://www.west-wind.com/XMLCursor.wwd?Company=A",.F.)
oHTTP.Send( lcXMLToPost )
lcRawResponse = oHTTP.ResponseText && Text
loDOM = oHTTP.ResponseXML && XMLDOM object
Notice that you can pull data down directly into an XMLDOM object, but you want to make sure you check for errors in the DOM document if you do go that route. XMLHTTP has another downside - it's COM based and while it should work fine for XML string it doesn't support binary data - it'll truncate at NULL characters. In addition the data returned from XMLHTTP and directly loaded into the parser must be properly encoded in UTF-8 or whatever the format specified in the XML document. If your code on the server generates XML as a string, make sure that all of the XML encoding rules for persisted (ie. file based) XML are respected.
Manual Parsing
Of course, using wwXML is optional for parsing XML. You can also manually parse the XML code using the XMLDOM parser and use it as you see fit. The following code walks through all the records of the table and echoes back all fields to the VFP console generically:
oxml = CREATEOBJECT('Microsoft.XMLDOM')
oXML.async = .f.
oXML.load("http://www.west-wind.com/XMLCursor.wwd?Company=A")
loRows = oXML.SelectNodes("/WebConnection/customers/customer")
FOR EACH loRow IN loRows
FOR EACH loField IN loRow.ChildNodes
? loField.Text
ENDFOR
? "---------"
ENDFOR
In this case, I’m relying on the actual structure of the document by knowing beforehand the names of the document, table and rows for use with the XSL method SelectNodes. It gets a little more confusing if you want to generically access nodes:
MSXML Bug
Microsoft shipped a bugged version of MSXML with the release version of Windows 2000 and Internet Explorer 5.01. This bug causes Incorrect Function errors (OLE Error 1) in Visual Foxpro to occur whenever an invalid node is accessed in an XML document. It also causes the Load and LoadXML methods to fail if the XML document is not valid. This bug has been acknowledged and fixed by Microsoft, and can be taken care of by installing a patch file.
wwXML uses the MSXML parser for importing data, and unless you run this patch, wwXML will fail upon importing any XML. wwXML includes a method that lets you check for the bug.
For more information, please visit: http://www.west-wind.com/webconnection/msxml.asp.
loRows = oXML.DocumentElement.ChildNodes(0).ChildNodes
Manual XML parsing is not terribly complex, but it is tedious and repetitive. It takes a trip to the XML reference before getting it right the first few times, and even now I have to look at some working code for reminders, and to get things working quickly. The main issue with this type of code is that it’s repetitive and potentially error-prone. Using a tool like wwXML to handle things generically is much cleaner. I encourage you to use or build (if you’re not using VFP) tools that wrap this type of behavior, to minimize XML hand-coding.
Objects make the world go ‘round
When we build applications, business objects that contain the structure and business logic of our application are key in making things work properly. They abstract and isolate this logic into a central structure that is easy to maintain and administer. Because objects often describe a particular piece of information about an application, they also make great containers for data, depending on how the business objects are structured.
For example, on my Web site most business objects include member objects that hold the actual data. An inventory object for my Web Store application has an oData member that contains the actual item information — in this case, simply the fields of the inventory item record added to the Item object with SCATTER NAME loItem.oData MEMO. This works for local data pulled from Visual FoxPro tables, or for data pulled from a SQL backend like SQL Server or Oracle via ODBC or ADO/OleDb. Once loaded, to get access to the data, I can use the following code:
loItem = CREATEOBJECT("cItem")
loItem.Load( 1241 ) && Customer PK
? loItem.oData.Sku
? loItem.oData.Descript
Methods on the object can then be used to perform operations on this business object. If I make a change to the item description and save the change, I’d use:
loItem.oData.Descript = "West Wind Web Connection 3.20"
loItem.Save()
Since this object is self-contained and has its own logic to deal with typical tasks of loading and saving, it’s very easy to use, and can be passed around as a message inside the local application, or even in COM method calls across application boundaries.
Wouldn’t it be nice if you could also pass an object like this around over the Web? The answer to doing this lies in persisting the object member data into XML, and then passing this XML over the wire. wwXML supports this functionality using the ObjectToXML method. On my Web store site most URLs can be accessed by adding a special flag to the HTML display URLs. For example, to display item data for a product on the Web site I can use the following URL:
http://www.west-wind.com/wwstore/item.wws?sku=WCONNECT
Figure 2 – Item object rendered as HTML using standard templates with ASP syntax
This gives me an HTML page. The Web application renders the data using a cItem object instance referenced in template tags (like <%= oItem.oData.Descript %> for example) that are expanded on the HTML page. I can also get that same information in XML format by adding &display=XML to the end of the URL:
http://www.west-wind.com/wwstore/item.wws?sku=WCONNECT&display=XML
Figure 3 – The same URL with the XML parameter to render the object as an XML document
Now, when the data is returned, the HTML template is not called, and the data is generated using wwXML's ObjectToXML method instead:
FUNCTION Item
LOCAL loItem
PRIVATE oItem, pcSKU
loItem = CREATE([cItem])
pcSKU = UPPER(Request.QueryString("Sku"))
IF EMPTY(pcSKU) OR ;
!loItem.GetItemBySku(pcSku)
THIS.ErrorMsg("Invalid Item")
RETURN
ENDIF
oItem = loItem.oData
IF THIS.nResultMode = 2
loXML = CREATE("wwXML")
loXML.cDocRootName = "wwstore"
lcXML = loXML.ObjectToXML(oItem,"item")
Response.ContentTypeHeader("text/xml")
Response.Write(lcXML)
RETURN
ENDIF
Response.ExpandTemplate( Request.GetPhysicalPath() )
Everything runs through the cItem business object. The decision to generate XML is set in the nResultMode flag, and is checked in the startup of the of the Process object, which handles the Web request (in the Process method):
*** Support XML retrieval of data on select request
THIS.nResultMode = IIF(Request.QueryString("Display") = "XML",2,0)
nResultmode is then forwarded to all request methods. If it’s set to 2, wwXML is used to create XML from the object. Otherwise, HTML is requested and the ExpandTemplate method of the Response object is used to evaluate the HTML template containing the <% %> script tags. If you used ASP instead of Web Connection, this business logic, as well as the check for the display mode, would be embedded inside the ASP page or a COM component that returns the result either in HTML or XML formats.
The wwXML syntax is very simple: You create the object and call the ObjectToXML method. The XML generated is shown in Figure 3. Note that we decided to return only the oData member (oItem = loItem.oData) rather than the entire item object, since the main object's (the generic business object properties) data is not appropriate here.
If I want to retrieve this data from a VFP client application that has this item object available, I can now run the following:
loXML=CREATE("wwXML")
lcXML=loXML.LoadUrl("http://…/item.wws?sku=wconnect&display=XML")
loItem = CREATE("cItem")
loItem.GetBlankRecord() && Init oData member
loXML.XMLToObject(lcXML,loItem.oData)
? loItem.oData.Descript
? loItem.oData.Price
Note that I pass loItem.oData as a second parameter to XMLToObject, as the object that is to be populated with the data from the XML. The passed object is used as a template for pulling the data from the XML, by walking through the object properties and looking for matching elements in the XML document. You can also optionally have the server side generate a DTD and let the client side assemble a generic container object that only contains the object's properties. This may be useful in some scenarios, but most commonly an object that receives the incoming data will likely exist on the client application.
Pretty slick isn’t it? With a handful of lines of code, you can persist an object from the server down to a client very, very easily! Note also that the XML is version-independent — the client application determines which data is used, so that if the server decides to add a new fields it won't break our client application. We've essentially decoupled the client and server through the intermediary XML. You can swap either end of the transaction with a different client or server and the other side would never know the difference!
Complex objects and hierarchical XML
If I take object persistence a step further, I can have more complex objects that combine multiple objects together. For example, this same Web Store application has an invoice object that has its own oData member containing the invoice header information, and also contains an oCustomer and oLineItems object for the detail info. When I load an invoice object, I can now access each of the subobjects through the invoice object, since it internally manages the subobjects as part of the load operation:
loInv = CREATEOBJECT("cInvoice")
loInv.Load( 100 ) && Invoice PK
? loInv.oData.InvDate && Invoice Data
? loInv.oData.InvTotal
? loInv.oCustomer.oData.LastName && Customer Data? loInv.oCustomer.oData.Company
? loInv.oLineItems.aRows[1].SKU && LineItem Data
? loInv.oLineItems.aRows[1].Description
? loInv.oLineItems.aRows[1].ItemTotal
The Invoice object has its own oData member, as does the oCustomer member object. loInv is simply a container that holds this object. The oLineItems member contains all of the lineitem detail in an object that has a member array of all the lineitem objects containing the actual lineitem data, which again simply maps to fields in the lineitems table.
Exporting an object into XML on the server, then downloading the XML and reassembling it on the client, becomes a matter of only a few lines of code
This object is very portable! You can take the entire object and simply pass it as a reference to a method or function, and let that code access all of the properties. The calling application (in this case an online e-Commerce app and an offline invoice manager application using the same object) can simply use the business object methods such as Load, Save, and New to manage the object.
Because of this structured object approach, it’s also possible to use this object on a Web page in an HTML template. In my application, template pages use syntax like:
<%= oInv.oCustomer.oData.LastName %>
to embed the actual data into the HTML form as shown in Figure 4.
Figure 4 – The invoice business object displayed in HTML format. All the elements displayed on this HTML form are using template expansion with ASP syntax to render the business object properties.
On the server side, the invoice is rendered either in HTML format or in XML format, according to the QueryString display mechanism discussed earlier. There’s one difference, though: This object is considerably more complex and requires a hierarchical XML document. To generate the XML on the server the following code is used:
lnPK = VAL( Request.QueryString("PK") )
IF lnPK = 0
THIS.ErrorMsg("Invalid PK")
RETURN
ENDIF
loInv = CREATE("cInvoice")
IF !loInv.Load(lnPK)
THIS.ErrorMsg("Invalid Invoice")
RETURN
ENDIF
loXML = CREATE("wwXML")
loXML.lRecurseObjects = .T.
loXML.cDocRootName = "wwstore"
lcXML = loXML.ObjectToXML(loInv,"invoice")
Response.ContentTypeHeader("text/xml")
Response.Write(lcXML)
Note the lRecurseObjects property, which is used to make sure that all the surrogate objects are also created into the XML:
<?xml version="1.0"?>
<wwstore>
<invoice>
<calias>wws_invoice</calias>
<cerrormsg></cerrormsg>
<cfilename>wws_invoice</cfilename>
…
<ocustomer>
<calias>wws_customers</calias>
<cerrormsg></cerrormsg>
<cfilename>wws_customers</cfilename>
…
<odata>
<address>32 Kaiea Place</address>
<city>Paia</city>
<company>West Wind Technologies</company>
<countryid>US</countryid>
<email>rstrahl@west-wind.com</email>
<entered>1999-11-19</entered>
…
<zip>96779</zip>
</odata>
</ocustomer>
<odata>
<cc>4123 1234 1234 1231</cc>
<ccexp>08/2002</ccexp>
<cctype>VI</cctype>
…
<tax>29.96</tax>
<weight>2</weight>
</odata>
<olineitems>
<ncount>2</ncount>
<arows>
<arows_item>
<descript>Web Connection</descript>
<invpk>110811</invpk>
<pk>106074</pk>
<price>699</price>
<qty>1</qty>
<sku>WCONNECT</sku>
<weight>0</weight>
</arows_item>
<arows_item>
<descript>HTML Help Builder</descript>
...
</arows_item>
</arows>
</olineitems>
</invoice>
</wwstore>
This time around, the object is generated with all of the base properties of the business objects — we get properties that are not required to be transferred over the wire, such as the data filename, the Alias, the error message property, and so on. These are base business object properties, which although superfluous, don’t hurt anything (other than making the result a little bigger). Incidentally, you can hide these properties during export by using the wwXML::cPropertyExclusions property.
Take a moment to think about how powerful this is: With a couple of lines of code you’ve generically persisted a rather complex and nested business object to XML! You can now transfer this object over the wire and — you guessed it — reassemble it on the client side just as easily. In fact, the Web store application includes an offline invoice manager, which retrieves orders placed online. The offline application can then be used to process credit card transactions (also using HTTP and XML) and send order confirmations to customers.
Figure 5 – The offline Invoice Manager uses the same business objects used on the server, and receives invoices via invoice objects sent over the wire as XML. Fields in this form are bound to the business object, such as THISFORM.oInv.oData.InvDate and THISFORM.oInv.oCustomer.oData.Company.
The offline manager has a download link, which runs in a loop to download any unprocessed invoices from the server. The relevant import code for an individual invoice looks like this:
loXML = CREATE("wwXML")
lcXML = loXML.LoadUrl("https://www.west-wind.com/wwstore/admin/MaintShowInvoices.wws?Action=Show&Display=XML&PK=110811",;
,"username","password")
loInv = CREATE("cInvoice")
loInv.Load(0) && Load empty invoice & subitems
loXML.lRecurseObjects = .T.
loXML.XMLToObject(lcXML,loInv)
? loInv.oCustomer.oData.LastName
? loInv.oCustomer.oData.Company
? loInv.oData.InvDate
? loInv.oData.InvTotal
? loInv.oLineItems.ARows[1].Sku
? loInv.oLineItems.ARows[1].Descript
Again, notice how little code is involved here — 4 lines of code to download and import the object. However, in order for the import of this hierarchical object to work, a couple of pre-requisites are required. The objects must exist on the client side in order for wwXML to be able to figure out the structure of the object to import to — hierarchical objects cannot be dynamically created from an export DTD. Furthermore, the object structure must be prefilled with empty values of the proper types in order for the import to work.
Member objects imported to cannot be NULL or otherwise undefined, as that would prevent them from being parsed and filled. wwXML requires an object structure, as it works by walking through the object and looking for matching elements in the XML document. In the case of the LineItem array, at least the first item of the array must be prefilled with the proper type — in this case, an item object.
Notice the special call to loInv.Load(0). This method call actually handles creating the properly formatted ‘empty’ objects. If we look at the business object code’s Load method we’ll see the following:
IF lnPK = 0
THIS.GetBlankRecord()
THIS.oCustomer = CREATE("cCustomer")
THIS.oCustomer.GetBlankRecord()
*** Create an empty item list
THIS.oLineItems = CREATE("cItemList")
loLItem = CREATE("cLineItems")
loLItem.GetBlankRecord()
THIS.oLineItems.aRows[1] = loLItem.oData
RETURN .F.
ENDIF
Each object is pre-initialized as an empty object. This type of behavior is not only useful for wwXML’s import parsing, but also for displaying the business objects in forms. For example, in the invoice manager form the actual field ControlSource values are bound to the object’s properties. NULL values would cause all sorts of problems for the databinding.
The EPS folks in Houston use the Invoice Manager to enter orders at their offices and post those invoices to the West Wind Web Server located in Oregon. West Wind can then download these orders in Hawaii like any other order placed online. This is a good example of the benefit of a distributed application in a small business environment!
POSTing data to the server
So far, I’ve shown how you can get data downloaded from the server and use it on the client side. You can also go the other way! Let’s look at the offline manager application, which is actually in use by several people in our organization. West Wind Technologies is associated with EPS Software, and EPS Software may take orders at their offices in Houston. However, West Wind manages all product sales, so all credit card transactions are processed by West Wind in Hawaii. The idea is that the folks in Houston can use the Invoice Manager to enter orders at their offices, and post those invoices to the West Wind Web Server located in Oregon. West Wind can then download the orders posted to the Web server, in the same way as online orders placed directly on the Web site. The concept could be extended to allow third party vendors to also post orders in XML format directly to the West Wind Web site. This is a good example of the benefit of a distributed application in a small business environment!
We already know how to create the invoice XML easily — the process in the Invoice Manager client application is identical to what's happening on the server. What’s different now is that we’re sending data to the server:
* frmInvoice :: UploadInvoice
LPARAMETER lnPk
LOCAL loXML, loIP, lcResult, loInv
IF !EMPTY(lnPk)
loInv = CREATE("cInvoice")
IF !loInv.Load(lnPK)
RETURN .F.
ENDIF
ELSE
loInv = THISFORM.oInv
ENDIF
*** Export invoice to XML hierarchically
loXMl = CREATE("wwXML")
loXMl.lRecurseObjects = .T.
lcXML = loXMl.ObjectToXML(loInv)
lcResult = loXML.LoadUrl(THIS.oConfig.cUploadINvoiceWebLink,;
lcXML,;
THIS.oConfig.cAdminUsername, ;
THIS.oConfig.cAdminPassword)
*** check for errors
IF lcResult # "OK"
IF loXML.lError
MESSAGEBOX(loXML.cErrorMsg,48,WWSTORE_APPNAME)
ELSE
MESSAGEBOX(lcResult,48,WWSTORE_APPNAME)
ENDIF
ELSE
MESSAGEBOX("Invoice uploaded.",64,WWSTORE_APPNAME)
ENDIF
The only new thing that happens here is that when we call wwXML::LoadUrl(), we’re passing the XML generated from the invoice export as the second lcXML parameter, which is POSTed to the Web server. LoadURL() posts data in raw format and doesn’t URLEncode the string. The server knows how to pick up this raw XML buffer. This simple bit of code handles posting an invoice from the client to the server. On the server, the code is straightforward:
FUNCTION SubmitXMLOrder
*** Security Check
IF !THIS.Login(Config.cAdminUser)
RETURN
ENDIF
*** Retrieve XML input if passed
lcXML = Request.FormXML()
*** Create an Invoice object and initialize
loInv = CREATE([cInvoice])
loInv.Load(0)
*** Import the object
loXML = CREATE("wwXML")
loXML.lRecurseObjects = .T.
loXML.XMLToObject(lcXML,loInv)
*** Check if data is Ok
IF !(loInv.oData.Pk # 0 AND loInv.oCustomer.oData.pk # 0)
Response.Write( "Error: Invalid Order Info")
RETURN
ENDIF
*** Must fix up invoice for PK conflicts
loInv.CheckInvoiceForPkConflict()
loInv.Save()
Response.Write("OK")
Keep in mind that these types of replication scenarios require thought about synchronization: You may run into duplication of primary keys. This invoice object includes a method that checks to see whether an invoice PK exists already, and if necessary, reassigns the PK for the invoice and updates the lineitems accordingly. For the customer, if a PK exists already, it double checks to see if the name and address match the original record. If it doesn’t, a new customer record is created instead. Conflict resolution is not always easy, as this type of online/offline application becomes an exercise in replication, along with all the issues related to this complex topic of synchronization of data.
More power to you!
I hope this article has given you some ideas on how you can utilize XML from Visual FoxPro client applications. Even if you’re not using Visual FoxPro, you can probably see that with the right tools for your development language of choice, you can perform powerful transfers of data with minimal coding effort. As the examples above show, you need only a few lines of code to handle convert tables and objects into XML, transfer that data over the wire, and reassemble the XML back into the appropriate data or object structure.
I’m sure that you can think of some useful applications for this stuff. Using these strategies, you can very easily build powerful applications with existing business objects, which can be transferred painlessly over the wire. The possibilities are endless.
And remember, XML can be useful for lots of things, even in non-distributed applications. By persisting objects and tables as XML, you can pass data around in applications, and even save object state into a table’s memo field.
In the next issue, we’ll look at how to build XML-based services that provide data access and remote code activation to a variety of clients generically. I'll talk more on how to access and consume XML in browser applications, as well as fat client applications, using the same application logic. We’ll also look at SOAP and how it figures in that space. Until then, give it a shot
Resources
Microsoft XML Web Site:
Using SOAP to call Remote Objects
http://www.west-wind.com/presentations/soap/
Building distributed Web Applications with Visual FoxPro
http://www.west-wind.com/presentations/wchttp.htm
msXML Bug update page
http://www.west-wind.com/webconnection/msxml.asp
wwXML Visual FoxPro XML
conversion class:
http://www.west-wind.com/wwxml.asp
wwIPStuff – Visual FoxPro HTTP
and other IP protocols class:
http://www.west-wind.com/wwipstuff.asp
http://www.west-wind.com/webconnection/
Rick Strahl is president of West Wind Technologies on Maui, Hawaii. The company specializes in Internet application development and tools focused on Internet Information Server, ISAPI, C++ and Visual FoxPro. Rick is author of West Wind Web Connection, a powerful and widely used Web application framework for Visual FoxPro, West Wind HTML Help Builder, co-author of Visual WebBuilder, a Microsoft Most Valuable Professional, and a frequent contributor to FoxPro magazines and books. His new book "Internet Applications with Visual FoxPro 6.0", was published April 1999 by Hentzenwerke Publishing. For more information please visit: http://www.west-wind.com/.
