West Wind Web Log

VFP String Concatenation Issue in Loops

Sunday, May 11, 2008, 6:13:00 AM

I got bit again by a nasty string concatenation problem in VFP. This isn't the first time I've hit this particular issue and I've known about it for years, but it's one that's easy to miss and so from time to time I hit it again. The problem occurs when you have string concatenation code that involves an expression that returns results from a function and where the function often returns a large amount of text.

For example, the following code - taken from Web Connection's wwWebRepeater control - fails:

   LOCAL lnX, lnCount

   lnCount = this.ChildControls.Count

   FOR lnX = 1 TO lnCount

       LOCAL loControl

       loControl = this.ChildControls.aItems(lnX,2)

       IF !ISNULL(loControl)

          lcOutput = lcOutput + loControl.Render()

       ENDIF

The code fails to properly build your concatenated string. The code in question is string concatenation where the code is appending to an existing string by appending an expression - typically a class method that is called to provide additional data. In each iteration of the loop lcOutput is set with the result of loControl.Render() rather than the concatenation of lcOutput + loControl.Render().

This makes for some interesting debugging - you step over the code with no problems but it takes a while to actually grok what the debugger is telling you - it's not the debugger truncating your string but VFP actually failing to concatenate.

This is an insidious bug in that sometimes the code works as expected and at other times it fails with creating the incorrect string format. The only way to really detect this problem is to step through with the debugger and see that the result is incorrect.

Luckily the workaround for this is fairly easy: you can create a variable and assign that to the concatenation expression. The following code produces the expected result:

   LOCAL lnX, lnCount, lcT

   lnCount = this.ChildControls.Count

   FOR lnX = 1 TO lnCount

       LOCAL loControl

       loControl = this.ChildControls.aItems(lnX,2)

       IF !ISNULL(loControl)

          lcT = loControl.Render()

          lcOutput = lcOutput + lcT

       ENDIF

   ENDFOR

This fixes the issue, but of course it results in the string being copied to a variable first, which depending on how much output is generated may be an expensive operation.

In Web Connection I had to use code like this in a lot of places and just last week I ran into this nasty 'bug' again while working with a client in a place where I missed the issue. Oddly, the problem occurred in a fairly high traffic area of the framework, so I'm surprised that nobody has actually hit this particular issue before. It's possible the behavior varies slightly with different versions of VFP 9 (ie. RTM, SP1, SP2) - I believe some of these behaviors were actually addressed in SP1 and later.

This 'behavior' was introduced with the string optimizations in VFP 7 which drastically improved string concatenation performance in VFP back then. When concatenating strings VFP will usually allocate more memory than it needs to concatenate the current string value to optimize string allocation. So rather than having to copy strings all the time for every concatenation, it allocates a bigger buffer and loads the content of the right had side of the concatenation directly into the end of the string buffer which improves string performance drastically.

I can only guess on why this doesn't always work with functions but my guess is that if the buffer size exceeds the pre allocated buffer that the string gets recreated and somewhere in that process the new string pointer for the result, rather than the concatenation is returned.

No matter what the problem is, it's a bug that you should be aware of since it's likely to be hit by a wide variety of code.

IIS 7 Default Request Filtering and Web Connection

FeedBack (0)

Sunday, April 06, 2008, 11:32:00 PM

IIS 7 has an extensive list of extensions and paths that it deems as restricted. This is generally a good thing as it blocks URL access to many common paths that are frequently used in Web applications to hold semi-private files like code and binary assemblies for example in ASP.NET applications.

As it turns out the default behavior also affects Web Connection because the default filtering completely disallows direct access to a BIN directory. In addition, IIS 7 blocks out access to many file extensions that you might previous have used for your own script mapes. For example, I just ran a demo and created a script map of .dd for my project only to find that it bombed with 404 everytime. It took some sleuthing to find out that .dd is a restricted extension and changing the extension immediately fixed the problem.

So what does it mean to your Web Connection Apps?

The biggest issue that you might run into with IIS 7 that if you have WC.DLL installed in the /Bin directory of your virtual or Web root, you cannot access the DLL directly. Urls like this:

/your Virtual/wc.dll?wwMaint~ShowStatus

Will fail to work and you'll get a 404 error like this:

HTTP Error 404.0 - Not Found

The resource you are looking for has been removed, had its name changed, or is temporarily unavailable.

Detailed Error Information

Module		IsapiFilterModule
Notification		MapPath
Handler		ISAPI-dll
Error Code		0x80070002
Requested URL	http://localhost:80/timetrakker/bin/wc.dll?wwMaint~ShowStatus
Physical Path	c:\westwind\TimeTrakker\bin\wc.dll
Logon Method	Not yet determined
Logon User	Not yet determined

Most likely causes:

· The directory or file specified does not exist on the Web server.

· The URL contains a typographical error.

· A custom filter or module, such as URLScan, restricts access to the file.

The restriction here lies in IIS 7's configuration for Request Filtering which can be found in ApplicationHost.config (in System32/inetsvr/config). In it you'll find a request filtering section:

<add fileExtension=".dd" allowed="false" />

</fileExtensions>

<add segment="bin " />

</hiddenSegments>

</requestFiltering>

The culprit for the direct WC.DLL execution is the hidden segment of bin filter in the hiddenSegments section. This filter basically prevents anything to be Web visible via URL that has a bin directory in its path.

<add segment="bin " />

If you absolutely need to run wc.dll directly and you don't or can't use scriptmaps – which I highly recommend anyway though – you can comment out this block



Which will then allow you to execute wc.dll out of the bin directory. NOTE: I would not advise this! It's a bad call to override these system settings because you'll have to remember to do it every time you install a new installation or move it.

Note that request filtering is a global setting – it must be set in ApplicationHost.config and cannot be delegated down to the web.config unless you override this setting:

And change the key to Allow.

Another option: Move the DLL out of the BIN directory into the root or another folder.

But I wouldn't recommend changing either of these options! The former mucks with default configuration settings that you have to remember to set each time the app gets reinstalled and the latter requires changing URLs anyway - and there's a better way to do that with scriptmaps.

So a better solution is to always use script maps. Create a script map or even use one of the default script maps that Web Connection installs into every installation (.WC, .WCSX are two of them) and replace every call to wc.dll with wc.wc and remove the /Bin path from the url. So

/myVirtual/bin/wc.dll?wwMaint~ShowStatus

Might become

/myVirtual/wc.wc?wwMaint~ShowStatus

In some situations this may cause pathing problems because if you used the DLL pages were pathed to the bin directory and relative links for image and other resources may have been relative to the bin folder.

But this is why we've tried for years to push our user to use script maps in the first place – script maps are much easier to manage both in terms of security as well as flexibilty.

Watch out for other blocked Extensions

When you create new projects and new script map extensions, you should be careful not to choose any blocked extensions.

For example when I tried to create an extension for .dd and then hit a page with this extension I got:

HTTP Error 404.7 - Not Found

The request filtering module is configured to deny the file extension.

Detailed Error Information

Module		RequestFilteringModule
Notification		BeginRequest
Handler		StaticFile
Error Code		0x00000000
Requested URL	http://localhost:80/timetrakker/default.dd
Physical Path	c:\westwind\TimeTrakker\default.dd
Logon Method	Not yet determined
Logon User	Not yet determined

Most likely causes:

· Request filtering is configured for the Web server and the file extension for this request is explicitly denied.

Things you can try:

· Verify the configuration/system.webServer/security/requestFiltering/fileExtensions settings in applicationhost.config and web.config.

Note that here the message points you right at the problem and where to look. It points right at the Request Filtering section in ApplicationHost.config. If you look back on the list of extensions you can see that .DD is indeed included in the list of restricted extensions.

Again the solution here is either to allow the extension or alternately choose a different extension.

This seems like a lot of new restrictions but I'd say these are a good thing. They are easy to fix or workaround as long as you know what the settings are. None of this is a problem for Web Connection applications that use script maps to begin with, so this is a reminder why script map formatting is the way to go with WWWC applications…

Detecting hung Objects in Visual FoxPro

FeedBack (1)

Monday, March 31, 2008, 2:44:00 PM

One of the big problems with Visual FoxPro is that the garbage collector makes it very difficult to completely release object references in some cases. It's well known that the VFP garbage collector will refuse to release objects if circular references exist – ie. You've passed a reference of one object to a second object which stores it on its own properties and when you then try to clear the second object, the object will appear to be null and cleared but in effect this object is still live in the background as its hanging on to the reference of the first object. Unless you first release the first object reference on the second, you'll end up with a hung reference that persists even after the object is 'released' in code by setting it to NULL or .F. You've effectively created a memory leak in your application.

In Web Connection and the Web Control Framework for example I use a rather large number of objects to define the page layout of an HTML page where each control is an object. These objects have interdependencies with properties like Parent and ChildControls that reference other controls and so there's a purposeful relationship that requires circular references.

This is a well known and fairly well understood problem and there are workaround for it, which basically involve having explicit code in place to release child references. In Web Connection WCF each object has an explicit Dispose method that is used to clean up itself and any child objects for example. Here's the core Dispose implementation on the base wwWebControl class:

************************************************************************

* wwWebControl :: Dispose

****************************************

FUNCTION Dispose()

LOCAL loCtl, lnX

*** Don't do if we were already here

IF THIS.lDisposeCalled

RETURN

ENDIF

this.lDisposeCalled = .t.

*** Explicitly release child controls

IF THIS.IsContainerControl AND !ISNULL(this.ChildControls)

FOR lnX = 1 TO this.ChildControls.Count

loCtl = this.ChildControls.aItems(lnX,2)

IF __DEBUGMODE && Public Var should be defined prior to using component

TRY

loCtl.Dispose() && This can be problematic during debug

CATCH

ENDTRY

ELSE

loCtl.Dispose() && This can be problematic during debug

ENDIF

loCtl = null

ENDFOR

this.ChildControls = null

ENDIF

*** Release all objects

THIS.Page = null

THIS.ParentControl = null

THIS.ViewState = null

THIS.PreservedProperties=null

*THIS.Context = null

THIS.Attributes = null

ENDFUNC

* wwWebControl :: Dispose

The method basically goes through and cleans up any objects that were explicitly declared and this method is explictly called from cleanup code of the page, which in turn drills into all child controls to clean up.

Note that Dispose() needs to be MANUALLY called from code – you can't rely on Destroy() to fire Dispose() for you. Therein lies the core of the FoxPro problem. Destroy for a class doesn't fire until all instances/references of that class have been released. If another reference exists anywhere else – even on an already 'destroyed' object the Destroy() method is not fired.

What this means is that in many situations code placed into the Destroy() method is not effective of actually release resources of a class. It also means that for applications like the Web Control Framework that purposefully use circular references (ie. Parent, Child Controls) have to explicitly build code – and error handling recovery – that can release resources. In the Web Control Framework there's a Dispose method in the top level wwWebPage class that is called as part of the page processing cycle. When Dispose() is fired on page it ends up calling dispose on all child controls in the entire page via the code above that recursively digs into all controls. If Dispose is called on all controls (and all controls clean up properly after themselves) then everything is fine.

The problem with this is that it requires a ton of discipline to do this right. You have to remember to implement your Dispose() methods and match any declarations. That's the easy part. The hard part – especially in desktop applications – is to figure out when an how to call the Dispose() method for each object. In desktop apps there's usually not a clean cleanup point, and instead you end up with events that create and cleanup objects so if there are interdepencies it's easy to end up with hung references.

Tip: How can you detect hung references?

So once you have hung references, how do you find them? Hung references by definition belong to objects that have been released from the variable stack, but that still are effectively live and holding references to other objects. IOW, there effectively invisible objects that you can't access via FoxPro code.

To detect a problem like this, one tool is to use the VFP SYS(1016) which returns memory usage for objects used in Visual Foxpro. If you run code and this memory usage keeps increasing even after you come back to a starting point you probably have a leak issue. Some fluctuation here is normal, but generally this level should stabilize once an application is running and most of the features have loaded. In Web apps you should definitely see this variable stay stable if everything releases properly between hits.

The next and harder question though is this: I know I have leak, where is it? There's no easy way to figure out which object is leaking. But there's a trick you can use to take a pretty good guess.

At the shutdown of your application – after all of your own application cleanup code has run – and most everything should have been cleaned up and released issue the following:

Login
Please enter a Username and Password.
Username:
Password:

Use weblog/test to access the Administration features.