Rick Strahl's FoxPro and Web Connection Web Log

VFP String Concatenation Issue in Loops

Sun, 11 May 2008 13:13:00 GMT

I got bit again by a nasty string concatenation problem in VFP. This isn't the first time I've hit this particular issue and I've known about it for years, but it's one that's easy to miss and so from time to time I hit it again. The problem occurs when you have string concatenation code that involves an expression that returns results from a function and where the function often returns a large amount of text.

For example, the following code - taken from Web Connection's wwWebRepeater control - fails:

   LOCAL lnX, lnCount

   lnCount = this.ChildControls.Count

   FOR lnX = 1 TO lnCount

       LOCAL loControl

       loControl = this.ChildControls.aItems(lnX,2)

       IF !ISNULL(loControl)

          lcOutput = lcOutput + loControl.Render()

       ENDIF

The code fails to properly build your concatenated string. The code in question is string concatenation where the code is appending to an existing string by appending an expression - typically a class method that is called to provide additional data. In each iteration of the loop lcOutput is set with the result of loControl.Render() rather than the concatenation of lcOutput + loControl.Render().

This makes for some interesting debugging - you step over the code with no problems but it takes a while to actually grok what the debugger is telling you - it's not the debugger truncating your string but VFP actually failing to concatenate.

This is an insidious bug in that sometimes the code works as expected and at other times it fails with creating the incorrect string format. The only way to really detect this problem is to step through with the debugger and see that the result is incorrect.

Luckily the workaround for this is fairly easy: you can create a variable and assign that to the concatenation expression. The following code produces the expected result:

   LOCAL lnX, lnCount, lcT

   lnCount = this.ChildControls.Count

   FOR lnX = 1 TO lnCount

       LOCAL loControl

       loControl = this.ChildControls.aItems(lnX,2)

       IF !ISNULL(loControl)

          lcT = loControl.Render()

          lcOutput = lcOutput + lcT

       ENDIF

   ENDFOR

This fixes the issue, but of course it results in the string being copied to a variable first, which depending on how much output is generated may be an expensive operation.

In Web Connection I had to use code like this in a lot of places and just last week I ran into this nasty 'bug' again while working with a client in a place where I missed the issue. Oddly, the problem occurred in a fairly high traffic area of the framework, so I'm surprised that nobody has actually hit this particular issue before. It's possible the behavior varies slightly with different versions of VFP 9 (ie. RTM, SP1, SP2) - I believe some of these behaviors were actually addressed in SP1 and later.

This 'behavior' was introduced with the string optimizations in VFP 7 which drastically improved string concatenation performance in VFP back then. When concatenating strings VFP will usually allocate more memory than it needs to concatenate the current string value to optimize string allocation. So rather than having to copy strings all the time for every concatenation, it allocates a bigger buffer and loads the content of the right had side of the concatenation directly into the end of the string buffer which improves string performance drastically.

I can only guess on why this doesn't always work with functions but my guess is that if the buffer size exceeds the pre allocated buffer that the string gets recreated and somewhere in that process the new string pointer for the result, rather than the concatenation is returned.

No matter what the problem is, it's a bug that you should be aware of since it's likely to be hit by a wide variety of code.

IIS 7 Default Request Filtering and Web Connection

Mon, 7 Apr 2008 06:32:00 GMT

IIS 7 has an extensive list of extensions and paths that it deems as restricted. This is generally a good thing as it blocks URL access to many common paths that are frequently used in Web applications to hold semi-private files like code and binary assemblies for example in ASP.NET applications.

As it turns out the default behavior also affects Web Connection because the default filtering completely disallows direct access to a BIN directory. In addition, IIS 7 blocks out access to many file extensions that you might previous have used for your own script mapes. For example, I just ran a demo and created a script map of .dd for my project only to find that it bombed with 404 everytime. It took some sleuthing to find out that .dd is a restricted extension and changing the extension immediately fixed the problem.

So what does it mean to your Web Connection Apps?

The biggest issue that you might run into with IIS 7 that if you have WC.DLL installed in the /Bin directory of your virtual or Web root, you cannot access the DLL directly. Urls like this:

/your Virtual/wc.dll?wwMaint~ShowStatus

Will fail to work and you'll get a 404 error like this:

HTTP Error 404.0 - Not Found

The resource you are looking for has been removed, had its name changed, or is temporarily unavailable.

Detailed Error Information

Module		IsapiFilterModule
Notification		MapPath
Handler		ISAPI-dll
Error Code		0x80070002
Requested URL	http://localhost:80/timetrakker/bin/wc.dll?wwMaint~ShowStatus
Physical Path	c:\westwind\TimeTrakker\bin\wc.dll
Logon Method	Not yet determined
Logon User	Not yet determined

Most likely causes:

· The directory or file specified does not exist on the Web server.

· The URL contains a typographical error.

· A custom filter or module, such as URLScan, restricts access to the file.

The restriction here lies in IIS 7's configuration for Request Filtering which can be found in ApplicationHost.config (in System32/inetsvr/config). In it you'll find a request filtering section:

<add fileExtension=".dd" allowed="false" />

</fileExtensions>

<add segment="bin " />

</hiddenSegments>

</requestFiltering>

The culprit for the direct WC.DLL execution is the hidden segment of bin filter in the hiddenSegments section. This filter basically prevents anything to be Web visible via URL that has a bin directory in its path.

<add segment="bin " />

If you absolutely need to run wc.dll directly and you don't or can't use scriptmaps – which I highly recommend anyway though – you can comment out this block



Which will then allow you to execute wc.dll out of the bin directory. NOTE: I would not advise this! It's a bad call to override these system settings because you'll have to remember to do it every time you install a new installation or move it.

Note that request filtering is a global setting – it must be set in ApplicationHost.config and cannot be delegated down to the web.config unless you override this setting:

And change the key to Allow.

Another option: Move the DLL out of the BIN directory into the root or another folder.

But I wouldn't recommend changing either of these options! The former mucks with default configuration settings that you have to remember to set each time the app gets reinstalled and the latter requires changing URLs anyway - and there's a better way to do that with scriptmaps.

So a better solution is to always use script maps. Create a script map or even use one of the default script maps that Web Connection installs into every installation (.WC, .WCSX are two of them) and replace every call to wc.dll with wc.wc and remove the /Bin path from the url. So

/myVirtual/bin/wc.dll?wwMaint~ShowStatus

Might become

/myVirtual/wc.wc?wwMaint~ShowStatus

In some situations this may cause pathing problems because if you used the DLL pages were pathed to the bin directory and relative links for image and other resources may have been relative to the bin folder.

But this is why we've tried for years to push our user to use script maps in the first place – script maps are much easier to manage both in terms of security as well as flexibilty.

Watch out for other blocked Extensions

When you create new projects and new script map extensions, you should be careful not to choose any blocked extensions.

For example when I tried to create an extension for .dd and then hit a page with this extension I got:

HTTP Error 404.7 - Not Found

The request filtering module is configured to deny the file extension.

Detailed Error Information

Module		RequestFilteringModule
Notification		BeginRequest
Handler		StaticFile
Error Code		0x00000000
Requested URL	http://localhost:80/timetrakker/default.dd
Physical Path	c:\westwind\TimeTrakker\default.dd
Logon Method	Not yet determined
Logon User	Not yet determined

Most likely causes:

· Request filtering is configured for the Web server and the file extension for this request is explicitly denied.

Things you can try:

· Verify the configuration/system.webServer/security/requestFiltering/fileExtensions settings in applicationhost.config and web.config.

Note that here the message points you right at the problem and where to look. It points right at the Request Filtering section in ApplicationHost.config. If you look back on the list of extensions you can see that .DD is indeed included in the list of restricted extensions.

Again the solution here is either to allow the extension or alternately choose a different extension.

This seems like a lot of new restrictions but I'd say these are a good thing. They are easy to fix or workaround as long as you know what the settings are. None of this is a problem for Web Connection applications that use script maps to begin with, so this is a reminder why script map formatting is the way to go with WWWC applications…