Rick Strahl's FoxPro and Web Connection Web Log
White Papers | Products | Message Board | News |

How to work around Visual FoxPro's 16 Megabyte String Limit


Thursday, January 19, 2012, 3:56:39 AM

If you take a look at the Visual FoxPro documentation and the System Limits you find that FoxPro's string length limit is somewhere around ~16mb. The following is from the FoxPro Documentation:

Maximum # of characters per character string or memory variable: 16,777,184

Now 16mb seems like a lot of data, but in certain environments like Web applications it's not uncommon to send or receive data larger than 16 megs. In fact, last week I got a message from a user of our Client Tools lamenting the fact that the HTTP Upload functionality does not allow for uploads larger than 16 megs. One of his applications is trying to occasionally upload rather huge files to a server using our wwHttp class. At the time I did not have a good solution for him due to the 16meg limit.

What does the 16meg Limit really mean?

The FoxPro documentation actually is not quite accurate! You can actually get strings much larger than 16megs into FoxPro. For example you can load up a huge file like the Office 2010 download from MSDN like this:

lcFile = FILETOSTR("e:\downloads\en_office_professional_plus_2010_x86_515486.exe")
? LEN(lcFile)

The size of this string: 681,876,016 bytes or 681megs! Ok that's a little extreme ?? but to my surprise that worked just fine; you can load up a really huge string in VFP if you need to. But when you get over 16megs the behavior of strings changes and you can't do all the things you normally do with strings.

Other operations do not work for example, the following which creates an 18meg string fails:

lcString = REPLICATE("1234567890",1800000)

with "String is too long to fit".

However the following which creates a 25 meg string does work:

lcString = REPLICATE("1234567890",1500000) && < 16 megs
lcString = lcString + REPLICATE("1234567890",1000000)
? LEN(lcString)  && 25,000,000

The following is almost identical except it copies the longer string to another string which does not work:

lcString = REPLICATE("1234567890",1500000) 
lcNewString = lcString + REPLICATE("1234567890",1000000)

And that my friends is the real sticking point with large strings. You can create them, but once they get bigger than 16megs you can no longer assign them to a new variable. That might sound easy to avoid but it's actually tough to do. If you pass string to mutable methods it's very likely that they are actually copied into temporary variables or added to another variable in a simulated buffer, and that is typically where large strings fail.

So what can we learn from this:

Doesn't Work:

  • Assigning a massive FoxPro string to another string fails
  • FoxPro commands that mutate strings like REPLICATE(), STRTRAN() can't create output larger than 16 megs

Works:

  • Assigning a massive string from a file using FileToString() works
  • Adding to the same large string (lcOutput = lcOutput + " more text") works and the string can grow
  • Calling methods that manipulate string work as long as the same string is assigned

There are some limitations but knowing that if you work with a single string instance that can grow large is actually good news. What this means is that if you're careful with how you use strings in FoxPro you can fairly easily get around the 16 meg string limit.

This actually worked well for me in the wwHttp class and the POST issue for larger than 16meg files but not string. Internally wwHttp uses a cPostBuffer property to hold the POST data. The failure was occuring in the send code which would copy the string to a temporary string and get the size, then pass that to the WinInet APIs. The fix for this was fairly easy: Rather than creating the temporary variables (which were redundant anyway) I simply used the class property directly throughout the code without any hand off and voila, now wwHttp supports POSTs for greater than 16 megs.

The code I use is kinda ugly because it's doing lots of string concatenation to build up the Post buffer. Something along these lines like this excerpt from wwHttp::AddPostKey:

************************************************************************
* wwHTTP :: AddPostKey
*********************************
***  Function: Adds POST variables to the HTTP request
***    Assume: depends on nHTTPPostMode setting
***      Pass: 
***    Return:
************************************************************************
FUNCTION AddPostKey(tcKey, tcValue, llFileName)
LOCAL lcOldAlias
tcKey=IIF(VARTYPE(tcKey)="C",tcKey,"")
tcValue=IIF(VARTYPE(tcValue)="C",tcValue,"")


IF tcKey="RESET" OR PCOUNT() = 0
   THIS.cPostBuffer = ""
   RETURN
ENDIF

*** If we post a raw buffer swap parms
IF PCOUNT() < 2
   tcValue = tcKey
   tcKey = ""
ENDIF

IF !EMPTY(tcKey)
   DO CASE
    *** Url Encoded
    CASE THIS.nhttppostmode = 1         
         THIS.cPostBuffer = this.cPostBuffer + IIF(!EMPTY(this.cPostBuffer),"&","") + ;
                            tcKey +"="+ URLEncode(tcValue) 
      *** Multi-part formvars and file
    CASE this.nHttpPostMode = 2
      *** Check for File Flag -  HTTP File Upload - Second parm is filename
      IF llFileName
           THIS.cPostBuffer = THIS.cPostBuffer + "--" + MULTIPART_BOUNDARY + CRLF + ;
            [Content-Disposition: form-data; name="]+tcKey+["; filename="] + JUSTFNAME(tcValue) + ["]+CRLF+CRLF
         this.cPostBuffer = this.cPostBuffer + FILETOSTR(FULLPATH(tcValue))
         this.cPostBuffer = this.cPostBuffer + CRLF
      ELSE
           this.cPostBuffer = this.cPostBuffer +"--" + MULTIPART_BOUNDARY + CRLF + ;
            [Content-Disposition: form-data; name="]+tcKey+["]+CRLF+CRLF
         this.cPostBuffer = this.cPostBuffer + tcValue
      ENDIF
   ENDCASE
ELSE
   *** If there's no Key post the raw buffer
   this.cPostBuffer = this.cPostBuffer +tcValue
ENDIF

ENDFUNC

AddPostKey can accept either a string value or a filename to load from. The file loading works by accepting the filename and then directly loading the file from within the function:

this.cPostBuffer = this.cPostBuffer + FILETOSTR(FULLPATH(tcValue))

This works fine because the file is directly loaded up into the buffer with no intermediate string variable.

You cannot however pass a string that is greater than 16 megs into this function because the code that adds the key basically does this with the tcValue parameter:

this.cPostBuffer = this.cPostBuffer + tcValue

which is assigning the larger than 16 meg string (tcValue in this case) to another variable and as discussed earlier that fails with "String too long to fit". Using a string to buffer your output to build up a larger string, there's no workaround for adding a larger than 16 meg string to another variable or buffer using variables. So my code now works with files loaded from disk, but not string parameters

Good but not good enough!

Files for Large Buffers

Based on the earlier examples I showed we know that we can easily load up massive content from a file. Thus FILETOSTR() offers an easy way to serve large files. Knowing that it's possible to build stream like class that allows you to accumulate string content in a file and then later retrieve it. To do this I created a wwFileStream class. Using the class looks like this:

*** Load library
DO wwapi

*** Create 20 meg string
lcString = REPLICATE("1234567890",1500000)
lcString = lcString + REPLICATE("1234567890",500000)

*** Create a stream
loStream = CREATEOBJECT("wwFileStream")

*** Write the 20meg  string
loStream.Write(lcString)

*** Add some more string data
loStream.WriteLine("...added content")

*** Now write a 16meg+ to the buffer as well
loStream.WriteFile("e:\downloads\ActiveReports3_5100158.zip")

*** Works
lcLongString = loStream.ToString()

*** 55+ megs
? loStream.nLength
? LEN(lcLongString)

*** Clear the file (auto when released)
loStream.Dispose()

Using this mechanism you can build up very large strings from files or strings regardless of what the size of the string is.

How wwFileStream works

Internally wwFileStream opens a low level file and tracks the handle. Each Write() operation does an FWRITE() to disk and the handle is released when the class goes out of scope.

The class implementation is pretty straight forward:

*************************************************************
DEFINE CLASS wwFileStream AS Custom
*************************************************************
*: Author: Rick Strahl
*:         (c) West Wind Technologies, 2012
*:Contact: http://www.west-wind.com
*:Created: 01/04/2012
*************************************************************

nHandle = 0
cFileName = "" 
nLength = 0


************************************************************************
*  Init
****************************************
FUNCTION Init()

this.cFileName = SYS(2023)  + "\" +  SYS(2015) + ".txt"
this.nHandle = FCREATE(this.cFileName)
this.nLength = 0

ENDFUNC
*   Init

************************************************************************
*  Destroy
****************************************
FUNCTION Destroy()
this.Dispose()
ENDFUNC
*   Destroy

************************************************************************
*  Dispose
****************************************
FUNCTION Dispose()

IF THIS.nHandle > 0
   TRY
   FCLOSE(this.nHandle)
   DELETE FILE (this.cFileName)
   CATCH
   ENDTRY
ENDIF
this.nLength = 0
ENDFUNC
*   Destroy

************************************************************************
*  Write
****************************************
FUNCTION Write(lcContent)
THIS.nLength = THIS.nLength + LEN(lcContent)
FWRITE(this.nHandle,lcContent)
ENDFUNC
*   Write

************************************************************************
*  WriteLine
****************************************
FUNCTION WriteLine(lcContent)
this.Write(lcContent)
this.Write(CHR(13) + CHR(10))
ENDFUNC
*   WriteLine

************************************************************************
*  WriteFile
****************************************
FUNCTION WriteFile(lcFileName)
lcFileName = FULLPATH(lcFileName)
this.Write(FILETOSTR( lcFileName ))
ENDFUNC
*   WriteFile

************************************************************************
*  ToString()
****************************************
FUNCTION ToString()
LOCAL lcOutput

FCLOSE(this.nHandle)
lcOutput = FILETOSTR(this.cFileName)

*** Reopen the file
this.nHandle = FOPEN(this.cFileName,1)
FSEEK(this.nHandle,0,2)

RETURN lcOutput
ENDFUNC
*   ToString()


************************************************************************
*  Clear
****************************************
FUNCTION Clear()

THIS.Dispose()
THIS.Init()

ENDFUNC
*   Clear

ENDDEFINE
*EOC wwFileStream 

The code is fairly self explanatory. The class creates a file in the temp folder and saves the handle. Any write operation then uses the file handle to FWRITE() either a string or the output from FILETOSTR(). ToString() can be called to retrieve the file, which closes the file, reads it then reopens it and points to the end. When the class is released the handle is closed and the handle released.

Using this class makes it easy to create large strings and hold onto them. The additional advantage is that memory usage is kept low as strings are loaded up only briefly and then immediately written to file and can be released. So if you're dealing with very large strings a class like this is actually highly recommended. In fact Web Connection uses this same approach for file based application output.

A matching MemoryStream Class

While the FileStream class works, it does have some overhead compared to memory based operation especially when you're dealing with small amounts of data. In the wwHttp class for example, I would not want to create a new wwFileStream for each POST operation. 99% of POST ops are going to be light weight, so it makes sense to only use the wwFileStream class selectively.

In order to do this I also created a wwMemoryStream class which has the same interface as wwFileStream and which uses a simple string property on the class to hold data. Since the classes have the same interface they are interchangable in use which makes them easily swappable.

The code for wwMemoryStream looks like this:

DEFINE CLASS wwMemoryStream AS Custom
*************************************************************
*: Author: Rick Strahl
*:         (c) West Wind Technologies, 2012
*:Contact: http://www.west-wind.com
*:Created: 01/05/2012
*************************************************************

cOutput = ""
nLength = 0

************************************************************************
*  Destroy
****************************************
FUNCTION Destroy()
THIS.Dispose()
ENDFUNC
*   Destroy

************************************************************************
*  Dispose
****************************************
FUNCTION Dispose()
this.cOutput = ""
this.nLength = 0
ENDFUNC
*   Dispose

************************************************************************
*  Clear
****************************************
FUNCTION Clear()
this.cOutput = ""
this.nLength = 0
ENDFUNC
*   Clear

************************************************************************
*  Write
****************************************
FUNCTION Write(lcContent)
this.nLength = this.nLength + LEN(lcContent)
this.cOutput = this.cOutput + lcContent
ENDFUNC
*   Write

************************************************************************
*  WriteLine
****************************************
FUNCTION WriteLine(lcContent)
this.Write(lcContent)
this.Write(CRLF)
ENDFUNC
*   WriteLine

************************************************************************
*  WriteFile
****************************************
FUNCTION WriteFile(lcFileName)
this.Write(FILETOSTR( FULLPATH(lcFileName) ))
ENDFUNC
*   WriteFile

************************************************************************
*  ToString()
****************************************
FUNCTION ToString()
RETURN this.cOutput
ENDFUNC
*   ToString()

ENDDEFINE
*EOC wwMemoryStream 

This way the user can easily chose which of the streams to use simply by specifying:

IF VARTYPE(this.oPostStream) != "O"
   this.oPostStream = CREATEOBJECT(this.cPostStreamClass)
ENDIF

What's also nice about this approach is that the mechanism becomes extensible. If you want to store POST vars in another storage format you can simply create another subclass that implements the same methods and now can store your post variables in an INI file or in structured storage etc. Unlikely scenario for POST data, but very useful for other potential data storage scenarios.

BTW, the wwFileStream class is also a fairly useful generic file output tool. If you ever need to write output to files it provides a real easy OO way to do so, cleaning up after itself when you close it. I've used classes like (wwResponseFile) for years in various applications that need to create file output. It's very useful in many situations.

Summary

Even though Visual FoxPro has a 16 meg string limit, you now have some tools in your arsenal to work around this limit and work with larger strings. While you can work with larger strings, keep in mind that once you go past 16 megs you can't assign that string to anything else. It also gets much harder (and slower) to string manipulation on that string once you're beyond VFP's legal limit.

Still it's nice to know that the limit is not a final one and there are ways to work around it.

Resources

Posted in: FoxPro

Feedback for this Weblog Entry


re: How to work around Visual FoxPro's 16 Megabyte String Limit



Sean Gowens
Rick - Thanks for looking into my original request and coming up with this really helpful article. Historically I used FTP for most everything inside of our desktop app, but found more and more issues with firewalls so I transitioned to HTTP. This is going to help tremendously.

re: How to work around Visual FoxPro's 16 Megabyte String Limit



JimM
Thanks Rick... I didn't know this...

re: How to work around Visual FoxPro's 16 Megabyte String Limit



Harvey
Is this this now an option on the FileUpload control?

re: How to work around Visual FoxPro's 16 Megabyte String Limit


@Jim - unfortunately not. The problem on the server side is that the Server Response comes in encoded. It's one buffer that has all the mime encoding in it and that needs to be decoded - ie. the string has to be manipulated.

re: How to work around Visual FoxPro's 16 Megabyte String Limit



MaroŇ° Klempa
--CodeMarker-- Hello Rick, there is simple workaround also for passing string greather than 16 meg to AddPostKey method. Solution is parse tcValue string into chunks smaller than 16,777,184 B using SUBSTRING() and add this chunks to THIS.cPostBuffer using THIS.cPostBuffer=THIS.cPostBuffer + lcChunk

re: How to work around Visual FoxPro's 16 Megabyte String Limit



Tobias Bartsch
Hello Rick,
thanks for sharing

Just wanted to say that when i use your wwFileStream with big files (250MB-500MB) it doesnt work probably. You will get the Error "There is not enough memory to complete this operation (Error 43)"

So i changed the WriteFile Methode to this, so at least this function will work again

--CodeMarker--.

Also the ToString() Method wont work with big files, so you have to avoid using this method. I tried to read the file with FREAD() but the Error still pops up.
Maybe this information helps somone

cheers Tobias

re: How to work around Visual FoxPro's 16 Megabyte String Limit


Hi Rick,

Can't find 'oPostStream' and 'cPostStreamClass' in WC6's wwHTTP.
Did you implement these properties somewhere else?

Thanks

re: How to work around Visual FoxPro's 16 Megabyte String Limit


Hi Rick,

Attempted to modify wc6/classes/wwAPI > EncodeDBF() to support files larger than 16Meg, couldn't work it out, any idea why?

--CodeMarker--

Long Path Tool



aiden carter
Thanks for this post,

The Long path tool is the very best program for error, unlock solution.
Try it and solved your problem.
I used the long path tool and I solved my error, unlocks problem solution.

re: How to work around Visual FoxPro's 16 Megabyte String Limit



liona1982
I have recently used a program called Long Path Tool & i am so happy with the results that this particular program does. I have more than 50 GB of duplicate files in my computer and i was going crazy with the hard disk space that was being consumed and thanks to the "Long Path Tool" where i don't need to search each & every file to delete. Long Path Tool did the job for me within a very short period of time. So no need to google each problem to remove duplicate & unwanted files, just grab the program called Long Path Tool.

re: How to work around Visual FoxPro's 16 Megabyte String Limit



liona1982
In other words well, according to me in my opinion Long Path Tool is the best because I am still using it now for more than 3 years & will continue to do so as it is so user friendly, simple to use & you can do anything with it & I would definitely recommend it to everyone out there.
 



© Rick Strahl, West Wind Technologies, 2003 - 2018