Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
Markdown Monster - The Markdown Editor for Windows

Server Errors when updating live Web Applications online


:P
On this page:

In the last few weeks I've been noticing some problems with site updates I've made to my live server. I have an application running locally and it's running fine without problems. The app is running as a Web Application Project (WAP) and so when I update the application most of the time I only update the BIN folder and possibly one or two of the ASPX markup pages.

So I upload all files out of the BIN directory to the server and I quite frequently end up with a yellow screen of death like this:

ErrorDisplay

which is not an unfamiliar dialog. If you've ever uploaded files to a live site you've probably been greeted by this dialog (or the safe redirected version thereof). When files are 'in transit' .NET can and often does detect the new file on the server even though it's incomplete and tries to load it, which fails and boom you get the above error. That's well understood (although not really an optimal way to handle this - .NET should be able to detect that this file is still being written to instead of blinding unloading and reloading assemblies - but that's another story for another day).

But what's happening to me now is that these this page keeps coming up even after the file upload to the server has been completed. If I refresh the above page I will see the app cycling through the BIN directory assemblies with similar errors. Somehow the files are locked or at least ASP.NET thinks they are.

It takes re-copying files (first impulse right? Must have screwed up the upload somehow!), a restart or editing of Web.Config to get the app back up and running. However on several occasions I've even restarted and not been able to get the assemblies to work. This even though the app runs locally. In that case I ended up copying the assemblies a couple more times (without changes mind you) and eventually it took.

It certainly looks like ASP.NET is somehow caching an invalid assembly (or all of them) and not letting them go after the upload is complete. In that case I get the dependency error, but no notice that the process cannot access a file. No secondary error message.

This is a new behavior for me. The uploads are relatively small (about 500k total) so the upload is pretty quick usually and I've been donig these sort of live updates forever. Even so I often used to get failures while files were copying but never once the files were on the server. This seems to be common now and it's thrown a small kink in my typical update routine. <shrug>

App_Offline.htm

So over the last couple of days I started using App_Offline.htm to put the application on hold when updating assemblies, which is a bit of a hassle. In case you don't know about App_Offline.htm: It's a file you can drop into the root of your ASP.NET Virtual which causes ASP.NET to basically hold requests and show the content of this HTML page. ASP.NET displays the content of this file - oddly as text and not HTML - for any URL that would normally access an ASP.NET based URL.

I've never been enamored with this approach of having to have an external file that you either have to copy or rename on the server which is a bit annoying to automate. It'd be nicer if there was some mechanism to do this from within ASP.NET so it can be driven from within an application, but I suppose it makes sense: The idea is that no .NET assemblies should be loaded while App_Offline.htm is in place. If the idea is to make a clean update App_Offline.htm is the only way short of shutting down the Virtual Application or Web Server.

Since I've used the update with App_Offline.htm I've had no more problems as expected. I manage this process through my FTP upload process (which I usually do manually since I tend to update only a few files at a time rather than doing a full file dump). So my normal process is to logon jump to bin and update my assemblies, then update any pages that need to be updated.

With App_Offline.htm I keep the page in the site root and I rename it over FTP as needed. Rename to make it active, then copy all assemblies, then rename it back to something else.

What about you? How are you managing small updates to live sites? Have you seen the kind of issues i've mentioned above and have they gotten more pronounced recently (maybe something's changed in the runtimes?). And how do you manage your file updates and if you use App_Offline.htm how do you use it?

Posted in ASP.NET  

The Voices of Reason


 

Rik Hemsley
August 11, 2008

# re: Server Errors when updating live Web Applications online

I'd guess it goes something like this:

You start copying files, ASP.NET notices a change, restarts.
You finish copying files, ASP.NET was busy restarting, didn't notice the last changes.

Seems most likely to me, anyway.

Tim Lloyd
August 11, 2008

# re: Server Errors when updating live Web Applications online

To help diagnose these issues and to keep a general log of shutdowns, I use something like the following called from Application_End (naughty but helpful...):

public static class HttpRuntimeUtil
{
private static readonly log4net.ILog log = log4net.LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);

public static void LogApplicationShutdownInfo()
{
HttpRuntime runtime = (HttpRuntime)typeof(HttpRuntime).InvokeMember(
"_theRuntime",
BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.GetField,
null, null, null);

if (runtime != null)
{
string shutDownMessage = (string)runtime.GetType().InvokeMember(
"_shutDownMessage",
BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField,
null, runtime, null);

string shutDownStack = (string)runtime.GetType().InvokeMember(
"_shutDownStack",
BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.GetField,
null, runtime, null);

string message = String.Format(
"{0}{0}HttpRuntime -> Application Shutdown{0}{0}_shutDownMessage:{0}{1}{0}{0}_shutDownStack:{0}{2}",
Environment.NewLine,
shutDownMessage,
shutDownStack);

log.Info(message);
}
}
}

TomDeleu
August 11, 2008

# re: Server Errors when updating live Web Applications online

From within Visual Studio, when you publish a site (FTP by example) it creates a App_offline.htm file itsself while it's publishing the files... So that's a bit less of a hassle...

Nick Berardi
August 11, 2008

# re: Server Errors when updating live Web Applications online

Hi Rick,

Did you say that the App_Offline.htm file just outputs the text and not the HTML for the browser to render. I have had a similar experience where this file is delivered like a text file instead of an HTML file. This became very annoying, because my procedure is much like yours, rename it through FTP, and upload. I have been trying to find a work around, but I think I am just going to need to create a Module to deliver the file correctly, because for some reason the IIS team could manage.

Nick

Lee
August 11, 2008

# re: Server Errors when updating live Web Applications online

I temporarily switch my home directory to a holding site on the server. I have found this eliminates most the problems you describe but it does mean you have to have easy access to the server.

Neslekkim
August 11, 2008

# re: Server Errors when updating live Web Applications online

Have been having the same trouble, I ended up deleting the content of the bin folder, which made aspnet flush the cache I think. Then copied new content.
Almost always if I copied new content over old content I had this problem.
Small sites though, newer got to using the App_Offline feature, but I guess it does the same as deleting te content in the binfolder.

Peter Bromberg
August 11, 2008

# re: Server Errors when updating live Web Applications online

I haven't done this yet, but my understanding is that you can put the <httpRuntime .. element in your web.config and override the default times ASP.NET uses for recycling when files are FTP-ed in - such as delayNotificationTimeout and maxWaitChangeNotification.

Did I mention that I haven't tried this yet?

Peter
August 11, 2008

# re: Server Errors when updating live Web Applications online

I've experience caching weirdness before, fortunately it's not that often. What I do to fix that is to delete the cache from c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\ -- basically remove everything, or at least the folder that matches the misbehaving application.

As far as the App_Offline it depends on what you use to upload the site. We use SyncBack which can handle this with little trick (I wish it did this in more straightforward way, but ohh well)

Mohammad Azam
August 11, 2008

# re: Server Errors when updating live Web Applications online

Hi Rick,

I also use the App_Offline.htm approach when updating the site. My only concern is that if something goes bad during the update then I cannot visit my site to check what happened this is because App_Offline.htm is in place. My App_Offline.htm contains text saying "Please wait the site is being updated". I think there should be some way so that admin can view the site even though the App_Offline is in place.

Steve Trefethen
August 11, 2008

# re: Server Errors when updating live Web Applications online

Be aware that if app_offline.htm isn't of a certain size that IE will display the "friendly Errors" if that feature is enabled on the client. To get your app_offline.htm to display you need to add enough content (comments count) to allow your page to appear on the client. I believe ScottGu blogged about this issue quite some time ago so you might want to Google on it.

Btw, I use exactly this process for updating my live sites.

Stephen
August 11, 2008

# re: Server Errors when updating live Web Applications online

Weird that "it only shows text, not HTML"

The app_offline.htm i have in my live project displays HTML content just fine.... maybe it's the before mentioned size requirement throwing that off?

Tom Groeger
August 11, 2008

# re: Server Errors when updating live Web Applications online

I also use app_offline.htm while updating my webs, and it shows html, not only text .. works very well, using the Windows Explorer with QTTabBar ( http://qttabbar.wikidot.com , highly recommended ), one Tab points to my local Web, the other one to my IIS root ( as a webfolder ). I simply rename app_offline.htm.xxx to app_offline.htm, copy the files, rename app_offline again, done.

Nicholas Piasecki
August 11, 2008

# re: Server Errors when updating live Web Applications online

I've seen this before at work. For us, it was related to shadow copy.

- Our application loads some DLLs after start up based on configuration. With shadow copying on the bin directory enabled, this could cause quite fantastic yellow screens of death as the processes managed to load new persistence layer DLLs while still running the older versions of business logic and presentation DLLs. So we disabled shadow copying by adding shadowCopyBinAssemblies="false" in our Web.config.

The [dis]advantage is that this locks the DLLs, so a simple copy and paste from Explorer won't work. We have to find a way to unload the application domain....

- We could use app_offline.htm, but app_offline.htm simply Does Not Work when using Web Gardens. (Since each worker process doesn't seem to check for the file until it's awakened by the next request, you essentially have no good way to force all of the currently active AppDomains to check for the file and spin down, short of visiting your site a whole bunch of times rapidly in succession.) Also, this file returns HTTP 404, which is dumb.

- Our solution is to activate an "offline Web site" that handles all ASP.NET requests with a "We're updating blah blah" page that returns an HTTP 503 Service Unavailable code, stop the AppPool in the real Web site, and update/replace the files in the bin directory. Then deactivate the updating site and reactivate the real site.

I'd love to hear of a better way, but this method seems to cause the least heartburn.

Eric Johnson
August 11, 2008

# re: Server Errors when updating live Web Applications online

I use the app_offline.htm for our .net 2.0 sites while making updates. I have our QA/staging sites available over FTP, remote desktop into the production server and then use Beyond Compare 3 to do a folder to FTP site sync to compare the staging site to the production site. It will do a comparison of the files, show the differences that need to be moved to production, and then copy the files up. Once done I remove the app_offline.htm file and verify. Works great.

Ran Davidovitz
August 11, 2008

# re: Server Errors when updating live Web Applications online

I use also the same procedure BUT i think that you should have a mirror site that is working - so that you can redirect to that site while working on maintenance (Sure it depends on the amount of time you do the work, as i guess you also do basic sanity post the upload)

Allan Laustsen
August 12, 2008

# re: Server Errors when updating live Web Applications online

We are using a robocopy bat script with a custom regex replacement to the web.config files, we change the web.config before the uploading.
This ensures that the app_pool is not starting to reload, and then misses any changes to the bin folder/web.config file, this is done from a shadow directory on the live server, to minimize the time it takes to copy the files, because doing it with FTP will sometimes cause the app_pool to not reload all dll's and web.config file :-(

I have also experimented with haveing 2 IIS websites that are working as a master/slave, this can help the app_pool to "warm" up, so that the update would be 100% transparent for the end user, but it also causes other problems like caching

So far the best method seems to be the robocopy

Florian Krüsch
August 12, 2008

# re: Server Errors when updating live Web Applications online

App_offline is bad. The server is supposed to respond with 503s during maintenance.

Ben Mills
August 12, 2008

# re: Server Errors when updating live Web Applications online

Hi Rick,

I found a great solution for uploading a new version of my site. I use the following tools:

* Gene6 FTP server on the web server (has MTDM support)
* WebDrive on my development PC (sets up an FTP connection as a drive letter)
* ViceVersa file sync tool.

WebDrive and Gene6 FTP server effectively make a folder on the web server act just like a local folder. I then use ViceVersa to do a basic file sync. Some notes about this solution:

* It's important for the FTP server to have MTDM support so that uploaded files have the right last modified date/time set for future file syncs.
* ViceVersa allows include/exclude filters, so that I only upload the necessary files. For example, I upload .aspx, .dll and .config, but I don't upload .cs.
* I have a batch file which pushes App_offline.htm up to the server. I then make any database schema changes (using the excellent RedGate SqlCompare). Finally, I run the file sync which also wipes out the App_offline.htm.
* ViceVersa can also run a script before and after the file sync. I have it run a script after the file sync to open the homepage, so that the first user won't have to wait during the first request.

Hope this helps.
Ben

Nik Kalyani
August 12, 2008

# re: Server Errors when updating live Web Applications online

I use the following approach:

1) I only keep the assemblies unlikely to change in bin (i.e. components, third-party libs etc.). I put my app's assemblies into a sub-folder of bin and name the folder according to the date (for example: bin\20080801)

2) My web.config has the following:

<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<probing privatePath="bin;bin\20080801;" />
</assemblyBinding>
</runtime>

When I need to update assemblies, I just upload them into a new sub-folder in bin, again with a name corresponding to the current date (bin\20080812). After the upload is complete, I upload web.config with a change to the sub-folder name. The app re-starts and picks up the new assemblies and forgets about the old sub-folder assemblies which I can leave or delete.

Seems to work and has the added benefit of speeding up the app start since the sub-folder assemblies are ignored until needed.

Steve from Pleasant Hill
August 13, 2008

# re: Server Errors when updating live Web Applications online

Nik,

I have no idea if yours is the best solution, but from reading it I like it a lot. This would make rolling back to a previous version pretty easy.

Nice!

Rick Strahl
August 14, 2008

# re: Server Errors when updating live Web Applications online

Looks like the App_Offline.htm is being returned without a content type. So IE will sniff the content and detect the HTML and render it. Firefox however doesn't see a content type and serves up the content as text.

Response Headers:
Server Microsoft-IIS/7.0
X-Powered-By ASP.NET
Date Fri, 15 Aug 2008 02:14:59 GMT
Content-Length 25

Speednet
August 16, 2008

# re: Server Errors when updating live Web Applications online

Hi Rick,

I use App_Offline.htm whenever I'm doing site changes that involve CSS, Script, and other stuff that I can't bang into the site in one quick stroke. I keep it in the root folder with ".exclude" at the end (i.e., "App_Offline.htm.exclude") and then toggle renaming it to turn it on/off. I guess the only thing quicker would be to create a batch file, but this is simpler for me to remember.

As far as site updates are concerned (with the warning message you're getting), I pre-compile the entire site -- even the markup -- so site updates involving just the ASP.NET stuff can be banged into the site instantly, without the need for App_Offline.htm.

I upload all the bin folder updates to a temp folder, and then copy/replace everything at once in the bin folder using Windows Explorer or DOS window. It happens instantly, with no downtime to users. Also, there is never any first-use compile delays that can happen with updateable sites.

Matt Davis
August 27, 2008

# re: Server Errors when updating live Web Applications online

Also check out: http://msdn.microsoft.com/en-us/library/e1f13641.aspx

You can set config values like 'waitChangeNotification', so the app won't recycle right away. Giving the file system a chance to catch up with the writes.

Milo
August 27, 2008

# re: Server Errors when updating live Web Applications online

When you "down" the site, I'm guessing you drop all the session with the .net reflection event when the files are updated. Can you not involved the "pause" action to allow the service to continue serving transient visitors and stop new sessions? Waiting for 90 second would possible allow the majority of visitors but not guarentee all of them...

Just forcing a pool reload isn't very elegent for visitors.

Does anyone address this?

Rick Strahl
August 27, 2008

# re: Server Errors when updating live Web Applications online

@Milo - there's really no way around downing the AppDomain when you are replacing assemblies because assemblies loaded must unload in order to be replaced. .NET cannot unload assemblies once they are loaded so no matter what you have to shut down the AppDomain for the changes to be recognized and that means taking InProc Sessions down with your app.

If Sessions are critical in your app you shouldn't be using InProc sessions anyway - using OutOfProc or Sql Sessions will let your sessions survive AppDomain and AppPool restarts and is a much better choice if Session data is essential to visitors of your site.

Simone Busoli
September 16, 2008

# re: Server Errors when updating live Web Applications online

Guys, the waitChangeNotification and maxWaitChangeNotification settings are no longer respected by IIS 7. That was a cool feature, and my primary deploy mechanism since it allowed hot swaps without the need to take nodes out of balancing or put the app offline.

majco333
October 21, 2008

# Problem App_Offline.htm with IIS 7.0

Hi, I have problem with App_Offline.htm in web application under IIS 7.0.

Firefox and Opera (IE show as html) show only text because server send to the browser this headers:

HTTP/1.0 404 Not Found
Server: Microsoft-IIS/7.0
X-Powered-By: ASP.NET
Content-Length: 1641
Content-Type: text/plain

It is bug of IIS 7.0??? Because IIS 5.1 and IIS 6.0 send Content-Type: text/html...

riverbed wan acceleration
December 29, 2008

# riverbed wan acceleration

Understanding Web 2. 0 Attacks Protecting the Crown Jewels With Database Security-- Rothman Chats With Ted Julian What\'s So Scary About CSRF? Plenty! Rothman Talks to Nitesh Dhanjani What You Need to Know About Source Code Analysis: Mike Rothman Talks to Brian Chess SQL Injection Rears Its Ugly Head Again Number One Threat to Web Applications: Mike Talks SQL Injection With White Hat Security Pros and Cons of Big Security: Mike Talks to Alan Shimel Is Big the New Small in Application Security? The Scourge of Cross- ...

moonriverlabs
January 14, 2009

# re: Server Errors when updating live Web Applications online

I routinely use app_offline.htm when deploying code these days, but another use I'm trying for it now is temporarily suspending sites for customers in arrears. The problem here is that app_offline.htm *only* blocks calls to pages that are processed by .NET (i.e. *.aspx, *.asmx etc) - it does not block access to non-.NET files (i.e. *.htm, *.html etc).

Is there any way to get IIS to block all types of content regardless of file type/extension when an app_offline.htm is present?

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024