Web Connection
Recycling a wc app in IIS
Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  All
  Oct 2, 2014 @ 02:37am
Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?



-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  Rick Strahl
  Thierry Nivelet (FoxInCloud)
  Oct 2, 2014 @ 04:02pm

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?



Rick Strahl
West Wind Technologies

Making waves on the Web
from Maui

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 3, 2014 @ 01:08am
Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?



Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  Rick Strahl
  Thierry Nivelet (FoxInCloud)
  Oct 3, 2014 @ 02:25pm

I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?






Rick Strahl
West Wind Technologies

Making waves on the Web
from Maui

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 6, 2014 @ 03:38am
OK, thanks Rick for this input.

I'll monitor memory usage, test a daily recycling and remove timeout


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?







-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 24, 2014 @ 04:33am
Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?







-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  Rick Strahl
  Thierry Nivelet (FoxInCloud)
  Oct 24, 2014 @ 03:29pm
Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?









Rick Strahl
West Wind Technologies

Making waves on the Web
from Maui

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 27, 2014 @ 04:19am
Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?










-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  Rick Strahl
  Thierry Nivelet (FoxInCloud)
  Oct 27, 2014 @ 12:18pm

Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?












Rick Strahl
West Wind Technologies

Making waves on the Web
from Maui

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 28, 2014 @ 01:47am
yes, single use
there must something wrong somewhere, we'll find out
thanks a lot for your help

Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?













-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 28, 2014 @ 02:08am
could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?













-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  Rick Strahl
  Thierry Nivelet (FoxInCloud)
  Oct 28, 2014 @ 02:01pm

We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?















Rick Strahl
West Wind Technologies

Making waves on the Web
from Maui

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 29, 2014 @ 03:03am
Rick,
Thanks for these clear explanations.
I think I'll remove the recycling @ 3:00 AM and see what happens, especially if the program below can stop the COM objects OK

We don't need a process list

useful at least for me to understand the sequence

This was a big problem on Windows 2000

Strange! Never ever had any problem with our previous Win 2k server - always stopped and restarted COM objects distantly to replace the tables behind http://foxincloud.com/demos.php using the program below, almost every day without any dangling COM object for more than 2 years.

loHTTP = NewObject('wwHTTP', 'wwHTTP.prg')
loFTP = NewObject('wwFTP', 'wwIPstuff.vcx')

* Set server on hold
loHTTP.HTTPget('<<a href="http://foxincloud.com/bin/wc.dll?_maintain~HoldRequests" target="top" >>http://foxincloud.com/bin/wc.dll?_maintain~HoldRequests<</a>>', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

* Upload customer table into server's \data folder
WITH m.loFTP AS wwFTP OF wwIPstuff.vcx

cFileSrce = DOS_INT + 'data\customer.dbf'
cFileTrgt = '/AbaqueApps/ttWeb/Data/customer.dbf'

.nConnectTimeout = 2 * 60 && réglage empirique car on ne sais pas exactement à quoi correspond cette valeur ...
.nFTPbinary = 1
llResult = .T.;
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'dbf'), ForceExt(m.cFileTrgt, 'dbf'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW));
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'cdx'), ForceExt(m.cFileTrgt, 'cdx'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW))

IF NOT m.llResult
? Chr(7)
? .cErrorMsg
ENDIF
ENDWITH

* Set server to process requests
loHTTP.HTTPget('<<a href="http://foxincloud.com/bin/wc.dll?_maintain~Load" target="top" >>http://foxincloud.com/bin/wc.dll?_maintain~Load<</a>>', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

RETURN m.llResult


We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?
















-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 29, 2014 @ 03:07am
Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

Will do, thanks

From what I understand, if I can run a WC4 app with the WC5 wc.dll, is it also possible using WC5 .net module?

We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?















Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  Rick Strahl
  Thierry Nivelet (FoxInCloud)
  Oct 29, 2014 @ 02:37pm
Thierry,

I know I do the same thing and I don't have any issues with dangling references either.

Can you try the following:

* Clear your wcErrors.txt
* Manually unload servers from the ISAPI page
* Do servers go away?
* What's in wcErrors.txt

IOW, do manual unloads work or can you see that fail. Are there messages in wcErrors.txt - if server unload you should actually see the unload messages (not errors). If they're not there that means the unload code is not actually firing which seems to point to the problem I mentinoed in my last message.

+++ Rick ---



Rick,
Thanks for these clear explanations.
I think I'll remove the recycling @ 3:00 AM and see what happens, especially if the program below can stop the COM objects OK

We don't need a process list

useful at least for me to understand the sequence

This was a big problem on Windows 2000

Strange! Never ever had any problem with our previous Win 2k server - always stopped and restarted COM objects distantly to replace the tables behind http://foxincloud.com/demos.php using the program below, almost every day without any dangling COM object for more than 2 years.

loHTTP = NewObject('wwHTTP', 'wwHTTP.prg')
loFTP = NewObject('wwFTP', 'wwIPstuff.vcx')

* Set server on hold
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~HoldRequests', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

* Upload customer table into server's \data folder
WITH m.loFTP AS wwFTP OF wwIPstuff.vcx

cFileSrce = DOS_INT + 'data\customer.dbf'
cFileTrgt = '/AbaqueApps/ttWeb/Data/customer.dbf'

.nConnectTimeout = 2 * 60 && réglage empirique car on ne sais pas exactement à quoi correspond cette valeur ...
.nFTPbinary = 1
llResult = .T.;
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'dbf'), ForceExt(m.cFileTrgt, 'dbf'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW));
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'cdx'), ForceExt(m.cFileTrgt, 'cdx'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW))

IF NOT m.llResult
? Chr(7)
? .cErrorMsg
ENDIF
ENDWITH

* Set server to process requests
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~Load', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

RETURN m.llResult


We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?


















Rick Strahl
West Wind Technologies

Making waves on the Web
from Maui

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 30, 2014 @ 01:27am
works as expected

Filename: C:\Temp\TT\wcErrors.txt
Clear

2014-10-30 09:26:24:017 Forced Release of Server - 0


Thierry,

I know I do the same thing and I don't have any issues with dangling references either.

Can you try the following:

* Clear your wcErrors.txt
* Manually unload servers from the ISAPI page
* Do servers go away?
* What's in wcErrors.txt

IOW, do manual unloads work or can you see that fail. Are there messages in wcErrors.txt - if server unload you should actually see the unload messages (not errors). If they're not there that means the unload code is not actually firing which seems to point to the problem I mentinoed in my last message.

+++ Rick ---



Rick,
Thanks for these clear explanations.
I think I'll remove the recycling @ 3:00 AM and see what happens, especially if the program below can stop the COM objects OK

We don't need a process list

useful at least for me to understand the sequence

This was a big problem on Windows 2000

Strange! Never ever had any problem with our previous Win 2k server - always stopped and restarted COM objects distantly to replace the tables behind http://foxincloud.com/demos.php using the program below, almost every day without any dangling COM object for more than 2 years.

loHTTP = NewObject('wwHTTP', 'wwHTTP.prg')
loFTP = NewObject('wwFTP', 'wwIPstuff.vcx')

* Set server on hold
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~HoldRequests', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

* Upload customer table into server's \data folder
WITH m.loFTP AS wwFTP OF wwIPstuff.vcx

cFileSrce = DOS_INT + 'data\customer.dbf'
cFileTrgt = '/AbaqueApps/ttWeb/Data/customer.dbf'

.nConnectTimeout = 2 * 60 && réglage empirique car on ne sais pas exactement à quoi correspond cette valeur ...
.nFTPbinary = 1
llResult = .T.;
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'dbf'), ForceExt(m.cFileTrgt, 'dbf'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW));
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'cdx'), ForceExt(m.cFileTrgt, 'cdx'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW))

IF NOT m.llResult
? Chr(7)
? .cErrorMsg
ENDIF
ENDWITH

* Set server to process requests
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~Load', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

RETURN m.llResult


We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?



















-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  Rick Strahl
  Thierry Nivelet (FoxInCloud)
  Oct 30, 2014 @ 01:48am

Ok, so the unload logistics in Web Connection are fine. It's the final unload that's a problem...

Try doing an IISReset (or restart just the Application Pool) and then check wcErrors.txt. Do servers go away then?
IOW, try an easier repro case than your server auto-recycling.

I also suggest you try the latest wc.dll from V5.70 (or any recent version) just to see if this maybe was addressed since the 4.x DLL was released over 10 years ago :-)

+++ Rick ---


works as expected

Filename: C:\Temp\TT\wcErrors.txt
Clear

2014-10-30 09:26:24:017 Forced Release of Server - 0


Thierry,

I know I do the same thing and I don't have any issues with dangling references either.

Can you try the following:

* Clear your wcErrors.txt
* Manually unload servers from the ISAPI page
* Do servers go away?
* What's in wcErrors.txt

IOW, do manual unloads work or can you see that fail. Are there messages in wcErrors.txt - if server unload you should actually see the unload messages (not errors). If they're not there that means the unload code is not actually firing which seems to point to the problem I mentinoed in my last message.

+++ Rick ---



Rick,
Thanks for these clear explanations.
I think I'll remove the recycling @ 3:00 AM and see what happens, especially if the program below can stop the COM objects OK

We don't need a process list

useful at least for me to understand the sequence

This was a big problem on Windows 2000

Strange! Never ever had any problem with our previous Win 2k server - always stopped and restarted COM objects distantly to replace the tables behind http://foxincloud.com/demos.php using the program below, almost every day without any dangling COM object for more than 2 years.

loHTTP = NewObject('wwHTTP', 'wwHTTP.prg')
loFTP = NewObject('wwFTP', 'wwIPstuff.vcx')

* Set server on hold
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~HoldRequests', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

* Upload customer table into server's \data folder
WITH m.loFTP AS wwFTP OF wwIPstuff.vcx

cFileSrce = DOS_INT + 'data\customer.dbf'
cFileTrgt = '/AbaqueApps/ttWeb/Data/customer.dbf'

.nConnectTimeout = 2 * 60 && réglage empirique car on ne sais pas exactement à quoi correspond cette valeur ...
.nFTPbinary = 1
llResult = .T.;
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'dbf'), ForceExt(m.cFileTrgt, 'dbf'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW));
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'cdx'), ForceExt(m.cFileTrgt, 'cdx'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW))

IF NOT m.llResult
? Chr(7)
? .cErrorMsg
ENDIF
ENDWITH

* Set server to process requests
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~Load', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

RETURN m.llResult


We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?





















Rick Strahl
West Wind Technologies

Making waves on the Web
from Maui

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 30, 2014 @ 01:51am
hold on Rick, I'm doing some extra tests under various scenarios
I think you'll like these results

Ok, so the unload logistics in Web Connection are fine. It's the final unload that's a problem...

Try doing an IISReset (or restart just the Application Pool) and then check wcErrors.txt. Do servers go away then?
IOW, try an easier repro case than your server auto-recycling.

I also suggest you try the latest wc.dll from V5.70 (or any recent version) just to see if this maybe was addressed since the 4.x DLL was released over 10 years ago :-)

+++ Rick ---


works as expected

Filename: C:\Temp\TT\wcErrors.txt
Clear

2014-10-30 09:26:24:017 Forced Release of Server - 0


Thierry,

I know I do the same thing and I don't have any issues with dangling references either.

Can you try the following:

* Clear your wcErrors.txt
* Manually unload servers from the ISAPI page
* Do servers go away?
* What's in wcErrors.txt

IOW, do manual unloads work or can you see that fail. Are there messages in wcErrors.txt - if server unload you should actually see the unload messages (not errors). If they're not there that means the unload code is not actually firing which seems to point to the problem I mentinoed in my last message.

+++ Rick ---



Rick,
Thanks for these clear explanations.
I think I'll remove the recycling @ 3:00 AM and see what happens, especially if the program below can stop the COM objects OK

We don't need a process list

useful at least for me to understand the sequence

This was a big problem on Windows 2000

Strange! Never ever had any problem with our previous Win 2k server - always stopped and restarted COM objects distantly to replace the tables behind http://foxincloud.com/demos.php using the program below, almost every day without any dangling COM object for more than 2 years.

loHTTP = NewObject('wwHTTP', 'wwHTTP.prg')
loFTP = NewObject('wwFTP', 'wwIPstuff.vcx')

* Set server on hold
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~HoldRequests', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

* Upload customer table into server's \data folder
WITH m.loFTP AS wwFTP OF wwIPstuff.vcx

cFileSrce = DOS_INT + 'data\customer.dbf'
cFileTrgt = '/AbaqueApps/ttWeb/Data/customer.dbf'

.nConnectTimeout = 2 * 60 && réglage empirique car on ne sais pas exactement à quoi correspond cette valeur ...
.nFTPbinary = 1
llResult = .T.;
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'dbf'), ForceExt(m.cFileTrgt, 'dbf'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW));
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'cdx'), ForceExt(m.cFileTrgt, 'cdx'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW))

IF NOT m.llResult
? Chr(7)
? .cErrorMsg
ENDIF
ENDWITH

* Set server to process requests
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~Load', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

RETURN m.llResult


We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?






















-- thn (FoxInCloud)

Gravatar is a globally recognized avatar based on your email address. Recycling a wc app in IIS
  FoxInCloud Support - Thierry N.
  Rick Strahl
  Oct 30, 2014 @ 02:28am
More complete test
Can you try the following:
* Clear your wcErrors.txt
* Manually unload servers from the ISAPI page
* Do servers go away?
* What's in wcErrors.txt

Test case

2 applications:
TTweb.exe (http://foxincloud.com/) runs as 'Administrator' - 2 COM servers
FICtuto.exe (http://foxincloud.com/tutotest/) runs as 'LocalSystem' - 1 COM server

- both belong to the 'FoxInCloud' app pool running under ’LocalSystem’
- same version of wc.dll (4.67)

_maintain~Release

TTweb.exe:


- servers unload
- wwMaint~wcDLLErrorLog:

No data returned from Web Connection DLL Error File
This means either the file is empty, or the file is in use.

FICtuto.exe:


- server unloads
- wwMaint~wcDLLErrorLog:

Filename: c:\Temp\FICtuto\wcErrors.txt
Clear
2014-10-30 09:56:33:603 Forced Release of Server - 0

looks like the account under which COM object runs has an impact

App Pool manual recycling


- clear both wcErrors.txt using wwMaint~wcDLLErrorLog~Delete
- click the ‘recycling’ button in IIS’s app pool panel > 'FoxInCloud'

TTweb.exe:


- servers unload
- wwMaint~wcDLLErrorLog:

No data returned from Web Connection DLL Error File
This means either the file is empty, or the file is in use.

FICtuto.exe:


- server does not unload
- wwMaint~wcDLLErrorLog:

No data returned from Web Connection DLL Error File
This means either the file is empty, or the file is in use.

Looks like we're approaching the sweet spot …



Thierry,

I know I do the same thing and I don't have any issues with dangling references either.

Can you try the following:

* Clear your wcErrors.txt
* Manually unload servers from the ISAPI page
* Do servers go away?
* What's in wcErrors.txt

IOW, do manual unloads work or can you see that fail. Are there messages in wcErrors.txt - if server unload you should actually see the unload messages (not errors). If they're not there that means the unload code is not actually firing which seems to point to the problem I mentinoed in my last message.

+++ Rick ---



Rick,
Thanks for these clear explanations.
I think I'll remove the recycling @ 3:00 AM and see what happens, especially if the program below can stop the COM objects OK

We don't need a process list

useful at least for me to understand the sequence

This was a big problem on Windows 2000

Strange! Never ever had any problem with our previous Win 2k server - always stopped and restarted COM objects distantly to replace the tables behind http://foxincloud.com/demos.php using the program below, almost every day without any dangling COM object for more than 2 years.

loHTTP = NewObject('wwHTTP', 'wwHTTP.prg')
loFTP = NewObject('wwFTP', 'wwIPstuff.vcx')

* Set server on hold
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~HoldRequests', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

* Upload customer table into server's \data folder
WITH m.loFTP AS wwFTP OF wwIPstuff.vcx

cFileSrce = DOS_INT + 'data\customer.dbf'
cFileTrgt = '/AbaqueApps/ttWeb/Data/customer.dbf'

.nConnectTimeout = 2 * 60 && réglage empirique car on ne sais pas exactement à quoi correspond cette valeur ...
.nFTPbinary = 1
llResult = .T.;
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'dbf'), ForceExt(m.cFileTrgt, 'dbf'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW));
AND Empty(.FTPsendFile('foxincloud.com', ForceExt(m.cFileSrce, 'cdx'), ForceExt(m.cFileTrgt, 'cdx'), ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW))

IF NOT m.llResult
? Chr(7)
? .cErrorMsg
ENDIF
ENDWITH

* Set server to process requests
loHTTP.HTTPget('http://foxincloud.com/bin/wc.dll?_maintain~Load', ABQ_ADMIN_LOGIN, ABQ_ADMIN_PASSW)

RETURN m.llResult


We don't need a process list. Web Connection knows the process Id of all the COM objects that are loaded and can in theory kill them. That's what is supposed to be happening. The problem is that the module unload code is probably not firing, due to some ISAPI timing issues that can cause the module to already be unloaded by the time the code runs which means the memory's been unload and the server ids are gone. This is a bug in ISAPI that causes the module unload operation to fire after the process has terminated so in some situations the notification process for Web Connection never gets called and then the servers don't get unloaded.

This was a big problem on Windows 2000 and early versions of (pre SP2) of Windows Server 2003 - I haven't seen it personally on newer servers and heard about it only very sporadically from customers.

But again the best solution for this is to use the module which doesn't have these particular issues as it runs at a higher level and has a clearer execution and termination pipeline plus determinisitc code that kills the servers when things go out of scope.

I know that doesn't help, but I don't know what else to tell you. Maybe you can try the wc.dll from Web Connection 5.5 - there have been some tweaks since the 4.x version that might help this issue.

+++ Rick ---



could list the processes using a WMI script (http://www.winhelponline.com/blog/list-running-processes-and-their-creation-times/)

FICtuto.exe (4296) 27/10/2014 12:23:00 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4200) 27/10/2014 12:23:36 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe
TTWeb.exe (4048) 27/10/2014 16:32:34 c:\program files (x86)\abaque\ttweb\ttweb.exe
w3wp.exe (3664) 28/10/2014 03:00:00 C:\Windows\SysWOW64\inetsrv\w3wp.exe
TTWeb.exe (772) 28/10/2014 03:05:30 c:\program files (x86)\abaque\ttweb\ttweb.exe
TTWeb.exe (4492) 28/10/2014 03:05:33 c:\program files (x86)\abaque\ttweb\ttweb.exe
FICtuto.exe (4756) 28/10/2014 04:39:37 c:\program files (x86)\abaque\ficsamples\fictuto\fictuto.exe
FICdemo.exe (4868) 28/10/2014 06:02:49 c:\program files (x86)\abaque\ficsamples\ficdemo\ficdemo.exe

Looks like *.exe were not stopped when w3wp.exe (www working process I guess) has restarted.

TTWeb, FICtuto and FICdemo fire upon the first request.
TTWeb has 2 COM instances, FICtuto and FICdemo a single instance.


Not sure then...

The error that I usually see related to this comes out of wc.dll - this is coming from within Windows itself (ntdll.dll) so this looks like it might be something completely different.

Make sure your ComServers are compile as single instance too - otherwise they won't kill.

+++ Rick ---



Hi Rick,

In this case, application pool and .exe use the 'localSystem' account.

- No application error
- Nothing in the ISAPI error log ('wcErrors.txt')
- No system restart

Found an error in the system event log, at the time when recycling occurs (3:00 AM):
Nom de l’application défaillante w3wp.exe, version : 8.0.9200.16384, horodatage : 0x5010885f
Nom du module défaillant : ntdll.dll, version : 6.2.9200.17046, horodatage : 0x53b485c4
Code d’exception : 0xc0000005
Décalage d’erreur : 0x00023e30
ID du processus défaillant : 0xbe0
Heure de début de l’application défaillante : 0x01cff0c08c95ca96
Chemin d’accès de l’application défaillante : C:\Windows\SysWOW64\inetsrv\w3wp.exe
Chemin d’accès du module défaillant: C:\Windows\SYSTEM32\ntdll.dll
ID de rapport : f6940901-5d7c-11e4-93f2-00155dc79808
Nom complet du package défaillant :
ID de l’application relative au package défaillant :

Not sure what I can do next ...
Thanks for your help anyway


Make sure the account you're running the COM servers under (in DCOM Config or inherited from the AppPool if you're using passthrough) has full rights on the machine - ISAPI typically will not completely unload COM servers and wc.dll will forcefull issue a KILL operation on the process, but you do need full Administrative Rights to do this.

And yes this should be much better with the managed module because the lack of thread marshalling the process of COM management is much much simpler and thus servers tend to actually unload correctly via COM.

FWIW, I never have issues with this on my server - I run the ISAPI manager mainly because I one single server with many applications running across multiple virtuals and that doesn't work with the .NET module unfortunately.



Still having dangling COM exe with a daily recycling (@ 3:00 AM)
easy to kill with admin page, still a pain when trying to update tables through FTP (dangling instances lock files)
never happened on our previous Win2K server
hope the managed .net module will help cure that.


I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

Automatic recycling of IIS Application Pools will shut down will cause a controlled shutdow which should trigger the servers to get unloaded. The behavior of that is no different than doing an application pool stop. Just make sure your App Pool is running high enough rights to be able to kill the instances (if you're running the ISAPI component - the .NET component will exit more cleanly from within the app and that what I highly recommend is used on Windows 2008 and later).

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

Yes I think it's a useful feature especially if you're running the ISAPI component and COM. The COM system in IIS unfortunately is somewhat flakey and if there are hard COM failures/disconnects over time there are small memory leaks that end up raising memory and can over time make COM loading unstable. Recycling IIS occasionally (like once a day maybe) can drastically help with this.

This is much less important for the .NET Module as it doesn't rely on COM marshalling to manage the thread pool and as such provides a much cleaner architecture. But regardless, it seems that apps tend to grow in memory size and handles eventually and an occasional recycle to 'clean house' is useful. Again infrequent recycles are a good idea if less critical for the .NET module but I have it on all my app pools at once a day in the early morning hours.


I notice in your App Pool config you have the Idle Timeout set - you should turn that off or to a really large value - that will also shut down the appDomain if the application is not accessed. And I suspect with many low volume bus apps this actually hits more frequently than the recycling timeout.

+++ Rick ---



Hi Rick,
Thanks for your detailed feedback; please see my responses below.

It shouldn't. When the Application Pool shuts down COM instances will be unloaded. Are you sure that you're not hitting the server or there are backed up requests that are restarting the instances immediately? Check the ProcIds and see if they change after the recycle.

I inadvertently saw dangling instances on our win 2012 (IIS8.0) server.
As I just moved these apps from a win2k server (without app pool and recycling), I drew the conclusion that the automatic, timed recycling was responsible for that.
I turned off recycling yesterday and no longer see any dangling instance - I'll report in a couple of weeks whether this solves the issue.

By the way, does anyone see a real interest in this recycling mechanism for wConnect applications?
Our previous win2k server always worked fine without this feature ...

It works for me here. In fact, it works even making a change to web.config (in Module mode) which also causes an internal AppDomain restart of the application. With the ISAPI handler recycling works, but not web.config changes.

I look forward to moving FoxInCloud to WC5

Make sure your Application Pool account is SYSTEM or some other high rights account otherwise it won't have rights to explicitly kill the instances. Also the .NET Module will do a better and faster job shutting down, than the ISAPI handler which typically has to actually kill the process to remove them (due to ISAPI shutdown timing).

Yes, LocalSystem is, by default, allowed to access to any COM object (execute, activate, etc.).
You could obtain the same result with another account to which you grant the same rights by DCOM config.
In this particular case, Application Pool account is indeed LocalSystem


+++ Rick ---



Hi,

Just realized that using the IIS 7+ Web Application recycling (defined in the Application Pool) leaves VFP exe (COM instances) running.
Turned off all automatic recycling options (ss below).
Any experience about this?