I'm always been intrigued when I review official design guidelines that call for Message Queues or any Queue related technology for various messaging scenarios. Queues seem to be the preferred mechanism that you see in Service Oriented designs. For .NET there's MSMQ and there's Sql Server 2005's Sql Service Broker which is basically a queue based implementation that can be driven directly through SQL code.

I think it's great that there 'in the box' solutions to handle this for you, but I've never been able to make Message Queues fit my distributed scenarios. The problem with queues is, well that they are queues. They are optimized for first in first out operation. Which in most scenarios that I end up with just doesn't work, or at the very least works very badly. Surely there are scenarios where queues are a good fit, but I find that I rarely need that particular scenario. A FIFO scenario works good for very structured scenarios and deliver once messages, but it works very badly for ongoing and two way messaging that seems to be much more prevalent (at least for me).

In most of my messaging scenarios I do need a dedicated reliable message store that allows applications to communicate with each other, but typically they need to be able to be random access. Specifically in many scenarios there's two way messaging where one application submits a message, another application processes a message and the first eventually gets a result returned to it. Now you can do this with queues where you have two queues set up – one for each of the messagengers, but that's both inefficient and complicated to work with (Ok, not that complicated but certainly not logical).

Let me explain one scenario I constantly see. I've been working with a customer who needs to have a daemon type process that's responsible for running some heavy duty reports. These reports need to be processed on a separate machine because they are serious CPU hogs and simply don't work well on the Web Server (or other application server for that matter). Most of the reports take about a minute or two to run and are typically served over the Web with an ASP.NET front end.

So it's obviously too long to run these requests right inline of the ASP.NET Page and because of the CPU load we are in fact offloading the processing to another machine. In addition, the server application should fire back some progress information about what's going on because clients will wait for the result to return. The timing is such that it's just quick enough to let users wait for results.

The way I've built this sort of application is by essentially creating a messaging interface that allows the ASP.NET application to communicate with the Report Server. This particular implementation uses SQL Server. It stores messages in a SQL table – each request has an id and the report server and ASP.NET communicate basically through this single record in a SQL table. Each record has information about when it was started and the last update time, fields for several types of messages (a message code, and a text message, a completion percentage) and of course cancel and completion flags. It also has a binary image field that can be used to hold any results which can either be the actual result or some sort of message or pointer to the actual results. In some situations the results are shipped back as binary serialized data (such as a serialized object or DataSet) at other times it may point to a disk resource or it may be an XML Message with more information.

I posted a sample a while back with my ATLAS examples that demonstrates a simplified version of the message handler class here:

http://www.west-wind.com/presentations/scriptcallbacks/sample/Atlas/LongProgress.aspx

The sample is overly simplistic and actually processes the request in the context of the page (on a separate thread) but it uses this sort of messaging class to communicate back status information for the current operation.

It actually works well, but I keep wondering why THIS sort of scenario isn't built into the OS in some way. In fact when Serivce Broker came out I was hopeful because it was SQL based that it would support better random access stores that can be accessed by multiple clients and can retrieve message out of order. Maybe this is possible but from what I've looked at so far it looks like messages are even more strictly queue based than MSMQ.

I've built this sort of interface with queues before too. It can be done, but it's a heck of a lot more work and much more complicated because queues are not designed to be accessed randomly (although with MSMQ at least you can – MSMQ can also peek messages which SB can't as far as I can tell). In most two-way or multi-way messaging scenarios you need multiple queues, and if multiple clients are listening to response queues a lot of same data gets written to the same queue multiple times because you can read data only once.

To top it off MSMQ is not really designed for large messages. MSMQ has max Message size of 4MB – if you have messages any larger you have to split them up. 4MB seems large until you have a result you need to queue up. Splitting and recombining that message is royal pain.

So, here I am wondering if I'm the only one who's wondering if there isn't a more generic solution out there that addresses this sort of random access messaging, communication problem that results in constant redesigning of custom solutions.

And if not I guess I should publish my Messaging class <g>