Oil is under $40 as I write this, which is down from over $147 sometime in July. So isn’t it high time to go out and buy those bargain basement Hummers to start burning up this cheap oil? I’m kidding of course, but some people never learn apparently as they are falling for lower oil prices and incentives from auto-manufacturers to continue down gas guzzling alley.

It’s not all that surprising actually. Gas is now down to the $1.30 range in some of the US. Even on Maui we’re down to $2.58 (we tend to have the most expensive gas in the country), which is as low as I can remember.  With gas this cheap, it seems easy to fall back into thinking that the recent rise in oil prices was nothing more than an unnatural energy futures bubble. While some of the price hikes were caused by speculation, another just as important aspect of high oil prices was that world supply for some time was running at full production capacity in relation to demand, with barely any oil to spare and national reserves drained near record low levels. Oil prices are largely determined by the interplay of supply and demand and as we all know by now supply can be a very fickle beast where the smallest interruption can cause price hikes. The current price drop is largely due to the projection that oil demand is going to be dropping drastically due to the current economic downturn with industrial production reduced and consumer spending curbed significantly.

With the drop in oil it’s back to the old days when people forget momentarily the pinch of $5 a gallon of gas and it’s written off as just another ploy by the oil companies, or what was it this time – oil speculators. But just because oil is cheap now doesn’t mean that it can’t go back up and go back up very, very quickly just as quickly as it went down. Certainly the current level of oil prices is a lot lower than where even our historical average should be (which based on historical averages is somewhere between $60-$70 a barrel + some offset for the low dollar) so I wouldn’t expect oil prices to stay this low for long, economic downturn or not.

Should we worry?

Our society and way of life is so intricately linked to oil, that it’s hard to not worry about the stability of oil prices.

From our society – especially in the US - that is so intricately dependent on cars and trucks for all transportation needs for lack of alternatives in most parts of the country (Suburbia Hell), to our agriculture that depends on oil based products for fertilizer, pesticides as well as the fuel to power farm machinery, to the myriad of consumer items that depend on oil based manufacturing as well as the energy inputs to produce actual products. Hi tech gadgets like our computers, phones, TVs etc. – even though often small -   require massive amounts energy to produce. There are very few industries that aren’t dependent on large scale energy usage and a lot of that energy comes from oil.

Oil is a near perfect energy source – it’s (at the moment at least) ridiculously cheap for the amount of energy it contains. It’s also easy to transport and store (as it’s liquid) so this energy can be easily transported around the world without significant loss of energy (other than transportation costs). To put this into perspective consider this quote from oil industry analyst Matthew R. Simmons:

“At $100 per barrel, oil costs about 15 cents a cup. A cup of oil converted to transportation fuel such as gasoline will carry a vehicle loaded with six people about 1.5 miles.”

Try getting that kind of output from human labor or even animal labor (think feed cost) at this rate or anywhere near it. Oil is damn convenient with its high energy output and portability. And that’s even with the inherent inefficiency of engine technology which wastes 20-30% of the energy available.

We’ve lived through 150 years of continuous growth in oil production with only a few small breaks in continuity – the biggest of which was in the 70’s during the energy embargo which forcibly reduced production, which also caused a severe recession, and brought about the first efforts of creating energy efficient cars, and household utilities etc which actually resulted in reduced energy usage for an extended period.

Oil usage started picking up early in the 20th century and since than has affected just about all aspects of the growth of our modern societies. It’s also been the main driving factor for population growth as the increased productivity of human population and our machines as well as oil based agriculture has driven up the count of the human population.

The dependence of our society on energy and specifically oil as the primary energy source has become a primary driving force of growth both real and economic. Imagine what would happen if that growth was no longer supported by an adequate supply of energy to go along with it.

Pure production doesn’t tell the whole story either. For the most part in this tremendous time of growth since the early 1900’s energy demand and supply were pretty closely matched. Our society grew from a mostly agricultural base to a mostly industrial one in a matter of 100 years. With a few exceptions – the oil shocks in the 70’s specifically – energy supply and demand have been largely in sync or at least in a positive balance with supply larger than demand. But earlier last year supply was once again starting to outrun demand which in large part resulted in the run up of oil prices up until the middle of the year.

This is a scary thought: it’s almost as if there was some anticipation – although not publicly made – that oil supply would continue to run short for the foreseeable future which in turn made in  or the speculation in oil futures that only made things worse. There’s been a lot of talk about demand increasing rapidly from emerging markets like China and India with production flattening out.

If there’s indeed some credence to the concept of Peak Oil (which says that oil production will peak and thereafter flatten and slowly decline when the half way point of oil reserves are produced), then even a moderate climb in demand quickly results in a large discrepancy between the actual production and world demand for oil which gets worse every year.

Last year’s divergence and shortage was fairly minor and mostly caused by interruptions in the supply chain rather than real shortages in production, but it gives a damn good taste how quickly the price of oil can shoot up and how adversely it ends up affecting the economy. Another disturbing sign is that oil production actually peaked in 2005 – since which production has stayed flat and has actually very slightly declined. In 2008 the numbers are going to be even lower although in this year the numbers aren’t going to be driven by production shortcomings but by the drastic drop-off of demand in the latter half of the year after the economy collapsed.

But imagine if you will that energy available is reduced even by very small percentages each year, especially in light of how our economy’s growth over the years that has pretty much always been accompanied by increased energy consumption. If you remove any amount of energy output from the economy it stands to reason that this reduction will also translate into reduced economic output and activity. Energy is extremely tightly coupled with our economy and whole way of life that we’ve been used to. Most people alive in the ‘western’ world at least don’t know any different – constant growth and no shortage of energy has been a given over the last 100 years when oil has been exploited.

Peak Oil?

The current situations of plummeting oil prices pretty much flies in the face of the Peak Oil theory which takes a fairly depressing view of oil production and when production of oil reaches its apex. The idea is that production (as well as discovery which proceeds it) follows a bell shaped curve the top of which is reached when half of the oil resources have been produced.

The production curve ramps up as the oil discovered is exploited and pulled out of the ground. At first both curves climb steeply as new discoveries of previously unexplored areas occur and production can pry off the easy pickings of top level easy to access oil (and often natural gas). As time goes on discoveries slow down and finally peak as ever more sophisticated (and expensive) discovery mechanism are used to track and find resources, that eventually are ever more difficult to find. In production the curve is similar – steep rise as the easy locations and wells are tapped and then slowly become more expensive and energy intensive to retrieve the oil out of the ground. The production curve eventually peaks where a maximum amount of oil has been retrieved after which the curve flattens out and then starts an steady and unstoppable decline.

The problem with Peak Oil is that it’s not easily quantifiable. Oil production rates are known, but the amount of reserves available in the world are obscured in that most of the big producers do not allow inspection of fields to determine actual production capacities. This has resulted in speculation on when the actual peak is supposed to occur. This naturally is an easy counter argument for this phenomen since it can’t be quantified exactly, so naysayers have plenty of counter arguments to field.

However, from a purely common sense perspective the idea of Peak Oil makes sense. The problem isn’t that we’re going to run out of oil, but that the oil that is left around the half way point – which most of the proponents of Peak Oil point to at this point in time or very soon – is going to be much more difficult to retrieve and get out of the ground. Whether it’s existing oil fields that have been drained and require pressurization to bring up oil, or new fields that are buried deep in the soil or under miles of ocean water (like some of the newer oil discoveries off Brazil and Mexico) procuring that oil is more difficult, expensive and most importantly more energy intensive.

The problem with energy production is that one has to overcome the energy deficit – the amount of energy required to retrieve the resource without expending large amounts of energy. Most of the ‘easy’ oil has been produced and now we’re into the phase of producing oil that requires more energy to retrieve. This means a barrel of oil produced today has a lesser net energy value than a barrel of oil produced say in the 1950s when the US was still the leading oil producer.

It wouldn’t do to expend more energy to retrieve oil than the energy that is contained in the actual oil retrieved. We're not there yet but a lot of the ‘alternative’ fossil fuel resources like extremely deep reserves or tar sands are very energy intensive compared to the easy oil that literally came spilling out of the ground in the early days of oil exploration. Half way in exhaustion of reserves is thus much further along the way, because 100% retrieval is likely to hit the point of diminishing returns way before resources are completely used up.

Nobody can say reliably when Peak Oil will hit but certainly I feel like there will be a peak. The question is whether it’s imminent, has already happened or still a few years off. Peak oil production currently is 3 years back in 2005 – 2006 through 2008 has not touched the 2005 peak production rates. However, it’s unclear whether that’s due to production limitations or whether it’s simply been due to supply chain issues – wars, natural disasters that got in the way of getting the oil to market. 2008 has seen a drastic drop in demand and usage that in turn has resulted in cut backs in production. If the economy continues its downward spiral demand is bound to continue low for some time to come producing a dip in the statistic similar to the one in the 70’s oil shocks and following economizing of energy consumption.

Still – while these are seem like small reprieve’s from a potentially disastrous problem, this slowdown is all but delaying the inevitable – which is that oil is a limited resource and it will get more scarce no matter what, even if it takes a little longer.

One of the scariest issues surrounding this problem is that modern society is so ill prepared to deal with even  modest reduction in energy resources available. Imagine a United States with gas at $10 a gallon or more and tell me that you can see things not get out of hand very quickly. Our suburban lifestyle, our food distribution network would break down very quickly. Again think about the last year prior to last summer has resulted in massive inflation in food prices. I know my food prices both on Maui and on the Mainland in Oregon had gone up by at least 40% and often much more year over year (while the gov’t numbers lie and exclude energy and food pricing in core inflation). Imagine what further rises along these lines could do to an economy that’s already in decline and where most people can’t do much else but spend their income on basic survival needs.

What alternatives?

What sucks about the current situation is that oil has plummeted so low, so quickly just at a time when serious thinking about alternative energy usage and more importantly the viability of alternative energies was finally gaining some steam. One way or another we need to re-focus on energy alternatives to get off the sinking ship that is massive oil consumption. There’s been a lot of talk about the general issue during this election cycle, but the discussion has been superficial at best with vague hints of addressing unspecified problems. But rarely does the discussion enter any state of urgency about how critical any reduction of energy resources would be for this country both on the national as well as on the individual, personal level.

There’s no way that the folks in the national security branch of the US don’t have this on their radar – they’re just not putting out the word to the public. Our energy dependence on foreign oil is a huge national security threat to say the least especially with most of our petro dollars going to foreign countries that are not friendly towards Western countries.

Yet even with the prospect of potentially disastrous changes in the energy balances of the world there are precious few alternatives that have any chance of providing a reasonable replacement for the cheap and easy to manage energy that comes from Oil. It’s no big surprise that innovation in energy has been shunned for so long while oil has been so relatively cheap because it’s impossible to compete with it on a calorie to dollar ratio.

The alternatives are slim and worse even once industries recognize the need to move to these alternatives they are very slow to do so. Bringing out hybrid cars is a baby step in the right direction but it sure seems like too little, too late. Look at how long cycle times are between a concept and actually having a car on the road. Even if a new technology was found, the replacement cost to produce the needed cars and infrastructure wouldn’t be able to ramp up for probably a decade or more. And that’s just for cars – factor all the aspects of the economy, agriculture etc. and the thought of switching to some other energy technology is staggering to the max.

Alternative fuels are practically non-existent – nothing more than vapor ware. There’s been a big PR push towards hydrogen power, but hydrogen is a storage medium rather than an energy source so something has to get the energy into it first. Volatility and energy inefficiency make hyrdogen a non-starter. Bio-fuels have horrible energy balances – it takes nearly as much energy to produce these non-waste bio-fuels than what they produce and the only reason that they are viable at all is that the crops in question (corn/sugarcane) are typically heavily gov’t subsidized. Without the subsidy there’s no way growing current bio fuels would make any sense. Nuclear power is often mentioned as an alternative and while it can produce a lot of power there’s of course the disposal issue plus the fact that large scale use would eat up all available uranium with a 20-30 years – it’s not a long term solution. Renewables like solar and wind have potential for small scale localized applications, unless adopted in large scale where every household and building becomes a part in a solar grid. Solar is getting very popular as a renewable, but currently it’s still so inefficient (with the best panels converting less than 40% of captured energy – with common more around 25%) and at the same time expensive. Solar equipment is also extremely environmentally unfriendly in production requiring much of the same processes used for computer/hardware production – it has a tremendous energy footprint that has to be made up.

Almost any alternative power proposed ends up having to do with electricity. But  electricity alone is not a complete replacement for what liquid fuels provide today. Heavy machinery – trucks, industrial machines, large airplanes etc. are not likely to run on electricity alone. Pure electricity also has the problem of not being very portable – the longer it has to be routed the more signal loss there is reducing the overall power output due to travel from site of generation to site of use. Storage of electricity is also a problem as power stored in batteries slowly drains and storage mediums available currently are very innefficient both in charging and then releasing energy resulting in significant power loss. Electricity use makes most sense near the source.

All in all the picture for alternatives look slim, yet it seems we – as a society – are relying on technology to save us from a potential energy shortage. If there is innovation in this space it’s happening behind closed doors and in baby steps. I doubt that if there was some great solution out there – even if it was in a pre-production stage - that we wouldn’t know about it. If it was private industry somebody would want to captilize on it. If it was the gov’t – somebody would want to take credit for it. :-}

It’s safe to say that if we run into problems with energy availability there’s not going to be one single technology that will help us out of the situation – it’ll have to be a combination of strong energy preservation and a combination of alternate energy technologies combined. But even then it will be hard to make up for the shortcomings that are likely to occur through oil shortages.

The problem is once such an energy shortage hits it will become that much harder to find a solutions to the energy problem as resources and any sort of funding will become less and less available in a declining energy and economic environment. Our best chance is to have some sort of Plan B in place before that happens, but it looks like this problem whether it hits soon or 5 years from now will catch us unprepared.

All of this may sound alarmist and unfounded and heck maybe it is. As mentioned above, the theory of Peak Oil can’t be pinned down exactly without reliable reserve numbers that are not available. But you can certainly search around on the Internet in relation to Peak Oil and find plenty of validated and scientifically backed analysis of this concept by many varied sources some of which would seem to have a conflict of interest even. The material is available, but in the end it is a very negative scenario and as we all know politicians are afraid of being harbingers of bad news. The other thing is that the Peak Oil concept has proven itself in many smaller scale scenarios of specific oil producing regions. The US was the prime example that M. King Hubbard (the originator of the Peak Oil theory) himself  used and predicted fairly accurately. There’s the North Sea example, Mexico, Russian production all of which are now in decline and all of which followed the Hubbart curve model.

If you want to find out more about peak oil there are a number of books and videos available. Here are a few I’ve looked at over the years that are worth mentioning (yes there’s also a lot of complete whacko stuff out there so reader/viewer beware holds).

The Party's Over: Oil, War and the Fate of Industrial Societies by Richard Heinberg New Society Publishers (April 1, 2003) Read more...

There are also a few videos you can check out including the 2006 Crude Awakening:

A Crude Awakening - The Oil Crash by n/a DOCURAMA Read more...

and The end of Suburbia:

The End of Suburbia: Oil Depletion and the Collapse of the American Dream by Matt Simmons, Colin Campbell, Steve Andrews, Michael Ruppert, Michael T. Klare Microcinema DVD Read more...

Both of these are not very recent and fairly alarmist, but they both provide the background on Peak Oil and discuss some of the issues at hand. Crude Awakening focuses more on the general issue of peak oil and where we are at in the world currently (or as of 2006 anyway), while The End of Suburbia is much more speculative and aggressively argues through some of the potential effects an end of oil scenario might have. You may not agree with the ideas and conclusions especially of the End of Suburbia, but both videos are interesting to start thinking about the what if implications that an end of oil scenario engenders.

What me worry?

I’ve long held the belief that our current path of development on this Earth is simply not sustainable. We can’t expect limitless growth to keep going forever and not to run out of the resources available on this finite planet’s resources. Additionally our use of conniving financing practices and debt to finance consumption and public expansion is only accelerating the dramatic explosion of populations and resource use of the world. Warnings about hitting some of our limits in expansion and never ending growth seem only reasonable. It’s happened many times before in history. Check out Collapse by Jared Diamond for some interesting case studies on how pissing into your own backyard doesn’t usually work out so well, even when in hindsight it only seems so extremely obvious with reality staring you in the face. We’re much on the same path today except that now we’re looking at it on a global scale for the first time.

The idea that oil resources running out are going to cause major changes in the balance of the world is the understatement of the millenium. The question is just whether this is something we really need to worry about or not at this moment in time. I think that some healthy skepticism at the very least is in order because a lot of the signs are pointing in the direction that there is a problem brewing…

For me personally I’ve thought about this quite a bit for some time which is why I’m writing about it here in this personal message on my blog. I think that while we can’t be sure whether there is an immediate problem in resource and energy shortages on the planet it stands to reason that at some point in the not too distant future we are going to be facing these issues head on. It seems incredibly childish not to address this possibility now while there is still time to soften the blow that a potential energy shock would have.

Think about it, even if you do nothing more than a little thought experiment with yourself. How would you live if you had to make do with a more primitive society that doesn’t run on power or power that is treated as a luxury rather than an abundant ever available resource as it is now? Or even one that doesn’t run with private cars? Do you live in the suburbs with no way to even get to a store by foot and no public transportation? How will you get to work if your job is in the city that’s 20, 50 or 80 miles away? Will you even have a job? In a drastically shrunken economy that has paid a heavy energy tax that is bound to deflate any economic growth that isn’t likely to come back, do you think you’ll still have a paper pushing job? Or an abstract job like programmer for example? How do you code when there’s no consistent electricity and which business would still need abstract work.

Once you start on this path of thinking there are a million other little things that pop up that make you realize how much of our lives depends on technology and most of all power to get us through the day. If nothing else, it might make you more energy aware, to show you how you are using energy in your daily life and you can possibly reduce your energy footprint.

The thoughts are not pleasant to say the least. Most of us have grown up in a society that has always been growing  – always with the assumption the next generation will have it better than the last (until fairly recently when we’ve flattened out and even have reversed the growth trend). Many of us have never worked with our hands. Instead we sit in offices or park our butts in front of computers and push paper and bits around rather than producing something tangible that has material value. Ask yourself, what would you do if you couldn’t do this sort of abstract work? What other skills do you have? Could you keep yourself alive self-sufficiently for any period of time without the comforts of modern day life – without electricity? Do you know how to build something, grow food? And would you die of boredom (hardly – survival tends to keep you awake?) not knowing what to do with out a computer, TV or video games to drain your brain in?

These are some uncomfortable questions at least to me, because I know the honest answers I have for them are not the way I would like them to be. Yet even a few years after thinking about this I’ve not been motivated enough to do a heck of a lot about it although it continues to be on my mind. Procrastination or lack of motivation – or to some degree seeing nobody else being the least bit concerned makes these kind of thoughts feel like ‘crazy thoughts’ at best. But they nag nevertheless.

Cody Lindley has a great post that explains some of the innards of the jQuery object returned from selector queries. As you may know if you are using jQuery, the jQuery object returned from a jQuery selector expression is an object that acts like an array, but is in fact an object with numeric properties that masquerade as an array. If this sounds a little confusing you’re not alone. Even though I had figured this out some time ago, every time I talk about jQuery I seem to get my terminology mixed up calling it the jQuery object and jQuery array interchangeably <snicker>... which is not correct – it’s definitely an object.

Anyway, check out Cody’s Post on the Learning jQuery site which explains in some detail how the jQuery object is constructed and how it is that you can treat an object like an array which is effectively a neat hack that might be useful to you in other JavaScript situations.

BTW the Learning jQuery site is a good site to keep bookmarked and RSS hooked since they frequently cover interesting content that explains some of the inner workings of jQuery… highly recommended.

In the last few months I’ve been experimenting around with watching streaming video online for movies and TV shows. I’ve been highly skeptical towards online streaming  and the hype that has preceded it for so many years, but I also have a few friends who love using streaming content. Given that nowadays you can rent videos and have them delivered to your door within a couple of days makes the effort of trying to stream a video and tie up your computer and internet connection seemed kind of silly of me. Still, it seems the technology to for online streaming has gotten a lot better in the last couple of years and video has become much more commonplace on the Web albeit mostly for lo bandwidth usage ala YouTube. So I started playing around with streaming video initially with Netflix streaming player while I was travelling in the last year. For the most part I’ve only been using two services Netflix and Hulu.com.

Netflix Online Streaming

I’ve been a NetFlix subscriber for a few years now and I’ve been very happy with the service they provide. I’m always amazed by the turnaround that manages to get videos returned back in two days from returning a previous video. I was also pleased to see NetFlix starting to offer streaming content with the Watch Instantly service. Although the selection for downloadable content is not anywhere near their full video collection, the selection is fairly decent (12,000 titles they say). What’s nice about the service is that Netflix makes it really easy to select movies add them to your queue and then play them on demand – usually once I start running out of videos rented physically.

The new Silverlight Netflix video player (currently in beta and an opt-in option until it gets released) is pretty slick and a big improvement over the old Flash based viewer in several ways. The UI for the player is much more flexible and more importantly it doesn’t suck CPU resources nearly as much as the old Flash player did.

Video quality steps up and down, depending on bandwidth available, which is actually unfortunate. There seems to be no option to determine your desired setting – Netflix does and can do so mid watching which really sucks when it happens and you get down throttled from TV quality 480p output back to whatever low bandwidth settings they use that pixelate heavily.

Hulu.com

Another site I’ve been using quite a bit is Hulu.com. Hulu provides streaming video content of a decent selection of movies and a ton of current TV serial content. Many currrent TV Shows are available on Hulu and you can watch them with only brief and bearable advertising breaks. For TV content – if you’re into that sort of thing - Hulu is hard to beat. Given that these days many TV shows provide higher quality content than the crap movies Hollywood is pumping out that’s not as far fetched as it once seemed to me.

Hulu also has a pretty nice flash based player that you can customize in a number of ways.

Quality is excellent in 480p mode but really sucks in the lower res mode as both video and sound quality are drastically reduced. The sound is actually more annoying than the lower video res in the low res mode.

One thing that’s a bummer about both Netflix and Hulu is that they cannot be accessed from out of country (or at least not easily and without IP spoofing). Due to licensing rules I suspect, but still a pisser – it sure would be nice to tap into online Netflix content while I’m out of the country but no such luck. Interesting side note: I’m in Austria at Markus Egger’s right now and he’s using XBox Live’s Netflix hookup from here to view streaming content. Apparently XBox gets around the international limitation when you sign in with a US account at least.

Other Video Sites

There are a few other sites I use occasionally. There’s obviously YouTube ,Google Video, Yahoo Video and MSN video which provide avenues for video sharing for everyday users as well as lots of current affairs and educational content. I like Google’s service as it has the advantage of allowing for longer videos vs. YouTube’s limited length videos that are limited to 10 minutes. Both sites really don’t fall into the same category as the others above – they are not for watching movies but for sharing video. Resolution is also much lower so video playback is never really a problem although quality is pretty weak even in the higher resolution modes offered by some videos.

Video is also becoming ever more popular for news and current affairs content. Most news sites and especially traditional news TV sites provide much of their content online. I for one have been pleased with this development as I travel quite a bit especially out of country and it’s nice to have access to US news source abroad as well as European sources when I’m in the US. Most news content is of surprisingly good video quality and runs reliably as well.

Then there are some kinda shady sites that offer movies like Watch Movies that provide links to movies including very recent ones. Shady – yeah definitely. You can watch movies that aren’t even out on video yet, and stored on diverse video sites under obscure names. I’ve checked this out mainly because a few people mentioned it to me – especially typically non-computer folks and kids – and I went just for novelty value and sure enough there’s lots of movie content of recent movies in all sorts of qualities from hand held bootlegs to fairly decent small screen quality. This is probably not worth anything except for novelty value, but not much beyond that.

Another thing I’ve noticed is that there are a lot of of YouTube like clones are popping up that provide specialized video content for a specific focused area of interest. Video content is getting much more popular all the time and content is popping up all over the place and in different contexts which is cool. As cool as all that is though it’s actually not as easy to find that content easily. Because it’s scattered all over the place and there is no official tagging mechanism with each site using its own search mechanism finding what’s available is a problem.

A few thoughts on High Quality Streaming

Having the ability to stream high quality video for watching movies like Netflix and Hulu is pretty cool mainly because it gives you choice. I tend to be a random TV watcher – usually I watch a movie or a show after I’m done at night to wind down. Having a physical DVD to watch is Ok, but you get to watch what you got home right now. With digital delivery the choice can be made based on how the mood strikes me and that’s a good thing. It’s also good to have access to some content for reference or be able to revisit is – useful for political documentaries I frequently watch.

But even though there’s a certain cool factor to watching full hi-res video on your laptop computer or piping the output from laptop to the big screen tv in the living room, the process of doing that is still pretty painful. Browsing for movies on these sites using the Web Browser and two screens is a pain in the ass if you’re watching on the big screen TV because the browsing and watching experiences are so separate.

Netflix is addressing this with software that’s getting built into many hardware devices that work directly with your TV. There’s the Roku console who’s sole purpose is to work with Netflix, and there are devices like a host Blue-Ray Net capable devices (Samsung, LG )that have network connections and can interface with the streaming service to pick and play movies directly from hardware you already have in the living room.

Using a computer in the living room might seem cool for the novelty of it, but realistically it is still a pain. So unless you have a full hardware setup that’s specific to a media center purpose it’s less than ideal as it fails the remote control UI test in my book.

Bandwidth and Service Issues

As nice as it is to watch digital content for me the experience has been a choppy one even on a fairly high bandwidth connection. Here on Maui, I have Roadrunner cable with what they call a 5 megabit connection and while I’ve never seen that kind of throughput it looks like 3-4 megabits, which is pretty good especially given the remote island gig and all (and that we had NO high speed at all until 2003!).

Even with this connection though, when running in 480p mode it’s barely enough to keep the video buffer stuffed. I’m not sure if the problem is on the inbound end or the outbound end, but it’s either way high definition streaming is choppy at best. Netflix is much worse in this respect as it frequently down shifts me into a lo-res mode and I have no control over this process. Hulu stays in whatever mode I selected, but it too stutters and loses frames from time to time.

It would really be nice if these streaming players would give you an option to buffer data for video up front – even allowing for downloading the entire movie up front possibly so that some of the choppiness could be mitigated, but alas the buffering is only real time so any network hiccup screws up playback.

Also if I put a strain on the bandwidth elsewhere on the network (even as trivial as checking email) and video play will definitely hiccup. For a typical internet connection the bandwidth is still not quite enough, unless you go for the gold plan of super high-bandwidth that some providers now offer (not in my area though <s>).

What’s also problematic is that recently many Internet Service Providers have started putting limits on the amount of bandwidth you are allowed on your ‘unlimited’ Internet access accounts. Video certainly doesn’t help bandwidth usage. When I started playing around with this stuff a few months back and watched a ton of online content after I returned from Europe I got a warning notice from RoadRunner that I went over some unspecified limit. Looks like more providers are going that route and that doesn’t bode well for video on the wire.

Incidentally, other than video my Internet Service is solid and plenty

CPU Usage

Also both Netflix and Hulu are major CPU hogs. If you look at the Hulu screen shot you can see process manager running and it’s got 1 of the dual core processors nearly pegged at 100% and the machine is audible running in overdrive. The CPU usage for Flash apparently is so high that on an older Pentium M class machine 480p conversion doesn’t play smooth at all with constant dropped frames which are apparently not due to the bandwidth alone as it plays smoother on my dual core machine. Heck even in lo-res mode the older machine can barely keep up even though it has a reasonably decent video card. Ouch.

The Netflix Silverlight player fares a bit better, but CPU usage is still pretty high in the upper 70’s% range on the core it runs on.

Content is lacking

The final road block is that although content is rapidly expanding, when it comes to commercial video online you’re bound to have a much more limited choice online than you get with DVDs from a place like Netflix. For example, Netflix’s on demand library is a fraction of the full DVD library which is surprising. Hulu.com’s library is frustrating in its incompleteness especially for serial content – usually you can view only most recent content back for about a year. The truly universal library that provides access to most or all video content doesn’t really exist yet, but that’s really the siren call for online, ondemand video streaming.

Ultimately this point is the biggest strike against online streaming I think – selection above all wins! This all comes back to the movie industry’s fear of digital distribution I suspect, copy protection and byzantine rights management agreements which are the most likely culprit that stand in the way of broader distribution. Bummer.

Bottom Line

The bottom line for me is this: Streaming video in hi-res with either of these sites uses tons of bandwidth and computer horsepower. As far as we’ve come with video on the computer, it still looks like the sweet spot is another generation of hardware and bandwidth away at least for mainstream computer hardware. Combined with the short

At the moment I wouldn’t want to trade in my DVD rentals for only online even if Netflix actually had all content available online, but still it’s interesting to see how far we’ve come. The new apps look and work slick but it feels like we need another generation of bandwidth improvements and video hardware improvements to make this all work a little smoother. Small scale video ala YouTube and News Feeds seems Ok right now – reliable and smooth, but the full screen hi-res movie experience is not there yet.

So what about you? What video services are you using if any and what’s you’re experience been for quality, bandwidth usage and limits from ISPs? I’d be curious to hear especially from folks who have those really high bandwidth accounts.

Ah thought I go back to basics today for a post. An operation I rely on a lot when I’m working with text is the ability to extract strings with delimiters from within another string. RegEx can be quite useful for many scenarios of finding matches and extracting text, but personally for the 10 times a year when I need to use RegEx expressions I tend to relearn most of the cryptic syntax each time. What should take me a couple minutes usually turns into a half an hour of experimenting with RegEx Buddy for me.

So a long time ago I created some helper functions that make it easier to help me with the common task of string extraction. One of them is ExtractString() which is something I use quite frequently to well extract a string from within another string. This can be in small strings or larger strings. The function accepts a source string and a couple of delimiters around which text gets extracted.

Usage looks something like this:

// json Date format is "\/Date(9221211)\/"
jsonDate = StringUtils.ExtractString(jsonDate, @"\/Date(", @")\/");

which extracts the only the milliseconds portion of the string inside of the date delimiter. Now here’s actually a good example of why this function is easier than using RegEx – try constructing that particular pattern with RegEx escape codes for some real fun and undecipherable gobbledygook. :-} Been there done that (for the JavaScript portion of the parser and it took a while to get right!).

ExtractString() also has a few additional parameters for case sensitivity and how to behave if the end delimiter is not found which is useful. For example in the following snippet a query string is passed and I need to find the value of value r parameter:

string res = StringUtils.ExtractString(Url,"?r=","&",false,true);

And no, before you ask: I couldn’t use Request.QueryString() because the string in this case is a parsed URL that doesn’t come from a query string input. The function call searches for a r= and & as delimiters, does so case insensitively (false) and specifies that the “&” delimiter can be missing which means the result returns up to the end of the string. In fact to create a generic query string parser you could now write:

public static string GetUrlEncodedKey(string urlEncodedString, string key)
{
    string res = StringUtils.ExtractString(urlEncodedString, key + "=", "&", false, true);
    return HttpUtility.UrlDecode(res);
}

Again RegEx can solve this fairly easily as well, but doing or’d  expressions are a pain to get right and since I use this functionality all the time I don’t want to fuck around with RegEx’s constant relearning curve (for me).

I certainly prefer:

string res = StringUtils.ExtractString(Url,"?r=","&",false,true);

to:

string sourceString = "ScriptCompression.ahx?r=ww.jquery.js";
Match match = Regex.Match(sourceString, @"r=(.*)(&|$)",RegexOptions.IgnoreCase);
string res = match.Groups[1].Value;

string expected = "ww.jquery.js";
Assert.AreEqual(expected, res, "Failed to extract string with missing end parameter");

I’m not saying that RegEx hasn’t got a place. Obviously for more complex searches RegEx is much more flexible. But for simple repetive scenarios RegEx is overkill and likely more resource intensive than a simple function that finds a couple of string indexes and extracts a substring. And to me at least RegEx usage seriously affects code readibility so if I have a chance to abstract behavior in such a way that I don’t have to use RegEx I will do so vehemently – either by wrapping the RegEx code or creating an alternative.

Anyway, here’s the implementation of ExtractString with various overloads:

/// <summary>
/// Extracts a string from between a pair of delimiters. Only the first 
/// instance is found.
/// </summary>
/// <param name="source">Input String to work on</param>
/// <param name="StartDelim">Beginning delimiter</param>
/// <param name="endDelim">ending delimiter</param>
/// <param name="CaseInsensitive">Determines whether the search for delimiters is case sensitive</param>
/// <returns>Extracted string or ""</returns>
public static string ExtractString(string source, string beginDelim, 
                                   string endDelim, bool caseSensitive, 
                                   bool allowMissingEndDelimiter)
{           
    int at1, at2;

    if (string.IsNullOrEmpty(source))
        return string.Empty;

    if (caseSensitive)
    {
        at1 = source.IndexOf(beginDelim);
        if (at1 == -1)
            return string.Empty;

        at2 = source.IndexOf(endDelim, at1 + beginDelim.Length);
    }
    else
    {
        //string Lower = source.ToLower();
        at1 = source.IndexOf(beginDelim,0,source.Length,StringComparison.OrdinalIgnoreCase);
        if (at1 == -1)
            return string.Empty;

        at2 = source.IndexOf(endDelim, at1 + beginDelim.Length, StringComparison.OrdinalIgnoreCase);
    }

    if (allowMissingEndDelimiter && at2 == -1)
        return source.Substring(at1 + beginDelim.Length);

    if (at1 > -1 && at2 > 1)
        return source.Substring(at1 + beginDelim.Length, at2 - at1 - beginDelim.Length);

    return string.Empty;
}

public static string ExtractString(string source, string beginDelim, string endDelim, bool caseSensitive)
{
    return ExtractString(source, beginDelim, endDelim, caseSensitive, false);
}

public static string ExtractString(string source, string beginDelim, string endDelim)
{
    return ExtractString(source, beginDelim, endDelim, false, false);
}

It’s pretty basic stuff, but it’s one of those handy utility functions that are useful so frequently it’s nice to have them in my Utility set of classes (StringUtils to be exact).

So I’m curious to hear what the RegEx aficionados will have to say. I wonder if RegEx could be used inside of the function – the big issue I always have with RegEx in reusable/generic scenarios is that patterns that contain the start and end delimiters have to escaped properly. If I have backslash in the starting delimiter a generic expression will die the painful RegEx parsing death. So (he asks mockingly) - could the generic internals of ExtractString be re-written to use RegEx instead of indexOf and SubString to retrieve the string match – frankly I don’t know how (and it’s not really necessary, but an interesting thought).

BTW, for some of the FoxPro folks lurking this function might look familiar. It closely matches West Wind Web Connection’s Extract method as well as VFP’s native StrExtract functions on which I had relied for many years. I think the above functions was one of the first things I ever created in .NET because its use was sorely missed when starting out in .NET.

Hopefully this method will be useful to some of you.

As a publisher of a Help Creation tool in Help Builder, I’ve seen a lot of problems with people not being able to run their help files properly. Here’s the scenario: You go ahead and happily build your fancy, schmanzy Help File and deploy it to your customer or let your customers download them off the Internet directly.

The customer downloads the file, opens the zip file and copies the help file contained in the zip file to disk, then opens the help file and finds the following help file:

...a help file that comes up with all topics in the tree on the left, but a Page Not found or Operation Aborted error in the IE content window; an IE error. The CHM file obviously opened since the topic list is there, but the Help Viewer refuses to display the content. Looks like a broken help file, right? But it's not - it's merely a Windows security 'feature' that tries to be overly helpful in protecting you.

The reason this happens is because files downloaded off the Internet - including ZIP files and files cotained in those zip files - are marked as potentially malicious and so do not get full browsing rights on the local machine – they can’t access local content, which is exactly what help topics are. If you look at the URL of a help topic you see something like this:

 
mk:@MSITStore:C:\wwapps\wwIPStuff\wwipstuff.chm::/indexpage.htm

which points at a special Microsoft Url Moniker that in turn points the CHM file and a relative path within that HTML help file. Although it looks weird this still equates to a call to the local computer zone, the same as if you had navigated to a local file in IE which by default is not allowed.  Unlike IE you don’t get an option to allow access via the yellow header security bar that IE usually shows to allow overriding or setting permanent security options for local content.

Instead you get a page like above or one that says that Navigation has failed.

How to Fix This

The easiest way to fix this issue is to set properties on the CHM Html Help file and allow it to be opened right when you download or open it for the first time:

There’s more detailed information of the issue and links to Microsoft KB articles as well as some nasty workarounds for this issue here: http://www.helpscribble.com/chmnetwork.html. Unfortunately on newer versions of Windows you won't even see this dialog - you'll get straight into the help file but with the topic not displaying and no hint as to what's going on.

Another option is to right click on the file's properties and 'Unblock' the file so that the security block is bypassed:

This latter approach is probably the better choice as it's not a global setting but specific to the particular help file. Once set Windows will display the help file properly and without any extra prompting for security. However, if you do the latter you have to do it to each file individually.

It's annoying as all hell that this sort of obtrusive marking is necessary, but it's admittedly a necessary evil because of Microsoft's use of the insecure Internet Explorer engine that drives the CHM Html Engine's topic viewer. Because help files are viewing local content and script is allowed to execute in CHM files there's potential for malicious code hiding in CHM files and the above precautions are supposed to avoid any issues.

I’ve been getting a lot of questions in the last few months regarding the URLs that WCF REST uses. WCF REST – new in .NET 3.5 – provides a pure Web based Binding (webHttpBinding specifically) that allows for creating REST based URLs that return non SOAP Http results in Xml, JSON, RSS, Atom or raw binary formats. One of the nice things that you can do with WCF REST endpoints is that you can specify a UriTemplate that allows you to customize the way the URL looks when accessing the endpoint.

For example I might have a method endpoint defined like this:

[OperationContract]
[WebInvoke(
    Method = "GET",
    ResponseFormat = WebMessageFormat.Xml,
    BodyStyle = WebMessageBodyStyle.Bare,
    UriTemplate="Quotes/{symbol}")  
]          
StockQuote GetStockQuote(string symbol);

Normally a WCF endpoint is accessed through the .SVC file plus a path that specifies the method name plus any parameters specified on the query string (or in POST data). Without the URI template to load a stock would look like this:

http://localhost/wcfAjax/RestStockService.svc/GetStockQuote?symbol=MSFT

Using the UriTemplate the URL becomes a little nice and more readable:

http://localhost/wcfAjax/RestStockService.svc/Quotes/MSFT

Note that once you apply a UriTemplate the method/parameter syntax no longer works – only a single endpoint URL maps to the specific endpoint.

Url formatting is of course mostly semantics – the latter URL is easier to parse by a human consumer having more of the feel of a breadcrumb that leads you down the path. You can also create your own ‘rules’ for this so if you also wanted to return that same data in a different format like Json (where the original returned XML) you might want to set up a second endpoint that does this and looks like this:

[OperationContract]
[WebInvoke(
    Method = "GET",
    ResponseFormat = WebMessageFormat.Json,
    BodyStyle = WebMessageBodyStyle.Wrapped,
    UriTemplate = "Quotes/{symbol}/json")
]        
StockQuote GetStockQuoteJson(string Symbol);

and can be called like this:

http://localhost/wcfAjax/RestStockService.svc/Quotes/MSFT/json

WCF REST allows for some welcome URL flexibility which is a nice feature when dealing with REST based endpoints.

One important limitation with UriTemplate to keep in mind that parameter mapping ( inside of {} ) only works with string parameters, so even simple type mapping like int, bool, or dates don’t work with them.

Getting rid of the pesky .SVC Extension

One of the first questions I usually get when I talk about the WCF REST features usually is: How do I get rid of the .SVC extension in those URLs? Back to semantics, eh? If you look at the URLs above you notice that each of the URLs references the .SVC file including its .SVC extension. The UriTemplate defined on the WCF endpoint method specifies the a path that follows the .svc file, so whatever template you set up it is appended to the main service URL which ends in the .Svc file.

Here are two approaches you can use to remove the .svc extension and both of them work through Url Redirection at the server level.

• In IIS 7 You can use the IIS 7 Url Rewrite Module
• In prior versions you use IIS Wildcard Mapping and an HttpModule

IIS 7 Rewrite Module

A couple of months ago Microsoft released the IIS 7 Rewrite module which allows you to easily and declaratively define Url Rewrite rules in your Web.config file. The module provides RegEx based replace syntax and is ideal for fixing up .SVC urls. What’s nice is that it also includes a nice UI for the IIS 7 Management console so that you can visually see all rewrite rules installed and even experiment and test them out.

To get a URL like this:

http://localhost/wcfAjax/Rest/RestStockService/Quotes/MSFT/json

You can use the following rule definition:

<?xml version="1.0" encoding="UTF-8"?><configuration>
    <system.webServer>
    <rewrite>
        <rules>
            <rule name="RestStockService" stopProcessing="true">
              <match url="^rest/reststockservice/(.*)$"/>
              <action type="Rewrite" url="rest/reststockservice.svc/{R:1}" />
            </rule>
        </rules>
    </rewrite>
    </system.webServer>  
</configuration>

You use RegEx expressions to match the service Url without the .svc extension. The RegEx expression is based on virtual directory as its base path – it’s important that you consider any subdirectory paths in the URL. As you can see my service lives in a Rest folder underneath the virtual directory root and that’s reflected in the match expression for the URL searched for. Note the RegEx group after the service name – an explicit group is used to capture the entire extra path of the URL so it can be used in the rewrite operation. You can reference any of the group matches as {R:1} for the first group match ( {R:0} is the entire match).

This is a pretty painless way to do URL routing that’s easy to modify declaratively and store in a configuration file.

You can download the IIS RewriteModule here:

http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1691

I’d highly recommend this module even if you’re not using it for WCF REST Urls. It’s been really useful to me in setting many URL rewriting tasks that I previously used custom modules for.

Custom Http Module

The IIS Rewrite module only work in IIS 7 only unfortunately so if you’re running IIS 6/5 or you need more sophisticated URL routing than RegEx expressions can provide you can create a custom module.

IIS 5/6 requires Wildcard Mapping

If you’re not running IIS 7 in order for this to work you need to enable Wildcard Script Mapping and map the wild card map to ASP.NET. Wildcard script mapping sets up all URLs to be routed through the ASP.NET pipeline so every resource – static or otherwise - is fired through ASP.NET so you can run HttpModules against them. This allows even extensionless Urls to (like http://localhost/wcfAjax/Rest/RestStockService/Quotes/MSFT/json ) fire through your ASP code and hit your modules.

Once you have requests firing into the ASP.NET pipeline you can then set up an HttpModule that reroutes requests. The following is a somewhat generic .svc rerouting mechanism where you can just specify a part fragment that is treated as a service.

/// <summary>
/// Redirect module that allows specifying a set of .svc urls
/// by stripping the svc extension off and accessing without it.
/// 
/// To use add any non-svc path segments (ie. service.svc should be service)
/// to the ServiceMap below.
/// 
/// Note that any path that uses one of these service map entries needs to 
/// end with a trailing backslash.
/// </summary>
public class  ServiceRedirector : IHttpModule
{

    static List<string> ServiceMap = new List<string> 
        {
              "reststockservice", 
              "ajaxservice",  
              "jsonstockservice"
        };        

    public void Dispose()
    { 
    
    }

    public void Init(HttpApplication app)
    {       
        app.BeginRequest += delegate
        {
            HttpContext ctx = HttpContext.Current;
            string path = ctx.Request.AppRelativeCurrentExecutionFilePath.ToLower();

            foreach (string mapPath in ServiceMap)
            {
                if (path.Contains("/" + mapPath + "/") || path.EndsWith( "/" + mapPath) )
                {
                    string newPath = path.Replace("/" + mapPath + "/", "/" + mapPath + ".svc/");
                    ctx.RewritePath(newPath, null, ctx.Request.QueryString.ToString(), false);
                    return;
                }
            }
        };
    }
}

The idea is that you provide a list of Urls (statically in this case but you could easily modify this to load the list dynamically either from config settings or some external source) that are to be routed as simple strings. In short the names of the URL that you want routed without the SVC extension. This code still assumes you want to route to the same name as the SVC file (ie. /reststockservice/ becomes /reststockservice.svc/).

Other than that the code just runs through list and and tries to match it to the current URL. If found it replace the match by adding the .svc extension to it and off it goes.

To hook up the module you’ll need to add it to your web.config:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.web>
        <httpModules>
            <add name="ServiceRedirector" type="WcfAjax.ServiceRedirector"/>
        </httpModules>
    </system.web>

    <system.webServer>
        <validation validateIntegratedModeConfiguration="false" />
        <modules>
          <add name="ServiceRedirector" type="WcfAjax.ServiceRedirector"/>
        </modules>
    </system.webServer>

</configuration>

The <system.web> section is for IIS 5/6, <system.webServer> for IIS 7. The module’s code is a pretty simplistic approach that uses simple string replacement to expand non-.SVC Urls to .SVC urls, but it should give you enough of an idea on how to create something more flexible should you need it. This approach works both in IIS 5/6 with the wildcard script mapping or in IIS 7 where no mapping is required as all requests already fire through the ASP.NET pipeline.

While it’d by nice if WCF had a  native mechanism for removing the .SVC extension from URLs, you can see that’s it’s pretty straightforward to do so either with the IIS 7 Rewrite Module or by creating a small custom module.

Does it really matter?

I’m not sure if changing a Url by removing an extension is really providing any benefits, perceived or otherwise. To me this is semantics. Sure a Url without the .svc looks nicer, but who cares? For the most part COMPUTERS not people access those Urls and computers have no sense of esthetics <s> at least last time I checked. And if it’s an API, it’s still only a developer who looks at it once and surely a developer can determine the difference between a URL with or without an extension buried in the extra path.

I think what it really comes down to is having a clean URL structure that's natural to follow. Certainly an API that is consistent will be easier to learn for a developer to use and easier to work with in the long run. Whether stripping of an .SVC extension helps that cause or whether you use a query string vs. a path separator is arguable at least in my mind. The biggest concern for an API is consistency - it should be logical, readable and most of all consistent. And frankly query strings are the easiest way to be consistent because they are non-positional unlike path separators which are.

However, as I mentioned at the outset of this post – a lot of people actually seem to care quite vehemently about the Url format, because it’s easily one of the most frequent requests I have heard since I started writing/speaking about WCF. Hence this post.

So, what’s it to you? Does the URL matter and if so why?

Last month I inherited an old laptop – an Averatec 2100 for selling which was to be put on Craigslist and sold for a few bucks. Well, it didn’t work out that way. It’s been a journey and I think eventually this thing is bound for the dump after all rather than going to a needing home.

It all started innocuous enough with a new computer being bought to replace this the Averatec. The Averatec is going on 3 years and it was actually quite useful for the person who’d used it mostly for email, web surfing, basic image editing, writing and little more. It’s small and lightweight, albeit a bit underpowered. When the call was made to ‘dispose’ of the machine, I offered to reformat the drive and reinstall Windows because that’s the main reason the machine had started to slow down and feel like an overweight sludgy snail. I figured re-installing would have been easier than selectively wiping data off and uninstalling apps and try to get the machine to run at reasonable speed.

Big mistake!

The issue is that the machine never came with any disks. Everything was – and I emphasize WAS – online and was meant for download if needed. Of course when the machine was bought everything was pre-installed so nobody thought of actually downloading the driver or recovery disks from the Averatec site which also was a big mistake.

Windows Key don’t work

I started with a reformat and reinstallation. No disk – no problem. Download the MSDN Windows XP Home Basic, start installing and use the Windows  key from the bottom of the machine. But immediately it became clear the key from the bottom doesn’t work even though I was installing the same version of Windows (XP Home Basic 64 bit). Apparently the difference is that this was a more recent version (Home Basic with SP2). I called Microsoft’s support number and the tech insisted that the key on the bottom of the machine was not valid downright accusing me of piracy. Thanks dude. Nice! Now I’m a pirate for re-installing Windows. Fine. I have other unused licenses of XP Home I never used  – so screw you too! Eventually (after two more tries) that worked.

Drivers oh Drivers

Next problem: Some drivers didn’t auto-install. Most things came down fine, but others didn’t. No video (ATI Drivers aren’t on Windows Update?), Wireless drivers (Broadcom fairly standard drivers either). Ok, that’s annoying but as long as you can figure out what hardware’s in use it’s not too bad to download drivers from the hardware vendors.

So I managed to get most things installed and got the machine functional with one exception: The Audio Driver. Could not for the life of me figure out what hardware is in the machine and no driver to install.

So as I mentioned the computer came with no CDs. So I went to the Averatec site which is absolute crap. That site has drivers for sale on CD, but not for the 2100. Averatec is also a subdivision of TriGem and they actually have a drivers download area, except – you guessed it – they don’t have drivers for the 2100. They have 1100 and 2200 downloads but not 2100. WTF? Nowhere is there any contact information on either of these sites. Nothing. No links, no phone numbers, no automated email nothing. Well, not true after digging around for about half an hour I found an email link (that I can’t find again now) that hinted at support questions. I emailed, but wasn’t disappointed when I didn’t get a response.

So after fucking around with this machine for a bit by using the 1100 and 2200 drivers I was actually able to get everything to work except for the Audio driver. That was two weeks ago and I’m still driverless. Without audio drivers this low powered machine which is useful for little more than Web Browsing and typing letters to Aunt Maya is basically worthless – not something I’d even want to give away to somebody. A Web machine with out audio is kind of pointless.

BTW, why is it that Audio drivers are sooo problematic on machines? If there is going to be a problem on any machine you install it’s most likely to be an audio driver. I’ve had Dells which I tried to install on Win 2003 and the one thing that never worked were the damn audio drivers. But the real issue with audio drivers is that the don’t auto detect or provide any information about themselves. They always show up as Unknown Multi-Media device, rather than the make of the device. So even if the manufacturer might have drivers figuring out exactly what the damn driver is painful.

Driver Sites Galore

While searching for drivers one thing that really struck out was that there’s just all sorts of commercial bullshit getting in the way in search results. If you search for Averatec 2100 Audio Drivers you get a mouthful of sites that claim to fix just about any computer driver related issue. You get everything EXCEPT the actual Averatec site.

After I had mucked around with this for some time I was about to give up. Via Twitter several people recommended some of the driver download sites like DriverAgent.com  and Driver Detective. I was hesitant at first because this is obviously hardly worth putting any money into. But after the hassles of trying to find a driver I figured a few dollars are cheaper than the time it had already taken me to get this far. Checked out the sites, but predictably they too failed to identify the audio hardware so they couldn’t make a driver recommendation. No go.

Other sites promised to provide full driver disks but most of these seem very shady at best. I actually broke down and tried to get a driver from the people at User-Guides.co.uk which looked no less shady but at least sounded like it would provide everything I need in one place. Well that site sucks too – it’s been 5 days and I still haven’t been able to download the drivers. I spoke to tech support and they claim a backlog and they will refund my money if I choose. I don’t hold up much hope that they’ll have more luck finding drivers than I have. Apparently these people build their own configuration/driver disks based on the specified computer. But of course there’s no mention on the site of this – they make it sound as if the disk and drivers are ready to download. Perfect example of a predatory business…

Predatory and just bad Business Practices

And that’s really what I got out of this experience: There’s so much crap out on the Internet and so much trash that is aiming to confuse, obfuscate and take advantage of people any way they can. This is nothing new and it’s easy to say – you gotta be careful where you go and who you give your information to (the order at User-guides was through PayPal – no way would I have given these folks a credit card!). Most of us as computer professionals are aware of the malice that exists on the net and if something looks even slightly shady chances are pretty good that it really is and a good sign to stay away.

And that’s all fine and good until you get into a situation where you really need something – like this damn audio driver for an Averatec 2100. Ok I don’t really need it, but by now this has become an obsession that has gone beyond any value proposition and I know it <s>. It still irks me to throw the computer away vs. trying to pass it on to somebody less fortunate so I would like to get this one small missing piece for this machine so it’s not a complete piece of junk.

Further it’s sad that a company – Averatec or TriGem – would be so callous as to not put the drivers onto their Web site. The TriGem site is complete piece of shit anyway and the driver portion of that site is actually broken with the fancy Javascript lists not actually working and depending on which browser you use not even be visible. But even forgiving that, why would you not publish the drivers for ALL machines you’ve ever sold? How hard could it be to get that online and keep it there? Apparently these drivers where available at some point in the past when drivers where found on the actual Averatec site (as opposed to the TriGem site) but since then – nothing. Insane.

Ok, I would have never bought a machine like that since it’s clearly a cheaply made consumer level machine and stuff like this isn’t entirely unexpected, but it still makes you wonder about how our resources and supply path are spread out all over the world and over different work and business ethics.

I was musing today on Microsoft’s obsession with OnSelectionChanged events on list controls in general and particularily in Silverlight, which is what I’m working with at the moment.In one of the apps I’m mucking around with I have a ton of sidebar menu selections that get displayed and the easiest way to drive these is with ListBoxes.

They look something like this:

As you can see this ‘menu’ is a collection of controls – a StackPanel with a border element at the top:

<StackPanel x:Name="SidebarMenu2" Margin="10,40,10,10">
    <Border x:Name="SidebarMenu2Header" Style="{StaticResource MenuHeaderStyle}" >
        <TextBlock Text="Update" Style="{StaticResource MenuHeaderTextStyle}"></TextBlock>
    </Border>

    <Border BorderBrush="Black" BorderThickness="1" >
        <StackPanel Background="White" >
            <ListBox x:Name="SidebarMenu2MenuItems" BorderThickness="0" Padding="5" 
                     SelectionChanged="SidebarMenu2MenuItems_SelectionChanged"
                     >
                <ListBox.Items>
                    <ListBoxItem Tag="Save" Content="Save" ></ListBoxItem>
                    <ListBoxItem Tag="Close" Content="Close"></ListBoxItem>
                    <ListBoxItem Tag="Delete" Content="Delete"></ListBoxItem>
                </ListBox.Items>
            </ListBox>
        </StackPanel>
    </Border>
</StackPanel>

I thought to use a listbox for this because it provides the default behavior of clickability as well highlighting the menu selection. The alternative would be having to create all of this behavior manually which is decidedly non-trivial. Additionally listboxes can also be data bound, which isn’t the example above,  but if menus get more complex using dictionary data sure would be easier to use than individual control markup for each item.

Ok, so using a listbox is handy, but as shown above with the SelectionChanged event the behavior is not working all that well for a menu. The selection change event is implemented like this:

private void SidebarMenu2MenuItems_SelectionChanged(object sender, SelectionChangedEventArgs e)
{
    object item = this.SidebarMenu2MenuItems.SelectedItem;
    if (item == null)
        return;

    ListBoxItem listItem = item as ListBoxItem;
    string tag = (string)listItem.Tag;

    if (tag == "Save")            
        this.SaveItem();            
    else if (tag == "Close")
        NavigatorPanel.Navigate(new ItemListPage());
    
}

So far so good. This actually works to make a single selection or another selection that is different which is the most common thing you might do.

But in this scenario SelectionChanged becomes a problem when you try to save more than once in a row. This is a common problem with SelectionChanged events. As the name implies SelectionChanged fires only – well when the selection changes. But what if I saved to accept my changes to commit the changes, then make some more edits and save again? I’m not  re-selecting the same selection and I’m expecting the save to fire again. Only it doesn’t…

This behavior is also prominent in list controls in ASP.NET and even Windows Forms and frankly the absence of an easy way to trap ItemClicks as opposed to SelectionChanged operations are annoying. I’d argue 90% the behavior that is expected is an ItemClick not a SelectionChange.

Ok, so how can this be fixed in the Silverlight code above? My first thought was to hook up a MouseLeftButtonDown event:

<ListBox x:Name="SidebarMenu2MenuItems" BorderThickness="0" 
         Padding="5" 
         MouseLeftButtonDown="MenuClickHandler"    
         >
    <ListBox.Items>
        <ListBoxItem Tag="Save" Content="Save" MouseLeftButtonDown="MenuClickHandler"></ListBoxItem>
        <ListBoxItem Tag="Close" Content="Close"></ListBoxItem>
        <ListBoxItem Tag="Delete" Content="Delete"></ListBoxItem>
    </ListBox.Items>
</ListBox>

But I was surprised that this didn’t work. Either by handling the event on the listbox itself or on each of the items. Apparently the listbox eats those events.

What did work in the end was the MouseLeftButtonUp event on the listbox:

<ListBox x:Name="SidebarMenu2MenuItems" BorderThickness="0" Padding="5" 
         MouseLeftButtonUp="MenuClickHandler">

which did handle the click. I now end up with a generic menu click handler that all the listbox ‘menus’ are hooked up to on the form:

private void MenuClickHandler(object sender, MouseButtonEventArgs e)
{
    ListBox list = sender as ListBox;
    object item = list.SelectedItem;
    if (item == null)
        return;

    ListBoxItem listItem = item as ListBoxItem;
    string tag = (string)listItem.Tag;

    if (tag == "Save")
        this.SaveItem();
    else if (tag == "Close")
        NavigatorPanel.Navigate(new ItemListPage());

    this.SidebarMenu2MenuItems.SelectedItem = null;
}

One thing to keep in mind with this approach is that it doesn’t account for keyboard navigation – so if you move the up and down arrows manually the menu will move, but nothing actually handles the selection. Incidentally this is another example of the shortcoming of SelectedChanged events as a mechanism for selection – when you use keyboard navigation you probably don’t want to fire the operations in the menu – only when you click or press space/enter should anything happen. But this is a pain in the ass to handle. Handling selection reliably on a control is not trivial as you have a host of events to capture (Mouse, Keyboard, plus navigation and possibly hotkeys – not trivial).

You ever run into these types of scenarios or am I making a mountain out of a molehill? I have the same issues in ASP.NET in many situations where you have to respond to SelectedIndexChanged events which often don’t handle the behavior I want to implement. This is why I more often than not implement my own list views these days. In Silverlight this is easy as well, but for basic behavior like the above this is overkill.

Not rocket science but searching around for workarounds earlier I saw several other posts on forums with similar issues, so I’d thought I post this. I’m struggling slowly through  the Silverlight UI learning curve at the moment so expect a few more of these type of basic WTF posts <s>…

Here on Maui electricity is rather on the pricey side and as my household here is hitting the 10 year mark a lot of the household appliances are starting to limp or downright fail. It’s getting to be time for replacements. As I do so I started to pay attention a little more to energy usage of the components.

So a week ago I picked up an electricity usage monitor as an exercise in better understanding energy usage in my house:

P3 International P4460 Kill A Watt EZ Electricity Usage Monitor by P3 INTERNATIONAL P3 INTERNATIONAL Read more...

and I’ve been plugging this thing into various outlets and hooking up a bunch of different appliances to them and monitoring electricity usage. It’s interesting  to see what household appliance and components actually consume in terms of energy – there are definitely quite a few surprises.

Here are a few devices and their energy consumption in a roughly 24 hour time span (unless otherwise noted):

The numbers are a bit surprising in some cases although this falls in line with my monthly energy usage that I see on my bill I use somewhere between 300 and 330 kwh a month in the house which is around 10/12 kwh a day.

Interesting is that the computer’s energy usage is fairly light even when in full operation for 10 hours or more and sleeping the rest. The monitor is the real energy suckage though which has 2.5 times the usage of the computer. All the little boxes we all have – router, cable modem, VOIP Modem, USB hub a couple of other switches all take up another 0.6kwh in that timeframe. It’s not a lot but it competes with the Refrigerator which I wouldn’t have thought – I always thought that office energy usage would be pretty light, but it’s amongst the most energy I use.

The Refrigerator – not surprisingly – uses the most energy of any appliance. It’s warm here on Maui with 80+ degree daytime temperatures and the fridge has to keep its base temperature through it so it’ll probably work a bit harder than it’s spec’d rating which is supposed to be 474kw a year – clearly that’s NOT going to happen (well if I go away for a half a year maybe <g>). I recently bought a new fridge and it’s interesting to compare the old and new ones. The new one uses less than half what the old one did.

Also not surprising is that my LCD TV (42”) uses a lot of energy. Those big screens eat up electricity regardless of that energy star logo on the thing. Also to be expected. What sucks though is that when the device is off sitting over night nearly a third as much energy doing nothing while in standby mode. Yup I had 0.3 kilowatt hours just letting the thing sit at night until 12 hours later in the morning. Everything was turned off, but it’s of course in ‘standby’ mode. The TV still has an LED in front, and stereo, DVD player and other things all have their LEDs on and burning. That probably doesn’t account for the energy usage alone, but it really bites that appliances these days don’t have a way to turn them off completely short of disconnecting them or plugging them into a power strip and turning everything off (which doesn’t work with anything that requires recording and time which I don’t have). As a result I started switching off the switch on the power strip – if I remember.

Either way all appliances these days seem to have this wasteful standby mode. My dishwasher has LEDs that say it’s clean (and it gets confused often enough when the door is closed to start up), the Microwave has blinking lights, the stove does too. All in standby mode with no way to really turn off except unplug or kill the power completely. Lame especially for things like TV, stereo that once off have really no reason to be in standby mode.

Worth looking at Energy Usage?

Now all of this might sound like overkill and it probably is. While electricity is pretty expensive here the actual cost of a kilowatt hour on a bill still amounts to little more than 30 cents. The biggest chunk of my bill are actually base charges, plus energy surcharges with the actual kilowatt hours not being that important (this is how they get you here anyway). In fact looking at my bill I can’t see a damn thing that actually tells me my actual per kilowatt energy costs. Everything is mumbo-jumbo’d with surcharges – probably just to keep people from figuring out exactly what the energy charges are.

I can guesstimate though since I have a few months without any energy usage (because I’m on the mainland then). So even if the entire amount is used the electricity still costs no more than 38 cents a kilowatt hour. But it’s actually less when removing surcharges and other crap that isn’t directly tied to usage. Most likely this will be a way for utilities to get around in the potential case where people start feeding back to the grid. Low kwh charges and high surcharges that aren’t paid back to users.

It’s hard to think that cutting back a few kwh here and there is going to make a big dent on the budget. But then it isn’t only about the bottom line, but becoming a little more aware of what the actual energy usage in the home is and what can be done to effectively reduce it within reason. It’s been useful to put some numbers with actual energy consumption and see them in relation to daily tasks and what energy usage is associated with them. Certainly it can’t hurt to reduce energy usage as much as possible when within reason.

One of the reasons I started looking into this about late last year is that I was thinking of setting up solar for my house here. Clearly we have plenty of sunshine and it would be nice to be somewhat/partially self-sufficient. But looking at my actual energy usage (which I suspect is fairly light compared to most people – especially if you live somewhere where house heating or cooling is required) even this small energy budget would be very expensive to cover with solar for self-sufficiency.

I suppose I’m glad I’ve put this off for some time. I suspect that there will be more improvements in solar technology and public interest in the solar energy sector in the not so distant future as there’s more politcal motivation to move to alternative energies (hopefully). Hopefully this means that prices for the technology will get a little closer to affordability within reasonable range of potential budget benefit <g> and not just as a good faith gesture that it is at this point.

I’ve posted part two in the (now) three part series of articles about jQuery with ASP.NET today. You can read the article here:

jQuery with ASP.NET (Part 2: Making Ajax Callbacks to the Server)

Part 2 of this article series deals entirely with various ways of interaction between jQuery and ASP.NET server side code for making AJAX callbacks.  If you have any questions or comments regarding this article please leave them below.

If you missed Part 1 – it is an introduction to jQuery and focuses primarily on the client side features of jQuery.

An Introduction to jQuery (Part1: The Client Side)

Although this article uses ASP.NET for the sample discussed the content in Part 1 is really server platform agnostic.

Part 2: Focus on Ajax and Server Callbacks

The new article first introduces the basic features available in jQuery and then proceeds to demonstrate a number of different ways to call ASP.NET code. A wide variety of topics are covered from basic .load() calls to separate URLs, to same Page callbacks with Html and JSON data. The bulk of the article focuses on interacting with WCF and ASMX services for raw AJAX callbacks that treat ASP.NET primarily as a data backend for a client application. There’s also some discussion of using client side templating for rendering and maintaining HTML in one place and several extensive sample applications are provided to demonstrate the concepts.

The sample project also includes a host of utilities and what is basically the West Wind Ajax Toolkit Version 2.0. The jQueryControls project contains a number of server controls that simplify a number of client scenarios and also includes a few useful utilities like the ClientScriptProxy, ScriptVariables and ScriptContainer components. The tools have all been updated to work with jQuery on the client.

The article is fairly long (yeah, yeah what else is new?) , mainly due to a fair number of code listings and detailed discussion of some of the examples. As I was writing this thing I was trying to incorporate a number of the questions and comments I’ve heard in the last few months. Hopefully some of you will find this useful.

I thought I was going to do only two parts in this series, but since I ran a bit long on the AJAX portions I decided to leave the server side integration topic (server control creation, wrapping client functionality etc.) for another article. Not only would this article have gotten too long but it would also have blurred the focus at the end. Eeek, two down one more to go…

I’m a big fan for using AJAX callback services in my AJAX enabled applications. I prefer the low level approach where the server acts as a service and only feeds data to the client and the client then manages updating the data and updating the UI with that data accordingly. On the .NET platform there are a lot of options available for doing this from using ASMX Web Services JSON support via the ASP.NET AJAX Web Services, to using WCF 3.5’s REST services to a number of free solutions that provide simple and easy to use connectivity from client to server.

A while back I’d talked about using WCF for REST services with jQuery and there was a ton of interest in this particular topic. I also noticed Dave Ward’s topic (which is great complement to mine) also has gotten a lot of traction and there certainly is a lot of interest in using WCF for AJAX callbacks. I get questions  from people almost daily who want to know something related to the WCF functionality or – almost as frequently – asking whether WCF is the right choice for AJAX Web Services.

WCF REST Services – worth the hassle?

Let me start off by saying that I’m a huge fan of WCF as a service framework for building Web Services and any sort of inter application communication interface. However, I’m much less enthusiastic about the WCF REST functionality and the AJAX functionality in particular. That’s not to say that I don’t use them – I think in the end WCF will replace ASMX for Ajax callbacks. But the reality for me is that WCF REST set up and operation is more error prone and requires more resources than the older and simpler ASMX services.

In the end what it comes down to is that using WCF doesn’t really provide any significant improvements over ASMX based AJAX services. There ARE new features in this part of the WCF stack, but most relate to the actual REST functionality rather than the AJAX features which realistically shouldn’t be lobbed into the same technology space. If you are using ASMX now and you’re moving to WCF for those same AJAX services don’t expect any new features other than ‘it just works’.

There are really two new features in the WCF 3.5: REST functionality which is based on the REST architectural guidelines of using URL based routing, plus usage of Http Verbs to specify intent on service operations. And then there is the RPC style interface functionality that we typically use for AJAX based applications to make AJAX style callbacks to the server from client applications. In WCF 3.5 both of these technologies sit under the same umbrella because they basically use the same functionality – non-SOAP based HTTP endpoints – although from an application architectural POV they are used quite differently.

REST is meant to be URL and Http Verb based, but typical AJAX callbacks – especially those that are based on ASP.NET AJAX - really only use a single endpoint URL and POST data to the the server. ASP.NET AJAX applications aren’t really REST in that they don’t follow typical REST recommendations (not that this semantic difference matters much in a pragmatic sense).

WCF REST is WCF – sort of

I was excited to see a dedicated HTTP REST and AJAX interface to WCF introduced in .NET 3.5. Prior to .NET 3.5 WCF provided only SOAP based services and SOAP based services are notoriously difficult to consume for ‘light’ client interfaces that don’t have native SOAP clients. Consuming a SOAP endpoint in HTTP for example would be rather painful. Instead many client applications today use either plain xml (POX) end points or more likely for Javascript client applications JSON endpoints.

WCF REST provides the pure HTTP based endpoint infrastructure that allows returning non-SOAP content in XML, JSON, RSS and Atom formats, plus the ability to return pure data via .NET streams which allow returning any sort of binary data like images or HTML or anything else so any type of content can be served from a REST endpoint. What this means in a nutshell is that you can use the standard WCF messaging model to return non-SOAP based HTTP data to clients.

I’ve already been using WCF and having this functionality extended to JSON services initially sounded like a great idea. But it quickly became clear that pure WCF REST services are very different than other services: An existing WCF service doesn’t work out of the box as is as a REST service. Unlike other service types that can switch between protocols with just a few configuration switch settings  REST requires that the service contract is marked up with REST specific service attributes on each operation like so:

[OperationContract]
[WebInvoke(
    Method="GET",                       
    ResponseFormat= WebMessageFormat.Json,
    BodyStyle = WebMessageBodyStyle.Bare,
    UriTemplate= "GetStockQuote/{symbol}")
]
StockQuote GetStockQuote(string symbol);

Each endpoint method specifies exactly the format that is used to return data to the client. There are quite a few custom attributes and they are mutually exclusive – any of the options unqiuely configure each endpoint so it’s not possible to create an endpoint that returns either Xml or Json, or a  bare or wrapped response based say based on the type of input that was received. Based on the design of WCF’s functionality that actually makes as one endpoint maps exactly to one messaging format. But it makes for a bit of a hack if you plan to expose and/or accept data in different formats. Essentially you have to create multiple methods with different names to expose data in different formats and – optionally – provide a custom UriTemplate template to make it seem seamless. For example if I want to return data both in JSON and XML I might have to do something like this:

[OperationContract]
[WebInvoke(
    Method = "GET",
    ResponseFormat = WebMessageFormat.Xml,
    BodyStyle = WebMessageBodyStyle.Wrapped,
    UriTemplate="Quotes/{symbol}")  
    ]          
StockQuote GetStockQuote(string symbol);


[OperationContract]
[WebInvoke(
    Method = "GET",
    ResponseFormat = WebMessageFormat.Json,
    BodyStyle = WebMessageBodyStyle.Bare,
    UriTemplate = "Quotes/{symbol}/json")
]        
StockQuote GetStockQuoteJson(string Symbol);

with two method implementations required that basically do exactly the same thing.  The only difference is the UriTemplate and the ResponseFormat. Again this makes sense for the way WCF works, but from a logistic implementation POV this means if you want to support 4 different message formats you end up exposing 4 different endpoints/methods (even if each just calls a base function to do the actual processing).

What’s even worse though is that you can map an endpoint only to a single HTTP verb and document input/output type. You can’t have a single endpoint to serve both GET and POST or based on parameters serve either Xml or JSON. While this might be OK for a pure REST implementation, for a typical AJAX service this is not optimal. When building an AJAX API it’s not uncommon to want to have clients access through both query string parameters or POST variables. AFAIK this can’t be done with a single WCF end point because again you’re specifying a whole bunch of service specific behavior on each method in the service contract.

If you add 4 output types plus maybe two different operation types for that endpoint  you end up with a plethora of methods to service that one endpoint. It gets messy REALLY quick and really makes you wonder: Wouldn’t it just have been easier to create an HttpHandler that has a single end point mapper and properly deserializes/serializes data based on a set of standardized input parameters? After all there is a DataContractJsonSerializer plus the XMLSerializer that can be used to serialize and de-serialize objects fairly easily. I’ve wondered this on several occasions even as I was putting together some of my relatively simple examples for my session. I imagine it only gets worse once you start building more complex services that expose a full complement of REST operations for REST manipulatable data or content.

Even if you stick with a single message format you are still required to specify the format on each Operation/Method. It sure would have been nice if WCF REST supported some mechanism to specify default formats for all those message format attributes that can be defined once at the Contract/Class level. But no – that’d be too easy.

All this service behavior markup too gets tedious as it has to be added to each endpoint method which means lots of repetition. If you decide later on you want to switch to a different message format you have to hunt through all service methods and change all of the attributes on them.  Unlike other services were behavior is set once on the service and only occasionally overridden the pure REST interface requires individual attributes.

No Authentication

WCF REST also doesn’t support any authentication schemes unless you enable enable ASP.NET Compatibility. You can use Windows Authentication with REST Urls but you can’t even easily retrieve the log in info unless you’re using ASP.NET compatibility. So your choices for authentication are use ASP.NET compatibility (described below)  and take advantage of Forms/Windows/Membership Authentication or roll your own. One of the big draws of WCF is that so much infrastructure is provided for you so you don’t have to reinvent the wheel for every protocol nuance. But WCF REST lacks most of that beyond the basic protocol implementations.

Single ASP.NET Authentication Mode Support
WCF REST only supports a single authentication mode in an IIS Web site. This means you can have EITHER anonymous or Windows Authentication going in the site, but not both. When you do both you get a nice yellow screen of death:

This is actually a potentially big deal if you plan on dropping WCF REST into an application that has to support both public and authenticated access. I haven’t seen a way to get around this issue and this will make WCF REST a non-starter in a number of situations – I know it has for me. You can’t always choose what format is used as the rest of the application might determine the security environment.

This seems like an arbitrary limitation, since obviously IIS supports mixing schemes. I can understand multiple actual auth schemes not being supported, but at the very least anonymous should always work. Alas it doesn’t and it’s a bummer.

No Support for Anonymous or object Types

This is huge especially if you are working with LINQ and potentially returning filtered data to the client. The following will compile but will blow up when you try to call the service:

[OperationContract]
public object GetAnonymousType()
{
    return new { Name = "Rick", Date = DateTime.Now, Value = 212.22 };
}

The problem is that the DataContractJsonSerializer that WCF uses to serialize to JSON doesn’t support non-specific types for serialization. While this makes some sense for WCF as a whole (because a non-specific type can’t be provided in a contract accurrately), it’s something that should work for JSON which doesn’t provide a real service contract metadata item to the client.

Incidentally ASMX services do support Anonymous Type serialization although you can’t pass them back from the client which makes sense since you can’t specify the type signature for the parameter. It’s silly that WCF doesn’t support this.

Error Handling

In non ASP.NET Compatibility mode, you’ll also find that the error handling is horrible. Instead of an response that is returned as JSON to the client you get an HTML error page that is completely useless to the client that can do nothing other than try to parse out the error message. This makes sense for pure REST scenarios, but in a JSON callback scenario an HTML message does not provide anything useful to the client – what’s really needed is like standard WCF an error returned in the appropriate message format (ie. JSON). There’s no built-in way to return an error in a JSON scenario as an error object – instead HTML is returned. You can roll  your own to do this (or again use the ASP.NET AJAX specific script factory) but that’s not a trivial task.

The REST Starter Kit improves this slightly. It provides some new WCF REST extensions that provide the abililty to throw errors from server code more easily and output error output pages that are much leaner than the typical Yellow Screen of Death you’d get by default. BTW, the starter kit provides a host of useful new features although one really has to wonder why this stuff wasn’t added to the core product in the first place.

Some of these limitations make sense for the platform – it’s the way WCF works, but maybe that’s precisely why the REST features are maybe not such good fit for WCF.

ASP.NET AJAX HostFactory makes things easier than ‘plain’ REST

What I described above mostly relates to pure REST interfaces where you specify every option you want to use explicitly on each method using the WebInvoke or WebGet attribute.This allows you maximum flexibility for your messages but it’s also tedious to do on each and every method. This is probably reasonable for ‘real’ REST based applications that are exposed to be accessed by the whole world , but for a typical AJAX callback service that gets only by your application to provide data for client pages that would be overkill. Typical AJAX applications aren’t concerned with REST semantics, but need to efficiently and consistently retrieve AJAX data.

Luckily Microsoft recognized this needs as well as the existing base of ASMX style AJA X services  that work with the ASP.NET AJAX infrastructure. If all you’re after is compatibility with ASP.NET AJAX and to replace AMSX with WCF  – there’s a shortcut that provides WCF REST the same functionality that ASMX based AJAX services provide. Microsoft was smart enough to release a SerivceFactory that provides in effect ASP.NET AJAX compatibility out of the box. It provides the same ‘wrapped’ JSON message format that ASP.NET AJAX uses, the script proxy (the /jsdebug option), error management by wrapping exceptions into objects that get marshalled back to the client as JSON objects. Some of these features – like the error management is only available to this specific ServiceFactory.

So to set up a pure AJAX service  you can use the WebScriptServiceHostFactory on the service definition in the Service markup:

<%@ ServiceHost Language="C#"       
                Service="WcfAjax.MsAjaxStockService"  
                CodeBehind="MsAjaxStockService.svc.cs"
                Factory="System.ServiceModel.Activation.WebScriptServiceHostFactory"                     
%>

which makes the service ‘config-less’ (the entries in Web.config can be removed), provides the /jsDebug proxy generation and serves and receives data in  Microsoft’s wrapped JSON format and error marshalling as JSON. No settings in web.config  are required and none of the endpoint methods require any attributes beyond the OperationContract to expose the method.

[OperationContract]          
StockQuote GetStockQuote(string symbol);

When using this model the input is always a POST JSON string in the MS parameter format which is in the mode of parameter key value pairs (ie. { “parm1”: value, “parm2”: “value”}. Output is always returned in the MS wrapped format which has a ‘root’ object with a single property that actually holds the result value. The wrapped format includes a root node, plus type information on each marshalled object value returned from the server to facilitate two-way serialization.

{"d":
     {"__type":"StockQuote:#WcfAjax",
      "Company":"Microsoft Corpora",
      "LastPrice":20.49,
      "LastQuoteTime":"\/Date(1227751200000-1000)\/",
      "LastQuoteTimeString":"Nov 26, 4:00PM",
      "NetChange":0.50,
      "OpenPrice":19.83,
      "Symbol":"MSFT"
     }
}

This format is specific to the WebScriptServiceHostFactory as far as I can tell. Although similar to the stadard BodymessageStyle.Wrapped the format is different as the base version does not include type information for objects and names the root node differently. IOW, this is a very ASP.NET AJAX specific implementation.

By default when using the WebScriptServiceHostFactory,  ASP.NET compatibilty – the abilty to access the HttpContext object and its intrinsic ASP.NET objects – is not enabled. If you have existing ASMX services you want to migrate to WCF and those services rely on things like Session or Request you need to add one more setting to your Web.Config:

<system.serviceModel>

  <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />

</system.serviceModel>    

Then on your service:

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public class RestStockService : StockServiceBase, IRestStockService

Options are Allowed, Required, NotAllowed. Note that this is specified on the implementation class rather than the service contract interface.

If you don’t need ASP.NET compatibility, it’s best to turn this option off, but if you have an existing ASMX service that uses say the session object to track users, it’s handy to be able to fall back onto ASP.NET objects. Realize when you do use ASP.NET features you are obviously tied to ASP.NET and IIS. REST services CAN be hosted in alternative hosts (self hosting for example) but if you use ASP.NET compatibility only ASP.NET app hosting is allowed.

Using the WebScriptHostFactory is a quick way to migrate existing ASMX services to WCF. However, keep in mind that realistically doing so will buy you nothing in terms of new functionality or features other than being able to say the app runs on WCF. However, going forward I suspect that any further improvements to the ASP.NET AJAX engine will be made to the WCF engine rather than ASMX, so going forward and especially for creating new services it’s probably not a bad idea to start using WCF services. It’s not really any more complicated than ASMX services as long as you use the WebScriptServiceHostFactory.

Stability

One big concern I’ve had with WCF is that I’ve had some stability problems with the .SVC file in my ASP.NET projects. The issues are mostly in the development environment, but it’s been a huge time sink. The problem I’ve run into is that the .svc file on the server somehow gets out of sync with the compiled code and causes a yellow screen of death for a compilation/temporary file error as I described here. These errors pop up randomly on apps that have been working just fine and even though the service file or the service related classes and interfaces have not changed.

Things get a little better if the following configuration setting is used in <system.web>:

        <compilation  debug="true" batch="false">

which makes this compilation error less of a problem. But it still happens to me from time to time and once it does it takes a few IIS restarts to get the application running again.

Do you need to use WCF rather than ASMX?

So where does that leave us? WCF REST provides new functionality but be prepared to spend some time with it, especially if you choose to not use the ASP.NET AJAX specific WebScriptHostFactory. Figuring out how to get WCF to return just the right contract can take a little tweaking.

As to switching from ASMX to WCF for AJAX services – if you have existing services that work with ASMX I really don’t see a compelling reason to switch to WCF. There are no new features so at best you get the same functionality you had before running WCF. If you do decide to go forward to WCF – updating is pretty easy. It basically involves migrating your classes by updating the various attributes [WebMethod] attributes to [OperationContract] attributes, so all in all that’s a pretty painless migration. Remember that if you use ASP.NET intrinsic objects anywhere in your service to enable AspNetCompatibilityEnabled in Web.config and set the attribute on the service class.

Going forward for new services using WCF is probably not a bad choice since it’s really no more complicated than ASMX to set up if you use the script factory.

Still I’m fairly wary of the new WCF features. I wanted to be exited and move forward, but ultimately WCF just doesn’t bring any benefit to the party, but rather makes things more complicated by requiring additional references and  requiring more resource. Most AJAX based ‘service’ interfaces are not real services but are very closely tied to the Web application. They tend to be private services used for internal use of the application, rather than widely accessed public services. For that task WCF REST services might have more appeal.

So I’m finally getting an opportunity to work with Silverlight in a real application. I had mentioned this application in the past a few weeks back – the original application architecture was created as a Flash front end to a .NET backend which provides service style XML data. In the original app the server infrastructure was used purely to serve data to the client – basically a complex, mostly meta data drive SQL Engine based on complex stored procedures feed the client Xml data. Originally our data format has been DataSets because those were the easiest way to consume the data in the Flash client since there’s native support for DataSets and it greatly facilitated the process of passing data to flash and the format is basic enough that it’s easy to send update data back from the client (manually created in this case).

In the original app I basically built the data engine – which is a custom HttpHandler based affair that basically is a glorified XML routing engine that maps inbound requests from the client who sends XML data to the appropriate business object/Stored Procedure routine which in turn responds with a data result that is streamed back to the client. The server app and business objects actually do very little work in this process – most of the work lives in the SQL backend with its stored procedures that manipulate the data. The business object’s tasks mostly is to pick up inbound query parameters and map those into the appropriate stored procedure calls. The results for now are returned were returned as DataSets and there’s a bunch of extra logic that can also return that same data out as either JSON or as simplified XML (without DS schemas).

This has all worked well, but it’s been based on the existing Flash expertise at the time, which have now become an issue.

So based on some back and forth discussions the thinking now is that since we’re starting from scratch Silverlight is probably a better choice due to the easier coding model and readily available developer pool. Which has landed me smack in the middle of Silverlight and not least of all a data access decision to make. :-}

Data Models

The main issue I’m working on at the moment is finding the right data model. The server side as it sits right now has a number of options for returning data to the client. Silverlight has many options for getting data to the client:

Raw Xml

Obviously we can retrieve the raw XML from the server and just parse that on the fly. While not difficult this would not be my choice for data access because it’s tedious and potentially error prone. Since the data is mostly dynamic and doesn’t lend itself to mapping well though this is not totally unreasonable because we’ll likely end up with some sort of untyped data representation on the client no matter what.

JSON or ADO.NET Data Services

As I mentioned our backend can actually feed data out as JSON so another option I was toying with is to use JSON marshalling to map onto objects. The problem with this approach is that we don’t really have a model we can map to. The problem is that the backend (over which I have no control at this point) is just a fairly large set of heavily parameterized stored procedures. While the actual simple tables in the system are fairly small and could be modelled, the way the data is manipulated and returned does not lend itself well to producing either a Linq to SQL or Entity Framework data model. We’d end up with about a hundred or more hand created types that would have to be kept in synch with the database. That doesn’t sound very promising either.

ADO.NET Data Services could be interesting for certain data scenarios, especially free form data access, but my current scenario isn’t one of them I think. Still digging through the docs on ADS but I think when dealing with stored procs there’s really nothing here to help effectively. Nor is the structure of this architecture really a good fit. To make this work would be square peg, round hole.

Web Services

Web Services provide another possible option. Silverlight can directly access WCF Web services and retrieve data that way and this is actually a pretty easy interface to work with. But again it requires fixed, typed result sets so the issues are somewhat similar as the previous  item. You need to be able to full type information published on the server in order to get the data back to the client.

One thing I was thinking about was to return a result back from a Web Service as a generic dictionary which actually serializes properly, preserving underlying type information:

[OperationContract]
public Dictionary<string, object> GetDictionary()
{
    Dictionary<string, object> items = new Dictionary<string, object>();

    items.Add("Field1", 10);
    items.Add("Field2", DateTime.Now);
    items.Add("Field3", "Rick");

    return items;
}

Properly returns in Silverlight as:

private void btnService_Click(object sender, RoutedEventArgs e)
{
    PraServiceClient service = this.GetService();
    service.GetDictionaryCompleted += 
        new EventHandler<GetDictionaryCompletedEventArgs>(service_GetDictionaryCompleted);
    service.GetDictionaryAsync();
}

void service_GetDictionaryCompleted(object sender, GetDictionaryCompletedEventArgs e)
{
    if (e.Error != null)
    {
        MessageBox.Show( e.Error.Message, "Service Error",MessageBoxButton.OK);
        return;
    }

    Dictionary<string, object> items = e.Result;

    StringBuilder sb = new StringBuilder();
    sb.Append(items["Field1"]);
    sb.Append(items["Field2"] + items["Field2"].GetType().ToString()) ;
    sb.Append(items["Field3"] + items["Field3"].GetType().ToString());

    MessageBox.Show(sb.ToString());
}

private PraServiceClient GetService()
{
    EndpointAddress addr = new EndpointAddress("http://rasnote/PraWebService/PraService.svc");
    BasicHttpBinding bind = new BasicHttpBinding();
    return new PraServiceClient(bind, addr);
}

The above surpringly works to marshal types in the dictionary down to the client even though the dictionary is Dictionary<string,object>. WCF does not like serializing generic object types however so results of type object or anonymous types fail outright. I suspect using these types as values for the dictionary too will fail.

Dynamic Data Access

The real issue here is that in lieu of a data model that can be used on the client side there’s no good data mechanism built into Silverlight that provides access to some dynamic data returned from the server. So say the server returns an arbitrary query where the user is allowed to pick the columns he wants to retrieve there’s really no clean way to return that data and present it on the client side. JSON serialization requires a type as an input to understand the structure and so a dynamic results can’t be represented. It’s possible to actually use JavaScript to create an object but even then that object comes back as a dynamic object which could only be accessed by Reflection (C# 4.0 dynamic oh where are you? <s>)

No DataSet Support

I know DataSets are kind of passé but when I really think about how this existing data engine works – DataSets with their weak typing and even with their update support are the perfect fit for this type of application. As many problems as DataSets have in typical pure data access scenarios, in internally controls disconnected scenarios Datasets have features that none of the newer data techologies have matched to date – mainly because DataSets allow for dynamic data to be returned and be accessed easily.

Alas – Silverlight has no support for DataSet/DataTable/DataRow. While looking at various third party components I actually ran into ComponentOne’s Silverlight Controls which include a Data assembly that provides a minimal set of the classic ADO.NET objects for Silverlight. But even with those the support is fairly limited (for example no support for schemaless XML or Diffgrams – the latter of which is kind of a bummer). Still for now that’s something to start with – if we decide to go this route I suppose it’s not terribly difficult to build my own abstraction that does support this functionality.

Something Custom?

I’ve also toyed with the idea of building a custom Dictionary type interface that I can deserialize from JSON which would basically allow me to use JSON from the server and get some sort of generic object back back on the client. Since the data is for the most part table based, object hierarchy isn’t really an issue either so it’d be fairly straight forward to build up something like this fairly easily.

Yes, C# 4.0 and DynamicObject would be pretty handy for this, but it’s going to be some time before that comes to pass.

Hmmm…

In some ways these issues are actually easier to work through in JavaScript/Html based interfaces. Because Javascript is a dynamic language it’s actually easier to retrieve JSON data from the server and use it on the client. The server can return arbitrary data and simply return it with the client having direct access to it. I suppose one could choose one of the dynamic languages (JavaScript, IronPython or IronRuby) on the client, but frankly that’s not really a good option either. The whole point of using Silverlight is to provide the richer, statically typed environment.

So I’m curious to hear what others are doing for their client side ‘data’ connectivity, especially when dealing with data that can’t be easily serialized back and forth between client and server. I suspect a lot of folks probably deal with raw data – manually parsed XML or possibly. If not what are you doing for data transfers? There are a many options but it seems that most of them have some shortcomings when it comes to effectively dealing with the data on both ends of the connection.