ASP.Net’s WebResource.axd and machineKey badness

Let me start from the beginning.

Yesterday, my company was updating our site with the usual round of fixes that hit with each release. Throughout the process, while updating some servers in our NLB cluster with my changes (Asp.net AJAX additions), we ran into some seriously weird errors.
Intermitant spews of the following:

SERVERNAME,SiteName,3/21/2007 2:47:26 PM,/ScriptResource.axd,Padding is invalid and cannot be removed.

I was baffled, as I had never seen such errors while developing on my machine, or after test deployment to our internal servers.

I soon discovered that the errors were caused by an incorrect d= value on WebResource.axd. The culprit appeared to be inconsistent encoded string generation across the NLB cluster. Through research, I also discovered that this problem also exists for persisting viewstate across server farms, which we had previously fixed by setting validationKey= in the machineKey node in Web.config. The fix for the WebResource error is similar, and appears to be a missing decryptionKey= attribute for the same machineKey node.

From what I understand, WebResource.axd will return a given client script based on any number of given scenarios (This is part of ASP.NET 2.0). Our scenario is to return client script for Atlas (asp.net’s ajax framework) in order to get our fancy asynchronous callback magic.

For whatever reason, we were not generating a consistent string across the farm without the attribute in place. So strings generated on Sever1, would not return the correct script on Server2, so on and so forth.

All 4 of our servers are now generating the same d= string.

You can reproduce the errors by altering the d= string.

More information on the string generation can be found here.

These incorrect d=[string here] values can surface for a number of reasons.

1) Search aggregates crawling the site and caching old machineKey values

2) Web Caching services, proxies, Google Cache, etc (We saw a ton of AOL proxy users throwing the exception)

3) Inconsistent machineKeys across farms

The only instance that pertains to us is the machineKey/NLB case.

I hope this helps someone, it was a headache to discover.