I have a creaky old Win 2003 Server dev box that I wanted to deploy my RIA solution to. I could not do a build and publish on the box itself because I did not want to install the RIA services, etc. on there.

So I did a publish on my local environment, which produced a folder with the required aspx files and a bin folder with all the RIA Dlls (include data annotations and domain services) and a ClientBin folder for the Silverlight XAP client payload.

I copied the folder as-is to my web server and created a virtual folder and an Application (in IIS) for it.

But when I browsed to it, I got an error in my browser stating:
Message: Unhandled Error in Silverlight Application
Code: 2104   
Category: InitializeError      
Message: Could not download the Silverlight application. Check web server settings    

This is slightly misleading since you might think there is something wrong with your application, but in fact it happens because the web server is not serving the required payload properly. It is because your web server may not have the required MIME types for Silverlight configured.
To fix this problem, right-click on your application, and click the MIME types button in the HTTP-headers tab.
Add the following:
.xaml    application/xaml+xml
.xap    application/x-silverlight-app
.xbap    application/x-ms-xbap

Now, the web server should be able to serve all the required files for the Silverlight client to run.