No, you can't add a reference to a Class Library project to a Silverlight application project. You can only add the reference of another Silverlight application project inside a Silverlight application project.
However, you can add a reference to a Web Service or WCF Service.