If I understood your question correctly then you require a function in ASP.net which can tell you if a url exist or not. Which can be used to findout for the following url if exist/not exist
<website>/sitemap.xml
<website>/robot.txt
Check this code which would be handy and will return true or false if provided url exist or not exist.
protected bool CheckUrlExists(string url)
{
// If the url does not contain Http. Add it.
if (!url.Contains("http://"))
{
url = "http://" + url;
}
try
{
var request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "HEAD";
using (var response = (HttpWebResponse)request.GetResponse())
{
return response.StatusCode == HttpStatusCode.OK;
}
}
catch
{
return false;
}
}
Please let me know if it does not satisfy your need.