How to read remote web page contents in ASP.NET?

Answer 1

We can see web page HTML contents of a web page by clicking on view source or view page source in browser. You might need to get web page contents in your ASP.NET code. You can get HTML source of a page by using ASP.NET classes. This can be achieved using different techniques. You can use WebRequest and WebResponse classes to along with StreamReader class to read remote web page contents. WebRequest class has a Create() method which takes the URL of a page and initialize a new instance of WebRequest class for specified URL. GetResponse() method of WebRequest class returns a response and we can assign this to WebResponse class object. Then we can use StreamReader class to read contents from WebResponse object.
You can also use a very simple technique to get page contents by using WebClient class. You can get byte data of page content by using DownloadData() method and you can also call DownloadString() method by providing page URL to it to directly get the HTML page contents of that page.

1.Open Visual Studio 2010
2.File > New > Website
3.Visual basic or Visual C# > ASP.NET Empty Web Site
4.Website > Add New Item > Web Form
5.Add code below in aspx page

<h3>Enter URL and get contents of the page</h3>
<asp:TextBox ID="TextBoxURL" runat="server" Height="20px" Width="250px"></asp:TextBox>
<asp:Button ID="Button1" runat="server" Text="Get Contents"
    onclick="Button1_Click" />
<br />
<br />
<asp:TextBox ID="TextBoxPageContents" runat="server" TextMode="MultiLine"
    Height="500px" Width="1000px"></asp:TextBox>

6.Now open code behind file and Include following namespaces

C#

using System.Net;
using System.IO;
using System.Text;

VB.NET

Imports System.Net
Imports System.IO
Imports System.Text

7.Add code below in Button Click event

C#

protected void Button1_Click(object sender, EventArgs e)
{
    string url = TextBoxURL.Text;

    WebRequest webReq = WebRequest.Create(url);
    WebResponse webRes = webReq.GetResponse();

    using (StreamReader reader = new StreamReader(webRes.GetResponseStream()))
    {
        TextBoxPageContents.Text = reader.ReadToEnd();
    }
}

VB.NET

Protected Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click
    Dim url As String = TextBoxURL.Text

    Dim webReq As WebRequest = WebRequest.Create(url)
    Dim webRes As WebResponse = webReq.GetResponse()

    Using reader As New StreamReader(webRes.GetResponseStream())
        TextBoxPageContents.Text = reader.ReadToEnd()
    End Using
End Sub

Here I have used WebRequest and Web Response classes to read remote web page content.

8.Now comment above code in Button click event and write code below in it

C#

protected void Button1_Click(object sender, EventArgs e)
{
    string url = TextBoxURL.Text;

    WebClient client = new WebClient();
    //TextBoxPageContents.Text = client.DownloadString(url);

    byte[] byteData = null;
    byteData = client.DownloadData(url);

    UTF8Encoding UTF8Encod = new UTF8Encoding();
    TextBoxPageContents.Text = UTF8Encod.GetString(byteData);
}

VB.NET

Protected Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click
    Dim url As String = TextBoxURL.Text

    Dim client As New WebClient()
    'TextBoxPageContents.Text = client.DownloadString(url)

    Dim byteData() As Byte
    byteData = client.DownloadData(url)

    Dim UTF8Encod As UTF8Encoding = New UTF8Encoding()
    TextBoxPageContents.Text = UTF8Encod.GetString(byteData)

End Sub

Here I have used WebClient class to read remote page contents. I have used two method of WebClient class here. First I have called the DownloadString() method to get directly HTML page contents as string. Second I have used DownloadData() method to get byte data and then I have converted this byte data to string using UTF8Encoding class.

9.Now see web site in browser. Provide URL of a page and click on button to read contents of that page.