top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Redirect search robots onto a dedicated cluster machine when using apache

0 votes
305 views

We have a small cluster of machines serving a wiki website. Search engine robots are allowed in to index the website and these create a lot of traffic that can affect the service. I would like to redirect all http requests from these robots to one of the machines in the cluster.

Any ideas for how to do this ?

posted Jul 2, 2013 by anonymous

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button

1 Answer

+1 vote

You should be able to use a rewrite rule that redirects to another server / port / worker.

https://www.google.com/search?q=apache+redirect+based+on+useragent

answer Jul 2, 2013 by anonymous
Similar Questions
0 votes
redirect based on internal or external IP using Apache

I am new to apache, so forgive me if this is a beginner questions.

I am trying to redirect request to apache based on the IP address. Depending on where the request originates from, the request will either have a external or internal IP address. External will be redirected to the external site, internal will be redirected to the internal site.

Request to www.site.com will be evaluated based on the IP address. Then redirected to either www.site.com/external or www.site.com/internal . I am able to accomplish this with the following. 

 RewriteCond %{REMOTE_ADDR} ^1.1.1.1 
 RewriteRule ^(.*)$ www.site.com/internal [R=301,L] 
 RewriteCond %{REMOTE_ADDR} ^2.2.2.2 
 RewriteRule ^(.*)$ www.site.com/external [R=301,L]

The problem I am running into is that this is in a loop. Every request will be evaluated and redirected. If possible, I need a way to redirect once, then let other request (www.site.com/external/a/b/c or www.site.com/internal/x/y/z )not be evaluated and then redirected

0 votes
Defining search depth for mod_ldap, using apache

when searching a user mod_ldap returns only parent node attributes. Is there any setting so that search will return parent node attributes as well.

LDAP URS is as follows:
LdapUrl=ldap://172.16.100.237/ou=radiusUserProfile,dc=mtnl,dc=com?uid,userPassword,cn,sn [1]?

Here I could get back only uid,userPassword. cn and sn were blank, which are attributes of child nodes.

+3 votes
Apache Proxy redirect to an application?

I want to redirect a url to an application on same apache server. My development environment is Ubuntu. I enables the mod_proxy. But my url isnt redirecting to the desired application. Following is the code I put inside the /etc/apache2/sites-available/default file. I put this bottom of the file but before the "VirtualHost" tag.

ProxyRequests On 
ProxyVia On  
Order deny,allow 
Deny from all 
Allow from localhost  ProxyPass /application_directory/ http://localhost/application/directory ProxyPassReverse /application_directory/ http://localhost/application/directory
+1 vote
Cluster creation and adding Services through Apache Ambari Restful APIs

I want to create Cluster and want to add services through Apache Ambari Restful APIs. I am unable to call POST,PUT and DELETE Web Services successfully.

I am using Resful APIs client to work and trying to use below URL with POST request but not working.

POST REQUEST

http://AmbariServerIP:8080/api/v1/clusters/c1
+1 vote
Question about mod_proxy_balancer when using apache

The Barracuda load balancer has feature which enables it to recover a broken web connection. Let me explain the scenario; suppose that there exist two web servers behind the Barracuda load balancer and the user is downloading a file from one of the back-end web servers. Now, suppose that the active web server goes down, the Barracuda load balancer is capable of continuing the download from where it was broken using the other web server. Is it possible to achieve such a behaviour using mod_proxy_balancer? Are there exist any open source alternatives which can provide the same capability? If not, is anybody aware of any roadmap to achieve the same functionality using open source tools? Considering mod_proxy_balancer, what are the required steps to follow in order to add such a functionality?

...