Watch out for the robots.txt link exchange trick…

I recently came across a method on accident that a lot of webmasters are doing to create one-way backlinks. I wanted to share this in the DP Search Engine Optimization section since it’s directly related to backlinks SEO value.

When you exchange links with someone you always do the obvious, you check for the nofollow attribute, robots meta tag and iframe injecting. Those methods are used to “block” or “not give value” to outbound links on the web page, which would make your link to them a one-way link and much more valuable.

I’ve started to see lately a lot of webmasters using the robots.txt file to block link pages when doing link exchanges, yes a web page can have a Pagerank value if it’s only blocked by robots.txt, it won’t crawl or index the web page, but the inbound link value is still passed (I noticed this on my own web site, see bottom for example.).

So, another method to check when doing link exchanges is to verify the page(s) your link is appearing on isn’t blocked via robots.txt, or when you check your link exchanges to make sure they are still valid. I have a feeling most of you will run out and check this and find atleast one webmaster pulling this manipulative method off.

Example of Pagerank passed to Blocked Documents

http://www.domain.com/ask.html

The above URL shows a PR0 in the Google toolbar, what you should notice is it’s not N\A but PR0, meaning it is recieving link juice from internal\external links. This web page is blocked via my robots.txt file.

We can check the robots.txt

User-agent: *
Disallow: /sitemap.google.php
Disallow: /sitemap.yahoo.php
Disallow: /xmlrpc.php
Disallow: /ask.html
Disallow: /open.html
Disallow: /addcontent.html
Disallow: /search.html
Disallow: /contact.html
Disallow: /help.html
Disallow: /instantresponse.html
Disallow: /showcat.html
Disallow: /feed/
Disallow: /admin/
Disallow: /data/
Disallow: /install/
Disallow: /attachments/
Disallow: /font/
Disallow: /inc/
Disallow: /pdf/
Disallow: /sitemap/
Disallow: /template/
Disallow: /xml/
sitemap: http://www.domain.com/sitemap.xml