Jim Henrik writes “For more and more websites you need to register or pay to have full access. The odd thing is that Google has the complete and full index of the website. So what’s going on here? Why must the regular user pay or register to have access when the google search engine bot has full access?. The reason is simple; every site wants to use the benefits of the wonderful world of Google, for webmasters free advertising is always welcome. But there is a simple way to be the Google (search)Bot. In this little article i will try to explain it.”
Link: linuxexposed.com
