Transparent Linking

We tested all twelve links for their suitability to serve as transparent links. Interestingly the results have been very similar for all these web spiders. Every spider was visiting the test area in irregular periods and just made a few requests every visit at maximum, probably in order to avoid to overflow the website and to demand too many resources. The only aspect changing were the times and time-periods when the different pages were accessed. All spiders have been switching between different IP addresses at some time during the crawling process.
The behaviour regarding the recognition and crawling of the links has been identical for all spiders: Of all twelve links that were provided only number 6, the hyperlink hidden in a comment, has not been recognized. All other elven links have been followed and all of the underlaying layers have been crawled. Also within the underlaying layers the only link type that has been disregarded was number 6.
The result might be surprising as one could expect other transparent links to be neglected as well. Especially link number 5 or 7 seem to be prone to disregard. But here we have to take into account that a web spider crawling a page does not necessarily mean that page will also be added to the search index. It is well known that search engine companies are blacklisting links which they consider to be an unwanted or forbidden manipulation of the search index. This means some link types may only gain a low ranking or may not be indexed at all. It is extremely difficult to gain reasonable information about the ranking a link creates, as many different aspects are involved: The number of links, the ranking of the source page the links are placed upon, the content of each page, the frequency of content changes and many others. Experience within the deployment of our honeypots showed that especially link type 2 and 12 result in reasonable ranking positions and are suitable to be used as transparent links.
Another aspect one has to consider for the interpretation of the results is that the test can only be seen as a single snapshot which cannot provide comprehensive information about the various behaviours of web spiders.