This is the latest data on identifying the search engine spiders. You can get more information by viewing your site's visitor log reports. Information on blocking the Search Engine Spiders with a Robots.txt file can be found at robotstxt.org.
| |
|
| Search Engine: |
Google Adsense |
| User Agent |
Mediapartners-Google/2.1 |
| Robots.txt Identifier |
User-agent: Mediapartners-Google |
| Details: |
https://www.google.com/bot.html Note: This and other Googlebots share crawled pages with each other to reduce the amount of crawling Google is required to do for their different services. |
| |
|
| Search Engine: |
LookSmart |
| User Agent |
Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; https://www.WISEnutbot.com) |
| Robots.txt Identifier |
User-agent: ZyBorg |
| Details: |
Powers LookSmart's Wisenut.com - https://www.WISEnutbot.com |
| |
|
| Search Engine: |
Microsoft / MSN / Live |
| User Agent(s) |
msnbot/1.0 (+https://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+https://search.msn.com/msnbot.htm)
msnbot-news/1.0 (+https://search.msn.com/msnbot.htm)
msnbot-products/1.0 (+https://search.msn.com/msnbot.htm) |
| Robots.txt Identifier(s) |
User-agent: msnbot
User-agent: msnbot-media
User-agent: msnbot-news
User-agent: msnbot-products |
| Details: |
MSN has not updated their page https://search.msn.com/msnbot.htm yet to officially address the specific bots they are using. If you wish to block a specific bot in your robots.txt file use the associated bot specific name as the User-agent |
| |
|
| Search Engine: |
Yahoo Mobile |
| User Agent |
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; https://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) |
| Robots.txt Identifier |
User-agent: YahooSeeker/M1A1-R2D2 |
| Details: |
https://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html |