The Bulk Submit 2.0 session at Search Engine Strategies in Chicago this week could have easily been called "You, Me and Sitemaps" - and probably should have been. The process of submitting sites to search engines has been an interesting evolutionary process, one which has ranged from bulk submission of URLs thanks to services such as Inktomi to the paid inclusion of sites. A recent agreement of the big three search engine networks (Google, Yahoo and MSN) that agreed to use the same standard for the use of sitemaps (find out more at sitemaps.org) - a welcome development for website marketers.
Amanda Camp of Google was up first. Camp made it clear that what she wasn't going to be talking about were Google Base, Local Business Center, Blogger, Page Creator or the Add URL feature as methods to notify Google of existing Web pages. What she did focus on were sitemaps and how they improve the comprehensiveness and freshness of your indexed pages and how they help improve the efficiency of Google itself. There are four accepted formats for Google sitemaps (which are now the same as a Yahoo or MSN sitemaps) and they are text files, rss feeds, the official sitemap protocol and OAI-PMH (open arvhices
initiative protocol for metadata harvesting - this is not used very often).
Camp presented some simple rules for success with sitemaps including:
- include the full URL path
- remove unnecessary parameters
- should be at the highest directory
- needs to match location
- name the file anything you want
- must use UTF8 encouding
- max is 50K URLs or 10MB per sitemap
- use Gzip for compression of neccessary
It's interesting to note that Camp mentioned that, when using the XML format, only the
,
and
are required. Optional data to incorporate into the XML file are the last modified date, the change frequency and the priority of each page. Camp suggested using the official Google Sitemap Generator (a python script) or one of the fifty plus third party generators listed on their site.
Next up was Anit Kumar of Yahoo. Kumar managed the Site Explorer team. Site Explorer is a popular tool among Web marketers for browsing pages and in-links, ensuring site authentication, enabling bulk submission through RSS and sitemaps and which will be offering up some new features very soon. Kumar emphasized that "the best was yet to come."
Eric Papczun was up next with a sitemaps case study from which he offered up some valuable tips.
- Do the due diligence, make sure you achieve clean URL submission
- Have an optimized named sitemap and links from a global footer
- Exclude redundant content (printer friendly pages, disembodied content (flash) and generally spammy stuff)
- Discussed what to expect - (the number of pages may go up or go down - which is neither a good or bad thing)
- Select URLs for more frequent crawls (news release pages, product release pages)
- Pay attention to unreachable URLs (in the Google Sitemaps system), uncrawlable URLs and URLs which were blocked by the robots.txt file.
Todd Friesen was up next and focused on comparison shopping engines which enables marketers to utilize RSS and XML to incorporate sites via paid inclusion. CSE feeds and their automation keep costs low but Friesen noted that a "human touch" brings the best performance. Friesen went into his own review of how well the big three search networks shopping services performed, highlighting that MSN Shopping converted very well at a reasonable cost per click, although he did note that there was a low volume comparatively. Google Base did convert well but offered little support which Yahoo Shopping featured a high volume of traffic which was expensive but did provide good conversions.