Google has officially announced that site load speed is a factor when determining rankings, not that it's a deal-breaker. According to Matt Cutts, "While site speed is a new signal, it doesn't carry as much weight as the relevance of a page. Currently, fewer than 1% of search queries are affected by the site speed signal in our implementation and the signal for site speed only applies for visitors searching in English on Google.com at this point."
So, the chances your site will be affected are slim. But read between the lines on this one. Or, let Google's Webmaster Central Blog do it for you: "But faster sites don't just improve user experience; recent data shows that improving site speed also reduces operating costs. Like us, our users place a lot of value in speed - that's why we've decided to take site speed into account in our search rankings."
It's only common sense that faster sites are more appealing to Web users. But knowing that Google places "a lot of value in speed" means that you should, too. The fact is, when Google decides to make a change to their search rankings -- any change -- website owners should take notice. While it might be a small factor in rankings today, it could loom much larger in the future.
Also from the Google blog are these suggested tools to check your site speed. Sometimes we can get caught up in the "cool" factor of Web page and lose sight of what's most important to users -- usability. Don't fall into that trap.