Also, I must offer a disclaimer. I've never actually written a successful query letter myself, and cannot remember if I've ever written an unsuccessful one. It would have been in the mid-1990s if I did. My process has been different — I first secured my agent relationship with arcaedia on the strength of my short fiction work (among my first words to her, ever: "I'm not ready for an agent"), and all my novels have sold either through her or through direct discussions with the publishers.
That being said, there's some key info that's worth keeping in mind if you write a query letter. First off, make a distinction between a query letter and a transmittal letter. The transmittal letter is another word for cover letter — it's what comes with the partial or full manuscript when that is sent to an agent or editor at their request. A transmittal letter is simple and businesslike. It reminds the recipient what you're sending and why, signs off politely and gets out of the way. More to the point, the transmittal letter is only indirectly a sales tool.
A query letter, on other hand, is first and foremost a sales tool. It needs to explain what you're offering and why your manuscript is interesting. It needs to pique the interest of the target audience and engage them. More to the point, it needs to not oversell, or focus on the wrong things. What goes into it?
Write to the right person.
There's no point in sending a letter to an agency or a publishing house if you're not going to address it to a specific agent or editor. Furthermore, be targeted — sending your hard science fiction novel to a romance agent doesn't make all that much sense.
The opening sentence is like the first moments of a blind date. They've opened the envelope (or email), they're glancing across the page, and those words hit their ears. You want to be professional, informative and interesting, without either pulling your punch or overwhelming your target. How you do that very much depends on your personal style, the style of your manuscript, and the approach you want to take. Just lead with what's strong and interesting, and don't tell them what to think.
Be informative but economical with your letter body. Explain the book in a little more detail, tell them why it's exciting and different, and give them just enough to want more without overselling or overexplaining. Remember it's not about you. It's about your book. Don't sell yourself, sell the manuscript. The only exception would be if you have some very specific qualification that has an obvious and important relationship to how the book can be promoted and sold. Keep it direct, simple and engaging.
That's it. You're done. A simple sign-off and you're out of there.
What You Didn't Write
You didn't go into detail about yourself. You didn't explain why the book was going to be good. You didn't make comments to or about the editor or agent you were trying to address. It's so easy to begin overselling, to begin spinning, to spew across the page. Just like blabbing on that first date, easy but almost always deadly.
In short, make your pitch and let the work sell itself. As always, your mileage may vary.