I recently corresponded with a colleague who has acquired a degree of "skepticism" regarding global warming. His comments to me specifically cited Freeman Dyson and Will Happer, distinguished physicists who are also well-known global-warming skeptics, and he contrasted their credentials with those of John Holdren (Obama's new science advisor) and Al Gore. The following is a lightly edited version of my response, which I'm posting here mainly as a place for some useful links on the subject.
I think I hold a pretty analytical view of the science now - like you I was quite unfamiliar with the science a couple of years ago and by default adopted the "consensus" position. However, I was challenged on this in several email lists and blog discussions and realized my understanding was pretty limited, so I've gotten into the science quite intensely over the past couple of years, including writing a couple of arXiv papers to clarify some points - the first one is this: http://arxiv.org/abs/0802.4324.
However, my study of the situation seems to have come to the opposite conclusion from you. On the simple question of the number of smart people on one side or the other of the "debate", Will Happer and Freeman Dyson are certainly distinguished, but they are also pretty much alone on their side near the top of this list of most-cited scientists who have commented in some way on climate issues.
Dyson (#47) and Happer (#85) are the highest-cited declared "skeptics", while there are 13 "activists" who have signed petitions in the other direction, and at least 20 IPCC authors, higher on the list than even Dyson. Of course climate science wasn't the central field of expertise for Dyson or Happer. Holdren, by the way, is way down the list (#668), and Gore isn't mentioned - in addressing the quality of the science based on credentials of those advocating one side or the other, one should really look at the scientists near the top of the list, I think. There are further details on who has said what and what their credentials are described on that page, and an interesting linked graph.
More important than the people is the science itself. Ray Pierrehumbert (#248 on the list) has an excellent (draft) online textbook on the subject which goes through a nice spiral of complexity in describing the atmospheric greenhouse effect, with the "gray gas model" (the assumption of my simple article mentioned above) covered in some detail, and then getting into the behavior of real gases, water vapor feedback effects, etc.
Another reference I have found extremely useful is Spencer Weart's comprehensive history of the subject, "The Discovery of Global Warming" - one thing you learn by reading that is the comprehensive collection of objections and challenges to the science that have been posed over the years. What is discouraging is seeing these same objections raised again and again, without those raising them seeming to be aware of the history and how these have been answered perfectly adequately decades ago. One site that I've found helpful in addressing recent instances of repeats of old arguments is Skeptical Science.
Of course the primary source on the state of the science has to be the IPCC reports, in particular the background science covered by IPCC Working Group 1. The 4th assessment report, IPCC AR4 WG1 can be found here:
I had a lot of questions about modeling, and found chapter 8 (Climate Models and their Evaluation) particularly helpful. Also follow some of the references found there - particularly Bony et al (Journal of Climate 19:3445 (2006)) is an excellent discussion of feedback analysis in the climate context. I even downloaded the NCAR "Community Climate Model" to take a look at the code and the documentation too, which is a textbook in itself.