The following is a collection and rearrangement of some of my comments on how we know about the radiative effects of greenhouse gases like carbon dioxide and how big they are, made on another blog. I'm posting these here as in working things out to my own satisfaction to try to respond to some rather egregiously wrong claims by the blogger there, I believe I clarified a few things in a way that's worth preserving.
The starting view here has to be the Kiehl-Trenberth diagram of Earth's energy flows, even though in some ways (which I'll get to below) it may be slightly misleading.
The problem is the transport of heat from the planet as a whole (or the surface where it's mainly absorbed) into space. Even with Earth's significant greenhouse effect as it is, some portion of Earth's heat flow never even touches the atmosphere, it goes directly out into space (this is partly why land cools off quickly on a clear dry night, for instance). In the above diagram that quantity is the 40 W/m^2 of the "atmospheric window".
To really understand what's going on one has to think about how this diagram would change under counterfactual conditions for a hypothetical version of Earth with different atmospheric constituents. If greenhouse gases (or infrared-absorbing clouds) were all suddenly removed, for instance, *all* of the thermal radiation leaving the surface would go straight into space; none would be absorbed by the atmosphere. There would be no "back radiation" returning to the surface. So the outgoing 396 W/m^2 radiated from the surface would greatly outweigh the 161 W/m^2 incoming (minus whatever convection and latent heat transports remained) and the surface would cool quickly until outgoing and incoming energy fluxes were closer to balance. In contrast, in the actual diagram the net heat flow associated with radiation is the difference between upward and downward fluxes - (396 - 333) or 63 W/m^2 (and note that 40 W/m^2 of this is never intercepted by the atmosphere). The remaining difference between incoming energy flux and outgoing radiative flux, (161 - 63) = 98 W/m^2, is accounted for by energy flows associated with convection ("thermals" and "latent heat" in the diagram) plus the 0.9 W/m^2 that the system is currently out of balance (causing the gradual warming of the surface we've been experiencing in recent decades).
Note that the 63 W/m^2 of current net radiative heat flow is less than the 161 W/m^2 it would be under a no-greenhouse-gas counter-factual condition. That is, the effect of greenhouse gases is to greatly *reduce* the net rate of radiative heat flow from the surface - and it is a reduction in outgoing heat flow, not an increase, that leads to warming. The bigger and more significant the greenhouse effect becomes, the *lower* the portion of surface cooling that happens through radiation.
Thermal radiation in the atmosphere is usually treated ballistically, as two large total energy flows (up and down), rather than the smaller net (heat) flow - a treatment repeated in the above diagram. This is because within nearly transparent systems the scattering length is long and the characteristics of source and sink can be quite different (not "local" enough for local thermodynamic equilibrium or local parametrizations to apply).
But it is really the net heat flow that matters to a thermodynamic evaluation such as the one presented here. One can approximately present it as similar to conductivity (and even convection given geometry) as
Q_rad = Q_rad0 + h_rad . (T_surf - T_trop)
Q_cond = k . (T_surf - T_trop)/l_trop
Q_conv = h_conv . (T_surf - T_trop)
(here each Q represents the rate per unit time of heat flow *out* of the surface). Q_rad0 here represents direct radiative heat flow from surface into space, while h_rad . delta T is the radiative heat flow from the surface intercepted by the atmosphere.
I.e. very roughly the total outgoing energy flow rate is represented by three contributions proportional to the surface-tropopause temperature difference, plus one that doesn't depend on that difference:
Q_out = Q_rad0 + (h_rad + h_conv + k/l_trop) (T_surf - T_trop)
The tropopause temperature is very roughly the effective radiating temperature of the planet; in any case close to fixed, so the surface temperature can be determined by balancing Q_out and Q_in:
T_surf = T_trop + (Q_in - Q_rad0)/(h_rad + h_conv + k/l_trop)
The effect of increasing greenhouse gases is:
* reducing Q_rad0 (more radiation from the surface is captured by the atmosphere)
* reducing h_rad (the effective scattering length is reduced, the atmosphere becomes more opaque to thermal radiation)
And both of these act to increase surface temperatures. There are also secondary impacts on the convective (and conductive) components through changes in the geometry (l_trop increases, for instance).
In short, GHG's act to reduce heat flow by radiation from the surface, bringing about warming.
In the limit of very high absorption the behavior of the radiation becomes diffusive - a random walk. Which is also the microscopic behavior of molecules and electrons conducting heat, so conductivity and radiative heat flow become functionally similar in that limit. The reason you can't treat it that way in the real atmosphere is because air is very far from that "high absorption" limit - and in particular some is not absorbed at all. One of the effects of adding greenhouse gases is to narrow the window of non-absorbed radiation.
Now, what is the specific effect of CO2? There is no single equation or a simple set of them going from the basic quantum behavior of CO2 to the amount of predicted warming. The reason for this is that several of the steps in the chain of logic involve high complexity: hundreds of thousands of individual absorption lines, pressure dependence and response issues dependent on the detailed profile of Earth's atmosphere. And climate modeling requires accounting for the detailed distribution of land and ocean on our planet as well. So there can be no simple equation or small set of them giving the result; any simple representation must be an approximation (of which the "sensitivity" relationship of about 3 K per doubling of CO2 usually discussed is the most useful).
But the details of the physics are generally known and accounted for with great precision; the issue is more that one needs to handle hundreds of thousands or millions of input parameters from spectroscopy observations etc, which requires computers to do the theory work properly (models). Essentially this is very straightforward physics, but it is highly complex and contingent on the properites of CO2 and Earth. It is most certainly not "ad hoc"!
Where this is "written down" in full is in model documentation - for example NCAR's Community Climate System Model has excellent details on the equations it uses. Or for a more general overview, there are the textbooks. David Archer has one with freely available online video lectures here, and Ray Pierrehumbert is also coming out with a new textbook sometime this year which I've already used in draft form myself, it's very clear on how things work.
So theoretical (model) calculations of the response of Earth to changes in CO2 are indeed the outcome of a complex computation but essentially all the ingredients are quite well known, not "unknown". The only real unknowns are a few of the response issues - which is why the results include a level of uncertainty. Those unknowns are, at least, acknowledged and quantified. And further information on the response can be found from both recent observations and historical records of how climate has behaved in the past.
In a similar fashion there is no "fundamental physics" explanation of, for example, the relation between stress and motion of, say, the Empire State Building. Nevertheless, such a relation can be assessed through basic theory, through computer and scale models, and through actual observations. Or, for example, going from the standard model of particle physics to the basic properties of interaction of atoms and molecules is similarly exceedingly computationally intensive, and is certainly not captured by just writing down a couple of equations. Nevertheless the foundations of chemistry in quantum physics have been proved time after time with detailed computations compared with observations.
So there is no contradiction between physics that is at heart simple, and having an exceedingly difficult time calculating meaningful numbers from that physics.
Now, one problem with the Kiehl-Trenberth diagram above is that it doesn't clearly relate to the terms often used to describe the effects of greenhouse gases - in particular the "radiative forcing" often talked about in relation to CO2, which has a value of roughly 4 W/m^2 when CO2 levels are doubled. Where does this number 4 enter into the diagram?
In fact, "radiative forcing" is a very different thing from "outgoing radiative energy flow from the surface" which (at 396 W/m^2) is the most visible thermal-radiation number in the diagram. For one thing, the energy flow change is calculated not at the surface, but at the tropopause. Surface energy flow changes will generally be very different from changes at the tropopause - not least because convection is involved as well as radiation, within the troposphere. This is something non-scientists like Monckton seem to be frequently confused about.
The approximation that makes "radiative forcing" a useful comparative concept at all is the 1-dimensional so-called "radiative convective" approach, first outlined in a paper by Ramanathan and Coakley (1978). Under that approximation, changes in solar forcing, aerosols, or GHGs all essentially manifest themselves through a change in radiative energy flux through the tropopause, and then given that number, the surface responds the same way to each forcing (i.e. feedbacks work the same way for any forcing change). The reason this works is that, to a good approximation, the zero gradient in temperature at the tropopause means convective and latent heat energy flows are essentially zero, so the only energy flux changes are radiative ones. It is a convenient way to divide the problem into two more tractable components: the raw effect of the CO2 change itself, and the response of the rest of the planet.
But note this is only an approximation used to better understand the problem, it is never in practice used to actually make the calculation - with models that is done by actually making the GHG changes in model atmospheres, for example.
Note again that total upwelling radiation from the surface is almost irrelevant to the level of warming if there is strong infrared absorption in the atmosphere. This is because however high the outgoing number is from the surface, the nearby atmosphere is almost as warm, and radiates almost as much back down, and other heat transport mechanisms (like latent heat flow) pick up the difference. What matters radiatively in determining warming of the surface is what gets off the planet completely - ie. if any simplistic Stefan-Boltzmann calculation is relevant it needs to be done high in the atmosphere, not at the surface. But wavelength dependence of emissivity makes that all much more complicated anyway. Which is why computer calculations are needed to get it right.
Now there is a further subdivision that can be made in the response of the planet to a given forcing, between the "bare" response, and feedbacks. This is discussed well in the 2006 review paper by Sandrine Bony et al, Journal of Climate 19:3445 (2006), "How Well Do We Understand and Evaluate Climate Change Feedback Processes?" - in particular see Appendix A. "How are Feedbacks Defined?"
In this case the simplest "bare" response is taken to be a uniform change in temperatures through the troposphere, with atmospheric composition held constant. This gives a value for the Planck response (for the change in radiative flux through the tropopause for a given uniform temperature shift) which is remarkably consistent across all the calculations that have been done - about 3.2 W/m^2K (see Brian J. Soden and Isaac M. Held, Journal of Climate 19:3354 (2006) - the first column of Table 1 shows the value from a variety of models, ranging from 3.13 to 3.26). This implies the bare temperature change to return to radiative balance at the tropopause with a forcing change of the order 4 W/m^2 would be 1.2 to 1.3 K.
And then there are the feedbacks on that bare response, discussed extensively in the Bony et al paper. Note that the "bare" response is actually a large negative, stabilizing feedback. I.e. if temperature increases 1 K, then outgoing radiation increases by 3.2 W/m^2 so that if incoming radiation has not changed, that increase in outgoing radiation creates an imbalance that cools the surface. Similarly a reduction in surface temperature creates a warming imbalance, so this bare radiative feedback is what makes temperatures close to stable. Then the other feedbacks need to be understood in the context of that main stabilizing negative feedback term.
How do we know that doubling CO2 will have that 4 W/m^2 effect on radiative energy at the tropopause and cause warming? That requires a computational analysis of the spectroscopic parameters of CO2 and other greenhouse gases in the atmosphere, but you can get a rough picture of the effect of increasing GHG levels by extending my earlier argument. I started above by introducing net radiative heat flow terms Q_rad0 and h_rad associated with radiative energy flow from the surface. Here h_rad is a bit like conductivity, while Q_rad0 completely bypasses the atmosphere. If Q_rad0 is large (and Q_rad0 is everything if there are no greenhouse gases at all) then a lot of energy escapes the planet directly into space by radiation, and warming of the surface is necessarily limited. The effect of increases in GHG's is to reduce Q_rad0 and also the conductivity-like factor h_rad - so the more GHGs there are, the lower the effective "heat conductivity of radiation" for the atmosphere.
And it is lower heat conductivity (in all senses of conduction, including convection etc) that causes raised temperatures - that's the principle behind all forms of insulation.
More explicitly, Q_rad depends on the frequency ν of radiation - Q_rad = Sum_ν Q_rad(ν). For frequencies that are completely absorbed by the atmosphere, there's a typical absorption length l_ν. The greater the GHG concentration, the more absorption, the shorter l_ν is. Very roughly, the radiative heat exchange from the surface then for frequency ν consists of outgoing radiation at a rate governed by the Planck function for the surface temperature, and then subtracting the Planck function for temperatures a distance l_ν above the surface.
The temperature at height l_ν is given roughly by:
T(l_ν) = T_surf + (T_surf - T_trop) * l_ν/l_trop
(assuming a constant lapse rate)
The Planck function B(T, ν) is given by
B(T, ν) = 2 h ν^3/c^2 . (1 / (e^(h ν/kT) - 1))
for relatively small temperature changes we can linearize it as
B(T, ν) = B(T0, ν) + (T - T0) B'(T0, ν)
so the radiative heat flow Q_rad(ν) from the surface is
Q_rad(ν) = B(T_surf, ν) - B(T(l_ν), ν)
= (T_surf - T_trop) . l_ν/l_trop . B'(T_surf, ν)
I.e. as GHG levels increase and l_ν decreases, Q_rad(ν) decreases for that particularly thermal frequency. This term contributes to the total effective radiative conductivity h_rad.
For frequencies ν that are not fully absorbed by the atmosphere, the absorbed fraction is a number ε which also governs emissions at that frequency. You then have
Q_rad(ν) = (1 - ε(ν)) B(T_surf, ν) + ε(ν) . (T_surf - T_trop) . B'(T_surf, ν)
(approximating the returning radiation as coming from the tropopause). So the ε(ν) term adds another contribution to h_rad (and this part increases as GHG levels increase), but the (1 - ε) term contributes to Q_rad0, and decreases with increasing GHG levels; the net effect of the two terms is to decrease Q_rad as GHG levels increase (higher ε). And as ε approaches 1 you get to the full absorption case where the absorption length comes into play.
So even a fully absorbing atmosphere has some continued effect on radiative heat flow from increasing greenhouse gas levels, through the decline in absorption length l_ν (reducing the temperature difference between absorption and emission points, ie. reducing the associated effective conductivity). And for partial absorption (ε < 1) the simple effect of increasing absorption (the value of ε) has that same role; for CO2 we have both windows of low absorption and regions of high absorption, so both effects are in play. And, as discussed, can be calculated quite precisely for given atmospheric composition and temperature profiles - for example via the online Modtran app.