This will necessarily be a somewhat rambling collection of my thoughts as I don't feel I've settled down to any solid conclusions on the matter. Perhaps just writing this down will clarify some of my thinking, or perhaps the ideas here will fetch some comments from others that will help point me in better directions. This is the development of some of the thinking from my earlier comments on measuring wrongness - the science I'm familiar with centers on measurement and quantification, and it certainly seems potentially fruitful to consider ways in which one could impose measures (of "wrongness" or "rightness" or just "uncertainty") on the world of ideas. In a sense that kind of measurement is what we impose under the banner of peer review, though the visible outcome is more a binary (publish/don't publish) than continuous measure. And that will be one of the issues - is the natural measure for the world of ideas a continuous one, or the binary true/false we're perhaps more naturally comfortable with, or something in between? Perhaps a three-valued measure: true, false, and subject to debate, as suggested by the Sphere of Legitimate Debate concept? But first - just what is this "world of ideas" that I propose to measure?
I was interested recently to run into the rationalist website Less Wrong - the title alone suggests my theme here. They have a nice collection of essays and specific examples that prove human concepts of probability and risk tend to be wildly inconsistent. Is this proof of natural human irrationality? Perhaps - but much of the debate seems more about meanings and understanding of concepts rather than clear illogic.
Along other lines, I recently read Douglas Hofstadter's book "I am a strange loop" and found it both inspiring and troubling, in a way. Loops are at the heart of nonlinearity and the complexity that makes the world interesting, but loops are also dangerous and eagerly avoided by the analyst. Circular reasoning is rightly condemned within pure logic and feedback loops are the bane of every sound system, but loops of self-consistency seem naturally at the heart of how we make sense of the world, and what true understanding is. I think one of Hofstadter's points is that it is only through level-crossing loops that you can break out of simple tautology into true understanding. Perhaps this is at the heart of the "inductive" reasoning that Popper persistently attacks in his "Logic of Scientific Discovery" (which I've only just started reading, and will likely have more comments on later...) Economic loops are central to growth; the familiar chicken-and-egg loop is at the heart of life itself. What exactly Hofstadter means by "strange" loops I'm not sure - it has something to do with the essence of subjective "I"-ness, and perhaps a little bit of quasi-religious sentiment on his part. In any case, the central message of the importance of self-referential loops to human thought and much else is clear.
Then Physics Today last month ran my old friend David Mermin's latest "Reference Frame" - What's bad about this habit - specifically on the problem with "inappropriately reifying our successful abstractions". We too often assume that the stories science gives us about the world are "true" in the sense of physically exactly describing the underlying reality. But what science gives us is not truth about reality, but rather what is useful in making predictions about the world. The epicycles of Ptolemy were certainly not a "true" description of the motions of the planets, but they were useful in predicting planetary motions. Asimov's example of the flat Earth was certainly not true, but it's not a bad approximation for local regions of Earth's surface, and the flatness assumption is still used in local mapmaking. The "caloric fluid" theory of heat was certainly wrong, but still useful enough for Carnot to work out the limits of the efficiency of heat engines and lead the way to the important concept of entropy. Galilean relativity and Newtonian gravity were not absolutely "true", as became clear after Einstein came along, but they are still highly useful in everyday engineering situations. And the same could just as well go for our current theories about quarks and quantum mechanics and even Einstein's ideas. Maybe some of these will persevere for as long as humans exist, and never be shown false under any conditions, but we can't know that now. If evidence for some more subtle underlying structure in nature comes to light (and is sufficiently reproducible etc. to be convincing) scientists would by and large be delighted to leave those old theories behind in favor of the latest understanding. What we can know is that these scientific theories are useful now in describing and making predictions about the world around us. And that's really all we can ask of the theories of science; expecting them to provide a complete description of reality is a misunderstanding of what science actually does for us.
Most recently, thanks to a "tweet" from Tim O'Reilly, I discovered the work of Alfred Korzybski, in particular his Science and Sanity. The thesis seems to be that our assumption of Boolean (or Aristotelean?) logic has led us astray and the world of ideas should be handled quite differently - this echoes some of my own thinking on reading Popper. But I've barely touched the surface of Korzybski's writings yet, it looks intriguing, but it'll take a while to actually get to understand what he's on about.
So, where do all these different strands of thought take us (or me, at least) so far in understanding the world of ideas, and how one could measure it? First there's the importance of context. Even a tautology is intrinsically true only within the context of the meaning we assign to the terms of the statement. Wittgenstein wrote a whole book that appeared to me to boil down to just this (his Tractactus). In all our ideas there is that base context of definition, reference to physical reality or perception - a statement is just a collection of symbols until we give those symbols meaning. For example the positive integers, as symbols, develop their meaning in any individual's mind through the experience of referring numbers to collections of objects - each of those references is in itself a statement about the world ("this box contains 3 marbles" or "I am holding 7 balloons") that could itself be true, false, or unknown (or along some continuum of confidence), and again has its own referential context. Popper takes care to distinguish between "universal" and "singular" statements, but I'm not sure that's really a meaningful distinction - the real issue for his argument is whether a statement is repeatedly testable or not, which even such a seemingly "singular" statement relating number symbols to the real world will often be (by repeatedly applying the "counting" test, for instance - at least until a balloons pops or a marble is lost). The process of counting itself is defined by reference to these sort of specific repeatable tests that convincingly give the same answer over and over, and so one has in addition to the symbols and their meanings for individual small numbers on their own, the concepts of "counting", "base-10 place-value notation", "addition" and other mathematical operations, and so forth.
That is, given the context of a given person's understanding of the symbols representing numbers, then all sorts of consequences about relationships between those numbers logically follow - the behavior of primes, Fermat's theorem, and so forth. The starting context is a collection of statements that person asserts to be true either from consistency of symbolic representation in experience ("3 refers to so many marbles") or as definitions ("a prime has no divisors between 1 and itself") where the terms are again defined in terms of more basic symbolic relations to experience.
Now the interesting cases are probably not the provably true (or false) statements of mathematics but the more ambiguous statements of the practical sciences. To return to Asimov's "flat Earth" example, the statement "the Earth is flat" isn't a bad approximation in appropriate contexts. Our experience of the pull of gravity (plus a small effective centrifugal force from Earth's rotation!) and the actions of a level or plumb line determine a local horizontal plane, and it's certainly useful to build structures based on those practical definitions without worrying about any underlying curvature. Whether the Earth is actually donut-shaped or apple-shaped or something else is a global property, and doesn't affect the existence of a local horizontal plane that is the contextual reference for Earth's local "flatness". Nevertheless, once you leave the local context, the implications of the "flat-Earth" assumption break down. The sun provably does not shine at the same angle relative to the horizontal at points far apart on Earth's surface. Going far enough in a straight line, you come back to the same point without falling off an edge. Following higher lines of latitude (defined by average Sun angle) it takes less and less distance to go all the way around. The logical consequences of an actually "flat Earth" would require none of these statements to be true. For those who have never heard or believed any of these statements and have no reason to test global properties of the planet themselves, continuing to accept "the Earth is flat" as applicable to the entire planet is logically consistent, possibly useful, and likely harmless. But accepting any one of these statements (or others with similar implications) into your personal context either forces rejection of "the Earth is flat" as globally applicable, or leaves some form of cognitive dissonance or logical inconsistency in the context - something the rational ought to avoid.
What I think I'm trying to get at here is that when we focus on the truth or otherwise of a single statement, like "the Earth is flat", there is a surrounding context of knowledge and accepted statements and definitions and relations to the external world that determines the degree to which such a statement is compatible or incompatible with reality as we otherwise understand it. That surrounding context could be defined by the sort of measure I've been talking about, the true/false/unknown status which a given person assigns to those statements. That measure could be mathematically represented as a function from the space of ideas to some small target space. The target space is most likely simply three discrete values (true, false, unknown) or a number between 0 and 1 as a probabilistic estimate of certainty, or perhaps another relevant representation of how people assess a statement.
I.e. we have a context function c that maps points of statement-space 'S' to the target-space 'T' -
c: S -> T
so that for any given statement 's' (about a specific event, a general idea, another person), a given person's contextual opinion at a given time is given by the value c(s) in the target space T.
This context will be continually changing as the person experiences new things about the world (and, in some cases, forgets or discounts old things they previously thought they knew). The context may also change through a reasoning process, reflection about the relative consistency of those things that are known - the self-consistency of the function 'c'. Adoption of a broad-ranging theory (s1 = "the Earth is flat on a global scale") together with other reasonable context implies a large number of consequent statements that must also be true (s2 = "Earth is flat to within a few km over a 100 km distance", s3 = "the sun's angle to the horizontal must be the same in Alexandria as in Syene"), so a rational context (with realistic knowledge about the Sun, Earth's topography, etc) where c(s1) = true must imply c(s2) = true and c(s3) = true. When a new experience comes along that convincingly demonstrates c(s3) is false, that one change in the context function 'c' has necessary ripple effects in returning to self-consistency - given present-day accepted truths about the world, we would expect showing c(s3) to be false would force c(s1) to be assessed as false also. But the logic of those ripples depends on the whole context function c, not just on the individual statements s1 and s3. It could well be, given the context, that a better logical conclusion for that individual would be to retain s1 and drop some other assumption about the distance of the Sun from the Earth.
That is, the forcing of a particular context change F(c): c -> c', where c(s3) = true and c'(s3) = false, with c' = c everywhere else, can introduce logical inconsistencies in that personal context. The person may be happily unaware of the inconsistency and do nothing about it. In fact, among the statements in statement-space 'S' are naturally some statements about the context 'c'; among these would be, say, sc1 = "context c is consistent", and sc2 = "context c' is inconsistent". Within c (and c') the person may be quite unaware of the logical implications and evaluate c(sc2) as unknown. But if the person becomes aware (or is aware from the start) that sc2 is true, then we may enter yet another contextual state (or have c(sc2) = true from the start) and there is motivation to resolve the inconsistency in the present context c'.
That is, the forcing f plus the awareness c("context c' is inconsistent") = true brings about a response r(f): c' -> c''. If this resolves all inconsistencies (that the person is aware of), then we're done, but in many cases the changes induced by the response may induce a further response r(r(f)) (adoption of an alternative theory, reassessment of the trustworthiness of the source of the original theory, etc.) and so forth. In other words, what we have here is a loop of self-referentiality associated with rational thought, characterized by the response function r.
And if the statement space 'S' is vast, the domain of the rational response function 'r' is infinitely vaster still. Remember c is a mapping from 'S' to 'T', so if 'T' is even just 3 discrete values, the number of possible such mappings c (the size of the context space 'C') is 3 to the power of the number of possible statements 'S' (c can have 3 values for each element of 'S'). Context changes like f map C to itself, so the size of that context-change space 'F' is the size of 'C' raised to the power of itself. And finally r, the rational response function, is a mapping from 'F' to itself.
I think the incredibly vast dimensions of this domain of rational response are a good part of the reason why we humans have such trouble talking about thought, and communicating in general. But I think a full accounting of that vastness needs to be the starting point for better understanding the (self-referential, loopy, and definitely strange) process of human understanding itself, both within science and in more common-sense areas of knowledge.
More to come, comments definitely appreciated!