Saturday, December 29, 2007

Foresight Exchange

I opened a Foresight Exchange account today (rolf.h.d.nelson). The main purpose is to challenge myself to become a better thinker by forcing myself to think through both the sides of why future events may or may not happen. My initial plan was to buy or sell one share in each of the "U.S. News" categories. I got through about six of the items before I gave into the temptation to put all the rest of my money down on "No" for the mispriced USAGeo.

The presence of mispriced items in this play-money exchange didn't surprise me, especially for "X will happen by the year Y." Presumably people put down "Buy if it goes down to price Z" orders, and as year Y comes closer and the price drops naturally well past Z, they have little incentive to log back in and rescind their now-insane orders (especially if they've abandoned their accounts.)

What did surprise me was how *boring* the brief experience was. Most of the decisions revolved, not around profound questions of philosophy or ideology, but around peering at graphs and trying to extrapolate probabilities of non-controversial events. The poor user interface added to the boredom factor as well.

Sunday, December 9, 2007

Am I corruptible?

Lord Acton said that "power corrupts." Others say that corrupt people in power were already corruptible to begin with; we just don't notice that people are prone to corruption before they gain power, because they never had the opportunity to benefit from corruption.

As a thought experiment, let's use the following model. 80% of the people are corruptible: that is, they will act corrupt if they become the King; there is no way of determining whether someone is corrupt before they become the King. Everyone publicly denies that they are corruptible. Two worlds exist, identical in every respect except:

In the "Self-Deceptive World", everyone has a self-image of themselves as incorruptible before they gain power.

In the "Self-Aware World", everyone is fully aware of whether they are corruptible; the corruptible people merely lie and claim that they are incorruptible.

These are the only two worlds that exist; to put it another way, the a priori odds that you live in one world rather than the other is 50%.

You have the self-image of yourself as someone who is incorruptible, but you have never been the King, and are unsure of which of the two worlds you live in. In this case, I would reason as follows:

Pick ten people at random, maybe five will be from the Self-Deceptive World, and five will be from the Self-Aware World. On average, there will be five self-deceptive people from the Self-Deceptive World with an incorruptible self-image, and one self-aware person from the Self-Aware World with an incorruptible self-image. Therefore, the odds are 5:1 that you live in the Self-Deceptive World, and are corruptible.

Sunday, November 25, 2007

Efficient Market Hypothesis

Suppose that you have a logical argument, which seems compelling to you, that publicly available information has not been reflected in an asset's price. (One example might be here, otherwise I'm sure you can pick out a different argument that has occurred to you at some point.) If you have funds to invest, should you focus investment funds in that area? I would argue, generally no, because of Copernican (majoritarian) considerations, including various forms of the Efficient Market Hypothesis.

If, instead, you have a partially Ptolemaic viewpoint, and are logically consistent, you would probably come to the conclusion that, any time you see everyone else make what seems to you like a logical mistake, you should spend significant effort in determining how you can profit from the mistake.

For example, suppose you believe that, with probability p, you are now in a 'privileged epistemological position' that will increase your expected annual returns, from 1.06 to 1.10, if you actively (rather than passively) manage you portfolio. (But with probability 1-p, there is no such thing as a privileged epistemological position. If you actively manage, but there is no such thing as a privileged position, your expected returns go down to 1.05 because of transaction costs.) If your probability p is above 0.2, you would want to actively manage rather than passively manage.

The problem with active management, of course, is that in the existing market, for every winner there must be a loser. So there's a "meta-level" where the above must be, on average, bad advice. It's not clear to me how to consistently avoid these types of traps without recourse to a Copernican Epistemology.

Saturday, November 10, 2007

Pure Copernican epistemologies

There are multiple people in the room, including you, who (even after discussion of the objective facts) all have different honest ("Experience") beliefs. You have to make a correct decision, based on those beliefs. Consider four algorithms to make the decision.

1. Always base your decision on your own ("Experience") beliefs.

2. Always go with the beliefs of whoever you judge has the most "common sense" in the room (which, by staggering coincidence, happens to be you.)

3. Always go with the beliefs of whoever's Social Security Number is 987-65-4320 (which, by staggering coincidence, happens to be your own Social Security Number.)

4. Everyone takes some sort of Good Decision-Making Test that measures your general GDM (Good Decision-Making) ability.

The first two are clearly not "Copernican Epistemologies", as they posit as axioms that you have 'privileged access' to truth. If you wish to adopt a purely Copernican Epistemology, you would reject (1) and (2). Would you have a preference between (3) and (4)? Both (3) and (4), on the face, Copernican. But the decision of what algorithm to choose make your current decision is, itself, a decision! If you apply a Copernican process to that decision, and so on recursively, you would (in theory) eventually come back to some small set of consistent axioms, and would reject (3).

I personally believe that a normative "Theory of Everything" epistemology would have to be purely Copernican, rather than partially Ptolemaic. To elaborate, it would have to be an epistemology where:
  1. There are a relatively small set of axioms (for example, there is no room for axioms that directly reference Social Security Numbers)
  2. None of these axioms explicitly reference yourself as a privileged source of knowledge, with the exception that I would allow some privileged access to your own current consciousness, and your own current thoughts and beliefs. (You do not have privileged access to your past feelings, thoughts, and beliefs; you have to infer those from your current thoughts and beliefs, like everyone else.) To be clear, this privileged access would not be of the form "I have privileged knowledge that my belief about X is correct," but rather, of the form "I have privileged knowledge that I know that 'I believe X is correct.' In contrast, I don't know whether Joe believes that 'X is correct'; he says he does, but for all I know, he's deliberately lying."

Saturday, November 3, 2007

Some hypothetical answers for the Wire Disagreement Dilemma

Here is a sampling of possible answers for the Wire Disagreement Dilemma:

1. Always go with your own "Experience" beliefs (cut the red wire).

2. Always go with the beliefs of whoever's Social Security Number is 987-65-4320 (which, by staggering coincidence, happens to be your own Social Security Number.)

3. Always go with the beliefs of whoever you judge has the most "common sense" in the room (which, by staggering coincidence, happens to be you.)

4. Always go with the majority belief.

5. Always go with the belief of the person who had the highest IQ test scores on his most recent test.

6. Always go with the person with the most education (as measured in years of schooling).

7. Assign a score, based on a preconceived formula that weights one or more of the previous considerations. Then go with whoever has the highest score, unless you really dislike the outcome, it which case go with your "Experience" beliefs.

Ptolemaic vs. Copernican Epistemologies. One of the differences between these solutions is the degree to which they presuppose that you have privileged access to the truth. For lack of a better term, I would call systems Copernican Epistemologies if they posit that have no privileged access to the truth, and Ptolemaic Epistemologies if they posit that you do have privileged access to the truth. This is a spectrum;
"Always go with your own 'Experience' beliefs" is the exemplar of Ptolemaic belief; "I have no privileged 'Experience' beliefs" is the exemplar of Copernican belief; there are plenty of gradients between.

Note that it is not possible for a human to actually implement a 100% pure Ptolemaic belief system, nor a 100% pure Copernican belief system. For example, your beliefs of "what I would have believed, apart from other peoples' opinions" will, in practice, be tainted by your knowledge of what other people believe.

Sunday, October 28, 2007

Wire Disagreement Dilemma

You are locked in a room with two other people and a time bomb. To disarm the bomb, you must choose correctly between cutting the red wire or the blue wire on the bomb; cutting the wrong wire, or failing to cut either of the wires in time, will trigger the bomb. Any one of the three of you can choose to lunge forward and cut one of the wires at any time.

Each of you puzzles over the circuit-wiring schematic. You find an airtight, 100% certain proof that the red wire is the wire that needs to be cut. But simultaneously, your two allies report that they have come up with airtight, 100% certain proofs the blue wire needs to be cut! You cannot come to a consensus, either because you do not have time, or because you simply cannot understand each others' proofs.

Your choices are:

1. Lunge forward and cut the red wire.

2. Allow your allies to cut the blue wire.

How do you make your decision? Call this the Wire Disagreement Dilemma.


1. According to the most straightforward application of classical logic, you should lunge forward and cut the red wire.

2. Philosophical Majoritarianism doesn't tell you exactly what to do. PM seems to be a heuristic that you use alongside other, sometimes conflicting, heuristics. As I've seen it outlined, it doesn't seem to tell you much about when the heuristic should be used and when it shouldn't.

3. There's a sense in which you never have an actual proof when you make a decision, you only have a memory that you had a proof.

4. Consider two people, Alice and Bob. Alice should not automatically give her own beliefs "magical precedence" over Bob's beliefs. However, there are many circumstances where Alice should give her own beliefs precedence over Bob's; there are also circumstances where Alice should defer to Bob.

5. This type of thinking is so rare, that (to my knowledge) we don't even have a short word to describe the difference between "I believe X because I reasoned it out myself" and "I believe X because someone smarter or more experienced than me told me X, even though, on my own, I would have believed Y."

In normal conversation, you have to use cumbersome phrases and idioms: for example, "it seems to me like X" in the former case and "my understanding is that X" in the latter case.

Experience vs. Hearing: As technical terms, I'd propose that in the former case we say "I Experience X" or "my Experience is X." In the latter case we can say "I Hear that X" or "my Hearing is X."

6. One asymmetry, when Alice is evaluating reality, is that she generally knows her own beliefs but doesn't necessarily know Bob's beliefs. Bob may be unavailable; Bob may be unable to correctly articulate his beliefs; Alice may misunderstand Bob's beliefs; there may not be time to ask Bob his beliefs; or Bob may deliberately deceive Alice about his beliefs.

Saturday, October 20, 2007

Occam's Meta-Razor

Let me define the Occam's Meta-Razor Problem as follows: What is the smallest and simplest set of basic philosophical postulates that a rational agent needs in order to act in a way that is intuitively satisfactory? The goal is that the behavior should satisfice, even if it's not necessarily optimal. Call this the Occam's Meta-Razor problem.

Intuitively, I think we want three items:

1. A simple way to analyze probabilities. Something like Solomonoff Induction might satisfice, if the Pascal's Mugging problem were solved.

2. A utility function. An initial start might be, “Maximize the expected amount of X in the Universe,” where X is some weighted combination of happiness, freedom from pain, autonomy, etc. A satisfactory but simple description for X would be difficult to unambiguously specify, especially in the case where the agent wields super-human intelligence. Two of many possible pitfalls:

  • For almost all X, the current set of humans who are alive (and humanity in general) are going to be sub-optimal, from the point-of-view of the agent. However, we want the agent to decide against wiping out humanity and replacing it with species that are “more worthy” according to its utility function.
  • We would want some portion of X to include the concept of “autonomy” and preserve our abilities to make informed, uncoerced decisions. But, a sufficiently smart agent could peacefully convince (trick?) me into making any number of ludicrous decisions. It's not clear how to unambiguously define “coercion” in the case of a super-intelligent agent.

3. A simple decision theory, such as Evidential Decision Theory (which I believe subsumes Hofstadter superrationality), or Causal Decision Theory (which is the standard in mainstream Game Theory.) Either should satisfice, though I regard Evidential Decision Theory as much simpler.

Being philosophical principles, obviously these can't be directly used to create a real, resource-limited AGI; for example, Solomonoff Induction is too slow for practical use.

But, as a set of normative philosophical principals for a human being to use, these seem like a reasonable starting point.

[edit -- decided to call it "Occam's Meta-Razor" rather than "Meta-Occam's Razor"]

Saturday, October 13, 2007

Superrationality and the placebo effect

Let me introduce a conjecture that I will call the Strong Biological Placebo Effect. The Strong Biological Placebo Effect states: if you believe a course of action can improve your health, then the mere belief invariably triggers biological changes that improve your health.

If the Strong Biological Placebo Effect is true, then it creates a situation where superrationality applies to human beings. You can consistently, and rationally, choose to believe that setting your alarm clock to prime numbers will increase your health; alternatively, you can consistently, and rationally, choose to believe that setting your alarm clock to prime numbers will not increase your health. If you are superrational, you will choose the latter option, and you will be healthier because of the superrationality.

(Caveat: the Strong Biological Placebo Effect is probably not even remotely true, so don't whip out your magnetic bracelets quite yet.)

Saturday, October 6, 2007

Wild Guess: Singularity in 2024

Suppose there is no World War III; suppose that there's no single disaster sufficient to wipe out more than, say, 10% of mankind in a single year. When will the Singularity arrive?

Kurzweil's scenario gives us affordable human-level hardware around 2024, according to my interpretation of his graph. I find his "accelerating exponential growth" model of pre-Singularity computer hardware to be more reasonable than straight Moore's Law, especially factoring in possible nanotech and biotech improvements. Note that Kurzweil states that his models gave him "10^14 to 10^16 cps for creating a functional recreation of all regions of the human brain, so (he) used 10^16 cps as a conservative estimate." I'm interested in "most likely" rather than "conservative", so I used 10^15, but that doesn't make a huge difference. I also picked a "most likely" spot on the gray error-bar rather than a conservative extreme, which does shift things significantly.

Kurzweil believes that the Singularity would arrive decades after we have cheap human-level hardware, but I think it's more likely to arrive a little bit ahead or a little bit behind. So, my wild guess is 2024: meaning that while it's unlikely to arrive in exactly that year, I give it a 50/50 odds of being before or after. Of course, it could end up being "never". It could end up being next year.

Saturday, September 29, 2007

Actively Open-Minded Thinking

Suppose you have decided to spend ten minutes on making a decision between A and B. Currently you're leaning towards A. Do you:

1. Spend most of the ten minutes thinking of reasons to do A?

2. Spend about equal amounts of time thinking of reasons to do A, and reasons to think of doing B?

3. Spend most of the ten minutes thinking of reasons to do B?

Our usual decision is to do (1). This is an aspect of the confirmation bias.

Part of Actively Open-Minded Thinking (AOMT), as advocated by Baron, is to try to instead do (2) or (3). This is hard to get in the habit of doing.

I find Actively Open-Minded Thinking very useful. For example, twice a day I think about an aspect of a current plan I have for the day, month, or lifetime, and then I briefly try to search for alternative strategies, or for reasons why my current strategy might be wrong; on multiple occasions this has caused me to adopt new courses of action. It's irrational, given that you're going to spend X minutes meditating on a decision anyway, to spend most of the time thinking of ways to rationalize your current decision. In fact, it's so clearly irrational that I'm retroactively surprised that AOMT never became widespread.

Sunday, September 23, 2007

...And, Scenarios Without a Strong AI

  1. A giant disaster or war occurs, say on a scale that kills off more than 20% of humanity in a single year.
  2. Getting the software right turns out to be harder than we thought. Sure, natural selection got it right on Earth, but it has an infinite universe to play with, and no one is around to observe the 10^10000 other planets where it failed.

Thursday, September 20, 2007

Strong AI Takeoff Scenarios...

Things that could cause the current Moore's Law curve to be greatly exceeded:
  1. Enormous nanotechnology advances.
  2. Enormous biotech advances, allowing dramatic intelligence augmentation to human brains.
Things that could cause a sudden, large (1000x) increase in hardware devoted to AI self-improvement:
  1. Self-improving AI, nowhere near *general* human-level intelligence yet, is suddenly put in charge of a botnet and swallows most of the Internet.
  2. Awareness of the promise or the threat of Strong AI becomes widespread. A government, or government consortium, launches an Apollo Project-scale activity to create a Strong AI, hopefully under controlled conditions!
Other Strong AI Takeoff scenarios:
  1. The usual "recursive self-improvement" scenario
  2. We were too conservative. Turns out creating a Strong AI on a mouse-level brain is easier than we thought; we just never had a mouse-level brain to experiment with before.

Tuesday, September 18, 2007

Total feasible human population over all time

On Usenet I recently, for fun, made a back-of-the envelope calculation for how many human beings could be born in one scenario:
  1. For simplicity of calculation, the only lifeforms are humans (no trans-humans on microchips).
  2. The limiting factor will be energy, rather than how much carbon etc. you have lying around to build things with. After all, you can fuse the useless hydrogen you come across into more useful elements, or delay peoples' births until room opens up for them.
  3. I put the energy efficiency at only .001 of total matter (dark and baryonic), partly because there will be inefficiencies with collecting/storing/transporting/transforming the energy, and partly because I'm not sure how well we can shove the dark matter into black holes. I used (3*10^-27 kg / (meter^3)) for the matter density.
  4. Having no source on how much matter we can grab or colonize before the accelerating expansion of the universe puts it out of reach, I arbitrarily guess that we can colonize all matter currently within 10 billion light-years of us.
  5. The miserly energy ration will be 300,000 nutritional Calories per day, for use for synthesizing/recycling food, and for all other pro-rated energy uses.

Google tells me:

(.01 * ((4 * pi) / 3) * ((10 billion light years)^3) * ((3 * ((10^(-27)) kg)) / (meter^3)) * (c^2) * .001) / ((300 000 (kilocalories / day)) = 10^52 person-years.

Sunday, September 16, 2007

The Virtues of Wild Guesses

Some probability distributions are very uncertain. For example, what are the odds of Strong AI by 2040? Not only is there a lot of uncertainty, but there is also "uncertainty to the uncertainty"; I have no rigorous way of answering the question.

Other questions are close to zero-knowledge. For example, what is the probability of a Singularity arising in the average star system? I'd put an upper-bound of 10^-16, based on the fact that no alien civilization has yet shown up to collect the unused sunlight pouring out uselessly into empty space. But, I have no lower bound.

Suppose today you have to make a decision based on one of these questions. There are three things you can do:
  1. Decide arbitrarily, or decide solely according to heuristics that are unconnected to the reality at hand. Examples: Always choose inaction. Continue doing whatever you were doing before you pondered the question. Do whatever makes you feel best today. Do whatever would result in the least embarrassment if you're wrong. Do whatever everyone else is doing.
  2. Make your best wild guess, but keep it to yourself to avoid political embarrassment. After all, everyone has some People Who Would Love To Take You Down A Notch in their lives; the last thing you want to do is say something in public that turns out to be embarrassingly wrong.
  3. Collaborate. Make your best wild guess, and share it freely, taking pains to label it as a wild guess of course! Just be aware that if you're wrong, the defense of "I *said* it was a wild guess, and anyway I was braver and more rational that the people who refused to guess at all!" is unlikely to be heard.
Choices (2) and (3) are reasonable approaches. Unfortunately, (1) is probably the most common; we tend to inertially push forward with our current paths until someone can prove that our current path is wrong.

Saturday, September 15, 2007

Diminishing Returns on Existential-Level Threats

I posted a vague suggestion as a comment to the Lifeboat blog:
When considering what problem to work on, one question is "how many other people are working on this problem"? If the answer is "a lot", you may stay away because of the Law of Diminishing Returns. (This is only partly mitigated by the fact that if a lot of other people agree that P is an important problem and are working on it by doing S, that somewhat increases the chances that your assessment that "P is an important problem and S is a good solution" is correct.)

In the course of figuring out where to spend resources, people and organizations like the Lifeboat Foundation are presumably tracking who else is working on what problems and how many resources are being spent by other organizations. Ideally, the Lifeboat Foundation should publish their order-of-magnitude estimates so that other people deciding what projects to work on can use that data as well.

Self-interested people have the same problem, of course. If there are already five companies selling apple-flavored toothpaste, you might not want your startup to sell the same thing. If no one is selling apple-flavored toothpaste, you might consider selling it, with the caveat that you'd first want to make sure there isn't a *really good reason* why no company has attempted to sell apple-flavored toothpaste. The difference between self-interested people and an (ideal) nonprofit is that self-interested people have less incentive to share their research with each other.
I don't expect an enthusiastic reception. Apart from everything else, there are strong political reasons why people and organizations (including you and me) don't like to publicly share all the Wild Guesses that inevitably make up the foundations of our long-range strategies.