There are many possible ways to judge the intelligence of a potential AGI. However, there are two criteria for intelligence that matter the most.
1. Is the AGI intelligent enough to destroy humanity in the pursuit of its goals?
2. Is the AGI intelligent enough to be able to steer the future into a region where no AGI will ever arise that would destroy humanity?
If an Unfriendly AGI reaches the level of intelligence described in (1) before a Friendly AGI reaches the level of intelligence described in (2), mankind perishes.
To paraphrase Glengarry Glen Ross: First prize in the race between FAI and UFAI is universal utopia. Second prize is, everyone dies. (No gift certificate, not even a lousy copy of our home game.)
Wednesday, March 26, 2008
Sunday, March 9, 2008
Making the Best of Incomparable Utility Functions
Suppose there are only two possible worlds, the small world w that contains a small number of people, and the large world W that contains a much larger number of people. You current estimate there is a .5 chance that w is the real world, and a .5 chance that W is the real world. You don't have an intuitive way of "comparing utilities" between these worlds, so as a muddled compromise you currently spend half your resources on activities AW that will maximize utility function UW if W is real, and the other half on Aw to maximize utility Uw if w is real.
This is not a pure utilitarian strategy, so we know that our activities may be suboptimal; that we can probably do better in terms of maximizing our total expected utility. Here is one utilitarian strategy, among many, that usually dominates our current muddled compromise.
Calculate expected utility EU' of a local state s as:
EU'(s) = P(W) U'(s|W) * P(w) U'(s|w)
With P being the probablity, and U' being any new utility function that obeys these constraints:
1. U'(s|W), the utility of s if W is real, is an affine transformation of UW.
2. U'(s|w), the utility of s if w is real, is an affine transformation of Uw.
3. If the only activities you could do were AW and Aw, and P(W) and P(w) are both .5, then EU' is maximized when you spend half of your resources on AW and half on Aw.
What have we gained?
1. We now have a way to evaluate the utility of a single action that will help in both W and w.
2. We now have a way to evaluate the utility of a single action that helps in W but harms in w, or vice versa.
3. As future evidence comes in about whether W or w is real, you have a way to optimally shift resources between AW activities and Aw activities.
This is not a pure utilitarian strategy, so we know that our activities may be suboptimal; that we can probably do better in terms of maximizing our total expected utility. Here is one utilitarian strategy, among many, that usually dominates our current muddled compromise.
Calculate expected utility EU' of a local state s as:
EU'(s) = P(W) U'(s|W) * P(w) U'(s|w)
With P being the probablity, and U' being any new utility function that obeys these constraints:
1. U'(s|W), the utility of s if W is real, is an affine transformation of UW.
2. U'(s|w), the utility of s if w is real, is an affine transformation of Uw.
3. If the only activities you could do were AW and Aw, and P(W) and P(w) are both .5, then EU' is maximized when you spend half of your resources on AW and half on Aw.
What have we gained?
1. We now have a way to evaluate the utility of a single action that will help in both W and w.
2. We now have a way to evaluate the utility of a single action that helps in W but harms in w, or vice versa.
3. As future evidence comes in about whether W or w is real, you have a way to optimally shift resources between AW activities and Aw activities.
Sunday, February 24, 2008
Disagreement Checklist
If you seem to disagree with someone about a factual statement, it's important to consider all four possible explanations:
1. The other person's conscious beliefs are incorrect.
2. Your conscious beliefs are incorrect.
3. The other person is deliberately deceiving you about his conscious beliefs.
4. You have the same conscious beliefs, but your understanding of what the other person is trying to say is incorrect, and you perceive a factual disagreement where none exists.
Obviously more than one of these can apply to a given situation.
1. The other person's conscious beliefs are incorrect.
2. Your conscious beliefs are incorrect.
3. The other person is deliberately deceiving you about his conscious beliefs.
4. You have the same conscious beliefs, but your understanding of what the other person is trying to say is incorrect, and you perceive a factual disagreement where none exists.
Obviously more than one of these can apply to a given situation.
Sunday, February 10, 2008
Majoritarianism and the Efficient Market
Posit four cases where your opinion differs from the majority:
1a. You are considering what checkout counter to go to at the supermarket; there are no lines. You are about to go to counter A, where the cashier looks like he'd be faster, when you are told that 3/4 of the people who shop here use counter B instead when there are no lines.
1b. The checkout counters have long lines; counter B's line is three times as long as counter A. The line at counter B looks to be moving only twice a fast, so you feel inclined to use counter A.
2a. You are, bizarrely, forced to make a 50-50 bet on whether Sun Microsystems will make a profit this year. You are about to bet "yes", when you are informed that 3/4 of the people offered this bet said "no".
2b. You are forced to either sell short or buy long on Sun Microsystem's stock. You are about to buy long when you are informed that three times as many people have shorted the stock as have gone long on it.
Philosophical Majoritarianism suggests you consider changing your mind in cases 1a and 2a, but not 1b and 2b. The reason is that, to the degree that Majoritarianism is valid, it doesn't matter which you line you pick in 1b, or whether you buy or sell in case 2b; the majority decision, if wise, has washed out the difference, so Majoritarianism doesn't provide direct guidance on what to do.
1a. You are considering what checkout counter to go to at the supermarket; there are no lines. You are about to go to counter A, where the cashier looks like he'd be faster, when you are told that 3/4 of the people who shop here use counter B instead when there are no lines.
1b. The checkout counters have long lines; counter B's line is three times as long as counter A. The line at counter B looks to be moving only twice a fast, so you feel inclined to use counter A.
2a. You are, bizarrely, forced to make a 50-50 bet on whether Sun Microsystems will make a profit this year. You are about to bet "yes", when you are informed that 3/4 of the people offered this bet said "no".
2b. You are forced to either sell short or buy long on Sun Microsystem's stock. You are about to buy long when you are informed that three times as many people have shorted the stock as have gone long on it.
Philosophical Majoritarianism suggests you consider changing your mind in cases 1a and 2a, but not 1b and 2b. The reason is that, to the degree that Majoritarianism is valid, it doesn't matter which you line you pick in 1b, or whether you buy or sell in case 2b; the majority decision, if wise, has washed out the difference, so Majoritarianism doesn't provide direct guidance on what to do.
Sunday, January 27, 2008
Free will and future actions
Suppose I chose to make an altruistic decision today. That decision was my choice via "free will", by which I mean that I choose my own actions, and that I take responsibility for my own actions. However, it is also the case that that decision was determined by some process, either by deterministic processes or by random quantum-mechanical fluctuations that could easily have turned out some other way. There is some set of mechanical preconditions that had to be satisfied before I could make that altruistic decision.
I intend to make reasonably altruistic decisions in the future, as well. However, if these mechanical preconditions fail to hold true for some reason in the future, I must acknowledge that, despite my current intentions, I will not make altruistic decisions in the future. I should take responsibility for the consequences of my future actions, in addition to my present actions.
Therefore, I should keep in mind the following:
* Where my present actions can influence my future actions from the mechanical point of view, there is some utility in deciding to take present actions that are likely to lead to positive future actions.
* Whenever contemplating taking on positions of power, I should avoid "moral overconfidence", that is, when considering whether power would corrupt me, I should try to objectively weigh all the data available to me, rather than believe that "free will" means that the Rolf who exists in the year 2007 can magically and infallibly determine the actions and goals of the Rolf who exists in the year 2017.
Obviously this does not mean that I can shirk responsibility for maintaining past promises; quite the opposite. In addition, the intent is not to go to an extreme and "declare war on my future self"; instead the goal is to take responsibility for all of the causal consequences of my current actions, including the consequences caused by the influence my current actions have on my future actions.
I intend to make reasonably altruistic decisions in the future, as well. However, if these mechanical preconditions fail to hold true for some reason in the future, I must acknowledge that, despite my current intentions, I will not make altruistic decisions in the future. I should take responsibility for the consequences of my future actions, in addition to my present actions.
Therefore, I should keep in mind the following:
* Where my present actions can influence my future actions from the mechanical point of view, there is some utility in deciding to take present actions that are likely to lead to positive future actions.
* Whenever contemplating taking on positions of power, I should avoid "moral overconfidence", that is, when considering whether power would corrupt me, I should try to objectively weigh all the data available to me, rather than believe that "free will" means that the Rolf who exists in the year 2007 can magically and infallibly determine the actions and goals of the Rolf who exists in the year 2017.
Obviously this does not mean that I can shirk responsibility for maintaining past promises; quite the opposite. In addition, the intent is not to go to an extreme and "declare war on my future self"; instead the goal is to take responsibility for all of the causal consequences of my current actions, including the consequences caused by the influence my current actions have on my future actions.
Saturday, January 5, 2008
Open letter on disenfranchisement in the Michigan Primary
To: the Democratic and Republican parties
In the past, I have considered myself independent. For example, I have voted for both Democratic and Republican candidates for President in the past, based on their individual merits. This may change soon, based on your actions.
You have until January 14, 2008 to resolve the threatened disenfranchisement of the Michigan primary voters. Satisfactory resolutions would include either of the following:
1. Your national party unambiguously withdraws the threat to punish the Michigan voters. Vague and unenforceable promises by the candidates to seat the Michigan delegates are not sufficient, since the candidates may break these promises if the convention vote ends up being close.
2. Your Michigan state party manages to move the primary back to February, thus making your threat moot.
On January 15th, if the Democrats have done a better job of resolving this crisis than the Republicans, I will vote a straight Democratic ticket for the next 20 years, at least in the usual case where there is no viable third-party candidate. If the Republicans have done a better job than the Democrats, I will vote a straight Republican ticket for the next 20 years, at least in the usual case where there is no viable third-party candidate.
Note that, if the current "status quo" continues, I will end up voting a straight Republican ticket, as the current Republican Party compromise of cutting the delegate count in half is marginally superior to the current Democratic Party threat of total disenfranchisement.
Sincerely,
Rolf Nelson
In the past, I have considered myself independent. For example, I have voted for both Democratic and Republican candidates for President in the past, based on their individual merits. This may change soon, based on your actions.
You have until January 14, 2008 to resolve the threatened disenfranchisement of the Michigan primary voters. Satisfactory resolutions would include either of the following:
1. Your national party unambiguously withdraws the threat to punish the Michigan voters. Vague and unenforceable promises by the candidates to seat the Michigan delegates are not sufficient, since the candidates may break these promises if the convention vote ends up being close.
2. Your Michigan state party manages to move the primary back to February, thus making your threat moot.
On January 15th, if the Democrats have done a better job of resolving this crisis than the Republicans, I will vote a straight Democratic ticket for the next 20 years, at least in the usual case where there is no viable third-party candidate. If the Republicans have done a better job than the Democrats, I will vote a straight Republican ticket for the next 20 years, at least in the usual case where there is no viable third-party candidate.
Note that, if the current "status quo" continues, I will end up voting a straight Republican ticket, as the current Republican Party compromise of cutting the delegate count in half is marginally superior to the current Democratic Party threat of total disenfranchisement.
Sincerely,
Rolf Nelson
Saturday, December 29, 2007
Foresight Exchange
I opened a Foresight Exchange account today (rolf.h.d.nelson). The main purpose is to challenge myself to become a better thinker by forcing myself to think through both the sides of why future events may or may not happen. My initial plan was to buy or sell one share in each of the "U.S. News" categories. I got through about six of the items before I gave into the temptation to put all the rest of my money down on "No" for the mispriced USAGeo.
The presence of mispriced items in this play-money exchange didn't surprise me, especially for "X will happen by the year Y." Presumably people put down "Buy if it goes down to price Z" orders, and as year Y comes closer and the price drops naturally well past Z, they have little incentive to log back in and rescind their now-insane orders (especially if they've abandoned their accounts.)
What did surprise me was how *boring* the brief experience was. Most of the decisions revolved, not around profound questions of philosophy or ideology, but around peering at graphs and trying to extrapolate probabilities of non-controversial events. The poor user interface added to the boredom factor as well.
The presence of mispriced items in this play-money exchange didn't surprise me, especially for "X will happen by the year Y." Presumably people put down "Buy if it goes down to price Z" orders, and as year Y comes closer and the price drops naturally well past Z, they have little incentive to log back in and rescind their now-insane orders (especially if they've abandoned their accounts.)
What did surprise me was how *boring* the brief experience was. Most of the decisions revolved, not around profound questions of philosophy or ideology, but around peering at graphs and trying to extrapolate probabilities of non-controversial events. The poor user interface added to the boredom factor as well.
Subscribe to:
Posts (Atom)