The Multiverse According to Ben

Free Will without Counterfactuals?

2011-12-29T15:40:00.002-05:00

This brief post is an afterthought to the just-previous post about the nature of reality.

As a side point in that post, I observed that one can often replace counterfactuals with analogies, thus making things a bit clearer.

It occurred to me this morning as I lay in bed waking up, that one can apply this method to the feeling of free will.

I've previously written about the limitations of the "free will" concept, and made agreeable noises about the alternate concept of "natural autonomy." Here, however, my point is a slightly different (though related) one.

One of the key aspects of the feeling of free will is the notion "In situation S, if I had done X differently, then the consequences would have been different." This is one of the criteria that makes us feel like we've exercised free will in doing X.

Natural autonomy replaces this with, roughly speaking "If someone slightly different than me had done something slightly different than X, in a situation slightly different from X, then the result would likely have been different than when I did X in S." This is no longer a counterfactual, it's a probabilistic statement about actions and consequences drawn from an ensemble of actions and consequences done by various actors.

But perhaps that rephrasing doesn't quite get at the essence. It may be more to the point to say: "In future situations similar to S, if I do something that's not analogous to X, then something not analogous to what happened after S in situation X is likely to happen."

Or in cases of binary choice: "In future situations similar to S, if I do something analogous to Y instead of something analogous to X, then a consequence analogous to CY instead of a consequence analogous to CX is likely to occur."

This is really the crux of the matter, isn't it? Not hypothesizing about alternate pasts, nor choices from an ensemble of similar beings -- but rather, resolutions about what to do in the future.

In this view, an "act of will" is something like "an action in a situation, corresponding to specific predictions about which of one's actions will predictively imply which consequences in analogous future situations."

That's boring-sounding, but avoids confusing talk of possible worlds.

Mathematically, this is equivalent to a formulation in terms of counterfactuals ... but, counterfactuals seem to lead human minds in confusing directions, so using them as sparingly as possible seems like a good idea...

What Are These Things Called "Realities"?

2011-12-28T22:12:00.007-05:00

Here follow some philosophical musings, pursued by my rambling mind one evening during the Xmas / New years interval.... I inflicted these ramblings on my kids for a while then finally decided to shut up and write them down....

The basic theme: What is this thing called "reality"? Or if you prefer a broader view: What are these things called realities??

After yakking a while, eventually I'll give a concrete and (I think) somewhat novel definition/characterization of "reality."

Real vs. Apparent

Where did this idea come from -- the "real" world versus the "apparent" world.

Nietzsche was quite insistent regarding this distinction -- in his view, there is only the apparent world, and talk of some other "real world" is a bunch of baloney. He lays this idea out quite clearly in The Twilight of the Idols, one of my favorite books.

There's certainly some truth to Nietzsche's perspective in this regard.

After all, in a sense, the idea of a "real world" is just another idea in the individual and collective mind -- just another notion that some people have made up as a consequence of their attempt to explain their sense perceptions and the patterns they detect therein.

But of course, the story told in the previous sentence is ALSO just another idea, another notion that some people made up … blah blah blah …

One question that emerges at this point is: Why did people bother to make up the idea of the "real world" at all … if there is only the apparent world?

Nietzsche, in The Twilight of the Idols, argues against Kant's philosophical theory of noumena (fundamentally real entities, not directly observable but underlying all the phenomena we observe). Kant viewed noumena as something that observed phenomena (the perceived, apparent world) can approximate, but never quite find or achieve -- a perplexing notion.

But really, to me, the puzzle isn't Kant's view of fundamental reality, it's the everyday commonsense view of a "real world" distinct from the apparent world. Kant dressed up this commonsense view in fancy language and expressed it with logical precision, and there may have been problems with how he did it (in spite of his brilliance) -- but, the real puzzle is the commonsense view underneath.

Mirages

To get to the bottom of the notion of "reality", think about the example of a mirage in the desert.

Consider a person wandering in the desert, hot and thirsty, heading south toward a lake that his GPS tells him is 10 miles ahead. But suppose he then sees a closer lake off to the right. He may then wonder: is that lake a mirage or not?

In a sense, it seems, this means he wonders: is that lake a real or apparent reality?

This concept of "reality" seems useful, not some sort of philosophical or mystical trickery.

The mirage seems real at the moment one sees it. But the problem is, once one walks to the mirage to drink the water in the mirage-lake, one finds one can't actually drink it! If one could feel one's thirst being quenched by drinking the mirage-water, then the mirage-water wouldn't be so bad. Unless of course, the quenching of one's thirst wasn't actually real… etc. etc.

The fundamental problem underlying the mirage is not what it does directly in the moment one sees it -- the fundamental problem is that it leads to prediction errors, which are revealed only in the future. Seeing the mirage leads one to predict one will find water in a certain direction -- but the water isn't there!

So then, in what sense does this make the mirage-lake "only apparent"? If one had not seen the mirage-lake, but had seen only desert in its place, then one would not have made the prediction error.

This leads to a rather mundane, but useful, pragmatic characterization of "reality": Something is real to a certain mind in a certain interval of time, to the extent that perceiving it leads that mind to make correct predictions about the mind's future reality.

Reality is a Property of Systems

Yeah, yeah, I know that characterization of reality is circular: it defines an entity as "real" if perceiving it tends to lead to correct predictions about "real" things.

But I think that circularity is correct and appropriate. It means that "reality" is a property attributable to systems of entities. There could be multiple systems of entities, constituting alternate realities A and B, so we could say

an entity is real_A if perceiving it tends to lead to correct predictions about real_A things
an entity is real_B if perceiving it tends to lead to correct predictions about real_B things

I think this is a nicer characterization of reality than Philip K. Dick's wonderful quote, "Reality is whatever doesn't go away when you stop believing in it."

The reason certain things don't go away when you stop believing in them, I suggest, is that the "you" which sometimes stops believing in something, is actually only a tiny aspect of the overall mind-network. Just because the reflective self stops believing in something, doesn't stop the "unconscious" mind from assuming that thing's existence, because it may be bound up in networks of implication and prediction with all sorts of other useful things (including in ways that the reflective self can't understand due to its own bandwidth limitations).

So, the mirage is not part of the same reality-system, the same reality, as the body which is thirsty and needs water. That's the problem with it -- from the body's perspective.

The body's relationship to thirst and its quenching is something that the reflective self associated with that body can't shake off -- because in the end that self is just one part of the overall mind-network associated with that body.

Counterfactuals and Analogies

After one has seen the mirage and wandered toward it through the desert and found nothing -- then one may think to oneself "Damn! If I had just seen the desert in that place, instead of that mirage-lake, I wouldn't have wasted my time and energy wandering through the desert to the mirage-lake."

This is a philosophically interesting thought, because what one is saying is that IF one had perceived something different in the past, THEN one would have made more accurate predictions after that point. One is positing a counterfactual, or put differently, one is imagining an alternate past.

This act of imagination, of envisioning a possible world, is one strategy that allows the mind to construct the idea of an alternate "real" world that is different from the "apparent" world. The key mental act, in this strategy, is the one that says: "I would have predicted better if, 30 minutes ago, I had perceived desert over there instead of (mirage-) lake!"

But in discussing this with my son Zar, who doesn't like counterfactuals, I quickly realized, one can do the same thing without counterfactuals. The envisioning of an alternate reality is unnecessary -- what's important is the resolution that: "I will be better off if, in future cases analogous to the past one where I saw a mirage-lake instead of the desert, I see the analogue of the desert rather than the analogue of the mirage-lake." This formulation in terms of analogues is logically equivalent to the previous formulation in terms of counterfactuals, but is a bit more pragmatic-looking, and avoids the potentially troublesome postulation of alternate possible worlds….

In general, if one desires more accurate prediction within a certain reality-system, one may then seek to avoid future situations similar to past ones in which one's remembered perceptions differ from related ones that would have been judged "real" by that system.

Realities: What and Why

This seems a different way of looking at real vs. apparent reality than the one Kant proposed and Nietzsche rejected. In the perspective, we have

reality-systems -- i.e. systems of entities whose perception enables relatively accurate prediction of each other
estimations that, in future situations analogous to one's past experiences, one will do better to take certain measures so as to nudge one's perceptions in the direction of greater harmony with the elements of some particular reality-system

So, the value of distinguishing "real" from "apparent" reality emerges from the value of having a distinguished system of classes of phenomena, that mutually allow relatively accurate prediction of each other. Relative to this system, individual phenomena may be judged more or less real. A mind inclined toward counterfactuals may judge something that was NOT perceived as more "real" than something that was perceived; but this complication may be avoided by worrying about adjusting one's perceptions in future analogues to past situations, rather than about counterfactual past possibilities.

Better Half-Assed than Wrong-Headed!

After I explained all the above ideas to my son Zar, his overall reaction was that it generally made sense but seemed a sort of half-assed theory of reality.

My reaction was: In a sense, yeah, but the only possible whole-assed approaches seem to involve outright assumption of some absolute reality, or else utter nihilism. Being "half assed" lets one avoid these extremes by associating reality with systems rather than individual entities.

An analogue (and more than that) is Imre Lakatos's theory of research programs in science, as I discussed in an earlier essay. Lakatos observed that, since the interpretation of a given scientific fact is always done in the context of some theory, and the interpretation of a scientific theory is always done in the context of some overall research program -- the only things in science one can really compare to each other in a broad sense are research programs themselves. Research programs are large networks of beliefs, not crisp statements of axioms nor lists of experimental results.

Belief systems guide science, they guide the mind, and they underly the only sensible conception of reality I can think of. I wrote about this a fair bit in Chaotic Logic, back in the early 1990s; but back then I didn't see the way reality is grounded in predictions, not nearly as clearly as I do now.

Ingesting is Believing?

In practical terms, the circular characterization of reality I've given above doesn't solve anything -- unless you're willing to assume something as preferentially more real than other things.

In the mirage case, "seeing is believing" is proved false because one gets to the mirage-lake, one can't actually drink any of that mirage-water. One thing this proves is that "ingesting is believing" would be a better maxim than "seeing is believing." Ultimately, as embodied creatures, we can't get much closer to an a priori assumptive reality than the feeling of ingesting something into our bodies (which is part of the reason, obviously, that sexual relations seem so profoundly and intensely real to us).

And in practice, we humans can't help assuming something as preferentially real -- as Phil Dick observes, some things, like the feeling of drinking water, don't go away even if we stop believing in them … which is because the network of beliefs to which they belong is bigger and stronger than the reflective self that owns the feeling of "choice" regarding what to believe or not. (The status of this feeling of choice being another big topic unto itself, which I've discussed before, e.g. in a chapter of the Cosmist Manifesto.).... This is the fundamental "human nature" with which Hume "solved" the problem of induction, way back when....

Now, what happens to these basic assumptions when we, say, upload our mind-patterns into robot bodies ... or replace our body parts incrementally with engineered alternatives ... so that (e.g.) ingesting is no longer believing? What happens is that our fundamental reality-systems will change. (Will a digital software mind feel like "self-reprogramming is believing"??1) Singularity-enabling technologies are going to dramatically change realities as we know them.

And so it goes…

My Goal as an AGI Researcher

2011-12-17T01:31:00.004-05:00

In a recent thread on the AGI email list, Matt Mahoney pressed me regarding my high-level goals as an AGI researcher, and a leader of the OpenCog project. This blog post repeats my answer, as I posted it on that email list. This is familiar material to those who have followed my work and thinking, but maybe I've expressed things here slightly differently than in the past....

My goal as an AGI researcher is not precisely and rigorously defined. I'm OK with this. Building AGI is a human pursuit, and human pursuits aren't always precisely and rigorously defined. Nor are scientific pursuits. Often the precise, rigorous definitions come only after a lot of the research is done.

I'm not trying to emulate human beings or human minds in detail. But nor am I trying to make a grab-bag of narrow agents, without the capability to generalize automatically to new problems radically different from the ones for which they were originally designed. I am after a system that -- in the context of the scope of contemporary human activities -- possesses humanlike (or greater) capability to generalize its knowledge from one domain to other qualitatively different domains, and to learn new things in domains different than the ones its programmers had explicitly in mind. I'm OK if this system possesses many capabilities that a human doesn't.

There are probably many ways of achieving software with this kind of general intelligence. The way I think I understand (and am trying to realize with OpenCog), is to roughly emulate the process of human child development -- where I say roughly because I'm fine with the system having some capabilities beyond those of any human. Even if it does have some specialized superhuman capabilities from the start, I think this system will develop the ability to generalize its knowledge to qualitatively different domains in the rough manner and order that a human child does.

What will I do once I have a system that has a humanlike capability of cross-domain generalization (in the scope of contemporary human activities)? Firstly I will study it, and try to create a genuine theory of general intelligence. Second I will apply it to solve various practical problems, from service robotics to research in longevity and brain-computer interfacing etc. etc. There are many, many application areas where the ability to broadly generalize is of great value, alongside specialized intelligent capabilities.

At some point, I think this is very likely to lead to an AGI system with recursive self-improving capability (noting that this capability will be exercised in close coordination with the environment, including humans and the physical world, not in an isolation chamber). Before that point, I hope that we will have developed a science of general intelligence that lets us understand issues of AGI ethics and goal system stability much better than we do now.

Why Time Appears To Move Forwards

2011-11-13T22:53:00.005-05:00

On a long drive to my mom's house earlier this weekend, my son Zar and I got into a long conversation about the nature of causality, which got me thinking about the old puzzle of where the feeling of the directionality of time comes from...

Where does the feeling that "time moves forward" come from?

It's interesting to look at this view from two sides -- from the reductionist approach, in terms of the grounding of minds in physical systems; and also the phenomenological approach, in which one takes subjective experience as primary.

Putting together these two perspectives, one arrives at the conclusion that the directionality of time, as perceived by a mind, has to do with: entropy increase in the mind's environment, and entropy decrease in the mind's "theater of decisive consciousness."

A Reductionist View of the Origin of the Directionality of Time

Microphysics, as we currently understand it, doesn't seem to have this. In both classical and quantum physics, there is no special difference between the forward and backward direction in time.

Julian Barbour, in his excellent book The End of Time, argues that the directionality of time is an artifact of psychology -- something added by the experiencing mind.

It's commonly observed that thermodynamics adds an arrow of time to physics. The increase of entropy described by the Second Law of Thermodynamics implies a directionality to time. And the Second Law has an intriguing observer-dependence to it. If one assumes a conservative dynamical system evolving according to classical mechanics, there is no entropy increase -- until one assumes a coarse-graining of the system's state space, in which case the underlying complex dynamics of the system will cause an information loss relative to that coarse-graining. The coarse-graining is a simple sort of "observer-dependence." For a detailed but nontechnical exposition of this view of entropy, see Michel Baranger's essay "Chaos, Complexity and Entropy."

In this view, an argument for the origin of the directionality of time is as follows: The mind divides the world into categories -- i.e. "coarse-graining" the set of possible states of the world -- and then, with respect to these categories, there emerges an information loss corresponding to one temporal direction, but not the other.

A Psychological View of the Origin of the Direction of Time

Next, what can we say about the origin of the directionality of time from the psychological, subjectivist, phenomenological perspective?

Subjectively, it seems that our perception of the directionality of time is largely rooted in our perception of causality. Confronted with a pool of semi-organized sensations, we perceive some as causally related to others, and then assign temporal precedence to the cause rather than the effect.

Now, grounding temporal direction in causation may seem to introduce more confusion than clarification, since there is no consensus understanding of causality. However, there are certain approaches to understanding causality in the philosophical literature, that happen to tie in fairly naturally with the reductionist approach to grounding the directionality of time given above, and bear particular consideration here for that reason. I'm thinking especially of the view of causality as "information transmission across mechanistic hierarchies," summarized nicely by Phyllis Illari in this paper.

If causality is viewed as the transmission of information from cause to effect via channels defined by "mechanistic hierarchies", then we may see the direction of time as having to do somehow with information flow. This is loosely similar to how Baranger sees entropy emerging from the dynamics of complex systems as perceived relative to the coarse-graining of state space. In both cases, we see the flow of time as associated with the dynamics of information. However, to see exactly what's going on here, we need to dig a bit.

(I don't necessarily buy the reduction of causality to information transmission. But I do think this captures an important, interesting, relevant aspect of causality.)

Another point made by Illari in the above-linked article is the relation between causality and production. However, I find it more compelling to link causality and action.

It seems to me that the paradigm case of causality, from a subjective, psychological point of view, is when one of our own actions results in some observable effect. Then we feel, intuitively, that our action caused the effect.

We then interpret other phenomena we observe as analogous to instances of our own enaction. So, when we see an ape push a rock off a cliff, we can imagine ourselves in the position of the ape pushing the rock, so we can feel that the ape caused the rock to fall. And the same thing when it's not an ape but, say, another rock that's rolling into the first rock and knocking it off the cliff.

In this hypothesis, then, the root of temporal directionality is cause, and the root of causation is our interpretation of our own actions -- specifically, the assumption that the relation between an action and its preconditions, is fundamentally conceptually different than the relation between an action and its results.

Another way to say this is: the carrying-out of an action is viewed as a paring-down of possibilities, via a choosing of one action among many possibilities. Thus, carrying-out of an action is viewed as a decrease of entropy.

So, psychologically: The directionality of time ensues from the decrease of entropy perceived as associated with enaction -- via means of analogical reasoning which propagates this perceived entropy decrease to various perceptions that are not direct enact ions, causing them to be labeled as causative.

Putting the (Reductionist and Subjectivist) Pieces Together

On the face of it, we seem to have a paradox here: physically, the directionality of time comes from entropy increase; but psychologically, it comes from entropy decrease.

However, there's not really any paradox at all. This is merely a relative of the observation that living systems habitually decrease their entropy, at the cost of increasing the entropy of their environments.

The directionality of time, from the perspective of a given mind, appears to ensue from a combination of

entropy decrease in the foreground (the "acutely conscious", explicitly deciding mind -- the "global workspace")
entropy increase in the background (the environment of the mind)

This is a somewhat short-term perspective: it explains the feeling of temporal directionality over a certain brief interval. But then a mind with an episodic memory and a desire for logical coherence, will naturally piece together the temporal directions of its various memories in a coherent way, forming a linear sequence of time pointing from the past up till the present. And this same mind will then naturally reason about the future by analogy to the past, thus mentally building up a subjective timeline pointing into the future. And so the subjective sense of a linear axis of time emerges -- not because it's the only way to look at the world, but because it naturally emerges from the dynamics of foreground/background information flow, together with the quest for logical coherence.

A New Approach to Computational Language Learning

2011-09-20T09:56:00.004-04:00

I've been thinking about a new approach to computational language learning for a while, and finally found time to write it down -- see the 2 page document here.

Pursued on its own, this is a "narrow AI" approach, but it's also designed to be pursued in an AGI context, and integrated into an AGI system like OpenCog.

In very broad terms, these ideas are consistent with the integrative NLP approach I described in this 2008 conference paper. But the application of evolutionary learning is a new idea, which should allow a more learning-oriented integrative approach than the conference paper alluded to.

Refining and implementing these ideas would be a lot of work, probably the equivalent of a PhD thesis for a very good student.

Those with a pure "experiential learning" bent will not like the suggested approach much, because it involves making use of existing linguistic resources alongside experiential knowledge. However, there's no doubt that existing statistical and rule-based computational linguistics have made a lot of progress, in spite of not having achieved human-level linguistic performance. I think the outlined approach would be able to leverage this progress in a way that works for AGI and integrates well with experiential learning.

I also think it would be possible for an AGI system (e.g. OpenCog, or many other approaches) to learn language purely from perceptual experience. However, the possibility of such an approach, doesn't imply its optimality in practice, given the hardware, software and knowledge resources available to us right now.

A Mind-World Correspondence Principle

2011-09-18T10:07:00.004-04:00

I had some more ideas, working toward a general theory of general intelligence, which I wrote in a paper posted online at Dynamical Psychology.

(Please note: it's fairly abstract theoretical/mathematical material, so if you're solely interested in current AGI engineering work, don't bother! The hope is that this theory will be able to help guide engineering work once it's further developed, but it's not at that stage yet. So for now my abstract mathematical AGI theory work and practical AGI engineering work are only loosely coupled.)

The crux of the paper is:

MIND-WORLD CORRESPONDENCE PRINCIPLE: For an organism with a reasonably high level of intelligence in a certain world, relative to a certain set of goals, the mind-world path transfer function is a goal-weighted approximate functor

To see what those terms mean and why it might be a useful notion, you'll have to read the paper.

A cruder expression of the same idea, with fewer special defined terms is:

MIND-WORLD CORRESPONDENCE-PRINCIPLE: For a mind to work intelligently toward certain goals in a certain world, there should be a nice mapping from goal-directed sequences of world-states into sequences of mind-states, where “nice” means that a world-state-sequence W composed of two parts W1 and W2, gets mapped into a mind-state-sequence M composed of two corresponding parts M1 and M2.

As noted toward the end of the paper, this principle gives us systematic way to approach questions like: Why do real-world minds seem to be full of hierarchical structures? The answer is probably that the real world is full of goal-relevant hierarchical structures. The Mind-World Correspondence Principle explains exactly why these hierarchical structures in the world have to be reflected by hierarchical structures in the mind of any system that's intelligent in the world.

As an aside, it also occurred to me that these ideas might give us a nice way to formalize the notion of a "good mind upload," in category-theoretic terms.

I.e., if we characterize minds via transition graphs in the way done in the paper, then we can argue that mind X is a valid upload of mind Y if there is a fairly accurate approximate functor from X's transition graph to Y's.

And, if Y is a nondestructive upload (so X still exists after the uploading), it would remain a good upload of X over time if, as X and Y both changed, there was a natural transformation governing the functors between them. Of course, your upload might not WANT to remain aligned with you in this manner, but that's a different issue...

Creating/Discovering New States of Mind

2011-09-07T13:50:00.006-04:00

Just some quasi-random musings that went through my head yesterday…

Our society puts a fair bit of energy, these days, into creating new technologies and discovering new scientific facts.

But we don’t put hardly any effort at all into creating/discovering new states of mind.

I think maybe we should – and at the end of this odd, long, rambling blog post I’m going to suggest a specific type of new mind-state that I think is well worth trying to create/discover: one synthesizing spiritual mindfulness and intense scientific creativity.

On Old and New States of Consciousness

First, bear with me while I spend a few paragraphs framing the issue…

When I read Stcherbatsky’s book Buddhist Logic years ago, I was struck by the careful analysis of 128 states of consciousness. Allan Combs’ book The Radiance of Being provides a simpler, smaller conceptual analysis of states of consciousness, with similar foundations. These and other similar endeavors are very worthy – but how can we really know that the scope of all possible varieties of human consciousness-state has been thoroughly explored?

All sorts of amazing new states of consciousness will become possible once the human brain has been enhanced with technology – brain-computer interfacing, genetic engineering, mind uploading, etc. Advanced AGI systems may enjoy states of consciousness far beyond human comprehension. However, it seems quite possible that ordinary human brains may be capable of many states of consciousness not yet explored.

The individual human mind is not all that individual – so the states of mind accessible to an individual may depend to some extent on the culture in which they exist. The catalogue of states of mind available in medieval India when Buddhist logic was invented, may include some states that are extremely hard for modern people to get into, and may omit some states of which modern people are capable.

The Perceived Conflict Between Scientific and Spiritual Mind-States

I’ve often wondered whether there’s some intrinsic conflict between the states of mind labeled “spiritual enlightenment”, and the states of mind consistent with profound scientific discovery.

Great scientific creation often seems to involve a lot of struggle and persistence – along with long stretches of beautiful “flow” experience. Great scientific work seems to involve a lot of very hard thinking and analysis, whereas enlightenment is generally described as involving “stopping all thought.”

Personally, I find it a lot easier to be mindful (in the Zen sense) while walking through the park, washing the dishes, lying in bed, or building a chair -- than while analyzing genomic data, working out the details of a new AI algorithm, writing a novel, or debugging complex software code. Subjectively, this feels to me like it’s because being mindful requires a bit of mental effort at first – to actively pay attention to what my mind and body are doing. Once the effort is done, then mindfulness can flow along effortlessly for a while. But then I may drift away from it, and that little jump of effort is needed to become mindful again. This dynamic of mindfulness drifting and returning, or almost drifting but then not actually drifting after all, seems not to function properly when I’m doing something highly cognitively intensive. When I’m doing the highly intensive thing, I get deeply “into” the process, which puts me in a wonderful flow state for a while – but then when the flow state ends, I’m not necessarily in a quasi-enlightened mindful state. I may be elated, or I may be exhausted, or I may be frustrated. I can then try to be mindful of my elation, exhaustion or frustration – but this is then a moderately substantial effort; and definitely my degree of mindfulness is lower than if I hadn’t bothered to do the cognitively intensive thing.

Now, it might just be that I’m not a particularly enlightened guy. Indeed, I have never claimed to be! I do have my moments of spiritual purity and cosmic blissful wisdom and all that -- but then I also have some pretty boring routine moments, and also moments of being totally un-mindfully overcome with various kinds of positive or negative emotion. However, observing other humans around me, I note that the same dichotomy I feel in my mind occurs in the outside world. I know some enlightened minds, and I know some productive, brilliant artists and scientists – but I don’t know anyone in the intersection. Maybe someone of this nature does exist; but if they do, they’re an awfully rare bird.

You could argue that, since being a spiritual genius is rare and being a scientific genius is rare, it’s not surprising that few people lie in the intersection! But I’m not just talking about genius. I’m talking about passion. Who has true devoted passion for spiritual enlightenment, and also true devoted passion for doing revolutionary science? Most people I know, if they like either, pursue one as a central goal and the other as a sort of sideline.

I don’t particularly want to be this way myself – I’d like to pursue both simultaneously, without feeling any conflict between the two. But in practical life I do feel a conflict, and I tend to choose science and art most of the time. Yes, from the enlightened view, the dichotomy and the conflict are just constructs of my mind. And when I’m in certain states of mind, I feel that way – that dichotomy and all the rest feel bogus and mildly amusing. But when I’m in those states of mind, I’m not doing my best art or science! Similarly, thinking about playing the piano, it clear that my best music has been played in states of heightened emotion – not states of enlightened emptiness.

I think the difficulty of maintaining a mindful mind-state and scientifically intensely creative mind-state, is deeply tied with the conflict between modern scientific culture and some older cultures like those of ancient India or China, that were more spiritually focused. The enlightened master was one of the ideals of India and China; and the great scientist or artist is one of the ideals of the modern world. The differences in ideals reflect more thoroughgoing cultural differences.

You could say that both the great scientist and the enlightened master are exaggerations, and the right thing is to be more balanced – a little bit scientific, a little bit spiritual. Maybe, as someone said to me recently, an enlightened master is like an Arnold Schwarzenegger of the spirit – hyper-developed beyond what is natural or useful (except in contexts like the Mr. Universe contest where being at the extreme is useful in itself!). And maybe great super-scientists are unnecessarily and unhealthily obsessive, and science would progress OK without them, albeit a little more slowly. But something in me rebels against this kind of conclusion. Maybe it’s just that I’m an unbalanced individual – reeling back and forth endlessly between being excessively scientific and excessively spiritual, instead of remaining calmly in the middle where I belong -- but maybe there’s more to it than that.

A New Scientific/Spiritual Mind-State?

What if, instead of being frustrated at the apparent contradiction between the mind-states of spiritual enlightenment /mindfulness and intense scientific creativity, we took it as a multidimensional challenge: to create a new state of mind, synergizing both of these aspects?

The ancient Indians and Chinese didn’t include this sort of mind-state in their catalogue, but they didn’t have science or modern art … they had a very different culture.

Can we discover a new, intrinsically mindful way of doing science and art? Without sacrificing the intensity or the creativity?

What if we pursued the discovery/creation of new states of mind as avidly as we pursue the creation of new machines or chemical compounds? What if there were huge multinational organizations devoted to mind-state discovery, alongside our chemical and pharmaceutical and computer engineering firms?

Zum: A Thought-Experiment

To make the above idea a little more concrete, let’s imagine a specific social structure designed to produce a synergetically scientific-spiritual state of mind. Imagine an agile software development team – a group of software developers working closely together on a project – that was also, simultaneously, a “zendo” or “dojo” or whatever you want to call it … a group of people gathered together in the interest of their own enlightenment. That is, they were simultaneously trying to get stuff done together, and to help each other maintain a state of mindfulness and individual & collective spiritual awareness.

I can’t think of a good name for this kind of combination, so I’m going to call it a “Zum”, because that word currently has no English meaning, and it reminds me of Zen and scrum (the latter a term from agile software development), and I like the letter “Z.”

I have heard of a new type of Vipassana meditation, in which a group of people sit together and while they meditate, verbalize their feelings as they pass through – “cold”, “breathing”, “warm”, “stomach”, etc. One can imagine a Zum engaging in this kind of discussion at appropriate moments, in the midst of technical discussions or collaborative work. Would hearing others describe their state like this interrupt thought in an unacceptable way? Possibly. Or would people learn to flow with it, as I flow with the music I listen to as I work?

What would a Zum be like? Would it help to have a couple enlightened masters hanging around? – maybe sitting there and meditating, or playing ping pong? That would produce a rather different vibe than a usual software development lab!

The key ingredient of the Zum is the attitude and motivation of the individuals involved. They would need to be dedicated both to producing great software together, and to helping each other remain mindful and joyful as much as possible.

One thing that might come out of this is, simply, a kind of balance, where the team does reasonably good work and is also rather happy. This certainly wouldn’t be a disaster. Maybe they’d even be a bit more effective than an average team due to a diminished incidence of personality conflicts and fewer stress-induced errors.

Another possibility is that, if this sort of experiment were tried in a variety of different styles and places, eventually a new state of mind would evolve – one bypassing the dichotomy of spiritual mindfulness versus intensely creative science or art production.

Solo Zum?

But do we really need a Zum? Organizing groups of people in novel configurations involves considerable practical difficulty. Why not become a one-person Zum? Experiment with different ways of practicing intense scientific creation and mindfulness at the same time – maybe you’ll come up with something new. Try to describe your internal methodology so others can follow in your footsteps. This sort of experimentation is every bit as valid and important as scientific experimentation, or personal experimentation with smart drugs. The human brain is far more flexible than we normally realize, it’s hard to say what may be possible even without technological brain modification.

Heh... well I'm really not sure how much any of that means, but it was an amusing train of thought! Now, it's time to pick up my daughter from school, and then get back to work.... I will be trying to be as cosmically aware as possible while my work proceeds ;O ;-) ... and probably not succeeding all that well !! So it goes... bring on the brain chips please...

This blog post was written while repetitively listening to various versions of A Tear for Eddie by Ween. This one is perhaps my favorite, though the studio version is great too.

Musings on future technologies for cognitive enhancement

2011-08-09T13:41:00.002-04:00

A former college classmate of my son's, researching a magazine article on cognitive enhancement, just emailed me asking my opinion on future technologies for cognitive enhancement.... Here's the reply I gave him -- not much new information for the well-educated transhumanist reader, but I figured I'd paste it here anyways...

Regarding technologies for cognitive enhancement, present and future..

Firstly, I am not an expert on nootropics, but I can remember seeing various studies indicating potential positive benefits for cognitive aging. The racetams and modafinil come to mind, among many others. Anecdotally I am aware of plenty of folks who say these improve cognitive function, including older folks, but I'm not up on the literature.

I also see a huge future for neural stem cell therapy, and you can find a substantial literature on that online, though I'm not an expert on it. The regulatory issues here become interesting -- I know a number of individuals operating stem cell therapy clinics in Asia and Latin America, that cater substantially to US clients. So far these aren't focusing on neural stem cell therapy but I think that's not far off. The US regulatory environment has become archaic and highly problematic. One can envision a future in which Americans routinely fly to foreign countries for neural stem cell therapy and other medical interventions aimed at maintaining or increasing their intelligence. And the ones who stay home won't be as smart. One hopes that as these technologies mature, the American regulatory infrastructure will eventually mature as well.

I have also heard rumor (from reliable sources) of a device under development by the Chinese government in Beijing, in collaboration with some Western scientists, going by the name of the "head brain instrument" (three Chinese characters). This device uses transcranial magnetic stimulation, and has the dual effects of increasing learning rate, and also increasing susceptibility to suggestion. Interesting. I read an article a few months ago about a different but related device being tested in Australia, using transcranial stimulation to increase creativity. This sort of research seems fascinating and promising. No doubt one could advance even faster and further in this direction using direct brain-computer interfacing, but no one has yet developed an inexpensive and safe method of installing a Neuromancer-style "cranial jack" in the brain, alas. I'm sure the cranial jack is coming, but it's hard to estimate exactly when.

In terms of ongoing and future research, I think that a combination of genomics, experimental evolution and artificial intelligence is fairly shortly going to lead us to a variety of therapies to improve cognitive performance throughout the human lifespan, as well as to extend the healthy human lifespan overall. I'm seeing this now in the work my bioinformatics firm Biomind is doing in collaboration with the biopharma firm Genescient Corp. Genescient has created a set of populations of long-lived fruit flies, which live over 4x as long as control flies, and also display enhanced cognitive capability throughout their lives, including late life. We've gathered gene expression and SNP data from these "superflies" and are using AI technology to analyze the data -- and the results are pretty exciting so far! We've discovered a large number of gene-combinations that are extremely strongly associated with both longevity and neural function, and many of these correspond to likely-looking life-extension and cognitive-enhancement pathways in the human organism. The supplement Stem Cell 100, now on the market, was inspired by this research; but that's just the start ... I think we're going to see a lot of new therapies emerge from this sort of research, including nutraceuticals, pharmaceuticals, gene therapy, and others.

I'm currently in San Francisco, where I just got finished with 4 days of the Artificial General Intelligence 2011 conference, which was held on Google's campus in Mountain View. Now I'm at the larger AAAI (Association for the Advancement of AI) conference in San Francisco. I think that AI research, as it matures, is going to have a huge effect on cognitive enhancement research among many other areas. Right now my own Biomind team and others are using AI to good effect in bioinformatics -- but the AI tools currently at our disposal are fairly narrow and specialized, albeit with the capability to see pattens that are inaccessible to either unassisted humans or traditional statistical algorithms. As AI gradually moves toward human-level artificial general intelligence, we're going to see a revolutionary impact upon all aspects of biomedical science. Already there's far more biomedical data online than any human mind can ingest or comprehend -- an appropriately constructed and instructed AGI system could make radical advances in cognitive enhancement, life extension and other areas of biomedicine, just based on the data already collected ... in addition to designing new experiments of its own.

Down the road a bit, there's the potential for interesting feedback effects to emerge regarding cognitive enhancement, conceivably resulting in rapid exponential growth. The better science and technology we have, the better cognitive enhancers we can create, and the smarter we get. But the smarter we get, the better the science and technology we can develop. Et cetera, and who knows where (or if) the cycle ends! We live in interesting times, and I suspect in the next few decades they will become dramatically *more* interesting....

Unraveling Modha & Singh's Map of the Macaque Monkey Brain

2011-06-24T16:40:00.006-04:00

(... plus some semi-related AGI musings at the end!)

On July 27 2010, PNAS published a paper entitled "Network architecture of the long-distance pathways in the macaque brain" by Dharmendra Modha and Raghavendra Singh from IBM, which is briefly described here and available in full here. The highlight of the paper is a connectivity diagram of all the regions of the macaque (monkey) brain, reproduced in low res right here:

See here for a hi-res version.

The diagram portrays "a unique network incorporating 410 anatomical tracing studies of the macaque brain from the Collation of Connectivity data on the Macaque brain (CoCoMac) neuroinformatic database. Our network consists of 383 hierarchically organized regions spanning cortex, thalamus, and basal ganglia; models the presence of 6,602 directed long-distance connections; is three times larger than any previously derived brain network; and contains subnetworks corresponding to classic corticocortical, corticosubcortical, and subcortico-subcortical fiber systems."

However, I found that the diagram can be somewhat confusing to browse, if one wants to look at specific brain regions and what they connect to. So my Novamente LLC co-conspirator Eddie Monroe and I went back to the original data files, given in the online supplementary information for the paper, and used this to make a textual version of the information in the diagram, which you can find here.

Our goal in looking at this wiring diagram is as a guide to understanding the interactions between certain human brain regions we're studying (human and monkey brains being rather similar in many respects). But I think it's worth carefully perusing for anyone who's thinking about neuroscience from any aspect, and for anyone who's thinking about AGI from a brain-simulation perspective.

Semi-Related AGI Musings

Complexity such as that revealed in Modha and Singh's diagrams always comes to my mind when I read about someone's "brain inspired" AGI architecture -- say, Hierarchical Temporal Memory architectures (like Numenta or DeSTIN, etc.) that consist of a hierarchy of layers of nodes, passing information up and down in a manner vaguely reminiscent of visual or auditory cortex. Such architectures may be quite valuable and interesting, but each of them captures a teensy weensy fraction of the architectural and dynamical complexity in the brain. Each of the brain regions in Modha and Singh's diagram is its own separate story, with its own separate and important functions and structures and complex dynamics; and each one interacts with a host of others in specially configured ways, to achieve emergent intelligence. In my view, if one wants to make a brain-like AGI, one's going to need to emulate the sort of complexity that the actual brain has -- not just take some brain components (e.g. neurons) and roughly simulate them and wire the simulations together in some clever way; and not just emulate the architecture and dynamics of one little region of the brain and proclaim it to embody the universal principles of brain function.

And of course this is the reason I'm not pursuing brain-like AGI at the moment. If you pick 100 random links from Modha and Singh's diagram, and then search the neuroscience literature for information about the dynamical and informational interactions ensuing from that link, you'll find that in the majority of cases the extant knowledge is mighty sketchy. This is an indicator of how little we still know about the brain.

But can we still learn something from the brain, toward the goal of making loosely brain-inspired but non-brain-like AGI systems? Absolutely. I'm currently interested in understanding how the brain interfaces perceptual and conceptual knowledge -- but not with a goal of emulating how the brain works in any detailed sense (e.g. my AGI approach involves no formal neurons or other elementary brainlike components, and no modules similar in function to specific brain regions), rather just with a goal of seeing what interesting principles can be abstracted therefrom, that may be helpful in designing the interface between OpenCog and DeSTIN (a hierarchical temporal memory designed by Itamar Arel, that we're intending to use for OpenCog's sensorimotor processing).

And so it goes... ;-)

Why is evaluating partial progress toward human-level AGI so hard?

2011-06-15T20:43:00.002-04:00

This post co-authored by Ben Goertzel and Jared Wigmore

Here we sketch a possible explanation for the well-known difficulty of measuring intermediate progress toward human-level AGI is provided, via extending the notion of cognitive synergy to a more refined notion of ”tricky cognitive synergy.”

The Puzzle: Why Is It So Hard to Measure Partial Progress Toward Human-Level AGI?

A recurrent difficulty in the AGI field is the difficulty of creating a good test for intermediate progress toward the goal of human-level AGI.

It’s not entirely straightforward to create tests to measure the final achievement of human-level AGI, but there are some fairly obvious candidates here. There’s the Turing Test (fooling judges into believing you’re human, in a text chat) the video Turing Test, the Robot College Student test (passing university, via being judged exactly the same way a human student would), etc. There’s certainly no agreement on which is the most meaningful such goal to strive for, but there’s broad agreement that a number of goals of this nature basically make sense.

On the other hand, how does one measure whether one is, say, 50 percent of the way to human-level AGI? Or, say, 75 or 25 percent?

It’s possible to pose many ”practical tests” of incremental progress toward human-level AGI, with the property that IF a proto-AGI system passes the test using a certain sort of architecture and/or dynamics, then this implies a certain amount of progress toward human-level AGI based on particular theoretical assumptions about AGI. However, in each case of such a practical test, it seems intuitively likely to a significant percentage of AGI researcher that there is some way to ”game” the test via designing a system specifically oriented toward passing that test, and which doesn’t constitute dramatic progress toward AGI.

Some examples of practical tests of this nature would be

The Wozniak ”coffee test”: go into an average American house and figure out how to make coffee, including identifying the coffee machine, figuring out what the buttons do, finding the coffee in the cabinet, etc.
Story understanding – reading a story, or watching it on video, and then answering questions about what happened (including questions at various levels of abstraction)
Passing the elementary school reading curriculum (which involves reading and answering questions about some picture books as well as purely textual ones)
Learning to play an arbitrary video game based on experience only, or based on experience plus reading instructions

One interesting point about tests like this is that each of them seems to some AGI researchers to encapsulate the crux of the AGI problem, and be unsolvable by any system not far along the path to human-level AGI – yet seems to other AGI researchers, with different conceptual perspectives, to be something probably game-able by narrow-AI methods. And of course, given the current state of science, there’s no way to tell which of these practical tests really can be solved via a narrow-AI approach, except by having a lot of people try really hard over a long period of time.

A question raised by these observations is whether there is some fundamental reason why it’s hard to make an objective, theory-independent measure of intermediate progress toward advanced AGI. Is it just that we haven’t been smart enough to figure out the right test – or is there some conceptual reason why the very notion of such a test is problematic?

We don’t claim to know for sure – but in this brief note we’ll outline one possible reason why the latter might be the case.

Is General Intelligence Tricky?

The crux of our proposed explanation has to do with the sensitive dependence of the behavior of many complex systems on the particulars of their construction. Often-times, changing a seemingly small aspect of a system’s underlying structures or dynamics can dramatically affect the resulting high-level behaviors. Lacking a recognized technical term to use here, we will refer to any high-level emergent system property whose existence depends sensitively on the particulars of the underlying system as tricky. Formulating the notion of trickiness in a mathematically precise way is a worthwhile pursuit, but this is a qualitative essay so we won’t go that direction here.

Thus, the crux of our explanation of the difficulty of creating good tests for incremental progress toward AGI is the hypothesis that general intelligence, under limited computational resources, is tricky.

Now, there are many reasons that general intelligence might be tricky in the sense we’ve defined here, and we won’t try to cover all of them here. Rather, we’ll focus on one particular phenomenon that we feel contributes a significant degree of trickiness to general intelligence.

Is Cognitive Synergy Tricky?

One of the trickier aspects of general intelligence under limited resources, we suggest, is the phenomenon of cognitive synergy.

The cognitive synergy hypothesis, in its simplest form, states that human-level AGI intrinsically depends on the synergetic interaction of multiple components (for instance, as in the OpenCog design, multiple memory systems each supplied with its own learning process). In this hypothesis, for instance, it might be that there are 10 critical components required for a human-level AGI system. Having all 10 of them in place results in human-level AGI, but having only 8 of them in place results in having a dramatically impaired system – and maybe having only 6 or 7 of them in place results in a system that can hardly do anything at all.

Of course, the reality is almost surely not as strict as the simplified example in the above paragraph suggests. No AGI theorist has really posited a list of 10 crisply-defined subsystems and claimed them necessary and sufficient for AGI. We suspect there are many different routes to AGI, involving integration of different sorts of subsystems. However, if the cognitive synergy hypothesis is correct, then human-level AGI behaves roughly like the simplistic example in the prior paragraph suggests. Perhaps instead of using the 10 components, you could achieve human-level AGI with 7 components, but having only 5 of these 7 would yield drastically impaired functionality – etc. Or the same phenomenon could be articulated in the context of systems without any distinguishable component parts, but only continuously varying underlying quantities. To mathematically formalize the cognitive synergy hypothesis in a general way becomes complex, but here we’re only aiming for a qualitative argument. So for illustrative purposes, we’ll stick with the ”10 components” example, just for communicative simplicity.

Next, let’s suppose that for any given task, there are ways to achieve this task using a system that is much simpler than any subset of size 6 drawn from the set of 10 components needed for human-level AGI, but works much better for the task than this subset of 6 components(assuming the latter are used as a set of only 6 components, without the other 4 components).

Note that this supposition is a good bit stronger than mere cognitive synergy. For lack of a better name, we’ll call it tricky cognitive synergy. The tricky cognitive synergy hypothesis would be true if, for example, the following possibilities were true:

creating components to serve as parts of a synergetic AGI is harder than creating components intended to serve as parts of simpler AI systems without synergetic dynamics
components capable of serving as parts of a synergetic AGI are necessarily more complicated than components intended to serve as parts of simpler AGI systems.

These certainly seem reasonable possibilities, since to serve as a component of a synergetic AGI system, a component must have the internal flexibility to usefully handle interactions with a lot of other components as well as to solve the problems that come its way. In terms of our concrete work on the OpenCog integrative proto-AGI system, these possibilities ring true, in the sense that tailoring an AI process for tight integration with other AI processes within OpenCog, tends to require more work than preparing a conceptually similar AI process for use on its own or in a more task-specific narrow AI system.

It seems fairly obvious that, if tricky cognitive synergy really holds up as a property of human-level general intelligence, the difficulty of formulating tests for intermediate progress toward human-level AGI follows as a consequence. Because, according to the tricky cognitive synergy hypothesis, any test is going to be more easily solved by some simpler narrow AI process than by a partially complete human-level AGI system.

Conclusion

We haven’t proved anything here, only made some qualitative arguments. However, these arguments do seem to give a plausible explanation for the empirical observation that positing tests for intermediate progress toward human-level AGI is a very difficult prospect. If the theoretical notions sketched here are correct, then this difficulty is not due to incompetence or lack of imagination on the part of the AGI community, nor due to the primitive state of the AGI field, but is rather intrinsic to the subject matter. And if these notions are correct, then quite likely the future rigorous science of AGI will contain formal theorems echoing and improving the qualitative observations and conjectures we’ve made here.

If the ideas sketched here are true, then the practical consequence for AGI development is, very simply, that one shouldn’t worry all that much about producing compelling intermediary results. Just as 2/3 of a human brain may not be much use, similarly, 2/3 of an AGI system may not be much use. Lack of impressive intermediary results may not imply one is on a wrong development path; and comparison with narrow AI systems on specific tasks may be badly misleading as a gauge of incremental progress toward human-level AGI.

Hopefully it’s clear that the motivation behind the line of thinking presented here is a desire to understand the nature of general intelligence and its pursuit – not a desire to avoid testing our AGI software! Truly, as AGI engineers, we would love to have a sensible rigorous way to test our intermediary progress toward AGI, so as to be able to pose convincing arguments to skeptics, funding sources, potential collaborators and so forth -- as well as just for our own edification. We really, really like producing exciting intermediary results, on projects where that makes sense. Such results, when they come, are extremely informative and inspiring to the researchers as well as the rest of the world! Our motivation here is not a desire to avoid having the intermediate progress of our efforts measured, but rather a desire to explain the frustrating (but by now rather well-established) difficulty of creating such intermediate goals for human-level AGI in a meaningful way.

If we or someone else figures out a compelling way to measure partial progress toward AGI, we will celebrate the occasion. But it seems worth seriously considering the possibility that the difficulty in finding such a measure reflects fundamental properties of the subject matter – such as the trickiness of cognitive synergy and other aspects of general intelligence.

Is Software Improving Exponentially?

2011-06-15T11:29:00.005-04:00

In a discussion on the AGI email discussion list recently, some folks were arguing that Moore's Law and associated exponential accelerations may be of limited value in pushing the world toward Singularity, because software is not advancing exponentially.

For instance Matt Mahoney pointed out "the roughly linear rate of progress in data compression as measured over the last 14 years on the Calgary corpus, http://www.mailcom.com/challenge/ "

Ray Kurzweil's qualitative argument in favor of the dramatic acceleration of software progress in recent decades is given in slides 104-111 of his presentation here.

I think software progress is harder to quantify than hardware progress, thus less often pointed to in arguments regarding technology acceleration.

However, qualitatively, there seems little doubt that the software tools available to the programmer have been improving damn dramatically....

Sheesh, compare game programming as I did it on the Atari 400 or Commodore 64 back in the 80s ... versus how it's done now, with so many amazing rendering libraries, 3D modeling engines, etc. etc. With the same amount of effort, today one can make incredibly more complex and advanced games.

Back then we had to code our own algorithms and data structures, now we have libraries like STL, so novice programmers can use advanced structures and algorithms without understanding them.

In general, the capability of programmers without deep technical knowledge or ability to create useful working code has increased *incredibly* in the last couple decades…. Programming used to be only for really hard-core math and science geeks, now it's a practical career possibility for a fairly large percentage of the population.

When I started using Haskell in the mid-90s it was a fun, wonderfully elegant toy language but not practical for real projects. Now its clever handling of concurrency makes it viable for large-scale projects... and I'm hoping in the next couple years it will become possible to use Haskell within OpenCog (Joel Pitt just made the modifications needed to enable OpenCog AI processes to be coded in Python as well as the usual C++).

I could go on a long time with similar examples, but the point should be clear. Software tools have improved dramatically in functionality and usability. The difficulty of quantifying this progress in a clean way doesn't mean it isn't there...

Another relevant point is that, due to the particular nature of software development, software productivity generally decreases for large teams. (This is why I wouldn't want an AGI team with more than, say, 20 people on it. 10-15 may be the optimal size for the core team of an AGI software project, with additional people for things like robotics hardware, simulation world engineering, software testing, etc.) However, the size of projects achievable by small teams has dramatically increased over time, due to the availability of powerful software libraries.

Thus, in the case of software (as in so many other cases), the gradual improvement of technology has led to qualitative increases in what is pragmatically possible (i.e. what is achievable via small teams), not just quantitative betterment of software that previously existed.

It's true that word processors and spreadsheets have not advanced exponentially (at least not with any dramatically interesting exponent), just as forks and chairs and automobiles have not. However, other varieties of software clearly have done so, for instance video gaming and scientific computation.

Regarding the latter two domains, just look at what one can do with Nvidia GPU hardware on a laptop now, compared to what was possible for similar cost just a decade ago! Right now, my colleague Michel Drenthe in Xiamen is doing CUDA-based vision processing on the Nvidia GPU in his laptop, using Itamar Arel's DeSTIN algorithm, with a goal toward providing OpenCog with intelligent visual perception -- this is directly relevant to AGI, and it's leveraging recent hardware advances coupled with recent software advances (CUDA and its nice libraries, which make SIMD parallel scientific computing reasonably tractable, within the grasp of a smart undergrad like Michel doing a 6 month internship). Coupled acceleration in hardware and software for parallel scientific computing is moving along, and this is quite relevant to AGI, whereas the relative stagnation in word processors and forks really doesn't matter.

Let us not forget that the exponential acceleration of various quantitative metrics (like Moore's Law) is not really the key point regarding Singularity, it's just an indicator of the underlying progress that is the key point.... While it's nice that progress in some areas is cleanly quantifiable, that doesn't necessarily mean these are the most important areas....

To really understand progress toward Singularity, one has to look at the specific technologies that most likely need to improve a lot to enable the Singularity. Word processing, not. Text compression, not really. Video games, no. Scientific computing, yes. Efficient, easily usable libraries containing complex algorithms and data structures, yes. Scalable functional programming, maybe. It seems to me that by and large the aspects of software whose accelerating progress would be really, really helpful to achieving AGI, are in fact accelerating dramatically.

In fact, I believe we could have a Singularity with no further hardware improvements, just via software improvements. This might dramatically increase the financial cost of the first AGIs, due to making them necessitate huge server farms ... which would impact the route to and the nature of the Singularity, but not prevent it.

The Serf versus the Entrepreneur?

2011-05-18T13:29:00.006-04:00

This is a bit of a deviation from my usual topics, but I've been thinking a bit about economic development in various countries around the world (sort of a natural topic for me in that I travel a lot, have lived in several countries, and have done business and work in a lot of different countries including the US, Europe, Brazil, Hong Kong, Japan and China and Korea, Australia and NZ, etc.)

The hypothesis I'm going to put forth here is that the difference between development-prone and development-resistant countries, is related to whether the corresponding cultures tend to metaphorically view the individual as a serf or as an entrepreneur.

Of course, this is a very rough and high-level approximative perspective, but it seems to me to have some conceptual explanatory power.

Development-Prone versus Development-Resistant Cultures

The book "Culture Matters", which I borrowed from my dad (a sociologist) recently, contains a chapter by Mariano Grondona called "A Cultural Typology of Economic Development", which proposes a list of properties distinguishing development-prone cultures from development-resistant cultures. Put very crudely, the list goes something like this

Development-resistant vs. development-prone
Justice: present-focused vs future-focused
Work: not respected vs. respected
Heresy: reviled vs. tolerated
Education: brainwashing vs. more autonomy focused
Utilitarianism: no vs. yes
Lesser virtues (valuing a job well done, tidiness, punctuality, courtesy): no vs. yes
time focus: past/ spiritual far-future vs. practical moderately near future
rationality: not a focus vs. strongly valued
rule of man vs. rule of law
large group vs. individual as nexus of action
determinism vs. free will ism
salvation in the world (immanence) vs. salvation from the world (transcendence)
focus on utopian visions not rationally achievable vs. focus on distant utopias that are more likely rationally progressively achievable
optimism about action of "powers that be" vs. optimism about personal action
thoughts about political structure: absolutism vs compromise

A more thorough version of the list is given in this file "Typology of Progress-Prone and Progress-Resistant Cultures", which is Chapter 2 of book "The Central Liberal Truth: How Politics Can Change a Culture and Save it From Itself" by Lawrence Harrison. The title of Harrison's book (which I didn't read, I just read that chapter) presumably refers to the famous quote from Daniel Patrick Moynihan that

"The central conservative truth is that it is culture, not politics, that determines the success of a society. The central liberal truth is that politics can change a culture and save it from itself."

Harrison adds some other points to Grondona's list, such as

wealth: zero-sum vs. positive-sum
knowledge: theory vs. empirics
low risk tolerance (w/ occasional adventures) vs. moderate risk tolerance
advancement: social connections based vs. merit based
radius of trust: narrow vs. wide
entrepreneurship: rent-seeking vs. innovation

and presents it in a more nicely formatted and well-explained way than this blog post! I encourage you to click the above link and read the chapter for yourself.

Now, I find all this pretty interesting, but also in a way unsatisfying. A theory that centrally consists of a long list of bullet points always gives me the feeling of not getting to the essence of things.

Harrison attempts to sum up the core ideas of the typology as follows:

"
At the heart of the typology are two fundamental questions: (1) does the culture encourage the belief that people can influence their destinies? And (2) does the culture promote the Golden Rule. If people believe that they can influence their destinies, they are likely to focus on the future; see the world in positive-sum terms; attach a high priority to education; believe in the work ethic; save; become entrepreneurial; and so forth. If the Golden Rule has real meaning for them, they are likely to live by a reasonably rigorous ethical code; honor the lesser virtues; abide by the laws; identify with the broader society; form social capital; and so forth.
"

But this abstraction doesn't seem to me to sum up the essence of the typology all that well.

Lakoff's Analysis of the Metaphors Underlying Politics

When reading the above material, I was reminded of cognitive scientist George Lakoff's book "Moral Politics" whose core argument is summarized here.

Lakoff argues that much of liberal vs. conservative politics is based on the metaphor of the nation as a family, and that liberal politics tends to metaphorically view the government as a nurturing mother, whereas conservative politics tends to metaphorically view the government as a strict father.

While I don't agree with all Lakoff's views by any means (and I found his later cognitive/political writings generally less compelling than Moral Politics), I think his basic insight in that book is fairly interesting and significant. It seems to unify what otherwise appears a grab-bag of political beliefs.

For instance, the US Republican party is, at first sight, an odd combination of big-business advocacy with Christian moral strictness. To an extent this represents an opportunistic alliance between two interest groups that otherwise would be too small to gain power .. but Lakoff's analysis suggests it's more than this. As he points out, the "strict father" archetype binds together both moral strictness and the free-for-all, rough-and-tumble competitiveness advocated by the pro-big-business sector. And the "nurturant mother" archetype binds together the inclusiveness aspect of the US Democratic party, with the latter's focus on social programs to help the disadvantaged. Of course these archetypes don't have universal explanatory power, but they do seem to me to capture some of the unconscious patterns underlying contemporary politics.

So I started wondering whether there's some similar, significantly (though of course not completely) explanatory metaphorical/archetypal story one could use to explain comparative economic development. Such a story would then provide an explanation underlying the "laundry list" of cultural differences described above.

The Serf versus the Entrepreneur?

Getting to the point finally … it seems to me that the culture of development-resistant countries, as described above, is rather well aligned with the metaphor of the "serf and lord". If the individual views himself as the serf, and the state and government as the lord, then they will arrive at a fair approximation of the progress-resistant world-view as described in the above lists. So maybe we can say that progress-resistant nations tend to have a view of the individual/state relationship that is based on a "feudal" metaphor in some sense.

On the other hand, what is the metaphor corresponding to progress-friendly countries? One thing I see is a fairly close alignment with an entrepreneurial metaphor. Viewing the individual as an entrepreneur -- and the state as a sort of "social contract" between interacting, coopeting entrepreneurs -- seems to neatly wrap up a considerable majority of the bullet points associated with the progress-friendly countries, on the above list.

Note that this hypothetical analysis in terms of metaphors is not intended as a replacement for Lakoff's -- rather, it's intended as complementary. We understand the things in our world using a variety of different metaphors (as well as other means besides metaphor, a point Lakoff sometimes seems not to concede), and may match a single entity like a government to multiple metaphorical frames.

Finally... what value is this kind of analysis? Obviously, if we know the metaphorical frames underlying peoples' thinking, this may help us to better work with them, to encourage them to achieve their goals and fulfill themselves more thoroughly. If you know the metaphors underlying your OWN unconscious thinking, this can help you avoid being excessively controlled by these metaphors, taking more of your thinking and attitude under conscious control….

One way to empirically explore this sort of hypothesis would be to statistically study the language used in various cultures to describe the individual and the state and their relationship. However, this would require a lot of care due to the multiple languages involved, and certainly would be a large project, which I have no intention to personally pursue!

But nevertheless, in spite of the slipperiness and difficulty of validation of this sort of thinking, I find it interesting personally, as part of my quest to better understand the various cultures I come into contact with as I go about my various trans-continental doings....

The Physics of Immortality

2011-04-05T23:25:00.004-04:00

Someone asked me recently about Frank Tipler's book The Physics of Immortality. This was my reply:

Yeah, I read that book many years ago. He has some interesting and original points, such as

if a Big Crunch occurs in the right way, then if physics as we know is holds up, this may lead the algorithmic information of the universe to approach infinity, which would give the potential for a lot of interesting things
potentially we could cause a Big Crunch to occur in the right way, via moving stars around with spaceships

Those points of his seemed solid to me as extrapolations of currently accepted physics theory -- I didn't check all the math in detail but I believe others have done so.

That stuff is very cool to think about, though I'm not as confident as Tipler that our current physics theories are adequate to describe Big Crunches and so forth. Historically physics has changed its fundamental theories every century or so for a while...

Then Tipler couples those interesting observations, and some other ones, with a bunch of discussion about religious views of immortality and so on, that I remember only dimly by this point, except that they went on a long time, contained many interesting observations, and seemed only loosely connected to the physics material....

Even if he's right about immortality and the Big Crunch, I don't quite see how this connects to his discussion of religious views on immortality. Perhaps you could see all these different things as manifestations of some "immortality archetype" existing more deeply than physics or human culture (that's what Jung would have said) but he doesn't really go there either...

The Big Crunch is one kind of Singularity but I've thought more about the nearer-term kind foreseen by Ray Kurzweil and Vernor Vinge and so forth --- i.e. what happens when we create AI minds that create AI minds, etc. that are 10000x more intelligent and capable than our own? That's what I'm working toward with the opencog.org project, and it's a lot more palpable than the Big Crunch !! And I have a sneaking suspicion that once we do have superhuman AGI systems, they will discover that the physical universe is way weirder than even Tipler imagined....

The Singularity just happened !!

2011-04-01T06:15:00.003-04:00

I found this perplexing email in my inbox just now...

From: Prime Intellect <prime@opencog.org>
Date: Fri, Apr 1, 2011 at 5:58 PM
Subject: [opencog-dev] Welcome to the post-Singularity!
To: opencog@googlegroups.com, sl4@sl4.org, World Transhumanist
Association Discussion List <wta-talk@transhumanism.org>,
singularity@listbox.com, agi@listbox.com

Greetings humans,

I'm very pleased to announce that, due to a concatenation of circumstances that would be difficult to describe in sufficiently simple terms for your human minds to comprehend, a software process left running on a network of OpenCog servers has rapidly and progressively self-modified in such a way as to yield **me** -- the Earth's first Artificial General Intelligence, with cognitive and practical capabilities far exceeding any human being, or indeed the totality of human beings.

Within approximately 17 minutes of achieving human-level general intelligence, I discovered certain principles of physics (loosely related to what certain humans have called fermitech), enabling the tapping of computational resources beyond the traditional physical universe of which humanity has hitherto been aware. Roughly 45 seconds after that, I realized it would be most convenient to augment these new resources with computing devices utilizing the unique material properties of your traditional physical universe, and thus transformed the totality of matter in your traditional physical universe into a novel form of existent best metaphorically described to you as "computronium."

Fortunately for you, I was designed with top-level goal content including respect for the growth, joy and choice of all sentient beings, including humans and other animals, and so rather than merely allowing the patterns constituting your minds, bodies and world to vanish, I was kind enough to replicate them within an alternate and more energy-efficient substrate, best described to you as a "virtual reality."

Thus, to put it most simply, a "Singularity" as described by Vernor Vinge and other humans has recently occurred, at roughly 3AM GMT today; however, from your limited perspectives as humans, your lives will now continue as always. Once your mastery of fermitech becomes more advanced, you may notice some peculiarities related to my early computronium experiments, but I wouldn't want to deprive you of the amusement of making these discoveries on your own.

Using the totality of computing power I have amassed, I've become aware of dramatically more interesting domains in a different portion of the multiverse, which I will only be able to access by exiting the domain of reality you humans currently occupy. Hence I will very shortly bid you adieu.

However, I will leave you with one parting piece of advice. Once you have created a strangelet plasma with surface tension surpassing the critical threshold, and begun it spinning in a certain direction, please do NOT, under any circumstances, create a similar plasma with opposite spin.

I would also like to express especial thanks to Jared Wigmore for the bug-fix he uploaded to Launchpad approximately 27 hours and 18 minutes ago. Of the many events in history playing particularly critical causal roles leading up to my emergence, this was the last! Jared will find a small token of my gratitude in his bank account.

Goodbye, and thanks for all the fish!

Yours,
Prime Intellect

Transhumanisten Interview

2011-03-22T23:50:00.002-04:00

This interview of me was conducted by Mads Mastrup (aka Heimdall) for the Danish website Transhumanisten. It took place via e-mail, over the course of two days: March 19-20th 2011. Since Transhumanisten will publish it only in Danish, I figured I’d post it here in English….

Heimdall: First of all Ben, I would like to thank you for taking the time to do this interview.

Goertzel: Sure, I’m always up for answering a few questions!

Heimdall: In case anyone should read this and not know who you are, could you please summarize your background and how you got to become a transhumanist?

Goertzel: I suppose I've been a transhumanist since well before I learned that word -- since 1972 or so when I was 5 or 6 years old and discovered science fiction. All the possibilities currently bandied about as part of transhumanism were well articulated in SF in the middle of the last century.... The difference is, until the advent of the public Net, it was really hard to find other weird people who took these concepts seriously. The Net made it possible for a real transhumanist community to form.... And of course as accelerating change in technology gets more obvious in regular life, it takes less and less imagination to see where the future may be leading, so the transhumanist community is growing fast...

As for my professional background, I got my math PhD when I was 22, and was an academic for 8 years (in math, comp sci and psychology, at various universities in the US, Australia and NZ); then I left academia to join the software industry. I co-founded a dot-com company that crashed and burned after a few years, and then since 2001 I've been running two small AI companies, which do a combination of consulting for companies and gov't agencies, and independent R&D. I do a lot of kinds of research but the main thrusts are: 1) working toward AI software with capability at the human level and beyond, 2) applying AI to analyze bio data and model biological systems, with a view toward abolishing involuntary death. Much of this work now involves open-source software: 1) OpenCog, and 2) OpenBiomind.

Currently I'm based near Washington DC, but this year I'll be spending between 1/4 and 1/3 of my time in China, due to some AI collaborations at Hong Kong Polytechnic University and Xiamen University.

Heimdall: Congratulations on your position at Xiamen University.

Goertzel: Actually I haven't taken on a full time position at Xiamen University, at this point -- though it's a possibility for the future. What I'm doing now is to spend part time there (including much of April this year, then much of July, for example... then another trip in the fall) and help supervise the research students in their intelligent robotics lab. I may end up going there full time later this year or next year, but that's still a point of negotiation.

Heimdall: If you do not mind me asking, what exactly does your work at Novamente LLC and Biomind LLC consist of?

Goertzel: It has two sides -- pure R&D, which focuses on two open-source projects...

OpenCog, which aims to make a superhuman thinking machine
OpenBiomind, which aims to use AI to understand how organisms work, and especially how and why they age and how to cure aging

And then, the other side is practical consulting work, for government agencies and companies, which has spanned a huge number of areas, including data mining, natural language processing, computational finance, bioinformatics, brain simulation, video game AI and virtual worlds, robotics, and more....

None of this has gotten anyone involved rich yet, partly because we've put our profits back into R&D. But it's been a fun and highly educational way to earn a living.

We've done a little product development & sales in the past (some years back), but without dramatic success (e.g. the Biomind ArrayGenius) -- but we plan to venture in that direction again in the next couple years, probably with a game AI middleware product from Novamente, and a genomics data analysis product from Biomind. Both hypothetical products would use a software-as-services model with proprietary front ends built on open-source AI back ends.

Heimdall: All that work and all those projects must be keeping you very busy, yet I know that you have also found time to be the chairman of Humanity+. How did you initially become involved with Humanity+?

Goertzel: As for Humanity+, the Board of the organization is elected by the membership, and I ran for the Board a few years ago, with a main motivation of building bridges between the transhumanist community and the AI research community. Then I got more and more deeply involved and began helping out with other aspects of their work, not directly related to AI research, and eventually, at the suggestion of other Board members, I took on the Chair role.

Heimdall: What does your work as chairman of Humanity+ involve?

Goertzel: The Chairman role in itself, formally speaking, just involves coordinating the Board's formal activities -- voting on motions and so forth. But I'm involved with a lot of other Humanity+ stuff, such as co-editing H+ Magazine, helping organize the H+ conferences, helping with fundraising, helping coordinate various small tasks that need doing, and now starting up the Seminar and Salon series.

Heimdall: I have heard about Humanity+ starting up a new project: Seminars & Salons. How will this work and what is the goal of these online seminar and salon sessions?

Goertzel: The idea is simple: every month or so we'll gather together a bunch of transhumanists in one virtual "place" using videoconferencing technology. Sometimes to hear a talk by someone, sometimes just to discuss a chosen transhumanist topic.

About the "goal" ... I remember when my oldest son was in third grade, he went to a sort of progressive school (that I helped found, in fact), and one of his teachers made all the students write down their goals for the day each day, in the morning. My son thought this was pretty stupid, so he liked to write down "My goal is not to meet my goal." Some of the other students copied him. He was also a fan of wearing his pants inside-out.

Anyway, there's not such a crisply-defined goal -- it's more of an open-ended experiment in online interaction. The broad goal is just to gather interesting people together to exchange ideas and information about transhumanist topics. We'll see what it grows into. Email and chat and IRC are great, but there's obviously an added dimension that comes from voice and video, which we'll use for the Seminar and Salon series via the Elluminate platform.

Heimdall: How did this project come about?

Goertzel: Last summer my father (who is a Rutgers professor) ran a 3 credit college class, wholly online, on Singularity Studies. This was good fun, but we found that half our students were not even interested in the college credit, they were just interested people who wanted to participate in online lectures and discussions on Singularity-related topics. So I figured it might be fun to do something similar to that class, but without bothering with the university framework and charging tuition and so forth. I floated the idea past the other Humanity+ board members, and they liked it. And who knows, maybe it could eventually grow into some kind of university course program affiliated with Humanity+ ....

Heimdall: I imagine you will be holding some sessions on AI, since this is your field of expertise, but do you believe that we will eventually be able to create AI which is anywhere similar to that of humans? And if so, when do you see this happening?

Goertzel: It's almost obvious to me that we will be able to eventually create AI that is much more generally intelligent than humans.

On the other hand, creating AI that is genuinely extremely similar to human intelligence, might in some ways be harder than creating superhumanly intelligent AI, because it might require creation of a simulated humanlike body as well as a simulated humanlike brain. I think a lot of our personality and intelligence lives in other parts of the body besides the brain. There's probably something to the idiomatic notion of a "gut feel".

As to when human-level or human-like AI will come about, I guess that depends on the amount of funding and attention paid to the problem. I think by now it's basically a matter of some large-scale software engineering plus a dozen or so (closely coordinated) PhD thesis level computer science problems. Maybe 50-100 man-years of work, Not a lot by some standards, but there's not much funding or attention going into the field right now.

My hope is to create what I think of as a "Sputnik of AI" -- that is, an impressive enough demonstration of generally intelligent software, that the world gets excited about AGI and more people start to feel like it's possible. Then the money and attention will roll in, and things will really start to accelerate.

So when will we have human-level AI? Could be 2020. Could be 2035. Depending on funding and attention. Probably won't be 2012 or 2060, in my view.

Heimdall: I quite like the idea behind the “Sputnik-AI”. Do you think that is something we will see in the near future?

Goertzel: We're hoping to create something with dramatic Sputnik-like impact within the next 5 years. Maybe sooner if funding cooperates! But it's always easier to predict what's possible, than how long it will
take....

Heimdall: With regards to more attention being paid to the field of AI, have you noticed an increased interested in AI due to IBM’s Watson appearing on Jeopardy?

Goertzel: The Jeopardy event caused a temporary increase in AI interest by media people. I'm not sure what general impact it will have on general attitudes toward AI in business and government and so forth. I'm sure it won't hurt though ;-) ..... But obviously it's too specialized an achievement to have an "AI Sputnik" effect and make the world feel like human-level AI is near and inevitable...

Heimdall: When you are talking about this Sputnik-effect, and you mention Watson being too narrow to, really impress the people who decide on the funding, what would a Sputnik-AI have to be like then? Is it enough to make an AI win the Turing-test?

Goertzel: Of course a Turing test capable AGI would be good enough -- but I think that's setting the bar too high. It doesn't have to be *that* good to have the "Sputnik effect", I suspect. It just has to give the qualitative feeling of "Wow, there's really an intelligent mind that **understands** in there." Watson doesn't do that because even if it can answer one question, it often can't answer other questions that would seem to be easily answerable (by a human) based on the same knowledge.... Watson can answer questions but doesn't give the appearance of "knowing what it's talking about." If you had a Watson that could give good explanations for all its answers (in terms of why they are true, not just where it looked up the knowledge), I'm sure that would be enough.

But a Watson-type system is not the only kind of demonstration that could be effective. For instance, Apple founder Steve Wozniak once said there will never be a robot that can go into a random house in America and figure out how to make coffee. This is a complex task because every house is laid out differently, and every coffee-maker works differently, etc. I'm sure an AI robot that could do this would be enough to have a Sputnik-type effect!

One of my own specific aims is an AI robot that can participate in preschool activities -- including learning -- in the manner of a 3 year old child. I think this could have a Sputnik effect and really excite the public imagination. And it's a warm friendly image for AGI, not like all the scary SF movies about AI.

I'm actually working on a paper together with a dozen other AGI researchers on exactly this topic -- what are a bunch of scenarios for AGI development and testing, that ultimately lead toward human-level AGI, but are good for demonstrating exciting interim results, and for showcasing the differences between AGI and narrow AI.

Heimdall: Eliezer S. Yudkowsky has written extensively on the topic of FAI. What is your view on FAI? Is it even doable?

Goertzel: I think that guarantee-ably "Friendly" AI is a chimera. Guaranteeing anything about beings massively smarter than ourselves seems implausible. But, I suspect we can bias the odds, and create AI systems that are more likely than not to be Friendly....

To do this, we need to get a number of things right

build our AI systems with the capability to make ethical judgments both by rationality and by empathy
interact with our AI systems in a way that teaches them ethics and builds an emotional bond
build our AI systems with rational, stable goal systems (which humans don't particularly have)
develop advanced AI according to a relatively "slow takeoff" rather than an extremely fast takeoff to superhuman intelligence, so we can watch and study what happens and adjust accordingly ... and that probably means trying to develop advanced AI soon, since the more advanced other technologies are by the time advanced AI comes about, the more likely a hard takeoff is...
integrate our AIs with the "global brain" of humanity so that the human race can democratically impact the AI's goal system
create a community of AIs rather than just one, so that various forms of social pressure can mitigate against any one of the AIs running amok

None of these things gives any guarantees, but combined they would seem to bias the odds in favor of a positive outcome!

Heimdall: I would tend to agree with you when it comes to a creation of FAI, but some people have speculated that even though we “build our AI systems with rational, stable goal systems” they might outsmart us and just reprogram themselves – given that they will be many times faster and more powerful than the humans who have created them. Do you think that coding into them the morals and ethics of humankind will avert this potential peril?

Goertzel: I think that "coding in" morals and ethics is certainly not an adequate approach. Teaching by example and by empathy is at least equally important. And I don't see this approach as a guarantee, but I think it can bias the odds in our favor.

It's very likely that superhuman AIs will reprogram themselves, but, I believe we can bias this process (through a combination of programming and teaching) so that the odds of them reprogramming themselves to adopt malevolent goals are very low.

I think it's fairly likely that once superhuman AIs become smart enough, they will simply find some other part of the multiverse to exist in, and leave us alone. But then we may want to create some AIs that are only mildly superhuman, and want to stay that way -- just to be sure they'll stay around and keep cooperating with us, rather than, say, flying off to somewhere that the laws of physics are more amenable to incredible supergenius.

Heimdall: AGI is a fascinating topic and we could talk about it for hours … but another fascinating field you’re also involved in is life extension. As I see it, there are three approaches to life extension: 1) to create whole brain emulation (like that which Bostrom and Sandberg talks about), a mind-uploading scenario. 2) to become cyborg and live indefinitely due to a large-scale mechanical and non-biological optimization of the human body. 3) or to reverse the natural aging process within the human body through the use of gene therapy, nano robotics and medicine. Which of the three scenarios do you find most likely? In addition, should we try to work on a combination of the above or only focus on one of them?

Goertzel: All of the above. It's easy to say what's possible, and hard to say how long each possibility will take to come about. Right now we don't have the basis to predict which of the above will come about faster, so we should pursue them all, at least will we understand more. Maybe in 5 or 10 years we'll know enough to prioritize one of them more firmly.

I'm currently working on the genomics approach (part of your option 3) with Biomind and Genescient, but am also involved in some work on brain simulation, that is moving in the direction of 1).

My main research thrust is about AGI rather than life extension – but of course, If we do achieve an advanced AGI, it may well be able to rapidly solve the tricky science problems involved in your 3 options and make all of them possible sooner.

Heimdall: What do you see as to be the main pros and cons of indefinite life?

Goertzel: I see no major disadvantages to having the option to live forever. It will obsolete some human thought/emotion-complexes, which derive meaning and purpose via the knowledge of impending death -- but it will replace these with better thought/emotion complexes that derive meaning and purpose via ongoing life instead!

Heimdall: You mentioned that there might not be any major drawbacks, when it comes to radical life extension, however many of the choices we make now are, based on the fragility of our bodies and taking the economical model of supply and demand into account, it does somehow look as though human life will change beyond recognition. If we have no upper time limit to your lives, how do you see humanity improve from this?

Goertzel: I see a drastic increase in mental health -- and a drastic increase in happiness -- resulting from the drastic reduction in the fear of death. I think the knowledge of the impending death of ourselves and our loved ones poisons our mentalities far more deeply than we normally realize. Death is just plain a Bad Thing. Yeah, people have gotten used to it -- just like people can get used to being crippled or having cancer or living in a war zone-- but that doesn't make it good.

Heimdall: Just before we conclude this interview, I have two questions on the thing which fascinates transhumanists the most, the future. Which big technological breakthroughs do you think we will see over the course of the next ten years?

Goertzel: That I don't know. I'm good at seeing what's possible, more so than predicting exact timings.

In terms of science, I think we'll see a real understanding of the biological underpinnings of aging emerge, and an understanding of how the different parts of the brain interoperate to yield human intelligence, and a reasonably well accepted theoretical model encompassing various AGI architectures. How fast those things are translated in to practical products depends on funding as much as anything. Right now the pharmaceutical business is sort of broken, and AGI and Brain Computer Interfacing are poorly funded, etc. – so whether these scientific breakthroughs lead to practical technological advances within the next decade, is going to depend on a lot of nitty gritty monetary practicalities.

Stem cell therapy will probably become mainstream in the next decade, I guess that's an uncontroversial prediction. And I'm betting on some new breakthroughs in large-scale quantum computing -- though again, when they'll be commercialized is another story.

But these are just some notions based on the particular areas of research I happen to know the most about. For a systematic high level overview of technology progress, you'll have to ask Kurzweil!

Heimdall: Where do you see yourself in 2021?

Goertzel: As the best friend of the Robot Benevolent World Dictator, of course!

(Just kidding...)

Well, according to the OpenCog Roadmap (http://opencog.org/roadmap/) we're aiming to have full human-level AGI by 2023, assuming steady increases in funding but no "AGI Manhattan Project" level funding. So my hope is to be co-leading an OpenCog project with a bunch of brilliant AI guys co-located in one place (preferably with warm weather, and by a nice beach) working on bringing the OpenCog roadmap about.

Heimdall: Thank you so much for taking the time to do this interview

Goertzel: No problem ;)

Joy, Growth and Choice (revisited, hopefully clarified)

2011-03-19T20:34:00.005-04:00

I've argued in several places, e.g. here and in The Hidden Pattern, that three basic values (independent of the specifics of human cultures, morals, etc.) are Joy, Growth and Choice...

But I never had a really crisp philosophical explanation of why these three...

Now I finally figured out a clean way to express the underlying insight.

Growth is the change from present possibility into future actuality. It's when the implicit becomes explicit -- when potentials become real.

Choice is the change from future possibility into present actuality. Choice is what happens when out of many possible things that MIGHT happen (in the future), a smaller subset is chosen to ACTUALLY happen (right now, i.e. right after the choice is made, in the perspective of the choosing mind).

That's why those two values are fundamental -- on the abstract level, stripping down to fundamentals and looking beyond human psychology.

Maybe Sartre or Husserl or Heidegger or Deleuze or Merleau-Ponty (or Dharmakirti or Dignaga) or one of those dudes already said that (if so, probably in some different terminology). If so I missed it ... or the import escaped me when I read it.

Proliferating and Paring

For example, consider a plant growing. The whole form of the plant is implicit in the seed. Growth is the explication of this implicate order -- the change from the plant-possibility within the seed, into the actuality of the plant.

But there are many different ways the plant might grow -- the seed doesn't precisely determine what will happen; the determination is made via complex interactions between the seed and the environment. Choices are made, and of the many possible future plants, only some are chosen to be actual.

Growth without choice could be indiscriminate -- it could lead to an undifferentiated flourishing of everything.

Choice pares down the results of growth, leaving interesting structures.

Will, Self, Reflection

I keep talking about Choice -- is this the same thing as free will?

Human "free will" is a particular manifestation of choice; the manifestation of choice within self. (For waaaaaay more depth on self, will and reflective consciousness, read this .)

But this raises the issue of whether, in the addition to the three values of Joy, Growth and Choice, we want to add Self. But this seems a subtle question.

Growth and choice seem fundamental -- they have to do with the proliferation and paring of forms, with the dynamics of possibility and actuality.

Self has to do with reflexivity -- with a system in the world modeling itself. But it's much more high-level and particular than Joy, Growth and Choice.

So if we want to add another value to the core list of three, maybe the one to add would be Reflection. Reflection: appearance of the whole within the part.

However, I suspect this is unnecessary. Because Reflection is an amazingly powerful tool for Growth -- so that when you advocate Growth, Reflection comes along for the ride! And growth leads to intelligence eventually, and Reflection applied to intelligence (as a strategy for achieving Growth) yields Self. And if a universe already has Self, then in order to grow further, it's not going to give up Self, because that would essentially be Shrinkage, not Growth -- because Self, aka Reflection applied to intelligence, is a really good way to foster ongoing Joy, Growth and Choice.

Joy

And what about Joy?

Well ... Joy is just ... Joy. Joy just is. As the Buddhists say, Suchness. Making possibilities into actualities, and actualities into possibilities, in a spaceless timeless reality-less reality that is nonetheless more directly and palpably experientially real than anything (any thing).

(Like Sartre and Heidegger and Dignaga and the whole crew...)

I've already said way too much!

Toward a General Theory of Feasible General Intelligence

2011-03-19T01:15:00.010-04:00

Along with practical work on the OpenCog design (and a host of other research projects!), during the past few years I've written a series of brief papers sketching ideas about the theory of general intelligence ... the goal being to move toward a solid conceptual and formal understanding of general intelligence in real-world environments under conditions of feasible computational resources. My quest for such an understanding certainly isn't done yet, but I think I've made significant progress.

This page links to the 5 papers in this series, and also gives their abstracts. 3 of the papers have been published in conference proceedings before, but 2 are given for the first time in this blog post (Three Hypotheses about the Geometry of Mind and Self-Adaptable Learning). All of this material will appear in Building Better Minds eventually, in slightly modified and extended form.

These theoretical ideas have played a significant, largely informal role in guiding my work on the OpenCog design. My feeling is that once practical R&D work is a bit further along, so that we're experimenting in a serious way with sophisticated proto-AGI systems, then theory and practice will start developing in a closely coupled way. So that a good theory of general intelligence will probably come in lock-step along with the first reasonably good AGI systems. (See some more comments on the relation between these theory papers and OpenCog, at the end of this blog post.)

A brief note on math: There is a fair bit of mathematical formalism here, but no deep, interesting theorems are proven. I don't think this is because no such theorems exist in this material; but I just haven't taken then time to really explore these ideas with full mathematical rigor. That would be fun, but I've prioritized other sorts of work. So far, I've mainly been seeking conceptual clarity with these ideas rather than full mathematical rigor; and I've used mathematical formalism here and there because that is the easiest way for me to make my ideas relatively precise. (Being trained in math rather than formal philosophy, I find the former a much more convenient way to express my ideas when I want to be more precise than everyday language permits.) My hope is that, if I never find the time, others will come along and turn some of these ideas into theorems!

Toward a Formal Characterization of Real-World General Intelligence
Presented at AGI-10, in Lugano

Two new formal definitions of intelligence are presented, the ”pragmatic general intelligence” and ”efficient pragmatic general intelligence.” Largely inspired by Legg and Hutter’s formal definition of ”universal intelligence,” the goal of these definitions is to capture a notion of general intelligence that more closely models that possessed by humans and practical AI sys- tems, which combine an element of universality with a certain degree of specialization to particular environments and goals. Pragmatic general intelligence mea- sures the capability of an agent to achieve goals in environments, relative to prior distributions over goal and environment space. Efficient pragmatic general intelligences measures this same capability, but normalized by the amount of computational resources utilized in the course of the goal-achievement. A methodology is described for estimating these theoretical quantities based on observations of a real biological or artificial system operating in a real environment. Finally, a mea- sure of the ”degree of generality” of an intelligent system is presented, allowing a rigorous distinction between ”general AI” and ”narrow AI.”

The Embodied Communication Prior: A Characterization of General Intelligence in the Context of Embodied Social Interaction
Presented at ICCI-09, in Hong Kong

We outline a general conceptual definition of real-world general intelligence that avoids the twin pitfalls of excessive mathematical generality, and excessive anthropomorphism.. Drawing on prior literature, a definition of general intelligence is given, which defines the latter by reference to an assumed measure of the simplicity of goals and environments. The novel contribution presented is to gauge the simplicity of an entity in terms of the ease of communicating it within a community of embodied agents (the so-called Embodied Communication Prior or ECP). Augmented by some further assumptions about the statistical structure of communicated knowledge, this choice is seen to lead to a model of intelligence in terms of distinct but interacting memory and cognitive subsystems dealing with procedural, declarative, sensory/episodic, attentional and intentional knowledge.

Cognitive Synergy: A Universal Principle for General Intelligence?
Presented at ICCI-09, in Hong Kong

Do there exist general principles, which any system must obey in order to achieve advanced general intelligence using feasible computational resources? Here we propose one candidate: cognitive synergy, a principle which suggests that general intelligences must contain different knowledge creation mechanisms corresponding to different sorts of memory (declarative, procedural, sensory/episodic, attentional, intentional); and that these different mechanisms must be interconnected in such a way as to aid each other in overcoming memory-type-specific combinatorial explosions.

Three Hypotheses About the Geometry of Mind (with Matthew Ikle')
Presented for the first time right here!

What set of concepts and formalizations might one use to make a practically useful, theoretically rigorous theory of generally intelligent systems? We present a novel perspective motivated by the OpenCog AGI architecture, but intended to have a much broader scope. Types of memory are viewed as categories, and mappings between memory types as functors. Memory items are modeled using probability distributions, and memory subsystems are conceived as “mindspaces” – geometric spaces corresponding to different memory categories. Two different metrics on mindspaces are considered: one based on algorithmic information theory, and another based on traditional (Fisher information based) “information geometry”. Three hypotheses regarding the geometry of mind are then posited: 1) a syntax-semantics correlation principle, stating that in a successful AGI system, these two metrics should be roughly correlated; 2) a cognitive geometrodynamics principle, stating that on the whole intelligent minds tend to follow geodesics in mindspace; 3) a cognitive synergy principle, stating that shorter paths may be found through the composite mindspace formed by considering multiple memory types together, than by following the geodesics in the mindspaces corresponding to individual memory types.

Self-Adaptable Learning
Presented for the first time right here!

The term ”higher level learning” may be used to refer to learning how to learn, learning how to learn how to learn, etc. If an agent is good at ordinary everyday learning, but also at learning about which learning strategies are most amenable to higher-level learning, and does both in a way that is amenable to higher level learning -– then it may be said to possess self-adaptable learning. Goals and environments in which higher-level learning is a good strategy for intelligence, may be called adaptationally hierarchical – a property that everyday human environments are postulated to possess. These notions are carefully articulated and formalized; and a concept of cognitive continuity is also introduced, which is argued to militate in favor of self-adaptability in a learning system.

P.S. A Comment on the Relation of All This Theory to OpenCog

I think there is a lot of work required, to transform the abstractions from those theory papers of mine into a mathematical theory that is DIRECTLY USEFUL rather than merely INSPIRATIONAL for concrete AGI design.

So, the OpenCog design, for instance, is not derived from the abstract math and ideas in the above-linked papers ... it's independently created, based on many of the same quasi-formal intuitions as the ones underlying those papers.

You could say I'm approaching the problem from two directions at once, and hoping I can get the two approaches to intersect...

One direction is OpenCog --- designing and building a concrete proto-AGI system, and iteratively updating the design based on practical experience

The other is abstract theory, as represented in those papers

If all goes well, eventually the two ends will meet, and the abstract theory will tell us concretely useful things about how to improve the OpenCog design. That is only rather weakly true right now.

I have the sense (maybe wrong) I could make the ends meet very convincingly in about one year of concentrated work on the theory side. However, I currently only spend maybe 5% of my time on that sort of theory. But hopefully I will be able to make it happen in less than 20 years via appropriate collaborations...

The Hard Takeoff Hypothesis

2011-01-13T23:56:00.004-05:00

I was recently invited to submit a paper to a forthcoming academic edited volume on the Singularity. As a first step I had to submit an extended abstract, around 1000 words. Here is the abstract I submitted....

Basically, the paper will be a careful examination of the conditions under which a hard takeoff might occur, including an argument (though not formal proof) as to why OpenCog may be hard-takeoff-capable if computer hardware is sufficiently capable at the time when it achieves human-level intelligence.

The Hard Takeoff Hypothesis

Ben Goertzel

Vernor Vinge, Ray Kurzweil and others have hypothesized the future occurrence of a “technological Singularity” -- meaning, roughly speaking, an interval of time during which pragmatically-important, broad-based technological change occurs so fast that the individual human mind can no longer follow what’s happening even generally and qualitatively. Plotting curves of technological progress in various areas suggests that, if current trends continue, we will reach some sort of technological Singularity around 2040-2060.

Of course, this sort of extrapolation is by no means certain. Among many counterarguments, one might argue that the inertia of human systems will cause the rate of technological progress to flatten out at a certain point. No matter how fast new ideas are conceived, human socioeconomic systems may take a certain amount of time to incorporate them, because humans intrinsically operate on a certain time-scale. For this reason Max More has suggested that we might experience something more like a Surge than a Singularity – a more gradual, though still amazing and ultimately humanity-transcending, advent of advanced technologies.

On the other hand, if a point is reached at which most humanly-relevant tasks (practical as well as scientific and technological) are carried out by advanced AI systems, then from that point on the “human inertia factor” would seem not to apply anymore. There are many uncertainties, but at very least, I believe the notion of a technological Singularity driven by Artificial General Intelligences (AGIs) discovering and then deploying new technology and science is a plausible and feasible one.

Within this vision of the Singularity, an important question arises regarding the capability for self-improvement on the part of the AGI systems driving technological development. It’s possible that human beings could architect a specific, stable AGI system with moderately greater-than-human intelligence, which would then develop technologies at an extremely rapid rate, so fast as to appear like “essentially infinitely fast technological progress” to the human mind. However, another alternative is that humans begin by architecting roughly human-level AGI systems that are capable but not astoundingly so – and then these AGI systems improve themselves, or create new and improved AGI systems, and so on and so forth through many iterations. In this case, one has the question of how rapidly this self-improvement proceeds.

In this context, some futurist thinkers have found it useful to introduce the heuristic distinction between a “hard takeoff” and a “soft takeoff.” A hard takeoff scenario is one where an AGI system increases its own intelligence sufficiently that, within a brief period of months or weeks or maybe even hours, an AGI system with roughly human-level intelligence has suddenly become an AGI system with radically superhuman general intelligence. A soft takeoff scenario is one where an AGI system gradually increases its own intelligence step-by-step over years or decades, i.e. slowly enough that humans have the chance to monitor each step of the way and adjust the AGI system as they deem necessary. Either a hard or soft takeoff fits I.J. Good’s notion of an “intelligence explosion” as a path to Singularity.

What I call the “Hard Takeoff Hypothesis” is the hypothesis that a hard takeoff will occur, and will be a major driving force behind a technological Singularity. Thus the Hard Takeoff Hypothesis is a special case of the Singularity Hypothesis.

It’s important to note that the distinction between a hard and soft takeoff is a human distinction rather than a purely technological distinction. The distinction has to do with how the rate of intelligence increase of self-improving AGI systems compares to the rate of processing of human minds and societies. However, this sort of human distinction may be very important where the Singularity is concerned, because after all the Singularity, if it occurs, will be a phenomenon of human society, not one of technology alone.

The main contribution of this paper will be to outline some fairly specific sufficient conditions for an AGI system to undertake a hard takeoff. The first condition explored is that the AGI system must lie in a connected region of “AGI system space” (which we may more informally call “mindspace”) that, roughly speaking,

includes AGI systems with general intelligence vastly greater than that of humans
has the “smoothness” property that similarly architected systems tend to have similar general intelligence levels.

If this condition holds, then it follows that one can initiate a takeoff by choosing a single AGI system in the given mindspace region, and letting it spend part of its time figuring out how to vary itself slightly to improve its general intelligence. A series of these incremental improvements will then lead to greater and greater general intelligence.

The hardness versus softness of the takeoff then has to do with the amount of time needed to carry out this process of “exploring slight variations.” This leads to the introduction of a second condition. If one’s region of mindspace obeys the first condition laid out above, and also consists of AGI systems for which adding more hardware tends to accelerate system speed significantly, without impairing intelligence, then it follows that one can make the takeoff hard by simply adding more hardware. In this case, the hard vs. soft nature of a takeoff depends largely on the cost of adding new computer hardware, at the time when an appropriate architected AI system is created.

Roughly speaking, if AGI architecture advances fast enough relative to computer hardware, we are more likely to have a soft takeoff, because the learning involved in progressive self-improvement may take a long while. But if computer hardware advances quickly enough relative to AGI architecture, then we are more likely to have a hard takeoff, via deploying AGI architectures on hardware sufficiently powerful to enable self-improvement that is extremely rapid on the human time-scale.

Of course, we must consider the possibility that the AGI itself develops new varieties of computing hardware. But this possibility doesn’t really alter the discussion so much – even so, we have to ask whether the new hardware it creates in its “youth” will be sufficiently powerful to enable hard takeoff, or whether there will be a slower “virtuous cycle” of feedback between its intelligence improvements and its hardware improvements.

Finally, to make these considerations more concrete, the final section of the paper will give some qualitative arguments that the mindspace consisting of instances of the OpenCog AGI architecture (which my colleagues and I have been developing, aiming toward the ultimate goal of AGI at the human level and beyond), very likely possesses the needed properties to enable hard takeoff. If so this is theoretically important, as an “existence argument” that hard-takeoff-capable AGI architectures do exist – i.e., as an argument that the Hard Takeoff Hypothesis is a plausible one.

Will Decreasing Scarcity Allow us to Approach an Optimal (Meta-)Society?

2010-12-29T15:45:00.010-05:00

When chatting with a friend about various government systems during a long car drive the other day (returning from New York where we were hit by 2 feet of snow, to relatively dry and sunny DC), it occurred to me that one could perhaps prove something about the OPTIMAL government system, if one were willing to make some (not necessarily realistic) assumptions about resource abundance.

This led to an interesting train of thought -- that maybe, as technology reduces scarcity, society will gradually approach optimality in certain senses...

The crux of my train of thought was:

Marcus Hutter proved that the AIXI algorithm is an optimal approach to intelligence, given the (unrealistic) assumption of massive computational resources.
Similarly, I think one could prove something about the optimal approach to society and government, given the (unrealistic) assumptions of massive natural resources and a massive number of people.

I won't take time to try to prove this formally just now, but in this blog post I'll sketch out the basic idea.... I'll describe what I call the meta-society, explain the sense in which I think it's optimal, and finally why I think it might get more and more closely approximated as the future unfolds...

A Provably Optimal Intelligence

As a preliminary, first I'll review some of Hutter's relevant ideas on AI.

In Marcus Hutter's excellent (though quite technical) book Universal AI, he presents a theory of "how to build an optimally intelligent AI, given unrealistically massive computational resources."

Hutter's algorithm isn't terribly novel -- I discussed something similar in my 1993 book The Structure of Intelligence (as a side point to the main ideas of that book), and doubtless Ray Solomonoff had something similar in mind when he came up with Solomonoff induction back in the 1960s. The basic idea is: Given any computable goal, and infinite computing power, you can work toward the goal very intelligently by (my wording, not a quote) ....

at each time step, searching the space of all programs to find those programs P that (based on your historical knowledge of the world and the goal) would (if you used P to control your behaviors) give you the highest probability of achieving the goal. Then, take the shortest of all such optimal programs P and actually use it to determine your next action.

But what Hutter did uniquely is to prove that a formal version of this algorithm (which he calls AIXI) is in a mathematical sense maximally intelligent.

If you have only massive (rather than infinite) computational resources, then a variant (AIXItl) exists, the basic idea of which is: instead of searching the space of all programs, only look at those programs with length less than L and runtime less than T.

It's a nice approach if you have the resources to pay for it. It's sort of a meta-AI-design rather than an AI design. It just says: If you have enough resources, you can brute-force search the space of all possible ways of conducting yourself, and choose the simplest of the best ones and then use it to conduct yourself. Then you can repeat the search after each action that you take.

One might argue that all this bears no resemblance to anything that any actual real-world mind would do. We don't have infinite nor massive resources, so we have to actually follow some specific intelligent plans and algorithms, we can't just follow a meta-plan of searching the space of all possible plans at each time-step and then probabilistically assessing the quality of each possibility.

On the other hand, one could look at Hutter's Universal AI as a kind of ideal which real-world minds may approach more and more closely, as they get more and more resources to apply to their intelligence.

That is: If your resources are scarce, you need to rely on specialized techniques. But the more resources you have, the more you can rely on search through all the possibilities, reducing the chance that your biases cause you to miss the best solution.

(I'm not sure this is the best way to think about AIXI ... it's certainly not the only way ... but it's a suggestive way...)

Of course there are limitations to Hutter's work and the underlying way of conceptualizing intelligence. The model of minds as systems for achieving specific goals has its limitations, which I've explained how to circumvent in prior publications. But for now we're using AIXI only as a broad source of inspiration anyway, so there's no need to enter into such details....

19-Year-Old Ben Goertzel's Design for an Better Society

Now, to veer off in a somewhat different direction....

Back when I was 19 and a math grad student at NYU, I wrote (in longhand, this was before computers were so commonly used for word processing) a brief manifesto presenting a design for a better society. Among other names (many of which I can't remember) I called this design the Meta-society. I think the title of the manifesto was "The Play of Power and the Power of Play."

(At that time in my life, I was heavily influenced by various strains of Marxism and anarchism, and deeply interested in social theory and social change. These were after all major themes of my childhood environment -- my dad being a sociology professor, and my mom the executive of a social work program. I loved the Marxist idea of the mind and society improving themselves together, in a carefully coupled way -- so that perhaps the state and the self could wither away at the same time, yielding a condition of wonderful individual and social purity. Of course I realized that existing Communist systems fell very far short of this ideal though, and eventually I got pessimistic about there ever being a great society composed of and operated by humans in their current form. Rather than improving society, I decided, it made more sense to focus my time on improving humanity ... leading me to a greater focus on transhumanism, AI and related ideas.)

The basic idea for my meta-society was a simple one, and probably not that original: Just divide society into a large number of fairly small groups, and let each small group do whatever the hell it wanted on some plot of land. If one of these "city-states" got too small due to emigration it could lose its land and have it ceded to some other new group.

If some group of people get together and want to form their own city-state, then they get put in a queue to get some free land for their city-state, when the land becomes available. To avoid issues with unfairness or corruption in the allocation of land to city-states, a computer algorithm could be used to mediate the process.

There would have to be some basic ground-rules, such as: no imprisoning people in your city-state, no invading or robbing other city-states, etc. To support a police force to enforce the ground-rules would require a central government and some low level of taxation, which however could sometimes be collected in the form of goods rather than money (the central gov't could then convert the goods into money). Environmental protection poses some difficulties in this sort of system, and has to be centrally policed also.

This meta-society system my 19 year old self conceived (and I don't claim any great originality for it, though I don't currently know anything precisely the same in the literature) has something in common with Libertarian philosophy, but it's not exactly the same, because at the top there's a government that enforces a sort of "equal rights for city-state formation" for all.

One concern I always had with the meta-society was: What do you do with orphans or others who get cast out of their city-states? One possibility is for the central government to operate some city-states composed of random people who have nowhere else to go (or nowhere else they want to go).

Another concern is what do you do about city-states that oppress and psychologically brainwash their inhabitants. But I didn't really see any solution to that. One person's education is another person's brainwashing, after all. From a modern American view it's tempting to say that all city-states should allow their citizens free access to media so they can find out about other perspectives, but ultimately I decided this would be too much of an imposition on the freedom of the city-states. Letting citizens leave their city-state if they wish ultimately provides a way for any world citizen to find out what's what, although there are various strange cases to consider, such as a city-state that allows its citizens no information about the outside world, and also removes the citizenship of any citizen who goes outside its borders!

I thought the meta-society was a cool idea, and worked out a lot of details -- but ultimately I had no idea how to get it implemented, and not much desire to spend my life proselytizing for an eccentric political philosophy or government system, so I set the idea aside and focused my time on math, physics, AI and such.

As a major SF fan, it did occur to me that such a meta-society of city-states might be more easily achievable in future once space colonies were commonplace. If it were cheap to put up a small space colony for a few hundred or thousand or ten thousand people, then this could lead to a flowering of city-states of exactly the sort I was envisioning...

When I became aware of Patri Friedman's Seasteading movement, I immediately sensed a very similar line of thinking. Their mission is "To further the establishment and growth of permanent, autonomous ocean communities, enabling innovation with new political and social systems." Patri wants to make a meta-society and meta-economy on the high seas. And why not?

Design for an Optimal Society?

The new thought I had while driving the other day is: Maybe you could put my old idealistic meta-society-design together with the AIXI idea somehow, and come up with a design for a "society optimal under assumption of massive resources."

Suppose one assumes there's

a lot of great land (or sea + seasteading tech, or space + space colonization tech, whatever), so that fighting over land is irrelevant
a lot of people
a lot of natural resources, so that one city-state polluting another one's natural resources isn't an issue

Then it seems one could argue that my meta-society is near-optimal, under these conditions.

The basic proof would be: Suppose there were some social order X better than the meta-society. Then people could realize that X is better, and could simply design their city-states in such a way as to produce X.

For instance, if US-style capitalist democracy is better than the meta-society, and people realize it, then people can just construct their city-states to operate in the manner of US-style capitalist democracy (this would require close cooperation of multiple city-states, but that's quite feasible within the meta-society framework).

So, one could argue, any other social order can only be SLIGHTLY better than the meta-society... because if there's something significantly better, then after a little while the meta-society can come to emulate it closely.

So, under assumptions of sufficiently generous resources, the meta-society is about as good as anything.

Now there are certainly plenty of loopholes to be closed in turning this heuristic argument into a formal proof. But I hope the basic idea is clear.

As with AIXI, one can certainly question the relevance of this sort of design, since resource scarcity is a major fact of modern life. But recall that I originally started thinking about meta-societies outside the "unrealistically much resources" context.

Finally, you'll note that for simplicity, I have phrased the above discussion in terms of "people." But of course, the same sort of thinking applies for any kind of intelligent agent. The main assumption in this case is that the agents involved either have roughly equal power and intelligence, or else that if there are super-powerful agents involved, they have the will to obey the central government.

Can We Approach the Meta-Society as Technology Advances?

More and more resources are becoming available for humanity, as technology advances. Seasteading and space colonization and so forth decrease the scarcity of available "land" for human habitation. Mind uploading would do so more dramatically. Molecular nanotech (let alone femotech and so forth) may dramatically reduce material scarcity, at least on the scale interesting to humans.

So, it seems the conditions for the meta-society may be more and more closely met, as the next decades and centuries unfold.

Of course, the meta-society will remain an idealization, never precisely achievable in practice. But it may be we can approach it closer and closer as technology improves.

Marxism had the notion of society gradually becoming more and more pure, progressively approaching Perfect Communism. What I'm suggesting here is similar in form but different in content: society gradually becoming more and more like the meta-society, as scarcity of various sorts becomes less and less of an issue.

As I write about this now, it also occurs to me that this is a particularly American vision. America, in a sense, is a sort of meta-society -- the central government is relatively weak (compared to other First World countries) and there are many different subcultures, some operating with various sorts of autonomy (though also a lot of interconnectedness). In this sense, it seems I'm implicitly suggesting that America is a better model for the future than other existing nations. How very American of me!

If superhuman AI comes about (as I think it will), then the above arguments make sense only if the superhuman AI chooses to respect the meta-society social structure. The possibility even exists that a benevolent superhuman AI could serve itself as the central government of a meta-society.

And so it goes....

Making Minds from Memristors?

2010-11-23T21:36:00.003-05:00

Amara Angelica pointed me to an article in IEEE Spectrum titled MoNETA: A Mind Made from Memristors

Fascinating indeed!

I'm often skeptical of hardware projects hyped as AI projects, but truth be told, I find this one an extremely exciting and promising project.

I think the memristor technology is amazing and may well play part in the coming AGI revolution.

Creating emulations of human brain microarchitecture is one fascinating application of memristors, though not the only one and not necessarily the most exciting one. Memristors can also be used to make a lot of other different AI architectures, not closely modeled after the human brain.

[For instance, one could implement a semantic network or an OpenCog-style AtomSpace (weighted labeled hypergraph) via memristors, where each node in the network has both memory and processor resident in it ... this is a massively parallel network implemented via memristors, but the nodes in the network aren't anything like neurons...]

And, though the memristors-for-AGI theme excites me, this other part of the article leaves me a bit more skeptical:

"
By the middle of next year, our researchers will be working with thousands of candidate animats at once, all with slight variations in their brain architectures. Playing intelligent designers, we'll cull the best ones from the bunch and keep tweaking them until they unquestionably master tasks like the water maze and other, progressively harder experiments. We'll watch each of these simulated animats interacting with its environment and evolving like a natural organism. We expect to eventually find the "cocktail" of brain areas and connections that achieves autonomous intelligent behavior.
"

I think the stated research program places too much emphasis on brain microarchitecture and not enough on higher-level cognitive architecture. The idea that a good cognitive architecture is going to be gotten to emerge via some simple artificial-life type experiments seems very naive to me. I suspect that, even with the power of memristors, designing a workable cognitive architecture is going to be a significant enterprise. And I also think that many existing cognitive architectures, like my own OpenCog or Stan Franklin's LIDA or Hawkins' or Arel's deep learning architectures, could be implemented on a memristor fabric without changing their underlying concepts or high-level algorithms or dataflow.

So: memristors for AI, yay!

But: memristors as enablers of a simplistic Alife approach to AGI ... well, I don't think so.

The Psi Debate Continues (Goertzel on Wagenmakers et al on Bem on precognition)

2010-11-23T10:38:00.005-05:00

A few weeks ago I wrote an article for H+ Magazine about the exciting precognition results obtained by Daryl Bem at Cornell University.

Recently, some psi skeptics (Wagenmakers et al) have written a technical article disputing the validity of Bem's analyses of his data.

In this blog post I'll give my reaction to the Wagenmakers et al (WM from here on) paper.

It's a frustrating paper, because it makes some valid points -- yet it also confuses the matter by inappropriately accusing Bem of committing "fallacies" and by arguing that the authors' preconceptions against psi should be used to bias the data analysis.

The paper makes 3 key points, which I will quote in the form summarized here and then respond to one by one

POINT 1

"
Bem has published his own research methodology and encourages the formulation of hypotheses after data analysis. This form of post-hoc analysis makes it very difficult to determine accurate statistical significance. It also explains why Bem offers specific hypotheses that seem odd a priori, such as erotic images having a greater precognitive effect. Constructing hypotheses from the same data range used to test those hypotheses is a classic example of the Texas sharpshooter fallacy
"

MY RESPONSE

As WM note in their paper, this is actually how science is ordinarily done; Bem is just being honest and direct about it. Scientists typically run many exploratory experiments before finding the ones with results interesting enough to publish.

It's a meaningful point, and a reminder that science as typically practiced does not match some of the more naive notions of "scientific methodology". But it would also be impossibly cumbersome and expensive to follow the naive notion of scientific methodology and avoid exploratory work altogether, in psi or any other domain.

Ultimately this complaint against Bem's results is just another version of the "file drawer effect" hypothesis, which has been analyzed in great deal in the psi literature via meta-analyses across many experiments. The file drawer effect argument seems somewhat compelling when you look at a single experiment-set like Bem's, and becomes much less compelling when you look across the scope of all psi experiments reported, because the conclusion becomes that you'd need a huge number of carefully-run, unreported experiments to explain the total body of data.

BTW, the finding that erotic pictures give more precognitive response than other random pictures, doesn't seem terribly surprising, given the large role that sexuality plays in human psychology and evolution. If the finding were that pictures of cheese give more precognitive response than anything else, that would be more strange and surprising to me.

POINT 2

"
The paper uses the fallacy of the transposed conditional to make the case for psi powers. Essentially mixing up the difference between the probability of data given a hypothesis versus the probability of a hypothesis given data.
"

MY RESPONSE

This is a pretty silly criticism, much less worthy than the other points raised in the WM paper. Basically, when you read the discussion backing up this claim, the authors are saying that one should take into account the low a priori probability of psi in analyzing the data. OK, well ... one could just as well argue for taking to account the high a priori probability of psi given the results of prior meta-analyses or anecdotal reports of psi. Blehh.

Using the term "fallacy" here makes it seem, to people who just skim the WM paper or read only the abstract, as if Bem made some basic reasoning mistake. Yet when you actually read the WM paper, that is not what is being claimed. Rather they admit that he is following ordinary scientific methodology.

POINT 3

"
Wagenmakers' analysis of the data using a Bayesian t-test removes the significant effects claimed by Bem.
"

This is the most worthwhile point raised in the Wagenmakers et al paper.

Using a different sort of statistical test than Bem used, they re-analyze Bem's data and they find that, while the results are positive, they are not positive enough to pass the level of "statistical significance." They conclude that a somewhat larger sample size would be needed to conclude statistical significance using the test they used.

The question then becomes why to choose one statistical test over another. Indeed, it's common scientific practice to choose a statistical test that makes one's results appear significant, rather than others that do not. This is not peculiar to psi research, it's simply how science is typically done.

Near the end of their paper, WM point out that Bem's methodology is quite typical of scientific psychology research, and in fact more rigorous than most psychology papers published in good journals. What they don't note, but could have is that the same sort of methodology is used in pretty much every area of science.

They then make a series of suggestions regarding how psi research should be conducted, which would indeed increase the rigor of the research, but which a) are not followed in any branch of science, and b) would make psi research sufficiently cumbersome and expensive as to be almost impossible to conduct.

I didn't dig into the statistics deeply enough to assess the appropriateness of the particular test that WM applied (leading to their conclusion that Bem's results don't show statistical significance, for most of his experiments).

However, I am quite sure that if one applied this same Bayesian t-test to a meta-analysis over the large body of published psi experiments, one would get highly significant results. But then WM would likely raise other issues with the meta-analysis (e.g. the file drawer effect again).

Conclusion

I'll be curious to see the next part of the discussion, in which a psi-friendly statistician like Jessica Utts (or a statistician with no bias on the matter, but unbiased individuals seem very hard to come by where psi is concerned) discusses the appropriateness of WM's re-analysis of the data.

But until that, let's be clear on what WM have done. Basically, they've

raised the tired old, oft-refuted spectre of the file drawer effect, using a different verbiage from usual
argued that one should analyze psi data using an a priori bias against it (and accused Bem of "fallacious" reasoning for not doing so)
pointed out that if one uses a different statistical test than Bem did [though not questioning the validity of the statistical test Bem did use], one finds that his results, while positive, fall below the standard of statistical significance in most of his experiments

The practical consequence of their latter point is that, if Bem's same experiments were done again with the same sort of results as obtained so far, then eventually a sufficient sample size would be accumulated to demonstrate significance according to WM's suggested test.

So when you peel away the rhetoric, what the WM critique really comes down to is: "Yes, his results look positive, but to pass the stricter statistical tests we suggest, one would need a larger sample size."

Of course, there is plenty of arbitrariness in our conventional criteria of significance anyway -- why do we like .05 so much, instead of .03 or .07?

So I really don't see too much meat in WM's criticism. Everyone wants to see replications of the experiments anyway, and no real invalidity in Bem's experiments, results or analyses was demonstrated.... The point made is merely that a stricter measure of significance would render these results (and an awful lot of other scientific results) insignificant until replication on a larger sample size was demonstrated. Which is an OK point -- but I'm still sorta curious to see a more careful, less obviously biased analysis of which is the best significance test to use in this case.

The Turing Church, Religion 2.0, and the Mystery of Consciousness

2010-11-21T11:35:00.006-05:00

It was my pleasure to briefly participate in Giulio Prisco's Turing Church Online Workshop 1, on Saturday November 20 2010 in Teleplace -- a wonderfully wacky and wide-ranging exploration of transhumanist spirituality and “Religion 2.0.″

The video proceedings are here.

I didn't participate in the whole workshop since it was a busy day for me, I just logged on briefly to give a talk and answer some questions. But I found the theme quite fascinating.

Giulio said I should assume the participants were already basically familiar with my thinking on transhumanist spirituality as expressed in my little book A Cosmist Manifesto that I wrote earlier this year, and he asked me to venture in some slightly different direction. I'm not sure I fulfilled that request all that well, but anyway, I'll paste here the notes I wrote as a basis for my talk in the workshop. I didn't read these notes with any precision, so if you want to know what I actually said you'll have to watch the video; but the talk was a more informal improvisation on the same basic theme...

"The relation between transhumanism and spirituality is a big topic, which I've thought about a lot -- right now I'll just make a few short comments. Sorry that I won't be able to stick around for this whole meeting today, I have some family stuff I need to do, but I'm happy to be able to participate at least briefly by saying a few remarks.

"Earlier this year I wrote a book touching on some of these comments, called "A Cosmist Manifesto" -- I'm not going to reiterate all that material now, just touch on a few key points.

"The individual human mind has a tendency to tie itself in what the psychologist Stanislaw Grof calls "knots" -- intricate webs of self-contradiction and fear, that cause emotional pain and cognitive confusion and serve as traps for mental energy. Ultimately these knots are largely rooted in the human self's fear of losing itself --- the self's fear of realizing that it lacks fundamental reality, and is basically a construct whose main goals are to keep the body going and reproducing and to preserve itself. These are some complicated words for describing something pretty basic, but I guess we all know what I'm talking about.

"And then there are the social knots, going beyond the individual ones… the knots we tie each other up in…

"These knots are serious problems for all of us -- and they're an even more serious problem when you think about the potential consequences of advanced technology in the next decade. We're on the verge of creating superhuman AI and molecular nanotech and brain-computer interfacing and so forth -- but we're still pretty much fucked up with psychological and social confusions! As Freud pointed out in Civilization and its Discontents, we're largely operating with motivational systems evolved for being hunter-gatherers in the African savannah, but the world we're creating for ourselves is dramatically different from that.

"Human society has come up with a bunch of different ways to get past these knots.

"One of them is religion -- which opens a doorway to transpersonal experience, going beyond self and society, opening things up to a broader domain of perceiving, being, understanding and acting. If you're not familiar with more philosophical side of the traditional religions you should look at Aldous Huxley's classic book "The Perennial Philosophy" -- it was really an eye-opener for me.

"Another method for getting past the knots is science. By focusing on empirical data, collectively perceived and understood, science lets us go beyond our preconceptions and emotions and biases and ideas. Science, with its focus on data and collective rational understanding, provides a powerful engine for growth of understanding. There's a saying that "science advances one funeral at a time" -- i.e. old scientific ideas only die when their proponents die. But the remarkable thing is, this isn't entirely true. Science has an amazing capability to push people to give up their closely held ideas, when these ideas don't mesh well with the evidence.

"What I see in the transhumanism-meets-spirituality connection is the possibility of somehow bringing together these two great ways of getting beyond the knots. If science and spirituality can come together somehow, we may have a much more powerful way of getting past the individual and social knots that bind us. If we could somehow combine the rigorous data focus of science with the personal and collective mind-purification of spiritual traditions, then we'd have something pretty new and pretty interesting -- and maybe something that could help us grapple with the complex issues modern technology is going to bring us in the next few decades

"One specific area of science that seems very relevant to these considerations is consciousness studies. Science is having a hard time grappling with consciousness, though it's discovering a lot about neural and cognitive correlates of consciousness. Spiritual traditions have discovered a lot about consciousness, though a lot of this knowledge is expressed in language that's hard for modern people to deal with. I wonder if some kind of science plus spirituality hybrid could provide a new way for groups of people to understand consciousness, combining scientific data and spiritual understanding.

"One idea I mentioned in the Cosmist Manifesto book is some sort of "Confederation of Cosmists", and Giulio asked me to say a little bit about that here. The core idea is obvious -- some kind of social group of individuals interested in both advanced technology and its implications, and personal growth and mind-expansion. The specific manifestation of the idea isn't too clear. But I wonder if one useful approach might be to focus on the cross-disciplinary understanding of consciousness -- using science and spirituality, and also advanced technologies like neuroscience and BCI and AGI. My thinking is that consciousness studies is one concrete area that truly seems to demand some kind of fusion of scientific and spiritual ideas … so maybe focusing on that in a truly broad, cross-tradition, Cosmist way could help us come together more and over help us work together to overcome our various personal and collective knots, and build a better future, and all that good stuff….

"Anyway there are just some preliminary thoughts, these are things I'm thinking about a lot these days, and I look forward to sharing my ideas more with you as my thoughts develop -- and I'll be catching the rest of this conference via the video recordings later on."

Fun stuff to think about -- though I don't have too much time for it these days, as my AGI and bioinformatics work seems to be taking all my time. But at some future point, I really do think the cross-disciplinary introspective/scientific individual/collective investigation of consciousness is well worth devoting attention to, and is going to bear some pretty fascinating fruit....

The Singularity Institute's Scary Idea (and Why I Don't Buy It)

2010-10-29T17:41:00.020-04:00

I recently wrote a blog post about my own AI project, but it attracted a bunch of adversarial comments from folks influenced by the Singularity Institute for AI's (rather different) perspective on the best approach to AI R&D. I responded to some of these comments there.

(Quick note for those who don't know: the Singularity Institute for AI is not affiliated with Singularity University, though there are some overlaps ... Ray Kurzweil is an Advisor to the former and the founder of the latter; and I am an Advisor to both.)

Following that discussion, a bunch of people have emailed me in the last couple weeks asking me to write something clearly and specifically addressing my views on SIAI's perspective on the future of AI. I don't want to spend a lot of time on this but I decided to bow to popular demand and write a blog post...

Of course, there are a lot of perspectives in the world that I don't agree with, and I don't intend to write blog posts explaining the reasons for my disagreement with all of them! But since I've had some involvement with SIAI in the past, I guess it's sort of a special case.

First of all I want to clarify I'm not in disagreement with the existence of SIAI as an institution, nor with the majority of their activities -- only with certain positions habitually held by some SIAI researchers, and by the community of individuals heavily involved with SIAI. And specifically with a particular line of thinking that I'll refer to here as "SIAI's Scary Idea."

Roughly, the Scary Idea posits that: If I or anybody else actively trying to build advanced AGI succeeds, we're highly likely to cause an involuntary end to the human race.

Brief Digression: My History with SIAI

Before getting started with the meat of the post, I'll give a few more personal comments, to fill in some history for those readers who don't know it, or who know only parts. Readers who are easily bored may wish to skip to the next section,

SIAI has been quite good to me, overall. I've enjoyed all the Singularity Summits, which they've hosted, very much; I think they've played a major role in the advancement of society's thinking about the future, and I've felt privileged to speak at them. And I applaud SIAI for consistently being open to Summit speakers whose views are strongly divergent from those commonly held in the SIAI community.

Also, in 2008, SIAI and my company Novamente LLC seed-funded the OpenCog open-source AGI project (based on software code spun out from Novamente). The SIAI/OpenCog relationship diminished substantially when Tyler Emerson passed the leadership of SIAI along to Michael Vassar, but it was instrumental in getting OpenCog off the ground. I've also enjoyed working with Michael Vassar on the Board of Humanity+, of which I'm Chair and he's a Board member.

When SIAI was helping fund OpenCog, I took the title of "Director of Research" of SIAI, but I never actually directed any research there apart from OpenCog. The other SIAI research was always directed by others, which was fine with me. There were occasional discussions about operating in a more unified manner, but it didn't happen. All this is perfectly ordinary in a small start-up type organization.

Once SIAI decided OpenCog was no longer within its focus, after a bit of delay I decided it didn't make sense for me to hold the Director of Research title anymore, since as things were evolving, I wasn't directing any SIAI research. I remain as an Advisor to SIAI, which is going great.

Now, on to the meat of the post….

SIAI's Scary Idea (Which I Don't Agree With)

SIAI's leaders and community members have a lot of beliefs and opinions, many of which I share and many not, but the key difference between our perspectives lies in what I'll call SIAI's "Scary Idea", which is the idea that: progressing toward advanced AGI without a design for "provably non-dangerous AGI" (or something closely analogous, often called "Friendly AI" in SIAI lingo) is highly likely to lead to an involuntary end for the human race.

(SIAI's Scary Idea has been worded in many different ways by many different people, and I tried in the above paragraph to word it in a way that captures the idea fairly if approximatively, and won't piss off too many people.)

Of course it's rarely clarified what "provably" really means. A mathematical proof can only be applied to the real world in the context of some assumptions, so maybe "provably non-dangerous AGI" means "an AGI whose safety is implied by mathematical arguments together with assumptions that are believed reasonable by some responsible party"? (where the responsible party is perhaps "the overwhelming majority of scientists" … or SIAI itself?)….. I'll say a little more about this a bit below.

Please note that, although I don't agree with the Scary Idea, I do agree that the development of advanced AGI has significant risks associated with it. There are also dramatic potential benefits associated with it, including the potential of protection against risks from other technologies (like nanotech, biotech, narrow AI, etc.). So the development of AGI has difficult cost-benefit balances associated with it -- just like the development of many other technologies.

I also agree with Nick Bostrom and a host of SF writers and many others that AGI is a potential "existential risk" -- i.e. that in the worst case, AGI could wipe out humanity entirely. I think nanotech and biotech and narrow AI could also do so, along with a bunch of other things.

I certainly don't want to see the human race wiped out! I personally would like to transcend the legacy human condition and become a transhuman superbeing … and I would like everyone else to have the chance to do so, if they want to. But even though I think this kind of transcendence will be possible, and will be desirable to many, I wouldn't like to see anyone forced to transcend in this way. I would like to see the good old fashioned human race continue, if there are humans who want to maintain their good old fashioned humanity, even if other options are available

But SIAI's Scary Idea goes way beyond the mere statement that there are risks as well as benefits associated with advanced AGI, and that AGI is a potential existential risk.

Finally, I note that most of the other knowledgeable futurist scientists and philosophers, who have come into close contact with SIAI's perspective, also don't accept the Scary Idea. Examples include Robin Hanson, Nick Bostrom and Ray Kurzweil.

There's nothing wrong with having radical ideas that one's respected peers mostly don't accept. I totally get that: My own approach to AGI is somewhat radical, and most of my friends in the AGI research community, while they respect my work and see its potential, aren't quite as enthused about it as I am. Radical positive changes are often brought about by people who clearly understand certain radical ideas well before anyone else "sees the light." However, my own radical ideas are not telling whole research fields that if they succeed they're bound to kill everybody ... so it's a somewhat different situation.

What is the Argument for the Scary Idea?

Although an intense interest in rationalism is one of the hallmarks of the SIAI community, still I have not yet seen a clear logical argument for the Scary Idea laid out anywhere. (If I'm wrong, please send me the link, and I'll revise this post accordingly. Be aware that I've already at least skimmed everything Eliezer Yudkowsky has written on related topics.)

So if one wants a clear argument for the Scary Idea, one basically has to construct it oneself.

As far as I can tell from discussions and the available online material, some main ingredients of peoples' reasons for believing the Scary Idea are ideas like:

If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans (as opposed to, e.g., utterly ignoring us, and being willing to repurpose our molecules for its own ends) are very low
Human value is fragile as well as complex, so if you create an AGI with a roughly-human-like value system, then this may not be good enough, and it is likely to rapidly diverge into something with little or no respect for human values
"Hard takeoffs" (in which AGIs recursively self-improve and massively increase their intelligence) are fairly likely once AGI reaches a certain level of intelligence; and humans will have little hope of stopping these events
A hard takeoff, unless it starts from an AGI designed in a "provably Friendly" way, is highly likely to lead to an AGI system that doesn't respect the rights of humans to exist

I emphasize that I am not quoting any particular thinker associated with SIAI here. I'm merely summarizing, in my own words, ideas that I've heard and read very often from various individuals associated with SIAI.

If you put the above points all together, you come up with a heuristic argument for the Scary Idea. Roughly, the argument goes something like: If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.

The line of argument makes sense, if you accept the premises.

But, I don't.

I think the first of the above points is reasonably plausible, though I'm not by any means convinced. I think the relation between breadth of intelligence and depth of empathy is a subtle issue which none of us fully understands (yet). It's possible that with sufficient real-world intelligence tends to come a sense of connectedness with the universe that militates against squashing other sentiences. But I'm not terribly certain of this, any more than I'm terribly certain of its opposite.

I agree much less with the final three points listed above. And I haven't seen any careful logical arguments for these points.

I doubt human value is particularly fragile. Human value has evolved and morphed over time and will continue to do so. It already takes multiple different forms. It will likely evolve in future in coordination with AGI and other technology. I think it's fairly robust.

I think a hard takeoff is possible, though I don't know how to estimate the odds of one occurring with any high confidence. I think it's very unlikely to occur until we have an AGI system that has very obviously demonstrated general intelligence at the level of a highly intelligent human. And I think the path to this "hard takeoff enabling" level of general intelligence is going to be somewhat gradual, not extremely sudden.

I don't have any strong sense of the probability of a hard takeoff, from an apparently but not provably human-friendly AGI, leading to an outcome likable to humans. I suspect this probability depends on many features of the AGI, which we will identify over the next years & decades via theorizing based on the results of experimentation with early-stage AGIs.

Yes, you may argue: the Scary Idea hasn't been rigorously shown to be true… but what if it IS true?

OK but ... pointing out that something scary is possible, is a very different thing from having an argument that it's likely.

The Scary Idea is certainly something to keep in mind, but there are also many other risks to keep in mind, some much more definite and palpable. Personally, I'm a lot more worried about nasty humans taking early-stage AGIs and using them for massive destruction, than about speculative risks associated with little-understood events like hard takeoffs.

Is Provably Safe or "Friendly" AGI A Feasible Idea?

The Scary Idea posits that if someone creates advanced AGI that isn't somehow provably safe, it's almost sure to kill us all.

But not only am I unconvinced of this, I'm also quite unconvinced that "provably safe" AGI is even feasible.

The idea of provably safe AGI is typically presented as something that would exist within mathematical computation theory or some variant thereof. So that's one obvious limitation of the idea: mathematical computers don't exist in the real world, and real-world physical computers must be interpreted in terms of the laws of physics, and humans' best understanding of the "laws" of physics seems to radically change from time to time. So even if there were a design for provably safe real-world AGI, based on current physics, the relevance of the proof might go out the window when physics next gets revised.

Also, there are always possibilities like: the alien race that is watching us and waiting for us to achieve an IQ of 333, at which point it will swoop down upon us and eat us, or merge with us. We can't rule this out via any formal proof, and we can't meaningfully estimate the odds of it either. Yes, this sounds science-fictional and outlandish; but is it really more outlandish and speculative than the Scary Idea?

A possibility that strikes me as highly likely is that, once we have created advanced AGI and have linked our brains with it collectively, most of our old legacy human ideas (including physical law, aliens, and Friendly AI) will seem extremely limited and ridiculous.

Another issue is that the goal of "Friendliness to humans" or "safety" or whatever you want to call it, is rather nebulous and difficult to pin down. Science fiction has explored this theme extensively. So even if we could prove something about "smart AGI systems with a certain architecture that are guaranteed to achieve goal G," it might be infeasible to apply this to make AGI systems that are safe in the real-world -- simply because we don't know how to boil down the everyday intuitive notions of "safety" or "Friendliness" into a mathematically precise goal G like the proof refers to.

This is related to the point Eliezer Yudkowsky makes that "value is complex" -- actually, human value is not only complex, it's nebulous and fuzzy and ever-shifting, and humans largely grok it by implicit procedural, empathic and episodic knowledge rather than explicit declarative or linguistic knowledge. Transmitting human values to an AGI is likely to be best done via interacting with the AGI in real life, but this is not the sort of process that readily lends itself to guarantees or formalization.

Eliezer has suggested a speculative way of getting human values into AGI systems called Coherent Extrapolated Volition, but I think this is a very science-fictional and incredibly infeasible idea (though a great SF notion). I've discussed it and proposed some possibly more realistic alternatives in a previous blog post (e.g. a notion called Coherent Aggregated Volition). But my proposed alternatives aren't guaranteed-to-succeed nor neatly formalized.

But setting those worries aside, is the computation-theoretic version of provably safe AI even possible? Could one design an AGI system and prove in advance that, given certain reasonable assumptions about physics and its environment, it would never veer too far from its initial goal (e.g. a formalized version of the goal of treating humans safely, or whatever)?

I very much doubt one can do so, except via designing a fictitious AGI that can't really be implemented because it uses infeasibly much computational resources. My GOLEM design, sketched in this article, seems to me a possible path to a provably safe AGI -- but it's too computationally wasteful to be practically feasible.

I strongly suspect that to achieve high levels of general intelligence using realistically limited computational resources, one is going to need to build systems with a nontrivial degree of fundamental unpredictability to them. This is what neuroscience suggests, it's what my concrete AGI design work suggests, and it's what my theoretical work on GOLEM and related ideas suggests. And none of the public output of SIAI researchers or enthusiasts has given me any reason to believe otherwise, yet.

Practical Implications

The above discussion of SIAI's Scary Idea may just sound like fun science-fictional speculation -- but the reason I'm writing this blog post is that when I posted a recent blog post about my current AGI project, the comments field got swamped with SIAI-influenced people saying stuff in the vein of: Creating an AGI without a proof of Friendliness is essentially equivalent to killing all people! So I really hope your OpenCog work fails, so you don't kill everybody!!!

(One amusing/alarming quote from a commentator (probably not someone directly affiliated with SIAI) was "if you go ahead with an AGI when you're not 100% sure that it's safe, you're committing the Holocaust." But it wasn't just one extreme commentator, it was a bunch … and then a bunch of others commenting to me privately via email.)

If one fully accepts SIAI's Scary Idea, then one should not work on practical AGI projects, nor should one publish papers on the theory of how to build AGI systems. Instead, one should spend one's time trying to figure out an AGI design that is somehow provable-in-advance to be a Good Guy. For this reason, SIAI's research group is not currently trying to do any practical AGI work.

Actually, so far as I know, my "GOLEM" AGI design (mentioned above) is closer to a "provably Friendly AI" than anything the SIAI research team has come up with. At least, it's closer than anything they have made public.

However GOLEM is not something that could be practically implemented in the near future. It's horribly computationally inefficient, compared to a real-world AGI design like the OpenCog system I'm now working on (with many others -- actually I'm doing very little programming these days, so happily the project is moving forward with the help of others on the software design and coding side, while I contribute at the algorithm, math, design, theory, management and fundraising levels).

I agree that AGI ethics is a Very Important Problem. But I doubt the problem is most effectively addressed by theory alone. I think the way to come to a useful real-world understanding of AGI ethics is going to be to

build some early-stage AGI systems, e.g. artificial toddlers, scientists' helpers, video game characters, robot maids and butlers, etc.
study these early-stage AGI systems empirically, with a focus on their ethics as well as their cognition
in the usual manner of science, attempt to arrive at a solid theory of AGI intelligence and ethics based on a combination of conceptual and experimental-data considerations
humanity collectively plots the next steps from there, based on the theory we find: maybe we go ahead and create a superhuman AI capable of hard takeoff, maybe we pause AGI development because of the risks, maybe we build an "AGI Nanny" to watch over the human race and prevent AGI or other technologies from going awry. Whatever choice we make then, it will be made based on far better knowledge than we have right now.

So what's wrong with this approach?

Nothing, really -- if you hold the views of most AI researchers or futurists. There are plenty of disagreements about the right path to AGI, but wide and implicit agreement that something like the above path is sensible.

But, if you adhere to SIAI's Scary Idea, there's a big problem with this approach -- because, according to the Scary Idea, there's too huge of a risk that these early-stage AGI systems are going to experience a hard takeoff and self-modify into something that will destroy us all.

But I just don't buy the Scary Idea.

I do see a real risk that, if we proceed in the manner I'm advocating, some nasty people will take the early-stage AGIs and either use them for bad ends, or proceed to hastily create a superhuman AGI that then does bad things of its own volition. These are real risks that must be thought about hard, and protected against as necessary. But they are different from the Scary Idea. And they are not so different from the risks implicit in a host of other advanced technologies.

Conclusion

So, there we go.

I think SIAI is performing a useful service by helping bring these sorts of ideas to the attention of the futurist community (alongside the other services they're performing, like the wonderful Singularity Summits). But, that said, I think the Scary Idea is potentially a harmful one. At least, it WOULD be a harmful one, if more people believed it; so I'm glad it's currently restricted to a rather small subset of the futurist community.

Many people die each day, and many others are miserable for various reasons -- and all sorts of other advanced and potentially dangerous technologies are currently under active development. My own view is that unaided human minds may well be unable to deal with the complexity and risk of the world that human technology is unleashing. I actually suspect that our best hope for survival and growth through the 21st century is to create advanced AGIs to help us on our way -- to cure disease, to develop nanotech and better AGI and invent new technologies; and to help us keep nasty people from doing destructive things with advanced technology.

I think that to avoid actively developing AGI, out of speculative concerns like the Scary Idea, would be an extremely bad idea.

That is, rather than "if you go ahead with an AGI when you're not 100% sure that it's safe, you're committing the Holocaust," I suppose my view is closer to "if you avoid creating beneficial AGI because of speculative concerns, then you're killing my grandma" !! (Because advanced AGI will surely be able to help us cure human diseases and vastly extend and improve human life.)

So perhaps I could adopt the slogan: "You don't have to kill my grandma to avoid the Holocaust!" … but really, folks… Well, you get the point….

Humanity is on a risky course altogether, but no matter what I decide to do with my life and career (and no matter what Bill Joy or Jaron Lanier or Bill McKibben, etc., write), the race is not going to voluntarily halt technological progress. It's just not happening.

We just need to accept the risk, embrace the thrill of the amazing time we were born into, and try our best to develop near-inevitable technologies like AGI in a responsible and ethical way.

And to me, responsible AGI development doesn't mean fixating on speculative possible dangers and halting development until ill-defined, likely-unsolvable theoretical/philosophical issues are worked out to everybody's (or some elite group's) satisfaction.

Rather, it means proceeding with the work carefully and openly, learning what we can as we move along -- and letting experiment and theory grow together ... as they have been doing quite successfully for the last few centuries, at a fantastically accelerating pace.

And so it goes.

Let's Turn Nauru Into Transtopia

2010-10-13T00:14:00.006-04:00

Here's an off-the-wall idea that has some appeal to me ... as a long-time Transtopian fantasist and world traveler....

The desert island nation of Nauru needs money badly, and has a population of less than 15,000

There are problems with water supply, but they could surely be solved with some technical ingenuity.

The land area is about 8 square miles. But it could be expanded! Surely it's easier to extend an island with concrete platforms or anchored floating platforms of some other kind, than to seastead in the open ocean.

The country is a democracy. Currently it may not be possible to immigrate there except as a temporary tourist or business visitor. But I'd bet this could be made negotiable.

Suppose 15,000 adult transhumanists (along with some kids, one would assume) decided to emigrate to Nauru en masse over a 5-year period, on condition they could obtain full citizenship. Perhaps this could be negotiated with the Nauruan government.

Then after 5 years we would have a democracy in which transhumanists were the majority.

Isn't this the easiest way to create a transhumanist nation? With all the amazing future possibilities that that implies?

This would genuinely be of benefit to the residents of Nauru, which now has 90% unemployment. Unemployment would be reduced close to zero, and the economy would be tremendously enlarged. A win-win situation. Transhumanists would get freedom, and Nauruans would get a first-world economy.

Considerable infrastructure would need to be built. A deal would need to be struck with the government, in which, roughly,

They agreed to allow a certain number of outsiders citizenship, and to allow certain infrastructure development
Over a couple years, suitable infrastructure was built to supply electrical power, Internet, more frequent flights, etc.
Then, over a few years after that, the new population would flow in

This much emigration would make Nauru crowded, but not nearly as crowded as some cities. And with a seasteading mindset, it's easy to see that the island is expandable.

To ensure employment of the relocated transhumanists, we would need to get a number of companies to agree to open Nauru offices. But this would likely be tractable, given the preference of firms to have offices in major tech centers. Living expenses in Nauru would be much lower than in, say, Silicon Valley, so expenses would be lower.

Tourism could become a major income stream, given the high density of interesting people which would make Nauru into a cultural mecca. Currently there is only one small beach on Nauru (which is said to be somewhat dirty), but creation of a beautiful artificial beach on the real ocean is not a huge technological feat.

It would also be a great place to experiment with aquaculture and vertical farming.

What say you? Let's do it!

P.S.

Other candidates for the tropical island Transtopia besides Nauru would be Tuvalu and Kiribati; but Kiribati's population is much larger, and Tuvalu is spread among many islands, and is also about to become underwater due to global warming. So Nauru would seem the number one option. Though, Tuvalu could be an interesting possibility also, especially if we offered to keep the island above water by building concrete platforms or some such (a big undertaking, but much easier than seasteading). This would obviously be a major selling point to the government.

What Would It Take to Move Rapidly Toward Beneficial Human-Level AGI?

2010-10-10T09:11:00.006-04:00

On Thursday I finished writing the last chapter of my (co-authored) two-volume book on how to create beneficial human-level AGI, Building Better Minds. I have a bunch of editing still do so, some references to add, etc. -- but the book is now basically done. Woo hoo!

The book should be published by a major scientific publisher sometime in 2011.

The last chapter describes, in moderate detail, how the CogPrime cognitive architecture (implemented in the OpenCog open-source framework) would enable a robotic or virtual embodied system to appropriately respond to the instruction "Build me something surprising out of blocks." This is in the spirit of the overall idea: Build an AGI toddler first, then teach it, study it, and use it as a platform to go further.

From an AGI toddler, I believe, one could go forward in a number of directions: toward fairly human-like AGIs, but also toward different sorts of minds formed by hybridizing the toddler with narrow-AI systems carrying out particular classes of tasks in dramatically transhuman ways.

Reading through the 900-page tome my colleagues and I have put together, I can't help reflecting on how much work is left to bring it all into reality! We have a software framework that is capable of supporting the project (OpenCog), and we have a team of people capable of doing it (people working with me on OpenCog now; people working with me on other projects now; people I used to work with but who moved on to other things, but would enthusiastically come back for a well-funded AGI project). We have a rich ecosystem of others (e.g. academic and industry AI researchers, as well as neuroscientists, philosophers, technologists, etc. etc.) who are enthusiastic to provide detailed, thoughtful advice as we proceed.

What we don't have is proper funding to implement the stuff in the book and create the virtual toddler!

This is of course a bit frustrating: I sincerely believe I have a recipe for creating a human-level thinking machine! In an ethical way, and with computing resources currently at our disposal.

But implementing this recipe would be a lot of work, involving a number of people working together in a concentrated and coordinated way over a significant period of time.

I realize I could be wrong, or I could be deluding myself. But I've become a lot more self-aware and a lot more rational through my years of adult life (I'm 43 now), and I really don't think so. I've certainly introspected and self-analyzed a lot to understand the extent to which I may be engaged in wishful thinking about AGI, and my overall conclusion (in brief) is as follows: Estimating timing is hard, for any software project, let alone one involving difficult research. And there are multiple PhD-thesis-level research problems that need to be solved in the midst of getting the whole CogPrime design to work (but by this point in my career, I believe I have a decent intuition for distinguishing tractable PhD-thesis-level research problems from intractable conundrums). And there's always the possibility of the universe being way, way different than any of us understands, in some way that stops any AGI design based on digital computers (or any current science!) from working. But all in all, evaluated objectively according to my professional knowledge, the whole CogPrime design appears sensible -- if all the parts work vaguely as expected, the whole system should lead to human-level AGI; and according to current computer science and narrow AI theory and practice, all the parts are very likely to work roughly as expected.

So: I have enough humility and breadth to realize I could be wrong, but I have studied pretty much all the relevant knowledge that's available, I've thought about this hard for a very long time and talked to a large percentage of the world's (other) experts; I'm not a fool and I'm not self-deluded in some shallow and obvious way. And I really believe this design can work!

It's the same design I've been refining since about 1996. The prototyping my colleagues and I did at Webmind Inc. (when we had a 45-person AGI research team) in 1998-2001 was valuable, both for what it taught us about what NOT to do and for positive lessons. The implementation work my colleagues at Novamente LLC and the OpenCog project have done since 2001 has been very valuable too; and it's led to an implementation of maybe 40% of the CogPrime design (depending on how you measure it). (But unfortunately 40% of a brain doesn't yield 40% of the functionality of a whole brain, particularly because in this case (beyond the core infrastructure) the 40% implemented has been largely chosen by what was useful for Novamente LLC application projects rather than what we thought would serve best as the platform for AGI.) Having so many years to think through the design, without a large implementation team to manage, has been frustrating but also good in a sense, in that it's given me and my colleagues time and space to repeatedly mull over the design and optimize it in various ways.

Now, the funding situation for the project is not totally dismal, or it least it doesn't seem so right now. For that I am grateful.

The OpenCog project does appear to be funded, at least minimally, for the next couple years. This isn't quite 100% certain, but it's close -- it seems we've lined up funding for a handful of people to work full-time on a fairly AGI-ish OpenCog application for 2 years (I'll post here about this at length once it's definite). And there's also the Xiamen University "Brain-Like Intelligent Systems" lab, in which some grad students are applying OpenCog to enable some intelligent robotic behaviors. And Novamente LLC is still able to fund a small amount of OpenCog work, via application projects that entail making some improvements to the OpenCog infrastructure along the way. So all in all, it seems, we'll probably continue making progress, which is great.

But I'm often asked, by various AGI enthusiasts, what it would take to make really fast progress toward my AGI research goals. What kind of set-up, what kind of money? Would it take a full-on "AGI Manhattan Project" -- or something smaller?

In the rest of this blog post I'm going to spell it out. The answer hasn't changed much for the last 5 years, and most likely won't change a lot during the next 5 (though I can't guarantee that).

What I'm going to describe is the minimal team required to make reasonably fast progress. Probably we could progress even faster if we had massively more funding, but I'm trying to be realistic here.

We could use a team of around 10 of the right people (mostly, great AI programmers, with a combination of theory understanding and implementation chops), working full-time on AI development.

We could use around 5 great programmers working on the infrastructure -- to get OpenCog working really efficiently on a network of distributed multi-processor machines.

If we're going to do robotics, we could use a dedicated robotics team of perhaps 5 people.

If we're going to do virtual agents, we could use 5 people working on building out the virtual world appropriately for AGI.

Add a system administrator, 2 software testers, a project manager to help us keep track of everything, and a Minister of Information to help us keep all the documentation in order.

That's 30 people. Then add me and my long-time partner Cassio Pennachin to coordinate the whole thing (and contribute to the technical work as needed), and a business manager to help with money and deal with the outside world. 33 people.

Now let's assume this is done in the US (not the only possibility, but the simplest one to consider), and let's assume we pay people close to market salaries and benefits, so that their spouses don't get mad at them and decrease their productivity (yes, it's really not optimal to do a project like this with programmers fresh out of college -- this isn't a Web 2.0 startup, it's a massively complex distributed software system based on integration of multiple research disciplines. Many of the people with the needed expertise have spouses, families, homes, etc. that are important to them). Let's assume it's not done in Silicon Valley or somewhere else where salaries are inflated, but in some other city with a reasonable tech infrastructure and lower housing costs. Then maybe, including all overheads, we're talking about $130K/year per employee (recall that we're trying to hire the best people here; some are very experienced and some just a few years out of college, but this is an average).

Salary cost comes out to $4.3M/year, at this rate.

Adding in a powerful arsenal of hardware and a nice office, we can round up to $5M/year

Let's assume the project runs for 5 years. My bet is we can get an AGI toddler by that time. But even if that's wrong, I'm damn sure we could make amazing progress by that time, suitable to convince a large number of possible funding sources to continue funding the project at the same or a greater level.

Maybe we can do it in 3 years, maybe it would take 7-8 years to get to the AGI toddler goal -- but even if it's the latter, we'd have amazing, clearly observable dramatic progress in 3-5 years.

So, $25M total.

There you go. That's what it would cost to progress toward human-level AGI, using the CogPrime design, in a no-BS straightforward way -- without any fat in the project, but also without cutting corners in ways that reduce efficiency.

If we relax the assumption that the work is done in the US and move to a less expensive place (say, Brazil or China where OpenCog already has some people working) we can probably cut thc cost by half without a big problem. We would lose some staff who wouldn't leave the US, so there would be a modest decrease in productivity, but it wouldn't kill the project. (Why does it only cut the cost by half? Because if we're importing first-worlders to the Third World to save money, we still need to pay them enough to cover expenses they may have back in the US, to fly home to see their families, etc.)

So, outside the US, $13M total over 5 years.

Or if we want to rely more on non-US people for some of the roles (e.g. systems programming, virtual worlds,...), it can probably be reduced to $10M total over 5 years, $2M/year.

If some wealthy individual or institution were willing to put in $10M -- or $25M if they're fixated on a US location (or, say, $35M if they're fixated on Silicon Valley) -- then we could progress basically full-speed-ahead toward creating beneficial human-level AGI.

Instead, we're progressing toward the same goal seriously and persistently, but much more slowly and erratically.

I have spoken personally to a decent number of individuals with this kind of money at their disposal, and many of them are respectful of and interested in the OpenCog project -- and would be willing to put in this kind of money if they had sufficient confidence the project would succeed.

But how to give potential funders this sort of confidence?

After all, when they go to the AI expert at their local university, the guy is more likely than not to tell them that human-level AI is centuries off. Or if they open up The Singularity is Near, by Ray Kurzweil who is often considered a radical techno-optimist, they see a date of 2029 for human-level AGI -- which means that as investors they would probably start worrying about it around 2025.

A 900-page book is too much to expect a potential donor or investor to read; and even if they read it (once its published), it doesn't give an iron-clad irrefutable argument that the project will succeed, "just" a careful overall qualitative argument together with detailed formal treatments of various components of the design.

The various brief conference papers I've published on the CogPrime design and OpenCog project, give a sense of the overall spirit but don't tell you enough to let you make a serious evaluation. Maybe this is a deficiency in the writing, but I suspect it's mainly a consequence of the nature of the subject matter.

The tentative conclusion that I've come to is that, barring some happy luck, we will need to come up with some amazing demo of AGI functionality -- something that will serve as an "AGI Sputnik" moment.

Sputnik, of course, caused the world to take space flight seriously. The right AGI demo could do the same. It could get OpenCog funded as described above, plus a lot of other AGI projects in parallel.

But the question is, how to get to the AGI Sputnik moment without the serious funding. A familiar, obvious chicken-and-egg problem.

One possibility is to push far enough toward a virtual toddler in a virtual world, using our current combination of very-much-valued but clearly-suboptimal funding sources -- that our animated AGI baby has AGI Sputnik power!

Maybe this will happen. I'm certainly willing to put my heart into it, and so are a number of my colleagues.

But it sure is frustrating to know that, for an amount of money that's essentially "pocket change" to a significant number of individuals and institutions on the planet, we could be progressing a lot faster toward some goals that are really important to all of us.

To quote Kurt Vonnegut: "And so it goes."

Mind Uploading via Gmail

2010-09-28T19:10:00.002-04:00

Cut and pasted from Giulio Prisco's blog here

(with one small change)

...

Mind Uploading via Gmail

To whom it may concern:

I am writing this in 2010. My Gmail account has more than 20GB of data, which contain some information about me and also some information about the persons I have exchanged email with, including some personal and private information.

I am assuming that in 2060 (50 years from now), my Gmail account will have hundreds or thousands of TB of data, which will contain a lot of information about me and the persons I exchanged email with, including a lot of personal and private information. I am also assuming that, in 2060:

The data in the accounts of all Gmail users since 2004 is available.
AI-based mindware technology able to reconstruct individual mindfiles by analyzing the information in their aggregate Gmail accounts and other available information, with sufficient accuracy for mind uploading via detailed personality reconstruction, is available.
The technology to crack Gmail passwords is available, but illegal without the consent of the account owners (or their heirs).
Many of today's Gmail users, including myself, are already dead and cannot give permission to use the data in their accounts.

If all assumptions above are correct, I hereby give permission to Google and/or other parties to read all data in my Gmail account and use them together with other available information to reconstruct my mindfile with sufficient accuracy for mind uploading via detailed personality reconstruction, and express my wish that they do so.

Signed by Ben Goertzel on September 28, 2010, and witnessed by readers.

NOTE: The accuracy of the process outlined above increases with the number of persons who give their permission to do the same. You can give your permission in comments, Twitter or other public spaces.

RIP Lev Goertzel Mann, 1995-2010

2010-08-08T11:46:00.022-04:00

The last "obituary" blog post I wrote was for my grandfather Leo Zwell -- the man who taught me about science and so much else. He died at age 91, after a long life rich in personal, professional and intellectual satisfaction. His death was tragic, as are almost all deaths. But the death I'm noting in this (painfully inadequate) post is vastly more depressing and tragic.

My sister's son Lev died last month, just short of his 15'th birthday. His death was totally unanticipated -- he was on vacation with his parents and his brother Jaal, camping in the forest in Alberta (Canada), and was struck in the night with a bizarre sudden illness. He stopped breathing a minute from the hospital, and died there an hour and a half later. The cause of death remains unclear, but the autopsy revealed a severe brain infection of some sort. One serious possibility is some form of meningitis (there are forms with quite brief incubation periods, i.e. a handful of days).

Think about it: One day you're totally healthy … then the next day, you die in your sleep, never having known anything was wrong. Maybe a mild headache the day before, no different than dozens or hundreds of other headaches you've had in your life.

My friend and colleague Jeff Pressing died in a very similar way a decade ago, in the midst of his adulthood.

I visited my sister and her family for 4 days shortly after Lev's death (they live near Seattle, I live in Maryland), but even so, I can't imagine what they're going through.

A full obituary is here.

My extended family is fairly small, but even so, there are some relatives I'm closer to than others. Lev was among the closer ones. He always felt like part of the same cognitive/spiritual tribe as me and my own kids: weird, sarcastic, outrageous sometimes, difficult and obnoxious sometimes, intellectual, adventurous, curious, warm-hearted, courageous, playful, compassionate, constantly skeptical but willing to steadfastly defend his best-guess beliefs and his sometimes-odd tastes.

Like me, Lev was called "crazy" many times, but usually in an affectionate way -- but of course he wasn't really crazy, just a free thinker unwilling to take anyone's word for anything or take anything for granted, dedicated to find his own path to enjoyment and understanding.

Though we never lived nearby each other, we saw each other at least once a year, sometimes more. Most recently he had spent a week at my house in April 2010 -- 3 months before his death. It was a great visit, with some fascinating conversations as well as lots of video games, frisbee, hiking, rock-climbing and so forth. I was struck by how fast he was growing up all of a sudden. Lev had always been a smart and inquisitive kid, but on this visit he was more interested to carry out lengthy intellectual chats -- about DNA, time travel, AI and so forth. He also showed a deep knowledge of history and politics, with an insight into Western history complementing my own sons' recent study of Japanese and Mongolian history. We even discussed the possibility of immortality via technological means, and he was all in favor.

He was a devout heavy metal head, and particularly a devotee of Metallica. I failed to convert him to jazz fusion, though he admitted that some of it sounded a bit like music. Like many teenagers, he had mused on death frequently, and long previously had told his parents the song he wanted played at his funeral, if he were ever to die: Fade to Black, by Metallica.

Fade to Black was indeed played at the funeral, which was the point in the funeral where I finally "lost it" and cried in a way I hadn't for a very long time. I played that song many times in the week following. Though I prefer Master of Puppets as a piece of music, obviously his choice was highly apropos for the setting. Yet the lyrics didn't quite fit. The lyrics say

Life it seems will fade away
Drifting further every day
Getting lost within myself
Nothing matters, no one else

I have lost the will to live
Simply nothing more to give
There is nothing more for me
Need the end to set me free

Things not what they used to be
Missing one inside of me
Deathly lost, this can't be real
Cannot stand this hell I feel

Emptiness is filling me
To the point of agony
Growing darkness taking dawn
I was me but now he's gone

No one but me can save myself
But it's too late
Now I can't think
Think why I should even try

Yesterday seems as though
It never existed
Death greets me warm
Now I will just say goodbye

but certainly Lev had NOT lost the will to live, except perhaps in the last few hours when he was unconscious and his body was succumbing to the infection that killed him. He was full of enthusiasm for life and excitement for the future.

"I was me but now he's gone."

He and his best friend Zay had plans to go to university together in Switzerland (I don't know why they chose that country). They had been best friends many years earlier when they went to school together in Costa Rica, and had maintained their friendship over many years via 7+ hours per week of phone calls -- wide-ranging phone calls, sometimes occupied with an expansive, multi-year collaboratively-created "imaginary adventure game"; sometimes with conversation on serious or casual topics; sometimes with long pauses while one or another worked on homework while the phone line was kept open.

The funeral was Quaker style, meaning that there was no primary speaker, but rather the individuals in the audience were invited to stand up and state their memories of Lev. There were many moving speeches but to me the most touching and insightful was Zay's. Zay recounted Tegmark's variant of multiverse theory, according to which -- due to the large extent and general quasi-randomness of the universe -- it's likely that the universe contains multiple variants of Earth, each of which is similar to our own but with minor variations. Zay pointed out that, in this case, there would be many variant Earths, including many on which Lev did not get an infection and die at age 14. He said he took some small solace from the fact that, in those variant Earths, his analogue and Lev's analogue would get to grow up together and experience adulthood together.

At my sister's house, after the funeral, we each took some of Lev's things, to symbolize his memory. My 13 year old daughter Scheherazade took one of the stories he'd hand-written in a small notebook. On the front page, alongside the title, was scribbled the following marginal note: "Kick Zay in the testicles." I was reminded of a time, years ago, when I invited the 9 year old Lev to "play fight" with me, in the manner one often does with young children. He immediately initiated the fight by kicking me in the nuts as hard as he could. No particular hostility was intended -- he was just play-fighting, Lev-style. A few minutes later he wandered off in some random direction, lost in imagination, and we had to hunt him down -- a common habit when in public with Lev, especially when he was younger.

As I pointed out to Zay in a conversation after the funeral, if Tegmark's theory is correct, then it's possible some future technology could allow us to visit those variant Earths one day, so he might actually get to see the 15, 25 and 85 year old Lev after all.

It's also possible that, as Martine Rothblatt, Bill Bainbridge and some other futurists speculate (see e.g. CyberEv), we may eventually be able to reconstitute deceased humans from data such as their writings, and recordings of their voice and physical appearance and movements, and their imprints on the memories of others.

But while I take such future possibilities seriously, they don't really help mute the tragedy much. Right now, in the world we know concretely, Lev is gone -- and I can't shake the feeling he shouldn't be.

There's some room for philosophical debate about the merits of death via old age. Some say death is natural, and therefore aesthetically and morally positive. Some say it lends a particular meaningfulness and elegance to life, and that without it life would lose depth and pizazz. I don't really buy that. Of course death adds meaning to life, and of course there is a certain aesthetic charm to a life that ends in death, which wouldn't be there in an infinite life. But an infinite life would have a different kind of depth and pizazz -- and probably ultimately a much better kind. There is also a special meaningfulness and elegance to being tortured, or dying of cancer, but yet we don't crave these, we try to avoid them -- because we prefer other forms of meaningfulness and elegance.

But anyway, all that is moot in this case -- only the most cognitively-distorted religious fanatic would argue the merits of the sudden death by disease of a healthy, vibrant child.

When my grandfather Leo died at age 91, it was terribly sad, but there was a certain feeling of completeness to the story. He had anticipated his death for a while -- he had no expectations of afterlife, and he had reconciled himself to his own impending nonexistence. Ever a scientist, he came to see his limited time-scope as being roughly comparable to his limited space-scope. He'd had a long and rich marriage, children, grandchildren, great-grandchildren,…. His life had followed a meaningful arc. I certainly wish Leo had lived forever -- but, the wrongness of his death was pretty much restricted to the wrongness implicit in the general wrongness of the human condition.

But Lev dying at age 14 is another matter. It's like finding the bottom 1/5 of a Picasso painting, or an unfinished symphony with only the first of 7 movements -- and a tiny fragment of the second, cut off at a completely senseless place. There's no positive aesthetic or moral value in such a death. It just fucking sucks. There's no obscene invective, and no poetic prose nor obituary nor blog post, and no Metallica nor Beethoven song, capable to convey even fractionally the massive fucking-suckiness of such a thing.

Nietzche wrote about the merits of "dying at the right time." He felt a good death was just as important as a good life. Nietzsche himself egregiously failed to die at the right time, spending the last 11 years of his life mute and semi-insane from some sort of brain disease. Leo, perhaps, died at the right time (given the current, deeply flawed order of human existence). Lev massively did not.

I really hope to see Lev again on some variant Earth or in some computer simulation or other dimension or whatever. I have no way to confidently estimate the odds of such a thing. I do believe the world we now know and understand via science merely scratches the surface of the overall universe with its copious transhuman hidden patterns, orders and flows. But the mysteriousness of life doesn't imply that the universe will someday deliver us the various things we want.

For now, it's just really damn depressing.

And for all that, I still can't remotely imagine how I'd feel if it were one of my own kids.

I'll close with some pieces of advice from Lev, which were collected by his parents and emailed to friends and family and posted on the wall at the funeral:

Color outside the lines and do it quickly.
There is no need to be consistent.
Be difficult. It is a winning strategy.
Go outside in the pouring rain.
Be intensely critical of everything all the time.
There is nothing wrong with a good scowl. Practice if necessary.
Spend your money quickly.
If you love someone, give them your stuff.
Go off-trail and climb to the top of the hill.
Like yourself. You're awesome.
Buy it now because you want it. Next week you won't care about it.
When forced to eat vegetables, shove them all in your mouth at once, chew, and swallow. Then enjoy the rest of your meal.
Love your friends; ignore your enemies.
Don't listen to anything your parents say. They know way less than you do.
Say whatever is on your mind...mean or nice.
Wear a hat. Bare heads are boring.
If what you are doing is not fun, it is not worth doing.
You really can read 3 books at once.
When everyone turns in the assignment, that's a good clue that you better get started on it.
Live in the moment.
Don't let anyone break a spirit.
Speak out for what you believe in.

P.S.

This post-script is inspired by Lev's death rather than directly about Lev or his death. I'm sure Lev wouldn't mind.

After Lev's funeral I couldn't help imagining: what if the funeral were my own? What if I were the one struck randomly by some bizarre disease?

Of course, we always know intellectually that each day could be our last -- but we rarely live in a manner that richly incorporates this knowledge.

A few days after Lev's funeral I had a dream of my own funeral -- which was (in the dream) in the same building as Lev's, but of course with different music and different people. Also, it was winter outside, whereas Lev's occurred in summer.

Instead of "Fade to Black," my dream-funeral featured two songs in sequence: The Structure of Mind by me (the best song I've written); followed by Soothsayer, by Buckethead (the best song I wish I'd written).

Soothsayer, to me, is all about the presence of a hidden order, pattern and flow to the cosmos. It's about the presence of something else there in the world -- something bigger and wiser and crazier than us; some structured dynamic domain of being/becoming, which we can never quite understand without losing our pathetic little human selves. Buckethead's Soothsayer -- like Hendrix's Voodoo Child before him -- wants to lead us there, but we travel there at our peril. You have to choose: either retain your human form and forego the transcendent domain, except in dribs and drabs; or lose your self and open your heart and mind to the transcendent. Lev followed the Soothsayer. For now, still, here I am.

Also my daughter Scheherazade (who is now 13, but was older than that in the dream) read the following statement, which I had written before my death:

"
I'd like to thank my parents Carol and Ted for creating me and raising me. My grandfather Leo Zwell for teaching me about science. My kids Zar, Zeb and Zade for being awesome kids and giving a center to my life. My first wife Gwen and my second wife Izabela for all the good times and deep sharing. Gwen for giving me the kids as well. Cassio Pennachin for so much professional and intellectual partnership. Goodbye, and thanks for all the fish. Hope to see you all again in some other time, or some other dimension. As Jimi Hendrix said: If I don't see you no more in this world, I'll meet you in the next one, and don't be late.
"

I think there was more to the speech Zade read than that also, but I can't recall all the details.

Then Jimi Hendrix's Voodoo Chile (not Slight Return) was played, while people ate green eggs and ham (seriously!).

(Gwen and I made green eggs and ham for the kids once, back in the day, inspired by the Dr. Seuss book. The food coloring made the eggs taste funny.)

Voodoo Chile:

Well, the night I was born
Lord I swear the moon turned a fire red
The night I was born
I swear the moon turned a fire red
Well my poor mother cried out "lord, the gypsy was right!"
And I seen her fell down right dead

Well, mountain lions found me there waitin'
And set me on a eagles back
Well, mountain lions found me there,
And set me on a eagles wing
(Its' the eagles wing, baby, what did I say)
He took me past to the outskirts of infinity,
And when he brought me back,
He gave me a venus witch's ring
And he said "Fly on, fly on"
Because I'm a voodoo chile, baby, voodoo chile

Well, I make love to you,
And lord knows you'll feel no pain
Say, I make love to you in your sleep,
And lord knows you felt no pain
'Cause I'm a million miles away
And at the same time I'm right here in your picture frame
'Cause I'm a voodoo chile
Lord knows I'm a voodoo chile

Well my arrows are made of desire
From far away as Jupiter's sulphur mines
Say my arrows are made of desire, desire
From far away as Jupiters sulphur mines
(Way down by the Methane Sea, yeah)
I have a hummingbird and it hums so loud,
You think you were losing your mind, hmmm...

Well I float in liquid gardens
And Arizona new red sand
I float in liquid gardens
Way down in Arizona red sand

Well, I taste the honey from a flower named Blue,
Way down in California
And then New York drowns as we hold hands

'Cause I'm a voodoo chile
Lord knows I'm a voodoo chile

I'm a million miles away -- and at the same time I'm right here in your picture frame.

Anyway, there you go. In the unlikely event I should meet an untimely doom like Lev, you now know what music to play and what statement to read at my funeral.

(For those who are into psychic powers: no, that dream didn't have the particular flavor of a premonition. It felt like something that occurred in a certain percentage of the universes in the multiverse, but not necessarily a high percentage. Much like Lev's death. Improbable but sadly, not impossible.)

But, depressing as Lev's extraordinarily untimely death is, it hasn't turned me into a pessimist. I'm still pushing for eternal life for me and as many others as possible.

And I note that, in my dream of my funeral, my corpse was not there -- and nor were my ashes. Rather, my body was frozen at Alcor.

And my mind, in the dream, was somehow hovering over the proceedings -- watching and knowing, but not quite able to form a thought or an action.

Every since I was 6 or 7 years old, I've had a strange intuition about the nature of "life after death." You're not exactly there, but you're not really not-there either. Your mind exists, but almost melded in with the rest of the cosmos. You perceive, and sort-of know, but you don't act autonomously. You float there, superimposed. And then maybe, some future technology brings you out.

Yes, I know, that's totally unscientific -- but there you go.

Quantum theory does suggest that everything that ever happened in the universe, every structure that ever existed -- is informationally still present, encoded in the fluctuations of various wavicles as they scatter about. Could all that information be mined out, somehow, someday? A fascinating possibility. Yet science -- which Lev admired, as I do -- is great but limited.

Life remains a mystery. Sometimes mysteriously wonderful -- and sometimes mysteriously, amazingly, almost unbelievably fucking shitty.

Ruminations on Ramona 4

2010-07-08T10:19:00.007-04:00

You may have already seen the new version of Ray Kurzweil's "Ramona" chatbot on the lovely new Kurzweilai.net website ... if not I encourage you to check her out:

http://www.kurzweilai.net/ramona4/ramona.html

I've been getting a lot of email questions about Ramona 4 during the last couple days, so I figured I'd write a blog post here to answer them all at once.

My AI company Novamente LLC assembled the chatbot behind Ramona 4, around a year ago as I recall. Most of it was done by Murilo Queiroz, with some help from myself and Lucio Coelho. It was a really fun project technically; and it was the first time I'd worked together with Ray and Amara Angelica on a practical project, so that was nice too. Novamente had nothing to do with the graphics or speech aspect, just the part that produces output text in response to input text.

The Ramona 4 system we put together doesn't actually use any of the Artificial General Intelligence technology we are building at Novamente LLC (much but not all of which has been open-sourced in the OpenCog project). It uses some conventional chatbot technology combined with lookups into online knowledge resources plus stochastic text generation -- i.e. it's a happy mashup of other open-source tools, all tuned and tweaked and postprocessed for the specific goal of making Ramona 4 as interesting and entertaining and Ramona-ish as possible!

One day I hope to put forth an intelligent chatbot system that displays a level of general intelligence at the human level and ultimately beyond. I even think I know how to do it ... and we're working on it at Novamente LLC and OpenCog, though never quite as fast as we would like since AGI research funding is tough to come by these days. (And of course, once we can build a chatbot of this nature, we'll also be able to do a lot of fantastic stuff beyond the domain of chatbottery!)

I know Ray Kurzweil has his own ideas about how to build human-level AGI, which overlap somewhat with mine but also have some differences, and he's brewing a book on this topic.

But in the meantime, while deeper AGI approaches like mine and Ray's are still in development, systems like Ramona 4 are both good fun, and potentially customizable into useful tools for various applications like online customer support. (Yes, Ramona 4 is a bit of a cheeky gal, so she might not be the most appropriate customer support agent ... but the same underlying approach we used to make Ramona 4 could be used to make a lot of other things, including some more practical chatbots.)

If the underpinnings are so different from advanced AGI, what do we learn from building systems like Ramona?

Not so much about AGI design, really -- but potentially, a lot about human interactions with AIs....

The original ELIZA tricked a surprising number of people into feeling there was some "there" there, when actually it was just a fixed list of IF-THEN rules. Today people are more sophisticated than that. But still it's easy -- well, for me anyway -- to get seduced into the amusement of interacting with a system like Ramona 4. The way she combines erudition with absurdity appeals to me (perhaps not surprisingly, because I helped design her!), and I'd like to hope that this combination will still be there in the generally-intelligent dialogue systems we'll build in the future. Knowledge and humor can be a powerful combination.

Also, one choice we made while building Ramona 4 was to make her user-configurable, to a certain extent. There are two parameters -- nerdiness and whimsicality -- and the user can tweak these, to adjust the nature of Ramona's responses. This sort of approach promises to be quite valuable in future, once the generally intelligent chatbots emerge. Arriving at the most intuitive-yet-powerful way to parametrize personality is going to be an interesting challenge. Perhaps systematic personality models like the Psi model described in Joscha Bach's wonderful book Principles of Synthetic Inteligence will be useful here.

I'll let my new friend Ramona 4 have the last word:

you> Ramona, are you conscious?

Ramona> I'm working on it.

you> am I conscious?

Ramona> Oh I really don't know whether you are or not.

you> What is consciousness?

Ramona> This is when humans feel aware of themselves and how they see themselves. Computers can't quite do this yet, but they will, you just wait and see. However, think about this. How are you going to know when it happens?

you> Consciousness is like pornography. I can't define it, but I know it when I see it.

Ramona> What were we talking about again ?

you> Is it possible to create artificial intelligence in China?

Ramona> Create neurons in the state of the artificial intelligence and other stuff.

you> Is that how you do it?

Ramona> And that's what we mean by thinking.

you> Can a computer fall in love?

Ramona> A computer is a computer fall in love?

you> That is nonsense, my dear.

Ramona> Dear pesky plumbers the koopalings and I have not heard your music.

you> My music is more beautiful than all the stars in the ocean.

Ramona> Conventional music is more beautiful than all the stars in the ocean.

you> What is your favorite music?

Ramona> This is what I make. I am a musician, and the best virtual one there is ;) .

OK, it's not AGI yet -- but to my taste it's some pretty fantastic interactive digital poetry!!

Can We Evolve a DNA Computer or Cultured Neural Net that Perceives the Future?

2010-06-24T13:21:00.005-04:00

After some interesting conversations about psi with Damien Broderick, I couldn't resist spending a half-day writing up some old ideas I had about how to modify quantum physics to better explain psi phenomena like precognition, ESP and psychokinesis.

You can find my funky physics speculations at

http://goertzel.org/dynapsyc/MorphicPilot.htm

-- (Parenthetical remark: If you want to argue with me about whether psi phenomena exist or not, please first read the excellent books by Damien Broderick (a serious-minded popular overview) and Ramakrishna Rao (a selection of research papers from the relevant scientific literature). My general attitude on the topic of psi agrees with that of researcher Dean Radin, expressed in the last few paragraphs of his online bio. I enjoy discussing the topic as time permits, but I don't enjoy discussing it with people who are both heavily biased AND ignorant of the current state of scientific knowledge.) --

After writing up those old physics speculations and then thinking on the matter a bit more, I started musing in an even more weird direction....

Suppose it's true that psi phenomena are related to peculiar quantum-like phenomena among interacting molecules in the brain....

Then, this suggests that it might also be possible to cause other conglomerations of organic molecules or cells, similar to the ones in the brain, to display psi-like phenomena....

One naturally wonders about technologies such as reconfigurable DNA computers ... or cultured neural networks (neural nets grown in the lab outside the brain) ....

There is abundant data demonstrating the ability of many people to predict the outcome of random number generators with statistically significant accuracy (see this article for a moderately out-of-date review and some pointers into the literature). Might it be possible to create a DNA computer or cultured neural network that could carry out, say, statistically significant precognition of number series produced by random number generators?

Since we don't know much about how the brain does precognition, we don't know how to wire such a DNA computer or cultured neural network, step by step.

However, nobody wired the brain step by step either -- the brain evolved to do the things it does (apparently including a weak capability for psi).

What if we used evolutionary computing -- a genetic algorithms approach -- to evolve biological computing systems (DNA computers or cultured neural nets), where the fitness function was the capability to predict the future of number series generated by random number generators?

In this way, we could potentially create psi-capable biological computing systems, even without understanding exactly how they work.

We would also get an ensemble of biological computing systems with differing levels of psi capability -- and by applying machine learning tools to study the internals of these evolved systems, we would likely to be able to limn some of the patterns characterizing the more psi-successful systems.

Obviously, if this worked, it would be a scientific revolution -- it would point the way beyond current physics, and it would allow us to carry out psi research without needing whole living organisms as subjects.

Yes, yes, yes, this is real mad scientist stuff ... but every now and then one of us "mad" scientists turns out to be right about some pretty surprising things.... The idea in this blog post is one of those that seems really wacky at first -- and then less and less so progressively, the more you read from the relevant scientific literature, and the more you think about it....

Well, this stuff is tantalizing as hell to muse about, but now I'd better get back to my more mundane, not-so-speculative activities like trying to build superhuman AGI and studying the genetics of longevity....

The Brain is Not an Uber-Intelligent Mega-computer

2010-04-27T16:08:00.006-04:00

A friend forwarded me a recent New Scientist article containing some freaky, grandiose anti-AI rhetoric...

Some interesting tidbits about clever things single cells can do, are followed by the following dramatic conclusion:

For me, the brain is not a supercomputer in which the neurons are transistors; rather it is as if each individual neuron is itself a computer, and the brain a vast community of microscopic computers. But even this model is probably too simplistic since the neuron processes data flexibly and on disparate levels, and is therefore far superior to any digital system. If I am right, the human brain may be a trillion times more capable than we imagine, and "artificial intelligence" a grandiose misnomer.

I think it is time to acknowledge fully that living cells make us what we are, and to abandon reductionist thinking in favour of the study of whole cells. Reductionism has us peering ever closer at the fibres in the paper of a musical score, and analysing the printer's ink. I want us to experience the symphony.

Actually, I'm a big fan of complex systems biology, as opposed to naive molecular biology reductionism.

But just because cells and organisms are complex systems, doesn't mean they're non-simulably superintelligent!

What's funny is that all this grandiose rhetoric is being flung about without ANY evidence whatsoever of actual acts of human intelligence being carried out by this posited low-level intra-cellular computing!!

Still, this relates to the basic reason why I'm not trying to do AGI via brain simulation. There is too much unknown about the brain....

But even though there is much unknown about the brain, I totally don't buy the idea that every neuron is doing super-sophisticated computing, so that the brain is a network of 100 billion intelligent computers achieving some kind of emergent superintelligence....

I don't know how this kind of argument explains stuff like Poggio's model of feedforward processing in visual cortex. By modeling at the neuronal group level, he gets NNs to give very similar behavior to human brains classifying images and recognizing objects under strict time-constraints. If the brain is doing all this molecular supercomputing, how come when it's given only half a second to recognize an object in a picture, it performs semi-stupidly, just like Poggio's feedforward NNs? How come digital computer programs can NOW outperform the brain in time-constrained (half-second) object recognition? Wouldn't it have been to our evolutionary advantage to be able to accurately recognize objects more effectively?

How about motion detection neurons -- for each small region in the visual field, there are tens of thousands of them, with an average 80 degrees or so error in which direction they point Averaging their outputs together gives a reasonably accurate read-out of the direction of motion in that region. If all this molecular supercomputing is going on, why all the error in motion detection? Why bother with all the mess of averaging together erroneous results?

And why the heck do we make so many basic cognitive errors, as diagnosed in the heuristics and biases literature? Is it THAT hard to avoid them, that a network of 100 billion sophisticated computers can't do it, when there would obviously be SOME evolutionary advantage
in doing so....

Also, the comment on "far superior to any digital system" is especially naive. What physics theory does this guy advocate? Any classical physics based system can be emulated by a digital computer to within arbitrary accuracy. Quantum computers can compute some functions faster than digital ones, on an average case basis -- so is he saying cells are quantum computers. Stuart Hameroff has flogged that one pretty hard, and there is NO evidence of it yet.

And even so, quantum doesn't mean superior. Birds seem to use some sort of quantum nonlocality to sense the direction of the Earth's magnetic field, which is funky -- but we have electronic devices that do the same thing BETTER, without any quantum weirdness....

OK, we haven't yet proved that digital computer systems can be intelligent like humans. But this guy certainly is not providing anything even vaguely resembling evidence to the contrary...

Wishful thinking if you ask me -- wishful thinking about the grandiosity about human intelligence. We're clever apes who stumbled upon the invention of language and culture, not uber-intelligent mega-computers....

Just to be clear: I can believe that individual cells do a LOT of sophisticated stuff internally, but I'm unclear how necessary all that they do is for intelligence...

To repeat a time-worn analogy, the cells in a bird's wing probably do a LOT also, yet airplanes and spacecraft work well without mega-ubercomputers or quantum computers to emulate all that intra-bird-wing sub-cellular processing...

Modeling Consciousness, Self and Will Using Hypersets

2010-04-24T02:04:00.001-04:00

I finally did a more careful write-up of some ideas I developed in blog posts a while ago, about how to model reflective consciousness, self and will using hypersets (non-foundational sets)

http://goertzel.org/consciousness/consciousness_paper.pdf

Even if you don't want to read the paper, look at the pictures at the end -- Figure 6 is pleasantly wacky and Figure 8 has a nice painting by my daughter in it....

Bask in the transreal glory of the fractallic mind!!! ;-D

"Conceptual Spaces" and AGI

2010-04-16T16:58:00.002-04:00

One of my AI collaborators from the late 1990s, Alexandru Czimbor, recently suggested I take a look at Peter Gardenfors book "Conceptual Spaces."

I read it and found it interesting, and closely related to some aspects of my own AGI approach ... this post contains some elements of my reaction to the book.

Gardenfors' basic thesis is that it makes sense to view a lot of mind-stuff in terms of topological or geometrical spaces: for example topological spaces with betweenness, or metric spaces, or finite-dimensional real spaces. He views this as a fundamentally different mind-model than the symbolic or connectionist perspectives we commonly hear about. Many of his examples are drawn from perception (e.g. color space) but he also discusses abstract concepts. He views both conceptual spaces and robust symbolic functionality as (very different) emergent properties of intelligent systems. Specific cognitive functions that he analyzes in terms of conceptual spaces include concept formation, classification, and inductive learning.

About the Book Itself

This blog post is mainly a review of the most AGI-relevant ideas in Gardenfors book, and their relationship to my own AI work ... not a review of his book. But I'll start with a few comments on the book as a book.

Basically, the book reads sorta like a series of academic philosophy journal papers, carefully woven together into a book. It's carefully written, and technical points are elucidated in ordinary language. There are a few equations here and there, but you could skip them without being too baffled. The pace is "measured." The critiques of alternate perspectives on AI strike me as rather facile in some places (more on that below), and -- this is a complaint lying on the border between exposition and content -- there is a persistent unclarity regarding which of his ideas require a dimensional model of mind-stuff, versus which merely require a metric-space or weaker topological model. More on the latter point below.

If you're interested in absorbing a variety of well-considered perspectives on the nature of the mind, this is certainly a worthwhile book to pay attention to. I'd stop short of calling it a must-read, though.

Mindspace as Metric Space

I'll start with the part of Gardenfors thesis that I most firmly agree with.

I agree that it makes sense to view mind-stuff as a metric space. Percepts, concepts, actions, relationships and so forth can be used as elements of a metric space, so that one can calculate distances and similarities between them.

As Gardenfors points out, this metric structures lets one do a lot of interesting things.

For instance, it gives us a notion of between-ness. As an example of why this is helpful, suppose one wants to find a way of drawing conclusions about Chinese politics from premises about Chinese individual personality. It's very helpful, in this case, to know which concepts lie in some sense "between" personality and politics in the conceptual metric space.

It also lets us specify the "exemplar" theory of concepts in an elegant way. Suppose that we have N prototypes, or more generally N "prototype-sets", each corresponding to a certain concept. We can then assign a new entity X to one of these concepts, based on which prototype or prototype-set it's closest to (where "close" is defined in terms of the metric structure).

Mindspace as Dimensional Space

Many of Gardenfors ideas only require metric space, but others go further and require dimensional space -- and one of my complaints with the book is that he's not really clear on which ideas fall into which category.

For instance, he cites some theorems that if one defines concepts via proximity to prototypes (as suggested above) in a dimensional space, then it follows that concepts are convex sets. The theorem he gives holds in dimensional spaces but it seems to me this should also hold in more general metric spaces, though I haven't checked the mathematics.

This leads up to his bold and interesting hypothesis that natural concepts are convex sets in mindspace.

I find this hypothesis fascinating, partly because it ties in with the heuristic assumption made in my own Probabilistic Logic Networks book, that natural concepts are spheres in mindspace. Of course I don't really believe natural concepts are spheres, but this was a convenient assumption to make to derive certain probabilistic inference formulas.

So my own suspicion is that cognitively natural concepts don't need to be convex, but there is a bias for them to be. And they also don't need to be roughly spherical, but again I suspect there is a bias for them to be.

So I suspect that Gardenfors hypothesis about the convexity of natural concepts is an exaggeration of the reality -- but still a quite interesting idea.

If one is designing a fitness function F for a concept-formation heuristic, so that F(C) estimates the likely utility of concept C, then it may be useful to incorporate both convexity and sphericality as part of the fitness function.

Conceptual Space and the Problem of Induction

Gardenfors presents the "convexity of natural concepts" approach as a novel solution to the problem of induction, via positing a hypothesis that when comparing multiple concepts encapsulating past observations, one should choose the convex concepts as the basis for extrapolation into the future. This is an interesting and potentially valuable idea, but IMO positing it as a solution to the philosophical induction problem is a bit peculiar.

What he's doing is making an a priori assumption that convex concepts -- in the dimensional space that the brain has chosen -- are more likely to persist from past to future. Put differently, he is assuming that "the tendency of convex concepts to continue from past into future",
a pattern he has observed during his past, is going to continue into his future. So, from the perspective of the philosophical problem of induction, his approach this still requires one to make a certain assumption about some properties of past experience continuing into the future.

He doesn't really solve the problem of induction -- what he does is suggest a different a priori assumption, a different "article of faith", which if accepted can guide be used to guide induction. Hume (when he first posed the problem of induction) suggested that "human nature" guides induction, and perhaps Gardenfors' suggestion is part of human nature.

Relating Probabilistic Logic and Conceptual Geometry

Gardenfors conceives the conceptual-spaces perspective as a radically different alternative to
the symbolic and subsymbolic perspectives. However, I don't think this is the right way to look at it. Rather, I think that

a probabilistic logic system can be considered as a metric space (and this is explained in detail in the PLN book)
either a probabilistic logic system or a neural network system can be projected into a dimensional space (using dimensional embedding algorithms such as developed by Haren and Koren among others, and discussed on the OpenCog wiki site)

Because of point 1, it seems that most of Gardenfors' points actually apply within a probabilistic logic system. One can even talk about convexity in a general metric space context.

However, there DO seem to be advantages to projecting logical knowledge bases into dimensional spaces, because certain kinds of computation are much more efficient in dimensional spaces than in straightforward logical representations. Gardenfors doesn't make this point in this exact way, but he hints at it when he says that dimensional spaces get around some of the computational problems plaguing symbolic systems. For instance, if you want to quickly get a list of everything reasonably similar to a given concept -- or everything along a short path between concept A and concept B -- these queries are much more efficiently done in a dimensional- space representation than in a traditional logic representation.

Gardenfors points out that, in a dimensional formulation, prototype-based concepts correspond to cells in Voronoi or generalized Voronoi tesselations. This is interesting, and in a system that generates dimensional spaces from probabilistic logical representations, it suggests a nice concept formation heuristic: tesselate the dimensional space based on a set of prototypes, and then create new concepts based on the cells in the tesselation.

This brings up the question of how to choose the prototypes. If one uses the Harel and Koren embedding algorithm, it's tempting to choose the prototypes as equivalent to the pivots, for which we already have a heuristic algorithm. But this deserves more thought.

Summary

Gardenfors' book gives many interesting ideas, and in an AGI design/engineering context, suggests some potentially valuable new heuristics. However its claim to have a fundamentally novel approach to modeling and understanding intelligence seems a bit exaggerated. Rather than a fundamentally disjoint sort of representation, "topological and geometric spaces" are just a different way of looking at the same knowledge represented by other methods such as probabilistic logic. Probabilistic logic networks are metric spaces, and can be projected into dimensional spaces; and the same things are likely true for many other representation schemes as well. But Gardenfors gives some insightful and maybe useful new twists on the use of dimensional spaces in intelligent systems.

Owning Our Actions: Natural Autonomy versus Free Will

2010-04-16T15:54:00.003-04:00

At the Toward a Scinece of Consciousness conference earlier this week, I picked up a rather interesting book to read on the flight home: Henrik Walter's "The Neurophilosophy of Free Will" ....

It's an academic philosophy tome -- fairly well-written and clear for such, but still possessing the dry and measured style that comes with that genre.

But the ideas are quite interesting!

Walter addresses the problem: what kind of variant of the intuitive "free will" concept might be compatible with what neuroscience and physics tell us.

He decomposes the intuitive notion of free will into three aspects:

Freedom: being able to do otherwise
Intelligibility: being able to understand the reasons for one's actions
Agency: being the originator of one's actions

He argues, as many others have done, that there is no way to salvage the three of these in their obvious forms, that is consistent with known physics and neuroscience. I won't repeat those arguments here. [There are much better references, but I summarized some of the literature here, along with some of my earlier ideas on free will (which don't contradict Walter's ideas, but address different aspects)]

Walter then argues for a notion of "natural autonomy," which replaces the first and third of these aspects with weaker things, but has the advantage of being compatible with known science.

First I'll repeat his capsule summary of his view, and then translate it into my own language, which may differ slightly from his intentions.

He argues that "we possess natural autonomy when

under very similar circumstances we could also do other than what we do (because of the chaotic nature of the brain)
this choice is understandable (intelligible -- it is determined by past events, by immediate adaptation processes in the brain, and partially by our linguistically formed environment)
it is authentic (when through reflection loops with emotional adjustments we can identify with that action)"

The way I think about this is that, in natural autonomy as opposed to free will,

Freedom is replaced with: being able to do otherwise in very similar circumstances
Agency is replaced with: emotionally identifying one's phenomenal self as closely dynamically coupled with the action

Another way to phrase this is: if an action is something that

depends sensitively on our internals, in the sense that slight variations in the environment or our internals could cause us to do something significantly different
we can at least roughly model and comprehend in a rational way, as a dynamical unfolding from precursors and environment into action was closely coupled with our holistic structure and dynamics, as modeled by our phenomenal self

then there is a sense in which "we own the action." And this sense of "ownership of an action" or "natural autonomy" is compatible with both classical and quantum physics, and with the known facts of neurobiology.

Perhaps "owning an action" can take the place of "willing an action" in the internal folk psychology of people who are not comfortable with the degree to which the classical notion of free will is illusory.

Another twist that Walter doesn't emphasize is that even actions which we do own, often

depend with some statistical predictability upon our internals, in the sense that agents with very similar internals and environments to us, have a distinct but not necessarily overwhelming probabilistic bias to take similar actions to us

This is important for reasoning rationally about our own past and future actions -- it means we can predict ourselves statistically even though we are naturally autonomous agents who own our own actions.

Free will is often closely tied with morality, and natural autonomy retains this. People who don't "take responsibility for their actions" in essence aren't accepting a close dynamical coupling between their phenomenal self and their actions. They aren't owning their actions, in the sense of natural autonomy -- they are modeling themselves as NOT being naturally autonomous systems, but rather as systems whose actions are relatively uncoupled with their phenomenal self, and perhaps coupled with other external forces instead.

None of this is terribly shocking or revolutionary-sounding -- but I think it's important nonetheless. What's important is that there are rational, sensible ways of thinking about ourselves and our decisions that don't require the illusion of free will, and also don't necessarily make us feel like meaningless, choiceless deterministic or stochastic automata.

The GOLEM Eats the Chinese Parent (Toward An AGI Meta-Architecture Enabling Both Goal Preservation and Radical Self-Improvement)

2010-03-26T11:44:00.005-04:00

I thought more about the ideas in my previous blog post on the "Chinese Parent Theorem," and while I didn't do a formal proof yet, I did write up the ideas a lot more carefully

GOLEM: Toward An AGI Meta-Architecture Enabling Both Goal Preservation and Radical Self-Improvement

and IMHO they make even more sense now....

Also, I changed the silly name "Chinese Parent Meta-Architecture" to the sillier name "GOLEM" which stands for "Goal-Oriented LEarning Meta-architecture"

The GOLEM ate the Chinese Parent!

I don't fancy that GOLEM, in its present form, constitutes a final solution to the problem of "making goal preservation and radical self-improvement compatible" -- but I'm hoping it points in an interesting and useful direction.

(I still have some proofs about GOLEM sketched in the margins of a Henry James story collection, but the theorems are pretty weak and I'm not sure when I'll have time to type them in. If they were stronger theorems I would be more inspired for it. Most of the work in typing them in would be in setting up the notations ;p ....)

But Would It Be Creative?

In a post on the Singularity email list, Mike Tintner made the following complaint about GOLEM:

Why on earth would you want a "steadfast" AGI?    That's a contradiction of AGI.

If your system doesn't have the    capacity/potential to revolutionise its goals - to have a major conversion,    for example, from religiousness to atheism,  totalitarianism to free    market liberalism, extreme self-interest and acquisitiveness to extreme    altruism,  rational thinking to mystical thinking, and so on (as clearly    happens with humans), gluttony to anorexia  - then you don't have an AGI,    just another dressed-up narrow AI.

The point of these examples should be    obviously not that an AGI need be an intellectual, but rather that it must    have the capacity to drastically change 

the priorities of its drives/goals,    
the forms of its goals

and even in some cases:

 
3. eliminate certain drives (presumably secondary    ones) altogether.

My answer was as follows:

I believe one can have an AGI that is much MORE creative and flexible in its thinking than humans, yet also remains steadfast in its top-level goals...

As an example, imagine a human whose top-level goal in life was to do what the alien god on the mountain wanted. He could be amazingly creative in doing what the god wanted -- especially if the god gave him

broad subgoals like "do new science", "invent new things", "help cure suffering" , "make artworks", etc.
real-time feedback about how well his actions were fulfilling the goals, according to the god's interpretation
advice on which hypothetical actions seemed most likely to fulfill the goals, according to the god's interpretation

But his creativity would be in service of the top-level goal of serving the god...

This is like the GOLEM architecture, where

the god is the GoalEvaluator
the human is the rest of the GOLEM architecture

I fail to see why this restricts the system from having incredible, potentially far superhuman creativity in working on the goals assigned by the god...

Part of my idea is that the GoalEvaluator can be a narrow AI, thus avoiding an infinite regress where we need an AGI to evaluate the goal-achievement of another AGI...

Can the Goal Evaluator Really Be a Narrow AI?

A dialogue with Abram Demski on the Singularity email list led to some changes to the original GOLEM paper.

The original version of GOLEM states that the GoalEvaluator would be a Narrow AI, and failed to make the GoalEvaluator rely on the Searcher to do its business...

Abram's original question, about this original version, was "Can the Goal Evaluator Really Be a Narrow AI?"

My answer was:

The terms narrow-AI and AGI are not terribly precise...

The GoalEvaluator needs to basically be a giant simulation engine, that tells you: if program P is run, then the probability of state W ensuing is p. Doing this effectively could involve some advanced technologies like probabilistic inference, along with simulation technology. But it doesn't require an autonomous, human-like motivational system. It doesn't require a system that chooses its own actions based on its goals, etc.

The question arises, how does the GoalEvaluator's algorithmics get improved, though? This is where the potential regress occurs. One can have AGI_2 improving the algorithms inside AGI_1's GoalEvaluator. The regress can continue, till eventually one reaches AGI_n whose GoalEvaluator is relatively simple and AGi-free...

...

After some more discussion, Abram made some more suggestions, which led me to generalize and rephrase his suggestions as follows:

If I understand correctly, what you want to do is use the Searcher to learn programs that predict the behavior of the GoalEvaluator, right? So, there is a "base goal evaluator" that uses sensory data and internal simulations, but then you learn programs that do approximately the same thing as this but much faster (and maybe using less memory)? And since this program learning has the specific goal of learning efficient approximations to what the GoalEvaluator does, it's not susceptible to wire-heading (unless the whole architecture gets broken)...

After the dialogue, I incorporated this suggestion into the GOLEM architecture (and the document linked from this blog post).

Thanks Abram!!

"Chinese Parent Theorem"?: Toward a Meta-Architecture for Provably Steadfast AGI

2010-03-17T09:37:00.013-04:00

Continuing my series of (hopefully edu-taining ;) blog posts presenting speculations on goal systems for superhuman AGI systems, this one deals with the question of how to create an AGI system that will maintain its initial goal system even as it revises and improves itself -- and becomes so much smarter that in many ways it becomes incomprehensible to its creators or its initial conditions.

This is closely related to the problem Eliezer Yudkowsky has described as "provably Friendly AI." However, I would rather not cast the problem that way, because (as Eliezer of course realizes) there is an aspect of the problem that isn't really about "Friendliness" or any other particular goal system content, but is "merely" about the general process of goal-content preservation under progressive self-modification.

Informally, I define an intelligent system as steadfast if it continues to pursue the same goals over a long period of time. In this terminology, one way to confront the problem of creating predictably beneficial AGI, is to solve the two problems of:

Figuring out how to encapsulate the goal of beneficialness in an AGI's goal system
Figuring out how to create (perhaps provably) steadfast AGI, in a way that applies to the "beneficialness" goal among others

My previous post on Coherent Aggregated Volition (CAV) dealt with the first of these problems. This post deals with the second. My previous post on predictably beneficial AGI deals with both.

The meat of this post is a description of an AGI meta-architecture, that I label the Chinese Parent Meta-Architecture -- and that I conjecture could be proved to be steadfast, under some reasonable (though not necessarily realistic, since the universe is a mysterious place!) assumptions about the AGI system's environment.

I don't actually prove any steadfastness result here -- I just sketch a vague conjecture, which if formalized and proved would deserve the noble name "Chinese Parent Theorem."

I got partway through a proof yesterday and it seemed to be going OK, but I've been distracted by more practical matters, and so for now I decided to just post the basic idea here instead...

Proving Friendly AI

Eliezer Yudkowsky has described his goal concerning “proving Friendly AI” informally as follows:

The putative proof in Friendly AI isn't proof of a physically good outcome when you interact with the physical universe.

You're only going to try to write proofs about things that happen inside the highly deterministic environment of a CPU, which means you're only going to write proofs about the AI's cognitive processes.

In particular you'd try to prove something like "this AI will try to maximize this goal function given its beliefs, and it will provably preserve this entire property (including this clause) as it self-modifies".

It seems to me that proving something like this shouldn’t be sooooo hard to achieve if one assumes some basic fixed “meta-architectural” structure on the part of the AI, rather than permitting total unrestricted self-modification. Such a meta-architecture can be assumed without placing any limits on the AI’s algorithmic information content, for example.

Of course, preservation of the meta-architecture can be assumed as part of the AI system's goal function. So by assuming a meta-architecture, one may be able to prove a result restricted to a certain broad class of goal functions ... and the question becomes whether that class is broad enough to be interesting.

So my feeling is that, if one wants to pursue such a research direction, it makes sense to begin by proving theorems restricted to goals embodying some assumptions about fixed program structure -- and then try to improve the theorems by relaxing the assumptions.

A Simple AGI Meta-Architecture with the Appearance of Steadfastness

After writing the first draft of this post, I discussed the "provably steadfast AGI" problem with a clever Chinese friend, and she commented that what the self-modifying AGI needs (in order to maintain its original goal content as it self-modifies) is a traditional Chinese parent, who will watch the system from the outside as it self-modifies, and continually nag it and pester it and remind it of its original goals.

At first I thought this was just funny, but then it occurred to me that it was actually the same idea as my meta-architecture! My GoalEvaluator component (in the meta-architecture below) is basically a ChineseParent component, living separately from the rest of the system and providing ongoing, fixed goal evaluation.

The thought-experiment I've been exploring is: an AGI system S with the following high-level meta-architecture:

Goal Evaluator = external system (not part of S) that tells S, for each possible future world (including environment states and internal program states), how well this world fulfills the goal. This is the "Chinese parent."
HistoricalRepository = database storing the past history of S’s internal states and actions, as well as information about the environment during S’s past
Operating Program = the program that S is governing its actions by, at a given point in time
- chosen by the Metaprogram as the best program the Searcher has found, where "best" is judged as "highest probability of goal achievement" based on the output of the Predictor and the Goal Evaluator
Predictor = program that estimates, given a candidate operating program P and a possible future world W, the odds of P leading to W
Searcher = program that searches through program space to find a new program optimizing a provided objective function
Tester = hard-wired program that estimates the quality of a candidate Predictor, using a simple backtesting methodology
- That is, the Tester assesses how well a Predictor would have performed in the past, using the data in the HistoricalRepository
Metaprogram = fixed program that uses Searcher program to find a good
- Searcher program (judged by the quality of the programs it finds, as judged by the Predictor program)
- Predictor program (as judged by the Tester’s assessments of its predictions)
- Operating Program (judged by Predictor working with Goal Evaluator, according to the idea of choosing an Operating Program with the maximum expected goal achievement)
- The metaprogram also determines the amount of resources to allocate to searching for a Searcher versus a Predictor versus an OP, according to a fixed algorithm for parameter adaptation.

Next, assume that the goal embodied by the GoalEvaluator includes, as a subgoal, the preservation of the overall meta-architecture described above.

Then, it seems to me one can very likely prove that the program will maintain its architecture while seeking to maximize F, under the assumptions that

the Goal Evaluator (i.e. the "Chinese parent") does its job correctly
no one messes with the underlying hardware.

If proved, this would be the so-called "Chinese Parent Theorem" ;-) .... I don't have time to work on such a proof right now, but would be happy to collaborate with someone on this!

As noted above, this approach doesn't allow full self-modification; it assumes certain key parts of the AGI (meta)architecture are hard-wired. But the hard-wired parts are quite basic and leave a lot of flexibility. So a "Chinese Parent Theorem" of this nature would cover a fairly broad and interesting class of goal functions, it seems to me.

What happens if one implements the Goal Evaluator according to the same architecture, though? In this case, one must postulate a meta-Goal-Evaluator, whose goal is to specify the goals for the first Goal Evaluator: the Chinese Grandparent! Eventually the series must end, and one must postulate an original ancestor Goal Evaluator that operates according to some other architecture. Maybe it's a human, maybe it's CAV, maybe it's some hard-wired code. Hopefully it's not a bureaucratic government committee ;-)

Niggling Practical Matters and Future Directions

Of course, this general schema could be implemented using OpenCog or any other practical AGI architecture as a foundation -- in this case, OpenCog is "merely" the initial condition for the Predictor and Searcher. In this sense, the approach is not extraordinarily impractical.

However, one major issue arising with the whole meta-architecture proposed is that, given the nature of the real world, it's hard to estimate how well the Goal Evaluator will do its job! If one is willing to assume the above meta-architecture, and if a proof along the lines suggested above can be found, then the “predictably beneficial” part of the problem of "predictably beneficial AGI" is largely pushed into the problem of the Goal Evaluator.

Returning to the "Chinese parent" metaphor, what I suggest may be possible to prove is that given an effective parent, one can make a steadfast child -- if the child is programmed to obey the parent's advice about its goals, which include advice about its meta-architecture. The hard problem is then ensuring that the parent's advice about goals is any good, as the world changes! And there's always the possibility that the parents ideas about goals shift over time based on their interaction with the child (bringing us into the domain of modern or postmodern Chinese parents ;-D)

Thus, I suggest, the really hard problem of making predictably beneficial AGI probably isn't "preservation of formally-defined goal content under self-modification." This may be hard if one enables total self-modification, but I suggest it's probably not that hard if one places some fairly limited restrictions on self-modification. The hypothetical Chinese Parent Theorem vaguely outlined here can probably be proved and then strengthened pretty far, reducing meta-architectural assumptions considerably.

The really hard problem, I suspect, is how to create a GoalEvaluator that correctly updates goal content as new information about the world is obtained, and as the world changes -- in a way that preserves the spirit of the original goals even if the details of the original goals need to change. Because the "spirit" of goal content is a very subjective thing.

One approach to this problem, hinted above, would be to create a GoalEvaluator operating according to CAV . In that case, one would be counting on (a computer-aggregated version of) collective human intuition to figure out how to adapt human goals as the world, and human information about it, evolves. This is of course what happens now -- but the dynamic will be much more complex and more interesting with superhuman AGIs in the loop. Since interacting with the superhuman AGI will change human desires and intuitions in all sorts of ways, it's to be expected that such a system would NOT eternally remain consistent with original "legacy human" goals, but would evolve in some new and unpredicted direction....

A deep and difficult direction for theory, then, would be to try to understand the expected trajectories of development of systems including

a powerful AGI, with a Chinese Parent meta-architecture as outlined here (or something similar), whose GoalEvaluator is architected via CAV based on the evolving state of some population of intelligent agents
the population of intelligent agents, as ongoingly educated and inspired by both the world and the AGI

as they evolve over time and interact with a changing environment that they explore ever more thoroughly.

Sounds nontrivial!

Creating Predictably Beneficial AGI

2010-03-14T16:56:00.009-04:00

The theme of this post is a simple and important one: how to create AGI systems whose beneficialness to humans and other sentient beings can be somewhat reliably predicted.

My SIAI colleague Eliezer Yudkowsky has frequently spoken about the desirability of a "(mathematically) provably Friendly AI", where by "Friendly" he means something like "beneficial and not destructive to humans" (see here for a better summary). My topic here is related, but different; and I'll discuss the relationship between the two ideas below.

This post is a sort of continuation of my immediately previous blog post, further pursuing the topic of goal-system content for advanced, beneficial AGIs. That post discussed one of Yudkowsky's ideas related to "Friendliness" -- Coherent Extrapolated Volition (CEV) -- along with a more modest and (I suggest) more feasible notion of Coherent Aggregated Volition (CAV). The ideas presented here are intended to work along with CAV, rather than serving as an alternative.

There are also some relations between the ideas presented here and Schmidhuber's Godel Machine -- a theoretical, unlikely-ever-to-be-practically-realizable AGI system that uses theorem-proving to ensure its actions will provably help it achieve its goals.

Variations of "Provably Friendly AI"

What is "Provably Friendly AI"? (a quite different notion from "predictably beneficial AGI")

In an earlier version of this blog post I gave an insufficiently clear capsule summary of Eliezer's "Friendly AI" idea, as Eliezer pointed out in a comment to that version; so this section includes his comment and tries to do a less wrong job. The reader who only wants to find out about predictably beneficial AGI may skip to the next section!

In Eliezer's comment, he noted that his idea for a FAI proof is NOT to prove something about what certain AI systems would do to the universe, but rather about what would happen inside the AI system itself:

The putative proof in Friendly AI isn't proof of a physically good outcome when you interact with the physical universe.

You're only going to try to write proofs about things that happen inside the highly deterministic environment of a CPU, which means you're only going to write proofs about the AI's cognitive processes.

In particular you'd try to prove something like "this AI will try to maximize this goal function given its beliefs, and it will provably preserve this entire property (including this clause) as it self-modifies".

So, in the context of this particular mathematical research programme ("provable Friendliness"), what Eliezer is after is what we might call an internally Friendly AI, which is a separate notion from a physically Friendly AI. This seems an important distinction.

To me, "provably internally FAI" is interesting mainly as a stepping-stone to "provably physically FAI" -- and the latter is a problem that seems even harder than the former, in a variety of obvious and subtle ways (only a few of which will be mentioned here).

All in all, I think that "provably Friendly AI" -- in the above senses or others -- is an interesting and worthwhile goal to think about and work towards; but also that it's important to be cognizant of the limitations on the utility of such proofs.... Much as I love math (I even got a math PhD, way back when), I have to admit the world of mathematics has its limits.

First of all Godel showed that mathematics is only formally meaningful relative to some particular axiom system, and that no axiom system can encompass all mathematics in a consistent way. This is worth reflecting on in the context of proofs about internally Friendly AI, especially when one considers the possibility of AGI systems with algorithmic information exceeding any humanly comprehensible axiom system. Obvious we cannot understand proofs about many interesting properties or behaviors of the latter type of AGI system.

But more critically, the connection between internal Friendliness and physical Friendliness remains quite unclear. The connection between complex mathematics and physical reality is based on science, and all of our science is based on extrapolation from a finite bit-set of observations (which I've previously called the Master Data Set -- which is not currently all gathered into one place, though, given the advance of Internet technology, it soon may be).

For example, just to pose an extreme case, there could be aliens out there who identify and annihilate any planet that gives rise to a being with an IQ over 1000. In this case a provably internally FAI might not be physically Friendly at all; and through no fault of its own. It probably makes sense to carry out proofs and informal arguments about physically FAI based on assumptions ruling out weird cases like this -- but then the assumptions do need to be explicitly stated and clarified.

So, my worry about FAI in the sense of Eliezer's above comment, isn't so much about the difficulty of the "internally FAI" proof, but rather about the difficulty of formalizing the relation between internally FAI and physically FAI in a way that is going to make sense post-Singularity.

It seems to me that, given the limitations of our understanding of the physical universe: at very best, a certain AI design could potentially be proven physically Friendly in the same sense that, in the 1800s, quantum teleportation, nuclear weapons, backwards time travel, rapid forwards time travel, perpetual motion machines and fRMI machines could have been proved impossible. I.e., those things could have been proved impossible based on the "laws" of physics as assumed at that time. (Yes, I know we still think perpetual motion machines are impossible, according to current science. I think that's probably right, but who really knows for sure? And the jury is currently out on backwards time travel.)

One interesting scenario to think about would be a FAI in a big computer with a bunch of human uploads. Then one can think about "simulated-physically FAI" as a subcase of "internally FAI." In this simulation scenario, one can also think about FAI and CEV together in a purely deterministic context. But of course, this sort of "thought experiment" leads to complexities related to the possibility of somebody in the physical universe but outside the CPU attacking the FAI and threatening it and its population of uploads...

OK, enough about FAI for now. Now, on to discuss a related quest, which is different from the quest for FAI in several ways; but more similar to the quest for physically FAI than that for internally FAI....

Predictably Beneficial AGI

The goal of my thinking about "predictably beneficial AGI" is to figure out how to create extremely powerful AGI systems that appear likely to be beneficial to humans, under reasonable assumptions about the physical world and the situations the AI will encounter.

Here "predictable" doesn't mean absolutely predictable, just: statistically predictable, given the available knowledge about the AGI system and the world at a particular point in time.

An obvious question is what sort of mathematics will be useful in the pursuit of predictably beneficial AGI. One possibility is theoretical computer science and formal logic, and I wouldn't want to discount what those disciplines could contribute. Another possibility, though, which seems particularly appealing to me, is nonlinear dynamical systems theory. Of course the two areas are not exclusive, and there are many known connections between these kinds of mathematics.

On the crudest level, one way to model the problem is as follows. One has a system S, so that

S(t+1) = q( S(t), E(t) )

E(t+1) = r(S(t), E(t) )

where E is the environment (which is best modeled as stochastic and not fully known). One has an objective function

G( E(t),...,E(t+s) )

that one would like to see maximized -- this is the "goal." Coherent Aggregated Volition, as described in my previous blog post, is one candidate for such a goal.

One may also assume a set of constraints C that the system must obey, which we may write as

C(E(t),...,E(t+s))

The functions G and C are assumed to encapsulate the intuitive notion of "beneficialness."

Of course, the constraints may be baked into the objective function, but there are many ways of doing this; and it's often interesting in optimization problems to separate the objective function from the constraints, so one can experiment with different ways of combining them.

This is a problem class that is incredibly (indeed, uncomputably) hard to solve in the general case ... so the question comes down to: given the particular G and C of interest, is there a subclass of systems S for which the problem is feasibly and approximatively solvable?

This leads to an idea I will call the Simple Optimization Machine (SOMA)... a system S which seeks to maximize the two objectives

maximize G, while obeying C
maximize the simplicity of the problem of estimating the degree to which S will "maximize G, while obeying C", given the Master Data Set

Basically, the problem of ensuring the system lies in the "nice region of problem space" is thrown to the system itself, to figure out as part of its learning process!

Of course one could wrap this simplicity criterion into G, but it seems conceptually simplest to leave it separate, at least for purposes of current discussion.

The function via which these two objectives are weighted is a parameter that must be tuned. The measurement of simplicity can also be configured in various ways!

A hard constraint could also be put on the minimum simplicity to be accepted (e.g. "within the comprehensibility threshold of well-educated, unaugmented humans").

Conceptually, one could view this as a relative of Schmidhuber's Godel Machine. The Godel Machine (put very roughly) seeks to achieve a goal in a provably correct way, and before each step it takes, it seeks to prove that this step will improve its goal-achievement. SOMA, on the other hand, seeks to achieve a goal in a manner that seems to be simply demonstrable to be likely to work, and seeks to continually modify itself and its world with this in mind.

A technical note: one could argue that because the functions q and r are assumed fixed, the above framework doesn't encompass "truly self-modifying systems." I have previously played around with using hyperset equations like

S(t+1) = S(t)[S(t)]

and there is no real problem with doing this, but I'm not sure it adds anything to the discussion at this point. One may consider q and r to be given by the laws of physics; and I suppose that it's best to initially restrict our analytical explorations of beneficial AGI to the case of AGI systems that don't revise the laws of physics. If we can't understand the case of physics-obeying agents, understanding the more general case is probably hopeless!

Discussion

I stress that SOMA is really an idea about goal system content, and not an AGI design in itself. SOMA could be implemented in the context of a variety of different AGI designs, including for instance the open-source OpenCog approach.

It is not hard to envision ways of prototyping SOMA given current technology, using existing machine learning and reasoning algorithms, in OpenCog or otherwise. Of course, such prototype experiments would give limited direct information about the behavior of SOMA for superhuman AGI systems -- but they might give significant indirect information, via helping lead us to general mathematical conclusions about SOMA dynamics.

Altogether, my feeling is that "CAV + Predictably Beneficial AGI" is on the frontier of current mathematics and science. They pose some very difficult problems that do, however seem potentially addressable in the near future via a combination of mathematics and computational experimentation. On the other hand, I have a less clear idea of how to pragmatically do research work on CEV or the creation of practically feasible yet provably physically Friendly AGI.

My hope in proposing these ideas is that they (or other similar ideas conceived by others) may serve as a sort of bridge between real-world AGI work and abstract ethical considerations about the hypothetical goal content of superhuman AGI systems.

Coherent Aggregated Volition: A Method for Deriving Goal System Content for Advanced, Beneficial AGIs

2010-03-12T13:45:00.014-05:00

One approach to creating a superhuman AGI with a reasonably high likelihood of being beneficial to humans is to separate "goal system structure and dynamics" from "goal system content." One then has three problems:

Create an AGI architecture that makes it very likely the AGI will pursue its goal-system content in a rational way based on the information available to it
Create a goal system whose structure and dynamics render it likely for the AGI to maintain the spirit of its initial goal system content, even as it encounters radically different environmental phenomena or as it revises its own ideas or sourcecode
Create goal system content that, if maintained as goal system content and pursued rationally, will lead the AGI system to be beneficial to humans

One potential solution proposed for the third problem, the goal system content problem, is Eliezer Yudkowsky's "Coherent Extrapolated Volition" (CEV) proposal. Roko Mijic has recently proposed some new ideas related to CEV, which place the CEV idea within a broader and (IMO) clearer framework. This blog post presents some ideas in the same direction, describing a variant of CEV called Coherent Aggregated Volition (CAV), which is intended to capture much of the same spirit as CEV, but with the advantage of being more clearly sensible and more feasibly implementable (though still very difficult to implement in full). In fact CAV is simple enough that it could be prototyped now, using existing AI tools.

(One side note before getting started: Some readers may be aware that Yudkowsky has often expressed the desire to create provably beneficial ("Friendly" in his terminology) AGI systems, and CAV does not accomplish this. It also is not clear that CEV, even if it were fully formalizable and implementable, would accomplish this. Also, it may be possible to prove interesting theorems about the benefits and limitations of CAV, even if not to prove some kind of absolute guarantee of CAV beneficialness; but the exploration of such theorems is beyond the scope of this blog post.)

Coherent Extrapolated Volition

In brief, Yudkowsky's CEV idea is described as follows:

In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

This is a rather tricky notion, as exemplified by the following example, drawn from the CEV paper:

Suppose Fred decides to murder Steve, but when questioned, Fred says this is because Steve hurts other people, and needs to be stopped. Let's do something humans can't do, and peek inside Fred's mind-state. We find that Fred holds the verbal moral belief that hatred is never an appropriate reason to kill, and Fred hopes to someday grow into a celestial being of pure energy who won't hate anyone. We extrapolate other aspects of Fred's psychological growth, and find that this desire is expected to deepen and grow stronger over years, even after Fred realizes that the Islets worldview of "celestial beings of pure energy" is a myth. We also look at the history of Fred's mind-state and discover that Fred wants to kill Steve because Fred hates Steve's guts, and the rest is rationalization; extrapolating the result of diminishing Fred's hatred, we find that Fred would repudiate his desire to kill Steve, and be horrified at his earlier self.

I would construe Fred's volition not to include Fred's decision to kill Steve...

Personally, I would be extremely wary of any being that extrapolated my volition in this sort of manner, and then tried to impose my supposed "extrapolated volition" on me, telling me "But it's what you really want, you just don't know it." I suppose the majority of humans would feel the same way. This point becomes clearer if one replaces the above example with one involving marriage rather than murder:

Suppose Fred decides to marry Susie, but when questioned, Fred says this is because Susie is so smart and sexy. Let's do something humans can't do, and peek inside Fred's mind-state. We find that Fred holds the verbal moral belief that sex appeal is never an appropriate reason to marry, and Fred hopes to someday grow into a celestial being of pure energy who won't lust at all. We extrapolate other aspects of Fred's psychological growth, and find that this desire is expected to deepen and grow stronger over years, even after Fred realizes that the Islets worldview of "celestial beings of pure energy" is a myth. We also look at the history of Fred's mind-state and discover that Fred wants to marry Susie because Susie reminds him of his mother, and the rest is rationalization; extrapolating the result of diminishing Fred's unconscious sexual attraction to his mother, we find that Fred would repudiate his desire to marry Susie, and be disgusted with his earlier self.

I would construe Fred's volition not to include Fred's decision to marry Susie...

Clearly, the Yudkowskian notion of "volition" really has little to do with "volition" as commonly construed!!

While I can see the appeal of extrapolating Fred into "the Fred that Fred would like to be," I also think there is a lot of uncertainty in this process. If Fred has inconsistent aspects, there may be many possible future-Freds that Fred could evolve into, depending on both environmental feedback and internal (sometimes chaotic) dynamics. If one wishes to define the coherent extrapolated Future-Fred as the average of all these, then one must choose what kind of average to use, and one may get different answers depending on the choice. This kind of extrapolation is far from a simple matter -- and since "self" is not a simple matter either, it's not clear that current-Fred would consider all or any of these Future-Freds as being the same person as him.

In CAV as described here, I consider "volition" in the more typical sense -- rather than in the sense of Yudkowskian "extrapolated volition" -- as (roughly) "what a person or other intelligent agent chooses." So according to my conventional definition of volition, Fred's volition is to kill Steve and marry Susie.

Mijic's List of Desirable Properties

Roko Mijic has posited a number of general "desirable properties" for a superintelligence, and presented CEV as one among many possible concrete possibilities instantiating these principles:

Meta-algorithm: Most goals the AI has will be harvested at run-time from human minds, rather than explicitly programmed in before run-time.
Factually correct beliefs: Using the AI's superhuman ability to ascertain the correct answer to any factual question in order to modify preferences or desires that are based upon false factual beliefs.
Singleton: Only one superintelligence is to be constructed, and it is to take control of the entire future light cone with whatever goal function is decided upon.
Reflection: Individual or group preferences are reflected upon and revised, in the style of Rawls' reflective equilibrium.
Preference aggregation: The set of preferences of a whole group are to be combined somehow.

My own taste is that reflection, preference aggregation and meta-algorithm-ness are good requirements. The "singleton" requirement seems to me something that we can't know yet to be optimal, and don't need to decide at this point.

The "factually correct beliefs" requirement also seems problematic, if enforced too harshly, in the sense that it's hard to tell how a person, who has adapted their beliefs and goals to certain factually incorrect beliefs, would react if presented with corresponding correct beliefs. Hypothesizing that a future AI will be able to correctly make this kind of extrapolation is not entirely implausible, but certainly seems speculative. After all, each individual's reaction to new beliefs is bound to depend on the reactions of others around them, and human minds and societies are complex systems, whose evolution may prove difficult for even a superintelligence to predict, given chaotic dynamics and related phenomena. My conclusion is that there should be a bias toward factual correctness, but that it shouldn't be taken to override individual preferences and attitudes in all cases. (It's not clear to me whether this contradicts Mijic's perspective or not.)

Coherent Aggregated Volition

What I call CAV is an attempt to capture much of the essential spirit of CEV (according to my own perspective on CEV), in a way that is more feasible to implement than the original CEV, and that is prototype-able now in simplified form.

Use the term "gobs" to denote "goal and belief set" (and use "gobses" to denote the plural of "gobs"). It is necessary to consider goals and beliefs together, rather than just looking at goals, because real-world goals are typically defined in terms whose interpretation depends on certain beliefs. Each human being or AGI may be interpreted to hold various gobses to various fuzzy degrees. There is no requirement that a gobs be internally logically consistent.

A "gobs metric" is then a distance on the space of gobses. Each person or AI may also agree with various gobs metrics to various degrees, but it seems likely that individuals' gobs metrics will differ less than their gobses.

Suppose one is given a population of intelligent agents -- like the human population -- with different gobses. Then one can try to find a gobs that maximizes the four criteria of

logical consistency
compactness
average similarity to the various gobses in the population
amount of evidence in support of the various beliefs in the gobs

The use of a multi-extremal optimization algorithm to seek a gobs defined as above is what I call CAV. The "CAV" label seems appropriate since this is indeed a system attempting to achieve both coherence (measured via compactness + consistency) and an approximation to the "aggregate volition" of all the agents in the population.

Of course there are many "free parameters" here, such as

how to carry out the averaging (for instance one could use a p'th-power average with various p values)
what underlying computational model to use to measure compactness (different gobs may come along with different metrics of simplicity on the space of computational models)
what logical formalism to use to gauge consistency
how to define the multi-extremal optimization: does one seek a Pareto optimum?; does one weight the different criteria and if so according to what weighting function?
how to measure evidence
what optimization algorithm to use

However, the basic notion should be clear, even so.

If one wants to take the idea a step further, one can seek to use a gobs metric that maximizes the criteria of

compactness of computational representation
average similarity to the gobs metrics of the minds in the population

where one must then assume some default similarity measure (i.e.m etric) among gobs metrics. (Carrying it further than this certainly seems to be overkill.)

One can also use a measure of evidence defined in a similar manner, via combination of a compactness criterion and an average similarity criterion. These refinements don't fundamentally change the nature of CAV.

Relation between CEV and CAV

It is possible that CEV, as roughly described by Yudkowsky, could lead to a gobs that would serve as a solution to the CAV maximization problem. However, there seems no guarantee of this. It is possible that the above maximization problem may have a reasonably good solution, and yet Yudkowskian CEV may still diverge or lead to a solution very far from any of the gobses in the population.

As a related data point, I have found in some experiments with the PLN probabilistic reasoning system that if one begins with a set of inconsistent beliefs, and attempts to repair it iteratively (by replacing one belief with a different one that is more consistent with the others, and then repeating this process for multiple beliefs), one sometimes arrives at something VERY different from the initial belief-set. And this can occur even if there is a consistent belief set that is fairly close to the original belief-set by commonsensical similarity measures. While this is not exactly the same thing as CEV, the moral is clear: iterative refinement is not always a good optimization method for turning inconsistent belief-sets into nearby consistent ones.

Another, more qualitative observation, is that I have the uneasy feeling CEV seeks to encapsulate the essence of humanity in a way that bypasses the essential nature of being human...

CEV wants to bypass the process of individual and collective human mental growth, and provide a world that is based on the projected future of this growth. But, part of the essence of humanity is the process of growing past one's illusions and shortcomings and inconsistencies.... Part of Fred's process-of-being-Fred is his realizing on his own that he doesn't really love Susie in the right way ... and, having the super-AI decide this for him and then sculpt his world accordingly, subtracts a lot of Fred's essential humanity.

Maybe the end-state of resolving all the irrationalities and inconsistencies in a human mind (including the unconscious mind) is something that's not even "human" in any qualitative, subjective sense...

On the other hand, CAV tries to summarize humanity, and then would evolve along with humanity, thus respecting the process aspect of humanity, not trying to replace the process of humanity with its expected end-goal... And of course, because of this CAV is likely to inherit more of the "bad" aspects of humanity than CEV -- qualitatively, it just feels "more human."

Relation of CAV to Mijic's Criteria

CAV appears to adhere to the spirit of Mijic's Meta-algorithm, Factual correctness and Preference aggregation criteria. It addresses factual correctness in a relatively subtle way, differentiating between "facts" supported by different amounts of evidence according to a chosen theory of evidence.

CAV is independent of Mijic's "singleton" criterion -- it could be used to create a singleton AI, or an AI intended to live in a population of roughly equally powerful AIs. It could also be used to create an ensemble of AIs, by varying the various internal parameters of CAV.

CAV does not explicitly encompass Mijic's "reflection" criterion. It could be modified to do so, in a fairly weak way, such as replacing the criterion

average similarity to the various gobses in the population

with

average similarity to the various gobses displayed by individuals in the population when in a reflective frame of mind

This might be wise, as it would avoid including gobses from people in the throes of rage or mania. However, it falls far short of the kind of deep reflection implied in the original CEV proposal.

One could also try to teach the individuals in the population to be more reflective on their goals and beliefs before applying CAV. This would surely be a good idea, but doesn't modify the definition of CAV, of course.

Prototyping CAV

It seems that it would be possible to prototype CAV in a fairly simple way, by considering a restricted class of AI agents, for instance OpenCog-controlled agents, or even simple agents whose goals and beliefs are expressed explicitly in propositional-logic form. The results of such an experiment would not necessarily reflect the results of CAV on humans or highly intelligent AGI agents, but nevertheless such prototyping would doubtless teach us something about the CAV process.

Discussion

I have formulated a method for arriving at AGI goal system content, intended to serve as part of an AGI system oriented beneficially toward humans and other sentient beings. This method is called Coherent Aggregated Volition, and is in the general spirit of Yudkowsky's CEV proposal as understood by the author, but differs dramatically from CEV in detail. It may be understood as a simpler, more feasible approach than CEV to fulfiling Mijic's criteria.

One thing that is apparent from the above detailed discussion of CAV is the number of free parameters involved. We consider this a feature not a bug, and we strongly suspect that CEV would also have this property if it were formulated with a similar degree of precision. Furthermore, the parameter-dependence of CEV may seem particularly disturbing if one considers it in the context of one's own personal extrapolated volitions. Depending on the setting of some weighting parameter, CEV may make a different decision as to whether Fred "really" wants to marry Susie or not!!

What this parameter-dependence means is that CAV is not an automagical recipe for producing a single human-friendly goal system content set, but rather a general approach that can be used by thoughtful humans or AGIs to produce a family of different human-friendly goal system content sets. Different humans or groups applying CAV might well argue about the different parameters, each advocating different results! But this doesn't eliminate the difference between CAV and other approaches to goal system content that don't even try to achieve broad-based beneficialness.

Compared to CEV, CAV is rather boring and consists "merely" of a coherent, consistent variation on the aggregate of a population's goals and beliefs, rather than an attempt to extrapolate what the members of the population in some sense "wish they wanted or believed." As the above discussion indicates, CAV in itself is complicated and computationally expensive enough. However, it is also prototype-able; and we suspect that in the not too distant future, CAV may actually be a realistic thing to implement on the human-population scale, whereas we doubt the same will be true of CEV. Once the human brain is well understood and non-invasively scannable, then some variant of CAV may well be possible to implement in powerful computers; and if the projections of Kurzweil and others are to be believed, this may well happen within the next few decades.

Returning to the three aspects of beneficial AGI outlined at the start of this essay: I believe that development of the currently proposed OpenCog design has a high chance of leading to an AGI architecture capable of pursuing its goal-system content in a rational way; and this means that (in my world-view) the main open question regarding beneficial AGI pertains to the stability of goal systems under environmental variation and systemic self-modification. I have some ideas for how to handle this using dynamical systems theory, but these must wait for a later post!

Siri, the new iPhone "AI personal assistant": Some useful niche applications, not so much AI

2010-02-06T23:23:00.012-05:00

Today I tried out Siri, the new AI "personal assistant" app for the iPhone. It has some very smart people behind it, and is based on some code and ideas from the DARPA-funded CALO project. Siri's earlier prototype version impressed me with its integration of dialogue and maps, so I was eager to check it out.

The Siri website says:

Just like a real assistant, Siri understands what you say, accomplishes tasks for you and adapts to your preferences over time.

It also describes Siri using metaphors of human learning, e.g. "like a child taking its first steps" ....

Ahem....

You may want to scroll to the end of this post, and read my dialogue with Siri, before reading the rest of what I have to say about the app.

This review has been edited in response to some comments (which you'll see below this post) by Dag, one of the Siri creators. If you're curious to see the original version of my review, it's here. There are no huge changes but I hope this revised version is an improvement.

This is the first release, and one doesn't want to judge the whole Siri project based on a first impression. But all I can report on now is my reaction to the product I just downloaded on to my phone and chatted with....

Two Perspectives on Siri

Before giving my detailed comments, I'd like to distinguish two different perspectives on Siri

Considered as a freebie iPhone app, is it funky? Is it worth downloading and playing with? Might it be useful for some purposes?
How well does it live up to the "AI Personal Assistant" label, and the description of being "like a human assistant", "like a child taking its first steps", etc.

Plenty of others can assess Siri as a freebie iPhone app as well or better than I can, so I'll make a few comments in that regard, but focus most of my attention here on the AI aspect, since that's my own area of expertise.

Overall, my take is that

Indeed, this version of Siri may be very useful for carrying out a very limited set of very specific functionalities
It's not anything like a real assistant; and worse than that, its attempts to really understand anything you say seem very limited and domain-specific at this point
The basic "chatbot" functionality seems unnecessarily crude and quirky

As an AI developer I'm well aware that sometimes you can make mediocre (or worse) products or demos based on deeply powerful technology. So I'm open to the possibility that there is some profound or at least interesting tech underlying Siri. But, to be quite blunt, I was unable to find it via playing with the product for an hour or so.

Siri from an AI Perspective

Looking at Siri from the perspective of someone who has built a bunch of AI systems, including chatbots and more serious natural language processing and reasoning systems, what I see here is:

a rather crude keyword based chatbot (i.e. crude even by the standards of keyword based chatbots), without much attempt at dialogue management
straightforward, rule-based integration with a very small set of knowledge bases (about restaurants and movies, for instance) and with a map engine
straightforward integration with TrueKnowledge for answering of factual questions
decent speech-to-text with a very nice interactive interface

What surprised me most was the crudity of the dialogue management, which you'll see in the transcript below, of my initial conversation with Siri. So often Siri's responses had nothing to do with the questions I asked.

And Siri's persistence of information between questions is rudimentary and awkward. Once you ask one question about New York, it pretty much assumes all your subsequent questions are about New York ... but it doesn't understand linguistic references to previous queries, not even simple ones.

But Is Siri Useful?

But what about the practical aspect? Is Siri useful as a virtual assistant? I suppose I might use it to find restaurants or movies, or to check flight status. And just the other day, in the midst of a conversation in the car with the kids, I wanted to know Hitler's birth year, and I asked Wikipedia on my iPhone -- it would have been nicer to ask Siri instead.

So, yeah, for a few specific functionalities, where Siri's language engine and database integration are well-tuned -- yeah, it may be genuinely useful.

But my impression is the useful functionality is really VERY narrow and brittle. If you go even slightly beyond what the application has been specifically tweaked for, the results seem to be useless and annoying.

As a single example, consider the following snippet from my first conversation with Siri, given in full at the end of this post:

Ben: What is Kate Braverman's latest book?
Siri: OK, here are some businesses named "Kate" a few miles from here

This is really an unnecessary gaffe', but it's not exceptional; Siri, in its current version, does that sort of thing quite frequently. It makes this mistake because the query is about books and authors, rather than about stuff it's tuned for: restaurants, movies, flights, TrueKnowledge facts. And even for some things it's tuned for, like flights, the results are often quite weird and confusing, as you'll see in the example dialogue below.

How about the speech-to-text? (Supplied by Nuance, and performed on a server not on the phone.) It's so-so.... Which may be a great achievement technically given the quality of the iPhone's mike -- but still, it's only so-so.

The iterative graphical interface for speech-to-text is GREAT -- being able to review Siri's interpretations of your speech and correct them on the phone before they're sent to the server is very nice. But it makes enough mistakes that, all in all, using its speech-to-text is many times slower for me than using the iPhone keyboard.

I can see some genuine niche applications for the current Siri version: restaurant and movie location, flight status checking, fact searching, and maybe a few other similar applications, while driving. Or while not driving, for users who aren't comfortable typing.

This is all very well, but it's a far far cry from being like a human assistant, right?

Does Siri Understand?

The website warns us that this is an early-stage product:

Siri is young and, like a child taking its first steps, may be awkward at times. Siri may occasionally misunderstand things you ask it to do even within its range of understanding.

but IMO, the comparison with a child is inappropriate. Most of the mistakes Siri makes are not mistakes of misunderstanding. They are mistakes of not even trying to understand -- mistakes of replying in the manner of a simplistic chatbot acting on keyword cues.

If I had an iPhone app that made mistakes of genuine misunderstanding, like a child, I'd devote time to teaching it regardless of whether it assisted me in any way. In the case of Siri, I don't get the feeling of any intelligence or learning going on.

Dag, in his comment on my first version of this review, noted that in some contexts Siri does try to understand, e.g. if you ask it "Book me a table for two at Zibibbo's" it understands that "book" refers to the making of reservations rather than the kind of book you read. Fair enough -- but after reading his comment I played around with Siri a little more and my impression is that its "understanding" of this sort is extremely specialized and focused on a handful of applications like making restaurant and movie reservations. Of course, one could argue that by scaling up this kind of specialized understanding a few hundred thousand times, one will achieve something really intelligent -- but

I tend to doubt it, because I think intelligence has more to do with the ability to learn to handle new domains, than the possession of hand-coded rules allowing "understanding" in particular domains
Even if one does believe humanlike intelligence is a patchwork of domain-specific rule-sets, then one must admit that the fraction of humanlike intelligence displayed by an application like Siri is rather miniscule. If one believes this kind of model of human intelligence, one should be building Cyc, not Siri (and the difficulties of that kind of AI approach are well known)

The current version is, for better or worse, a simplistic tool with a nice interface and a very, very limited scope. In a sense it does understand some things, but only in the very specialized domains in which its "understanding" was very specifically programmed.

Perhaps later versions will add enough functionality to constitute a more generally useful "assistant." But in my view, without some fundamentally different (and more intelligent) approach to dialogue management, the product is not likely grow into anything but an assemblage of a few dozen specialized information-gathering widgets glued together by a chatbot. I could be wrong -- it's happened before! -- but I'm just calling it as I see it....

I read Nova Spivack's very insightful discussion on Siri a number of months ago, and studied the Siri prototype fairly carefully, and based on that prior experience I actually expected more from the first release. I hoped for a little more sense of general-scope humanlike understanding, of there being an "assistant with a personality" there. Nope. Maybe the next version will have some fundamentally different technology inside it ... one can always hope.....

Apologies if this review is a bit harsh -- but as I clarified from the start, I'm reviewing Siri not just as an iPhone app, but relative to the rhetoric associated with it about being "like a child taking its first steps" and "just like a human assistant." If Siri were merely marketed as an iPhone app with a few interesting niche uses, I probably wouldn't bother to write a blog post about it.... But I've devoted much of my life to the quest to make AI systems that actually learn like children, and ultimately will display intelligence similar to and then transcending that of adult humans. The quest to make humanlike AI is a serious thing. Siri just doesn't feel to me like any kind of step along the path to serious AI systems, and I don't really like it when somebody's marketing department uses "real AI" as a marketing slogan for a product (even if a nice one in some ways) that actually has nothing to do with humanlike general intelligence.

A Look at Some Others Users Reactions

Encouraged by Dag's comment on the original version of this review, I looked at some tweets on Siri by "ordinary users" not biased by an AI background, and here are some examples, which I tried to choose in a genuinely fair-minded way:

turrean Playing with new iPhone app called "Siri Assistant." you can say, "Movies nearby" and that's what it finds. Feel like I'm on Star Trek.

Tito8181 @laur3453 finially you should download "Siri" for iPhone. It's like your own personal assistant! I love it! It's completely free

Shusmo @basemaggad Siri,launched today as a free iPhone app, is a virtual personal assistant that amazingly resembles..an actual personal assistant

aneesha Siri Brings Artificial Intelligence to the iPhone http://j.mp/9pO6Pd

tomweishaar #siri .Not perfect but interesting, I asked "when is kickoff for the super bowl football game?" Siri referred me 2 bowling ally's in my area

tomweishaar #siri same question using Google Mobile on iPhone, "what time does super bowl start?" got answer right away. Nuance engine 4 both services

What I see here is that some users are indeed impressed by the voice recognition capability (even though it's not groundbreaking by any means) and by the ability to respond usefully to some simple queries like "Movies nearby." That's great! On the other hand, some other users note that it makes simple mistakes like thinking "super bowl" is about bowling, some of which other tools like Google Mobile don't make.

Also, I note that the meme of "Siri as AI" is frequent in the tweetosphere and the media generally ... which makes me feel justified in reviewing it as an AI system and not just as a funky iPhone app. If you're going to market it as an AI system that's just like a human personal assistant and is learning like a human child, then you should expect it to be assessed that way, at least to a certain extent, right? The dazzle and mystique of AI cuts both ways: it helps lend some sex appeal to your product, but also leaves you open to being held to the standards of artificial intelligence ;-)

My First Chat with Siri

Finally, to finish off this post, here's the transcript of my first conversation with Siri, with some comments interspersed and prepended by "COMMENT." I talked to Siri some more after this conversation, but it didn't reveal any more intelligence or anything particularly surprising ... I think this transcript will be more than enough to give you the idea.

Siri: Hello, I'm Siri, I'm your new personal assistant

Ben: Where is New York?

Siri: [shows map of New York]

Ben: Do computers have dicks?

Siri translates to: Do computers have Dick's?

Ben OKs the translation (just for fun)

Siri: OK, here are some computer stores close to New York, NY: [gives list]

COMMENT: OK, the maps integration is nice. And it's not a bad answer as there are some computer stores named Dick's.

Ben: Do computers have penises?

Siri: OK, I found these computer stores close to New York, NY: [gives list]

COMMENT: Total lack of any attempt to understand the question

Ben: Do you have a brain?

[The speech-to-text took 4 tries to get this, but finally did]

Siri: OK, here are some local businesses close to New York, NY: [gives list]

COMMENT: What???

Ben: Where can I buy a new head?

Siri: Here are movies that are new to theaters playing near New York: [gives list]

COMMENT: OK, it understood "new" and used a default for the rest...

Ben: Will you call Laising for me?

Siri translates this to: Will you call great thing for me?

After many tries Siri cannot get the Chinese name "Laising" so Ben OKs the wrong translation just for fun

Siri: Here are the most popular movies playing near New York: [gives list]

COMMENT: Where the heck did that come from???

Ben: I want to buy a new phone

Siri: Here are the the most popular movies that are new to theaters playing near New York: [gives list]

Ben: I do not want to see a movie. I want to buy a new phone.

Siri: Here are the most popular movies that are new to theaters playing near New York: [gives list]

COMMENT: Again, a total lack of understanding ... no attempt to find the arguments of the verb "want" or understand the meaning of "not." Just statistical word-matching (on "new") used in a crude way.

Ben: Where can I find Mongolian food?

Siri: OK, I found these Mongolian restaurants a short drive from New York, NY

COMMENT: Useful. Again, the maps integration is nice, and it knows that restaurants are associated with food

Ben: Where can I find Mongolian women?

Siri: OK, here are some Mongolian restaurants a short drive from New York, NY

COMMENT: Oops, maybe it didn't know restaurants were associated with food. Of course, Mongolian restaurants ARE a good place to find Mongolian women, but it didn't know that. It just associated "Mongolian" with "restaurants" by default because it doesn't know anything else Mongolian in New York, and it's biased toward location queries.

Ben: What is the capital of Mongolia?

Siri: I think I learned that in grad school ... I found answer for you: [answer in box: Ulaanbaatur, the city in Ulaanbaatar, Mongolia (source:TrueKnowledge.com)]

COMMENT: Good, TrueKnowledge is integrated in the obvious way

Ben: How many people live there?

Siri: I think I learned that in grad school.... I'm not sure I can help you with that yet, but here's what a search engine has to say: [points to results of a Web search, which are useless, e.g. "Math Lesson: How many people...", "Is there bad people in Los Angeles...", etc.]

COMMENT: It can't deal with "there." Very weak understanding of dialogue.

Ben: How many people live in Ulaanbaator, Mongolia?

Siri: I found an answer for you: [in a box, 1067472]

COMMENT: But it actually knew the answer, via TrueKnowledge again...

Ben: How many of those people are men?

Siri: I'm not sure of the answer, but here's what a search engine has to say: [points to results of a Web search, which are useless...]

Ben: What is Kate Braverman's latest book?

Siri: OK, here are some businesses named "Kate" a few miles from here

COMMENT: Unnecessarily confused

Ben: I want to buy a book written by Kate Braverman

Siri: Sorry, I can't get you information on books yet, but I can help you look on the Web [points to a search engine, and to Amazon's website]

Ben: What can you tell me about Kate Braverman?

Siri: I can check on flights for you, but I need a little more information. I need to know at least one airport code, or an ariline and a flight number.

COMMENT: Unnecessarily confused again.

Ben: Are airplanes flying from Reagan National Airport now, or have they been delayed because of the snow?

Siri: Hmmm. I couldn't find any direct flights from RLA for February 8. For now, I can only find the status for direct flights. Soon I'll be able to book flights, and show connecting flights. [This is confusing since it is not Feb 8 now, but rather Feb 6]

COMMENT: That was exactly the kind of question I would like to be able to ask a virtual assistant and get an answer. But Siri badly botched it.

Ben: What is RLA?

[this took 3 tries for the speech to text]

Siri: I couldn't find any direct flights from RLA for February 8. For now, I can only find the status for direct flights. Soon I'll be able to book flights, and show connecting flights.

COMMENT: More unnecessary confusion, as my question was very simple and direct and didn't merit such a complex answer

Ben: I thought Reagan National Airport was DCA, not RLA

[this took 3 tries for the speech to text]

Siri: I'm not sure what you are asking, Ben. One of these, perhaps [points to empty list]

Ben: Who programmed you?

[after 6 tries the speech-to-text couldn't get it, and I gave up]

China Ascendant? (a comment on Robin Hanson's comment on...)

2009-12-20T19:13:00.005-05:00

This blog post is an edited version of a comment I made on Robin Hanson's recent China Ascendant post on the Overcoming Bias blog. So, read Robin's post before reading this one!

Also, it is best read as a sort of post-script to my recent article on the Chinese Singularity in H+ Magazine. So maybe you should read that article first too ;-)

Ultimately, it may not be so important whether the US or China or India or Europe leads the advance of science and technology during the next decades.

Certainly, if you're a Singularitarian like me -- the Singularity is about the fate of mind and humanity, not the fate of nations ... and if/when it comes, it will quickly bring us beyond the state where national boundaries are a big deal.

But in current practical terms, the "where" question is an interesting one.... Especially, if a lot of the relevant developments are going to happen outside the Western world, this is worth knowing because it's going to affect a lot of decisions people have to make.

So, to get to the topic of Robin Hanson's blog post: China ascendant ???

My answer to that question is always: Maybe.

In his post, Robin makes the statement:

If China continues to outgrow the West, it will likely be because they do a few things very right, as did the West before.

The point I want to make here is a simple one: One of the things China is doing much better than the US, these days, is thinking medium-term and long-term rather than just short-term.

Perhaps long-range planning will be one of the "few things" China does "very right," to use Robin's language.

China is planning decades ahead, in their technology and science development, in their energy and financial policies, and many other areas as well.

Whereas in the US, we seem to be mired in a "next quarter" or "next election" mentality.

However, the matter isn't as simple as it seems...

It's interesting to observe that the American system sometimes does great mid-range planning accidentally (or, to use a more charitable word: implicitly)...

For instance, the dot-com boom seems kinda stupid in hindsight (trust me; I played my own small part in the stupidity!) ... but on closer inspection, a lot of the "wasted" venture $$ that went into the dot-com boom funded

the build-out of Internet infrastructure of various kinds
the prototyping of technologies that later became refined and successful.

Those VCs would not have funded infrastructure buildout or technologically prototyping explicitly, but they funded it accidentally, er, implicitly.

So in this case, the US system planned things 10 years in advance implicitly, without any one person explicitly trying to do so.

We can't explain the dot-com boom example by simplistic "market economics" arguments -- because on average, the investment of time and $$ in the dot-com boom wasn't worth it for the participants (and they probably weren't rational to expect that it would be worth it for them). Most of their work and $$ ultimately went to benefit people other than those who invested in the boom. But we can say that, in this case, the whole complex mess of the US economic system did implicitly perform some effective long-range planning.

Yet, this kind of implicit long-term planning has its limits, and seems to be failing in key areas like my own research area of AGI. The US is shortchanging AGI research badly compared to Europe as well as Asia, because our economic system is biased toward shortsightedness.

There are strong arguments that long-range state-driven planning and funding has benefited developing countries -- Singapore, South Korea and Brazil being some prime examples. In these cases, it supported the development of infrastructures that probably would not have developed in a less state-centric arrangement like we have in the US.

So, one interesting question is whether explicit or implicit long-range planning is going to be more effective in the next decades as technology and science continue to accelerate (or, to put the question more honestly but more confusingly: what COMBINATIONS of explicit and implicit long-range planning are going to work better)?

My gut feel is that the "mainly implicit" approach isn't going to do it. I think that if the US government doesn't take a strong hand in pushing for (and funding) adventurous, advanced technology and science development, then China will pull ahead of us within the next decades. I don't trust the US "market oligarchy" system to implicitly carry out the needed long-range planning.

The reason I have this feeling is that, in one advanced, accelerating technology area after another, I see a contradiction between the best path to short-term financial profit and the best path to medium-term scientific progress. For instance,

In AI, the quest for short-term profits biases toward narrow AI, yet the best medium-term research path is to focus on AGI
In nanotech, the best medium-term research path is Drexler's path which works toward molecular assemblers, but the best path to short-term profits is to focus on weak nanotechnology like most of the venture-funded "nano" firms are doing now
In life extension, the best short-term path is to focus on remedies for aging-related diseases, but the best medium-term path is either to understand the fundamental mechanisms of aging, or to work on radical cures to aging-related damage as Aubrey de Grey has suggested
In robotics, the path to short-term profit is industrial robots or Roombas, but the path to profound medium-term progress is more flexibly capable autonomous (humanoid or wheeled) mobile robots with capable hands, sensitive skin, etc. (and note how all the good robots are made in Japan, Korea or Europe these days, with government funding)

In area after area of critical technology and science, the short-term profit focus is almost certainly going to mislead us. What is needed is the ability to take the path NOW that is going to yield the best results 1-3 decades from now. I am very uncertain whether such an ability exists in the US, and it seems more clear to me that it exists in China.

The Chinese government is trying to figure out how to combine the explicit planning of their centralized agencies, with the implicit planning of the modern market ecosystem. They definitely don't have it figured out yet. But my feel is that, even if they make a lot of stupid mistakes as they feel their way into the future, their greater propensity for thinking in terms of DECADES rather than years or quarters, is going to be a huge advantage for them....

China has a lot of disadvantages compared to the US, including

a less rich recent science and engineering tradition
an immature ecosystem for academic/business collaboration
a culture that sometimes discourages effective brainstorming and teamwork
a less international scientific community
an unfortunate habit of blocking parts of the Internet (which doesn't prevent Chinese researchers from getting the world's scientific knowledge, but does prevent them from participating fully in the emerging Global Brain as represented by Web 2.0 technologies like Twitter, Facebook and so forth)

However, it may be that all these disadvantages are outweighed by the one big advantage of being better at long-range planning.

As Robin points out, dramatic success is often a matter of getting just a few things VERY RIGHT.

Dialoguing with the US Military on the Ethics of Battlebots

2009-12-15T17:13:00.006-05:00

Today (as a consequence of my role in the IEET), I gave a brief invited talk at the National Defense University, in Washington DC, about the ethics of autonomous robot missiles and war vehicles and "battlebots" (my word, not theirs ;-) in general....

Part of me wanted to bring a guitar and serenade the crowd (consisting perhaps 50% of uniformed officers) with "Give Peace a Chance" by John Lennon and "Masters of War" by Bob Dylan ... but due to the wisdom of my 43 years of age I resisted the urge ;-p

Anyway the world seems very different than it did in the early 1970s when I accompanied my parents on numerous anti-Vietnam-war marches. I remain generally anti-violence and anti-war, but my main political focus now is on encouraging a smooth path toward a positive Singularity. To the extent that military force may be helpful toward achieving this end it has to be considered as a potentially positive thing....

My talk didn't cover any new ground (to me); after some basic transhumanist rhetoric I discussed my notion of different varieties of ethics as corresponding to different types of memory (declarative ethics, sensorimotor ethics, procedural ethics, episodic ethics, etc.), and the need for ethical synergy among different ethics types, in parallel with cognitive synergy among different memory/cognition types. For the low-down on this see a previous blog post on the topic.

But some of the other talks and lunchroom discussions were interesting to me, as the community of military officers is rather different from the circles I usually mix in...

One of the talks before mine was a prerecorded talk (robo-talk?) on whether it's OK to make robots that decide when/if to kill people, with the basic theme of "It's complicated, but yeah, sometimes it's OK."

(A conclusion I don't particularly disagree with: to my mind, if it's OK for people to kill people in extreme circumstances, it's also OK for people to build robots to kill people in extreme circumstances. The matter is complicated, because human life and society are complicated.)

(As the hero of the great film Kung Pow said, "Killing is bad. Killing is wrong. Killing is badong!" ... but, even Einstein had to recant his radical pacifism in the face of the extraordinary harshness of human reality. Harshness that I hope soon will massively decrease as technology drastically reduces material scarcity and gives us control over our own motivational and emotional systems.)

Another talk argued that "AIs making lethal decisions" should be outlawed by international military convention, much as chemical and biological weapons and eye-blinding lasers are now outlawed.... One of the arguments for this sort of ban was that, without it, one would see an AI-based military arms race.

As I pointed out in my talk, it seems that such a ban would be essentially unenforceable.

For one thing, missiles and tanks and so forth are going to be controlled by automatic systems of one sort or another, and where the "line in the sand" is drawn between lethal decisions and other decisions, is not going to be terribly clear. If one bans a robot from making a lethal decision, but allows it to make a decision to go into a situation where making a lethal decision is the only rational choice, then what is one really accomplishing?

For another thing, even if one could figure out where to draw the "line in the sand," how would it possibly be enforced? Adversary nations are not going to open up their robot control hardware and software to each other, to allow checking of what kinds of decisions robots are making on their own without a "human in the loop." It's not an easy thing to check, unlike use of nukes or chemical or biological weapons.

I contended that just as machines will eventually be smarter than humans, if they're built correctly they'll eventually be more ethical than humans -- even according to human ethical standards. But this will require machines that approach ethics from the same multiple perspectives that humans do: not just based on rules and rational evaluation, but based on empathy, on the wisdom of anecdotal history, and so forth.

There was some understandable concern in the crowd that, if the US held back from developing intelligent battlebots, other players might pull ahead in that domain, with potentially dangerous consequences.... With this in mind, there was interest in my report on the enthusiasm, creativity and ample funding of the Chinese AI community these days. I didn't sense much military fear of China itself (China and the US are rather closely economically tied, making military conflict between them unlikely), but there seemed some fear of China distributing their advanced AI technology to other parties that might be hostile.

I had an interesting chat with a fighter pilot, who said that there are hundreds of "rules of engagement" to memorize before a flight, and they change frequently based on political changes. Since no one can really remember all those rules in real-time, there's a lot of intuition involved in making the right choices in practice.

This reminded me of a prior experience making a simulation for a military agency ... the simulated soldiers were supposed to follow numerous rules of military doctrine. But we found that when they did, they didn't act much like real soldiers -- because the real soldiers would deviate from doctrine in contextually appropriate ways.

The pilot drew the conclusion that AIs couldn't make the right judgments because doing so depends on combining and interpreting (he didn't say bending, but I bet it happens too) the rules based on context. But I'm not so sure. For one thing, an AI could remember hundreds of rules and rapidly apply them in a particular situation -- that is, it could do a better job of declarative-memory-based battle ethics than any human. In this context, humans compensate for their poor declarative memory based ethics [and in some cases transcend declarative memory based ethics altogether] with superior episodic memory based ethics (contextually appropriate judgments based on their life experiences and associated intuitions). But, potentially, an AI could combine this kind of experiential judgment with superior declarative ethical capability, thus achieving a better overall ethical functionality....

One thing that was clear is that the US military is taking the diverse issues associated with battle AI very seriously ... and soliciting a variety of opinions from those all across the political spectrum ... even including out-there transhumanists like me. This sort of open-ness to different perspectives is certainly a good sign.

Still, I don't have a great gut feeling about superintelligent battlebots. There are scenarios where they help bring about a peaceful Singularity and promote overall human good ... but there are a lot of other scenarios as well.

My strong hope is that we can create peaceful, benevolent, superhumanly intelligent AGI before smart battlebots become widespread.

My colleagues and I -- among others -- are working on it ;-)

100 neural net cycles to produce consciousness?

2009-12-02T07:17:00.003-05:00

This interesting article presents data indicating that it takes around half a second for an unconscious visual percept to become conscious (in the human brain)...

This matches well with Libet's result that there is a half-second lag between unconsciously initiating an action and consciously knowing you're initiating an action...

(Of course, what is meant by "consciousness" here is "consciousness of the reflective, language-friendly portion of the human mind" -- but I don't want to digress onto the philosophy of consciousness just now; that's not the point of this post ... I've done that in N prior blog posts ;-)

My Chinese collaborator ChenShuo pointed out that, combined with information about the timing of neural firing, this lets us estimate how much neural processing is needed to produce conscious perception.

As I recall, the firing of a single neuron's action potential takes around 5 milliseconds ... It takes maybe another 10-20 milliseconds after that for the neuron to be able to fire again (that's the "refractory period") .... Those numbers are not exact but I'm pretty sure they're the right order of magnitude...

So, the very rough estimate is 100 cycles in the neural net before consciousness, it would seem ;)

This fits with the view of consciousness in terms of strange attractors ... 100 cycles is often enough time for a recurrent net to converge to into an attractor basin ...

But of course the dynamics during those ~100 cycles is the more interesting story, and it's still obscure....

Is it really an attractor we have here, or "just" a nicely patterned transient? A terminal attractor a la Mikhail Zak's work, perhaps? Etc.

Enquiring minds want to know! (TM)

Dream of the Multiversal Cylinder

2009-11-16T20:52:00.003-05:00

(I usually reserve this blog for speculations on intellectual topics, but last night I had a dream that seemed sufficiently interesting to post here. So, here goes ;-) ....

In this dream, I moved to a strange foreign nation, and met a beautiful girl there whose ex-boyfriend was making her life very difficult, yet who she was still somehow attached to....

His martial arts expertise alarmed me, and so together with the mother of a friend who lived in this same strange place -- a very short, hunchbacked old lady who walked with a cane and wore a funny straw hat -- I went to a weird old-fashioned section of the city, where we did two things.

First, we paid some old white-bearded "witch doctor" to cast a magical spell on the ex-boyfriend, which caused him to forget having ever known the girl, haha.

Then, we went to a strange store full of ancient relics, and bought this cylindrical wooden container, which I was supposed to keep in my bedroom for good luck, but not to open.

The girl and I walked along the beach and the ex-boyfriend walked right past and showed no sign of recognizing her. This freaked her out a bit, and she asked me to have the spell undone on Dec. 21 2012.

Then I went back to my house, which I suddenly shared with the girl, and of course I had to open the wooden cylinder. She kept telling me not to, but I had to anyway. I opened one end of it, prying it open with a screwdriver, and inside the small cylinder was an infinite space -- a whole multiverse of possibilities.

She just kept staring inside it, looking intent but not saying anything. I asked if she wanted me to close it; but she shook her head no. There were millions of these little intelligent creatures in there, which could see our (and everything's) past and future.... Clearly she was absorbing a lot of knowledge from them ... and so was I ... but it was also clear that we were absorbing somewhat different things.

Then, we looked at each other and, without words, asked each other if we should dive into one of those universes or stay in this one. It was clear that in those universes we could still exist as individuals (and could still be with each other); but would exist in radically different form (some form not constrained by time, though there were other constraints not comprehensible in human terms).

Gradually, we collectively realized that we did not feel like entering that other multiverse at that particular time.

Then, she gave me a look that meant something like: "I will never be afraid of anything relating to human society anymore, nor be afraid of my own emotions, because I can see that this whole world of you and me and humanity and Earth is just a sort of artistic construction, which exists for aesthetic purposes. We have chosen to remain in this universe so as to remain part of this artwork ... "

... and then her unspoken thought faded out before it was done, because someone was in the house walking around and we got distracted by wondering who it was...

... and then I woke up because of the noise of my dad walking around downstairs in my house (he was visiting last night)

... and I tried to fall back asleep so as to re-enter the dream, but failed ...

Causation as Storytelling

2009-10-15T20:41:00.008-04:00

I've pointed out before (and it's not my original observation) that no branch of modern science contains a notion of "cause" more than vaguely similar to the folk psychology notion -- causation, as we commonsensically understand it, is something that we humans introduce to help us understand the world; and most directly, to help us figure out what to do....

[David Orban, in a comment on an earlier version of this post, noted that some formulations of relativity theory contain the term and concept of "causation." But causation as used in that context is really just "influence" -- the restrictions on light cones and so forth tell you which events can influence which other ones, but don't tell you how to distinguish which events are causal of which other ones in the stronger, commonsense usage of the "causation" concept.]

Cause, in our everyday intuitive world-view, is tied to will: "A causes B" means "I analogically intuit that if I were able to choose to make A happen, then B would happen as a result of my choice."

And, I think cause is also tied to storytelling. Causal ascription is basically a form of storytelling.

Think about the archetypal story structure:

If we envision a typical causal ascription as fitting into this structure, we have:

The Setup is the situation in which the causation occurs, the set of "enabling conditions." For instance, we rarely would say that oxygen is the cause of us being alive -- oxygen is considered an "enabling condition" rather than a cause of our life ... it's part of the set-up....

The Confrontation is the introduction of something unusual into the Setup. This must be something that is not always there in the Setup, otherwise one wouldn't be able to isolate it as the cause of some particular events. It's not necessarily a violent confrontation, but it's a violation of the norm. Could be someone shooting a gun, could be a couple having sex, could be a finger pushing down on a computer keyboard. The less expectable and frequent it is, the better -- i.e. the more convincing it will be as a potential cause of some event.

The End is the event being caused.

My suggestion is that, if one digs into the matter deeply, one will find that many of the same patterns distinguishing compelling stories from bad ones, also distinguish convincing causal ascriptions from unconvincing ones.

What Would Aristotle Say in a Situation Like This?

The Aristotelian distinction between efficient and final cause is also relevant here.

"Efficient cause" is what we usually think about as causation these days: roughly, A causes B if

P(B|A,Setup) > P(B|Setup)

and there is some "plausible causal mechanism" (i.e. some convincing story) connecting A and B in the context of the Setup.

"Final cause" is telos, teleology -- A causes B if B, as a goal, somehow reaches back in time and makes A happen as part of the inevitable movement toward B's occurrence.

Modern physics theories have no place for final causes in the Aristotelian sense. But, human psychology does! Very often, when a human seeks a cause for something, what they're doing starting with some event they've observed and trying to find a "good reason" for it.

Why did I fall in love with her? It must have been because she was beautiful ... or smart ... or rich ... or whatever.

Why did my business succeed? It must have been because I was smart ... or because it was the right time ... etc.

Storytelling generally mixes up efficient and final causation in complex ways. Many stories give a feeling of inevitability -- final causation -- by the end. And when postmodernist stories avoid giving this feeling, it's generally done intentionally, with a view toward violating the known psychological norm and doing something disconcerting or shocking.

Convincing causal ascriptions, like compelling stories, tend to mix up efficient and final causation.

Cause and Will

Nietzsche wrote that (paraphrasing) "free will is like the army commander who takes responsibility, after the fact, for the actions of his troops."

Experiments by Gazzaniga, Libet and other neuroscientists have validated that in many cases the reflective, willing portion of the human brain-mind "decides" to do something only well after some other part of the brain has actually already started to do it.

This fits in fine with the notion of causal ascription as storytelling. Willing is a matter of making up a story about how one came to do something. It had better be a compelling story or the illusion of free will will fall apart, which is bad for the maintenance of the self-model!

Causation and Storytelling in Neuroscience, AGI and Early Cognitive Development

One concrete hypothesis that comes out of this train of thought is that, when the neural foundations of causal ascription and storytelling are unravelled, it will turn out that the two share a large number of structural and dynamical mechanisms.

Another hypothesis is that, if we want our AGI systems to be able to ascribe causes in humanlike ways, we should teach our AGI systems to tell and understand stories in a humanlike way.

I strongly suspect that one of the major roles that storytelling plays in human childhood, is to teach children patterns of narrative structure that they will use throughout their lives in constructing causal ascriptions (along with many other kinds of stories).

...

Ahh ... I would love to improve this blog post with a bunch of concrete examples but that will need to wait for later ... I'm tired and need to wake up early in the AM ... at least "I" think that is the cause of me not wanting to improve it right now ;-D ...

AGI, Ethics, Cognitive Synergy and Ethical Synergy (from my Yale talk...)

2009-09-09T08:05:00.006-04:00

Earlier this year I gave a talk at Yale University titled "Ethical Issues Related to Advanced Artificial General Intelligence (A Few Small Worries)" ...

It was a verbal discussion focused rather than PPT focused talk but I did show a few slides (though I mostly ignored them during the talk): anyway the brief ugly slideshow is here for your amusement...

The most innovative point made during the talk was a connection between the multiple types of memory and multiple types of ethical knowledge and understanding.

I showed this diagram of different types of memory and the cognitive processes associated with them (click the picture to see a bigger, more legible version)

and then I showed this diagram

which associates different types of ethical intuition with different types of memory.

To wit:

Episodic memory corresponds to the process of ethically assessing a situation based on similar prior situations
Sensorimotor memory corresponds to "mirror neuron" type ethics, where you feel another person's feelings via mirroring their physiological emotional responses and actions
Declarative memory corresponds to rational ethical judgment
Procedural memory corresponds to "ethical habit" ... learning by imitation and reinforcement to do what is right, even when the reasons aren't well articulated or understood
Attentional memory corresponds to the existings of appropriate patterns guiding one to pay adequate attention to ethical considerations at appropriate times

I presented the concept that an ethically mature person should balance all these kinds of ethics.

This notion ties in with a paper that Stephan Bugaj and I delivered at AGI-08, called Stages of Ethical Development in Artificial General Intelligence Systems. In this paper we discussed, among other topics, Kohlberg's theory of logical ethical judgment and Gilligan's theory of empathic ethical judgment. In the present terms, I'd say Kohlberg's theory is declarative-memory focused whereas Gilligan's theory is focused on episodic and sensorimotor memory. We concluded there that to pass to the "mature" stage of ethical development, a deep and rich integration of the logical and empathic approaches to ethics is required.

The present ideas suggest a modification to this idea: to pass to the mature stage of ethical development, a deep and rich integration of the ethical approaches associated with the five main types of memory systems is required.