Kevin Kelly: "superhuman" AI is bullshit

Silicon was already a pretty old-school substrate thirty years ago. But it’s relatively cheap, and there are a lot of sunk costs.

The first part, yes, definitely. The second is a semantic issue, as to whether or not one associates free will with sense-of-self. The meta-self can still have a sort of free will which is separate from one’s feeling of personal identity, but without an ego as such the meta-self is not concerned about it one way or another.

Not speaking for @anon81034786, but I think it is a matter of different “signal processing” domains. They just don’t work similarly. It is easy-ish to make a binary approximation of something, but it is only a static model. Much more involved is real-time modelling of an approximation of an analog physical system - modern computers still struggle to implement even fairly simple ones in real time.

It is kind of like making a 1:1 scale map of some vast complicated landscape. Even succeeding for an instant would be terribly expensive and complicated (as well as arguably unnecessary!). And then there’s the fact that the landscape is a changing ecology, an emergent output of larger chaotic meta-systems, which one would also need to model. So to get a functional 1:1 map which changes over time you either need to make a system much larger and more complex than what you hope to model - or else drastically simplify it somehow. That’s problematic when dealing with a quality such as consciousness. Most handwavium about human-level AI (as if that were A Thing) is to realistically model the living human brain. Yet there is little consensus about how - or even IF - a brain is actually conscious.

When we instead just try to figure out what might be complex and yet balanced enough to make an interesting information system, things get more promising. But less likely to resemble human. The efficient way to go about it is to exploit binary circuits and algorithms for what they are good at, instead of trying to replicate billions of years of squishy happy accidents. The results are unlikely to resemble anything like even life/biology. It doesn’t need to maintain its own body, or survive, or reproduce. It is probably the first “alien” we will really “meet” - if we are even capable of perceiving each others existence.