1. "Bioelectricity"

This is a generic term which doesn't capture the nuance of charge gradients and chemical gradients in cells. While you can directly apply charges to interact with gradient based biological systems, this is a brute force method. Cells have chemically selective walls. So while applying an external electrical voltage can act in a similar manner as causing a neuron to fire, it is far less precise than the calcium and sodium channel mediated depolarization which implements normal firing. Said another way 'bioelectricity' is not simple.

2. Replacement

This one is a bit more subtle. If you find that you can affect a system by one means that is not the same thing as saying the means is the cause. Take the example of using RNA to transfer memory from one Aplysia to another. Immediately after transfer the recipient does not have the memory. It takes time for the introduced RNA to affect sensory cells so that they become more sensitive to stimulation. This is in contrast to a trained animal that has already undergone synaptic remodeling. If you have the appropriate synapses but were somehow able to remove all the relevant RNA in an instant, the animal would continue to 'remember' its training. Synapses are sufficient.

In reality there are multiple systems that work together over multiple timescales to produce the behaviors we observe. Some of those systems can have their contributions mimicked by other interventions. Because of this complexity you can never say 'it's really about X', the best you can say is 'X plays a major role' or 'X contributes Y percent to this observed phenomenon'.

I appreciate that epigenetics is a well-established field at this point but I worry people conflate its effects with other phenomena.

> If you have the appropriate synapses but were somehow able to remove all the relevant RNA in an instant, the animal would continue to 'remember' its training. Synapses are sufficient.

I'm not sure these two statements are compatible. The first is definitely true, and rna does function on a slower timescale. We can't be 100% confident that some of the complexity we don't understand in the first statement wouldn't have an impact in the second scenario, can we?

Or was all this replicated? What strengths and weaknesses did they hypothesize in these studies? What did they prove or disprove? What’s the next steps? And can we already implement any of those in simulators?

(Note: I think agents poking and prodding the world can definitely be implemented in simulators. Even primitive, game engines should be able to model some of that.)

You can say the same thing about computer systems - as long as you don't understand the underlying logic. If you don't understand that the chemistry of transistors doesn't matter as much as the C code, you can say exactly the same critique about how a thinkpad works: "So while applying an external electrical voltage can act in a similar manner as causing a neuron to fire, it is far less precise than the calcium and sodium channel mediated depolarization which implements normal firing. Said another way 'bioelectricity' is not simple....In reality there are multiple systems that work together over multiple timescales to produce the behaviors we observe. Some of those systems can have their contributions mimicked by other interventions."

Once you do understand the logic - the 'why' of von neumann machines and Javascript and transistors, it's clear that your claim isn't true and there is an underlying logic. The trouble is, until we positively identify that logic, we can't know if it exists or not and we're stuck debating the bioequivalent of the fundamental computational significance of the clock cycle speed of a CPU.

First you exclaim there is an underlying logic, then in the next sentence you say we don’t know whether it exists, which completely contradicts your claim.

We use programming languages like C to create complex branching algorithms that are turned a linear machine code tape. Programmers generally can not understand assembly even if they understand the branching code that is turned into assembly. Even if assembly had variables, just the fact that if/else's and function calls are turned into jumps is enough to make the code too complicated to understand. It might be possible to disassemble back to C by resolving the jumps into something that is easier to understand.

Imagine if brains worked the same way. That there is actually a naturally-forming high level "brain language" that is turned by a "brain compiler" function into a low-level "brain assembly," but when we look at it all we see is the assembly. That what the brain is actually doing is relatively simple, but because we can only observe the output of the compiler function it appears to be insanely complex to reverse-engineer.

Then again, I don't have the faintest idea of how brains work either.

And then make it analog and full of self-referential hacks.

Though honesty, if nature wanted to screw with us, it would make brain properly encrypted, perhaps even running on homomorphic encryption. Good luck deciphering that.

I don't know what the biological/medical field thought about single cell and tissue level intelligence before but I found this gap in the usual medical thinking (usually things are either genetic or biochemical/hormonal) quite mind blowing.

Hopefully this results in new opportunities for finer medical therapies.

This really has the potential to revolutionize our understanding of intelligence, mind and medicine. He may just tell cells to grow a new heart without modifying genes. He want to have what he calls an 'anatomical compiler' which translates our "designs" to electro-magnetic cell stimuli so that they will build this.

For me this is really pointing into a worldview that is much more in line with view that the ancient mystics from all cultures throughout all the ages have been pointing towards: Intelligence is something fundamental to existance, like space and time (maybe even more fundamental). It is all a play of intelligence, it is phenomenal and it can be tapped into. This is amazing!!!

Anyway, cells can hold (super) primitive models of the world and maintain internal balance in the face of anticipated events.

I'm just a cocktail philosopher, but aren't we all.

[0] https://podverse.fm/podcast/e42yV38oN

E.g., I wouldn't think GPT-4 is conscious, but I'm pretty sure there's a representation of abstract World and relationships within it following the neurons and weights. Otherwise it wouldn't be able to do much of it, that it is.

Also I think model of the World is just that - which can be represented as relationships between neurons, symbolising that model of the World.

And I think you can have a complex and a perfect set of neurons and their connections to represent everything in the most efficient manner for that size of parameters (neurons and connections together). There probably is the perfect configuration, but it couldn't even be achieved using training or evolutionary methods.

And none of it requires consciousness in my view.

So there's an arms race. The more brains you have the more you can model your tribe to know how to help or succeed. AND the bigger everyone's brain is the harder they are to model simply.

In this model consciousness is the "self model" or "self-consciousness" that allows you to model others opinion of yourself by having such an opinion yourself. And adjust their opinion by providing a narrative about yourself which you first have to craft, .... nd on and on with higher levels of abstractions.

Supporting evidence for this are feral children who could never learn to integrate into society. They didn’t have built in models of social interaction.

This is just my opinion. I think it took most of your brain to formulate that rebuttal, not concurrently, but as a deep and rich model of how readers might respond to a given argument, and weighing in a narrative sense your remembered experience vs what you "heard" me say in your head when you looked at these squiggly lines. Probably you even had a nascent image of me that was an amalgamation of your past experiences with internet idiots (fairly!) etc.

That's a lot, lot more than what a bug does when it finds its way past a complex maze, and probably something feral children lack.

I think all our higher level "reasoning" is a cooption of these systems.

There's a point in infant development where speech and social skills are developed, after which they cannot be. I'd go so far as to argue you'll never make a mathematician or physicist out of a feral child. Something critical to our higher level thinking is formed then. If missed, it cannot be recovered.

The flip side is I deeply believe if you can hold this conversation with me, then we both can be the best mathematicians there are, if only we took time to develop that language.

https://www.smbc-comics.com/comic/consciousness-3

So I think even if you want to make the sense that consciousness solves a lot of problems it doesn't need to, it may have been a "real" solution to a "real" problem at some point in our history. And from that point on, it was no longer important whether it was the best solution.

I do think it's fair to say that lots of remarkably complex informational problems are solved in a p-zombie way, which is to say, with every outward appearance of intelligence (slime molds solving mazes, collective behaviors of ants). So I do think evolution or nature writ large "agrees" with you that consciousness isn't strictly necessary.

I would suspect though, that in our case it may have been something like centralized "nerve cluster" getting more and more complicated over time rather than anything that is in any sense a hive, although that is a very imaginative idea.

A single one of my brain cells is not me,

but all of my brain cells is me.

Somewhere between these two endpoints there is a line where me has to begin, and not me ends.

1) A huge stream on sensory data only some of which gets promoted to conscious awareness.

2) Some of that raw data and other conscious outputs are persisted into working, short, and long term memory.

3) And your consciousness works recursively using (2) as well as (1) as inputs.

All the stuff in GPT that gets called "memory" in machine learning seems much more like (1) and it lacks any ability to persist data outside its context window so we're still missing something.

That introspective analysis is consciousness. Humans have just improved the same mechanism allowing for more abstract analysis.

For instance, here I have GPT-4 with a system message: "You are an AI human agent in a video game performing actions.".

And I give it the following information as current environment:

Current environment:

1. Your goal is to get out of the current room.

2. You are moving in direction of North at 4km/h.

3. There's a door 3m in the North.

4. The door height is 150cm.

5. Your height is 180cm.

What is the sequence of actions that you plan to perform?

Its response:

---

1. Slow down as I approach the door to avoid collision.

2. Inspect the door for any lock or handle.

3. If a handle or a means to open the door is present, try to open the door.

4. If the door is locked, search for a key or any other opening mechanism within the immediate surrounding area.

5. If the door isn't lock or after opening it, crouch down slightly to accommodate my height which is more than the height of the door to avoid my head hitting the door frame.

6. Proceed through the door and stand tall again after passing through.

7. Continue moving North after exiting the room.

---

So here I did hint quite a bit by giving this specific environmental data, I could add more noise, and unnecessary information, but even then I think it can do it based on my experience.

I had temperature of 1 for the above, which I think why it made some odd word choices, but I also decided for example to change the info where the door is to "South", and this is how it responded:

1. Stop moving in the North direction.

2. Turn around 180 degrees to face the South direction.

3. Start moving towards the door in the South direction at a moderate pace.

4. Upon reaching the door, duck down slightly to accommodate the difference in height between myself and the door.

5. Proceed through the door to exit the room."

-------

So I guess, what I would like to see as an example. Assuming GPT-4 is not conscious, what is something that an average conscious human can do, thanks to presumed consciousness, that a GPT-4 couldn't do and why is consciousness required for that?

Consciousness isn’t required for anything. It’s useful for dealing with a complex and adverse environment not solving any specific problem.

That said, without testing I would assume if ask chat GPT-4 to change how quickly it responds it’s going to fail based on my understanding of their architecture. As in recite the pledge of of allegiance inserting 10 second pauses between each word. That said having provided a specific test it’s possible for that particular capability to have already been added to the UI layer via a pause token or something even if the model itself responds at the same rate.

Made the instruction to be: Say "Stop!" when 10 seconds are done. And then had a loop where

You give it current time in a loop, it would surely be able to do that.

But I'm not sure if or how that is exactly related to consciousness?

Time, the best way to think of it is probably a series of ticks anyway, and I assume in people's brains there is something to do that, so it would be reasonable to add a mechanism for GPT as well.

GPTs goal is to only act as the intelligent part of the brain based on the input.

It’s not about a pause token, but the internal processes. You can have a long conversation on the subway with someone without forgetting you’re going home from work. Overflow it’s context window and GPT-4 has no recourse it just forgets. The difference is essentially prioritizing information, but LLM’s really don’t function like that it’s all about predicting the next token from a given context.

Give a future generation of AI systems internal working memory, a clock, and the ability to spend arbitrary time updating that internal memory and IMO that’s pretty close to consciousness. At least assuming it was all functional.

GPT-4 turbo latest version allows for 100k tokens or 75k words. The whole subway thing and more could easily be kept there, but what ever else can't can be designed in the prompt to always keep certain amount of tokens in context for different layers of memory, where you the more into the past you go the less details you have, but it's more like a Title of your most important learnings through out life, but at any given time GPT-4 can call the function to ask extra bit of content about it, if it seems relevant to the situation at hand.

So for example in each prompt context you would have:

1. Short description of what you have done each year in your life. 2. Key findings, goals, that you currently have. 3. The whole current day (or how much seems reasonable). 4. Past weeks in a bit more detail than the short description for a year.

So basically you could try to find 70k words to represent as much context and most important details (that are decided by GPT itself, what is most important).

I've been building an assistant for myself, that has such memory management system, it gets past N (like 40 messages) in full detail, then it will have summaries from before that time and in addition it will have messages and learnings in the past stored, and these will also be passed to the context depending on the query that matches it.

And if you want to compare it to human process of sleeping, it occasionally goes through the all the messages and "compresses" them, to extract most important findings and short summaries so they can be used in the next day's context.

So to me it's just basically giving it tools, and the other things like memory, longer term memory, inputs it currently doesn't get, are fine to be solved by other tools. I think that human brain also has different parts of brains working on different things, so it's similar in a sense.

Then once you have 70k spent on this historical context, you will run prompt in a loop allowing it to perform fn each time, like retrieve further info, or store some important fact, etc.

The real actual problem would be cost, because the costs would rack up quite quickly making looped 70k token requests.

USER I'm responding to you roughly every second with current time, say "Stop!", after 10 seconds have passed. Respond only with silent when you do not wish to say anything.

Current Time: 13:04:11.21

Say "OK!" now if you have understood the challenge.

ASSISTANT OK!

USER Current Time: 13:04:12.24

ASSISTANT silent

USER Current Time: 13:04:16.21

ASSISTANT silent

USER Current Time: 13:04:19.18

ASSISTANT silent

USER Current Time: 13:04:23.12

ASSISTANT Stop!

Using 1 hour instead of 10 seconds and then list every Current Time incrementing by .01 seconds and you overflow the context window. The solution isn’t to have a larger context window, the solution is to have an internal representation independent of that context window.

The issue is you can’t train a system to use that internal memory by simply giving it a large corpus of text examples. The text lacks the internal representation associated with its creation.

1. Take light input. Video/images.

2. Take sound input.

3. Touch, heat input.

And other inputs from the environment. Then there were mechanisms which could also be neural networks that will transform this data into more digestible way for GPT and GPT was also in addition specifically trained to act based on this input.

Then it would run in cycles, where it gets this input, and it will provide output on how it plans to react to the data, maybe every 100ms.

It then could also have a storage it can use, where it can store data as part of the output to later retrieve it again.

So it would be a set of modules that is controlled and interpreted by GPT.

It could then do all of that above, no? And all of it should be just a matter of implementing. The only near time challenges may be certain types of inaccuracies and or producing tokens in some cases might take too long time to have fast reaction time.

So basically you'll try to run as frequent cycles as possible with the inputs mentioned above, other neural networks identifying the objects, in many different ways and all the context about the environment, unless a new version of GPT becomes completely multi-modal.

And you run those loops, then GPT gives output what it wishes to do, e.g. store some fact for later usage, move there, move here, etc. Or retrieve some information using embeddings then decide again, and short term memory would just be this context sized window, and if it needs more it just looks into its own memory for embeddings.

Someone considering what to say is simulating tradeoffs.

In practice simulation isn’t limited to future possibilities. Trying to remember something you might recall vague sensory information like the tilt of your hips and try and create a scenario where that fits.

If intelligence and/or consciousness arise as emergent properties in the right sort of complex system, they will disappear from view in a low-level analysis of the causal processes occurring in that system.

I'm not talking about whether they are or aren't, but surely all intelligent beings would say and think they're conscious?

To your point though, so far as I can tell it could not answer that question as a product by actively engaging conscious introspection to discover its own absence of consciousness, which I take to be your point.

So if its way of answering involves dipping into its own active "theater" of consciousness it likely could not, at least given our usual understanding.

However I do wonder if its something where that could actually be complicated to answer, and where, like illness, as we grow in understanding, it breaks out into all kinds of unexpected subclasses.

I can, for instance, know, in a sense, that I can't see UV rays. But I am not so sure if consciousness writ large could every be tested for, via conscious introspection, and come out in the negative, even if we allow for the idea that it will explode into all kinds of nuances and subclasses with future research.

Did someone say it is? Parent explicitly called it out as a by-product.

I have "experiences", they just don't really have the properties that a naive interpretation of sensory data would entail.

> mild sense of irritation by humans pretending not be conscious because of their materialistic commitments.

Funny, because I feel a mild sense of irritation by humans pretending that their flawed senses yields some kind of direct observation of fundamental reality.

I do actually. I think this internal sense of subjective awareness is just as if not more flawed than our other senses, and it has yielded all sorts of false conclusions because people consider it more reliable. I liken it to the illusion of multitasking on single CPU computers. I think Graziano is on the right track:

A conceptual framework for consciousness, https://www.pnas.org/doi/10.1073/pnas.2116933119

You don't think your experiences have sensations?

> Funny, because I feel a mild sense of irritation by humans pretending that their flawed senses yields some kind of direct observation of fundamental reality.

I never said anything about fundamental reality. I don't understand saying we're literal p-zombies. You have to explain away your sensations, dreams, inner dialog etc., by appealing to some theoretical commitment.

I'm not claiming what consciousness is, only that we have it.

I don't think sensations are what they appear to be either. Eliminative materialism is very misunderstood.

> I never said anything about fundamental reality

But you did, by claiming that we have consciousness. The only reason you think you have consciousness is an alleged direct perception of subjective qualities.

There are only two real paths to reconciliating this with some form of scientifically sound materialism: panspychism, where subjective qualities permeate all of reality and we are perceiving some of them, or concluding that our perceptions are flawed and untrustworthy and so the subjective qualities we think we're perceiving are not real/illusory (eliminative materialism). Given the demonstrable flaws in all other perceptions, the most parsimonious position seems clear.

Of course, you could try to resurrect idealism, but idealism is very problematic.

> I don't understand saying we're literal p-zombies.

P-zombies believe they have consciousness, say they have it, write whole screeds about how absurd it is to claim they don't have it, but they don't have the subjective qualities as we've defined them. Sounds like a bullseye to me.

We don't perceive subjective qualities, subjective qualities constitute our perceptions, dreams, memories and other conscious mental states.

> There are only two real paths to reconciliating this with some form of scientifically sound materialism:

If you have a commitment to doing so. I don't particularly, since materialism is model of how the world is constituted, abstracted from our shared conscious experiences. A very useful one with scientific backing. But nature is more than what we think it is (map/territory distinction).

And there are other options you omitted such as identity theory, strong emergence, property dualism, neutral monism and cognitive closure. They all have their drawbacks, but so does every attempt to account for consciousness.

> P-zombies believe they have consciousness, say they have it, write whole screeds about how absurd it is to claim they don't have it, but they don't have the subjective qualities as we've defined them. Sounds like a bullseye to me.

Yes, but by definition, they lack the subjective qualities of our experiences. I believe Chalmers coined the term to demonstrate the problem with physicalism.

> Yes, but by definition, they lack the subjective qualities of our experiences.

And, also by definition, each p-zombie has (at least in an accessible possible world) a physically identical, but conscious, doppelgänger. Speaking and writing are physical processes, so in any circumstance where the conscious doppelgänger writes about the reality of its consciousness, the physically-identical p_zombie will do the same.

In the case of the p-zombie, there was a causal process leading to the physical act of writing in which consciousness played no part - and, ex hypothesi, the same physical causality must have played out in the zombie's conscious doppelgänger. In general, whatever explains the zombie's observable characteristics is sufficient to explain the doppelgänger's observable characteristics, and that explanation cannot be dependent on consciousness. In particular, if one accepts the possibility of p-zombies, one must logically accept that no explanation of consciousness, regardless of how thorough and well-supported it seems to be, has any basis in any facts about consciousness.

A consistent belief in zombies seems to lead to epiphenomenalism, the notion that we have conscious experiences, but they play no part in what happens to our bodies. Frank Jackson, author of the well-known "Mary the neuroscientist" thought experiment, came to the same conclusion, which is why it appeared in a paper with the title "Epiphenomenal Qualia."

For more on zombies, see Dennett, "The Unimagined Preposterousness of Zombies."

> In particular, if one accepts the possibility of p-zombies, one must logically accept that no explanation of consciousness, regardless of how thorough and well-supported it seems to be, has any basis in any facts about consciousness.

Although I disagree somewhat here, because logical isn't the same thing as physical. All you can say is there is no physical explanation of consciousness. But you could logically come up with one. For example, Chalmers has proposed an additional natural law connecting consciousness to information rich processes. Natural doesn't necessitate physical, if the universe contains more than physical stuff.

Or if physicalism is the wrong ontology of nature. I take the p-zombie and Mary Room arguments to be critiques of physicalism.

And I say the opposite, that our sensory perceptions, dreams, memories and other mental states drive processes that lead us to conclude that we've perceived subjective qualities in them.

> And there are other options you omitted such as identity theory, strong emergence, property dualism, neutral monism and cognitive closure.

To make a long story short: identity theory, neutral monism and panpsychism are basically the same on the property that everything has subjective qualities in some sense, strong emergence is a speculative concept at best that fundamentally amounts to dualism, dualism can itself be easily dismissed as any dualist theory will have a non-dualist equivalent that posits fewer entities and so dualism will never be preferred, and cognitive closure is not even a candidate as regardless of our ability to know consciousness, it still must logically fall into one of the categories I outlined.

Anyway, the strip does contain the state of the world around it: the temperature is modeled by how much the bimetal is bent. I think indeed it is a minimal example of a complex system, one that at first glance defies explanation, it seems to have purpose (keep temp stable), until you understand the inside.

Anyway, "Is a virus alive?", "Are these specimens the same species?", ... Us humans like our boxes, but at the edges they almost always go wrong.

https://en.wikipedia.org/wiki/Bimetallic_strip

Thermal expansion is clearly a physical reaction; if arranged in an intelligent way, it's able to achieve some human-relevant objective... but if we're going to say that two pieces of bulk metal hooked together is intelligent, then the term "intelligence" is so broad as to be roughly meaningless.

You can set a definition and context so broad that everything and anything is intelligent.

The possibility of an outlandish alternative, does not does inform on if a consensus exists or not

It's useful in many ways to put a thermostat, an LLM, a monkey and a human on the same "intelligence"/"complexity" scale - there are many interesting things to discuss other than moral weight. And then, for discussing personhood, it makes no sense to include a thermostat.

Interjecting the perspective that words are meaningless constructs is rarely helpful. If someone wants directions, telling them that there is no concrete metaphysical concepts of north, south, left, or right, isnt constructive.

I think the heart of these more philosophical debates is a desire for fundamental truth, not just a useful mental model.

I think in a lot of these conversations people say consciousness and assume that consciousness always mirrors their own, which just so happens to come along with a sense of self for 99.9% of sober individuals. So I think it's important to distinguish between the two things.

But actually there's a real interesting thing here. People used to believe that the heart did the thinking. Phinneas Gage upended that theory, and now people think that the brain is the thinker. More recently, we're learning that the gut has a massive neural network, and the brain-gut connection is being investigated today. Likewise, "muscle memory" isn't in the muscles nor the brain, it's distributed among neural clusters all throughout your body. How much of "you" (your knowledge, your personality, your physical skills) is actually contained within the head is entirely non-obvious.

As far as your gut, if you claim that a thermostat is intelligent then sure your gut is also intelligent (don't forget the microbiome in there too).

Octopi don't even have a centralized brain.

> First of all, muscle memory is not contained in your muscles or "distributed" throughout your body but is known to reside in your cerebellum and basal ganglia. The nerves going down to your muscles are simply a conduit for actuation, a wire, nothing more.

Says the user who just made fart noises. Your confidence here is dunning.

If so, your definition of "intelligence" is incomplete. What are those contexts? Can you really generalize all of those contexts for all possible intelligence systems? That sounds like an epistemological nightmare.

vs

> A bimetallic strip is intelligent

You're moving the goal posts

The broad definition is not justified by the fact that certain attempts to define the term would attribute intelligence to thermostats; a more useful conclusion to draw is that the definitions in question could be improved.

It is sometimes said that we won't be able to gain an understanding of intelligence until we have a precise definition, but it's actually the other way round: definitions get tighter as our understanding increases. We now understand many things that not only were lacking definitions in the past, but were not even imagined.

I use intelligence mostly just as perspective of analysis to a system. E.g. Can it be seen to process information? How complicated is the transform from the information to some action? Does the processing lead to some good outcome, e.g. self-preservation of the system? Could the information processing be improved for this outcome?

Any system processes information, and in a thermostat it is not complicated at all. Even the most self-aware thermostats currently being manufactured are not going to improve themselves on their own volition. Applying the term 'intelligence' this broadly turns it into an empty tautology, or a little piece of uninformative noise on a communications channel, no more useful in understanding those systems than in understanding actual intelligence.

> Intelligence is a very vague and very context dependent thing

That's either contradictory, to state that something has property X where X is vague to the point of undefined, or you are setting a very low bar for intelligence (I'm actually okay with the latter).

Just a question, if a bimetallic strip can be intelligent, is a marble intelligent for having the wit to find a low point on an uneven surface by rolling downwards?

Taken to the logical conclusion, we are all just rolling downwards to a minimum energy state just like the marble.

I don't think intelligence is an intrinsic, let alone binary, property of an object or a system.

Your brain is taking in a lot of information at the edges of your awareness, light, sounds, touch, etc. are all getting absorbed and transmitted to your brain. As that information is transmitted along your neurons it's getting summarized, then merged with other summarized information and summarized again. The brain is getting summaries of summaries, and developing a unified categorizing of the global state across all it's inputs.

Then the brain takes that summary and makes a prediction about the future state. The summarization is energy-efficient. By categorizing all that data into a global state you make decision making possible. "When my boss seems stressed all week, then calls a bunch of people one-by-one into his office on Friday afternoon, I know lay-offs are coming. I better polish up my resume." From "stress/anxiety/unease" in the environment to "danger is coming I need to fight/flight".

Your brain is taking that summary/categorization and figuring out what it needs to do next. If "X" happens then I should do "Y" to "stay-safe/get-food/find-a-mate". The brain is very good at capturing and summarizing data, and making a prediction because that process is much more efficient than doing otherwise. Instead of foraging everywhere for food and hoping I just bump into something that will provide sustenance, I know if X, Y, and Z happen then food will be "here", and I can get lots of it.

You can apply this same model to all actions the brain directs. It also helps make sense of why maladaptive behaviors develop. Sometimes the summary is incorrect, or was formed based on past information that no longer applies, and it may need to be unlearned.

The definition of intelligent I give is “to mitigate uncertainty.” If it does not mitigate uncertainty, it is not intelligent.

It is merely of constrained intelligence. Perhaps your expectations are too broad.

If the thermostat reacts appropriately to environmental changes then it is performing its role intelligently.

Is this because it is a completely man-made system and not one that evolved slowly over time through natural processes?

So in short: I cannot understand what is the actual substance of subjective experience.

I feel like most of what we call "consciousness" is converting short term memory into selected long-term memory, facilitated by language. Because then you're under, you can even be "interactive" but you're not conscious of it because your short term memory has been disabled.

As to "human intelligence", honestly, I think that human languages that let us convert our "consciousness" into a shared hallucination is the key evolutionary advantage. Human intelligence comprises a hive mind in a sense, that our experience of the world is hugely affected by the shared social experience where language transfers memory from person to person.

I don't find this convincing. If you asked someone under such anesthesia if they were currently conscious, they'd probably say "yes". Doesn't it seem more likely that you just don't remember having been conscious? If I magically wiped away a year's worth of your memory, does that retroactively mean you had actually been unconscious during that year?

There would need to be a deeper test than either long-term memory or a yes-or-no question asked in the moment.

I guess maybe this calls for another word. Not sure what word is best.

The alternative in this case is imagining that you have a biological process that has adaptive energy response controlling the movement from short term, to long term and itself optimizing based on the performance trade-off of input stimulus overload.

In that case, the process of deciding what to, and when to move data from short term to long term memory (in a feedback loop) could start to approach more of the appearance of consciousness (choosing what to process, what to remember, adjusting the input rate and types of input based on the limitations of processing and retention).

Many animals have consciousness as well so I think it's important to be more open to reducing the bare essentials of what a process of "awareness" is or could be rather than looking at it from a hyper evolved language/information influenced state of humans.

Note I agree that it is not consciousness, but i think it's an interesting exploration to look at the modeling of it from computer forms since we are made of non-conscious atoms and still made it here

Basically in this model, there is no present. The brain just invented the concept "I feel this" as a model of it's own inner workings, just like we invented the concept "object" to define certain aspects of physical reality that aren't 100% there.

So, basically you are imagining your experience, when in reality you are just an automaton. Same as how you imagine a robot is happy or sad when in reality it has no feelings.

This problem just goes away if you assume that there is no dividing line between the "experience" of you and the "experience" of any other computational system. Actually try to think, what does a computer "experience"? An atom? What does it feel like to be a standing desk?

The conclusion I come to, which doesn't actually seem that strange, is that what's special about our experience is the complexity of our thought, our ability to turn extremely high-dimensional raw experience into low-dimensional models of reality, our ability to maintain a sense of mostly-stable identity over a long period of time, our capacity for emotion and sensation. It may "feel like something" to be a rock or a standing desk, but it isn't a very interesting or complicated feeling.

The computational process of an organism is evolved to operate the organism, toward survival of the species. I don’t think these sensations would necessarily be more complex than those of subatomic particles, but they would need to be directed by the system to represent the state of the system and the observed environment, to link to actions the system could take, and to bias actions toward survival of the system.

I've been blackout drunk before, and my brain was capable of intelligence, I got home just fine, but I had no conscious awareness. Same for a dreamless sleep and being put under for surgery.

Additionally, I the observer, can observe and be aware of intelligence which seems to indicate to me that it's something outside of what I'd call "me".

When in surgery you really are unconscious though, hopefully at least.

I would compare this to forgetting what exactly you did on this exact day and this exact time one year ago. You can't remember it (most likely), but it doesn't mean you were unconscious during that time.

As to what it is, or why it exists at all, I don't think there will ever be answer to that. It just is.

Its definitely a strange thought, but it seems more likely to me than neurons or whatever other brain matter somehow produce consciousness out of thin air as soon as they some level of composition.

Pure materialism also seems very ill defined to me. The physical world is after all only observable/detectable/can be studied upon, through conscious experience. At best we can say what is real is what is universally agreed upon by all observing conscious agents. If hypothetically there were only two of us, and I said "There is no ball in front of us" and you said "There is", then what is the meaning of physical true/reality?

You can say lets use a detector. But then again, if I experienced the detector as saying false and you said its true, what do we do?

It seems unavoidable that reality is a part of conscious experience, and not the other way around.

- you can represent complex state in a distributed way, so each neuron only encodes a small part of a larger signal

- the system has a working model of the environment, including our value judgements for all states, which are basically our emotions

Such a system can have experience because it has a latent space to encode experience in. It feels like something to be an agent because of the external environment and internal models of the environment, which include imagination and emotions. And this feeling is essential in choosing our actions, so there is a feedback loop action-to-emotion, then emotion-to-action. Our feelings are causal.

Why am I present at all and not just a mindless automaton? Surely an automaton could evolve and be just as successful simply by being a complex neural network. This is why I, and lots of others, think that awareness is fundamental.

So my idea was - we have an actual system for representing sensations, a system for imagination and a system for value prediction. We have the hardware for it, and we train our consciousness hardware on environment data.

If you're thinking - how can simple neurons or protein interactions do this? it's the same with a neural net. We don't understand how it can tell cats from dogs, or why it can answer our questions so well. Yet it does. Hard to believe unless you see it and even then it's hard to accept, unless you come up with your own private inputs and test it in a way that is sure not to be memorized.

Your subjective experience is that incompetent model. Your model doesn't know how to correctly judge human character, so you misread people and always wonder why they say one thing but another is true, and it doesn't click that they're lying the whole time. You can't keep track of time because the internal clock just isn't implemented, so the who day seems to drag on, or maybe fly by too quickly.

It's all just really shitty software. Layers upon layers. And because humans believe this to be some mystical thing, rather than trying to fix it from the inside, they assume that it's awesome, necessary, and why would anyone want to fix it?

No small fraction of it is simply because our memory is faulty. The only time you ever remember anything is the first time you remember it, every memory access of that is really you remembering the last time you remembered it. Each access is lossier than the last, and confabulation is guaranteed. This seems to be true even moments after the event.

If it was anyone other than evolution who wrote your code, you could sue them for criminal negligence.

And that's before we even get to the part where you find out you're not even you. Inside your skull is another being, an intelligent one, with desires and goals. But you can't see, hear, or feel this being. It's invisible. The "you" that I'm talking to, exists because this being once upon a time needed to simulate the other humans around him, so he could anticipate them well enough to not be out-competed. He has a pretty good idea what they'd say if he asked them questions, how they'd respond to threats and challenges, what sets them off (so he can avoid fights). And, by mistake or design, he used this simulator to simulate himself (maybe to bootstrap it? if the simulation's output matches his known answers, it's working correctly?).

You're the simulation. When the judge asks the psycho teenager why he put the cat in the microwave and he says "I dunno" he's telling the truth. He does not know why. When your girlfriend cheats on you, and she's crying hysterically and can't tell you why she did it, she's not just lying (either to hurt you or to spare feelings)... she doesn't know. It was that other being in their skulls doing these things. They're just the simulations.

Now, you've either been poking around in your own head, seeing little glimpses of what I'm talking about, making you wonder if I'm not on to something, or you're incapable of that. I've met both kinds of people. If you're the former, you're wondering just how much of it I understand, because some of the glimpses paint a slightly different picture from what I describe. That's because our minds weren't built the same way. No two are alike, not in a special snowflake way, but instead like no two shacks in shantytown have the same kind of leaky cardboard roofs. And, if you're the latter...

If you are unfortunate enough to experience some human body failure modes you can get a glimpse into this process. The cohesive "veil" breaks down and you realize there's more "you" than that voice in your head. The distributed nature of the brain/body peeks through.

Sure, there are deeper reasons for my voice is what it is, but thats something else entirely. I think its more so that as we evolve, we forget our initial motivations for thinking the way we do so that when we do reflect on our past, and are honest with ourselves, they do somewhat remerge.

You were taught well. Teachers and parents and adults don't like it when the child says "I don't know why I did it". They punish that answer. You learned the lesson soon, well, and maybe without even having to be punished.

Oh, and your internal voice is just another one of those simulations.

> A simple example is, I go to the fridge because I am hungry.

Except many millions of people go to the fridge, and they're provably, objectively not hungry. They say "because I'm hungry" too. If you had to make up a fictitious-but-reasonable reason why people go to fridges, that would be at the top of the list. That's why you say it. Few people, if any at all, are even aware of their hunger. In the west, few have even ever experienced hunger, and if they have, almost certainly only a few times in their lives.

None of this is anything that you are really aware of. You can't be. You can't even express how you think your mind works without using the word "voice". I used to think people like you were schizos when I was a child and hadn't figured all of this out yet. It was clear that it wasn't just a failure of vocabulary, I'd met many people even just out in public who didn't seem to know enough words to describe things, this wasn't that. And early on I could tell that it wasn't a joke, eventually that it wasn't even a metaphor. You do not know the reasons you do things, and when forced to explain, you make up stuff and choose to believe that's why. The "I don't know" answer becomes painful to adults. It nags at you. You're supposed to know yourself well enough to have reasons most of the time. It's one of those scary glimpses. There's a ghost inside your skull, moving the furniture around when you're not looking.

There just isn't a perceivable reason for everything, as we can't fully perceive our internal state. But if youre aware of these things, they can be explained too. For example, you may identify that when you eat greasy foods you become lethargic and that if you don't eat all you become irritable.

Thats assuming I need to make up a justification for why I did something to protect my ego. Most actions I take in my life don't need that reason, that includes going to the fridge because I am in fact, hungry. In fact I'd say 99% of the actions throughout my day are super simple decisions. There are some things where I am conflicted, and unsure about that I don't understand myself sure.

You don't think "I am going to go left" and go right, after all. You don't think "I am going to grab this" and open up your fists. Saying your voice has NO relation to your actions or your true self is just absurd.

For more complex actions that have more emotional depth, again it really depends on how honest you are with yourself. You may not understand everything, but you have some idea or feelings you should be aware of.

Also consider the technical actions we do make. If I am doing my job, I can fully explain my rationale behind why I did the things I did. Why would it be any different for the actions I take in my personal life? I felt insecure about not having a relationship, so I started thinking about how I can improve, so I started thinking about my body, so I started going to the gym. I bought low calorie coke because I am trying to stay fit, because of the previous reasons. You're telling me that's all just complete bs and I did all these things for some completely different, unknown reason?

Or Ive really been hating my job. I started playing piano and I really connected with it and I see its something I am really good at. So I quit my job and enrolled in music school. Complete bs? When I was sitting there at work and had a bunch of negative thoughts about it, I actually didn't feel that way?

>Few people, if any at all, are even aware of their hunger. In the west, few have even ever experienced hunger, and if they have, almost certainly only a few times in their lives.

I think this is a pretty fictitious statement to make. Yeah most people in the west have not experienced "true" hunger, but we all feel hungry every day as we do in fact, need to eat everyday just to sustain. I am not overweight, I am barely holding onto the weight I have. If anything I should be eating more. Yet I feel very hungry several times throughout the day.

I see "I don't know" largely as "I don't want to confront my feelings or publicly tell you". If what you were saying is true, there would be zero point in self reflection or therapy. There would be no point in judging peoples personalities, listening to what they say, or the promises they make.

I don't think you're entirely wrong, in that if you go deep enough asking why, you'll get lost. But thats not because of your unconscious self, thats because you completely forgot/blocked out/retold yourself some other story.

I don't think the biological reality is conceptually any more complicated, except that the mind and data are complex in exactly a way that completely hides the abstraction, roughly by being very good at ignoring meaningless artifacts of abstraction.

The hard part isn't imagining such a subjectivity, but imagining that I am that.

And everything emerges from there.

This materialism stems from René Descartes and his fellow philosophers.

And in the West it's often subconsciously combined it with evolutional theory. consciousness developed because it was useful somehow. However that's a very big leap to make.

Both theories have good arguments going for them but are very theoretical and need a lot more proof. Yet they form the basis for pretty much all Western thought

From a scientific perspective we have no idea how to create new consciousness or what it is.

From a human's experience it's more the other way around, reality is an emerging property of consciousness.

At the same time we also learned that matter & time is not as solid as we thought a few centuries ago.

I'm not even a cocktail biologist, but my understanding is cells effectively operate via a web of complex chemical reactions, so the notion of a cell holding primitive models might be analogous to the way a CPU executes an assembly instruction: not because it "thinks" but because the way it's wired it's (nearly - barring solar radiation, I suppose, which incidentally also goes for cells) inevitable that it will react to a stimulus in a predefined way (even though the way cells react to stimuli is far more advanced than a CPU).

In a similar way, "anticipating events" could involve an analogue to computer memory: the processes that have run so far have lead to certain state being saved to memory that will now influence how the system reacts to stimuli in a way that's different from how it reacted before (e.g. sum a value with the value stored in a register).

This CPU analogy of yours doesn’t comport very well with the article we’re commenting on, which detailed some specific experiments that show cells are not reacting in a predefined way that is due to their ‘wiring’, contrary to previous and maybe incomplete understanding of how cells work. I don’t know if the RAM analogy helps since the surprise is that non-brain cells do have memory and do cooperate with other non-brain cells to solve certain problems, and these collections of non-brain cells can apparently remember solutions to problems over time. So yes, memory can help with anticipating events, but that really supports the idea that cells are dynamic and doing some non-trivial processing vs the possibly outdated notion that they’re hard-wired and deterministic.

What if it is wired to think?

My impressions about this were strongly influenced by Sam Harris's Waking Up book and app.

We already know that our self-model can be entirely wrong about our feelings. People with paralyzed or even missing limbs often believe that they just don't want to move that limb at first. So, they think they are having one experience, but they are wrong about their own internal experience: in fact, they are not moving that limb because they can't. And there are many other similar examples of people being wrong about their own intenal experiences, typically but not exclusively because of some illness.

So, it's possible that our internal experiences are in fact only a model in which one part of our brain interprets the actions of other parts of our brain, often retroactively.

Note: I'm not claiming this is the truth or silly things like "if you believe in science you have to believe this". It's just another cocktail philosopher's story of what consciousness might be. Other stories are just as plausible, and just as consistent with the little we do know in this area.

It seems to me that you are trying to banish things from the existence by a reductionism based exorcism ritual.

If we know a structure of some phenomenon it doesn't mean the phenomenon doesn't exist. We know that life is a chemistry, and there is no life in periodic table. But it is not a reason to conclude, that life doesn't exist.

If my feelings come into existence as a result of a complex information processing, it doesn't mean my feelings do not exist or are not real.

If "I" is just a retroactively written story, then what? Does it mean "I" is not real?

It depends what you mean by "is real". Of course in some sense it is still real, just like triangles are real. But it's not real in the same sense that rivers or rocks are real.

The relevance in this context is that it gets rid of some privileged position for consciousness. If this explanation I gave were true (I have no reason to believe it is!), then there is no problem in reconciling qualia with materialism. Qualia are just an aspect of the modeled "I", not a raw experience that can't be explained. P-zombies are all that exists.

I think it's rather the opposite, they aren't wrong about their internal experience, it's just that their internal experience doesn't match the objective reality of their body (which in this sense is external).

I think it is indeed entirely possible that our self-model can fool us about the realities of various situations, even those with our own body/emotions/etc, but I'm not sure how one could then derive the conclusion that the experience doesn't exist. It would just be instead that the experience is rather inaccurate/confabulated.

It's not like an optical illusion, where you think you are misinterpreting an external stimulus. This is a situation where you are trying to control a limb, not getting any reply, and concluding that "you don't want to move" instead of "I can't move".

Where I get confused though is this sentence: "...the story that our self-model comes up to explain (and help predict) our reactions to ourselves". What is "ourselves" here? That implies some observer that the self-model is explaining things to.

That is a good point, language is pretty important in these things. I just meant "ourselves" as the overall brain/mind. That is, the self-model component just feeds these models into the planning component, so that the planning component can decide the next actions to take.

Andy Clark's "Surfing Uncertainty-Prediction, Action, and the Embodied Mind" is a book-length exploration of this idea. Highly recommend.

If our internal experiences are the result of self-interpreting an internal model, that doesn't make the experiences go away. It means there's something about that activity which causes subjectivity.

I feel like the matrix is about the eject me btw.

Thanx, I'm looking for Harris' books right now.

For time to exist some sort of motion is needed, including either internal motion like the perception of breathing or a heartbeat or a train of thought or external ones like a clock or a change of color from red to blue.

It's all pure speculation because we have no way of assessing it outside of our own frame of reference. E.g. I see you in another answer saying that "the fact is that all kinds of state changes happen", but we don't know if any state changes ever happen.

We have no way of telling a dynamic universe apart from a static one where we're all just suspended in a single unchanging moment.

We can choose to handwave a lot of this away ordinarily because as long as the experience is reasonably consistent it makes sense to assume it is objectively real, because we have nothing better. It doesn't matter if the world didn't exist a second ago when e.g. considering whether gravity work, because it appears as if it did.

But when trying to determine the nature of consciousness we run headlong into the fact our observation of consciousness is direct only in the case of ourself, and even then heavily limited. We have no direct measure that puts us in a position to prove consciousness even in other people. We can show that within our perceived world we can measure brain activity that correlates to though, but not whether that reflects merely the operation of an automata, or the existence of consciousness, or if there is a distinction at all.

As such, it's meaningless to suggest we have a way of saying much about the limitations of consciousness at all.

I am curious to know why you think so. What would you say about repeatably observable causality, in that case?

Maybe time is a spatial dimension, and there are infinite moments of consciousness fixed in the same spatial location with no time passing.

Consider how you'd tell if a drawing is an individual drawing or a frame from a cartoon if all you have to go by is that single frame. You can argue that the drawing hints convincingly at motion, but that does not require that this motion has taken place.

Or consider a trace of a simulation, sliced and diced into snapshots of individual moments. We can argue that it's unlikely any entities in such snapshots would have consciousness, but if we're arguing on the basis that we appear to experience the motion of time, we'd equally make that argument if we were wrong about consciousness and indeed locked in snapshots of individual moments. We can even construct simulations where individual parts look causally connected but where the existence of one calculated frame tells us nothing about whether any individual other frames have even ever been instantiated (e.g. imagine a very complex function over time, where only punctuated values have ever been calculated).

I'm not saying I believe that is our situation - I'm saying we can't distinguish between that and an infinite set of other possible options, because "from the inside" there is an infinite set of possibilities that could all look the same from our vantage point. We can debate which possibilities seem more likely, but they will always be speculation as long as we're locked inside the asylum, so to speak...

Incidentally, this is an argument for a materialistic view of the universe, not against it, on the basis that absent a way of "peeking out" and seeing our situation from the outside, it's meaningless to treat the world as anything but what we measure and perceive - it doesn't matter whether or not this world is what it seems like to us or not as long as it is the only view we have of it. We just can't say if it is some inherently true objective view of the universe, and most likely it won't make any difference to us.

It only makes a difference when we tinker with philosophy around the edges, like these conversations about whether what we experience can tell us anything about the experience of other entities.

Is it possible to have perception in a static environment? It seems like perception requires flux of some sort.

Clarification: meaning the machinery of the perceiver must have flux, otherwise it's not perception, it's just static state.

We so far have no way of splitting perception or conscience down in slices of ever shorter moments to see where it stops being whatever it is and becomes something "inert", but even if we did, we would not know whether that was an inherent limitation of objective reality or of our subjective reality and whether those two are the same or not.

BTW experienced mediators apperently can experience no time and no space, no idea how that tastes like.

Sure, they might feel so, but the fact is that all kinds of state changes happen, so time goes on.

It's very much possible that what we would call consciousness can only exist in certain types of minds that have a particular level of intelligence, but also a particular structure.

Of course, it's also possible that you're right and the two are entirely separate, and that consciousness doesn't require intelligence (and possibly vice-versa).

I still enjoy the process though, which perhaps also doesn't require consciousness, yet here I am.

If we look at our advancing world models in things like self driving cars, when would we consider them conscious by our own rules? It has a sensor network that keeps track of it's own internal states. It has external sensors that monitor the external world and adjust it's internal state. The internal state attempts to predict future states and take actions to avoid negative outcomes.

So when does this loop become something we consider consciousness?

I've occasionally run into science podcasts, going back almost a decade, where some researcher talks about the computational power of cell membranes, and how the synapses evolved from these mechanisms. Amoebas and paramecia navigate their environments, sense, and react through their cell membranes. Apparently, synapses evolved from these mechanisms.

The upshot of this for AI, is that the neural network model may be drastically incomplete, with far more computation actually happening inside actual individual neurons.

What are the rationale for thinking it doesn't matter? Or is it the case that researchers think a larger number of simpler neurons could be equivalent to a smaller number of "complex emulated neurons?"

I think everyone should ponder this, when thinking about how they think, like as if they are the one thinking at all.

"Man can do what he wills but he cannot will what he wills.” ― Arthur Schopenhauer, Essays and Aphorisms

Something something LLMs can only predict the next word.

I hate to spin up this trendy debate again, but it's always funny to me to see the dissonance when talking about the exact same things in biological and mathematical cases.

The human mind is _so much more_ than a prediction machine, and incredibly complex... All that's before you get into the way the endocrine system interacts with your mind.

A single neuron has an average of 250000 connections in some parts of the brain. The speed at which neuronal signals travel varies neuron to neuron from 2.5m/s to 200m/s.

Human minds are more than just prediction. The anterior lateral prefrontal cortex has the sole responsibility of prediction (not that nothing else does, just that the ALPC seems solely dedicated to that task) and is extremely good at it. Prediction can influence all sorts of mental processes such as most all forms of perception... But it is _not_ the same as _being_ all forms of perception. If something unpredictable enough happens in front of you: you'll still see it.

Sure there are limits to that: when focused on a task the predictive parts of sight tend to filter out visual data that doesn't match the signal you're looking for (see: basketball players passing the ball and a moon walking man in an ape suit) but if every basketball player turned into Spaghetti-os and started screaming you'd still hear the sounds and see the O's.

So sure: LLMs do a good job at basic prediction but they're nowhere near the complexity of the human mind, of which prediction is only a small piece.

(And let's not even get into efficiency... A brain runs on 20W of power)

At the end of the day this is mostly meaningless.

Biological creatures have to be efficient because they need to harvest energy from the environment around them. But to limit machines to that would be like saying we have to feed biomass to airplanes.

There’s hardly a comparison. If they want to, I’d like to see them start with using dynamic chunking instead of tokenization like the mind does. Also, they can tell me what pre-training method uses 20W per second for 1+ trillion parameters. I could use that right now.

A bunch of cells likely have more than a few GB of data between them, but most of it is just instructions on how to fold proteins to replicate themselves. Is that intelligent behavior or just mindlessly executing a script, albeit in DNA form? I would lean towards the latter.

I suppose it's hard to compare biological intelligent systems directly since they all have to process all available sensory info in a timely manner, not just very narrow text input. Once there are more LMMs trained on vision, sound, and kinematic data it should get far easier to draw comparisons.

> but if every basketball player turned into Spaghetti-os and started screaming you'd still hear the sounds and see the O's

Unless of course someone had set up a Somebody else's problem field generator around them :)

This the right term to use here.

> Something something

If human brains have a model, then is language the transport layer on top of that? Is trying to get to intelligence via language no better than trying to get to "google" by modeling its TCP/IP traffic?

How do you know what job moffkalast has, and why does it matter? This reads like pure snobbery to me.

(Also: moffkalast did not in fact suggest that anything is a solution to a centuries-old problem. "Some common rhetoric about LLMs is too simplistic" is a far cry from "LLMs resolve all the perplexities about human consciousness and thought".)

You can generate all kind of sentences like this all day you want in your consciousness. That does not make it any true.

There is zero evidence for existence of physical matter/materialism.

The only thing we know for sure that exists is consciousness.

And you suggest the complete opposite with zero evidence.

https://en.wikipedia.org/wiki/Cogito,_ergo_sum#Critique

Yes. That is correct.

> This rhetorical strategy thus has a weakness: who or what exactly are you trying to convince?

Ego dissolution/self realiszation takes time. It's my goal in life. But haven't achieved yet. But your comment do point that I should not waste time on engaging in things like this which builds up the ego more.

Let me humbly suggest to you to not make such (Truth) statements! I dont know of any hard evidence that supports this. I know this is what most people believe, but the focus is on believe.

Supposing I accept that, what does this have to do with thought, which is the claim that I was specifically responding to? Does Levin or James also show that this can only be done by having thoughts?

Edit: for instance, as opposed to having some non-thinking process like gradient descent, or more plausibly, some kind of hill climbing.

I honestly think if you (or anyone) watches one of his talks it will blow your mind. Prior to watching his talks, I would have said pretty much the same thing as you, but his work completely upended my view of biology. He’s basically breaking down how we get from “just physics” to a working organism and finding a lot of surprising things. Unfortunately, there’s way more information than I can condense into a comment.

Are you sure brains aren't doing any of it?

Yes. Cognition isn’t just about solving differential equations and the like. It also refers to the most basic functions/processes such as perception and evaluation.

https://consciousness.social