let's zoom in at this very particular question of computation on a processor and communication between processors so what does this system look like that you're envisioning one of the places you're envisioning it is in the paper on optoelectronic intelligence so what are we talking about are we talking about something that starts to look a lot like the human brain or does it still look a lot like a computer what are the size of this thing is it go inside a smartphone or as you said does it go inside something that's more like a house like uh what should we be imagining what are you thinking about when you're thinking about these fundamental systems let me introduce the word neuromorphic there's this concept of neuromorphic computing where what that broadly refers to is computing based on the information processing principles of the brain and as digital computing seems to be pushing towards some fundamental performance limits people are considering architectural advances drawing inspiration from the brain more distributed parallel network kind of architectures and stuff and so there's this continuum of neuromorphic from things that are pretty similar to digital computers but maybe there are more cores and the way they send messages is a little bit more like the way brain neurons send spikes but for the most part it's still digital electronics and then you know you have some things in between where maybe you're you're using transistors but now you're starting to use them instead of in a digital way in an analog way and so you're trying to get those circuits to behave more like neurons and then that's a little bit quite quite a bit more on the neuromorphic side of things you're trying to get your circuits although they're still based on silicon you're trying to get them to perform operations that are highly analogous to the operations in the brain that's where a great deal of work is in neuromorphic computing people like yakimo and davari and gert kaunbergs jennifer hasler countless others it's it's a rich and exciting field uh going back to carver mead in the late 1980s and then all the way on the other extreme of the continuum is where you say i'll give up anything related to transistors or semiconductors or anything like that i'm not not starting with the assumption that i'm going to use any kind of conventional computing hardware and instead what i want to do is try and understand what makes the brain powerful at the kind of information processing it does and i want to think from first principles about what hardware is best going to enable us to capture those information processing principles in an artificial system and that's where i live that's where that's where i'm doing my exploration these days so uh what are the first principles of brain like computation communication right yeah this is this is so important and i'm glad we booked 14 hours for this because uh i only have 13 i'm sorry okay so the brain is notoriously complicated and i think that's a an important part of why it why it can do what it does but okay let me let me try to break it down starting with the devices neurons as i as i said before they're they're sophisticated devices in and of themselves and synapses are too they they can um change their state based on the activity so they they adapt over time that's crucial to the way the brain works they don't just adapt on one time scale they can adapt on myriad time scales from the the spacing between pulses the spacing between spikes that come from neurons all the way to the age of the organism um also relevant perhaps i think the most important thing that's guided my thinking is the the network structure of the brain so which can also be adjusted yes on different scales absolutely yes so so you're you're making new con you're changing the strength of contacts you're changing the the spatial distribution of them all those spatial distribution doesn't change that much once you're a mature organism but that network structure is is really crucial so let me dwell on that for a second um you can't talk about the brain without emphasizing that most of the neurons in the the neocortex or the prefrontal cortex the part of the brain that we think is most responsible for high-level reasoning and things like that those neurons make thousands of connections so you have this network that is highly interconnected and i think it's safe to say that one of the primary reasons that they make so many different connections is that allows information to be communicated very rapidly from any spot in the network to any other spot in the network so that's a that's a sort of spatial aspect of it you can quantify this in terms of concepts that are related to fractals and scale invariants which i think is is a very beautiful concept so what i mean by that is kind of no matter what spatial scale you're looking at in the brain within certain bounds you see the same general statistical patterns so if i draw a box around some region of my cortex most of the connections that those neurons within that box make are going to be within the box to each other in their local neighborhood and that's sort of called clustering loosely speaking but a non-negligible fraction is going to go outside of that box and then if i draw a bigger box the pattern is going to be exactly the same so you have the scale and variance and you also have a non-vanishing probability of a neuron making connection very far away so suppose you you want to plot the probability of a neuron making a connection as a function of distance if that were an exponential function it would go e to the minus radius over some characteristic radius and it would it would drop off up to some certain radius the probability would be reasonable close to one and then a beyond that characteristic length r zero it would it would drop off sharply and so that would mean that the neurons in your brain are really localized and that's not what we observe instead what you see is that the probability of making a longer distance connection it does drop off but it drops off as a power law so the probability that you're going to have a connection at some radius r goes as r to the minus some power and that's more that's what we see with with forces in nature like the electromagnetic force between two particles or gravity goes as one over the radius squared so you can see this in fractals i love that there's a like a fractal dynamics of the brain that if you zoom out you draw the box and you increase that box by certain step sizes you're going to see the same statistics i think that's probably very important to the way the brain processes information it's not just in the spatial domain it's also in the temporal domain and what i mean by that is that's incredible that this emerged through the evolutionary process that potentially somehow connected to the way the physics of the universe works yeah i i couldn't agree more that it's a deep and fascinating subject that i i hope to be able to spend my life studying you think you need to solve understand this this fractal nature in order to understand intelligence and company i do think so i think they're deeply intertwined yes i think power laws are right at the heart of it so just to just to push that one through the same thing happens in the temporal domain so suppose you had um suppose your neurons in your brain were always oscillating at the same frequency then the probability of finding a neuron oscillating as a function of frequency would be this narrowly peaked function around that certain characteristic frequency that's not at all what we see the probability of finding neurons oscillating or pulsing producing spikes at a certain frequency is again a power law which means there's no there's no defined scale of the temporal activity in the brain it's you don't what at what speed do your thoughts occur well there's a there's a fastest speed they can occur and that is limited by communication and other other things but there's not a characteristic scale we have thoughts on all temporal scales from you know a few tens of milliseconds which is physiologically limited by our devices compare that to tens of picoseconds that i talked about in superconductors all the way up to the lifetime of the organism you can still think about things that happened to you when you were a kid or if you want to be really trippy then across multiple organisms in the entirety of human civilization you have thoughts that span organisms right yes taking it to that level if you're willing to see the entirety of the human species as a single organism with the collective intelligence and that too on a spatial and temporal scale there's thoughts occurring and then if you look at not just the human species but the entirety of life on earth as as an organism with thoughts that occurring that are greater and greater sophisticated thoughts there's a different spatial and temporal scale there this is getting very suspicious hold on though before we're done i just want to just tie the bow yes and say that the the spatial and temporal aspects are intimately interrelated with each other so activity between neurons that are very close to each other is more likely to happen on this this faster time scale and information is going to propagate and encompass more of the brain more of your cortices different modules in the brain are going to be engaged in information processing on longer time scales so there's this concept of information integration where most neurons are neurons are specialized any given neuron or any cluster of neuron has its specific purpose but they're also um they're also very much integrated so you you have neurons that specialize but share their information and so that happens through these fractal nested oscillations that occur across spatial and temporal scales i think capturing those dynamics in in hardware to me that's the goal of of neuromorphic computing so does it need to look so first of all that's fascinating we stated some clear principles here now does it have to look like the brain outside of those principles as well like what other characteristics have to look like the human brain or can it be something very different well it depends on what you're trying to use it for and so i i think a lot of the community asks that question a lot what are you going to do with it and i i completely get it i think that's a very important question and it's also sometimes not the most helpful question what if what you want to do with it is study it what if you just want to see um what does it what do you have to build into your hardware in order to observe these dynamical principles so and also i ask sometimes i ask myself that question every day and i'm not sure i'm able to answer that it's like what are you what are you gonna do with this particular neuromorphic machine so suppose what we're trying to do with it is build something that thinks we're not trying to get it to make us any money or drive a car maybe we'll be able to do that but that's not our goal our goal is to see if we can get the same types of behaviors that we observe in our own brain and by behaviors in this sense what i mean the behaviors of the components the neurons the network that kind of stuff i think there's another element that i didn't really hit on that that you also have to build into this and those are architectural principles they have to do with the the hierarchical modular construction of the network and without getting too lost in jargon the the main point that i think is relevant there let me try and illustrate it with a cartoon picture of the architecture of the brain so in the brain you have the the cortex which is sort of this outer sheet um it's actually a you can it's a layered structure you can if you could take it out of your brain you could unroll it on the table and it would be about the size of a of a pizza sitting there and um that's a module it it does certain things it it processes as yorgi buzaki would say it processes the what of of what's going on around you but you have another really crucial module that's called the hippocampus and that that network is structured entirely differently first of all this this cortex that had described 10 billion neurons in there so numbers matter here and they're they're organized in that sort of power law distribution where the probability of making a connection drops off as a power law in space the hippocampus is another module that's important for understanding how where you are when you are um keeping track of of your your position in space and time and that network is very much random so the probability of making a connection it it almost doesn't even drop off as a function of distance it's the same probability that you'll make it here to over there but there are only about 100 million neurons there so you can have that huge densely connected module because it's not so big and the the neocortex or the cortex and the hippocampus they talk to each other constantly and that communication is largely facilitated by what's called the thalamus i'm not a neuroscientist here i'm trying to do my best to recite this cartoon picture of the brain i gotcha yeah something like that so this thalamus is is coordinating the activity between the neocortex and the hippocampus and making sure that they they talk to each other at the right time and send messages that will be useful to one another so this all taken together is called the thalamocortical complex and it seems like building something like that is going to be crucial to capturing the types of activity we're looking for because though those responsibilities those separate modules they do different things that's got to be central to achieving these states of efficient information integration across space and time by the way i am able to achieve this state by watching simulations visualizations of the thelma cortical complex there's a few people i forget from where they've created these incredible visual illustrations of like visual stimulation from the eye or something like that it this in this image like flowing through the brain wow i haven't seen that i got to check that out so it's one of those things you you find this stuff in the world and you see like on youtube it has like 1000 views these like these visualizations of the human brain processing information and like because there's uh there's chemistry there like because this is act from actual human brains i don't know how they're doing the coloring but they're able to actually trace the uh like different the the chemical and the electrical signals throughout the brain and the visual thing it's like whoa because it looks kind of like the universe i mean the whole thing is just incred i recommend it highly i'll probably post a link to it but you can just look for uh um one of the things they simulate is the uh thelma cortical uh complex and just visualization you can find that yourself on youtube but it's it's beautiful um the other question i have for you is um how does memory play into all this because all the signals sending back and forth that's kind of like uh that's computation and communication but that's kind of like uh you know processing of inputs and outputs to produce outputs in the system that's kind of like maybe reasoning maybe there's some kind of recurrence but like is there a storage mechanism that you think about in the context of neuromorphic computing yeah absolutely so that's got to be central you have to have a way that you can store memories and there are a lot of different kinds of memory in the brain that's yet another example of how it's not a simple system so there's one kind of memory one way of talking about memory uh usually starts in the context of hopfield networks you were lucky to talk to john hopfield on this program but the the basic idea there is uh working memory is stored in the dynamical patterns of activity between neurons and you can you can think of a certain pattern of activity as an attractor meaning if you put in some signal that's similar enough to other previously experienced signals like that then you're going to converge to the same network dynamics and you will see these neurons participate in the same network patterns of activity that they have in the past so you can talk about the probability that different inputs will allow you to converge to different basins of attraction and you might think of that as oh i saw this face and then i excited this network pattern of activity because last time i saw that face i was at you know what some movie and that that's a famous person on the screen or something like that so so that's one memory storage mechanism but crucial to the ability to imprint those memories in your brain is the ability to change the strength of connection between one neuron and another that synaptic connection between them so synaptic weight update is a massive field of neuroscience and neuromorphic computing as well so there are two poles to that on that spectrum one in okay so in more in the language of machine learning we would talk about supervised and unsupervised learning in when i'm trying to tie that down to neuromorphic computing i will use a definition of supervised learning which basically means the external user the person who's controlling this hardware has some knob that they can tune to change each of the synaptic weights depending on whether or not the network's doing what you want it to do whereas what i mean in this conversation when i say unsupervised learning is that those synaptic weights are are dynamically changing in your network based on nothing that the user is doing nothing that there's no wire from the outside going into any of those synapses the network itself is reconfiguring those synaptic weights based on physical properties that you've built into the devices so if if the synapse receives a pulse from here and that causes the neuron to spike some circuit built in there with no help from me or anybody else adjust the weight in a way that makes it more likely to store the useful information and excite the useful network patterns and makes it less likely that random noise useless communication events will have an important uh effect on the network activity so there's memory encoded in the weights uh the synaptic weights what about the formation of something that's not often done in machine learning the formation of new synaptic connections right well that seems to so again not not a neuroscientist here but my reading of the literature is that that's particularly crucial in early stages of brain development where a newborn is uh born with tons of extra synaptic connections and it's actually pruned over time so the number of synapses decreases as opposed to growing new long-distance connections it is possible in the brain to grow new neurons and um assign new synaptic connections but it doesn't seem to be the primary mechanism by which the brain is learning so for example like right now sitting here talking to you you say lots of interesting things and i learn what i learn from you and i can remember things that you just said and i didn't grow new axonal connections down to new synapses to to enable those it's plasticity mechanisms in the between the synaptic connections between neurons that enable me to learn on that time scale so at the very least that you can sufficiently approximate that with just weight updates you don't need to form new connections i would say weight updates are a big part of it i also think there's more because broadly speaking when we're doing machine learning our networks say we're talking about feed forward deep neural networks the temporal domain is not really part of it okay you're going to put in an image and you're going to get out of classification and you're going to do that as fast as possible so you care about time but time is not part of the essence of this thing really whereas in spiking neural networks what we see in the brain time is as crucial as space and they're intimately intertwined as i've tried to say and so adaptation on on different time scales is important not not just in memory for formation although it plays a key role there but also in just keeping the activity in a useful dynamic range so you have other plasticity mechanisms not just weight update or at least not on the time scale of many action potentials but even on the shorter time scale so a synapse can become much less efficacious it can it can transmit a weaker signal after the second third fourth that can uh second third fourth action potential to occur in a sequence so that's what's called short-term synaptic plasticity which is a form of learning you're learning that i'm getting too much stimulus from looking at something bright right now so i need to tone that down you know um there's also another really important mechanism in learning it's called metaplasticity what that seems to be is a a way that you change not the weights themselves but the rate at which the weights change so when i am in say a lecture hall and my this is a potentially terrible cartoon example but let's say i'm in a lecture hall and uh it's time to learn right so my brain will release more perhaps dopamine or some neuromodulator that's going to change the rate at which synaptic plasticity occurs so that can make me more sensitive to learning at certain times more sensitive to overwriting previous information and less sensitive at other times and finally as long as i'm rattling off the list i think another concept that falls in the category of learning or memory adaptation is homeostasis or homeostatic adaptation where neurons have the ability to control their firing rate so if if one neuron is just like blasting way too much it will naturally tone itself down it's its threshold will adjust so that it's it stays in a useful dynamical range and we see that that's that's captured in in deep neural networks where you don't just change the synaptic weights but you can also move the thresholds of of simple neurons in those models and so to uh to achieve this spiking neural networks you want to use like you want to implement the first principles that you mentioned of the temporal and the spatial fractal dynamics here so you can you can communicate locally you can communicate across much greater distances and do the same thing in space and do the same thing in time you