## Neural Networks and Partial Derivatives, why?

A place to discuss the science of computers and programs, from algorithms to computability.

Formal proofs preferred.

Moderators: phlip, Moderators General, Prelates

pedroj012
Posts: 49
Joined: Thu Jul 12, 2012 12:49 am UTC

### Neural Networks and Partial Derivatives, why?

I started taking a course through coursera or maybe audacity that explained how one type of neural network works --- and the readjustment of the network, where the strengths of the connections are adjusted, involved solving partial differential equations, and maybe this is really stupid, but it made me think - neural networks are based on neurons and the connections between them, and obviously our neurons aren't solving differential equations as they adjust synaptic strength - so there has to be a simpler way right? Like a much less computationally intense way.

TheAmazingRando
Posts: 2308
Joined: Thu Jan 03, 2008 9:58 am UTC
Location: San Diego, CA

### Re: Neural Networks and Partial Derivatives, why?

We use calculus to understand electrical circuits too, but that doesn't mean the wires are solving differential equations.

Xanthir
My HERO!!!
Posts: 5311
Joined: Tue Feb 20, 2007 12:49 am UTC
Contact:

### Re: Neural Networks and Partial Derivatives, why?

You're getting confused by assuming that physical systems and computers work the same way. Physical systems "compute" by performing ridiculous numbers of very simple calculations in a massively parallel fashion (subatomic particles solving the field equations defined by physics). Computers compute in a complete different manner, serially attacking complicated problems. Your language makes it clear that you're thinking of a neuron as a "thing" which does "computation". It's not - it's a loosely-bound collection of trillions of quantum variables interacting via quantum physics. "Being a neuron" only shows up when you zoom *way* out and abstract a lot, from quantum variables to subatomic particles to atoms to molecules to cells.

There are lots of things that are very simple to do in the "physical" way, but difficult to do in the "computer" way, and vice versa.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

ConMan
Shepherd's Pie?
Posts: 1663
Joined: Tue Jan 01, 2008 11:56 am UTC
Location: Beacon Alpha

### Re: Neural Networks and Partial Derivatives, why?

The PDEs, and especially things like the gradient-descent method of trying to find a solution/equilibrium point to them, are just part of working with the abstraction. It's been a while since I did the Stanford ML course (the predecessor to the Coursera one), but the general idea is that a real neuron takes some set of inputs (that are varying strengths of an electrical signal) and produces an output, and that there is some process that modifies how the output relates to the input based on some idea of how it "should" respond. So, in creating our "virtual neurons", we find that they can be defined as elements of a rather large function - but, since we know that we want this function to respond in a particular way to our training data, we can state the difference between the current state of the network and this desired state as a kind of differential equation, which we can then solve numerically through a method like gradient-descent.

It's similar in ways to the point of how we can describe the trajectory of a ball through differential equations, but our brains don't have to actually perform the calculus in order for us to catch it.
pollywog wrote:
Wikihow wrote:* Smile a lot! Give a gay girl a knowing "Hey, I'm a lesbian too!" smile.
I want to learn this smile, perfect it, and then go around smiling at lesbians and freaking them out.

pedroj012
Posts: 49
Joined: Thu Jul 12, 2012 12:49 am UTC

### Re: Neural Networks and Partial Derivatives, why?

it's a loosely-bound collection of trillions of quantum variables interacting via quantum physics. "Being a neuron" only shows up when you zoom *way* out and abstract a lot, from quantum variables to subatomic particles to atoms to molecules to cells.

But that's anything, right? Billiard balls are a loosely-bound collection of trillions of quantum variable interacting via quantum physics - and yet they act in very predictable ways at the zoomed out perspective of seeing billiard balls collide instead of trillions of molecules.

You're saying the biological equivalent of solving the differential equations is just inherent in the makeup of neurons/synapses, right?
I guess what I'm saying is that it seems like there should be a better/simpler way to abstract molecular learning. --- somehow make it inherent in neural networks rather than a step that requires shutting down the network and computing all of the differential equations.

Are neural networks used or thought to be useful for something like a robot trying to catch a ball?

Xanthir
My HERO!!!
Posts: 5311
Joined: Tue Feb 20, 2007 12:49 am UTC
Contact:

### Re: Neural Networks and Partial Derivatives, why?

pedroj012 wrote:
it's a loosely-bound collection of trillions of quantum variables interacting via quantum physics. "Being a neuron" only shows up when you zoom *way* out and abstract a lot, from quantum variables to subatomic particles to atoms to molecules to cells.

But that's anything, right? Billiard balls are a loosely-bound collection of trillions of quantum variable interacting via quantum physics - and yet they act in very predictable ways at the zoomed out perspective of seeing billiard balls collide instead of trillions of molecules.

Yes, but billiard balls aren't "solving simple linear equations" when they collide, either, though that's what we use when we model their interactions with math. The physics behind billiard balls, neurons, and flowing fluids are all essentially the same; it's only the tiniest of details that make them so different.

You're saying the biological equivalent of solving the differential equations is just inherent in the makeup of neurons/synapses, right?
I guess what I'm saying is that it seems like there should be a better/simpler way to abstract molecular learning. --- somehow make it inherent in neural networks rather than a step that requires shutting down the network and computing all of the differential equations.

No, I'm saying that the action of lots and lots of tiny, local physics calculations in parallel can be modeled with a differential equation. We use a single, complicated equation because it's easier to work with than doing a trillion-particle simulation. The particles aren't "solving a differential equation", they're doing something else that can also be represented by a diffeq.

Are neural networks used or thought to be useful for something like a robot trying to catch a ball?

Neural networks have nothing to do with brains; go ahead and remove that thought from your head right now. Their name just comes from the fact that their structure is vaguely similar, with lots of small, simple, independent units sending values to each other based on how much value they've received.

Neural networks are useful just because they can be "trained" pretty easily (really, twerk the parameters of the connections between units based on whether the unit contributed to a "correct" or "incorrect" answer), and once you cross a certain complexity threshold, they can represent any computable function. So, if you have enough training data to run tests on, you can work up a decent solver for your problem without having to do much actual work yourself.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

pedroj012
Posts: 49
Joined: Thu Jul 12, 2012 12:49 am UTC

### Re: Neural Networks and Partial Derivatives, why?

gotcha. So AI researchers don't work with neural networks?

Xanthir
My HERO!!!
Posts: 5311
Joined: Tue Feb 20, 2007 12:49 am UTC
Contact:

### Re: Neural Networks and Partial Derivatives, why?

Some do - they're useful tools, but so are plenty of other things. However, it's correct to say that NNs have nothing intrinsically to do with artificial intelligence.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

Posts: 3072
Joined: Mon Oct 22, 2007 5:28 pm UTC
Location: Beaming you up

### Re: Neural Networks and Partial Derivatives, why?

An artificial neural network is something along the lines of a data structure for bounded approximation. If you wanted to computationally find the highest point of some arbitrary curve, you might pick a point at random (x), then slowly move that point in the "up" direction until you get to a point on the curve where both directions lead you "down", finding the maximum f(x).

A neural network is ultimately a function from n inputs to m outputs, which are all real numbers. There's "inner neurons" and connections between them, which are ultimately just additional variables to play with, along the lines of x above.

A very important note here is that you are only changing the innards, not the input or output. The hill-climbing example above can be thought of as a way of getting a pair of (x, f(x)) out of a function f.

In the same vein, one application of neural networks is generating a function f from a set of inputs (x, f(x)). You pass your dataset to the function, which is expressed by the innards of the neural network. This is where partial derivatives come in. Solving those equations is how you find "up". Your neural network (f) travels "up" when you change the innards to be closer to matching the original dataset.

Solving partial derivatives is not an intrinsic part of a neural network, or particularly necessary to using them. They are used in many operations on neural networks, but there's nothing else special going on.

Ultimately, neural networks are metaphorical "dudes climbing a hill". Don't let the magical name distract you. Just follow the math, and keep in mind that operations on neural networks are their own separate things. After all, we don't ask how numbers are able to solve systems of linear equations... ;)
<quintopia> You're not crazy. you're the goddamn headprogrammingspock!