Skip to content

Instantly share code, notes, and snippets.

@InnovArul
Last active June 17, 2023 22:09
Show Gist options
  • Save InnovArul/14af4577b6d8cd19efd6a5b275cfc189 to your computer and use it in GitHub Desktop.
Save InnovArul/14af4577b6d8cd19efd6a5b275cfc189 to your computer and use it in GitHub Desktop.

Answering https://discuss.pytorch.org/t/a-set-of-data-sum-as-the-dividend-how-to-find-grad/182303:

We can find $grad_x = \frac{dz}{dx}$ in the following way:

$$grad_x = \frac{dz}{dx} = \left[\frac{dz}{dx_0},\frac{dz}{dx_1}, \frac{dz}{dx_2}\right]$$

$$ = \frac{dz}{dx} = \left[\frac{dy}{dx_0}\frac{dz}{dy},\frac{dy}{dx_1}\frac{dz}{dy},\frac{dy}{dx_2}\frac{dz}{dy}\right]$$

Now let's concentrate on finding one element $\frac{dz}{dx_0} = \frac{dy}{dx_0}\frac{dz}{dy}$

Here $z$ is just $y[1].sum()$ (=1st case) or $y.sum()$ (=2nd case). It is a 3x1 vector. In the first case, $\frac{dz}{dy} = [0, 1, 0]^T$.

In the second case, $\frac{dz}{dy} = [1,1,1]^T$.

To find $\frac{dy}{dx_0}$, we have to differentiate the vector $[y_0, y_1, y_2] = \left[\frac{x_0^2}{sum(x)}, \frac{x_1^2}{sum(x)}, \frac{x_2^3}{sum(x)}\right]$ with respect to $x_1$.

$\frac{d\frac{x_0^2}{sum(x)}}{dx_0} = \frac{sum(x).2x_0 - x_0^2.1}{sum(x)^2} = \frac{6.(2.1) - 1^2.1}{36} = \frac{11}{36} = 0.30555$

$\frac{d\frac{x_1^2}{sum(x)}}{dx_0} = \frac{sum(x).0 - x_1^2.1}{sum(x)^2} = \frac{6.(0) - 2^2.1}{36} = \frac{-4}{36} = -0.11111$

$\frac{d\frac{x_2^2}{sum(x)}}{dx_0} = \frac{sum(x).0 - x_2^2.1}{sum(x)^2} = \frac{6.(0) - 3^2.1}{36} = \frac{-9}{36} = -0.25$

Now, $\frac{dy}{dx_0} = [0.30555, -0.11111, -0.16666]$

Further,

in $1^{st}$ case, $\frac{dz}{dx_0} = \frac{dz}{dy}\frac{dy}{dx_0} = [0.30555, -0.11111, -0.25].[0, 1, 0]^T = -0.11111$,

in $2^{nd}$ case, $\frac{dz}{dx_0} = \frac{dz}{dy}\frac{dy}{dx_0} = [0.30555, -0.11111, -0.25].[1, 1, 1]^T = -0.05556$

Similarly we can calculate for $\frac{dz}{dx_1}$ and $\frac{dz}{dx_2}$.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment