This is a proof of the vector decomposition omitted in Raytracing in One Weekend, section 10.2(Snell's Law)
Let $R_{p}'$ be the decomposed vector perpendicular to the normal $n$, and $R_{n}'$ be the other decomposed vector parallel to $n$. Without loss of generality, we can assume $|R| = |R'| = 1$(this is omitted in the book). Also, let $p$ be a unit vector with the same direction as $R_{p}'$(so it is perpendicular to $n$ as well).
Now we can simply decompose $R' = R_{p}' + R_{n}'$ such that
$$R_{p}' = \sin\theta'p, \text{ } R_{n}' = -\cos\theta'n$$
Since the book shows the result of $R_{p}'$ first, let's investigate this one first.
By Snell's law, we have $R_{p}' = \frac{\eta}{\eta'}\sin\theta p$. Also, if we recall how we get the reflected vector in the above section 9.4 in the book, we can easily see that $\sin\theta p = R + \langle R, n \rangle n = R + \cos\theta n$(remember that $|R| = 1$!). Therefore, we have
$$R_{p}' = \frac{\eta}{\eta'}\sin\theta p = (\frac{\eta}{\eta'}\sin\theta)(\frac{1}{\sin\theta}(R + \cos\theta n)) = \frac{\eta}{\eta'}(R + \cos\theta n)$$ as desired.
On the other hand, $R_{n}' = -\cos\theta'n = -\sqrt{1 - \sin^{2}\theta'}n$(since we assume that $\theta$ and $\theta'$ are both acute angles), and $\sin\theta' = |R_{p}'|$, so we have
$$R_{n}' = -\sqrt{1 - \sin^{2}\theta'}n = -\sqrt{1 - |R_{p}'|^{2}}n$$
which is the same as what the book shows.
