Most SRP calculations happen in the group of integers modulo a large number N. Basically whenever a number x grow beyond N you use x mod N instead.
Here's an example:
N = 7
x = 3
x² = 3*3 = 9 % 7 = 2
x³ = x²*x = 2*3 = 6
x^4 = x²*x² = 2*2 = 4
x^4 = x³*x = 6*3 = 18 % 7 = 4
As you can see in the last examples you can still apply your known math tricks.
Finding the exponent b in x^b = y in this group is called the discrete logarithm problem https://en.wikipedia.org/wiki/Discrete_logarithm . Computing discrete logarithms is believed to be difficult. No efficient general method for computing discrete logarithms on conventional computers is known, and SRP bases it's security on the assumption that the discrete logarithm problem has no efficient solution.
It's fairly easy to agree upon a shared key and prevent anyone listening from calculating the same key without solving the discrete log problem (Diffie Hellman Key Exchange):
client picks random number a
server picks random number b
client calculates A=g^a
server calculates B=g^b
client sends A to server
server sends B to client
server calculates S=A^b=g^(a*b)
client calculates S=B^a=g^(a*b)
As you can see both sides now know a shared secret S. Anyone who only knows A and B will have a hard time calculating S.
SRP includes the password in the key exchange and thus allows the server to authenticate the client.