Skip to content

Instantly share code, notes, and snippets.

@raychorn
Created October 17, 2020 14:17
Show Gist options
  • Save raychorn/64afddc58139e867a6496689f13db6d7 to your computer and use it in GitHub Desktop.
Save raychorn/64afddc58139e867a6496689f13db6d7 to your computer and use it in GitHub Desktop.
Make Python faster by using GO as efficiently as possible with minimal effort. (with a bit of RUST).
Problem Statement:
Use Go to augment the processing power of Python.
Solutions:
(1) Call GO from Python:
This can be done but it is a bit tedious as shown below:
# loading shared object
lib = cdll.LoadLibrary("main.so")
# go type
class GoSlice(Structure):
_fields_ = [("data", POINTER(c_void_p)), ("len", c_longlong), ("cap", c_longlong)]
lib.Foo.argtypes = [GoSlice]
lib.Foo.restype = c_void_p
t = GoSlice((c_void_p * 5)(1, 2, 3, 4, 5), 5, 5)
f = lib.Foo(t)
print(f)
This might not be practical if one wants to manipulate more than just a small bit of data.
(2). Build a small fast Go Web Server attached to localhost and issue web services with blobs of JSON.
This may seem less efficient than (1) but... JSON tends to make data more transportable and easier to handle.
And GO can easily ingest JSON and make sense of it. GO does have micro-threading which means GO can be more easily used to
process large masses of data quickly. And (2) does tend to leverage parallel processing because one can deploy many
GO Web Servers each able to process blobs of data quickly in parallel.
(2a). Make it Secure.
TOTP -> https://github.com/raychorn/go_totp
Client in Python uses TOTP to generate a one-time token for the client which is GO.
Server in GO uses TOTP to validate the one-time token from Python then GO generates a response with a one-time token that Python
will use to validate the response.
Both Client and Server use the same Pass-Phrase which means they can cross-validate for each other.
Tokens are hidden inside GUID values so any observers will "think" they are seeing GUID values being passed back and forth.
Tokens are typically 6 numbers so easy to blend into a typical GUID.
Communications can be secured via self-signed SSL for a bit of added security but not all that necessary unless one is working
to achieve a bit more security. Personally, I would just use SSL and be done with this part of the architecture because it's
easy and minimally secure.
Pass-phrases can be auto-generated by Python and stored in a JSON file for ingestion by GO and then updated by Python by restarting
the GO processes.
TCP/IP ports can be managed by Python and changed for each run.
RUST can be used in the form of Python extensions (see also: https://github.com/PyO3/pyo3) to augment data analysis, as-required.
(c). Copyright Ray C Horn, All RIghts Reserved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment