Last active
July 16, 2018 07:44
-
-
Save santiagobasulto/141cb786aa789892a3dd14cabc473482 to your computer and use it in GitHub Desktop.
Despite multiple efforts, I couldn't figure out how to evaluate generators in parallel. Ideas are welcome.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def get_repo_stars(org, repo): | |
url = 'https://api.github.com/repos/{org}/{repo}'.format( | |
org=org, repo=repo) | |
print("GET ", url) | |
resp = requests.get(url) | |
return resp.json()['stargazers_count'] | |
params = [ | |
('requests', 'requests'), | |
('requests', 'httpbin'), | |
('django', 'django'), | |
('Lukasa', 'hyper'), | |
] | |
generator = (get_repo_stars(org, repo) for org, repo in params) | |
def parallel(generator): | |
def _next(): | |
try: | |
val = next(generator) | |
print(val) | |
except StopIteration: | |
pass | |
threads = [threading.Thread(target=_next) for _ in range(2)] | |
[t.start() for t in threads] | |
[t.join() for t in threads] | |
parallel(generator) |
knowsuchagency
commented
Jul 15, 2018
•
Hey, thanks! Yes, I know about map of concurrent.futures
and multiprocessing.Pool
. I was just trying to see if it was possible to make generators parallel.
Generators are always synchronous, but you can make the request async (and therefore do it concurrently).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment