Skip to content

Instantly share code, notes, and snippets.

@EdOverflow
Created April 5, 2018 14:06
Show Gist options
  • Star 8 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save EdOverflow/8e12e8c26b6bc96168e6b55324b91fa1 to your computer and use it in GitHub Desktop.
Save EdOverflow/8e12e8c26b6bc96168e6b55324b91fa1 to your computer and use it in GitHub Desktop.
Find a public Google group for a particular host. Some of these groups contain sensitive information. The tool runs against a list of hosts and returns all public groups.
#!/bin/bash
# Find a public Google group for a particular host.
# Some of these groups contain sensitive information.
# The tool runs against a list of hosts and returns all public groups.
while read domain; do
if curl -LIs "https://groups.google.com/a/$domain" | grep "overview" > /dev/null; then
echo "[+] https://groups.google.com/a/$domain/forum/#!overview"
fi
done < $1
@EdOverflow
Copy link
Author

Or just use this bookmarklet to quickly check the host your are currently viewing:

javascript:window.location="https://groups.google.com/a/"+document.domain+"/forum/#!overview";

@0xmilan
Copy link

0xmilan commented Jan 8, 2020

This script can run a lot faster.
Currently curl makes 3 requests per domain because of 2 redirects.

curl -LIs "https://groups.google.com/a/python.org"

HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 08 Jan 2020 19:55:31 GMT
Location: https://groups.google.com/a/python.org/d/
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
Accept-Ranges: none
Vary: Accept-Encoding

HTTP/1.1 302 Moved Temporarily
Content-Type: text/html; charset=UTF-8
Location: /a/python.org/forum/#!overview
Date: Wed, 08 Jan 2020 19:55:32 GMT
Expires: Wed, 08 Jan 2020 19:55:32 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
Accept-Ranges: none
Vary: Accept-Encoding

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 08 Jan 2020 19:55:33 GMT
Content-Length: 24363
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Set-Cookie: NID=195=VKxxxxxxxxxjYgTxts4; expires=Thu, 09-Jul-2020 19:55:33 GMT; path=/; domain=.google.com; HttpOnly
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000

This is not necessary. You can go straight to "/a/$domain/forum/#!overview" in 1 request:
curl -Is "https://groups.google.com/a/python.org/forum/#\!overview"

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 08 Jan 2020 20:00:58 GMT
Content-Length: 24363
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Set-Cookie: NID=195=kmRxxxxxxxeB0PKQGco; expires=Thu, 09-Jul-2020 20:00:58 GMT; path=/; domain=.google.com; HttpOnly
Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000

Without the redirects we don't even need the -L option.
How do you check if a Google Groups instance is public?
grep for the "Set-Cookie" header, instead of "overview" which appears in the "Location: " header during the redirect.

The Set-Cookie header is only present for public domains, for example:
github.com - exists, but private >> no output after grep
curl -Is "https://groups.google.com/a/github.com/forum/#\!overview" | grep "Set-Cookie"

github123.com - 404 >> no output after grep
curl -Is "https://groups.google.com/a/github123.com/forum/#\!overview" | grep "Set-Cookie"

python.com - public >> grep finds the Set-Cookie header
curl -Is "https://groups.google.com/a/python.org/forum/#\!overview" | grep "Set-Cookie"
Set-Cookie: NID=195=Xpe8xxxxxxxxxxxxxxxx9xir2CevaRUv-IpIT_4anvlJ-PUOISCeoRMJVvoRL5y1RDq8f_wEVxYhU; expires=Thu, 09-Jul-2020 20:16:05 GMT; path=/; domain=.google.com; HttpOnly

I've also changed the echoed URL to https://groups.google.com/a/$domain/forum/#!forumsearch/ so you go straight to the list of groups and don't have to click "Browse all" on the overview page.

So the final script looks something like this:

#!/bin/bash

# Find a public Google Groups for a particular host. 
# Some of these groups contain sensitive information. 
# The tool runs against a list of hosts and returns all public groups.

while read domain; do
    if curl -Is "https://groups.google.com/a/$domain/forum/#\!overview" | grep "Set-Cookie:" > /dev/null; then
        echo "https://groups.google.com/a/$domain/forum/#!forumsearch/"
    fi
done < $1

Time test on 100 domains:
Before:

time ./googlegroups.sh 100domains
./googlegroups.sh 100domains  6,17s user 1,62s system 4% cpu 2:36,17 total

After:

time ./googlegroups.sh 100domains
./googlegroups.sh 100domains  6,09s user 1,60s system 8% cpu 1:34,91 total 

2:36 > 1:34, shaved off a minute.

@0xmilan
Copy link

0xmilan commented Jan 9, 2020

The results above are with 100 domains that are either private or 404. For those curl makes 2 request (1 redirect), not 3.
If you take 100 public domains, then the difference in run time is higher (3 requests / domain vs. 1 request).

Before:
./googlegroups_old.sh 100public 6,29s user 1,50s system 3% cpu 3:48,46 total

After:
./googlegroups.sh 100public 6,20s user 1,56s system 7% cpu 1:38,67 total

3:48 > 1:38
200 requests and more than 2 minutes saved, only on 100 domains.

@EdOverflow
Copy link
Author

Nice work, @milangfx! I wrote this merely as a proof of concept — not focusing on performance. If you really want performance, don't write a while loop in the script itself. Just have the script issue the requests and then run it using GNU parallel.

@0xmilan
Copy link

0xmilan commented Jan 17, 2020

Thanks for the feedback. I was focusing on efficiency in a single threaded case. Making the same request run in parallel would be faster, but still inefficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment