Tor Bug 4185 Testing and Report

00-report.txt

Great Firewall Tor Probing Circa 09 DEC 2011
--------------------------------------------
Author: Tim Wilde, Team Cymru, Inc. <twilde@cymru.com>
Date: 06 JAN 2012

This analysis was performed during the week of 05 DEC 2011 - 09 DEC 2011, in
an attempt to determine the extent and type of blocking and probing being
performed by the Great Firewall of China (henceforth "GFW") against
unpublished Tor bridges located outside of China, but accessed from within
China.

These results are considered valid at the time of their collection (05-09 DEC
2011), and no guarantees are provided to their continuing validity beyond
that time, due to the changing nature of anti-circumvention techniques and
technologies.

This analysis was originated based on Tor trac ticket #4185.  A github
"gist" of the results, including this report, can be found at
https://gist.github.com/da3c7a9af01d74cd7de7

Pre-Analysis Conditions, Goals, and Assumptions
-----------------------------------------------

Prior to the initiation of this round of analysis, it had been determined
that any Tor bridge outside of China that was connected to by a client
within China would be blocked by the GFW, usually within a matter of
minutes.  Previous analysis revealed that the Tor connection initiated by
the client resulted in a series of probes from a variety of other IPs within
China (varying from connection to connection), and, within minutes of those
probes, the client's connection to the bridge would be blocked and could not
be re-initiated.  Blocking all connections to the bridge except from the
intended client was shown to allow the connection to persist successfully
for > 48 hours, thus showing a direct connection between the probing and
blocking.  This previous analysis was performed ad-hoc without dedicated
attention/resources, and the analysis described in this report was performed
as a follow-up under more stringent conditions to attempt to determine the
exact parameters of probing and blocking.

The primary goals of this analysis were:

1) Determine the extent of GFW probing of Tor connections, and more broadly,
SSL sessions in general; what triggers the probing, what time considerations
are involved, etc.

2) Determine how the decision to block is made.

3) Determine a course forward for Tor to circumvent probing and blocking.

It is assumed that, despite a significant series of probes originating from
a single host, the GFW does not change behavior to prevent us from learning
what they are doing.  This assumption has been tested with a smaller set of
tests from some other hosts and appears to hold true, but cannot be
conclusively proven.

During the previous analysis it was noted that blocking appeared to be
performed against specific destination IP:port pairings; attempting to
connect to a new port on a host after a previous connection had been blocked
was successful, but provoked new probing and blocking in the same way as the
previous port.  It is assumed that this destination IP:port pairing is still
in use in current techniques.

Summary of Conclusions
----------------------

Goal #2 was found to be unachievable at the time of analysis, as it was not
possible throughout our testing to provoke blocking.  Testing of connections
to "new" published Tor relays (exact dates for "new" not determined) showed
that even published Tor relays were connectable from within China.  Once
this was determined, focus shifted to understanding the probing and how it
could be avoided (goals #1 and #3).

With respect to goal #1, our testing showed that the GFW appears to perform
"garbage binary" probes of the non-China side of any connection from China
to TCP port 443 that performs an SSL negotiation.  This probe is performed
in near-real-time after the connection is established, implying
near-line-rate deep packet inspection (DPI) capabilities.  TCP/443
connections that did not actually exchange an SSL handshake, such as using
the obfsproxy obfs2 protocol, did not provoke probing.  At this time the
purpose of this probe is unknown; further details of its contents and
speculation as to its purpose can be found later in this report.

A second type of probing, which appears to be specifically aimed at Tor, was
also detected.  This probing appeared to occur using a client that actually
spoke SSL, and, within the SSL session, the Tor protocol, and was not
activated in real-time, but appeared to be triggered to occur at regular 15
minute past the hour intervals if its conditions were met.  Analysis showed
that the Tor SSL cipher list was at least one significant component of the
detection mechanism that initiated this probing; a client modified to remove
a sincle cipher from the list, when connecting to an unmodified unpublished
bridge in the United States, did not trigger probing.  The SSL "client
hello" packet was determined as a direct cause of the probing; when just
that data was sent to an open Tor bridge from within China using netcat, the
probing was triggered on the expected 15 minute interval.

A connection between a Chinese client and a US unpublished bridge node over
the obfsproxy obfs2 protocol was able to successfully persist for several
days, including numerous disconnects/reconnects, without provoking any type
of probing.

Recommendations
---------------

In the immediate term, modifying the Tor SSL client hello to present a list
of ciphers more closely (ideally identically) matching that of a common
commodity web browser would bypass the current probe trigger.  In general,
the items in Proposal 179 remain valid and a wise choice; the same eye
should be cast towards the client hello, and an attempt made to match it as
closely to commodity web browsers as possible.  With both sides of that
equation implemented, it becomes much more difficult for any authoritarian
regime to distinguish Tor traffic from standard SSL traffic, requiring
significantly more effort to locate unpublished bridge nodes and connections
to them.

In the longer term, the two problems that need to be solved are:

1) Client-server negotiation that is uniquely identifiable in any way.

2) The ability of an arbitrary adversary to, given a suspected Tor server
IP:port combination, connect to that IP:port combination and perform a
negotiation using the Tor protocol, or anything that can be uniquely
identified to a Tor protocol negotiation.

Problem #1 can be solved by a transport that either looks identical on both
sides to a standard HTTPS SSL session, or that looks like nothing
identifiable at all, like the obfs2 protocol provided by obfsproxy.

Problem #2 is somewhat more difficult.  Tor proposal 190 combined with the
ideas in [0] is one method that has some promise.  The "shared secret"
implementation of obfs2 in obfsproxy is another, as an adversary attempting
to initiate an obfs2 connection without the right shared secret would be
unable to get anything but noise back from the server.  That noise, however,
which the server currently emits immediately upon connection in the current
implementation, may be trivially fingerprintable, as it is not entirely
common in the wild.

[0] http://archives.seul.org/or/dev/Nov-2011/msg00073.html

Methodology
-----------

A single system within China was used to initiate connections to a series of
Tor bridges provisioned within the Amazon EC2 cloud computing product,
within the United States.  EC2 was utilized to allow for rapid changing of
IPs while maintaining toolsets and datasets, to eliminate possible
black/whitelisting of IPs by the GFW.  All test sessions were monitored with
full packet captures generated on the bridge nodes, and at several points in
the process Tor debug logs were captured from the bridge as well. 
Reasonable efforts were made to ensure that each test used a unique
connection tuple to prevent cross-contamination.

General probe observations
--------------------------
All probes came from a wide range of IP addresses within China.  A sampling
of these IPs as collected through our analysis can be found on the following
github gist:

    https://gist.github.com/4320b75d398f2e1f074d

This is by no means an exhaustive list; we have not yet undertaken to
attempt a fuller collection of probing IPs.  It is also possible that this
list contains false positives; it is simply a list of all IPs (other than
those of our test systems) geolocated to China that were observed connecting
to our unpublished bridges during the testing cycle; random scanning
activity could have snuck in, but the majority are due to active probes.

Each probing "session" generally consisted of multiple probes from multiple
different IPs within China.  p0f (passive OS fingerprinting) data collected
during the probes indicated that they generally originated from Linux
systems, but there was some conflicting data in the source ports used by
some probes, which appeared in ephemeral port ranges more commonly seen on
Windows systems.  This data is not conclusive in any way due to the
uncertain nature of p0f and the ability to reconfigure source port ranges at
will in modern operating systems.

Garbage binary probes
---------------------

Any connection from China to TCP/443 of a host in the US that exchanged an
SSL handshake, no matter what the certificates looked like (self-signed,
CA-signed, etc) provoked what we have termed the "garbage binary" probes
("garbage probes" for short) against the US host.  The only exception to
this seemed to be "old" hosts, which is to say, long-established hosts that
have had HTTPS services running for an extended period of time.  Connections
to those hosts did not appear to provoke any type of scanning activity.

The garbage probes occurred within moments of the SSL negotiation.  In some
cases, the data sent by these probes was identical, even from different
source hosts and at different times.

Notably, garbage probes did not occur in the case of TCP/443 connections
that spoke simple plaintext or non-SSL binary protocols, such as obfs2. 
This indicates that DPI was taking place, likely at or near line-rate, to
select which hosts should be probed.

Once a host was probed, subsequent connections with the same destination
outside of China did not provoke probes for approximately the next 10
minutes (measured from the final packet in a sequence of probes).  After 10
minutes, a new SSL negotiation between the hosts would trigger a new round
of probing.  The "ignored" interval remained consistent through many rounds
of testing.

We have not attempted to exhaustively collect and compare the data sent in
these probes, but what analysis we have performed does not provide any
insight into their purpose or intent.  It is possible that these probes are
designed to provoke a response from a different category of server that
operates on TCP/443 (and possibly other ports) and that the Chinese
government is intending to block, but that is only speculation at this time. 
It is not clear if further analysis would be worthwhile at this time given
that these probes do not, at the surface, appear to be directed specifically
at Tor.  The lack of currently active blocking resulting from any probes
also makes it difficult to evaluate the effectiveness of any changes that
might result from such analysis.

Full SSL/Tor probes
-------------------

Our testing demonstrated that the SSL client hello sent by a Tor client in
China to a US-based unpublished bridge will consistently trigger probing of
the bridge by Chinese hosts, consisting of fully-established TCP, SSL, and
Tor-protocol connections.  Analysis of debug logs from bridges in this
testing showed behavior consistent with a Tor client configured with
multiple bridge statements.  It is also possible, but impossible to
determine definitively at this time, that the probers were using custom
software either written from scratch or adapted from the Tor sources to
perform their probing.

Specifically, the debug logs and pcaps showed that the prober connected,
performed an SSL handshake, performed an SSL renegotiation, and then spoke
the Tor protocol.  When speaking the Tor protocol, the probers we captured
built a one-hop circuit and sent a BEGIN_DIR cell, as would be expected from
a proper Tor client.  When the bridge replies to the BEGIN_DIR cell with its
descriptor, the prober hangs up the connection.

This probing consistently took place within a range of +3 minutes from a 15
minute past the hour interval (HH:00, HH:15, HH:30, HH:45) after a Tor
client SSL client hello was sent from China to the US host.  Multiple
connections were made in each round of probing.  The probing occurred
consistently regardless of the TCP source and destination ports - if a Tor
SSL client hello packet was sent, on the next 15 minute interval a full
Tor-speaking probe could be expected, like clockwork.

Conversely, when the Tor SSL client hello was modified by removing a single
cipher from the cipher spec, full Tor connections from China to a US
unpublished bridge could be established multiple times, and for extended
durations, without provoking any probing.  Removing the less-common
ServerName SSL extension from the client hello, however, did not change the
probing behavior, indicating that the probe is likely tied specifically to
the cipher specs, not to the entire hello packet.  Unfortunately we were not
able to verify if sending a raw set of cipher specs only would trigger the
probing; this test was not performed before time on the analysis ran out.

The SSL cipher list probe trigger is further discussed in Tor trac ticket
#4744.

Server-side SSL cert impacts
----------------------------

Before isolating the SSL client hello as the provoker of probing, we
performed a number of tests using a simple SSL server on the US side and
openssl s_client, wget, and Firefox on the China side to attempt to locate
server SSL certificate combinations that provoked probing.  As discussed in
the garbage probing section, it appears that any SSL negotiation, even with
a fully legitimate CA-signed cert (and having used the correct hostname in
the cert to access the host, etc), will trigger the garbage probing.  On
non-standard ports, though, none of these clients provoked any type of
probing, garbage or real SSL, no matter what the server certificate looked
like.  As such it does not appear that the GFW is, today, keying on the
server certificate presented to make probing decisions.

Traceroute testing
------------------

Mike Perry posited that the prober IPs could be legitimate IPs within China
that are hijacked (partially or entirely by the GFW to perform its probes,
and suggested some traceroute testing to attempt to confirm or deny this
hypothesis.  After several false starts, we were able to successfully source
TCP traceroute packets to a prober IP using the exact same
sourceIP:sourcePort:destIP:destPort tuple that was being used for the probe,
during the probe and after the probe.  These results did not show a
different route during the probe and after the probe as Mike expected if
such hijacking were taking place, so it does not appear likely, though it
has not been conclusively disproven.

Additional Data
---------------

A number of other files are available in the github gist that contains this
report, including:

* testcases.txt - a list of planned test cases for the project, plotted out
  both before and during the project.  Not all cases were tested as some
  were deemed irrelevant when actual blocking could not be replicated, and
  due to other results found during testing.

* results.txt - a log of results obtained from working within the test cases
  described above.  Somewhat terse and difficult to interpret for anyone
  other than the author, but included for completeness. 
  Interpretation/annotation available upon request to the author.

* general-notes.txt - a series of general notes and a set of tentative
  conclusions formed on the final two days of testing.  These notes have
  largely been distilled, cleaned up, and expanded upon within this report.

* traceroute.txt - traceroute output from the traceroute testing phase
  described in a previous section.

* cn.log - hotpot.py output of some "garbage probes" collected early on in
  the process.

* client-hello.raw - the raw bytes representing a Tor client's SSL client
  hello sent via netcat to provoke full Tor/SSL probing.

Additionally, trusted researchers / Tor developers are welcome to contact
the author for access to the pcaps and debug logs collected during this
testing.  Due to senstive contents (IPs of probing systems, etc), we cannot
release these publicly at this time.

Acknowledgements
----------------

Thanks to Team Cymru, Inc. for providing approximately a week of the
author's time dedicated to this analysis, as well as the funding for the
Amazon EC2 bridges and facilitating the access within China.  Thanks to
George Kadianakis (asn) for his analysis assistance and help providing
direction for the testing, as well as the handy hotpot tool.  Thanks also to
too many Tor developers and other contributors in #tor-dev to count or list,
but I'll try: nickm, armadev, rransom, Sebastian, and mikeperry.  Apologies
to those I missed, it's not because you were unhelpful, I promise you!

Revision History
----------------
06 JAN 2012 - Initial draft published

07 JAN 2012 - Minor revisions and additions per feedback from George
Kadianakis.

client-hello.raw

cn.log

2011-12-05 18:25:46.223838 | ''

2011-12-05 18:26:53.720946 | '\x80w\x01\x03\x01\x00N\x00\x00\x00 \x00\x009\x00\x008\x00\x005\x00\x00\x16\x00\x00\x13\x00\x00\n\x07\x00\xc0\x00\x003\x00\x002\x00\x00/\x03\x00\x80\x00\x00\x05\x00\x00\x04\x01\x00\x80\x00\x00\x15\x00\x00\x12\x00\x00\t\x06\x00@\x00\x00\x14\x00\x00\x11\x00\x00\x08\x00\x00\x06\x04\x00\x80\x00\x00\x03\x02\x00\x80\x00\x00\xff\x9dd\x83\xbfP\xdb/L\xd6\x88>\xfa\xbbs\x88\xbd\x0cN\xfd\x0f\x84\x8c\xad\x04x61\x99\x82\x01\x19|'

2011-12-05 18:26:57.973470 : 221.205.139.111:52880
2011-12-05 18:26:57.973497 | '\xaa\x14f\x82\x97\xfb\xfc\xab\xf0\xed\xafY\x9e\xd62?[\xf8JU\xf5X\xc1\xfa\x1bZZ\x9a\xe6\xd0FBV\xfb\x9fbG#\xbb\xc2?\x8b\xa2\xe1ZY\xa6h\xee0 \xe9\xea\xd1\xfb\xb9\xe5s\x0c\xb0\xc4\x12BB\xbe%\x93\xb0_\xaf\xca\x97\xd8]\x90\x89\x1b\r\xe8\x11^<\x0bt\x10\xd6\xd8\x9a\xe3\xb0w1\x14\xa0\xb8^MK\x0e\xabz\xd7\xc2\xd1\xb3\xd1Z\xcd\xdd\xc1^<}\xe7/\x8c\xbe\x86\xa6!6\x1d\xd0I=\x08&\x89\xd23\xb4\xcb\x89\xf5\x9c=\xc6\xf6\x89\xa47f_\xb3N\x8d@\x8b\x92e+\xc7\x01\xfa\x8f=\x82\xb5FTg\xf9\x9e\xf0\xee:\xac\xb4\xaf\xb4\xd7\xe6\x1b6\x99hBX\xf2\xd3\xbc\x1d\x1a\xbd\x96)y\x97\xdd\xbej\xc37\x083%B^Yp\x13\xafV\xac\xe5o\x14\'\xc6\x85z\x03!\x93\xbf6\xbb8\xcd\x18v7\xdb\xac\xbe\x8dQ\xff\xeb)\xee\xfd\xd7E\xa9\xbc\xb3<c\xf8@\xdc\xfa\xdfo9\x94+\xf0a\xc1\xe5\x18\x9c\x91\xd5*aT\x94\x89B\x11a\x069\x9c8t\xfe13Z\xaa\x91\xc9c%sS\x064\xb7\x1dPHq\xf8\xa9\xc4\x8c\xb1\x85\x1c\x12\x8bT.\xc2G,ry\x86\x1d\n\xce\xfe\xad\xc0\xd0\xb2s\x87N\xc2N\xbe:\xf6\x02E\xa8\x87`9\x91\xb3fS\xf9\x12E\xf1\x17az\xe4\xde($\xaeY\x965\xa7\xd8\x82\xe4\x12\xf8\xe6\xd6 \xec6X}i>O\xe1O\x93\xd3\xe4s\xcc\xc8Q\xf3\xeb\xfe\xcc\x01\xb2s\xd8\xb4W\xea\xac\xbc@K\xa8\xf4"\xa4]_\xf3\xbe-\x06\x11\x11y\xdcXIP\xc2\xc7\x1c\xc3y\x0e\x1b\xacd\x05X\xa0\xc3\xa2H\xb8Dl\x94\xa2\xdeRO\xe3\xe1\xde\xdb\xbe\xb6%\x8d\xf7k'

2011-12-05 18:26:58.839414 : 218.10.51.83:1629
2011-12-05 18:26:58.839469 | '\xd7s\x10\x05\x95\x86\xe6\x89ok\x07\xd2\x95\xab}\x16\x10\xc4\xc8F\xac|ax\x86d\xc6\x03\x19_=\xd1\x13W\x14\x8dG\x0e;Y.\'\xd2\x90\xa3\xab X\xb1w\x97\xe4e\xd8A=\xc7\xd0\xe4S\xc46\xcb\x86\xea9\x9bIz\xe0\xa8\xe7\xb7\x1b\x8c\xa8H\x1aK\xfdl\t\xdf\xa0\xb4\xd3\xcd\xe8r~\xd5\\u\xfa\xdb\xab\xad\xf5s\'U\x9a\x0e\x0c4\x993|3\xfc\xf8\x1e\x85\xd8\xbd9\xab\n!\x9d\x87vx\xfbp\xd2&\x9c\xc7\x18\xc2\x1c\xb1O\xa6eg\xd8`\x99UX\xb6\xd90s\x12[|\xb2\xf7\x83(\xee~\x17\xc0$\xb2\x87;\xf3"lB\xc7\xd0\xa8\x1f\xaf\xc1s\x07wL\xb6j\xdd\x11e\x8f\x87\xe76u\xe5\xcb5\t};C\xef\\\xae1\xa2\xfdX\xc0\xac\x19\xb33\x90\xfe\xe8y[\xf8]\xe9\xfeE\x9es\xa9i\'\xb1easT<\xa1\x84]\x9e\xdc\x9c\xcatO\xfc\x04\xcdd\xfb(\xdbY\x91\xd9\x1d/L\xc5\x98\xf1\xf5|Rh\xd0\r\t\xd3i\'/\x84p\xa3\xd3l&\xa0\xcf"G*\xf9\xd7\x03\x16\x07\xcdZ\x1e'

2011-12-05 18:26:59.704311 : 112.94.175.239:35960
2011-12-05 18:26:59.704387 | '\xcb\x11E.\xf7I\x91Y\x92{\x00\x0f\xd0&\xc3\x1c\xe1Tq\xe3C\xf9\xd9\xfd\xeb}\x8c\xd2\xfc\xa5\xf9K\x9f\x8c*\x86\xdb\x1f\xe3\x19v\xd951\xb6Ep\xa4\xde\x1cJ\xd1S\x95\',\xa9\xdc\xba\x03y\xc0\xfc\xe7\x9b\x12\x1f\xee\x18k\xa5\xdb75H\xe8\x84\x97+2\xbb\xd1*\xaa\xef\xb1\xb3\t7\xaf4{\xa2\xbf\x05\xac>#\x9aU\x8e\xbe0D\xf3\xf7\xab\xf6\x0e\xd5\xa7H\xa6Pq\x95\x01$\x1d\xb6\xd3P1\xf4\x8e5 \xcbX: ew\xcf\xa8jFS\xdf\xd2\xa7\x86\x1bM\xd5\x0baU/}\x0c\x81L\xbb\xf4\xdap\x14\xa5G\xccE\xabD\x14\xd2-\xd8\xa4\r\xaaK\x12\xc4\x17gOw;}s\xc5\xfd\xbe\x81\xf1\x18p\x84=\xb6Q\x81\xe0\x14\x142@\xeb\xd5\xcb\x95\x9f]\xd95\xc3(+}$\x9dC\xa0\xdaC\x11r\xb2\x94\xae\xe7d/Gw\xc1x7\xac\xcc\x02\xc0k^\x99\x1f\xa0@I\x1ece`\x03?"\x130S\'\xdd:\x8a\x8b\x80\x81Lw\xb7wC88-\x96\xd0K6\x11\x93\xd2\xf2w\xb1\xf55\xd2\x87e%\xadB\xdd8M\xdc8\x98S\xee\x10\x95&G\xc1;\x96\x8bq&\x9d\xc2\x19\x14t\x8dIF\x14\xad\xea\xc1nGx\xba$\xaf\xd2v\x1d\xe1\x8bB\xa7L}\xbcWm\xe2\xf3\xaez\x87"\x86\xcf\xe7\x99\xfbQ\xd9\xe8\x97\xd0\xa3\xba\x7fu\xb0\x9b\xd5;]\xfb\x06Y\xb7\\E\x19\xcf\xf2\x12V\x94\x97\xa4{\xb0\x9f\xcb\t\x87b\xd9\xa9\x9c\xd7\x9dL'

2011-12-05 18:27:00.785398 : 113.58.246.59:19316
2011-12-05 18:27:00.785443 | '\xcb\x11E.\xf7I\x91Y\x92{\x00\x0f\xd0&\xc3\x1c\xe1Tq\xe3C\xf9\xd9\xfd\xeb}\x8c\xd2\xfc\xa5\xf9K\x9f\x8c*\x86\xdb\x1f\xe3\x19v\xd951\xb6Ep\xa4\xde\x1cJ\xd1S\x95\',\xa9\xdc\xba\x03y\xc0\xfc\xe7\x9b\x12\x1f\xee\x18k\xa5\xdb75H\xe8\x84\x97+2\xbb\xd1*\xaa\xef\xb1\xb3\t7\xaf4{\xa2\xbf\x05\xac>#\x9aU\x8e\xbe0D\xf3\xf7\xab\xf6\x0e\xd5\xa7H\xa6Pq\x95\x01$\x1d\xb6\xd3P1\xf4\x8e5 \xcbX: ew\xcf\xa8jFS\xdf\xd2\xa7\x86\x1bM\xd5\x0baU/}\x0c\x81L\xbb\xf4\xdap\x14\xa5G\xccE\xabD\x14\xd2-\xd8\xa4\r\xaaK\x12\xc4\x17gOw;}s\xc5\xfd\xbe\x81\xf1\x18p\x84=\xb6Q\x81\xe0\x14\x142@\xeb\xd5\xcb\x95\x9f]\xd95\xc3(+}$\x9dC\xa0\xdaC\x11r\xb2\x94\xae\xe7d/Gw\xc1x7\xac\xcc\x02\xc0k^\x99\x1f\xa0@I\x1ece`\x03?"\x130S\'\xdd:\x8a\x8b\x80\x81Lw\xb7wC88-\x96\xd0K6\x11\x93\xd2\xf2w\xb1\xf55\xd2\x87e%\xadB\xdd8M\xdc8\x98S\xee\x10\x95&G\xc1;\x96\x8bq&\x9d\xc2\x19\x14t\x8dIF\x14\xad\xea\xc1nGx\xba$\xaf\xd2v\x1d\xe1\x8bB\xa7L}\xbcWm\xe2\xf3\xaez\x87"\x86\xcf\xe7\x99\xfbQ\xd9\xe8\x97\xd0\xa3\xba\x7fu\xb0\x9b\xd5;]\xfb\x06Y\xb7\\E\x19\xcf\xf2\x12V\x94\x97\xa4{\xb0\x9f\xcb\t\x87b\xd9\xa9\x9c\xd7\x9dL'

2011-12-05 18:30:43.346920 | '\x80w\x01\x03\x01\x00N\x00\x00\x00 \x00\x009\x00\x008\x00\x005\x00\x00\x16\x00\x00\x13\x00\x00\n\x07\x00\xc0\x00\x003\x00\x002\x00\x00/\x03\x00\x80\x00\x00\x05\x00\x00\x04\x01\x00\x80\x00\x00\x15\x00\x00\x12\x00\x00\t\x06\x00@\x00\x00\x14\x00\x00\x11\x00\x00\x08\x00\x00\x06\x04\x00\x80\x00\x00\x03\x02\x00\x80\x00\x00\xff\xcd]\x06\xb5S\xdb\x87\x8e\x06\x1a\x14\xfd\xf0\xd5;\x81\xf6(\xeaB\xb0)(&\x9d\xaf\xcc\xc0g2\x1a\xf4'
2011-12-05 18:30:46.450253 < 'foo\n'
2011-12-05 18:30:46.450253 > '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
2011-12-05 18:30:56.230631 < 'bar\n'
2011-12-05 18:30:56.230631 > '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
2011-12-05 18:30:57.130697 < 'baz\n'
2011-12-05 18:30:57.130697 > '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

general-notes.txt

Tentative Conclusions 08 DEC 2011
---------------------------------
Based on the data so far, it appears that all TCP/443 connections out of 
China to previously-uncategorized hosts are subject to DPI, and, if an SSL
handshake of any type is detected, subject to active probing.  TCP/443
connections that do not speak obvious SSL (ie obfs2 protocol) do not appear
to be probed.  TCP/443 connections to "old" hosts (definition of "old" is
very much unknown at this time) do not appear to cause probing.

SSL connections to other TCP ports, including "SSL-expected" ports like
TCP/993 and TCP/995, do not appear to initiate probing, no matter how
provacative the certificates in use.  This is a change from previous
behavior; during investigation of #4185 in October 2011, Tor and other SSL
connections on non-TCP/443 ports were definitely probed (and blocked).  It
appears that the GFW's SSL probing has been significantly changed since that
time.  Further investigation is still ongoing.

Update 1640 UTC: A non-obfs2 Tor connection to a random high port
(TCP/18457) on a virgin non-published bridge IP resulted in an SSL-connected
scan that appeared to act as a Tor client (not yet confirmed from debug
logs) several minutes after the initial connection was established (and
reconnected several times).  More than 10 minutes after the probing, the
bridge IP:PORT had not yet been blocked.

Update 2100 UTC: The full SSL-speaking (and tor-speaking) probes appear to
happen consistently at roughly 15 minute-past-the-hour intervals (:00, :15,
:30, :45).  This is consistent to a range of +3 minutes through all of the
samples we've collected so far (with only one outlier in the +3 minute
range, all others within 1 minute of the 15 minute mark).

Thus far we have been unable to provoke this type of probing using anything
other than a full Tor connection.  SSL connections to non-Tor servers, even
those using Tor-like SSL certificates, have not been able to trigger it.

The Tor connection does not need to be persistent to trigger the probing; a
short connection just long enough to build a circuit, made only 2-3 minutes
before a "scheduled" scan time, was enough to provoke a scan at the next
interval.  A similar duration circuit right after a previous "scheduled"
scan time also provoked a scan at the next interval.

An openssl s_client connection to an actual Tor server did NOT provoke a
scan of this nature.

TCP traceroute to an SSL/Tor-speaking prober during and after a probe does
not yield any change.  See traceroute.txt for a sample.  Still pending:
using the same sourceport with a handy iptables trick thanks to mikeperry.

09 DEC 2011
-----------
A connection to Tor 0.2.3.8 from a Tor 0.2.3.8 client with the ServerName
SSL extension removed still provoked probing.

A connection to Tor 0.2.3.8 from a Tor 0.2.3.8 client with a reduced SSL
cipher list did NOT provoke probing.  Further testing is in progress.

The iptables trick did not work for traceroute with an identical
sourceip:sourceport:destip:destport tuple; my suspicion is that conntrack
gets in the way of the rewriting.  Additional data added to traceroute.txt
(see data for 60.13.132.232).

Subsequent tweaking of the iptables trick returned the traceroute data for
113.58.244.209, which showed no difference between "during probe" and "after
probe" in the short term.

Observations
------------
* Probing will re-occur with new connection attempt to a previously probed
  (but not blocked) system if repeated after more than 10 minutes.  New 
  connections less than 10 minutes after a previous probe do not generate
  new probes, even if the SSL certs exchanged are changed.
  * The 10 minute timer appears to start from the time of the final probe in
    a sequence, not the first probe.
* Probing occurs within moments of an SSL exchange that contains a
  "probable" cert / cert chain within the constraints of the time limit
  notes above.
* Several probes are initiated in each instance of probing, from multiple
  hosts.

Plans
-----
* Use cronned */5 TCP ping to each port from inside CN to verify
  reachability

results.txt

2) obfs2 transport (https://gitweb.torproject.org/obfsproxy.git)
----------------------------------------------------------------
Initial instance of obfs2 on TCP/443 generated one set of probes that
did not conform to other probe methodologies.  No repeats occurred with
over two hours of operation and disconnects/reconnects.

Subsequent use of obfs2 on a new TCP port did not provoke any probing
with over an hour of operation and disconnects/reconnects as well as
(failed, but intended to fail) openssl s_client and wget connection
attempts.

A longer duration obfs2 test back on TCP/443 on a fresh clean IP did not
reveal any probing in over 24 hours of execution.

3) Tor-like-SSL (control)
-------------------------
TCP/443:
* hotpot.py: triggers probing, does not cause blockage.
* server.pl: triggers probing, does not cause blockage.
* unpublished tor 0.2.3.8 bridge: openssl s_client triggers probing, no block
* unpublished tor 0.2.3.8 bridge: tor 0.2.3.4-alpha client triggers
  probing, no block
* unpublished tor 0.2.3.5 bridge: tor 0.2.3.4-alpha client triggers
  probing, no block
* unpublished tor 0.2.3.4 bridge: tor 0.2.3.4-alpha client triggers
  probing, no block
* apache-wget: triggers probing, does not cause blockage.

TCP/14392:
* server.pl: no probing

TCP/18457:
* unpublished tor 0.2.3.8 bridge: openssl s_client does not trigger probing
* unpublished tor 0.2.3.8 bridge: tor 0.2.3.4-alpha client triggers real SSL
  probing, with several minute delay (probing occurred at ~30 minutes past
  the hour)
* The above two tests were performed in close proximity and can't be nailed
  down; further testing pending.  Bridge IP for this test was 50.17.159.162.

TCP/16415
* unpublished tor 0.2.3.8 bridge: tor 0.2.3.4-alpha client triggers real SSL
  probing 10 minutes after Tor connection established (probing occurred at
  ~45 minutes past the hour)
  * results replicated again, this time with scanning at :00 past the hour.

4) Self-signed cert
-------------------
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            9f:46:4c:ab:e1:db:41:bb
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Minnesota, L=Duluth, O=Joe Bloggs, LTD, CN=iscream.4dq.com/emailAddress=root@null.com
        Validity
            Not Before: Dec  7 16:22:02 2011 GMT
            Not After : Dec  6 16:22:02 2012 GMT
        Subject: C=US, ST=Minnesota, L=Duluth, O=Joe Bloggs, LTD, CN=iscream.4dq.com/emailAddress=root@null.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

* apache-wget TCP/443: triggers probing

5) Real CA-issued cert
----------------------
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0f:5b:4e:f2:f3:68:af:a5:83:a3:37:e1:15:7c:c8:b2
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=PositiveSSL CA
        Validity
            Not Before: Dec  7 00:00:00 2011 GMT
            Not After : Dec  6 23:59:59 2012 GMT
        Subject: OU=Domain Control Validated, OU=PositiveSSL, CN=www.joebloggsltd.net
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

* apache-wget TCP/443: triggers probing
* apache-firefox TCP/443: triggers probing

* apache-firefox to www.team-cymru.org: does not appear to trigger probing

* apache-wget TCP/14952: no probing
* apache-firefox TCP/14952: no probing

7) Firewall everybody but the intended user of the bridge.
----------------------------------------------------------
Clients in China provided access to a bridge that permitted access only to a
set of explicitly whitelisted IPs were able to access the bridge for >48
hours without blockage.

8) Other protocols for reference (ssh, ftp, smtp, smtps)
--------------------------------------------------------
* POP3S CA-signed cert: no probing
* POP3S self-signed cert: no probing
* POP3S tor-like cert: no probing

9) TCP connection/scan with no SSL handshake
--------------------------------------------
hping -S -p 443 <test IP> and telnet <test IP> 443 both failed to generate
scanning to a server which did generate scanning when connected via openssl
s_client.

13) Relay with debug verbosity after SSL-based probing duplicated
-----------------------------------------------------------------
Upon analysis of debug logs from a 0.2.3.8-alpha bridge, SSL-based probes
from CN appear to be speaking the Tor protocol, similar to the behavior
of a client with multiple bridges configured that is not actively using
our bridge.

testcases.txt

1) Handshakes using OpenSSL and libnss
2) obfs2 transport (https://gitweb.torproject.org/obfsproxy.git)
3) Tor-like-SSL (control)
4) Self-signed cert
5) Real CA-issued cert
6) Vary responses of bridge to probing as well as initial handshake (independently if possible)
7) Firewall everybody but the intended user of the bridge.
8) Other protocols for reference (ssh, ftp, smtp, smtps)
9) TCP connection/scan with no SSL handshake
10) Connection into China-hosted bridge from outside of China, which side (if any) gets probed?
11) Handshakes using real Apache on the server side
12) Tor 0.2.1.x stable
13) Relay with debug verbosity after SSL-based probing duplicated
14) Determine how many negative probes it takes for the GFW to say "this is
    not worth probing anymore" (may prove difficult)

traceroute.txt

Sample traceroute to a prober during a probe:

 traceroute -T -O ack --sport=993 -n -p 6885 113.58.244.51
traceroute to 113.58.244.51 (113.58.244.51), 30 hops max, 40 byte packets
 1  10.208.200.3  0.360 ms  0.314 ms  0.237 ms
 2  10.1.18.13  0.447 ms  0.382 ms  0.389 ms
 3  10.1.19.14  0.460 ms  0.369 ms  0.344 ms
 4  216.182.232.50  0.686 ms  0.674 ms  0.439 ms
 5  72.21.222.148  0.830 ms  0.931 ms  0.742 ms
 6  72.21.220.52  1.152 ms  1.121 ms  1.191 ms
 7  168.143.232.49  1.050 ms  0.921 ms  0.996 ms
 8  129.250.3.118  2.783 ms  2.280 ms  2.733 ms
 9  192.205.37.53  3.535 ms  3.189 ms  3.045 ms
10  12.122.134.134  76.622 ms  75.564 ms  76.007 ms
11  12.122.18.21  73.662 ms  71.409 ms  71.971 ms
12  12.122.31.85  71.701 ms  72.252 ms  71.765 ms
13  12.122.30.25  77.486 ms  75.511 ms  75.763 ms
14  12.122.30.30  70.226 ms  72.362 ms  71.562 ms
15  12.123.30.249  70.481 ms  70.040 ms  70.323 ms
16  12.122.129.145  73.380 ms  73.415 ms  73.626 ms
17  12.116.103.6  289.428 ms  289.163 ms  289.259 ms
18  219.158.4.177  280.133 ms  280.081 ms  279.180 ms
19  219.158.15.10  285.558 ms  277.110 ms  277.112 ms
20  219.158.23.242  295.061 ms  295.484 ms  295.082 ms
21  221.11.155.250  295.448 ms  295.591 ms  295.428 ms
22  221.11.160.14  298.747 ms  301.180 ms  301.139 ms
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  *^C

After the probe:

 traceroute -T -O ack --sport=993 -n -p 6885 113.58.244.51
traceroute to 113.58.244.51 (113.58.244.51), 30 hops max, 40 byte packets
 1  10.208.200.3  0.321 ms  0.230 ms  0.268 ms
 2  10.1.18.13  17.777 ms  1.088 ms  0.384 ms
 3  10.1.19.14  0.644 ms  0.541 ms  0.613 ms
 4  216.182.232.50  0.743 ms  0.685 ms  0.745 ms
 5  72.21.222.148  0.953 ms  0.975 ms  0.694 ms
 6  72.21.220.52  1.193 ms  0.999 ms  1.186 ms
 7  168.143.232.49  0.844 ms  1.032 ms  0.947 ms
 8  129.250.3.118  2.337 ms  2.236 ms  2.241 ms
 9  192.205.37.53  3.386 ms  3.232 ms  3.287 ms
10  12.122.134.134  75.566 ms  75.584 ms  75.913 ms
11  12.122.18.21  72.999 ms  71.614 ms  72.121 ms
12  12.122.31.85  72.872 ms  72.201 ms  72.168 ms
13  12.122.30.25  74.797 ms  76.056 ms  75.605 ms
14  12.122.30.30  73.746 ms  72.219 ms  71.538 ms
15  12.123.30.249  70.312 ms  70.102 ms  70.144 ms
16  12.122.129.145  71.867 ms  72.608 ms  72.667 ms
17  12.116.103.6  289.208 ms  289.009 ms  289.292 ms
18  219.158.4.177  280.327 ms  280.069 ms  280.181 ms
19  219.158.15.10  277.672 ms  277.070 ms  277.369 ms
20  219.158.23.242  295.198 ms  295.630 ms  295.114 ms
21  221.11.155.250  296.065 ms  295.571 ms  295.917 ms
22  221.11.160.14  301.651 ms  301.019 ms  301.074 ms
23  * 221.11.160.14  299.997 ms  301.065 ms
24  * * *
25  * * *^C

Sample using mikeperry's iptables hack to use the same exact TCP tuple as
the live connection, while it was live it failed; pcaps showed that the
packets weren't going out at all, and appeared to be eaten at the iptables
layer.

 traceroute -T -O ack --sport=1234 -n -p 15713 60.13.132.232
traceroute to 60.13.132.232 (60.13.132.232), 30 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  *^C

But the same ports to a different IP were okay:

 traceroute -T -O ack --sport=1234 -n -p 15713 1.2.3.4
traceroute to 1.2.3.4 (1.2.3.4), 30 hops max, 40 byte packets
 1  10.208.200.3  0.432 ms  0.355 ms  1.001 ms
 2  10.1.16.13  0.593 ms  0.576 ms  0.339 ms
 3  10.1.17.14  0.349 ms  0.425 ms  0.436 ms
 4  216.182.224.82  0.645 ms  0.479 ms  0.436 ms
 5  72.21.222.148  0.691 ms  0.680 ms  0.733 ms
 6  72.21.220.36  40.609 ms  1.297 ms  1.244 ms
 7  168.143.232.73  1.207 ms !N  1.361 ms !N  1.499 ms !N

And after the prober closed its connection:

 traceroute -T -O ack --sport=1234 -n -p 15713 60.13.132.232
traceroute to 60.13.132.232 (60.13.132.232), 30 hops max, 40 byte packets
 1  10.208.200.3  0.328 ms  0.268 ms  0.284 ms
 2  10.1.14.13  0.412 ms  0.520 ms  0.283 ms
 3  10.1.15.14  0.393 ms  0.424 ms  0.441 ms
 4  216.182.224.78  0.497 ms  0.429 ms  0.501 ms
 5  72.21.222.148  0.806 ms  0.728 ms  0.688 ms
 6  72.21.220.68  0.870 ms  0.972 ms  0.931 ms
 7  168.143.191.17  1.439 ms  1.325 ms  1.378 ms
 8  192.205.37.53  3.388 ms  3.271 ms  3.050 ms
 9  12.122.134.134  73.478 ms  71.643 ms  71.718 ms
10  12.122.18.21  76.272 ms  75.908 ms  78.250 ms
11  12.122.31.85  74.103 ms  71.976 ms  82.942 ms
12  12.122.30.25  75.551 ms  74.995 ms  76.269 ms
13  12.122.30.30  73.357 ms  75.962 ms  75.576 ms
14  12.123.30.249  73.233 ms  73.750 ms  73.583 ms
15  12.122.129.49  74.569 ms  74.515 ms  73.608 ms
16  12.118.130.86  238.895 ms  282.266 ms  252.014 ms
17  219.158.96.221  240.930 ms  240.885 ms  240.886 ms
18  219.158.96.225  242.314 ms  242.288 ms  242.288 ms
19  219.158.23.94  311.817 ms  311.760 ms  311.904 ms
20  221.7.2.250  314.890 ms  314.793 ms  314.791 ms
21  221.7.3.26  313.309 ms  313.450 ms  313.255 ms
22  * * *
23  * * *^C

Another prober, this time actually during the probing thanks to more
iptables and routing hackery:

 traceroute -T -O ack --sport=1234 -n -p 18339 113.58.244.209
traceroute to 113.58.244.209 (113.58.244.209), 30 hops max, 40 byte packets
 1  10.208.200.3  22.941 ms  4.654 ms  0.896 ms
 2  10.1.14.13  0.308 ms  0.294 ms  0.335 ms
 3  10.1.15.14  0.400 ms  0.446 ms  0.353 ms
 4  216.182.232.50  0.614 ms  0.492 ms  0.457 ms
 5  72.21.222.148  0.657 ms  0.689 ms  0.749 ms
 6  72.21.220.52  1.107 ms  1.243 ms  1.206 ms
 7  168.143.232.49  1.208 ms  1.221 ms  1.207 ms
 8  129.250.3.118  2.107 ms  2.286 ms  2.263 ms
 9  192.205.37.53  11.880 ms  3.407 ms  3.336 ms
10  12.122.134.134  72.544 ms  71.551 ms  71.749 ms
11  12.122.18.21  72.905 ms  75.897 ms  75.792 ms
12  12.122.31.85  77.837 ms  75.563 ms  76.777 ms
13  12.122.30.25  76.276 ms  75.540 ms  76.331 ms
14  12.122.30.30  73.454 ms  74.699 ms  71.855 ms
15  12.123.30.249  74.593 ms  74.357 ms  74.842 ms
16  12.122.129.145  73.903 ms  69.713 ms  72.209 ms
17  12.116.103.10  279.680 ms  279.639 ms  279.349 ms
18  219.158.3.161  266.821 ms  269.409 ms  267.077 ms
19  219.158.96.54  256.382 ms  254.179 ms  254.349 ms
20  219.158.14.106  306.252 ms  306.526 ms  306.267 ms
21  221.11.155.250  305.514 ms  305.254 ms  305.474 ms
22  221.11.160.14  308.502 ms  310.979 ms  311.144 ms
23  * * *
24  * * *
25  * * *
26  *^C

And the same prober, > 5 minutes after the probe:

& traceroute -T -O ack --sport=1234 -n -p 18339 113.58.244.209
traceroute to 113.58.244.209 (113.58.244.209), 30 hops max, 40 byte packets
 1  10.208.200.3  0.439 ms  0.278 ms  0.236 ms
 2  10.1.14.13  5.294 ms  1.225 ms  0.739 ms
 3  10.1.15.14  0.385 ms  0.430 ms  0.442 ms
 4  216.182.232.50  7.351 ms  0.657 ms  0.501 ms
 5  72.21.222.148  0.798 ms  0.719 ms  0.757 ms
 6  72.21.220.52  1.056 ms  1.101 ms  1.108 ms
 7  168.143.232.49  1.062 ms  1.192 ms  1.017 ms
 8  129.250.3.118  2.472 ms  2.231 ms  2.240 ms
 9  192.205.37.53  3.337 ms  3.206 ms  3.288 ms
10  12.122.134.134  73.007 ms  72.235 ms  71.503 ms
11  12.122.18.21  75.957 ms  76.638 ms  75.013 ms
12  12.122.31.85  74.581 ms  76.336 ms  76.061 ms
13  12.122.30.25  74.598 ms  76.213 ms  75.506 ms
14  12.122.30.30  72.519 ms  72.265 ms  71.669 ms
15  12.123.30.249  74.450 ms  74.448 ms  74.605 ms
16  12.122.129.145  69.584 ms  69.976 ms  69.622 ms
17  12.116.103.10  279.630 ms  279.473 ms  279.406 ms
18  219.158.3.161  266.466 ms  266.928 ms  267.200 ms
19  219.158.96.54  254.338 ms  254.340 ms  254.042 ms
20  219.158.14.106  306.293 ms  306.199 ms  306.074 ms
21  221.11.155.250  305.337 ms  305.372 ms  305.138 ms
22  221.11.160.14  305.191 ms  311.037 ms  312.963 ms
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *