Skip to content

Instantly share code, notes, and snippets.

@Lixivial
Created June 7, 2009 12:30
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Lixivial/125309 to your computer and use it in GitHub Desktop.
Save Lixivial/125309 to your computer and use it in GitHub Desktop.
a quick and dirty fallback downloader and load balancer
################
# INSTALLATION #
################
There should be no weird php.ini variables to set, so just move this to a server and set the appropriate permissions.
If you want to make it so a URL can be parsed as http://foo.bar/proxy/param1/param2/paramn, you can utilise a .htaccess
file to do this.
First, rename proxy.php to proxy and then put a .htaccess file in the same directory as proxy with the following information:
<Files proxy>
ForceType application/x-httpd-php
</Files>
This will force apache to handle any requests to proxy with a content-type of application/x-httpd-php, forcing apache to execute it as
a php script.
#########
# USAGE #
#########
To use this in darkplaces/nexuiz, the following is a sample example curl_urls.txt that was used as a base case:
teambubble-colours.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
robot_v2.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
nxmdls-fel-jag-lei-sui.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
nxmdl-abyss.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
nxmdl-oa_angelyss_md3_tagfixed.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
nxmdl-spy.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
zuriastrad-bot.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
zz-keyhuntfix080813.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
zz-genderfix080817.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
nexball-ballmodel.pk3 http://pavlvs.nexuizninjaz.com/servrarz/models/
spidflisk_effectinfo_* http://pavlvs.nexuizninjaz.com/servrarz/models/
* http://lixi.nexfiles.com/proxy.php/map/
This allows a user to specify finer granularity for static files, and use the proxy as a fallback. The idea
is that explicit file definitions should be guaranteed to at least not 404; though I'd like to allow proxy.cfg to be able
to load balance and fallback explicit files (per-file sets of providers). See CONFIG for where I want to take this.
In generic terms, the URL request takes this form:
http://foo.bar/proxy.php/key1/value1/key2/value2/keyn/valuen
And the four understood keys thus far are:
string "map" (defaults to null)
boolean "load_balance" (defaults to true, if it's set to anything, it's false)
int "retries" (not yet used; will default to 0)
boolean "debug" (*DO NOT USE* this for direct downloads; only use this to test the script from a direct request in the browser)
So an example might be:
http://maps.somedomain.ext/proxy.php/retries/2/load_balance/false/map/test.pk3
or
http://maps.somedomain.ext/proxy.php/retries/2/map/test.pk3/debug/true
Please note that for darkplaces "map" has to be the last parameter, because the map's filename is automatically appended to request URL before
being dispatched. Other game servers might have different requirements on this; please check the server's documentation for clarification on this.
##########
# CONFIG #
##########
By default, the script expects a config file of config.cfg to reside in the same directory as the proxy script. However, this can be configured
by means of editing the proxy.php file and editing:
define("INI_FILE", "proxy.cfg");
to be a relative or absolute path to your config file.
The config file uses a standard Windows or PHP ini structure, for example:
[section 1]
parameter1 = "value";
parameter2 = "value2";
[section 2]
parameter3 = "value3";
parameter4 = "value4";
The sample proxy.cfg has two example URLs in it, to define multiple values for one key, a [] must be used in the key name. For example:
[filename.pk3]
urls[] = "http://www.google.com";
urls[] = "http://www.microsoft.com";
urls[] = "http://www.apple.com";
[filename2.pk3]
urls[] = "http://www.yahoo.com";
urls[] = "http://www.opera.com";
[filename3.pk3]
urls[] = "http://www.fark.com";
urls[] = "http://news.google.com";
The script doesn't currently support grouping (with [filenamen.blah]), but grouping these is something I want to do, so that multiple values
can be assigned on a per-file basis (and namespacing with inheritance, too).
urls[]="http://maps.nexuizninjaz.com/"
urls[]="http://maps.nexfiles.com/"
urls[]="http://maps.nexfiles.com/spidflisk/"
urls[]="http://maps.praeclan.com/"
urls[]="http://maps.praeclan.com/spidflisk/"
urls[]="http://rm.endoftheinternet.org/~nexuiz/maps/"
<?php
/**
* proxy.php -- A simple little script to pseudo load balance/failover multiple map servers.
* by Lixivial (Jesse Pearson)
* contact: jesall@gmail.com
* or irc.quakenet.org/#prae.nexuiz
* URL parameter construct:
*
* http://foo.bar/install.path/debug/{true|false}/load_balance/{true|false}/retries/{n-maxn}/map/{filename}
*
* This will then be accessible in an associative array by the given key (load_balance, retries, map).
* param defaults are as follows:
* debug false
* load_balance true
* retries 0
* map null
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
define("INI_FILE", "proxy.cfg");
define("MAX_PARAM_LENGTH", 254);
// The number used to accommodate pings that were unable to be retrieved, or local servers,
// to push them to the bottom of the load balanced sorted list.
define("ARTIFICIAL_PING", rand(125, 200));
// You can use this variable to set a global redirection on error.
define("ERROR_REDIRECT_LOCATION", null);
// If debugging is enabled, this is the level of error reporting PHP will report.
// E_ALL ^ E_NOTICE is default PHP error reporting (all errors except notice).
define("ERROR_REPORTING_LEVEL", E_ALL ^ E_NOTICE);
/* If a user wants to use ICMP as a ping, set this to TRUE.
Note that it'll probably be less reliable to get data back from the server
as many people block these requests, but if this is set to true, it'll fallback
to using the method used when this is defined as FALSE.
Also, even though this is set to TRUE, there's no guarantee the web user will
have sufficient permissions to open a raw socket.*/
define("USE_ICMP", FALSE);
define("MAX_ICMP_BYTES", 512);
$params = get_parameters();
$config = parse_ini_file(INI_FILE);
/* If debugging is set on the URL, then allow the specified error reporting level,
otherwise silently eat all errors, so as not to offer a corrupted download
to a client. */
error_reporting(isset($params['debug']) ? ERROR_REPORTING_LEVEL : 0);
// Perform pseudo load balancing?
$load_balance = isset($params['load_balance']) ? FALSE : TRUE;
// NOT YET USED: Perform a given number of retries on a host before moving on.
$retries = isset($params['retries']) ? $params['retries'] : 1;
// Construct our list of servers, and, if loadbalancing, then order them by their response times.
$servers = get_server_list($load_balance, $config["urls"]);
// Ensure that a map variable was given and that we have access to the curl extension.
if ((isset($params['map']) && $params['map'] != "") && extension_loaded('curl')) {
// Instantiate a null curl handle.
$chandle = curl_init();
if (!isset($params['debug'])) {
$count = 0;
/* Since this is already sorted by lowest ping, we'll take them 1-by-1
and take the first to give a valid response.
NOTE: This has to be done as close to the actual redirect such that we get the most accurate
representation of the actual server response (which is more important than pinging). */
foreach($servers as $key => $value) {
$temp_url = $key.$params['map'];
$count++;
if (get_status_code($chandle, $temp_url) == 200) {
header("Location: ".$temp_url);
break;
} else if ($count + 1 >= count($servers)) {
die_404();
}
}
} else {
if (PHP_SAPI == "cli") {
echo " Map | Server | Ping | Status |\r\n";
foreach($servers as $key => $value) {
$temp_url = $key.$params['map'];
echo $params['map']." | ".$key." | ".$value." | ".get_status_code($chandle, $temp_url)."\r\n";
}
} else {
echo "<table><tr><td>Map</td><td>Server</td><td>Ping</td><td>Status</td>";
foreach($servers as $key => $value) {
$temp_url = $key.$params['map'];
echo "<tr><td>".$params['map']."</td><td>".$key."</td><td>".$value."</td><td>".get_status_code($chandle, $temp_url)."</td>";
}
echo "</table>";
}
}
// Close the curl handle.
curl_close($chandle);
} else {
die_404();
}
exit;
/**
* Throw a 404 response so that any cURL or any file downloaders will get a standard error response.
* rather than a blank page or a page with a message, which will result in a 200.
*
* It can take an optional parameter to override the default static variable
* ERROR_REDIRECT_LOCATION, in case there are instances where we want to redirect
* requests for some reason.
*
* @param string $redirect
*/
function die_404($redirect=null) {
if ($redirect != null) {
header("Location: ".$redirect);
} else if (ERROR_REDIRECT_LOCATION != null) {
header("Location: ".ERROR_REDIRECT_LOCATION);
} else {
header("HTTP/1.1 404 Not Found", true, 404);
header("Status: 404 Not Found", true, 404);
}
}
/**
* Tokenize the parameters in the form of /param1/paramn/
*
* Parameters take this form, rather than param1=test&paramn=testn to accommodate
* crawler-safe URLs.
*
* The URL should take the form of http://foo.bar/key1/value1/key2/value2/keyn/valuen
* It makes the URL longer than, say, http://foo.bar/value1/value2/value3, but offers
* reliable access to values in other areas, without worrying about the order in which
* they were typed in the request.
*
* @return associative array $return_params
* Array of parameters. Note that this can be empty.
*/
function get_parameters() {
if(isset($_SERVER['PATH_INFO'])) {
$return_params = array();
// Split the URL on /
$params = explode('/', $_SERVER['PATH_INFO']);
array_shift($params);
reset($params);
// loop every other one (every even will be a key, every odd will be a value)
for ($i=0; $i<count($params); $i=$i+2) {
$key = $params[$i];
$value = $params[$i + 1];
// if it has a value, place it in the associative array with its key.
// Also prevent sending overly long value or keys.
if ((isset($value) && strlen($value) < MAX_PARAM_LENGTH) && strlen($key) < MAX_PARAM_LENGTH) {
$return_params[ $key ] = $value;
}
}
return $return_params;
} else {
die_404();
}
}
/**
* Construct an associative array of sites and their pings.
*
* @param boolean $load_balance
* Whether or not to attempt rudimentary load balancing (via ping).
* @param array $server_list
* A list of servers, in the form of http://www.google.com or www.google.com.
* @return associative array $hash_table
* A sorted associative array whose value of ping is keyed by the server name.
*/
function get_server_list($load_balance, $server_list) {
$hash_table = array();
// Go over each of the server_list URLs in order to build an key/value pair out of them.
if (count($server_list) > 0) {
foreach($server_list as $server_url) {
if ($load_balance) {
/* Check to see if it's a local server. If it is, we're going to put it at the bottom of the list.
This is a stupid stupid stupid check, it should become more intelligent through some constructive
and clever querying. */
if (strpos($server_url,$_SERVER['SERVER_NAME']) === false) {
$ping = ping(split_url($server_url));
# Set the ping, if it's found, else place it at the end of the list.
if ($ping != null) {
$hash_table[ $server_url ] = $ping;
} else {
$hash_table[ $server_url ] = ARTIFICIAL_PING;
}
} else {
$hash_table[ $server_url ] = ARTIFICIAL_PING;
}
} else {
$hash_table[ $server_url ] = ARTIFICIAL_PING;
}
}
if ($load_balance) {
uasort($hash_table, "comparator");
}
return $hash_table;
} else {
die_404();
}
}
/**
* Checks a site's HTTP status.
*
* @param resource $handle
* A null or already init'd curl handle.
* @param string $url
* The URL to check, in the form of http://foo.bar.com/path/to/file.ext
* @return int
* The status as returned from the target web server (404, 503, 200, etc)
*/
function get_status_code($handle, $url) {
// Ensure that we can actually perform the action...
if (!ini_get('safe_mode') &&
(strpos(ini_get('disable_functions'), 'curl_exec') == 0) &&
(strpos(ini_get('disable_functions'), 'curl_get_info') == 0) &&
(strpos(ini_get('disable_functions'), 'curl_setopt_array') == 0)) {
/* Reset the URL, and ensure that the header and results are returned.
Also, limit our byte download to just 1 byte, so that we can get our status code without
having to download the file ourselves. */
curl_setopt_array($handle, array(CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER=>TRUE,
CURLOPT_HEADER=>TRUE,
CURLOPT_RANGE=>"0-1"));
// Perform the connection in order to determine the server's response.
$response = curl_exec($handle);
// Return the status code.
$status = curl_getinfo($handle, CURLINFO_HTTP_CODE);
// Since we chopped off our download, we'll get a 206; force it to be a 200.
return $status == 206 ? 200 : $status;
} else {
return 404;
}
}
/**
* Pings a domain using ICMP, TCP or UDP.
*
* If USE_ICMP is defined as being TRUE, then it attempts to create a socket connection, which
* requires privilege escalation on most platforms. If this fails, it parses the output of the
* ping utility. If *that* fails, it falls back to a simple fsockopen() ping.
*
* @param associative array $url
* The array of the parsed requested URL.
* @return int $status
* The time it took fsockopen to respond between $starttime and $stoptime assignments.
*/
function ping($url) {
if (USE_ICMP) {
// First ensure that the sockets extension has been loaded and compiled into PHP.
if(extension_loaded('sockets')) {
$operating_system = strtolower(PHP_OS);
$socket = null;
// If we're on Windows, handle socket creation attempt differently; and the ping command.
if (substr_count($operating_system, "win")) {
// Try to create a socket.
if ($socket = socket_create(AF_INET, SOCK_RAW, 1)) {
return perform_icmp_ping($socket, $url['site']);
} else {
// TODO: Add parsing of the Windows ping command here.
return null;
}
} else {
// Attempt to perform a setuid as root.
if(posix_seteuid(0)) {
// This could potentially be quite dangerous since we're sending this out as root.
// We need to determine what user we're running as, and perform another seteuid.
if ($socket = socket_create(AF_INET, SOCK_RAW, 1)) {
return perform_icmp_ping($socket, $url['site']);
} else {
return perform_fsock_ping($url);
}
} else {
// Only perform this if we can be assured that this function is callable.
if(!ini_get('safe_mode') &&
(strpos(ini_get('disable_functions'), 'shell_exec')) == 0) {
// Accommodate the different ping syntax.
// 100 second timeout, 1 attempt.
if (substr_count(strtolower(PHP_OS), "hpux")) {
$result = shell_exec("ping -m 10 -n 1 " . $url['site']);
} else if (substr_count(strtolower(PHP_OS), "aix")) {
$result = shell_exec("ping -i 10 -c 1 " . $url['site']);
} else if (substr_count(strtolower(PHP_OS), "mac") ||
substr_count(strtolower(PHP_OS), "darwin")) {
$result = shell_exec("ping -t 10 -c 1 " . $url['site']);
} else if (substr_count(strtolower(PHP_OS), "bsd")) {
$result = shell_exec("ping -w 10 -c 1 " . $url['site']);
} else {
$result = shell_exec("ping -W 10 -c 1 " . $url['site']);
}
if ($result == null || $result == "") {
return perform_fsock_ping($url);
} else {
// Parse the output.
$position = strpos($result, "min/avg/max");
if ($position > 0) {
$output = trim(str_replace(" ms", "", substr($result, $position)));
$pieces = explode("=", $output);
$results = explode("/", $pieces[1]);
return round($results[1], 0);
} else {
return null;
}
}
} else {
return perform_fsock_ping($url);
}
}
}
} else {
return perform_fsock_ping($url);
}
} else {
return perform_fsock_ping($url);
}
}
/**
* Splits a URL into its core components.
*
* based on: http://snipplr.com/view/6526/split-an-url-into-protocol-site-and-resource-parts/
*
* @param string $url
* A page URL to be parsed. (eg. http://www.google.com/search?q=test)
* @return associative array $result
* Construction:
* string $result['protocol'] = "http"
* int $result['port'] = 80
* string $result['site'] = "www.google.com"
* string $result['resource'] = "/search?q=test"
*/
function split_url($url) {
$result = array();
$regex = '#^(.*?//)*([\w\.\d]*)(:(\d+))*(/*)(.*)$#';
$matches = array();
preg_match($regex, $url, $matches);
// Assign the matched parts of url to the result array
$result['protocol'] = $matches[1];
$result['port'] = $matches[4] == NULL ? 80 : intval($matches[4]);
$result['site'] = $matches[2];
$result['resource'] = $matches[6];
// clean up the site portion by removing the trailing /
$result['site'] = preg_replace('#/$#', '', $result['site']);
// clean up the protocol portion by removing the trailing ://
$result['protocol'] = preg_replace('#://$#', '', $result['protocol']);
return $result;
}
/**
* A basic array comparator.
*
* As with any comparator, it takes two values, compares the two, and then returns whether or not
* the value should move up or down the array (or stay in place: 0)
*
* @param string $a
* Value to be compared against.
* @param string $b
* Value to be compared to.
* @return int
* The direction in which the element whose value is $a should move.
*/
function comparator($a, $b) {
if ($a == $b) {
return 0;
}
return ($a < $b) ? -1 : 1;
}
/**
* An fsockopen() ping.
*
* This will perform a rudimentary attempt at a connection and time it
* in order to attempt a guess at how well the server is responding.
*
* This can support either UDP or TCP, depending on how the server is defined in
* the INI_FILE.
*
* @param associative array $url
* The array of URL request elements.
* @return int $status
* The calculated time in ms. If null, it means an error occurred.
*/
function perform_fsock_ping($url) {
if (!ini_get('safe_mode') &&
(strpos(ini_get('disable_functions'), 'fsockopen') == 0)) {
$start_time = microtime(true);
// Allow per-site UDP/TCP handling.
if (strtolower($url['protocol']) == "udp") {
$url['site'] = "udp://".$url['site'];
}
$file = fsockopen($url['site'], $url['port'], $errno, $errstr, 10);
$stop_time = microtime(true);
$status = null;
// The site is not responding.
if (!$file) {
return null;
} else {
fclose($file);
$status = ($stop_time - $start_time) * 1000;
$status = floor($status);
}
return $status;
} else {
return null;
}
}
/**
* An ICMP packet ping.
*
* This method has multiple facilities:
* 1. An attempt is made to blindly create a raw socket.
* 2. If the above somehow fails, perhaps due to Vista or Windows 7,
* it tries to use the built-in ping, and takes the final avg of that.
* 3. Otherwise, it will return null. This is so an error can be differentiated from a response.
*
* @param string $domain
* The domain name to be pinged.
*/
function perform_icmp_ping($socket, $domain) {
// Set the timeout values to 2 seconds.
socket_set_option($socket,SOL_SOCKET,SO_RCVTIMEO, array("sec"=>2,
"usec"=>0));
// Attempt a connection.
if(socket_connect($socket, $domain, null)) {
$packet = create_icmp_packet();
// Attempt to send the packet.
if(socket_write($socket, $packet, strlen($packet))) {
$start_time = microtime(TRUE);
// Grab the socket success status while building our read array, and timeouts.
$response = socket_select($read_array = array($socket), $write_array = NULL, $f = NULL, $timeout_seconds, $timeout_milliseconds);
// Only move forward if the socket states it is OK (responded within given timeout).
if ($response > 0) {
$response_time = round((microtime(TRUE) - $start_time) * 1000, 0);
/** TODO: Perform sequence validation to avoid man-in-the-middle attacks.
* $result = socket_read($socket, MAX_ICMP_BYTES);
* $response_sequence = substr($result, 26, 2);
* if ($response_sequence != $packet->sequence) {
* return null;
* } */
socket_close($socket);
return $response_time;
} else {
socket_close($socket);
return null;
}
} else {
socket_close($socket);
return null;
}
} else {
return null;
}
}
/**
* Build a basic ICMP packet.
*
* But it goes something like, where each '-' is a bit, and ~ is variable length.
* |--------|--------|----------------|~~~~~~~~|
* Type Code Checksum Data
*
* For further definition see http://tools.ietf.org/rfc/rfc0792.txt.
*/
function create_icmp_packet() {
// Qualification of the packet.
$data = "ping from a file download proxy";
// We want to send an echo request packet.
$type = "\x08";
$code = "\x00";
// As specified by RFC-792, we need to 0 out this checksum before calculating it.
$checksum = "\x00\x00";
// Generate a random 16-bit identifier and sequence.
$identifier = chr(rand(0, 255)).chr(rand(0, 255));
$sequence = "\x00\x00";
// Generate a checksum of the almost completed packet.
$checksum = perform_icmp_checksum($type.$code.$checksum.$identifier.$sequence.$data);
// Return the built packet.
return ($type.$code.$checksum.$identifier.$sequence.$data);
}
/**
* Generate a checksum of an almost complete packet (generating this is the final piece)
*
* This checksum generation is based off of in_cksum from
* http://ws.edu.isoc.org/materials/src/ping.c
*
* @param short $packet
* The packet containing everything but the checksum.
*
* @return short $checksum
* The checksum of the given packet.
*/
function perform_icmp_checksum($packet) {
// Ensure there are no trailing bits, if so, add another bit.
if(strlen($packet) % 2) {
$packet.="\x00";
}
// First, create a new array of unsigned shorts from our packet
$bit_array = unpack("n*",$packet);
// Now get a sum total of the array.
$bit_sum = array_sum($bit_array);
// Now do some bit shifting
while($bit_sum >> 16) {
$bit_sum = ($bit_sum >> 16) + ($bit_sum & 65535);
}
// Return the unsigned short truncated sum.
return pack('n*', ~$bit_sum);
}
?>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment