Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Script to find official Debian mirrors that support HTTPS
#!/usr/bin/env python3
import urllib.request
import re
import ssl
import sys
# # find generic mirrors
mirrors = urllib.request.urlopen('http://www.debian.org/mirror/list')
https = []
for line in mirrors.readlines():
m = re.match(b'.*<td valign="top"><a rel="nofollow" href="http(.*)">.*', line)
if m:
url = 'https' + m.group(1).decode()
print('trying: %s...' % url)
sys.stdout.flush()
try:
response=urllib.request.urlopen(url, timeout=1)
https.append(url)
print('success!')
except urllib.request.URLError as err:
print('fail!')
except ssl.SSLError as err:
print('bad SSL!')
except:
print('bad stuff!!!')
# print('HTTPS apt repos:'
#for url in https:
# print(url)
# # find security mirrors
mirrors = urllib.request.urlopen('http://www.debian.org/mirror/list-full')
securitys = []
for line in mirrors.readlines():
m = re.match(b'.*</tt><br>Security updates over HTTP: <tt><a rel="nofollow" href="http(.*)">.*/debian-security/</a>.*', line)
if m:
url = 'https' + m.group(1).decode()
print('trying: %s...' % url)
sys.stdout.flush()
try:
response=urllib.request.urlopen(url, timeout=1)
securitys.append(url)
print('success!')
except urllib.request.URLError as err:
print('fail!')
except ssl.SSLError as err:
print('bad SSL!')
# print('HTTPS security repos:'
# for url in securitys:
# print(url)
# now find the backports mirrors
try:
mirrors = urllib.request.urlopen('http://backports-master.debian.org/Mirrors/')
except:
print('URL open failed!!!')
backports = []
for line in mirrors.readlines():
#<td><a href="http://be.mirror.eurid.eu/debian-backports/">/debian-backports/</a>
m = re.match(b'.*<td><a href="http(.*)">.*/debian-backports/</a>.*', line)
if m:
url = 'https' + m.group(1).decode()
print('trying: %s...' % url)
sys.stdout.flush()
try:
response=urllib.request.urlopen(url, timeout=1)
backports.append(url)
print('success!')
except urllib.request.URLError as err:
print('fail!')
except ssl.SSLError as err:
print('bad SSL!')
#print('HTTPS backports repos:'
#for url in backports:
# print(url)
# now find the CD image mirrors
mirrors = urllib.request.urlopen('http://www.debian.org/CD/http-ftp/')
cds = []
for line in mirrors.readlines():
# <a rel="nofollow" href="http://mirror.easyspeedy.com/debian-cd/">HTTP</a></li>
m = re.match(b'.*<a rel="nofollow" href="http(:.*)">HTTP</a></li>.*', line)
if m:
url = 'https' + m.group(1).decode()
print('trying: %s...' % url)
sys.stdout.flush()
try:
response=urllib.request.urlopen(url, timeout=1)
cds.append(url)
print('success!')
except urllib.request.URLError as err:
print('fail!')
except ssl.SSLError as err:
print('bad SSL!')
except:
print('bad stuff!')
print('HTTPS CD image repos:')
for url in cds:
print(url)
# now write everything to a file
dst_filename='/tmp/https-debian-archives.txt'
f = open(dst_filename, 'w')
print('Outputfile: "%s"' % dst_filename)
f.write('HTTPS apt repos\n')
f.write('---------------\n')
for url in https:
f.write(url + '\n')
f.write('\n\nHTTPS security repos\n')
f.write('---------------\n')
for url in securitys:
f.write(url + '\n')
f.write('\n\nHTTPS backports repos\n')
f.write('--------------------\n')
for url in backports:
f.write(url + '\n')
f.write('\n\nHTTPS CD image repos\n')
f.write('--------------------\n')
for url in cds:
f.write(url + '\n')
f.close()
@tryingsomestuff

This comment has been minimized.

Copy link

@tryingsomestuff tryingsomestuff commented Oct 16, 2014

And did you find some https mirrors ? :-)

@eighthave

This comment has been minimized.

Copy link
Owner Author

@eighthave eighthave commented Oct 17, 2014

Sure, there are many. I regularly use:

  • mirrors.ece.ubc.ca
  • mirrors.kernel.org
  • mirror.cse.unsw.edu.au
  • spout.ussg.indiana.edu
@sipdbg2

This comment has been minimized.

Copy link

@sipdbg2 sipdbg2 commented Sep 26, 2016

Hey eighthave, i tried your script and liked it, i added threading in my fork 👍

@frn4pi

This comment has been minimized.

Copy link

@frn4pi frn4pi commented Feb 11, 2017

Hey,

+add line(s)
except ssl.CertificateError as err:
print 'CertificateError: BAD CERT DOMAIN'

for example:

https://ftp.be.debian.org => SSLCert for => https://mirror.as35701.net*

mercy
bye

@abeluck

This comment has been minimized.

Copy link

@abeluck abeluck commented Nov 11, 2017

@eighthave Why not use https for the mirror list?

mirrors = urllib.request.urlopen('httpS://www.debian.org/mirror/list')  #<----
@dkg

This comment has been minimized.

Copy link

@dkg dkg commented Jan 22, 2019

@eighthave thanks for writing this! it would be great if you could publish the current results someplace. We should encourage more mirror operators to offer https.

@thomsh

This comment has been minimized.

Copy link

@thomsh thomsh commented Jan 22, 2019

CVE-2019-3462 : Remote Code Execution in apt/apt-get

Supporting http is fine. I just think it’s worth making https repositories the default – the safer default – and allowing users to downgrade their security at a later time if they choose to do so. I wouldn’t have been able to exploit the Dockerfile at the top of this post if the default package servers had been using https.

sources:
https://justi.cz/security/2019/01/22/apt-rce.html
https://www.debian.org/security/2019/dsa-4371

@whalehub

This comment has been minimized.

Copy link

@whalehub whalehub commented Jan 26, 2019

@tryingsomestuff @dkg Here is the full output of the Python script, filtered by mirrors that were reachable via HTTPS:

https://debian.unnoba.edu.ar/debian/

https://mirror.aarnet.edu.au/debian/

https://debian.anexia.at/debian/

https://ftp.tu-graz.ac.at/mirror/debian/

https://mirror.internex.at/debian/

https://mirror.datacenter.by/debian/

https://ftp.belnet.be/debian/

https://mirror.as35701.net/debian/

https://repositorio.nti.ufal.br/debian/

https://debian.ludost.net/debian/

https://ftp.uni-sofia.bg/debian/

https://debian.ca-west.mirror.fullhost.com/debian/

https://mirror.csclub.uwaterloo.ca/debian/

https://mirrors.163.com/debian/

https://mirrors.tuna.tsinghua.edu.cn/debian/

https://mirrors.ustc.edu.cn/debian/

https://mirrors.ucr.ac.cr/debian/

https://ftp.debian.cz/debian/

https://ftp.sh.cvut.cz/debian/

https://ftp.zcu.cz/debian/

https://merlin.fit.vutbr.cz/debian/

https://mirror.dkm.cz/debian/

https://mirror.one.com/debian/

https://mirrors.dotsrc.org/debian/

https://debian.ues.edu.sv/debian/

https://ftp.eenet.ee/debian/

https://debian.mirror.ate.info/

https://ftp.lip6.fr/pub/linux/distributions/debian/

https://ftp.u-strasbg.fr/debian/

https://mirror.plusserver.com/debian/debian/

https://mirrors.ircam.fr/pub/debian/

https://debian.grena.ge/debian/

https://artfiles.org/debian/

https://debian.charite.de/debian/

https://debian.inf.tu-dresden.de/debian/

https://debian.tu-bs.de/debian/

https://ftp.fau.de/debian/

https://ftp.halifax.rwth-aachen.de/debian/

https://ftp.mpi-sb.mpg.de/pub/linux/debian/

https://ftp.plusline.net/debian/

https://ftp-stud.hs-esslingen.de/debian/

https://ftp.tu-chemnitz.de/debian/

https://mirror.eu.oneandone.net/debian/

https://mirror.netcologne.de/debian/

https://mirror.wtnet.de/debian/

https://packages.hs-regensburg.de/debian/

https://mirror.xtom.com.hk/debian/

https://ftp.fsn.hu/debian/

https://debian.simnet.is/debian/

https://mirror.isoc.org.il/pub/debian/

https://debian.mirror.garr.it/debian/

https://ftp.linux.it/debian/

https://dennou-k.gfd-dennou.org/debian/

https://dennou-q.gfd-dennou.org/debian/

https://ftp.jaist.ac.jp/debian/

https://ftp.yz.yamagata-u.ac.jp/debian/

https://mirror.hoster.kz/debian/

https://mirror.ps.kz/debian/

https://debian.mirror.liquidtelecom.com/debian/

https://ftp.harukasan.org/debian/

https://ftp.lanet.kr/debian/

https://debian.koyanet.lv/debian/

https://mirror.litnet.lt/debian/

https://debmirror.tuxis.nl/debian/

https://ftp.nluug.nl/debian/

https://mirror.i3d.net/pub/debian/

https://mirror.neostrada.nl/debian/

https://mirror.nforce.com/debian/

https://mirror.nl.datapacket.com/debian/

https://mirror.novg.net/debian/

https://mirror.serverius.net/debian/

https://mirror.vpgrp.io/debian/

https://mirror.fsmg.org.nz/debian/

https://mirror.pregi.net/debian/

https://ftp.icm.edu.pl/pub/Linux/debian/

https://ftp.eq.uc.pt/software/Linux/debian/

https://ftp.rnl.tecnico.ulisboa.pt/pub/debian/

https://mirrors.up.pt/debian/

https://mirrors.nav.ro/debian/

https://mirrors.nxthost.com/debian/

https://mirrors.pidginhost.com/debian/

https://ftp.psn.ru/debian/

https://mirror.corbina.net/debian/

https://mirror.0x.sg/debian/

https://ftp.arnes.si/debian/

https://debian.redimadrid.es/debian/

https://debian.redparra.com/debian/

https://ftp.caliu.cat/debian/

https://ftp.cica.es/debian/

https://ftp.acc.umu.se/debian/

https://mirror.linux.pizza/debian/

https://mirror.zetup.net/debian/

https://debian.ethz.ch/debian/

https://mirror.init7.net/debian/

https://mirror.iway.ch/debian/

https://mirror.sinavps.ch/debian/

https://pkg.adfinis-sygroup.ch/debian/

https://debian.csie.ntu.edu.tw/debian/

https://ftp.ntou.edu.tw/debian/

https://mirror.applebred.net/debian/

https://mirror.kku.ac.th/debian/

https://ftp.metu.edu.tr/debian/

https://debian.volia.net/debian/

https://debian.mirror.uk.sargasso.net/debian/

https://mirror.bytemark.co.uk/debian/

https://mirror.mythic-beasts.com/debian/

https://mirror.sax.uk.as61049.net/debian/

https://mirrorservice.org/sites/ftp.debian.org/debian/

https://mirror.sucs.swan.ac.uk/pub/linux/debian/

https://mirror.vorboss.net/debian/

https://debian.osuosl.org/debian/

https://mirror.hmc.edu/debian/

https://mirror.math.princeton.edu/pub/debian/

https://mirrors.bloomu.edu/debian/

https://mirrors.cat.pdx.edu/debian/

https://mirrors.edge.kernel.org/debian/

https://mirrors.gigenet.com/debian/

https://mirror.sjc02.svwh.net/debian/

https://mirrors.lug.mtu.edu/debian/

https://mirrors.namecheap.com/debian/

https://mirrors.ocf.berkeley.edu/debian/

https://mirrors.syringanetworks.net/debian/

https://mirror.steadfast.net/debian/

https://mirrors.wikimedia.org/debian/

https://mirrors.xmission.com/debian/

https://repo.cure.edu.uy/debian/

@HacKanCuBa

This comment has been minimized.

Copy link

@HacKanCuBa HacKanCuBa commented Jan 2, 2020

I forked this and made it asynchronous and parallel (for Python 3.7+): https://gist.github.com/HacKanCuBa/e3a998d68a82f81dbf11f2cce4f26d04

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment