Skip to content

Instantly share code, notes, and snippets.

@tobek
Last active August 29, 2015 14:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tobek/9edceef229e5ad5e7ed1 to your computer and use it in GitHub Desktop.
Save tobek/9edceef229e5ad5e7ed1 to your computer and use it in GitHub Desktop.
Unfurler QS blacklist checker
LEGEND:
= Match
_ No canonical URL found in either
~ Both requests failed
X Mismatch
========================================================================X=XXXX====================== ✔ {"domain":"www.marketwatch.com"}
======_===============~===========================================_=====_=~_=================_=====X ✔ {"domain":"nymag.com"}
____________________________________________________________________________________________________ ✖ {"domain":"luufy.com"}
===__________=====X====_===========___==============_====_______===__===============_=____=_======== ✔ {"domain":"www.zazzle.com"}
==========================================================_X======================================== ✔ {"domain":"www.dailymail.co.uk"}
____________________________________________________________________________________________________ ✖ {"domain":"news.feedzilla.com"}
==================================================================================================== ✔ {"domain":"youtube.com","param":"feature"}
==================================================================================================== ✔ {"domain":"youtube.com","param":"a"}
====================_================_=========_==============================_==============_=====_ ✔ {"domain":"www.youtube.com","param":"feature"}
====================================================================_=============================== ✔ {"domain":"www.youtube.com","param":"sns"}
=_==========================================================_========_=====_=============_========== ✔ {"domain":"www.youtube.com","param":"app"}
_=================================================_=========_=_====____====================___====== ✔ {"domain":"www.youtube.com","param":"hd"}
==================================================================================================== ✔ {"domain":"www.youtube.com","param":"autoplay"}
==_=============_=_===================================_=_==_==================_=========_=====_===== ✔ {"domain":"www.youtube.com","param":"a"}
==================================================================================================== ✔ {"domain":"www.huffingtonpost.co.uk","param":"ir"}
==================================================================================================== ✔ {"domain":"www.huffingtonpost.com","param":"cps"}
==X=================================_=========================X==========_==X======================= ✔ {"domain":"www.linkedin.com","param":"trk"}
X========_===_=__XXX===X=X==X=XX___=_=_X===_=X___XX=_===X===_==_=====X==_=_XX____X=__===X=_====X_=X= ✖ {"domain":"www.linkedin.com","param":"goback"}
=_X=X==X==X==X====_=_X===X=X==X==X=X=X_=X_=XX==XX====_X==X=_X_XX=X===XX=X_===X==XXX_=== ✖ {"domain":"www.linkedin.com","param":"qid"}
======= ✔ {"domain":"www.linkedin.com","param":"anchorTime"}
==================================================================================================== ✔ {"domain":"www.linkedin.com","param":"_mSplash"}
=_================================================================================================== ✔ {"domain":"uk.reuters.com","param":"feedName"}
==_================================================================================================= ✔ {"domain":"uk.reuters.com","param":"feedType"}
==================================_================================================================= ✔ {"domain":"uk.reuters.com","param":"WT.tsrc"}
==================================================================================================== ✔ {"domain":"br.reuters.com","param":"feedName"}
==================================================================================================== ✔ {"domain":"br.reuters.com","param":"feedType"}
==============================================================================================_===== ✔ {"domain":"jp.reuters.com","param":"feedName"}
==============================================================================================_===== ✔ {"domain":"jp.reuters.com","param":"feedType"}
==================_================================================================================= ✔ {"domain":"in.reuters.com","param":"feedName"}
=================_================================================================================== ✔ {"domain":"in.reuters.com","param":"feedType"}
==_==============================================_======_=======_=================================== ✔ {"domain":"www.reuters.com","param":"feedName"}
==_==============================================_=====_========_=================================== ✔ {"domain":"www.reuters.com","param":"feedType"}
____________________________________________________________________________________________________ ✖ {"domain":"m.facebook.com","param":"type"}
____________________________________________________________________________________________________ ✖ {"domain":"m.facebook.com","param":"theater"}
____________________________________________________________________________________________________ ✖ {"domain":"m.facebook.com","param":"set"}
____________________________________________________________________________________________________ ✖ {"domain":"m.facebook.com","param":"fref"}
____________________________________________________________________________________________________ ✖ {"domain":"www.facebook.com","param":"permPage"}
____________________________________________________________________________________________________ ✖ {"domain":"www.facebook.com","param":"type"}
____________________________________________________________________________________________________ ✖ {"domain":"www.facebook.com","param":"theater"}
_=___=________==__=____=___===_=____=__________===_________=____________________=___=_==____====__=_ ✖ {"domain":"www.facebook.com","param":"brandloc"}
____________________________________________________________________________________________________ ✖ {"domain":"www.facebook.com","param":"set"}
________=_________________________=______________________________=______=_______=____=__=___=_______ ✖ {"domain":"www.facebook.com","param":"fref"}
==_=_=_===_==============_=====~===========_==================================_==X~==_=X===~X======= ✔ {"domain":"www.amazon.co.uk","param":"tag"}
_=====================X===_=======_=================~_=_=============_===_==_======================= ✔ {"domain":"www.amazon.co.uk","param":"qid"}
_=========================_====X==_================_===_===========_======_==_====================== ✔ {"domain":"www.amazon.co.uk","param":"sr"}
==================================================================================================== ✔ {"domain":"www.amazon.co.uk","param":"keywords"}
_==_=_=_==========_====~=====_=====================_X=~_=X~X====================____==_~===_==~====_ ✔ {"domain":"www.amazon.co.uk","param":"linkCode"}
==_==_=========_====~======X===================X==X~=~===================_==X===~=~===X=======~===== ✔ {"domain":"www.amazon.co.uk","param":"camp"}
==_==_==========_===~========================X~==X==~===================_==~=X==~==_=========~====~= ✔ {"domain":"www.amazon.co.uk","param":"creative"}
_===============_============================================================================_====== ✔ {"domain":"www.amazon.co.uk","param":"s"}
_=_=~_=_X===X=X==~=X~=_X_XXX~_=____~~~_X=_~==X==~===~==_~~=====X==X====~X=XX====X_===~=XXXX==~X==~X~ ✖ {"domain":"www.amazon.co.uk","param":"_encoding"}
_=========_=X======_===============_=_=X====_===_===========_==============================_======== ✔ {"domain":"www.amazon.co.uk","param":"ie"}
==================X==================================~_========X=========_=====_==X============X==== ✔ {"domain":"www.amazon.com","param":"tag"}
============_===========================_===========_=============================================== ✔ {"domain":"www.amazon.com","param":"qid"}
============_===========================_==========_================================================ ✔ {"domain":"www.amazon.com","param":"sr"}
=========_===================_=======X===X===========X======X=============X====X=======X==X========= ✔ {"domain":"www.amazon.com","param":"keywords"}
====X=======_=====_=====X============================X========_=~===_X====~=_====_================== ✔ {"domain":"www.amazon.com","param":"linkCode"}
===X========_=====_=====X============================X========~==X=====~_======_==================== ✔ {"domain":"www.amazon.com","param":"camp"}
==X=======_=====_======X========================X=======~====X====~=_=====_========================= ✔ {"domain":"www.amazon.com","param":"creative"}
==================================================================================================== ✔ {"domain":"www.amazon.com","param":"s"}
===========XX==_=~=======~_==========X=========_~_====X=___=_=_=======X==__==_==X==_~======__======_ ✔ {"domain":"www.amazon.com","param":"_encoding"}
===========X==========_=============================_=====_===X_===_======X=======X===========_===== ✔ {"domain":"www.amazon.com","param":"ie"}
=======================X============================================================================ ✔ {"domain":"www.ebay.com","param":"lgeo"}
==================================================================================================== ✔ {"domain":"www.ebay.com","param":"vectorid"}
==================================================================__================================ ✔ {"domain":"www.ebay.com","param":"roken"}
==================================================================================================== ✔ {"domain":"www.ebay.com","param":"ssPageName"}
XXXXXXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXX_XXXX=XXXXX=XXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXXX=X ✖ {"domain":"www.ebay.com","param":"_nkw"}
=================================================================================X========X=X====X== ✔ {"domain":"www.ebay.com","param":"_sacat"}
===__==================================_==================_========================================= ✔ {"domain":"itunes.apple.com","param":"mt"}
=__===========_==========================================================_=X=_=XX===X==_============ ✔ {"domain":"itunes.apple.com","param":"ign-mpt"}
____________________________________________________________________________________________________ ✖ {"domain":"play.spotify.com","param":"play"}
==================================================================================================== ✔ {"domain":"www.ft.com","param":"_i_referer"}
==============================================================================================_===== ✔ {"domain":"www.ft.com","param":"siteedition"}
==================================================================================================== ✔ {"domain":"www.ft.com","param":"ftcamp"}
==X==X===X=X=X=X=====X======X=======XXX====X==X===========X===X===X=X==X===X=======X======X=======_= ✖ {"domain":"www.inc.com","param":"cid"}
==================================================================================================== ✔ {"domain":"www.cbc.ca","param":"cmp"}
==================================================================================================== ✔ {"domain":"www.buzzfeed.com","param":"bftw"}
==================================================================================================== ✔ {"domain":"www.theglobeandmail.com","param":"click"}
_=========X=XXX_XXXXXXX=_XX=_==__X=XX_XXXXX_XXX=XXXXXXXX__XX==XXXXXXXXXXXXXXX_XXXXX=_X=X=XXXXXXXX_X_ ✖ {"domain":"dragplus.com","param":"title"}
=======XXX_XXXX==XXX_XXX_X=XXXXXX=XXXXXXXXXXX=XXX=X=__X_X_=_XXX=X=_X==__X=XX=_=XXXXXXXX====XXX===XXX ✖ {"domain":"dragplus.com","param":"clid"}
==================================================================================================== ✔ {"domain":"dragplus.com","param":"ct2"}
==================================================================================================== ✔ {"domain":"dragplus.com","param":"fd"}
==================================================================================================== ✔ {"domain":"dragplus.com","param":"ei"}
==================================================================================================== ✔ {"domain":"dragplus.com","param":"usg"}
___=_=_=___=___=__=__=_=______=___=___=____=_====_=_===__=______==_____=___________=__===___==__==__ ✔ {"domain":"www.careerbuilder.com","param":"siteid"}
___________________=_______=_____=_=___=__=_____________=__________________________________=________ ✖ {"domain":"www.careerbuilder.com","param":"SiteID"}
___=______=___________=__________=____==____=__=_______=__=____________=_________=_________ ✖ {"domain":"www.careerbuilder.com","param":"IPath"}
___=_=_=___=___=__=_=_=______=___=___=____=_===_==_===__=_____==_____=__________=___===__==_==___=__ ✔ {"domain":"www.careerbuilder.com","param":"ipath"}
__=_=____=_____==_=____===__=___=____=__ ✔ {"domain":"www.careerbuilder.com","param":"APath"}
__=_=__=___=__=___=_=_=______=__=_____=___=_===_=_====___=____=__=___=___________=_=_=_=_==_=__=__=_ ✔ {"domain":"www.careerbuilder.com","param":"showNewJDP"}
__=_=__=___=___=__=_==_______=__=______=___====_===_==___=____=_=____=__________=___===__==_=_=__=__ ✔ {"domain":"www.careerbuilder.com","param":"lr"}
____________________________________________________________________________________________________ ✖ {"domain":"klout.com","param":"n"}
____________________________________________________________________________________________________ ✖ {"domain":"klout.com","param":"v"}
____________________________________________________________________________________________________ ✖ {"domain":"instagram.com","param":"modal"}
======_=========_====_==================================_===_=====================_================= ✔ {"domain":"www.flickr.com","param":"rb"}
=_========================================================_=_==========_========================_=== ✔ {"domain":"www.flickr.com","param":"WT.mc_id"}
==_=======================================================================================_====== ✔ {"domain":"www.flickr.com","param":"WT.cg_n"}
======================_============================================================================= ✔ {"domain":"www.elmundo.es","param":"cid"}
=======================_============================================================================ ✔ {"domain":"www.elmundo.es","param":"s_kw"}
=====____=__====_======__===_===__=======_==========_=_=_=_==_=============_=_=====_=_=====_=__==__= ✔ {"domain":"www.usatoday.com","param":"csp"}
____________________________________________________________________________________________________ ✖ {"domain":"www.shorouknews.com","param":"cdate"}
====_========_======================_===__=_============_=========__==========_=====_=_============= ✔ {"domain":"www.ibm.com","param":"cmp"}
====_======_====================_=_=_=_============_======_===_==========_=====__=============_==_== ✔ {"domain":"www.ibm.com","param":"cr"}
====_======_====================_===_=_===_=========_=====_===_==========_=====_=_=============_===_ ✔ {"domain":"www.ibm.com","param":"ct"}
====_======_====================__===__=============_=====_==_===========_======_=_==========_=_==== ✔ {"domain":"www.ibm.com","param":"cm"}
====_======_==================_==_==_=_=============_=====_==_==========_======_=_============_==_== ✔ {"domain":"www.ibm.com","param":"ccy"}
=====_=====_==================_==_==__==============_=====_==_============_====_=_============_=_=== ✔ {"domain":"www.ibm.com","param":"ce"}
===========_=============_==_=__======_====_==_========_====_========_===_=======_=====__=========== ✔ {"domain":"www.ibm.com","param":"ca"}
==================================================================================================== ✔ {"domain":"money.cnn.com","param":"hpt"}
==================================================================================================== ✔ {"domain":"money.cnn.com","param":"sr"}
==================================================================================================== ✔ {"domain":"money.cnn.com","param":"iid"}
==================================================================================================== ✔ {"domain":"money.cnn.com","param":"section"}
==========X======X======================================X======XX===X===X=========================== ✔ {"domain":"edition.cnn.com","param":"eref"}
===============================================================X=X=====X==========X=======~========X ✔ {"domain":"edition.cnn.com","param":"hpt"}
===X==X==============================================X======================X==========X===========X ✔ {"domain":"edition.cnn.com","param":"sr"}
=====__===_==_=======_==_=_=___===____=_____=_=__=_=______==_========________=_==========_=___==_=__ ✔ {"domain":"edition.cnn.com","param":"video_referrer"}
____________________________________________________________________________________________________ ✖ {"domain":"news.yahoo.com","param":"soc_src"}
___________________________________________________X________________________________________________ ✖ {"domain":"news.yahoo.com","param":".tsrc"}
=================================================_================================================== ✔ {"domain":"www.latimes.com","param":"cid"}
==================================================================================================== ✔ {"domain":"www.latimes.com","param":"track"}
======================================================================================X============= ✔ {"domain":"www.upworthy.com","param":"c"}
==================================================================================================== ✔ {"domain":"www.upworthy.com","param":"g"}
==================================================================================================== ✔ {"domain":"raptr.com","param":"type"}
======_=====_==_======_=========_===_==_======__=====_=====_=====_====_====_====_===_========_====== ✔ {"domain":"espn.go.com","param":"ex_cid"}
==================================================================================================== ✔ {"domain":"www.dw.de","param":"maca"}
==================================================================================================== ✔ {"domain":"www.cracked.com","param":"wa_ibsrc"}
==================================================================================================== ✔ {"domain":"www.vulture.com","param":"mid"}
==================================================================================================== ✔ {"domain":"www.politico.com","param":"hp"}
==========================================_========================================================= ✔ {"domain":"www.politico.com","param":"ml"}
===================================================_==========================_======_============== ✔ {"domain":"item.rakuten.co.jp","param":"scid"}
==========X===~====================~=============================_=====_====X======================= ✔ {"domain":"runkeeper.com","param":"channel"}
=_=_________=_=___=_=_____===_____===____=__=________=_=___=_=_=__=_==_===__===_=__=__=__=_====_____ ✔ {"domain":"www.yelp.com","param":"pt"}
_=__________=_____=____=____________=________=____=____=___==__==____=_=_==_=_____=__=________=_____ ✖ {"domain":"www.yelp.com","param":"v"}
==================================================================================================== ✔ {"domain":"www.npr.org","param":"ft"}
==================================================================================================== ✔ {"domain":"www.npr.org","param":"f"}
_______=_==_==__=___=_=_===___=_=___=___==____X_====___=====_===_==X==__=_=_=_____===_======_=_=_=== ✔ {"param":"utm_content"}
_____=__=________===__=_===_===_=_=_=X===__=____=____=_=__==_=====___==__=____=_=____===_____==__=_= ✔ {"param":"utm_medium"}
_____==___________===___=====_===___X=====__=_=_______==_=__=_====__=_=_=___=__=__==_=_==_____==___= ✔ {"param":"utm_source"}
_=_______==_=======_===__=__==__====_==__=_=__=__=__=__=_=_____==__=_==_=___=__X_=__==_===__=_==_=== ✔ {"param":"utm_campaign"}
________=__=_===_=X==__XX___=__=___=_=_===_===================_=_=_=_===_=__=_=_====_____==__==_~=== ✔ {"param":"utm_term"}
__=__=_=____=__===_=_===X_=_==__==== ✔ {"param":"utm_referrer"}
===~======~==X=_===X==_===_===X== ✔ {"param":"utm_reader"}
__=================_ ✔ {"param":"utm_hp_ref"}
_====_=====__====X==__========_===========__=X=_==_===_===X==_=~===========X========_===_==__=_====_ ✔ {"param":"utm_keyword"}
___==__=_____===_=_=_== ✔ {"param":"utm_name"}
___=____=_==___===_=_====_X_=___=__=__=_===__ ✔ {"param":"utm_expid"}
=====____==_==_======_==____=_ ✔ {"param":"utm_destname"}
====___=_==__========_==___=__ ✔ {"param":"utm_desttype"}
=====_= ✔ {"param":"curator"}
_=====______=___=____=___=___=_=__=____=__==__~_________=_==_=_==__====_===__X__=__=_=X_=__==_X_=___ ✔ {"param":"campaign"}
___===____====__===_=_______===__=____=_=___===__=_=_==_==_=__=_===_===_======_=_==========_==_=_==_ ✔ {"param":"cmpid"}
==================~===X__=___=___=___=____===_==__=_=___X___===_X_____======____=_==_____X__=_=_X=== ✔ {"param":"linkId"}
=X~=======X=_=========~_===~~===~==~=~~~~=_=~=~=====~=~=_~_~=_~~~=~===~_=_=====_~_=X=X=_==_==_==__== ✔ {"param":"dlvrit"}
_==_====_==_==X==_=___====__==____==_=____=___===_=____=__==__=====___==__==_X======___===_=_=_===__ ✔ {"param":"hootPostID"}
____=_________=_====_________________=___=____=_________=_X__=_____________=_====_=___=_=_=X___==___ ✖ {"param":"src"}
__=_=__________=_=_=_____=_=______=_____=_=====__===================__=__=_____~_=___==______=__X=__ ✔ {"param":"source"}
_=____=__=__~___X_==_= ✔ {"param":"share"}
_==__=___=__X_=_______=_______=__=_=_______=_=====_=__=________=___=_=_=___==__=___=_X_______==__=_= ✔ {"param":"ref"}
_==========================X=======================_================================================ ✔ {"param":"spref"}
Done! Checked 159 and found problems with 30
Problem with: {"domain":"luufy.com"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"news.feedzilla.com"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"www.linkedin.com","param":"goback"} | checked: 100 | mismatched: 21 | no canonical: 26 | requests failed: 0
Problem with: {"domain":"www.linkedin.com","param":"qid"} | checked: 87 | mismatched: 30 | no canonical: 10 | requests failed: 0
Problem with: {"domain":"m.facebook.com","param":"type"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"m.facebook.com","param":"theater"} | checked: 89 | mismatched: 0 | no canonical: 89 | requests failed: 0
Problem with: {"domain":"m.facebook.com","param":"set"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"m.facebook.com","param":"fref"} | checked: 70 | mismatched: 0 | no canonical: 70 | requests failed: 0
Problem with: {"domain":"www.facebook.com","param":"permPage"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"www.facebook.com","param":"type"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"www.facebook.com","param":"theater"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"www.facebook.com","param":"brandloc"} | checked: 100 | mismatched: 0 | no canonical: 76 | requests failed: 0
Problem with: {"domain":"www.facebook.com","param":"set"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"www.facebook.com","param":"fref"} | checked: 100 | mismatched: 0 | no canonical: 92 | requests failed: 0
Problem with: {"domain":"www.amazon.co.uk","param":"_encoding"} | checked: 100 | mismatched: 22 | no canonical: 15 | requests failed: 17
Problem with: {"domain":"www.ebay.com","param":"_nkw"} | checked: 100 | mismatched: 94 | no canonical: 2 | requests failed: 0
Problem with: {"domain":"play.spotify.com","param":"play"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"www.inc.com","param":"cid"} | checked: 100 | mismatched: 21 | no canonical: 1 | requests failed: 0
Problem with: {"domain":"dragplus.com","param":"title"} | checked: 100 | mismatched: 65 | no canonical: 14 | requests failed: 0
Problem with: {"domain":"dragplus.com","param":"clid"} | checked: 100 | mismatched: 59 | no canonical: 12 | requests failed: 0
Problem with: {"domain":"www.careerbuilder.com","param":"SiteID"} | checked: 100 | mismatched: 0 | no canonical: 92 | requests failed: 0
Problem with: {"domain":"www.careerbuilder.com","param":"IPath"} | checked: 91 | mismatched: 0 | no canonical: 79 | requests failed: 0
Problem with: {"domain":"klout.com","param":"n"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"klout.com","param":"v"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"instagram.com","param":"modal"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"www.shorouknews.com","param":"cdate"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"news.yahoo.com","param":"soc_src"} | checked: 100 | mismatched: 0 | no canonical: 100 | requests failed: 0
Problem with: {"domain":"news.yahoo.com","param":".tsrc"} | checked: 100 | mismatched: 1 | no canonical: 99 | requests failed: 0
Problem with: {"domain":"www.yelp.com","param":"v"} | checked: 100 | mismatched: 0 | no canonical: 80 | requests failed: 0
Problem with: {"param":"src"} | checked: 100 | mismatched: 2 | no canonical: 76 | requests failed: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment