This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from datetime import datetime | |
from datetime import datetime | |
import requests | |
from bs4 import BeautifulSoup as BSoup | |
from lxml import html | |
def get_html(): | |
url = 'https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import requests | |
import shutil | |
r = requests.get('url', stream=True) | |
if r.status_code == 200: | |
with open(path, 'wb') as f: | |
r.raw.decode_content = True | |
shutil.copyfileobj(r.raw, f) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# ONLY UBUNTU < 20.04 | |
$ sudo apt-get install chromium-chromedriver chromium-browser | |
$ sudo ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/chromedriver |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import unicodedata | |
city = "České Budějovice" | |
normalized = unicodedata.normalize('NFD', city) | |
new_city = u"".join([c for c in normalized if not unicodedata.combining(c)]) | |
print(new_city) # Ceske Budejovice |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def clear_text(func): | |
"""Remove non visible characters (\\n, \\r, \\t) | |
Example: | |
>>> @clear_text | |
>>> def foo(): | |
... return "blah \\n blah" | |
... | |
>>> foo() | |
... blah blah |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
map $http_upgrade $connection_upgrade { | |
default upgrade; | |
'' close; | |
} | |
upstream backend { | |
server 127.0.0.1:3000 fail_timeout=0; | |
} | |
upstream streaming { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# Mastodon v4.3.x docker character limit change | |
# Does *NOT* word with >= 4.3.4 as the handling was changed | |
# Based on https://mpp-service.de/en/mastodon-character-limit-increase-in-a-docker-environment/ | |
# Based on https://fouquet.me/2024/10/10/mastodon-4-3-char-limit | |
# ./postLimit.sh 2500 → change post limit to 2500 | |
# Must be executed in the directory Mastodons docker-compose.yaml is located | |
set -e |