tdmalone

#!/usr/bin/env python

import json, sys
from urllib.request import urlopen


def green(str):  return f'\033[92m{str}\033[0m'
def orange(str): return f'\033[38;5;214m{str}\033[0m'
def red(str):    return f'\033[91m{str}\033[0m'
def yellow(str): return f'\033[93m{str}\033[0m'

justmarkham

MaxHalford

ustayready

import openai
import boto3
import json
import time
from typing import Dict, List

openai.api_key = '### SET YOUR OPENAPI API KEY HERE ###'
session = boto3.session.Session()
client = session.client('iam')

akhan619

Exploring Tokenizers from Hugging Face

Hugging Face (HF) has made NLP (Natural Language Processing) a breeze. In this post, we are going to take a look at tokenization using a hands on approach with the help of the Tokenizers library. We are going to load a real world dataset containing 10-K filings of public firms and see how to train a tokenizer from scratch based on the BERT tokenization scheme. In the process we will understand tokenization in detail and some gotchas to keep an eye out for.

Background on NLP (Optional)

If you already have an understanding of the NLP pipeline, you can safely skip this section.

For any NLP task, one of the first steps is pre-processing the data so that it can be fed into our NLP models. For those new to NLP, the general pipeline for any NLP task (text classification, question answering, etc.) is as follows:

rasbt

# Sebastian Raschka 09/24/2022
# Create a new conda environment and packages
#   conda create -n whisper python=3.9
#   conda activate whisper
#   conda install mlxtend -c conda-forge

# Install ffmpeg
# macOS & homebrew
#   brew install ffmpeg
# Ubuntu

codewithbas

import json
from TwitterAPI import (
    TwitterAPI,
    TwitterPager
)

consumer_key        = "<YOUR API KEY>"
consumer_secret     = "<YOUR API KEY SECRET>"
access_token        = "<YOUR ACCESS TOKEN>"
access_token_secret = "<YOUR ACCESS TOKEN SECRET>"

NewscatcherAPI

summary = [article['summary'] for article in articles]
sentence = summary[0]

ParthTrambadiya

import logging
import base64
import boto3
import os

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3_client = boto3.client('s3')

aquilax

<a href="javascript:(function(){Array.from(document.querySelectorAll('textarea')).map(function(b){var a=document.createElement('div');var d=document.createElement('button');d.textContent='↑';d.addEventListener('click',function(f){f.preventDefault();b.value=Array.from(new Set(b.value.split('\n'))).sort().join('\n')});var c=document.createElement('button');c.textContent='↓';c.addEventListener('click',function(f){f.preventDefault();b.value=Array.from(new Set(b.value.split('\n'))).sort().reverse().join('\n')});a.appendChild(d);a.appendChild(c);b.parentNode.insertBefore(a,b)})})();">Sort textarea unique</a>