Skip to content

Instantly share code, notes, and snippets.

View thanakijwanavit's full-sized avatar

Nic Wanavit thanakijwanavit

View GitHub Profile
@thanakijwanavit
thanakijwanavit / findip
Last active January 11, 2020 14:53
a script to get ip address
#!/bin/bash
curl ipinfo.io/ip
@thanakijwanavit
thanakijwanavit / selenium_in_colab.py
Last active August 13, 2022 03:10
how to run a headless selenium process in colab
#installation
!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
@thanakijwanavit
thanakijwanavit / selenium.py
Last active April 21, 2020 04:41 — forked from korakot/selenium.py
Use selenium in Colab
# install chromium, its driver, and selenium
!apt install chromium-chromedriver
!pip install selenium
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
# set options to be headless, ..
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
@thanakijwanavit
thanakijwanavit / lambda_selenium_template.yaml
Created April 22, 2020 05:23
Deploy serverless for selenium
#https://github.com/smithclay/lambdium/blob/master/template.yaml
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: selenium with headless chromium
Resources:
DeploymentPermission:
Type: "AWS::Lambda::LayerVersionPermission"
Properties:
Action: lambda:GetLayerVersion
LayerVersionArn: !Ref ChromiumLayer
# from avail.aws.cloudsearch import get_cloudsearch_endpoint_from_settings
import time
import json
def splitList(l,n):
x = [l[i:i + n] for i in range(0, len(l), n)]
return x
# get all items from dynamodb
import boto3
@thanakijwanavit
thanakijwanavit / splitList.py
Created April 28, 2020 09:27
split list into sublists
def splitList(l,n):
x = [l[i:i + n] for i in range(0, len(l), n)]
return x
splitList(l,n)
@thanakijwanavit
thanakijwanavit / trigger_every_hour.yml
Created April 29, 2020 00:47
sam template to add a trigger event every hour
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
NotificationEmail:
Type: String
Resources:
CheckWebsitePeriodically:
Type: AWS::Serverless::Function
Properties:
Handler: LambdaFunctionOverHttps.handler
@thanakijwanavit
thanakijwanavit / pd_to_dynamodict.py
Created April 29, 2020 04:44
get dict from pandas dataframe for ingestion by dynamodb
myl=df.T.to_dict().values()
@thanakijwanavit
thanakijwanavit / invokerLambda.py
Created April 29, 2020 18:02
invoke lambda function
import boto3
lambda_ = boto3.client('lambda',
aws_access_key_id = MY_ACCESS_KEY_ID,
aws_secret_access_key = MY_SECRET_ACCESS_KEY,
region_name = 'us-east-1' )
response = lambda_.invoke(
FunctionName='arn:aws:lambda:us-east-1:xxxxxx',
InvocationType='RequestResponse',
LogType='Tail',
Payload= json.dumps({'test': 'testPayload'})
@thanakijwanavit
thanakijwanavit / pynamodbModel.py
Last active July 14, 2021 09:22
pynamodb model for uploading to dynamodb
!pip install --quiet pynamodb
from pynamodb.models import Model
from pynamodb.attributes import UnicodeAttribute, JSONAttribute
class PynamoStocks(Model):
class Meta:
# aws_access_key_id = MY_ACCESS_KEY_ID
# aws_secret_access_key = MY_SECRET_ACCESS_KEY
table_name = 'tops-inv-backlog-2'
region = 'us-east-1'
billing_mode='PAY_PER_REQUEST'