Helper Python Functions related to CI/CD and production. In this post, I will outline and provide links to various sources and guides that will help working with Python in production.
-
Suppose you have curl formatted file
curl 'http://fiddle.jshell.net/echo/html/' -H 'Origin: http://fiddle.jshell.net' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: */*' -H 'Referer: http://fiddle.jshell.net/_display/' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --data 'msg1=wow&msg2=such' --compressed
-
You can convert it using the following website. The result will look like:
import requests headers = { 'Accept': '*/*', 'Referer': 'http://fiddle.jshell.net/_display/', } data = [ ('msg1', 'wow'), ('msg2', 'such'), ] response = requests.post('http://fiddle.jshell.net/echo/html/', headers=headers, data=data)
Link: source_1
-
Instead of typing sensitive information in .py file, you can hide your credentials in Yaml file
-
Here is how .yml file looks like:
key1: key1_1: value1_1 key1_2: value1_2
-
How to import in .py file
import yaml yaml_file = yaml.load(open('<location of .yaml file>')) value1_1_reference = yaml_file['key1']['key1_1']
Link: source_1
-
To set up Python environment variables from .py file
import os os.environ["name_of_varible"] = "variable_value"
- To reference variable
import os var_to_store = os.environ["name_of_variable"]
Suppose you have dictionary defined as: data = {'key':'value', 'key2':'value2'}
and you want to pass it to function processing_dictionary()
where function is defined as:
```python
def main_calling():
data = {'key':'value', 'key2':'value2'}
processing_dictionary(**data)
def processing_dictionary(data):
print(data)
```
The snippet above illustrates that func(**data)
is equivalent to func(key1='value1', key2='value2')
Link: source_1
- Python Doc that with guide on how to handle Python Requests
- How to make Python Requests quickguide
- Example
import requests import json data = {'key':'value'} r = requests.get('http://httpbin.org/get', params=data) print(r.json())
-
Great article with Python code examples on why testing is important.
-
Resource and examples on how unittest test fixture functions and methods are written
-
Python Unit Test CheatSheet
-
How to run multiple/all .py unitTest files: link
-
SetUp and tearDown
- If in your test case functions you are creating/initating variables and/or objects repetitively, you can create 2 functions: setUp and tearDown
- For more information refer to Python Doc
- Example used from Python Doc:
import unittest class WidgetTestCase(unittest.TestCase): def setUp(self): self.widget = Widget('The widget') def tearDown(self): self.widget.dispose() self.widget = None def test_default_size(self): self.assertEqual(self.widget.size(), (50,50), 'incorrect default size') def test_resize(self): self.widget.resize(100,150) self.assertEqual(self.widget.size(), (100,150), 'wrong size after resize')
-
SetUpClass
- Suppose you have
- Example below illustrates how to reference variables created in setUpClass method in Unittest:
import unittest import sys import mymodule class BookTests(unittest.TestCase): @classmethod def setUpClass(cls): cls._mine = mymodule.myclass('<my variables>') def test_something(self): self.assertEqual(self._mine.attribute, 'myAttribute')
Link: source_1
- Framework Nose that allows you to mock test API calls
- In other words, if you need to unitTest your API calls, you can use this framework
- Here are resources I found very useful and easy to understand how to use Python Mocking:
- This blog was the best resource in explaining in simple terms what mocking is and how it acts
- Mocking In Python Part I
- Blog post that provides excellent examples on how to use Mock libary
- Another blog post on Medium that explains how to use Mock Library
- Cheatseet on how to use Mock Library
- JSON Placeholder website in case you need to make API calls and experiment with JSON format
Below I provided some sample code for reference
# Assume we are trying to test function -> def function_to_test <-
# and we are writing unittest function -> def test_function_to_test -<
# Also, assume that function_to_test calls method_name_1 and method_name_2 from within
# In this case, we want to test function_to_test and simulate method_name_1 and _2
# Below is how to use Patch in order to simulate them:
@patch('package_name.module_name.class_name.method_name_1')
@patch('package_name.module_name.class_name.method_name_2')
def test_function_to_test(self, mockObject_1, mockObject_2)
dict_1 = {'k1': 'v1'}
list_2 = [1, 2, 3, 4, 5]
# mockObject_1 and _2 are objects where you can specify return values that -- ideally should-- be returned
mockObject_1.return_value = dict_1
mockObject_2.return_value = list_2
...some test manipulation goes here...
try:
result = function_to_test(param_1, param_2, ...)
self.assertTrue(result)
except ValueError:
print("Wrong Value")
-
Example:
- Place all the unittesting codes into 1 folder. Suppose we name our folder
/test
. Then we have:
/test -- test_code_1.py -- test_code_2.py -- test_code_3.py
- Navigate to
/test
folder and type in Terminal
$ coverage.py run -m unittest discover $ coverage report -m
- Result will be a table which will show you how much code you have tested (covered)
- Place all the unittesting codes into 1 folder. Suppose we name our folder
-
If you want to test all 'test_*.py' files, type following command:
$ coverage run -m unittest discover
-
If you want to test selected 'test_*.py' files, type:
$ coverage run --source=folder_1/,folder_2/,folder_3/ -m unittest discover
-
StackOverflow post on how to run unittest in case you have nested directories
- List of resources to learn PostgreSQL for beginners
- PostgreSQL Tutorial for quick look up
- TutorialsPoint guide for quick references
- Tutorial on how connect to PostgreSQL remotely
- Some useful commands. Type in Terminal:
Enter your credentials. Once you are logged in$ <path_to_PostGreSQL>/runpsql.sh; exit
<your_db_name>=# <your_db_name>=# \l -> lists all the DBs <your_db_name>=# \dt -> lists all the tables in the DB selected <your_db_name>=# \c <DB name> -> connect to DB <your_db_name>=# \d+ <tablename> -> list all the columns in the table
- Tutorials on Python + Django
- Tutorial by agatha-codes on Medium
- TutorialsPoint guide
- Building Django App by Amos Omondi
-
ORM layer can be used to to interact with application data from various relational databases such as SQLite, PostgreSQL and MySQL without having to type SQL Queries.
- In other words, instead of typing
select * from database_table where database_column = "matching_value"
You can refer todatabase_table
as a Python object
- In other words, instead of typing
-
Here is an article that discusses what ORM is, the benefits, and what frameworks you can use
- Frameworks:
- Django
- Django Resources: fullstackpython, opensource, djangogirls (most comprehensive one, which includes intro to Python), article by Prateek, real python
- Django textbook
- How to create models: github post by ggcarrots, digitalocean
- How make queries using Django tutorial
- Handling relationships in ORM: one-to-one, many-to-one
- How to connect Django and PostgreSQL in Python source code
- Peewee
- Peewee documentation
- How to create models and fields for Peewee doc
- How to do queries in Peewee guide
- Example how to use Peewee
- Django
- Frameworks:
To perform various operations on your DB through Terminal you can do following commands:
$ python manage.py shell
>>>> from yourDjangoApp.models import *
# To create objects
>>>> <yourTable>.objects.create(<key1>='value1', <key2>='value2'...)
# To list all objects in the table. In SQL equivalent to: select * from yourTable;
>>>> <yourTable>.objects.all()
#To list all objects and filter. In SQL equivalent to: select * from yourTable where column_name==value;
<yourTable>.objects.filter(<column_name>==<value>
To create objects with foreignKey
$ python manage.py shell
>>>> from <yourDjangoApp>.models import *
>>> p = <yourTable>(title='Sample title', sampleId=1234, content='Test')
>>> p.save()
>>> <yourNewTable>.objects.create(sampleId=p, key2="34", key3="56")
Every time you make changes to model.py
file, you have migrate
and makemigrations
as follows:
$ python manage.py makemigrations <yourDjangoApp>
$ python manage.py migrate <yourDjangoApp>
- Introductory guides in various blogs on git: GitLab Documentation, meghan nelson, ariel camus
- I wrote a cheatsheet on Github with various Git commands
- Git Documentation on GitLab
- Why there is
git add
andgit commit
operations article
-
Create empty file and name it
.gitignore
in the project directory.$ cd project_dir $ nano .gitignore
-
Now you add files or folder you want git to ignore when adding or committing changes.
- To add folders, type:
name_of_folder/
- To add file, type:
name_of_file.file_extension
- Additionally, you can add sign (*) anywhere in order to make it more generic. For example, suppose I want to ignore all files that have .pyc extension. I will type
*.pyc
- Additionally, you can add sign (*) anywhere in order to make it more generic. For example, suppose I want to ignore all files that have .pyc extension. I will type
- Each folder or file goes on a new line
- To add folders, type:
- Introduction to Pipelining and Jobs on GitLab doc
- Set up gitlab-ci.yml file to run your codes on gitlab CI/CD server
- You can use GitLab API + Python
- How to work with YAML file in GitLab doc
- Continuous Integration and Development within Gitlab article
-
Comprehensive guide on how to use Docker and Python
-
Install Docker
-
Create
root_folder
$ mkdir root_folder
-
In order to install all the dependencies from your virtual environment to Docker container export by typing in terminal:
$ cd root_folder $ pip freeze > requirements.txt
-
Prepare Dockerfile
- Create Dockerfile
$ cd root_folder $ nano Dockerfile
- In the Dockerfile paste:
FROM python:3 COPY . /root_folder WORKDIR /root_folder RUN pip install -r requirements.txt
-
Here is how folder hierarchy will look like:
/root_folder -- requirements.txt -- Dockerfile
-
Open Terminal and in directory
/root_folder
type (don't forget the dot):$ docker build .
-
Pushing and Pulling from Docker Hub:
- Create profile to create user on Docker Hub
$ docker login <enter your credentials info> $ docker images returns (new image with <none> tags and imageID) $ docker tag <imageID> <yourUserName/package> $ docker push <yourUserName/package>
Link: source_1
-
Create
docker-compose.yml
file in your directory -
In the docker-compose.yml file:
version: '3' services: postgres: image: postgres:11-alpine environment: - POSTGRES_USER=<postgresUser> - POSTGRES_PASSWORD=<yourPassword> ports: - '<localhost port like 8080>:<postgres port: like 5432>' <app name>: build: ./ environment: - POSTGRES_DB=<postgresDatabase> - POSTGRES_USER=<postgresUser> - POSTGRES_PASSWORD=<yourPassword> - POSTGRES_HOST=postgres
-
Note on
POSTGRES_HOST
environment variable: make sure the name of thePOSTGRES_HOST
is the same as database you are setting up (i.e.postgres
right under services in the above provided code snipped) -
To run
$ docker-compose up
-
To kill and clean processes and images:
$ docker-compose down $ docker rm $(docker ps -a -q) $ docker rmi $(docker images -q)
-
Additional info can be found in these resources: Docker Compose For Python, official Docker Hub PostgreSQL Documentation
- Gitlab Documentation about Docker
- Articles on how to run Docker Image on Gitlab: medium, freeCodeCamp
- Create Docker container and uplod it to Docker Hub. I assume that docker image on Hub is set to public
- Refer back to gitlab-ci.yml file and change
image: python-alpine
toimage: <yourUserName/package>