AlmasM/Python_Development_Resources.md

## Python_Development_Resources.md

      
    Raw
  

              Python_Development_Resources.md
            
          
    Python In Production

Helper Python Functions related to CI/CD and production. In this post, I will outline and provide links to various sources and guides that will help working with Python in production.
Table of Content


Supplimentary Production Related Functions


Converting from CURL to Python Formatted Syntax
Navigating YAML Formatted File
*Args and **Kwargs in Python
Python and API Calls


Unit Tests


Unit Testing in Python and Exceptions
Using Frameworks for Unit Testing
Mocking API Calls and Mocking Python Functions for Unit Testing
Coverage of Python Code


PostgreSQL and ORMs


PostgreSQL
Object Relational Mapping (ORM)
Django Commands in Terminal


Git, GitLab and GitHub


How to Use Git
GitIgnore File And How to Use It
GitLab CI/CD
Running Gitlab CI with Docker


Docker


Docker
Docker docker-compose.yaml


Converting from CURL to Python Formatted Syntax


Suppose you have curl formatted file
curl 'http://fiddle.jshell.net/echo/html/' 
-H 'Origin: http://fiddle.jshell.net' 
-H 'Accept-Encoding: gzip, deflate' 
-H 'Accept-Language: en-US,en;q=0.8' 
-H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: */*' -H 'Referer: http://fiddle.jshell.net/_display/' 
-H 'X-Requested-With: XMLHttpRequest' 
-H 'Connection: keep-alive' --data 'msg1=wow&msg2=such' --compressed


You can convert it using the following website. The result will look like:
import requests

headers = {
    'Accept': '*/*',
    'Referer': 'http://fiddle.jshell.net/_display/',
}

data = [
  ('msg1', 'wow'),
  ('msg2', 'such'),
]

response = requests.post('http://fiddle.jshell.net/echo/html/', headers=headers, data=data)


Link: source_1

Navigating YAML Formatted File


Instead of typing sensitive information in .py file, you can hide your credentials in Yaml file


Here is how .yml file looks like:
key1:
    key1_1: value1_1
    key1_2: value1_2


How to import in .py file
import yaml
yaml_file = yaml.load(open('<location of .yaml file>'))
value1_1_reference = yaml_file['key1']['key1_1']


Link: source_1

Set up Python Environment Variables


To set up Python environment variables from .py file
import os
os.environ["name_of_varible"] = "variable_value"

To reference variable

import os
var_to_store = os.environ["name_of_variable"]


Links source_1, source_2

*Args and **Kwargs in Python

Suppose you have dictionary defined as: data = {'key':'value', 'key2':'value2'} and you want to pass it to function processing_dictionary() where function is defined as:
```python 
def main_calling():
    data = {'key':'value', 'key2':'value2'}
    processing_dictionary(**data)

def processing_dictionary(data):
    print(data)
```

The snippet above illustrates that func(**data) is equivalent to func(key1='value1', key2='value2')
Link: source_1

Python and API Calls


Python Doc that with guide on how to handle Python Requests
How to make Python Requests quickguide
Example
import requests
import json
data = {'key':'value'}
r = requests.get('http://httpbin.org/get', params=data)
print(r.json())


Unit Testing in Python and Exceptions


Great article with Python code examples on why testing is important.


List of Standard Exceptions


List of Standard Assertions


List of Different Exception Handling Types


Resource and examples on how unittest test fixture functions and methods are written


Python Unit Test CheatSheet


How to run multiple/all .py unitTest files: link


SetUp and tearDown

If in your test case functions you are creating/initating variables and/or objects repetitively, you can create 2 functions: setUp and tearDown
For more information refer to Python Doc
Example used from Python Doc:
import unittest

class WidgetTestCase(unittest.TestCase):
   def setUp(self):
       self.widget = Widget('The widget')

   def tearDown(self):
       self.widget.dispose()
       self.widget = None

   def test_default_size(self):
       self.assertEqual(self.widget.size(), (50,50),
                        'incorrect default size')

   def test_resize(self):
       self.widget.resize(100,150)
       self.assertEqual(self.widget.size(), (100,150),
                        'wrong size after resize')


SetUpClass

Suppose you have
Example below illustrates how to reference variables created in setUpClass method in Unittest:

import unittest  
import sys  
import mymodule  

class BookTests(unittest.TestCase):  
    @classmethod  
    def setUpClass(cls):  
        cls._mine = mymodule.myclass('<my variables>') 

    def test_something(self):
       self.assertEqual(self._mine.attribute, 'myAttribute')


Link: source_1

Using Frameworks for Unit Testing


Framework Nose that allows you to mock test API calls

In other words, if you need to unitTest your API calls, you can use this framework


Mocking API Calls and Mocking Python Functions for Unit Testing


Here are resources I found very useful and easy to understand how to use Python Mocking:

This blog was the best resource in explaining in simple terms what mocking is and how it acts
Mocking In Python Part I
Blog post that provides excellent examples on how to use Mock libary
Another blog post on Medium that explains how to use Mock Library
Cheatseet on how to use Mock Library


JSON Placeholder website in case you need to make API calls and experiment with JSON format

Below I provided some sample code for reference
# Assume we are trying to test function -> def function_to_test <- 
# and we are writing unittest function -> def test_function_to_test -<
# Also, assume that function_to_test calls method_name_1 and method_name_2 from within 
# In this case, we want to test function_to_test and simulate method_name_1 and _2
# Below is how to use Patch in order to simulate them:

@patch('package_name.module_name.class_name.method_name_1')
@patch('package_name.module_name.class_name.method_name_2')
def test_function_to_test(self, mockObject_1, mockObject_2)

  dict_1 = {'k1': 'v1'}
  list_2 = [1, 2, 3, 4, 5]
  # mockObject_1 and _2 are objects where you can specify return values that -- ideally should-- be returned
  mockObject_1.return_value = dict_1
  mockObject_2.return_value = list_2
  
  ...some test manipulation goes here...
  
  try:
      result = function_to_test(param_1, param_2, ...)
      
      self.assertTrue(result)
  except ValueError:
      print("Wrong Value")
  

Coverage of Python Code


Coverage Doc


Example:

Place all the unittesting codes into 1 folder. Suppose we name our folder /test. Then we have:

/test
  -- test_code_1.py
  -- test_code_2.py
  -- test_code_3.py


Navigate to /test folder and type in Terminal

$ coverage.py run -m unittest discover
$ coverage report -m


Result will be a table which will show you how much code you have tested (covered)


If you want to test all 'test_*.py' files, type following command:
 $ coverage run -m unittest discover


If you want to test selected 'test_*.py' files, type:
$ coverage run --source=folder_1/,folder_2/,folder_3/ -m unittest discover


StackOverflow post on how to run unittest in case you have nested directories


PostgreSQL


List of resources to learn PostgreSQL for beginners
PostgreSQL Tutorial for quick look up
TutorialsPoint guide for quick references
Tutorial on how connect to PostgreSQL remotely
Some useful commands. Type in Terminal:
$ <path_to_PostGreSQL>/runpsql.sh; exit

Enter your credentials. Once you are logged in
<your_db_name>=#
<your_db_name>=# \l -> lists all the DBs
<your_db_name>=# \dt -> lists all the tables in the DB selected
<your_db_name>=# \c <DB name> -> connect to DB 
<your_db_name>=# \d+ <tablename> -> list all the columns in the table


Tutorials on Python + Django

Tutorial by agatha-codes on Medium
TutorialsPoint guide
Building Django App by Amos Omondi


Object Relational Mapping (ORM)


ORM layer can be used to to interact with application data from various relational databases such as SQLite, PostgreSQL and MySQL without having to type SQL Queries.

In other words, instead of typing  select * from database_table where database_column = "matching_value" You can refer to database_table as a Python object


Here is an article that discusses what ORM is, the benefits, and what frameworks you can use

Frameworks:

Django

Django Resources: fullstackpython, opensource, djangogirls (most comprehensive one, which includes intro to Python), article by Prateek, real python
Django textbook
How to create models: github post by ggcarrots, digitalocean
How make queries using Django tutorial
Handling relationships in ORM: one-to-one, many-to-one
How to connect Django and PostgreSQL in Python source code


Peewee

Peewee documentation
How to create models and fields for Peewee doc
How to do queries in Peewee guide
Example how to use Peewee


Django Commands in Terminal

To perform various operations on your DB through Terminal you can do following commands:
$ python manage.py shell
>>>> from yourDjangoApp.models import *

# To create objects
>>>> <yourTable>.objects.create(<key1>='value1', <key2>='value2'...)

# To list all objects in the table. In SQL equivalent to: select * from yourTable;
>>>> <yourTable>.objects.all() 

#To list all objects and filter. In SQL equivalent to: select * from yourTable where column_name==value;
<yourTable>.objects.filter(<column_name>==<value>

To create objects with foreignKey
$ python manage.py shell
>>>> from <yourDjangoApp>.models import *
>>> p = <yourTable>(title='Sample title', sampleId=1234, content='Test')
>>> p.save()
>>> <yourNewTable>.objects.create(sampleId=p, key2="34", key3="56")

Every time you make changes to model.py file, you have migrate and makemigrations as follows:
$ python manage.py makemigrations <yourDjangoApp>
$ python manage.py migrate <yourDjangoApp>


How to Use Git


Introductory guides in various blogs on git: GitLab Documentation, meghan nelson, ariel camus
I wrote a cheatsheet on Github with various Git commands
Git Documentation on GitLab
Why there is git add and git commit operations article


GitIgnore File And How to Use It


Create empty file and name it .gitignore in the project directory.
$ cd project_dir
$ nano .gitignore


Now you add files or folder you want git to ignore when adding or committing changes.

To add folders, type: name_of_folder/
To add file, type: name_of_file.file_extension

Additionally, you can add sign (*) anywhere in order to make it more generic. For example, suppose I want to ignore all files that have .pyc extension. I will type *.pyc


Each folder or file goes on a new line


GitLab CI/CD


Introduction to Pipelining and Jobs on GitLab doc
Set up gitlab-ci.yml file to run your codes on gitlab CI/CD server

Read this tutorial, which explains how gitlab-ci.yml file operates and how to structure your files
Set up gitLab CI/CD on Ubuntu guide


You can use GitLab API + Python
How to work with YAML file in GitLab doc

Simple YAML file example code
Python Example + YAML file code


Continuous Integration and Development within Gitlab article


Docker


Comprehensive guide on how to use Docker and Python


Install Docker


Create root_folder
$ mkdir root_folder


In order to install all the dependencies from your virtual environment to Docker container export by typing in terminal:
$ cd root_folder
$ pip freeze > requirements.txt


Prepare Dockerfile

Create Dockerfile

 $ cd root_folder
 $ nano Dockerfile


In the Dockerfile paste:

 FROM python:3
 COPY . /root_folder
 WORKDIR /root_folder
 RUN pip install -r requirements.txt


Here is how folder hierarchy will look like:
/root_folder
 -- requirements.txt
 -- Dockerfile


Open Terminal and in directory /root_folder type (don't forget the dot):
$ docker build .


Pushing and Pulling from Docker Hub:

Create profile to create user on Docker Hub

$ docker login 
    <enter your credentials info>  
$ docker images
    returns (new image with <none> tags and imageID)
$ docker tag <imageID> <yourUserName/package>
$ docker push <yourUserName/package>


Link: source_1
Docker: docker-compose.yml


Create docker-compose.yml file in your directory


In the docker-compose.yml file:
version: '3'

services:
  postgres:
    image: postgres:11-alpine
    environment:
      - POSTGRES_USER=<postgresUser>
      - POSTGRES_PASSWORD=<yourPassword>
    ports:
      - '<localhost port like 8080>:<postgres port: like 5432>'
  <app name>:
    build: ./
    environment:
      - POSTGRES_DB=<postgresDatabase>
      - POSTGRES_USER=<postgresUser>
      - POSTGRES_PASSWORD=<yourPassword>
      - POSTGRES_HOST=postgres


Note on POSTGRES_HOST environment variable: make sure the name of the POSTGRES_HOST is the same as database you are setting up (i.e. postgres right under services in the above provided code snipped)


To run $ docker-compose up


To kill and clean processes and images:
$ docker-compose down
$ docker rm $(docker ps -a -q)
$ docker rmi $(docker images -q)


Additional info can be found in these resources: Docker Compose For Python, official Docker Hub PostgreSQL Documentation


Running Gitlab CI with Docker


Gitlab Documentation about Docker
Articles on how to run Docker Image on Gitlab: medium, freeCodeCamp
Create Docker container and uplod it to Docker Hub. I assume that docker image on Hub is set to public
Refer back to gitlab-ci.yml file and change image: python-alpine to image: <yourUserName/package>

Unsorted


How to promote GitHub Project
DevOps Roadmap

REST + GRAPHQL


REST vs GraphQL
Intro to GraphQL
REST + Flask