python tips and tricks #python #study #tips

certificates.md

Motivation

When connecting to the internet we could be doing that :

1. to install new packages/modules, or
2. to make requests within the application, using requests module.

By either of them we could get SSLCertVerficationError, but each requires a different solution.

PIP Trust

For the first scenario, using pip to install new modules, one could configure the .ini file and add the following lines

[global]
cert = C:\CA\my-ca-cert.crt
       C:\CA\another-ca-cert.crt

trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org

Application Trust

I have tried the python-certifi-win32, which was not going smooth as it injected itself to all the calls, and when it made python, I wasn't even able to uninstall it except when I commented out the initialization in the module's .pth file

I also tried pip-system-certs, which simply didn't work, but I think that's related to being only working for the first scenario, AKA: pip.

The solution was to add a new environment variable REQUESTS_CA_BUNDLE and set it to the .crt file that contains all the CA certificates, that I need requests to consider alongside with certifi's certificates.

config.md

Python system configuration notes.

Site Packages

Configuration of site packages are found in the following paths, for example for python v3.10:

• Windows: C:\Users\$USER\AppData\Roaming\Python\Python310\site-packages
• Linux: TBD

Note: When a virtualenv is used, the venv\Lib\site-packages is the correct location

Config File

Could be found in the docs for pip

Windows

• Global: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
• User: %APPDATA%\pip\pip.ini
• Site: %VIRTUAL_ENV%\pip.ini

Linux

• Global: /etc/xdg/pip/pip.conf
• User: $HOME/.config/pip/pip.conf
• Site: $VIRTUAL_ENV/pip.conf

debugging.md

Production

After installing a package, one may need to debug it while running in production, one way would include editing the files in the production environment directly. To do so, one shall reach the installation dir of the packages.
Run the following commands, to figure that out

python
import blinkstick
print(blinkstick.__file__)

Reference: SO comment

Inspection

Available Modules

• Get all imported modules at some point Useful in detecting if caching happened

import sys
print(sys.modules.keys())    # Only the module names
print(sys.modules.values())  # Further module info, mainly the path of the pyc being in use
print(sys.modules['alpha'])  # <-- example

One could use the output values to reload the modules

• Get all imported modules for the current module only

import types
def imports():
    for name, val in globals().items():
        if isinstance(val, types.ModuleType):
            yield val.__name__

This won't return local imports, or non-module imports like from x import y. Note that this returns val.name so you get the original module name if you used import module as alias; yield name instead if you want the alias. Reference: SOA

Module Content

• help(MODULE_NAME): Get an interative help of a module run .
• dir(MODULE_NAME): Get all members (functions and variables) of a module.
• for i in dir(module): print i: Get all members (1 per line)
• Get all the functions of a module

from inspect import getmembers, isfunction

from somemodule import foo
print(getmembers(foo, isfunction))

• Use ast to parse a module without importing it.

dicts_vs_namespaces.md

I resorted eventually to using dictionaries instead of namespaces for clarity,
for example, I had to make this decision when I wanted to use variables either coming from:

• args (the script is being called as the __main__ file)
• or settings (the script is called from another script)

However, in case it was ever needed to use namespaces instead following is my tip

Python's native Simple namespaces are not recursive, so using them would be limiting, thus one would need to create his own recursive namespace.
Using the following snippet from TaqKarim#Extending SimpleNamespace for Nested Dictionaries

class RecursiveNamespace2: # without extending SimpleNamespace!

  @staticmethod
  def map_entry(entry):
    if isinstance(entry, dict):
      return RecursiveNamespace(**entry)

    return entry

  def __init__(self, **kwargs):
    for key, val in kwargs.items():
      if type(val) == dict:
        setattr(self, key, RecursiveNamespace(**val))
      elif type(val) == list:
        setattr(self, key, list(map(self.map_entry, val)))
      else: # this is the only addition
        setattr(self, key, val)

dynamic_class.py

# This code generates a python class named tmpika which has
# the method OnStart, and objects OnStarts, which ideally would be a list of handlers
# reference: https://www.geeksforgeeks.org/create-classes-dynamically-in-python/

@classmethod
def OnStartTemplate(cls):
  for onStart in cls.OnStarts:
    print(onStart)

index = "001"

generatedClass = type(f"myclass{index}", (object, ), {
    "OnStarts": [],
    "OnStart": OnStartTemplate
  })

print(generatedClass)
a = generatedClass()
a.OnStarts.append("hello")
a.OnStarts.append("hello2")
a.OnStart()

hints.md

• To install a library in an offline way, download the wheel file first using pip download PACKAGE_NAME then install it using pip install c://path/to/file.wheel

lib_injection.md

python-certifi-win32 is an interesting package as it injects itself into the external calls just before certifi to ensure that all our external requests are passed through it.
that occurs by creating a .pth file in the site-packages for example in env/lib/site-packages which adds it as a module before every call.
The packages would get imported from the site, user or global packages afterwords.

Notes:

• If you ever installed python-certifi-win32 and got stuck by an issue in it like me, comment the line in python-certifi-win32.init.pth then uninstall it.
• python-certifi-win32 is not maintained anymore, in favor of the cross platform pip-system-certs

pip.md

Indexes

• To list all indexes use python -m pip config list
• Typical content of a pypirc file
• If you got the error unsupported schema pypi setup.py, check whether you have the .pypirc file in the hom dir or not
• Uploading to pypi

python_tips.md

Environment setup

• Repl.it: an online compiler, that works for many languages.

Project template

• setting up different python environments gist: an intro to start off for python module development.
• python3_module_template: an amazing project template by xavierdupre which is both advanced and has everything preset for us.

Testing

• PyQuickHelper: a useful assistant for testing.

Fill an instance list with the same object

instancelist = [ MyClass() for i in range(29)]

Arguments

• To understand args and kwargs go through this article.
• To properly use the kwargs with default values check this SOA, the comments are helpful too.