Skip to content
On this page


Python is a good language for writing software. It encourages clean, readable code. It's a nice interface to the underlying system.

A great choice for synchronous workflows. Synchronous workflows can help processes on the system side. For example, walking a file system and reporting when the job finishes.

Python doesn't run client-side on web browsers. For that you'll need Javascript (JS). Python shows up frequently on the API side or server side.

New Project

Best practices for starting a python project

Files and Directories that should be included:

I like starting with a template script to include things I like when getting started. A boilerplate.

#!/usr/bin/env python3
# By: 
# On: [date]
# License: MIT 

Usage: [source] [destination]

import os
import sys

def usage():

def read_file(source):
    with open(source, 'r') as f:
        for line in f.readlines():

def merge_logs(source, destination):
    if source and os.path.exists(source):
        options = os.listdir(source)
        print("Couldn't find source path: %s" % source)

if __name__ == '__main__':
    source = None
    destination = None
    if len(sys.argv) > 1:
        helps = ['--help', 'help', '-h']
        for i in helps:
            if i in sys.argv:
        source = sys.argv[1]
        if len(sys.argv) > 2:
            destination = sys.argv[2]

    merge_logs(source, destination)

Walking Directories

def walk_files(source, destination):
    if os.path.exists(source):
        for root, dirs, files in os.walk(source):
            for f in files:
                if'\?', f):
                    sname = os.path.join(root, f)
        print("Couldn't find path: %s" % source)


Python 3.6 introduced a handy new syntax for substituting variables in a string. Prefix with an f and use curly brackets to designate the variables in the string:

name = "human"
print(f"Hello {name}")


Use the built in library:

import json


def read_metadata(source):
    items = []
    if pathlib.Path(source).is_file():
        with open(source, 'r+') as f:
            items = json.load(f)
    return items


def write_metadata(items, destination):
    with open(destination, "w") as f:
        json.dump(items, f, indent=2, default=str)


CSV Files


OS modules

Sleep / Wait

from time import sleep 


Use UTC in the system to simplify

from datetime import datetime,timezone
now_utc =

Then, to print it in a format understood in JS:




Sooner or later it's a good idea to keep track of what was done when
and ensure that those activities are written no matter what happens in the code (e.g. a crash)

enter logging
Logging HOWTO — Python 3.10.4 documentation

import logging
import os
log_dest = os.path.join(metadata_dir, "example.log")
logging.basicConfig(filename=log_dest, level=logging.DEBUG, format='%(asctime)s: %(levelname)-8s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

logging.debug('This message should go to the log file')'So should this')
logging.warning('And this, too')
logging.error('And non-ASCII stuff, too, like Øresund and Malmö')

Initialize your logging instance in the main script to control the destination.

If all settings should be the same across an application, use a common library that gets imported to other scripts like or

Multiple Logs


    # log results from each validation separately for easier tracking
    LOG_FORMAT = ('%(asctime)s: %(levelname)-8s: %(message)s')
    LOG_LEVEL = logging.DEBUG

    tar_file = os.path.basename(tar_path)

    now_utc =
    destination_name = tar_file.replace('.tar', '.validate.')
    destination_name += now_utc.strftime("%Y%m%d")
    log_destination = os.path.join(
        metadata_dir, snapshot, destination_name)

    print("Logging to:", log_destination)

    validate_logger = logging.getLogger("validate.details")
    validate_logger_file_handler = logging.FileHandler(log_destination)
        logging.Formatter(LOG_FORMAT, datefmt='%Y-%m-%d %H:%M:%S'))

Then pass the validate_logger around to use just like logger. Maybe there's a better way?


Set up a test to describe the behavior you want to see. It's okay to use a template.

Then run the test.

Helps to have this process built in early.

I have used nosetests in the past, but it may be sufficient to just use the built in unit testing at this point. py.test is also recommeded Testing Your Code — The Hitchhiker's Guide to Python

Make it easy to run tests.

Run tests often. Before you start making changes. After changes are complete. And all along the way in between.


py.test is a more popular alternative to nose(2) test:

These simplify the syntax of the native unnittest library with shortcuts like 'assert'.

pip3 install pytest

pwd /data/data/com.termux/files/home/public/moments $ pip3 install pytest

then in directory with tests:



Many great introductions to the language exist. Here are some resources that looked strong. There may be new options available. Try searching for "python learning resourxrs". How to Think Like a Computer Scientist — How to Think Like a Computer Scientist: Learning with Python v2nd Edition documentation Dive Into Python

Regular Expressions

The group() function returns the string matched by the re


how to install a module for development with pip?

sudo pip install -e sortable/


The context you are running in will determine the python environment.


For virtualization, I prefer running a container and describe the requirements in a Dockerfile.

Create a container with docker-compose.yml

      context: python
      dockerfile: Dockerfile
    container_name: boilerplate_python
    # restart: unless-stopped
      - ./python:/srv/python
    #  - 8080:8080
    entrypoint: ["tail", "-f", "/dev/null"]

Choose versions that line up with what you have available in production.
In python/Dockerfile:

FROM python:3

# Do this so the requirements are available when the image is built
COPY requirements.txt /srv/python/requirements.txt
WORKDIR /srv/python
RUN pip install -r requirements.txt

If you need access to both node/js and python in the same container (e.g. bullmq + python worker):!

python3.9-nodejs12 3.9.12 12.22.11 buster
python3.9-nodejs12-bullseye 3.9.12 12.22.11 bullseye
python3.9-nodejs12-slim 3.9.12 12.22.11 slim
python3.9-nodejs12-alpine 3.9.12 12.22.11 alpine

FROM nikolaik/python-nodejs:python3.9-nodejs12

# Do this so the requirements are available when the image is built
COPY requirements.txt /srv/python/requirements.txt
WORKDIR /srv/python
RUN pip install -r requirements.txt

requirements.txt TODO: best way to specify versions?


-e git://

Shared System

On shared systems, you may not have the ability run containers or to install your own python packages if using the system python. In these situations, a self contained Python environment can come in handy.

cd /srv/path/to/code

wget \
    && bash ./ -b -p /srv/path/to/code/bin/miniconda

cd bin

miniconda/bin/pip3 install pymongo


Ubuntu systems come with Python installed by default. Is pip available?

Pip is useful:

sudo apt install python3-pip

Add .local/bin to the PATH by editing ~/.bashrc

export PATH

All other dependencies should be installed as part of the container so they can be tracked with the project.

Virtual Environment

Virtual Environments