Python
Python is a good language for writing software. It encourages clean, readable code. It's a nice interface to the underlying system.
A great choice for synchronous workflows. Synchronous workflows can help processes on the system side. For example, walking a file system and reporting when the job finishes.
Python doesn't run client-side on web browsers. For that you'll need Javascript (JS). Python shows up frequently on the API side or server side.
Installation
The context you are running in will determine the python environment.
Docker
For virtualization, I prefer running a container and describe the requirements in a Dockerfile.
Create a container with docker-compose.yml
python:
build:
context: python
dockerfile: Dockerfile
container_name: boilerplate_python
# restart: unless-stopped
volumes:
- ./python:/srv/python
#ports:
# - 8080:8080
entrypoint: ["tail", "-f", "/dev/null"]
Choose versions that line up with what you have available in production.
In python/Dockerfile:
FROM python:3
# Do this so the requirements are available when the image is built
COPY requirements.txt /srv/python/requirements.txt
WORKDIR /srv/python
RUN pip install -r requirements.txt
If you need access to both node/js and python in the same container (e.g. bullmq + python worker):
https://hub.docker.com/r/nikolaik/python-nodejs/#!
python3.9-nodejs12 3.9.12 12.22.11 buster
python3.9-nodejs12-bullseye 3.9.12 12.22.11 bullseye
python3.9-nodejs12-slim 3.9.12 12.22.11 slim
python3.9-nodejs12-alpine 3.9.12 12.22.11 alpine
FROM nikolaik/python-nodejs:python3.9-nodejs12
# Do this so the requirements are available when the image is built
COPY requirements.txt /srv/python/requirements.txt
WORKDIR /srv/python
RUN pip install -r requirements.txt
requirements.txt TODO: best way to specify versions?
# https://stackoverflow.com/questions/16584552/how-to-state-in-requirements-txt-a-direct-github-source
-e git://github.com/charlesbrandt/sortable#egg=sortable
requests
Shared System
On shared systems, you may not have the ability run containers or to install your own python packages if using the system python. In these situations, a self contained Python environment can come in handy.
cd /srv/path/to/code
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& bash ./Miniconda3-latest-Linux-x86_64.sh -b -p /srv/path/to/code/bin/miniconda
cd bin
miniconda/bin/pip3 install pymongo
Local
Ubuntu systems come with Python installed by default. Is pip available?
Pip is useful:
sudo apt install python3-pip
Add .local/bin to the PATH by editing ~/.bashrc
PATH="/opt/local/bin:~/.yarn/bin:~/.local/bin:${PATH}"
export PATH
All other dependencies should be installed as part of the container so they can be tracked with the project.
Virtual Environment
New Project
Best practices for starting a python project
Files and Directories that should be included:
https://docs.python-guide.org/writing/structure/
https://gist.github.com/sloria/7001839
I like starting with a template script to include things I like when getting started. A boilerplate.
#!/usr/bin/env python3
"""
#!/usr/bin/env python3
"""
# By:
# On: [date]
# License: MIT
Usage:
template_script.py [source] [destination]
"""
import os
import sys
def usage():
print(__doc__)
def read_file(source):
with open(source, 'r') as f:
for line in f.readlines():
print(line)
def merge_logs(source, destination):
if source and os.path.exists(source):
options = os.listdir(source)
print(options)
else:
usage()
print("Couldn't find source path: %s" % source)
exit()
if __name__ == '__main__':
source = None
destination = None
if len(sys.argv) > 1:
helps = ['--help', 'help', '-h']
for i in helps:
if i in sys.argv:
usage()
exit()
source = sys.argv[1]
if len(sys.argv) > 2:
destination = sys.argv[2]
merge_logs(source, destination)
Walking Directories
def walk_files(source, destination):
if os.path.exists(source):
for root, dirs, files in os.walk(source):
for f in files:
if re.search('\?', f):
sname = os.path.join(root, f)
else:
print("Couldn't find path: %s" % source)
exit()
f-strings
Python 3.6 introduced a handy new syntax for substituting variables in a string. Prefix with an f and use curly brackets to designate the variables in the string:
name = "human"
print(f"Hello {name}")
JSON
Use the built in library:
import json
Read
if Path(source).is_file():
with open(source, 'r+') as f:
items = json.load(f)
Write
def add_items(items, destination):
with open(destination, "w") as f:
json.dump(items, f, indent=2, default=str)
Testing
Set up a test to describe the behavior you want to see. It's okay to use a template.
Then run the test.
Helps to have this process built in early.
I have used nosetests in the past, but it may be sufficient to just use the built in unit testing at this point. py.test is also recommeded
https://docs.python-guide.org/writing/tests/ Testing Your Code — The Hitchhiker's Guide to Python
Make it easy to run tests.
Run tests often. Before you start making changes. After changes are complete. And all along the way in between.
https://docs.python.org/3/library/unittest.html
py.test
py.test is a more popular alternative to nose(2) test:
https://docs.pytest.org/en/latest/
These simplify the syntax of the native unnittest library with shortcuts like 'assert'.
pip3 install pytest
pwd /data/data/com.termux/files/home/public/moments $ pip3 install pytest
then in directory with tests:
pytest
Learning
Many great introductions to the language exist. Here are some resources that looked strong. There may be new options available. Try searching for "python learning resourxrs".
http://www.greenteapress.com/thinkpython/thinkpython.html
http://openbookproject.net//thinkCSpy/ How to Think Like a Computer Scientist — How to Think Like a Computer Scientist: Learning with Python v2nd Edition documentation
http://diveintopython.org/ Dive Into Python
Regular Expressions
https://www.debuggex.com/cheatsheet/regex/python
https://www.datacamp.com/community/tutorials/python-regular-expression-tutorial
The group() function returns the string matched by the re
Development
how to install a module for development with pip?
https://pip.pypa.io/en/stable/reference/pip_install/
sudo pip install -e sortable/