Scripting Success: Reproducible Code Habits for Python Scientists#

Python Installation and Package Management#

The first step in ensuring a reproducible environment is to manage your Python installation. Common ways to handle this include:

• Using the official CPython distribution and manually managing packages.
• Installing python via system-level package managers (e.g., apt, yum, brew).
• Using the Anaconda or Miniconda distributions, which come with a pre-packaged environment manager.

When starting fresh, Miniconda is often a lightweight, flexible choice that lets you create multiple, independent environments. For instance, you could install Miniconda and then create an environment named myenv:

1
# Install Miniconda from:
2
# https://docs.conda.io/en/latest/miniconda.html
3

4
# Create a new environment
5
conda create --name myenv python=3.10
6
# Activate the newly created environment
7
conda activate myenv

Using Virtual Environments#

Even if you’re not using Conda, Python provides a built-in module called venv that creates isolated environments. By activating a venv environment, you avoid clashes with global system packages and ensure you can replicate the same environment on any machine.

Below is a quick example:

1
# Create a venv environment named .venv
2
python3 -m venv .venv
3

4
# Activate it
5
source .venv/bin/activate  # On Linux/MacOS
6
# .venv\Scripts\activate  # On Windows
7

8
# Install packages
9
pip install numpy pandas matplotlib

By keeping a requirements file (requirements.txt) or an environment file (environment.yml for Conda), you ensure all collaborators can install the same versions of your dependencies. For example, a simple requirements.txt might look like:

1
numpy==1.23.5
2
pandas==1.5.3
3
matplotlib==3.6.0
4
scipy==1.9.3

The next user can simply run pip install -r requirements.txt to mirror your setup.

Project Directory Structure#

Project structure helps collaborators quickly find the relevant scripts, data, and results. A well-organized layout also makes the difference between a disentangled fiasco and a reproducible pipeline. Below is a typical arrangement:

1
my_project/
2
   ├── data/
3
   �?   ├── raw/
4
   �?   └── processed/
5
   ├── scripts/
6
   �?   ├── analysis.py
7
   �?   ├── helpers.py
8
   �?   └── utils/
9
   ├── tests/
10
   �?   ├── test_analysis.py
11
   �?   └── test_helpers.py
12
   ├── notebooks/
13
   �?   └── exploration.ipynb
14
   ├── environment.yml  # or requirements.txt
15
   ├── README.md
16
   └── LICENSE

Feel free to modify this to suit your project’s needs, but ensure that:

1. Raw data is separated from processed data.
2. Scripts are modularized (with separate helper scripts and main scripts).
3. Tests are in their own directory.
4. Documentation files are easily accessible (top-level).

Basic Git Workflow#

Here’s the simplest Git workflow:

1
# Initialize Git in your project
2
git init
3

4
# Add your files and commit
5
git add .
6
git commit -m "Initial commit"
7

8
# Make some changes, add, and commit again
9
git add .
10
git commit -m "Add analysis script"

Push your project to a remote hosting platform like GitHub or GitLab:

1
git remote add origin https://github.com/username/my_project.git
2
git push -u origin main

Remember to .gitignore files or directories that don’t belong in version control—like data files or environment files that are too large or automatically generated. Typically:

1
.venv/
2
__pycache__/
3
*.pyc
4
.ipynb_checkpoints/

Collaborating with Branches#

Branches in Git allow you to work on new features or bug fixes without disturbing the main codebase. Common branch strategies include:

• Feature branches: For new features.
• Bug-fix branches: For fixing specific issues.
• Release branches: For preparing stable releases with version tags.

A typical branching workflow:

1
# Create and switch to a new branch for a feature
2
git checkout -b feature-add-plotting
3

4
# Make changes, commit them
5
git add .
6
git commit -m "Add a new plotting function"
7

8
# Switch back to main, merge
9
git checkout main
10
git merge feature-add-plotting
11
# Resolve conflicts if any, then commit.
12
git push origin main

Git Hooks for Code Quality#

Git hooks allow you to automate tasks at key points in the Git workflow. For instance, pre-commit hooks can check if your code is well-formatted or passes all tests before committing:

• Pre-commit: Runs formatting tools (e.g., black, flake8) or quick tests.
• Pre-push: Ensures the full test suite passes before any push.

An example .pre-commit-config.yaml snippet:

1
repos:
2
  - repo: https://github.com/psf/black
3
    rev: 22.8.0
4
    hooks:
5
      - id: black
6
  - repo: https://github.com/PyCQA/flake8
7
    rev: 4.0.1
8
    hooks:
9
      - id: flake8

This helps keep your repository clean and enforce coding standards automatically.

PEP 8 and Readability#

PEP 8 is Python’s style guide. Some of its key recommendations include:

• Use 4 spaces per indentation level.
• Keep line length to ~79 characters.
• Use snake_case for function and variable names, PascalCase for class names.
• Insert spaces after commas, around operators, and around assignments for readability.

Using an auto-formatter like black or autopep8 ensures your code meets PEP 8 standards with minimal effort.

Example of a well-formatted function:

1
def process_data(input_path: str) -> pd.DataFrame:
2
    """Load and process data from a CSV file."""
3
    df = pd.read_csv(input_path)
4
    df = df.dropna()
5
    df['some_feature'] = df['some_feature'] * 100
6
    return df

Docstrings and Documentation#

Docstrings are multiline strings that serve as in-code documentation. Tools like Sphinx or MkDocs can automatically parse docstrings to generate documentation websites, increasing the discoverability of your code’s functionality.

Docstring example (NumPy style):

1
def compute_statistics(data: pd.DataFrame) -> dict:
2
    """
3
    Compute mean and standard deviation for a DataFrame column.
4

5
    Parameters
6
    ----------
7
    data : pd.DataFrame
8
        Input DataFrame with numeric columns.
9

10
    Returns
11
    -------
12
    results : dict
13
        Dictionary containing mean and standard deviation.
14
    """
15
    mean_val = data["column"].mean()
16
    std_val = data["column"].std()
17
    return {"mean": mean_val, "std": std_val}

Good docstrings make your code more self-explanatory, lowering the effort later if someone else (or future you) needs to remember how a function works.

Logging for Diagnostics#

Instead of relying on print statements, use Python’s logging module to track the flow of your program. Logging allows you to keep different levels of log messages (e.g., debug, info, warning, error, critical) and control output easily.

Simple logging example:

1
import logging
2

3
# Configure logging
4
logging.basicConfig(level=logging.INFO)
5

6
def run_analysis(data_file):
7
    logging.info("Starting analysis")
8
    try:
9
        df = pd.read_csv(data_file)
10
        logging.debug(f"Data shape: {df.shape}")
11
        # ... analysis steps
12
        logging.info("Analysis completed successfully")
13
    except Exception as e:
14
        logging.error(f"Analysis failed: {e}")

Logs are especially helpful for diagnosing issues in large analyses or long-running computations; by setting different levels of verbosity, you can switch between seeing every detail or just high-level status updates.

Unit Tests with pytest#

While Python’s built-in unittest is sufficient for many projects, pytest is one of the most popular testing frameworks due to its simplicity and powerful features. A minimal unit test using pytest might look like:

1
from scripts.helpers import compute_statistics
2
import pandas as pd
3

4
def test_compute_statistics():
5
    data = pd.DataFrame({"column": [1, 2, 3, 4, 5]})
6
    results = compute_statistics(data)
7
    assert results["mean"] == 3
8
    assert round(results["std"], 2) == 1.58

You simply run pytest in your project’s root directory, and it will find and execute any file that starts with test_ or ends with _test.py.

Integration Tests and Test Organization#

Integration tests ensure that multiple components of your code work together as expected—e.g., verifying an entire pipeline from data loading to final result. These tests might take longer to run and can involve more complex setups:

• Integration with external services (e.g., an API).
• End-to-end data processing.
• Performance tests for certain data sizes.

Organize your tests in logical subfolders if your project grows large:

1
tests/
2
   ├── unit/
3
   ├── integration/
4
   └── system/

Take advantage of fixtures in pytest to share setup and teardown logic across multiple tests.

Setting Up Continuous Integration (CI)#

Popular CI platforms include:

• GitHub Actions
• GitLab CI
• Travis CI
• CircleCI

They generally require a YAML config file that specifies the environment and commands to run. For instance, a minimal GitHub Actions workflow (.github/workflows/tests.yml) might look like:

1
name: Tests
2
on: [push, pull_request]
3
jobs:
4
  build:
5
    runs-on: ubuntu-latest
6
    steps:
7
      - uses: actions/checkout@v2
8
      - name: Set up Python
9
        uses: actions/setup-python@v2
10
        with:
11
          python-version: '3.10'
12
      - name: Install dependencies
13
        run: |
14
          pip install --upgrade pip
15
          pip install -r requirements.txt
16
      - name: Run tests
17
        run: pytest

Whenever you push or create a pull request, the tests will be run automatically, and you’ll get quick feedback on whether the code still works as expected.

Data Versioning#

Research data can be huge and can change frequently. To keep track of which version of the data was used for a particular analysis, consider:

• Storing small data in Git if feasible.
• Using Git LFS for large data files.
• Using tools like DVC (Data Version Control) if your data is very large or is updated frequently.

If your dataset is in the gigabyte to terabyte range, external versioning solutions with cloud storage integration become almost essential. They let you revert to previous versions of data to confirm or replicate results.

Snakemake and Makefiles#

Scientific pipelines often involve multiple steps: data cleaning, transformation, modeling, generating figures, etc. Snakemake is a powerful workflow management system inspired by GNU Make, tailored for bioinformatics but applicable in many contexts:

• Declarative rule-based approach: You specify input, output, and steps.
• Automatic dependency resolution: Snakemake knows which tasks are out of date and reruns only those.
• Scalability: It can run locally or across HPC clusters.

A minimal Snakefile example:

1
rule all:
2
    input:
3
        "results/analysis.txt"
4

5
rule analyze_data:
6
    input:
7
        "data/processed/data_clean.csv"
8
    output:
9
        "results/analysis.txt"
10
    shell:
11
        """
12
        python scripts/analysis.py --input {input} --output {output}
13
        """

Then, running snakemake in the command line will automatically run analyze_data if the output analysis.txt does not exist or if its input changed.

Docker and Containerization#

Docker allows you to package your entire environment—including Python version, system libraries, and your code—into a single container image. With Docker, you can be confident that your code runs identically on any machine that has Docker installed.

A basic Dockerfile could look like:

1
# Start from a Python base image
2
FROM python:3.10-slim
3

4
# Install dependencies
5
RUN pip install --upgrade pip
6
COPY requirements.txt /tmp/requirements.txt
7
RUN pip install -r /tmp/requirements.txt
8

9
# Copy your code
10
WORKDIR /app
11
COPY . /app
12

13
# Specify the command to run
14
CMD ["python", "scripts/analysis.py"]

Then build and run it:

1
docker build -t my_analysis .
2
docker run --rm my_analysis

You can even share this image on Docker Hub or a private registry, literally shipping your environment around. Collaborators can pull the image and have the exact same setup.

Conda Environments for Scientific Computing#

Beyond Docker, Conda is almost a standard in the Python scientific ecosystem. Conda environments are highly configurable and can manage not just Python packages but also system libraries (like libxml2, curl, etc.). This is particularly useful in fields like bioinformatics or machine learning, where specialized libraries might not have easy Pip equivalents.

Creating a conda environment with pinned library versions:

1
conda create --name analysis_env python=3.10 numpy=1.23 pandas=1.5
2
conda activate analysis_env

You can export an environment file that captures every dependency and version:

1
conda env export > environment.yml

Collaborators simply run conda env create -f environment.yml to get the exact environment setup.

Reproducible Notebooks and Jupyter Extensions#

Jupyter notebooks are powerful for demonstrations, interactive analysis, and data exploration. Keeping them reproducible involves:

1. Clearing outputs before committing: This ensures results must be rerun, guaranteeing they are generated from the code, not static from a prior run.
2. Using %run or external scripts: Instead of writing all logic in the notebook, import your tested Python modules.
3. nbconvert or papermill: Convert notebooks to scripts, or parameterize them for batch runs.

Tools like nbdime also help with diffs and merges of notebooks in Git.

Code Reviews#

A code review is a structured process where a colleague reviews your changes (via pull or merge requests) before merging them into the main branch:

• Ensures quality: Catches bugs and style issues early.
• Teaches best practices: Less experienced colleagues learn from the feedback of senior developers, and vice versa.
• Increases truck factor: More than one person understands every part of the codebase, reducing risk if someone leaves.

Pair Programming and Mob Programming#

Pair programming involves two developers working together at the same workstation—one person writes the code (driver), while the other reviews each line of code as it is typed (navigator). Mob programming extends this idea to a larger group. These practices can be highly effective for complex scientific code, ensuring fewer mistakes and more collective understanding.

Project Documentation and READMEs#

A good README.md is often the first point of contact for new collaborators or for your future self. It should contain:

• Project purpose and overview.
• Instructions on environment setup.
• How to run analyses or tests.
• Contact or citation information.

For larger projects, a dedicated docs/ folder might be warranted. Using a documentation generator like Sphinx can help produce a website for your code’s API and usage instructions.

Automated Deployment and Container Registries#

When your workloads grow beyond a single server or workstation, you might need to deploy your code on multiple machines or in the cloud. Automated deployment tools (like Jenkins, GitLab CI/CD, or GitHub Actions with custom scripts) can build your Docker images, push them to a container registry, and then deploy them:

1. Build: The CI pipeline builds the Docker image for your analysis.
2. Test: The pipeline runs your entire test suite against the image.
3. Push: If tests pass, the image is pushed to a registry like Docker Hub or an internal registry.
4. Deploy: A cluster manager (e.g., Kubernetes) automatically deploys the new container.

Advanced Testing Frameworks and Profiling#

Beyond simple unit tests, advanced scenarios might call for:

• Hypothesis: A property-based testing library that generates test cases to explore edge scenarios.
• Performance profiling: Tools like cProfile, line_profiler, or even advanced solutions like Py-Spy and scalene to identify bottlenecks.
• Coverage reports: Tools like coverage.py show how much of your code is tested.

Example coverage command:

1
coverage run -m pytest
2
coverage report -m

This ensures your test suite covers as many branches and lines as possible.

Continuous Delivery (CD), Airflow, and Beyond#

While CI ensures code merges are always tested and stable, Continuous Delivery (CD) extends this to automatically release validated changes to production or staging environments. Tools like Apache Airflow orchestrate complex data pipelines, ensuring tasks run in the correct order, handle failures gracefully, and retry where necessary.

In a scientific context, you might create an Airflow DAG (Directed Acyclic Graph) that runs your data cleaning, modeling, and result generation tasks in sequence. If a step fails, the pipeline can notify you and pause. This level of automation can drastically reduce repeated manual tasks, freeing you to focus on new analyses.