How to Test Bartholomew

Install the Bartholomew evaluation and diagnostic CLI directly from PyPI:

pip install bartholomew-eval

Verify the install worked:

python -m bartholomew_eval.cli --help

Diagnose a failing test file: Point Bartholomew at any Python file with a known failure and it will identify the root cause and suggest a minimal fix.

python -m bartholomew_eval.cli diagnose ./path/to/test_file.py

Detect and quarantine a flaky test by running it multiple times and analyzing the variance:

# Replace with your actual test function name
python -m bartholomew_eval.cli quarantine test_payment_webhook_timeout

Benchmark AI repair models side-by-side across real AST defect scenarios:

python -m bartholomew_eval.cli eval-models

You will see a ranked leaderboard comparing Gemini 2.0 Flash, Claude 3.7 Sonnet, GPT-4o, and DeepSeek R1 on pass rate, latency, and cost per fix.

The Bartholomew Simulator lets you watch the full autonomous repair loop execute against three real production failure scenarios without connecting a GitHub repository:

1. Go to bartholomew.info/simulator
2. Select a failure scenario (Python Asyncio, TypeScript Vitest, or Go Data Race)
3. Click Run Full Auto-Fix Cycle
4. Watch the 5-stage repair loop execute step by step in the terminal

The simulator faithfully demonstrates: Webhook Ingest → Isolated Reproduction → Minimal AST Patch → 100% Test Verification → Green PR Dispatch.

This connects your repository directly to the Bartholomew backend so it receives real CI failure events.

Step A — Copy the Webhook URL:

https://acn-fastapi-backend-k72r.a.run.app/api/webhooks/github

Step B — Add it to your GitHub Repository:

1. Go to your GitHub repository → Settings → Webhooks
2. Click Add webhook
3. Paste the Webhook URL above into the Payload URL field
4. Set Content type to application/json
5. Under Events, choose Workflow runs (or "Send me everything")
6. Click Add webhook

Once your webhook is connected, introduce a known failure into your repository to trigger the full repair loop:

# Paste this into any test file in your repo
# This simulates the exact asyncio failure Bartholomew knows how to repair

import asyncio

def test_async_broken():
    loop = asyncio.get_event_loop()  # Deprecated in Python 3.10+
    result = loop.run_until_complete(some_async_task())
    assert result == True

async def some_async_task():
    return True

Push to your main branch and let CI run. When it fails, Bartholomew will:

1. Receive the GitHub workflow_run failure webhook
2. Clone the failing commit into an isolated sandbox
3. Synthesize a standalone reproduction proof
4. Apply the minimal AST patch (asyncio.get_running_loop())
5. Verify 100% of tests pass locally
6. Open a Pull Request on your repository with the fix attached

You can monitor all live repair activity in the Operations Workspace.