🧪 Testing Guide

Complete guide for testing ai-slop-gate.

Quick Start

# Run all tests
python -m pytest ai_slop_gate/tests -v

# Run with coverage
python -m pytest ai_slop_gate/tests \
  --cov=ai_slop_gate \
  --cov-report=term-missing \
  --cov-report=html

# View HTML coverage report
xdg-open htmlcov/index.html  # Linux
open htmlcov/index.html       # macOS
start htmlcov\index.html      # Windows

Test Structure

ai_slop_gate/
├── tests/
│   ├── __init__.py
│   ├── conftest.py              # Shared fixtures
│   ├── test_cli.py              # CLI tests
│   ├── test_providers/
│   │   ├── test_static.py       # Static analysis tests
│   │   ├── test_gemini.py       # Gemini provider tests
│   │   ├── test_groq.py         # Groq provider tests
│   │   └── test_ollama.py       # Ollama provider tests
│   ├── test_compliance/
│   │   ├── test_license.py      # License checks
│   │   ├── test_gdpr.py         # GDPR compliance
│   │   └── test_supply_chain.py # Supply chain tests
│   └── test_integration/
│       ├── test_github.py       # GitHub integration
│       ├── test_gitlab.py       # GitLab integration
│       └── test_cache.py        # Cache integration

Running Tests

All Tests

python -m pytest ai_slop_gate/tests -v

Specific Test File

# Test static provider
python -m pytest ai_slop_gate/tests/test_providers/test_static.py -v

# Test CLI
python -m pytest ai_slop_gate/tests/test_cli.py -v

Specific Test Function

python -m pytest ai_slop_gate/tests/test_cli.py::test_run_static -v

By Marker

# Only unit tests
python -m pytest -m unit

# Only integration tests
python -m pytest -m integration

# Skip slow tests
python -m pytest -m "not slow"

Test Coverage

Generate Coverage Report

source .venv/bin/activate

python -m pytest ai_slop_gate/tests \
  --cov=ai_slop_gate \
  --cov-report=term-missing \
  --cov-report=html \
  --cov-report=xml

View Coverage

# Terminal report (summary)
python -m pytest --cov=ai_slop_gate --cov-report=term

# HTML report (detailed)
python -m pytest --cov=ai_slop_gate --cov-report=html
xdg-open htmlcov/index.html

# XML report (for CI/CD)
python -m pytest --cov=ai_slop_gate --cov-report=xml

Coverage Targets

Component	Target	Current
Overall	80%	Check `htmlcov/index.html`
Core CLI	90%	-
Providers	75%	-
Compliance	85%	-

Cache Integration Tests

Quick Test (30 seconds)

./scripts/quick_cache_test.sh

Expected output:

✅ Cache system initialized
✅ First run: API called
✅ Second run: Cache hit
✅ All integration tests passed!

Full Smoke Test

Requires API keys:

export GEMINI_API_KEY="your-key"
./scripts/verify_cache_v2.sh

Tests:

Cache initialization
First run (API call)
Second run (cache hit)
Cache invalidation
Cache statistics

Test Demo Repository

Test detection on intentionally bad code:

git clone https://github.com/SergUdo/slop_test
cd slop_test

# Run static analysis
python -m ai_slop_gate.cli run --provider static --path .

# Run LLM analysis
python -m ai_slop_gate.cli run --provider gemini --llm-local --path .

Expected: Multiple violations detected (secrets, AI slop, license issues)

Writing Tests

Unit Test Example

import pytest
from ai_slop_gate.providers.static import StaticProvider

def test_secret_detection():
    """Test that hardcoded secrets are detected."""
    code = 'API_KEY = "sk-1234567890abcdef"'
    
    provider = StaticProvider()
    results = provider.analyze(code)
    
    assert len(results) > 0
    assert any('secret' in r.message.lower() for r in results)

Integration Test Example

import pytest
from ai_slop_gate.cli import run_analysis

@pytest.mark.integration
def test_full_analysis(tmp_path):
    """Test full analysis pipeline."""
    # Create test file
    test_file = tmp_path / "test.py"
    test_file.write_text('eval("malicious_code")')
    
    # Run analysis
    results = run_analysis(
        provider='static',
        path=str(tmp_path)
    )
    
    assert results.has_violations()
    assert 'eval' in results.violations[0].message

Cache Test Example

import pytest
from ai_slop_gate.cache import LLMCache

def test_cache_hit():
    """Test cache hit on repeated analysis."""
    cache = LLMCache()
    
    # First call
    result1 = cache.get_or_compute(
        key='test_key',
        compute_fn=lambda: expensive_llm_call()
    )
    
    # Second call (should use cache)
    result2 = cache.get_or_compute(
        key='test_key',
        compute_fn=lambda: expensive_llm_call()
    )
    
    assert result1 == result2
    assert cache.stats['hits'] == 1

Test Fixtures

Common Fixtures (`conftest.py`)

import pytest
from pathlib import Path

@pytest.fixture
def test_project(tmp_path):
    """Create a temporary test project."""
    project = tmp_path / "test_project"
    project.mkdir()
    
    # Create test files
    (project / "main.py").write_text("print('hello')")
    (project / "policy.yml").write_text("enforcement: advisory")
    
    return project

@pytest.fixture
def mock_api_response():
    """Mock LLM API response."""
    return {
        'severity': 'high',
        'confidence': 0.9,
        'message': 'AI-generated code detected'
    }

Mocking External APIs

Mock Gemini API

import pytest
from unittest.mock import patch

@pytest.mark.unit
def test_gemini_provider(mock_api_response):
    """Test Gemini provider with mocked API."""
    with patch('ai_slop_gate.providers.gemini.call_api') as mock:
        mock.return_value = mock_api_response
        
        provider = GeminiProvider()
        result = provider.analyze("test code")
        
        assert result.severity == 'high'
        mock.assert_called_once()

Mock GitHub API

@pytest.mark.integration
def test_github_comment():
    """Test PR comment posting."""
    with patch('github.PullRequest.create_comment') as mock:
        post_pr_comment(
            repo='owner/repo',
            pr_id=123,
            message='Test comment'
        )
        
        mock.assert_called_once()

Performance Testing

Benchmark Test

import pytest
import time

@pytest.mark.benchmark
def test_static_analysis_performance(benchmark):
    """Benchmark static analysis speed."""
    def analyze():
        provider = StaticProvider()
        return provider.analyze(large_codebase)
    
    result = benchmark(analyze)
    assert result.duration < 5.0  # Must complete in 5s

Cache Performance

# Measure cache improvement
./scripts/benchmark_cache.sh

# Expected output:
# Without cache: 15.2s
# With cache: 0.8s
# Speedup: 19x

Test Markers

Available Markers

@pytest.mark.unit          # Unit tests (fast, no external deps)
@pytest.mark.integration   # Integration tests (may need API keys)
@pytest.mark.slow          # Slow tests (>5 seconds)
@pytest.mark.benchmark     # Performance benchmarks
@pytest.mark.requires_api  # Requires API keys

Run by Marker

# Only fast unit tests
pytest -m "unit and not slow"

# Integration tests only
pytest -m integration

# Skip tests requiring API
pytest -m "not requires_api"

CI/CD Testing

GitHub Actions

name: Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.12'
      
      - name: Install dependencies
        run: |
          pip install -e .
          pip install pytest pytest-cov
      
      - name: Run tests
        run: |
          python -m pytest ai_slop_gate/tests \
            --cov=ai_slop_gate \
            --cov-report=xml \
            --cov-report=term
      
      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          file: ./coverage.xml

GitLab CI

test:
  stage: test
  image: python:3.12
  script:
    - pip install -e .
    - pip install pytest pytest-cov
    - pytest ai_slop_gate/tests --cov=ai_slop_gate --cov-report=term
  coverage: '/TOTAL.*\s+(\d+%)$/'

Test Data

Fixtures Location

ai_slop_gate/tests/fixtures/
├── code_samples/
│   ├── good_code.py
│   ├── bad_code.py
│   ├── ai_slop.py
│   └── secrets.py
├── policies/
│   ├── strict.yml
│   ├── advisory.yml
│   └── permissive.yml
└── responses/
    ├── gemini_response.json
    └── groq_response.json

Using Test Fixtures

import pytest
from pathlib import Path

@pytest.fixture
def ai_slop_sample():
    """Load AI slop code sample."""
    fixture_path = Path(__file__).parent / 'fixtures' / 'code_samples' / 'ai_slop.py'
    return fixture_path.read_text()

def test_ai_slop_detection(ai_slop_sample):
    provider = GeminiProvider()
    result = provider.analyze(ai_slop_sample)
    assert result.has_ai_slop

Debugging Tests

Run with Verbose Output

pytest -vv

Show Print Statements

pytest -s

Stop on First Failure

pytest -x

Run Last Failed Tests

pytest --lf

Debug with PDB

def test_something():
    import pdb; pdb.set_trace()
    assert some_condition

Or:

pytest --pdb  # Drop into debugger on failure

Test Best Practices

1. Keep Tests Fast

# ✅ GOOD: Mock external calls
@patch('ai_slop_gate.providers.gemini.call_api')
def test_provider(mock_api):
    mock_api.return_value = {...}

# ❌ BAD: Real API calls in tests
def test_provider():
    result = call_real_api()  # Slow and flaky

2. Test Edge Cases

def test_empty_input():
    assert analyze("") == []

def test_large_input():
    assert analyze("x" * 1_000_000) is not None

def test_unicode():
    assert analyze("🚀 code") is not None

3. Use Descriptive Names

# ✅ GOOD
def test_secret_detection_in_environment_variables():
    ...

# ❌ BAD
def test_secrets():
    ...

4. One Assertion Per Test

# ✅ GOOD
def test_severity_is_high():
    assert result.severity == 'high'

def test_confidence_is_above_threshold():
    assert result.confidence > 0.8

# ❌ BAD
def test_result():
    assert result.severity == 'high'
    assert result.confidence > 0.8
    assert result.message == 'test'

Troubleshooting

Issue: “ModuleNotFoundError”

Solution: Install in editable mode

pip install -e .

Issue: Tests hang indefinitely

Solution: Use timeout

pytest --timeout=60  # 60 second timeout per test

Issue: “No tests collected”

Solution: Check test discovery

# Ensure test files start with "test_"
# Ensure test functions start with "test_"
pytest --collect-only  # See what pytest finds

Issue: Cache tests failing

Solution: Clear cache before tests

rm -rf .ai-slop-cache/
pytest ai_slop_gate/tests/test_integration/test_cache.py

Coverage Goals

Priority Areas

Core CLI (90%+)
- Command parsing
- Configuration loading
- Provider orchestration
Static Providers (85%+)
- Secret detection
- Dangerous function detection
- License checks
LLM Providers (75%+)
- API integration
- Response parsing
- Cache integration
Compliance (85%+)
- GDPR checks
- License validation
- Supply chain audit

Support

Testing issues?

Check GitHub Issues
Read Documentation