Testing Guide¶
This guide covers APILeak's comprehensive testing strategy, including unit tests, integration tests, and property-based testing.
📋 Table of Contents¶
- Testing Philosophy
- Test Structure
- Running Tests
- Unit Testing
- Integration Testing
- Property-Based Testing
- Test Coverage
- Writing Tests
- Continuous Integration
🎯 Testing Philosophy¶
APILeak follows a comprehensive testing approach that ensures:
- Correctness: All functionality works as specified
- Reliability: System behaves consistently under various conditions
- Security: Security features work correctly and don't introduce vulnerabilities
- Performance: System meets performance requirements
- Maintainability: Tests are easy to understand and maintain
Testing Pyramid¶
🏗️ Test Structure¶
Directory Structure¶
tests/
├── unit/ # Unit tests
│ ├── test_core_engine.py # Core engine tests
│ ├── test_findings_collector.py # Findings collector tests
│ ├── test_http_client.py # HTTP client tests
│ └── test_owasp_modules.py # OWASP module tests
├── integration/ # Integration tests
│ ├── test_core_engine_integration.py
│ ├── test_fuzzing_integration.py
│ └── test_owasp_integration.py
├── properties/ # Property-based tests
│ ├── test_property_level_auth_properties.py
│ └── test_bola_properties.py
├── fixtures/ # Test fixtures and data
│ ├── sample_responses.json
│ └── test_configs.yaml
└── conftest.py # Pytest configuration
Test Categories¶
- Unit Tests: Test individual components in isolation
- Integration Tests: Test component interactions
- Property-Based Tests: Test universal properties with generated data
- End-to-End Tests: Test complete workflows
- Performance Tests: Test performance characteristics
🚀 Running Tests¶
Basic Test Execution¶
# Run all tests
make test
# Run specific test file
python -m pytest tests/test_findings_collector.py -v
# Run tests with specific pattern
python -m pytest tests/ -k "test_bola" -v
# Run tests in parallel
python -m pytest tests/ -n auto
Test Options¶
# Verbose output
python -m pytest tests/ -v
# Show test coverage
python -m pytest tests/ --cov=core --cov=utils --cov=modules
# Generate HTML coverage report
python -m pytest tests/ --cov=core --cov-report=html
# Run only failed tests
python -m pytest tests/ --lf
# Stop on first failure
python -m pytest tests/ -x
# Run tests with specific markers
python -m pytest tests/ -m "not slow"
Make Commands¶
make test # Run all tests
make test-coverage # Run tests with coverage
make test-unit # Run only unit tests
make test-integration # Run only integration tests
make test-properties # Run only property-based tests
🔧 Unit Testing¶
Unit tests verify individual components work correctly in isolation.
Example Unit Test¶
"""Unit tests for FindingsCollector"""
import pytest
from unittest.mock import Mock, patch
from datetime import datetime
from utils.findings import FindingsCollector, Finding
from core.config import Severity
class TestFindingsCollector:
"""Test cases for FindingsCollector"""
def setup_method(self):
"""Setup test fixtures"""
self.scan_id = "test-scan-123"
self.collector = FindingsCollector(self.scan_id)
def test_initialization(self):
"""Test collector initialization"""
assert self.collector.scan_id == self.scan_id
assert len(self.collector.findings) == 0
assert len(self.collector._deduplication_cache) == 0
def test_add_finding_with_auto_classification(self):
"""Test adding finding with automatic classification"""
finding = self.collector.add_finding(
category="BOLA_ANONYMOUS_ACCESS",
severity=None, # Will be auto-classified
endpoint="/api/users/123",
method="GET",
evidence="Anonymous access detected",
recommendation="Implement authentication"
)
assert finding.severity == Severity.CRITICAL
assert finding.owasp_category == "API1"
assert finding.scan_id == self.scan_id
assert len(self.collector.findings) == 1
@pytest.mark.parametrize("category,expected_severity", [
("BOLA_ANONYMOUS_ACCESS", Severity.CRITICAL),
("WEAK_JWT_ALGORITHM", Severity.HIGH),
("MISSING_RATE_LIMITING", Severity.MEDIUM),
("INFORMATION_DISCLOSURE", Severity.LOW),
("ENDPOINT_DISCOVERED", Severity.INFO)
])
def test_severity_classification_rules(self, category, expected_severity):
"""Test automatic severity classification"""
finding = self.collector.add_finding(
category=category,
severity=None,
endpoint="/api/test",
method="GET",
evidence="Test evidence",
recommendation="Test recommendation"
)
assert finding.severity == expected_severity
Unit Test Best Practices¶
- Isolation: Mock external dependencies
- Fast: Tests should run quickly
- Deterministic: Same input always produces same output
- Focused: Test one thing at a time
- Clear: Test names describe what is being tested
🔗 Integration Testing¶
Integration tests verify that components work correctly together.
Example Integration Test¶
"""Integration tests for Core Engine"""
import pytest
import asyncio
from unittest.mock import AsyncMock, Mock
from core.engine import APILeakCore
from core.config import APILeakConfig, TargetConfig
class TestCoreEngineIntegration:
"""Integration tests for APILeak Core Engine"""
@pytest.fixture
def config(self):
"""Create test configuration"""
return APILeakConfig(
target=TargetConfig(base_url="https://httpbin.org"),
# ... other config
)
@pytest.fixture
def core(self, config):
"""Create APILeak core instance"""
return APILeakCore(config)
@pytest.mark.asyncio
async def test_complete_scan_workflow(self, core):
"""Test complete scan workflow"""
# Mock external dependencies
with patch('modules.fuzzing.orchestrator.FuzzingOrchestrator') as mock_fuzzer:
mock_fuzzer.return_value.discover_endpoints = AsyncMock(return_value=[])
# Execute scan
results = await core.run_scan("https://httpbin.org")
# Verify results
assert results.scan_id is not None
assert results.target_url == "https://httpbin.org"
assert results.findings_collector is not None
@pytest.mark.asyncio
async def test_findings_integration(self, core):
"""Test findings collection integration"""
# Add test findings
core.findings_collector.add_finding(
category="TEST_FINDING",
severity=None,
endpoint="/test",
method="GET",
evidence="Test evidence",
recommendation="Test recommendation"
)
# Verify integration
stats = core.findings_collector.get_statistics()
assert stats["total_findings"] == 1
Integration Test Patterns¶
- Real Dependencies: Use real components where possible
- Controlled Environment: Use test databases/services
- End-to-End Flows: Test complete user workflows
- Error Scenarios: Test error handling and recovery
🎲 Property-Based Testing¶
Property-based tests use generated data to verify universal properties.
Example Property-Based Test¶
"""Property-based tests for Property Level Authorization Module"""
from hypothesis import given, strategies as st, settings
import hypothesis
from modules.owasp.property_level_auth import PropertyLevelAuthModule
class TestPropertyLevelAuthProperties:
"""Property-based tests for Property Level Authorization Module"""
@given(
field_names=st.lists(
st.text(min_size=1, max_size=30),
min_size=1,
max_size=20
)
)
@settings(max_examples=100, deadline=5000)
def test_undocumented_field_filtering_property(self, field_names):
"""
**Feature: apileak-owasp-enhancement, Property 9: Undocumented Field Detection**
**Validates: Requirements 3.4**
For any list of field names, undocumented field filtering should consistently
identify potentially undocumented fields while filtering out common metadata fields.
"""
module = PropertyLevelAuthModule(self.config, self.mock_http_client, self.auth_contexts)
for field_name in field_names:
result = module._is_potentially_undocumented(field_name)
# Property: Result should always be a boolean
assert isinstance(result, bool)
# Property: Common metadata fields should always be filtered out
field_lower = field_name.lower()
common_fields = ['timestamp', 'created_at', 'updated_at', 'id']
for common_field in common_fields:
if common_field in field_lower:
assert result is False, f"Common field '{field_name}' should be filtered out"
break
Property-Based Testing Benefits¶
- Comprehensive Coverage: Tests many more cases than manual examples
- Edge Case Discovery: Finds edge cases you might not think of
- Regression Prevention: Catches regressions across wide input space
- Specification Verification: Verifies properties hold universally
Property Test Guidelines¶
- Clear Properties: Define clear, testable properties
- Good Generators: Create generators that produce valid test data
- Assume Wisely: Use
assume()sparingly to avoid filtering too much - Shrinking: Let Hypothesis find minimal failing examples
📊 Test Coverage¶
Coverage Goals¶
- Unit Tests: 90%+ coverage for core components
- Integration Tests: 80%+ coverage for critical paths
- Property Tests: 100% coverage for correctness properties
- Overall: 85%+ total coverage
Measuring Coverage¶
# Generate coverage report
python -m pytest tests/ --cov=core --cov=utils --cov=modules --cov-report=html
# View coverage in terminal
python -m pytest tests/ --cov=core --cov-report=term-missing
# Generate XML report for CI
python -m pytest tests/ --cov=core --cov-report=xml
Coverage Analysis¶
# View HTML report
open htmlcov/index.html
# Check coverage thresholds
coverage report --fail-under=85
✍️ Writing Tests¶
Test Structure (AAA Pattern)¶
def test_feature_functionality():
"""Test that feature works correctly"""
# Arrange - Set up test data and conditions
config = create_test_config()
module = TestModule(config)
test_data = generate_test_data()
# Act - Execute the functionality being tested
result = module.process_data(test_data)
# Assert - Verify the results
assert result.is_valid()
assert result.count == len(test_data)
assert all(item.processed for item in result.items)
Test Naming Conventions¶
# Good test names
def test_add_finding_with_valid_data_creates_finding():
def test_mass_assignment_detection_with_dangerous_fields_returns_true():
def test_http_client_with_rate_limiting_respects_limits():
# Poor test names
def test_finding():
def test_mass_assignment():
def test_http():
Fixtures and Mocking¶
@pytest.fixture
def sample_config():
"""Create sample configuration for tests"""
return APILeakConfig(
target=TargetConfig(base_url="https://test.example.com"),
# ... other config
)
@pytest.fixture
def mock_http_client():
"""Create mock HTTP client"""
client = Mock()
client.request = AsyncMock()
return client
def test_with_mocked_dependencies(sample_config, mock_http_client):
"""Test using fixtures and mocks"""
# Configure mock
mock_http_client.request.return_value = Mock(status_code=200)
# Test functionality
module = TestModule(sample_config, mock_http_client)
result = module.process_request()
# Verify mock was called correctly
mock_http_client.request.assert_called_once()
assert result.success is True
Parametrized Tests¶
@pytest.mark.parametrize("input_data,expected_result", [
("valid_input", True),
("invalid_input", False),
("", False),
(None, False),
])
def test_validation_with_various_inputs(input_data, expected_result):
"""Test validation with various input types"""
result = validate_input(input_data)
assert result == expected_result
Async Testing¶
@pytest.mark.asyncio
async def test_async_functionality():
"""Test asynchronous functionality"""
client = AsyncHTTPClient()
response = await client.get("https://httpbin.org/get")
assert response.status_code == 200
assert response.json is not None
🔄 Continuous Integration¶
GitHub Actions Workflow¶
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.11, 3.12]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: |
python -m pytest tests/ --cov=core --cov=utils --cov=modules
- name: Upload coverage
uses: codecov/codecov-action@v3
Test Automation¶
- Pre-commit Hooks: Run tests before commits
- Pull Request Checks: Require tests to pass before merge
- Coverage Reporting: Track coverage trends over time
- Performance Monitoring: Track test execution time
Quality Gates¶
- All tests must pass
- Coverage must meet minimum thresholds
- No new linting errors
- Performance tests within acceptable limits
🐛 Debugging Tests¶
Common Issues¶
# Issue: Test is flaky
def test_flaky_behavior():
# Problem: Depends on timing or external state
time.sleep(0.1) # Don't do this
# Solution: Use proper synchronization
await wait_for_condition(lambda: system.is_ready())
# Issue: Test is too slow
def test_slow_operation():
# Problem: Does real I/O or computation
result = expensive_operation() # Don't do this
# Solution: Mock expensive operations
with patch('module.expensive_operation') as mock_op:
mock_op.return_value = expected_result
result = function_under_test()
# Issue: Test is brittle
def test_brittle_assertion():
# Problem: Tests implementation details
assert len(internal_cache) == 5 # Don't do this
# Solution: Test behavior, not implementation
assert system.can_handle_request()
Debugging Techniques¶
# Use pytest debugging features
def test_with_debugging():
import pdb; pdb.set_trace() # Debugger breakpoint
# Or use pytest's built-in debugging
pytest.set_trace()
# Capture output for debugging
def test_with_output_capture(capfd):
function_that_prints()
captured = capfd.readouterr()
assert "expected output" in captured.out
# Use temporary directories
def test_with_temp_files(tmp_path):
test_file = tmp_path / "test.txt"
test_file.write_text("test content")
result = process_file(test_file)
assert result.success
📚 Additional Resources¶
Happy Testing! 🧪