Implementation Plan: Article Collection
Branch: 001-article-collection | Date: 2026-05-28 | Spec: spec.md
Note: This is a brownfield plan. All described architecture already exists in production. The purpose of this plan is to document decisions for future reference and to guide verification tasks generated by /speckit-tasks.
Summary
The article collection capability discovers, fetches, deduplicates, and forwards articles from three source types (RSS feeds, arXiv API, tech blogs) to the downstream intelligence pipeline via an in-process event bus. The architecture follows hexagonal/DDD: a domain Scraper interface is implemented by infrastructure scrapers; business rules (dedup, keyword filtering, event publishing) live in the application and domain layers.
Technical Context
Language/Version: Python 3.11
Primary Dependencies: feedparser (RSS), requests (HTTP), BeautifulSoup4 + lxml (HTML parsing), pypdf / pdfplumber (PDF parsing), SQLAlchemy 2.0 (persistence), structlog (logging), OpenTelemetry (metrics)
Storage: PostgreSQL 15 via SQLAlchemy 2.0 (NullPool — cron job model, no connection pooling)
Testing: pytest; unit tests in src/tests/unit/; integration tests in src/tests/integration/ with isolated test_integration schema and per-test savepoint rollback. All tests run inside Docker (make test, make test-integration).
Target Platform: Linux container (Railway cron job); container starts, runs, exits.
Project Type: Scheduled batch pipeline service
Performance Goals: Complete a full pipeline run (all sources) within 50 minutes; per-host concurrent requests capped at 1 to avoid rate-limiting.
Constraints: No 24/7 worker; NullPool prevents connection leak across cron invocations; 50-minute hard timeout enforced by entrypoint.
Scale/Scope: Tens of sources; hundreds of articles per run.
Constitution Check
GATE: Checked against constitution v1.1.0
| Principle | Status | Notes |
|---|---|---|
| I. DDD layer separation | ✅ Pass | Scraper ABC in domain/services/; implementations in infrastructure/; use case in application/; zero infra imports in domain |
| II. Atomic Frontend Architecture | N/A | No frontend components in this capability |
| III. Test Discipline | ✅ Pass | Unit tests must not require DB; integration tests use isolated schema + savepoints |
| IV. Docker-First Development | ✅ Pass | Tests run via make test / make test-integration inside Docker |
| V. CI-Only Deployment Boundary | N/A | No deployment artifacts in this capability |
| VI. Observability as First-Class | ✅ Pass | OTel metrics (SCRAPER_ARTICLES_FOUND), structlog throughout, no silent error swallowing |
| VII. Code Style & Quality | ✅ Pass | PEP 8, uv, no TODO comments in production code |
No violations. No complexity justification required.
Project Structure
Documentation (this feature)
specs/001-article-collection/
├── plan.md ← This file
├── research.md ← Phase 0: architectural decision record
├── data-model.md ← Phase 1: domain entities and relationships
├── checklists/
│ └── requirements.md ← Spec quality checklist (all pass)
└── tasks.md ← Phase 2 output (generated by /speckit-tasks)Source Code (existing layout)
src/modules/collection/
├── domain/
│ ├── entities/ ← ScrapeJob, ScrapeJobMetadata, ArxivMetadata, ScraperSetting, FailedTask
│ ├── value_objects/ ← ScrapedArticle, UrlHash, ScraperKeyword, SelectorConfig, Url
│ ├── services/ ← Scraper (ABC), DedupService
│ ├── repositories/ ← ArxivMetadataRepository (ABC), ScraperSettingRepository (ABC)
│ └── factories/ ← ScraperFactory (domain factory interface)
├── application/
│ ├── use_cases/ ← ProcessScrapedArticleUseCase, DiscoverScrapeJobsUseCase, ArticleOutcome
│ ├── event_handlers/ ← ArticleScrapedHandler
│ └── events/ ← ArticleScrapedEvent, PipelineCompletedEvent
src/infrastructure/collection/
├── scrapers/ ← BaseScraper, RssScraper, ArxivScraper, BlogScraper, ScraperFactory
├── executor/ ← ScrapeExecutor, DiscoverTask, FetchTask, HostQueueMap,
│ QueueRouter, QueueSelector (WeightedRoundRobin)
├── parsers/ ← BaseContentParser, HtmlArticleParser, PdfParser, SanitizeService
├── clients/ ← RssClient, ArxivClient (ArxivRateLimitedError)
└── handlers/ ← OtelMetricsHandler
src/tests/unit/ ← Unit tests (no DB)
src/tests/integration/ ← Integration tests (isolated schema)Complexity Tracking
No constitution violations to justify.