Article Analyzer

Appearance

Feature Specification: LLM Article Analysis ​

Feature Branch: 003-llm-analysis

Created: 2026-05-29

Status: Draft

Input: User description: "LLM analysis capability — AnalyzeArticleUseCase + ResilientLLMService + SlidingWindowStrategy"

User Scenarios & Testing (mandatory) ​

User Story 1 - Analyze a Scraped Article (Priority: P1) ​

A system operator runs the scraper pipeline. After an article is scraped and saved, the system automatically analyzes it using a configured LLM provider to extract a summary, pain points, insights, innovations, and topic-relevant tags.

Why this priority: Core value of the pipeline. Without analysis, scraped articles provide no structured insight to end users.

Independent Test: Can be tested by injecting a single Article entity into AnalyzeArticleUseCase.execute() and asserting that an Analysis record with all required fields is persisted to the database.

Acceptance Scenarios:

  1. Given a scraped Article with content and a configured LLM provider, When AnalyzeArticleUseCase.execute() is called, Then an Analysis record is persisted with non-empty summary, pain_points, insights, innovations, and at least one tag group.
  2. Given an ArXiv article with more than 15,000 characters of content, When the analysis is triggered, Then the LLM receives at most 15,000 characters of content.
  3. Given a successfully analyzed article, When the analysis is persisted, Then the model name, input token count, and output token count are recorded alongside the analysis.
  4. Given an article already analyzed (duplicate analysis_id FK), When analysis is triggered again, Then the system returns a failure result without overwriting the existing analysis.

User Story 2 - Topic-Mode Aware Tagging (Priority: P1) ​

The system classifies each article's tags according to the topic's tagging mode. For supervised topics, tags must come from a predefined set. For unsupervised topics, the LLM creates its own tag groups. Semi-supervised allows both.

Why this priority: Tag quality and consistency directly affect downstream filtering and user discovery. Incorrect mode handling corrupts the tag taxonomy.

Independent Test: Can be tested by running AnalyzeArticleUseCase for articles associated with topics in each of the three modes (SUPERVISED, SEMI_SUPERVISED, UNSUPERVISED) and asserting the correct tag group behavior.

Acceptance Scenarios:

  1. Given an article with a SUPERVISED topic and predefined tag groups in the database, When analysis runs, Then all returned tag keys are restricted to the predefined set.
  2. Given an article with an UNSUPERVISED topic, When analysis runs, Then new tag group names generated by the LLM are upserted to the database with embeddings.
  3. Given an article with a SEMI_SUPERVISED topic and some existing tag groups, When analysis runs, Then the LLM may reuse existing groups or create new ones; all new groups are upserted.
  4. Given an article with no topic_id, When analysis runs, Then all active topics are merged into the prompt and the LLM generates tag groups freely (auto-mode).
  5. Given a SUPERVISED topic with no tag groups in the database, When analysis runs, Then the system falls back to auto-mode (unsupervised prompt) and continues normally.

User Story 3 - Provider Fallback on Rate Limit or Failure (Priority: P2) ​

When the primary LLM provider is rate-limited or returns an error, the system automatically falls back to the next configured provider in priority order, ensuring analysis continues without operator intervention.

Why this priority: Provider outages or quota exhaustion are frequent in production. Without fallback, analysis stalls and creates backlogs.

Independent Test: Can be tested by mocking the primary provider to raise RateLimitExhausted and asserting that the secondary provider is called and returns a successful result.

Acceptance Scenarios:

  1. Given a primary provider that is rate-limited, When analysis is requested, Then the system automatically retries with the next provider in priority order without surfacing an error to the caller.
  2. Given a primary provider that raises a transient error, When the retry limit is reached, Then the system falls back to the next provider.
  3. Given all configured providers are exhausted or rate-limited, When analysis is requested, Then the system returns a failure result (not an exception) and logs that all providers failed.
  4. Given a provider that was demoted due to rate-limiting, When subsequent analysis requests are made, Then that provider is tried last until its rate limit window resets.
  5. Given a provider returns a malformed JSON response, When the response is received, Then the provider returns None, the system logs a warning, and moves to the next provider without retrying.

User Story 4 - Rate-Limit Enforcement Per Provider (Priority: P2) ​

The system enforces per-provider rate limits (requests per minute, tokens per minute, requests per day) using a sliding window approach. When a provider nears its limit, requests are briefly queued rather than dropped.

Why this priority: Without rate enforcement, the system triggers HTTP 429 errors or provider bans, which are harder to recover from than managed throttling.

Independent Test: Can be tested by configuring a provider with a low RPM limit and issuing bursts of analysis requests, asserting that requests are appropriately paced.

Acceptance Scenarios:

  1. Given a provider with an RPM limit of N, When N requests are made within 60 seconds, Then the (N+1)th request is held until the oldest request leaves the 60-second window.
  2. Given a provider with a TPM limit, When the token usage estimate exceeds the limit, Then new requests are delayed until enough tokens clear the window.
  3. Given a provider with an RPD (requests per day) limit that has been reached, When a new analysis request arrives, Then RateLimitExhausted is raised immediately and the provider is demoted in the fallback chain.
  4. Given a provider with no rate-limit configuration (rpm, tpm, rpd all null), When analysis is requested, Then requests pass through immediately with no throttling.

User Story 5 - Analysis Failure Isolation (Priority: P3) ​

Analysis failures are contained and reported without crashing the pipeline. Each article's analysis is independent; one failure does not block others.

Why this priority: Pipeline resilience is important but less critical than the core analysis behavior.

Independent Test: Can be tested by triggering all providers to fail and asserting AnalysisResult.success is False with a meaningful exception_type field.

Acceptance Scenarios:

  1. Given a provider that fails after all retries, When analysis is executed, Then AnalyzeArticleUseCase returns AnalysisResult(success=False) with exception_type and exception_message set.
  2. Given the analysis database write fails, When analysis content is returned from the LLM, Then AnalyzeArticleUseCase returns AnalysisResult(success=False) with the database error details; no exception propagates to the caller.
  3. Given a tag group embedding fails, When analysis is saved, Then the analysis is still persisted successfully and only a warning is logged.

Edge Cases ​

  • What happens when an article has empty or null content? → LLM receives empty string; expected to return minimal or error response from provider; system returns AnalysisResult(success=False).
  • What happens when a provider's API key environment variable is missing? → Provider initialization fails at bootstrap time; system raises ValueError and refuses to start.
  • What happens when the LLM returns a response missing required JSON fields? → Provider returns None after logging a validation warning; treated as a provider failure and fallback triggers.
  • What happens when the same article is analyzed twice concurrently? → The database unique constraint on article_id prevents duplicate analysis rows; the second write fails and returns AnalysisResult(success=False).
  • What happens when tag group embedding service is unavailable? → Analysis persists successfully without embeddings; upserted tag groups have no embedding vector.

Requirements (mandatory) ​

Functional Requirements ​

  • FR-001: The system MUST analyze a scraped article and produce a structured result containing summary, pain points, insights, innovations, and tag groups.
  • FR-002: The system MUST select the analysis prompt based on the article's topic tagging mode (SUPERVISED, SEMI_SUPERVISED, UNSUPERVISED, or no topic).
  • FR-003: In SUPERVISED mode, the system MUST constrain generated tags to the predefined tag group keys registered for the topic.
  • FR-004: In UNSUPERVISED and SEMI_SUPERVISED modes, the system MUST upsert newly generated tag group names to the database, including embedding vectors where an embedding service is available.
  • FR-005: The system MUST record which LLM model produced each analysis along with input and output token counts.
  • FR-006: The system MUST attempt analysis through multiple LLM providers in configured priority order, falling back automatically when a provider fails or is rate-limited.
  • FR-007: When a provider is rate-limited (RPD exhausted), the system MUST deprioritize it to the end of the fallback chain for subsequent requests.
  • FR-008: The system MUST enforce per-provider sliding-window rate limits (RPM, TPM) by queuing requests rather than immediately rejecting them.
  • FR-009: The system MUST enforce daily request limits (RPD) per provider and raise a specific rate-limit error when the limit is exceeded.
  • FR-010: The system MUST return a structured failure result (not raise an exception) when analysis cannot be completed, including the exception type and message.
  • FR-011: For ArXiv articles, the system MUST truncate analysis input to a maximum of 15,000 characters.
  • FR-012: The system MUST store analysis content and metadata in a persistent store, keyed uniquely to the article.
  • FR-013: Tag group embedding failures MUST NOT block analysis persistence; they are logged as warnings and the pipeline continues.
  • FR-014: The system MUST support providers with no rate-limit configuration, treating them as unlimited.

Key Entities ​

  • Analysis: The result of applying an LLM to an article. Contains content (summary, pain_points, insights, innovations, tag_groups) and metadata (model_used, input_tokens, output_tokens). One-to-one with Article.
  • AnalysisTagGroup: A grouping of tags generated or selected during analysis. Has a snake_case group key and a list of tag strings. May be predefined (SUPERVISED) or generated (UNSUPERVISED/SEMI_SUPERVISED).
  • LLMProvider: A configured backend (Claude, Gemini, OpenRouter) with a priority, model identifier, API credentials reference, and optional rate-limit parameters (RPM, TPM, RPD).
  • RateLimitWindow: A rolling 60-second window tracking request count and token consumption for a single provider. Resets naturally as events age out.
  • AnalysisPrompt: A rendered, topic-specific instruction sent to the LLM alongside article content. Three variants: auto (unsupervised), fixed (supervised), semi.

Success Criteria (mandatory) ​

Measurable Outcomes ​

  • SC-001: 95% of article analyses complete successfully end-to-end (LLM call + persistence) in a healthy pipeline run.
  • SC-002: When the primary provider is unavailable, the system falls back to a secondary provider without operator intervention and completes analysis within twice the normal latency.
  • SC-003: No more than 1 HTTP 429 rate-limit response is received per provider per minute when rate-limit configuration is correctly set.
  • SC-004: Analysis pipeline continues processing remaining articles even when up to 20% of individual analyses fail.
  • SC-005: Every persisted analysis record has a non-null model_used, input_tokens, and output_tokens value.
  • SC-006: In SUPERVISED mode, 100% of generated tag keys match the predefined set for the topic; no out-of-vocabulary tags appear in the database.

Assumptions ​

  • The LLM provider configurations (API keys, models, rate limits) are managed in the database and loaded at pipeline startup; runtime changes to provider config require a restart.
  • Each article is analyzed exactly once under normal operation; re-analysis of failed articles is handled by a separate retry mechanism (outside this feature scope).
  • The embedding service used for tag group vectors is an optional dependency; the analysis pipeline is fully functional without it.
  • The pipeline is single-threaded within a single process; rate-limit state is in-process only and is not shared across multiple pipeline instances.
  • Content truncation rules (15,000 character cap for ArXiv) are fixed and not configurable at runtime.
  • English is the primary analysis language; translated variants are produced by a separate translation pipeline (004-translation spec).
  • A topic's tagging mode is fixed at analysis time; mode changes after articles are analyzed do not retroactively re-tag existing analyses.