Technical Whitepaper

Lucid is a news analysis platform that helps readers see how different outlets cover the same event. It gathers articles from many publishers and presents clear summaries and context so people can compare perspectives. The approach draws on research showing that diverse sources reveal a fuller picture and that bias often appears through language and framing rather than a single simple label.

Type
Press release
Published
2026-03-14
Last updated
2026-08-06
Reading time
12 min

Lucid is a transparency‐oriented media intelligence platform that aggregates coverage of the same event from multiple outlets and enriches it with explanatory context. Its goal is to help readers “see the whole picture” rather than feed them content for maximum engagement. Lucid’s design emphasises user agency, legibility, and neutrality: it surfaces raw source text alongside analysis cues (such as summaries, bias indicators, and narrative framing), but does not tell users what to think. Under the hood, Lucid combines modern retrieval and AI techniques with privacy‐by‐design and explainability safeguards. In particular, Lucid adopts multi-stage retrieval and clustering methods (as in Tang et al.’s NEWSCOPE framework) to maximize diverse perspectives on an event. Its AI summarisation explicitly targets “emotional fingerprints” of bias (building on Liu et al.’s NeutraSum approach) so that generated summaries are closer to a neutral baseline. Finally, Lucid’s source‐context features are organized by a rigorous bias taxonomy (from Spinde et al.) that links computational signals to user-friendly categories. By following current best practices (NIST’s AI risk framework, EU privacy and AI regulations, ICO guidance on explainability, etc.), this whitepaper describes Lucid’s architecture, data flow, and principles in public‐safe terms, omitting proprietary thresholds or security details (see Appendix for deliberate redactions).

Why Lucid was built:

Modern information systems often optimise for capture and engagement (clicks, shares, outraged reactions), which can amplify sensational or polarising content. As studies show, false and emotion‐charged news can spread faster than neutral reporting, and algorithmic feeds tend to re-echo popular narratives (potentially reinforcing filter bubbles). In this attention economy, individual readers struggle to form independent judgements because they see only narrow slices of events. Lucid’s mission is to counteract this trend by intentionally surfac(ing) context and multiple viewpoints: instead of ranking by popularity, Lucid’s retrieval seeks complementary coverage of an event. The system exposes when language is subjective or emotionally loaded and shows how framing differs between sources. This helps readers recognise bias patterns (linguistic and contextual) rather than having those patterns hidden in the app’s black box.

Public product overview:

A user can, for example, enter a news topic or click on a cluster and see the following layers:

  • Raw articles: Lucid displays article titles, publisher attribution, and quotes or excerpts from each source’s reporting. Each excerpt clearly links back to the original article.
  • Publisher context panel: For each outlet, Lucid shows a brief profile drawn from public data (e.g. declared editorial stance, known ownership, correction policy). These are coarse signals, with links to their sources, intended as context rather than judgement. (For instance, if The Example Times is widely rated as “center” or “left‐center” on Allsides or MBFC, Lucid will note that with citation.)
  • Comparative cues: Lucid clusters articles that describe the same event. Within each cluster, users can easily compare how different outlets cover the story. Differences in headlines, highlighted quotes, or image choices are flagged. For instance, Lucid may tag a sentence in one article as a “claimed fact” and in another as “asserted speculation.”
  • Machine-generated assistance: Lucid also provides AI‐generated summaries and analysis, clearly marked as such. For example, a brief summary may appear at the top, and emotional or framing cues may highlight portions of text. An on-screen legend explains that “⋯ Lucid summarised this text and indicates frames like ‘aggressive tone’ here” or “we flagged these terms as subjective”.

What Lucid is not:

It does not assert absolute truth or replace careful reading. It does not censor content or editorialize on its own. It is not a personalized filter that only shows you like-minded news; on the contrary, its default is to show breadth and let you navigate. It is not a moderation layer — for example, it does not remove extremist content (though it may warn “This language is inflammatory”). Its “neutralisation” of bias (see below) only applies to its own summaries; it does not rewrite source articles or label them with ideological slurs. Lucid provides signals and context, but the reader always retains judgement. This stance aligns with emerging regulation that AI systems must disclose automated content and should assist rather than decide for users.

High-level system architecture and data flow

Lucid follows a pipeline architecture with clear modularity (allowing independent assessment and redaction of private details):

  • Ingestion: Lucid periodically fetches news articles via licensed news APIs and RSS feeds. Each article’s text and metadata (title, author, date, URL) are stored. Only necessary fields are kept to respect data minimisation: by default Lucid does not collect user tracking cookies or extraneous system logs. (Following the GDPR’s data protection by default principle, only user- provided or public information is stored.)
  • Normalization: Incoming articles are normalized to a common schema. Lucid identifies the publisher identity (resolving any aliases or mirror sites) and normalizes date formats and language codes. It also deduplicates exact copies (for example wire copy syndicated across sites) at this stage.
  • High-relevance retrieval (stage 1): When a user queries a topic, Lucid first uses traditional IR/dense‐retrieval (via an embedding index) to retrieve a broad set of relevant articles from recent content. This mirrors standard news search (like a query on “climate summit”). The result set prioritizes topical match.
  • Diversity re-ranking (stage 2): Following Tang et al.’s NEWSCOPE model, Lucid then applies a second pass aimed at maximizing perspective diversity. It segments the retrieved articles at the sentence level, computes semantic embeddings, clusters similar sentences, and then selects/boosts sentences and their source articles to cover distinct clusters. In effect, if five articles say the same key facts, Lucid will de-emphasize four of them and highlight an article that adds unique detail. This two-stage (relevance then diversity) approach ensures broad coverage without sacrificing core relevance. Lucid evaluates diversity internally using measures analogous to NEWSCOPE’s (e.g. pairwise semantic distance between kept sentences). The system’s retrieval indices and models can be updated continuously, but are kept separate from user-specific data to preserve privacy.
  • Storage: The system maintains a database of (1) raw article text and metadata, (2) processed analysis outputs (summaries, indicators), and (3) provenance links. For example, each summary has a timestamp and a link back to the source articles. No raw user queries or personal data are stored beyond immediate session needs.
  • Presentation layer: The front-end (a web app) requests content via an API. All AI outputs are rendered with provenance banners (“Lucid AI generated”) and hyperlinks to source. No sensitive internal endpoints are exposed. Rate limiting and abuse detection are in place but are considered internal operational details (and are omitted here).

This high-level flow—multi‐stage retrieval, enrichment, and transparent presentation—exposes what is coming from where at each step. Readers can trust Lucid’s intent because key inputs (source text, known publisher attributes) are plain, and analytical steps are accompanied by explanatory labels.

AI analysis and explainability

Types of analysis: Lucid applies natural language processing to each cluster of articles. Common tasks include:

  • Neutral summarization: Collates multiple articles and generates a concise summary. The summary is explicitly labeled “AI‐assisted summary” and is presented with a short explanation (e.g. “Summarises key facts without adding new opinions”).
  • Entity and event extraction: Identifies key entities (people, places, organisations) and events, highlighting them in context (with a note “This is an extracted entity”).
  • Rhetoric and framing cues: Detects subjective or charged language (e.g. “This phrase was flagged as emotionally negative”). This draws on linguistically interpretable features like sentiment or valence.
  • Coverage analysis: Indicates what percentage of sources mention a claim. For instance, if only one out of five articles claims a side-effect, Lucid may mark that claim as minority coverage.

All AI outputs are designed to be interpretable. For instance, when summarizing, Lucid’s system (built on a framework similar to Liu et al.’s NeutraSum) explicitly measures the summary’s emotional attributes against the source texts. By targeting and reducing the “Valence-Arousal-Dominance” (VAD) signature of biased language, Lucid makes sure its summaries are emotionally neutral. It may even report an “emotional bias score” so the reader can see that the summary’s tone is milder than the articles’ median tone.

Explainability: Lucid follows standard principles (e.g. UK ICO guidance) that outputs should come with context. Concretely:

  • Every highlight or summary is accompanied by an icon/tooltip like “AI: neutral summary” or “AI: framing analysis” to clearly differentiate machine inference from original text.
  • The system includes brief plain-language notes about methodology. For example, it might say: “To avoid echo chambers, this summary was generated by first finding diverse article fragments and then rewriting neutrally.” This kind of meta-commentary is limited to a sentence or two.
  • Uncertainty is surfaced where possible: if the analysis is low-confidence (e.g. the clustering is inconclusive), Lucid will show multiple possible summaries or simply note that the event is unsettled.

Lucid avoids “black box” claims. It does not display raw model internals or confidence percentages numerically. Instead, it visualizes reasoning steps: e.g. it might color-coded sentences to show which perspective cluster they came from. This is in line with research advice that transparency means clarity about what was done, not disclosing secret weights.

Limits and accountability: Lucid explicitly states what it does not do. It is not a fact-checker or content moderator. It does not guarantee that any information is true; it relies on sources, which are themselves fallible. It also does not assign an overall “bias score” to articles, since such an aggregate could be misleading. Instead, it breaks bias into categories (as below) so users understand nuance. The system supports user feedback: if a summary or tag seems wrong, users can flag it. Such feedback is used to retrain models internally but is not publicly shown.

Lucid’s development follows the NIST AI Risk Management Framework approach: documentation and testing of models, tracking of “versions” of analysis pipelines, and monitoring for unintended patterns. The platform publishes a transparency report (Appendix) that summarizes its datasets and methodologies in high-level terms.

Research Foundations Informing Lucid

Lucid’s analytical approach is informed by several strands of academic research in computational journalism, information retrieval, and bias detection. While the platform’s implementation is proprietary, the conceptual design draws on publicly available research that explores how news events can be reconstructed from multiple sources, how partisan language manifests in emotional patterns, and how media bias can be systematically categorised.

Diverse News Retrieval and Event Reconstruction

Research on diverse news retrieval demonstrates that effective event understanding requires more than retrieving the most relevant article. Instead, systems must balance relevance with diversity in order to surface distinct viewpoints and contextual information. Studies such as Uncovering the Bigger Picture: Comprehensive Event Understanding via Diverse News Retrieval show that retrieval systems relying only on semantic similarity tend to produce redundant results centred on the same narrative angle. By contrast, diversity-aware retrieval strategies reveal additional dimensions of an event, including criticism, policy implications, international reactions, and historical context.