Phase 2: Autonomous Research Agent · ~90 minutes · Python

Literature Retrieval

A hypothesis is cheap. Knowing whether someone already proved it is the expensive part. Build the retrieval layer that answers that question before the runner spins up a sandbox.

Hiring signal: Can build literature retrieval end to end

What you will learn

Model a small paper record with the fields the loop will read downstream.
Build a BM25 index over abstracts with stdlib data structures only.
Walk a citation graph to surface papers the lexical search missed.
Deduplicate hits across the lexical and graph passes by stable paper id.
Wrap two mock external APIs behind a single client so the upstream call site stays the same when real endpoints land.

Introduction

Type: Build Languages: Python Prerequisites: Phase 19 Track A lessons 20-29 Time: ~90 minutes

Objective

Learning objectives

Model a small paper record with the fields the loop will read downstream.
Build a BM25 index over abstracts with stdlib data structures only.
Walk a citation graph to surface papers the lexical search missed.
Deduplicate hits across the lexical and graph passes by stable paper id.
Wrap two mock external APIs behind a single client so the upstream call site stays the same when real endpoints land.

Why two retrieval passes

A keyword search over abstracts returns papers that share vocabulary with the query. That covers most of the surface. It misses two cases. The first is when the foundational paper uses different vocabulary; for example a query for "sparse attention" misses a paper titled "block selection in transformer routing." The second is when the relevant paper is a follow up that cites a known anchor; it is more efficient to find the anchor and walk forward than to brute force the abstract pool.

The lesson builds both passes. BM25 over abstracts catches the lexical hits. A citation graph traversal expands a seed set forward and backward by one or two hops. The union is deduplicated by paper id and ranked by a small combined score.

The Paper shape

Paper
  id          : str           (stable identifier, "p001" for the mock corpus)
  title       : str
  abstract    : str
  year        : int
  authors     : list[str]
  references  : list[str]     (paper ids this paper cites)
  citations   : list[str]     (paper ids that cite this paper)
  source      : str           (which mock api supplied it, "arxiv" or "s2")

The references and citations fields form the directed citation graph. The two mock APIs return overlapping but not identical fields, so the corpus loader unions them on id.

Unlock the full lesson

You've read the first 2 sections. The rest of this lesson covers The Problem, Build, Check Yourself, Key Terms & Next — plus a hands-on lab, quiz, and project artifact.

Create a free account to unlock Phase 0 and Phase 1 of every course — no credit card.