Zain Dana Harpergather · research intake

Collect research material from hard sources.

Gather collects research material from web pages, including JavaScript-rendered ones, APIs, scholarly records, PDFs, scans, audio, video, feeds and local files. It saves each item with its source in a local corpus you can search and verify.

See it work, step by step: an animated walkthrough with the real commands and output from the repository.

gather · v2.5.0 · pip install gather-engine==2.5.0
Plate 01 Layered strata with arched gaps.

Sources it reads

Each source is an adapter. Every item it returns records the source, the reference, the method used, the time and a SHA-256 of the content, so you can tell a caption transcript from an OCR scan from a direct page read.

video captions, metadata and comments through yt-dlp
web the static HTML a server returns, without running JavaScript
browser JavaScript-rendered pages through a headless browser
feed · docs RSS or Atom feeds, and local files or folders
arxiv papers from the arXiv API by id or query
pdf · ocr text from PDFs through pdftotext and from scans through tesseract
transcribe audio transcripts through a Whisper-style CLI
api authenticated JSON APIs, with the token read from the environment and never written to disk

The browser adapter guards only the first navigation; the page then follows its own redirects and sub-requests. Do not point it at untrusted URLs on a network where internal services are reachable.

What a run does

gather run fetches from the sources you name, filters to your scope, stores items in a local corpus that removes duplicates by hash, and writes one record of the run. gather corpus verify re-hashes every stored item, and recall reports any item that is missing or changed. The core needs only the Python standard library; optional adapters call their external tool only when you select that source.

Install and try it

Version 2.5.0 installs from PyPI as gather-engine, and gives you the gather command. Optional adapters need their own tool installed: yt-dlp, pdftotext, headless chromium, tesseract or a Whisper-style CLI.

Version 1.8.1 added protection for retained corpus reads against directory replacement on unsupported Linux mounts. On WSL, keep corpora on native Linux storage or use native Windows Python: Windows-drive 9p/v9fs mounts are refused with UNSAFE_PATH. Python hosts can pass a retained same-process corpus descriptor; CLI and MCP continue to accept directory strings.

Install 1.9.1 or later. Every earlier release is inside the range of at least one of the four Gather advisories listed on the Security page. A tool found only inside the working folder now needs its absolute path in its GATHER_<TOOL> variable, and an MCP host that relied on the old gather.run or gather.pilot behavior adds a grant to its launch configuration.

Create sample.txt with this exact two-line body, then store it and select only the useful slice.

sample.txt body
Offline quickstart: Gather keeps acquired context separate from conclusions.
Decision fact: the release can select useful text from a stored corpus.

After the inspect command, replace ROW_REF and CORPUS_DIGEST with the values from the inspect JSON. 0:76 selects the first line of this fixture only.

$ pip install gather-engine==2.5.0
$ gather docs sample.txt --store corpus --json
$ gather corpus context corpus --json --excerpt-chars 90
$ gather corpus context corpus --json --select ROW_REF:0:76 --expect-digest CORPUS_DIGEST

Sample selected text: Offline quickstart: Gather keeps acquired context separate from conclusions.

Release evidence and context limits

Version 1.9.1, released on 27 September 2026, fixes two more advisories, GHSA-r38f-cr69-jpp8 (high) and GHSA-j6j7-39vh-qrp4 (medium). A PATH entry that reaches the working folder, whether it names the folder, spells it another way or links to it, no longer lets a program planted there start in place of a tool. The folder of the Python that runs Gather stays on PATH, and on Windows so do the Windows, System32 and SysWOW64 folders. When Gather runs from a filesystem root, or from the home folder or a folder above it, the guard drops only the entry naming that folder itself. On Windows, the file sources and every MCP path argument refuse network shares and device paths. Every version before 1.9.1 is affected by at least one of the four advisories, so install 1.9.1 or later.

Version 1.9.0, released on 26 September 2026, fixes two high-severity security advisories, GHSA-pxvv-rg3f-4v5w and GHSA-r4f3-9xrf-72m5. Before gather.run or gather.pilot runs a command, reaches a network source or sends a credential, the MCP server now needs a grant given at launch. Every external tool now starts from an absolute path in a private folder. Every version before 1.9.0 is affected by both.

New corpus writes preserve exact UTF-8 source text while selection uses a readable LF-normalized view. Selections stay pinned to the witnessed corpus digest you pass with --expect-digest.

The 1.9.1 wheel installed from PyPI on 27 September 2026 and ran the quickstart above. The checks below were run on earlier releases.

The public 1.8.1 wheel passed isolated Windows CLI, MCP, retained-descriptor and tamper checks. On WSL x86_64, native reads passed and an actual 9p directory-replacement attempt was refused; descendant-directory and leaf controls were instrumented. The downloaded packages match the release commit: 64 Python modules and 133 tracked source archive files. Linux mount classification uses the opened descriptor and refuses unavailable mount information. Other POSIX platforms retain no-follow and type checks without a mount-detection claim.

The earlier released 1.7.1 wheel smoke run selected a newline-spanning DECISION/part fixture through both CLI and MCP. The payload used schema gather.readable-context/v1, returned verified: true, kept source SHA-256 b1736b2f3f466fdf55d900ccf4c28ba9a2170c60914ede77173873f447dba6d3 separate from readable view SHA-256 bfac793415f57cdb3aadb174993a58dc4767abe8005b78855daae6ab110b93cf, and produced selection digest 0450040a624224de25249cc42f1a68099898d8d8accf68d286e71c4771b7324b.

The publication receipt verifies the GitHub release, PyPI files, GitHub and PyPI download-back hashes, a clean Windows wheel install, CLI and MCP context selection, and tamper refusal. Selection offsets belong to the readable view. To detect changed storage later, pin the corpus digest now.

Recorded demo → 29-second demo and full workflow  ·  GitHub → v2.5.0 release  ·  Source → HarperZ9/gather  ·  PyPI → gather-engine 2.5.0  ·  ← all flagships  ·  the index