crawly: the literature assistant

Search 8+ Open Access scholarly databases simultaneously, download full-text PDFs with automatic DOI resolution, export BibTeX citations, and generate AI literature reviews using local Ollama / LM Studio or cloud LLMs — configured via config.json and packaged as a zero-dependency executable for Windows & Linux/WSL.

⚡ Fast One-Line Install 📊 Live Sample Report 📦 Releases & Binaries GitHub

╭──────────────────────────────────────────────────────────────────────────────╮ │ crawly: the literature assistant — Multi-Source Academic Search, Open-Access │ │ PDF Downloader & AI Literature Review │ │ Open Databases: OpenAlex · Unpaywall · Semantic Scholar · arXiv · Europe │ │ PMC · Crossref · DOAJ · HAL · OpenAIRE │ │ Compliance Mode: ● Strict Legal Open-Access Mode (Default) │ │ AI Engine: Ollama connected (http://localhost:11434) - Models: llama3.2 │ │ Config File: config.json │ ╰──────────────────────────────────────────────────────────────────────────────╯ Main Menu: 1. Search & Discover Papers (OpenAlex, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE) 2. Download Full-Text PDFs (Legal Open Access Default + Optional Risk-Gated Third-Party Fallback) 3. AI Literature Review & Q&A (Mode 1: Summaries · Mode 2: Literature Review Matrix · Mode 3: Chat with PDFs) 4. Run End-to-End Automated Pipeline (Search ➔ Download PDFs ➔ AI Literature Review) 5. Configure AI Models & Settings (Updates config.json: Ollama, LM Studio, OpenAI API, Gemini, Anthropic) 6. Export Dataset to BibTeX (.bib) (For Zotero, Mendeley, EndNote & Overleaf/LaTeX) 7. Run System & API Health Doctor

Zero-Setup Downloads for Windows & Linux / WSL

Normal users never need to run pip install or touch requirements.txt. Run the one-line terminal installer or download standalone executables directly from GitHub:

🐧 Linux / macOS / WSL (One-Liner)

Downloads the portable release, creates a symlink in ~/.local/bin, and configures PATH automatically.

curl -fsSL https://lokeshsk.github.io/crawly/install.sh | bash

🪟 Windows PowerShell (One-Liner)

Extracts the release into %LOCALAPPDATA%\crawly and configures User PATH automatically.

irm https://lokeshsk.github.io/crawly/install.ps1 | iex

🐍 PyPI Package (Standard Python Virtual Environments)

Install directly into any Python 3.9+ virtual environment via standard pip. Automatically configures the global crawly command.

pip install crawly-ai

📦 Prefer Manual Downloads or Standalone Executables?

Looking for the raw native .exe, native Linux x86_64 binary, single-file crawly.pyz, or universal cross-platform .zip bundle? Grab all pre-compiled assets directly from the official GitHub Release.

🚀 Download Standalone Binaries on GitHub Releases ⭐ Star Repository on GitHub

Built for Serious Literature Reviews

Everything a researcher needs from initial keyword search to citation networks, full-text synthesis, and Zotero integration.

🌐 8+ Legal Open-Access APIs

Queries OpenAlex, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL Open Archive, and OpenAIRE concurrently with live rich progress bars.

📥 Smart DOI & PDF Waterfall

Resolves missing DOIs from existing CSVs via URL parsing, OpenAlex, and Crossref, then checks Unpaywall, arXiv preprints, Europe PMC, and OpenAlex OA with strict %PDF- verification.

🌲 Citation Graph Snowballing

Given an anchor DOI or paper, crawly snowball traces foundational works cited (backward) and recent papers citing it (forward) via OpenAlex & Semantic Scholar.

📊 Interactive HTML Dashboard

Run crawly report to export a self-contained interactive HTML review dashboard with live search, filters, paper cards, dark mode, and BibTeX copy.

🌐 Open Sample Report Demo ➔

🗂️ Direct Local Zotero Sync

Run crawly zotero to sync your discovered dataset and downloaded PDFs directly into your local Zotero desktop library via connector API or linked PDF bundle.

🎯 Research Gaps & Roadmap

Mode gaps focuses on methodology constraints, assumptions, and limitations across papers to produce an actionable matrix of open challenges for researchers.

🐕 Literature Watchdog Alerts

Run crawly watch to scan recent preprints and journal articles from the past 7 days matching your research topics and generate an alert digest.

⚙️ Single `config.json` Setup

Configure Ollama, LM Studio, OpenAI-compatible servers (vLLM, Groq, DeepSeek), Gemini, Claude, temperature, and limits in one labeled file.

📚 Automatic BibTeX (.bib) Export

Every search and download batch automatically exports a clean .bib file ready for Overleaf, LaTeX, Zotero, Mendeley, and JabRef.

Interactive CLI Command Builder

Customize your research workflow and copy the exact command for Windows or Linux/WSL.

Workflow Action

Keywords / DOI / CSV Dataset

AI Review Mode

AI Backend

./dist/crawly pipeline "synthetic aperture radar deep learning" -o sar_review --limit 10 --mode all

⚠ Legal Compliance Framework & Non-Liability Disclaimer

Not Recommended by Developers: The developers, authors, and contributors of crawly (crawly: the literature assistant) do NOT recommend, endorse, or encourage the use of Sci-Hub, shadow libraries, or automated scraping of websites that prohibit automated access or bypass copyright/paywall protections.
Strict Legal Open-Access Default: By default, crawly operates strictly within a legal framework using official Open Access APIs and repositories (OpenAlex, Unpaywall, Semantic Scholar, arXiv, Europe PMC, Crossref, DOAJ, HAL, OpenAIRE).
Use at Your Own Sole Risk & Zero Developer Liability: Any optional feature that goes beyond official Open Access APIs is disabled by default and guarded by an explicit interactive risk-acceptance prompt. If a user chooses to enable or continue with such external websites or tools, they do so strictly at their own sole risk and legal responsibility, and the developers shall not be held liable for any copyright claims, Terms of Service violations, ISP restrictions, or any other harm or damages.