Your Data Pipeline Is One YAML File
The open-source command line for production collection: describe sources, filters, and storage in YAML — the CLI collects in batch, on schedule, straight into your database.
Why Teams Pick It
Declarative YAML Pipelines
Chain sources with dependencies, set filters, storage, and schedule in one file. Six templates cover common patterns; incremental runs track a cursor so nothing repeats.
Batch at Production Scale
Parallel execution, per-source error strategies, and resume after interruption — batches of 10,000+ records run unattended and pick up where they stopped.
Your Database, Not Ours
Results load straight into SQLite, PostgreSQL, or ClickHouse — or land as Parquet files — then answer to plain SQL through DuckDB.
LLM Enrichment Built In
Classify, summarize, enrich, and deduplicate records as a pipeline step — sentiment on mentions, categories on posts — without exporting to another tool.
Agent-Ready by Design
Your agent discovers endpoints, writes the YAML, prices the run with --dry-run, and executes — clean JSON responses and exit codes it can act on.
Only Config Touches Context
Collection runs locally; records never pass through the model. Workflow tools that push every record through context burn ~500K tokens per 1,000 records; the CLI holds at ~1K — an efficiency gain of 500×, rising to 50,000× at 100,000 records.
How It Works
Install the CLI
One package, MIT-licensed. Set your API key and pull the endpoint schema — the whole setup is three commands.
pip install anysite-cli
Describe the Pipeline in YAML
Name your sources, chain them with depends_on, set filters and storage. Or brief your agent in plain language and let it write the file.
anysite dataset init prospect-pipeline
Run It — or Put It on a Schedule
Preview cost with a dry run, execute, then hand it to cron. Incremental mode collects only what changed since the last run.
anysite dataset collect pipeline.yaml --dry-run
Data Lands in Your Database
Records arrive in SQLite, PostgreSQL, or ClickHouse, ready for SQL — and for LLM steps like classify and summarize when you want a read, not rows.
anysite llm classify pipeline.yaml --source posts
The Pipeline Is One YAML File
name: prospect-pipeline
sources:
target_companies:
endpoint: /api/companies/search
input:
industry: "SaaS"
employee_count: "51-200"
parallel: 3
decision_makers:
endpoint: /api/company/employees
depends_on: target_companies
input:
company: ${target_companies.urn}
keywords: "VP Sales, Director Sales"
on_error: skip
recent_posts:
endpoint: /api/people/posts
depends_on: decision_makers
storage:
format: parquet
path: ./data/prospectsThe live example, trimmed for length — endpoint paths shortened to generic form.
# Preview costs before running anysite dataset collect pipeline.yaml --dry-run # Execute the full pipeline anysite dataset collect pipeline.yaml # Run incremental updates anysite dataset collect pipeline.yaml --incremental # Query results with SQL anysite dataset query pipeline.yaml \ --sql "SELECT * FROM decision_makers WHERE title LIKE '%CTO%'" # Classify posts with LLM anysite llm classify pipeline.yaml --source recent_posts \ --categories "product_update,hiring,thought_leadership"
One file, five commands — preview the cost, run, refresh, query, classify.
# Configure your API key anysite config set api_key YOUR_API_KEY # Update the schema anysite schema update # Make your first request anysite api /api/people/profile user=satyanadella # Create your first pipeline anysite dataset init my-first-pipeline
Step zero is pip install anysite-cli. Endpoint path shortened to generic form.
What You Ask. What You Get.
“Every morning, collect our three competitors’ new posts on Twitter/X and Reddit, classify by topic, and flag spikes.”
What happens: A scheduled pipeline collects each source in parallel, dataset diff picks out what changed since yesterday, and an LLM step labels every post by topic and sentiment.
You get: A daily table in PostgreSQL — new posts, topic, sentiment, engagement — plus a webhook ping when the run completes.
“Build a list of seed-stage companies from Y Combinator, pull the founders, and keep it fresh every Monday.”
What happens: The pipeline chains company search into founder lookups with depends_on, runs incrementally, and skips records it already holds.
You get: A deduplicated founders table in SQLite with company, batch, role, and profile fields — current as of this morning.
Common Questions
They run on the same engine. MCP is conversational — explore sources, prototype a workflow, ask one-off questions from Claude or Cursor. The CLI is operational — the same workflow as a YAML pipeline, batch-sized, scheduled, and writing to your database. Most teams explore with MCP, then execute with the CLI.
The Web Is the Database. The Agent Is the Data Engineer.
Open source, MIT-licensed, one pip install away. Describe the pipeline tonight; query the table tomorrow.