Add webscraper MCP server (7 tools, httpx+bs4, 15+ tests)

2026-04-03 13:40:50 +02:00
parent 6623fe0337
commit 38a2b89bd3
9 changed files with 2266 additions and 0 deletions
@@ -0,0 +1,42 @@
+# Webscraper MCP Server
+
+MCP server for web scraping operations: fetch pages, extract links/tables, parse sitemaps.
+
+## Tools
+
+- `webscraper_fetch(url, max_chars=5000)` — Title + markdown body + metadata
+- `webscraper_fetch_links(url, deduplicate=True)` — Extract all hrefs
+- `webscraper_fetch_tables(url)` — HTML tables as markdown
+- `webscraper_fetch_all(url, max_chars=5000)` — Everything in one call
+- `webscraper_fetch_section(url, selector)` — Specific CSS section
+- `webscraper_fetch_meta(url)` — Title, description, OG tags
+- `webscraper_fetch_sitemap(url, max_urls=100)` — Sitemap URL list
+
+## Stack
+
+- httpx (HTTP client)
+- BeautifulSoup4 + lxml (HTML parsing)
+- html2text (HTML to markdown)
+
+## Run
+
+```bash
+./run.sh  # uv sync && uv run src/server.py
+```
+
+## Tests
+
+```bash
+uv run pytest tests/ --cov=src
+```
+
+## MCP Config
+
+Add to `.roo/mcp.json`:
+
+```json
+"webscraper": {
+  "command": "uv",
+  "args": ["run", "--directory", "/home/pplate/pi_mcps/webscraper", "src/server.py"]
+}
+```