ai-tools-searxng-firecrawl-hhem-api/README.md

AI Tools Stack — SearXNG + Firecrawl + HHEM

Self-hosted Docker Compose stack for the DGX Spark (ARM64/aarch64, CPU-only).
Runs alongside the existing n8n automation stack and exposes three services for agent use.

Quick Start

cd ~/Docker/searXNG\ Firecrawl\ HHEM\ \ 

# First run: build Firecrawl from source (takes ~5–10 min)
docker compose build

# Start everything
docker compose up -d

# Check status
docker compose ps

Services & Ports

Service	Port	Container name	Purpose
SearXNG	8888	searxng	Meta-search (JSON API)
Firecrawl	3002	firecrawl-api	Web scraper / crawler
HHEM API	8881	hhem-api	Hallucination evaluator

All services share the ai-tools Docker network.
n8n reaches them at http://host.docker.internal:<port> (or http://<DGX-IP>:<port>).

---

1. SearXNG — Meta-Search

Base URL: http://localhost:8888

Search (JSON)

GET /search?q=<query>&format=json&categories=general

Example curl:

curl "http://localhost:8888/search?q=Claude+AI&format=json" | jq '.results[0]'

Response shape:

{
  "query": "Claude AI",
  "results": [
    {
      "title": "...",
      "url": "https://...",
      "content": "...",
      "engine": "google",
      "score": 1.0
    }
  ]
}

n8n HTTP Request node

Field	Value
Method	GET
URL	http://host.docker.internal:8888/search
Query params	q = {{ $json.query }}, format = json, categories = general
Response	JSON

---

2. Firecrawl — Web Scraper

Base URL: http://localhost:3002
No API key required (self-hosted, USE_DB_AUTHENTICATION=false).

Scrape a single URL

POST /v1/scrape
Content-Type: application/json

{
  "url": "https://example.com",
  "formats": ["markdown"]
}

Example curl:

curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

Response shape:

{
  "success": true,
  "data": {
    "markdown": "# Example Domain\n\n...",
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com"
    }
  }
}

Crawl a site (async)

POST /v1/crawl
Content-Type: application/json

{
  "url": "https://example.com",
  "limit": 10,
  "scrapeOptions": { "formats": ["markdown"] }
}

Returns a jobId. Poll GET /v1/crawl/<jobId> for results.

n8n HTTP Request node (scrape)

Field	Value
Method	POST
URL	http://host.docker.internal:3002/v1/scrape
Body (JSON)	{"url": "{{ $json.url }}", "formats": ["markdown"]}
Response	JSON

Queue admin UI

http://localhost:3002/admin/aitools-bull-changeme/queues

---

3. HHEM API — Hallucination Evaluator

Base URL: http://localhost:8881
Model: vectara/hallucination_evaluation_model (183 M BERT-based, CPU-only)

On first start the model is downloaded from HuggingFace (~700 MB).
Cached in the hhem-model-cache volume for subsequent restarts.

Score

POST /score
Content-Type: application/json

{
  "source": "The Eiffel Tower is in Paris, France.",
  "generated": "The Eiffel Tower is located in Paris."
}

Response:

{
  "score": 0.9741,
  "label": "grounded"
}

Score range	Meaning
> 0.5	Grounded — generated text is faithful to source
≤ 0.5	Hallucinated — generated text contradicts or invents facts

Health check:

curl http://localhost:8881/health
# {"status":"ok","model":"vectara/hallucination_evaluation_model","loaded":true}

n8n HTTP Request node

Field	Value
Method	POST
URL	http://host.docker.internal:8881/score
Body (JSON)	{"source": "{{ $json.scrapedContent }}", "generated": "{{ $json.llmAnswer }}"}
Response	JSON

---

n8n Agentic Pipeline Example

This stack is designed for a Search → Scrape → Validate pipeline:

1. [HTTP Request] → SearXNG: search for the user's query, extract top URL
2. [HTTP Request] → Firecrawl /v1/scrape: scrape the top result into markdown
3. [LLM Node]    → Generate an answer grounded in the scraped content
4. [HTTP Request] → HHEM /score: validate the answer against the scraped source
5. [If]          → Route: score > 0.5 → return answer | score ≤ 0.5 → flag/retry

n8n expression for step 4 body:

{
  "source": "{{ $('Firecrawl Scrape').item.json.data.markdown }}",
  "generated": "{{ $('LLM').item.json.text }}"
}

---

Build Notes (ARM64 / aarch64)

• SearXNG: official image is multi-arch (ARM64 supported natively).
• Redis: redis:7-alpine is multi-arch.
• Firecrawl: built from source at /home/sparky/Docker/firecrawl/git/firecrawl/.
  Dockerfile targets Node 22 slim which supports ARM64.
• HHEM API: python:3.11-slim + PyTorch CPU. PyPI provides linux_aarch64 wheels for torch 2.x.

Rebuilding after Firecrawl updates

git -C ~/Docker/firecrawl/git/firecrawl pull
docker compose build firecrawl-playwright firecrawl-api
docker compose up -d firecrawl-playwright firecrawl-api

Connecting n8n to this stack

If n8n does not already have host.docker.internal available, add to the n8n service in its compose file:

extra_hosts:
  - "host.docker.internal:host-gateway"

Then use http://host.docker.internal:<port> in all n8n HTTP Request nodes.