ai-tools-searxng-firecrawl-hhem-api/README.md

# AI Tools Stack — SearXNG + Firecrawl + HHEM

Self-hosted Docker Compose stack for the DGX Spark (ARM64/aarch64, CPU-only).  
Runs alongside the existing n8n automation stack and exposes three services for agent use.

## Quick Start

```bash
cd ~/Docker/searXNG\ Firecrawl\ HHEM\ \ 

# First run: build Firecrawl from source (takes ~5–10 min)
docker compose build

# Start everything
docker compose up -d

# Check status
docker compose ps
```

## Services & Ports

| Service    | Port | Container name    | Purpose |
|------------|------|-------------------|---------|
| SearXNG    | 8888 | `searxng`         | Meta-search (JSON API) |
| Firecrawl  | 3002 | `firecrawl-api`   | Web scraper / crawler |
| HHEM API   | 8881 | `hhem-api`        | Hallucination evaluator |

All services share the `ai-tools` Docker network.  
n8n reaches them at `http://host.docker.internal:<port>` (or `http://<DGX-IP>:<port>`).

---

## 1. SearXNG — Meta-Search

**Base URL:** `http://localhost:8888`

### Search (JSON)

```
GET /search?q=<query>&format=json&categories=general
```

**Example curl:**
```bash
curl "http://localhost:8888/search?q=Claude+AI&format=json" | jq '.results[0]'
```

**Response shape:**
```json
{
  "query": "Claude AI",
  "results": [
    {
      "title": "...",
      "url": "https://...",
      "content": "...",
      "engine": "google",
      "score": 1.0
    }
  ]
}
```

### n8n HTTP Request node

| Field | Value |
|-------|-------|
| Method | GET |
| URL | `http://host.docker.internal:8888/search` |
| Query params | `q` = `{{ $json.query }}`, `format` = `json`, `categories` = `general` |
| Response | JSON |

---

## 2. Firecrawl — Web Scraper

**Base URL:** `http://localhost:3002`  
**No API key required** (self-hosted, `USE_DB_AUTHENTICATION=false`).

### Scrape a single URL

```
POST /v1/scrape
Content-Type: application/json

{
  "url": "https://example.com",
  "formats": ["markdown"]
}
```

**Example curl:**
```bash
curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'
```

**Response shape:**
```json
{
  "success": true,
  "data": {
    "markdown": "# Example Domain\n\n...",
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com"
    }
  }
}
```

### Crawl a site (async)

```
POST /v1/crawl
Content-Type: application/json

{
  "url": "https://example.com",
  "limit": 10,
  "scrapeOptions": { "formats": ["markdown"] }
}
```

Returns a `jobId`. Poll `GET /v1/crawl/<jobId>` for results.

### n8n HTTP Request node (scrape)

| Field | Value |
|-------|-------|
| Method | POST |
| URL | `http://host.docker.internal:3002/v1/scrape` |
| Body (JSON) | `{"url": "{{ $json.url }}", "formats": ["markdown"]}` |
| Response | JSON |

### Queue admin UI

`http://localhost:3002/admin/aitools-bull-changeme/queues`

---

## 3. HHEM API — Hallucination Evaluator

**Base URL:** `http://localhost:8881`  
**Model:** `vectara/hallucination_evaluation_model` (183 M BERT-based, CPU-only)

> On first start the model is downloaded from HuggingFace (~700 MB).  
> Cached in the `hhem-model-cache` volume for subsequent restarts.

### Score

```
POST /score
Content-Type: application/json

{
  "source": "The Eiffel Tower is in Paris, France.",
  "generated": "The Eiffel Tower is located in Paris."
}
```

**Response:**
```json
{
  "score": 0.9741,
  "label": "grounded"
}
```

| Score range | Meaning |
|-------------|---------|
| > 0.5 | **Grounded** — generated text is faithful to source |
| ≤ 0.5 | **Hallucinated** — generated text contradicts or invents facts |

**Health check:**
```bash
curl http://localhost:8881/health
# {"status":"ok","model":"vectara/hallucination_evaluation_model","loaded":true}
```

### n8n HTTP Request node

| Field | Value |
|-------|-------|
| Method | POST |
| URL | `http://host.docker.internal:8881/score` |
| Body (JSON) | `{"source": "{{ $json.scrapedContent }}", "generated": "{{ $json.llmAnswer }}"}` |
| Response | JSON |

---

## n8n Agentic Pipeline Example

This stack is designed for a **Search → Scrape → Validate** pipeline:

```
1. [HTTP Request] → SearXNG: search for the user's query, extract top URL
2. [HTTP Request] → Firecrawl /v1/scrape: scrape the top result into markdown
3. [LLM Node]    → Generate an answer grounded in the scraped content
4. [HTTP Request] → HHEM /score: validate the answer against the scraped source
5. [If]          → Route: score > 0.5 → return answer | score ≤ 0.5 → flag/retry
```

### n8n expression for step 4 body:
```json
{
  "source": "{{ $('Firecrawl Scrape').item.json.data.markdown }}",
  "generated": "{{ $('LLM').item.json.text }}"
}
```

---

## Build Notes (ARM64 / aarch64)

- **SearXNG**: official image is multi-arch (ARM64 supported natively).
- **Redis**: `redis:7-alpine` is multi-arch.
- **Firecrawl**: built from source at `/home/sparky/Docker/firecrawl/git/firecrawl/`.  
  Dockerfile targets Node 22 slim which supports ARM64.
- **HHEM API**: `python:3.11-slim` + PyTorch CPU. PyPI provides `linux_aarch64` wheels for torch 2.x.

## Rebuilding after Firecrawl updates

```bash
git -C ~/Docker/firecrawl/git/firecrawl pull
docker compose build firecrawl-playwright firecrawl-api
docker compose up -d firecrawl-playwright firecrawl-api
```

## Connecting n8n to this stack

If n8n does not already have `host.docker.internal` available, add to the n8n service in its compose file:

```yaml
extra_hosts:
  - "host.docker.internal:host-gateway"
```

Then use `http://host.docker.internal:<port>` in all n8n HTTP Request nodes.