tinyweb/README.md

# TinyWeb

A personal, decentralized search engine built on the [Reticulum](https://reticulum.network/) mesh network. You save pages you find. They are stored locally and shared over a mesh network so other people can find them too.

## Contents

- [About this project](#about-this-project)
- [Features](#features)
- [Performance & Scale](#performance--scale)
- [Getting started](#getting-started)
- [Data storage](#data-storage)
- [Remote gateway](#remote-gateway)
- [How it works](#how-it-works)
- [Known rough edges](#known-rough-edges)
- [Forum plugin](#forum-plugin)
- [Project structure](#project-structure)
- [Security](#security)
- [Maintenance](#maintenance)
- [Dependencies](#dependencies)

## About this project

Code generated by LLMs. Built by one person.

## Features

- **Personal search index** — Save pages you find valuable, search them with full-text search (SQLite FTS5)
- **Tagging** — Organize saved pages with comma-separated tags
- **Bookmarklet** — One-click indexing from any browser tab
- **Subscriptions** — Subscribe to friends' TinyWeb instances over Reticulum and search their indexes alongside yours
- **Custom templates** — Full HTML/CSS/JS template editor to personalize your instance
- **Import/export** — JSON-based backup and restore
- **Mesh-native** — Works over Reticulum without the internet; encrypted and decentralized by default
- **Forum plugin** — Optional link-sharing discussion board over the mesh (see Forum section below)

### What sharing means

Once a subscriber syncs your pages, you have no control over their copy.
No revocation, no DRM, no expiry. If you shared a page, assume it's out
there permanently.

Tag-based sharing (`private`, `public`) is advisory. The software
respects these tags in its API response, but there is no technical
mechanism preventing a subscriber from re-sharing your data.

Deletion is local only. Removing a page from your index does not
propagate to subscribers.

## Performance & Scale

### Search Speed

| Pages indexed | Search speed | Notes |
|--------------|-------------|-------|
| 1,000 | ~50ms | Fast local FTS5 |
| 10,000 | ~50-100ms | Full-text search |
| 100,000 | ~100-200ms | Combined BM25 + semantic |
| 500,000 | ~200-400ms | With semantic enabled |
| 1,000,000 | ~300-500ms | Hybrid search |

*Times are estimates for combined BM25 + semantic search. Actual performance varies by hardware, storage type (SSD/HDD), and search complexity.*

### Concurrent Connections

- Database pool: 16 simultaneous connections
- Suitable for single-user + a few subscriptions

### Export

- Paginated at 10,000 pages per request
- Use `?batch=N` to export in chunks: `/export?batch=0`, `/export?batch=1`, etc.

## Getting started

**Requirements:** Python 3.10+ and pip.

```bash
git clone https://codeberg.org/tinyweb/tinyweb.git
cd tinyweb
pip install -r requirements.txt
python app.py
```

Open `http://127.0.0.1:8080` in your browser. The web UI is localhost-only by default.

Your destination hash is printed on startup — share it with friends so they can subscribe to your index.

### Command line options

```bash
python app.py -p 9000        # Use port 9000 instead of default 8080
python app.py --bind 0.0.0.0 # Expose to your LAN (no auth — see Security)
```

### Keeping it running

```bash
# Terminal session (closes when you log out):
python app.py

# Background with tmux:
tmux new-session -d -s tinyweb 'python app.py'

# Background with nohup:
nohup python app.py &
```

### Docker

A `docker-compose.yml` is included for containerized setups. Build and run:

```bash
docker compose up -d
```

Data persists in the `tinyweb-data` named volume. On Linux with LAN auto-discovery it works as-is; on macOS or remote setups, see `docker-compose.yml` comments for TCP transport config.

## Storage Estimates

Average web page content is ~15KB per page:

| Pages | Database | Embeddings* | Total |
|-------|----------|------------|-------|
| 10,000 | 150MB | 80MB | ~250MB |
| 100,000 | 1.5GB | 800MB | ~2.5GB |
| 500,000 | 7.5GB | 4GB | ~12GB |
| 1,000,000 | 15GB | 8GB | ~25GB |

*Embeddings require semantic search to be enabled. With compression enabled (Settings > Search > AI), embeddings use ~50% less storage.

Enable optional compression in Settings > Search > AI to reduce embedding storage by ~50%.

## Data storage

### Local (Python/binary)

Your data is stored in `~/.tinyweb/`:

| File | Description |
|------|-------------|
| `index.db` | SQLite database with your indexed pages |
| `tinyweb_identity` | Your Reticulum identity (keep safe!) |
| `forum.db` | Forum plugin database (only if forum is enabled) |
| `models/` | Downloaded AI models for semantic search |
| `index.hnsw` | Semantic search index |

This allows your data to persist between upgrades and stay separate from the application.

### Backups

Back up the whole `~/.tinyweb/` directory periodically. The two files that matter:

- **`tinyweb_identity`** is your permanent mesh identity. If you lose it, your destination hash changes and every subscriber has to re-subscribe to the new one. Keep it somewhere you trust; the file is `0600` by default.
- **`index.db`** is your full reading history — every page, note, tag, and synced remote page. Losing it loses everything you've curated.
- **`forum.db`** (if the forum plugin is enabled) — all threads, posts, upvotes, and moderation settings. Losing it loses your forum data.

`models/` and `index.hnsw` are re-derivable (the model will re-download, and the HNSW index rebuilds from the database on next startup with semantic search enabled) so they don't need to be backed up.

The `/export` page produces a JSON dump of your pages. It's a migration aid — it doesn't preserve your identity file, your custom template, or subscription state. A full restore needs a copy of `~/.tinyweb/`.

## Remote gateway

To browse a remote TinyWeb instance without running your own index:

```bash
python -m tinyweb.gateway <destination_hash>
```

This connects over Reticulum and serves the remote instance at `http://localhost:8080`.

## How it works

1. **Save pages** — Use the `/add` form or the bookmarklet (found on `/style`) to index any URL
2. **Search** — Full-text search across your saved pages, linked pages from trusted sites, and synced subscriptions
3. **Subscribe** — Add a friend's destination hash on `/subscriptions` to sync their shared index
4. **Customize** — Edit your site name, HTML template, and sharing settings on `/style`

## Known rough edges

- Single-user UI
- All-or-nothing sharing per mode
- Manual sync (except optional forum auto-sync)
- No recrawling
- No browser extension — bookmarklet only
- Desktop-oriented
- JSON-only import
- Forum threads prune after 30 days by default
- Forum discovery is through the subscription graph rather than topic-based blooms
- Best-effort maintenance

## Forum plugin

TinyWeb ships with an optional [tinyweb-forum](https://codeberg.org/tinyweb/tinyweb-forum) plugin — a decentralized link-sharing discussion board that runs in-process alongside TinyWeb.

### Install

```bash
pip install tinyweb-forum
```

Enable it on the `/style` page under "Forum". A "Forum" link will appear in the navigation bar.

### How it works

- Threads and posts are stored in `~/.tinyweb/forum.db` (separate from your search index)
- Instances are discovered automatically via mesh announces — no manual setup needed
- Sync is manual by default: click "sync now" on the forum page. Auto-sync every 5 minutes is optional (toggle on moderation page)
- At scale, sync uses epidemic gossip: 20 random peers per cycle, converging globally within ~O(log N) cycles
- Authors are identified by a short pseudonymous identity hash (no accounts, no sign-up)
- Auto-discovery can be disabled in the moderation page
- Threads are auto-pruned after 30 days (configurable, or set to 0 to keep everything)
- Moderation is local: block authors, mute threads, keyword filters, and gossip block lists with peers (auto-block after 3 peer reports)
- **Trust circle** — Content is gated by a trust graph derived from your TinyWeb subscriptions. Each subscribed peer's forum content (and their transitive trust network) enters your view automatically. Blocking a peer cascades: their downstream trust network is removed from your view.

For full feature docs, see the [tinyweb-forum README](https://codeberg.org/tinyweb/tinyweb-forum).

## Project structure

```
app.py                  — Entry point (shim, imports from tinyweb.app)
pyproject.toml          — Package configuration (src layout)
src/tinyweb/
  __init__.py           — Package marker
  app.py                — Boots Reticulum, starts HTTP gateway
  db.py                 — SQLite database, FTS5, URL fetching, SSRF protection
  gateway.py            — HTTP-to-RNS bridge (local or remote dispatch)
  templates.py          — HTML template rendering and escaping
  embeddings.py         — Semantic search: ONNX, HNSW, reranking
  rns_client.py         — Reticulum client for fetching remote site lists
  handlers/
    __init__.py         — Dispatch logic + re-exports
    _helpers.py         — CSRF, FTS sanitizer, pagination, response builders
    search.py           — Search (BM25, hybrid, trusted/remote results)
    pages.py            — Add/edit/delete/bulk/bookmark handlers
    subscriptions.py    — Sync, sharing, API, subscription CRUD
    customize.py        — Settings form, about page
    tags.py             — Tag list and browse
    data.py             — Export, import, semantic reindex
themes/                 — Saved HTML templates (e.g. default.html, junimo.html)
```

## Security

**The web UI has no authentication.** It is bound to `127.0.0.1` by default, so only processes on the local machine can reach it. If you pass `--bind 0.0.0.0` (or run inside a container with a published port), anyone who can reach that address can fully control your instance — reading private entries, changing settings, and modifying the HTML template (which runs in your browser). Put TinyWeb behind a reverse proxy with auth before exposing it beyond localhost.

Other hardening measures:

- **CSRF protection** — All POST forms use per-session tokens via double-submit cookies
- **SSRF prevention** — URL fetching validates hostnames against private IP ranges, with redirect re-validation
- **FTS5 injection prevention** — Search queries are sanitized before passing to SQLite MATCH
- **Content Security Policy** — CSP headers on all HTML responses restrict script/style/frame sources
- **XSS escaping** — All user-supplied content is HTML-escaped before rendering
- **Bookmark authentication** — The bookmarklet endpoint requires a secret token
- **Identity file protection** — The Reticulum identity key is restricted to owner-only permissions (0600)
- **Forum caveats** — See [tinyweb-forum Security](https://codeberg.org/tinyweb/tinyweb-forum#security) for forum-specific risks (voluntary retractions, block gossip manipulation, no rate limiting)

#### Not hardened

- No HTTPS
- No authentication
- No encryption-at-rest
- No rate limiting
- Bookmarklet token sent as a plain URL parameter
- Forum moderation is gossip-based — block lists can be manipulated

## Maintenance

### Database Vacuum

Over time, deleted pages leave empty space in the database. Run the vacuum tool periodically to reclaim space:

1. Go to `/style` in your browser
2. Click "vacuum database" at the bottom of the page

### Optional Compression

To reduce storage for semantic search embeddings (~50% savings):

1. Go to `/style` > Search > AI
2. Enable "compress embeddings"
3. Re-index your existing pages for the compression to apply to existing embeddings

## Dependencies

- [requests](https://docs.python-requests.org/) — HTTP fetching
- [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing and link extraction
- [rns](https://reticulum.network/) — Reticulum mesh networking