TinyWeb

A personal, decentralized search engine built on the Reticulum mesh network. You save pages you find. They are stored locally and shared over a mesh network so other people can find them too.

Contents

• Features
• Performance & Scale
• Getting started
• Data storage
• Remote gateway
• How it works
• Known rough edges
• Forum plugin
• Project structure
• Security
• Maintenance
• Dependencies

Features

• Personal search index — Save pages you find valuable, search them with full-text search (SQLite FTS5)
• RNS live browsing — Browse any RNS site through your instance via <base> tag injection, with LRU caching
• Unified add form — Add HTTP URLs or RNS destination hashes through a single form; auto-detection handles both
• Tagging — Organize saved pages with comma-separated tags
• Bookmarklet — One-click indexing from any browser tab
• Subscriptions — Subscribe to friends' TinyWeb instances over Reticulum and search their indexes alongside yours
• Custom templates — Full HTML/CSS/JS template editor to personalize your instance
• Import/export — JSON-based backup and restore
• Mesh-native — Works over Reticulum without the internet; encrypted and decentralized by default
• Forum plugin — Optional link-sharing discussion board over the mesh (see Forum section below)

What sharing means

Once a subscriber syncs your pages, you have no control over their copy. No revocation, no DRM, no expiry. If you shared a page, assume it's out there permanently.

Tag-based sharing (private, public) is advisory. The software respects these tags in its API response, but there is no technical mechanism preventing a subscriber from re-sharing your data.

Deletion is local only. Removing a page from your index does not propagate to subscribers.

Performance & Scale

Search Speed

Pages indexed	Search speed	Notes
1,000	~50ms	Fast local FTS5
10,000	~50-100ms	Full-text search
100,000	~100-200ms	Combined BM25 + semantic
500,000	~200-400ms	With semantic enabled
1,000,000	~300-500ms	Hybrid search

Times are estimates for combined BM25 + semantic search. Actual performance varies by hardware, storage type (SSD/HDD), and search complexity.

Concurrent Connections

• Database pool: 16 simultaneous connections
• Suitable for single-user + a few subscriptions

Export

• Paginated at 10,000 pages per request
• Use ?batch=N to export in chunks: /export?batch=0, /export?batch=1, etc.

Getting started

Requirements: Python 3.10+ and pip.

git clone https://codeberg.org/tinyweb/tinyweb.git
cd tinyweb
pip install -r requirements.txt
python app.py

Open http://127.0.0.1:8080 in your browser. The web UI is localhost-only by default.

Your destination hash is printed on startup — share it with friends so they can subscribe to your index.

Command line options

python app.py -p 9000        # Use port 9000 instead of default 8080
python app.py --bind 0.0.0.0 # Expose to your LAN (no auth — see Security)

Keeping it running

# Terminal session (closes when you log out):
python app.py

# Background with tmux:
tmux new-session -d -s tinyweb 'python app.py'

# Background with nohup:
nohup python app.py &

Docker

A docker-compose.yml is included for containerized setups. Build and run:

docker compose up -d

Data persists in the tinyweb-data named volume. On Linux with LAN auto-discovery it works as-is; on macOS or remote setups, see docker-compose.yml comments for TCP transport config.

Storage Estimates

Pages are stored as cleaned text (HTML tags stripped, boilerplate removed) — typically 5-15 KB per page across both HTTP and RNS sources:

Pages	Database	Embeddings*	Total
10,000	~100MB	80MB	~180MB
100,000	~1GB	800MB	~1.8GB
500,000	~5GB	4GB	~9GB
1,000,000	~10GB	8GB	~18GB

*Embeddings require semantic search to be enabled. With compression enabled (Settings > Search > AI), embeddings use ~50% less storage.

Enable optional compression in Settings > Search > AI to reduce embedding storage by ~50%.

Data storage

Local (Python/binary)

Your data is stored in ~/.tinyweb/:

File	Description
index.db	SQLite database with your indexed pages
tinyweb_identity	Your Reticulum identity (keep safe!)
forum.db	Forum plugin database (only if forum is enabled)
models/	Downloaded AI models for semantic search
index.hnsw	Semantic search index

This allows your data to persist between upgrades and stay separate from the application.

Backups

Back up the whole ~/.tinyweb/ directory periodically. The two files that matter:

• tinyweb_identity is your permanent mesh identity. If you lose it, your destination hash changes and every subscriber has to re-subscribe to the new one. Keep it somewhere you trust; the file is 0600 by default.
• index.db is your full reading history — every page, note, tag, and synced remote page. Losing it loses everything you've curated.
• forum.db (if the forum plugin is enabled) — all threads, posts, upvotes, and moderation settings. Losing it loses your forum data.

models/ and index.hnsw are re-derivable (the model will re-download, and the HNSW index rebuilds from the database on next startup with semantic search enabled) so they don't need to be backed up.

The /export page produces a JSON dump of your pages. It's a migration aid — it doesn't preserve your identity file, your custom template, or subscription state. A full restore needs a copy of ~/.tinyweb/.

Remote gateway

To browse a remote TinyWeb instance without running your own index:

python -m tinyweb.gateway <destination_hash>

This connects over Reticulum and serves the remote instance at http://localhost:8080.

How it works

1. Save pages — Use the /add form or the bookmarklet (found on /style) to index any URL
2. Search — Full-text search across your saved pages, linked pages from trusted sites, and synced subscriptions
3. Subscribe — Add a friend's destination hash on /subscriptions to sync their shared index
4. Customize — Edit your site name, HTML template, and sharing settings on /style

Known rough edges

• Single-user UI
• All-or-nothing sharing per mode
• Manual sync (except optional forum auto-sync)
• No recrawling
• No browser extension — bookmarklet only
• Desktop-oriented
• JSON-only import
• Forum threads prune after 30 days by default
• Forum discovery is through the subscription graph rather than topic-based blooms
• Best-effort maintenance

Forum plugin

TinyWeb ships with an optional tinyweb-forum plugin — a decentralized link-sharing discussion board that runs in-process alongside TinyWeb.

Install

pip install tinyweb-forum

Enable it on the /style page under "Forum". A "Forum" link will appear in the navigation bar.

How it works

• Threads and posts are stored in ~/.tinyweb/forum.db (separate from your search index)
• Instances are discovered automatically via mesh announces — no manual setup needed
• Sync is manual by default: click "sync now" on the forum page. Auto-sync every 5 minutes is optional (toggle on moderation page)
• At scale, sync uses epidemic gossip: 20 random peers per cycle, converging globally within ~O(log N) cycles
• Authors are identified by a short pseudonymous identity hash (no accounts, no sign-up)
• Auto-discovery can be disabled in the moderation page
• Threads are auto-pruned after 30 days (configurable, or set to 0 to keep everything)
• Moderation is local: block authors, mute threads, keyword filters, and gossip block lists with peers (auto-block after 3 peer reports)
• Trust circle — Content is gated by a trust graph derived from your TinyWeb subscriptions. Each subscribed peer's forum content (and their transitive trust network) enters your view automatically. Blocking a peer cascades: their downstream trust network is removed from your view.

For full feature docs, see the tinyweb-forum README.

Project structure

app.py                  — Entry point (shim, imports from tinyweb.app)
pyproject.toml          — Package configuration (src layout)
src/tinyweb/
  __init__.py           — Package marker
  app.py                — Boots Reticulum, starts HTTP gateway
  db.py                 — SQLite database, FTS5, URL fetching, SSRF protection
  gateway.py            — HTTP-to-RNS bridge (local or remote dispatch)
  templates.py          — HTML template rendering and escaping
  embeddings.py         — Semantic search: ONNX, HNSW, reranking
  rns_client.py         — Reticulum client for fetching remote site lists
  handlers/
    __init__.py         — Dispatch logic + re-exports
    _helpers.py         — CSRF, FTS sanitizer, pagination, response builders
    search.py           — Search (BM25, hybrid, trusted/remote results)
    pages.py            — Add/edit/delete/bulk/bookmark handlers
    subscriptions.py    — Sync, sharing, API, subscription CRUD
    customize.py        — Settings form, about page
    tags.py             — Tag list and browse
    data.py             — Export, import, semantic reindex
themes/                 — Saved HTML templates (e.g. default.html, junimo.html)

Security

The web UI has no authentication. It is bound to 127.0.0.1 by default, so only processes on the local machine can reach it. If you pass --bind 0.0.0.0 (or run inside a container with a published port), anyone who can reach that address can fully control your instance — reading private entries, changing settings, and modifying the HTML template (which runs in your browser). Put TinyWeb behind a reverse proxy with auth before exposing it beyond localhost.

Other hardening measures:

• CSRF protection — All POST forms use per-session tokens via double-submit cookies
• SSRF prevention — URL fetching validates hostnames against private IP ranges, with redirect re-validation
• FTS5 injection prevention — Search queries are sanitized before passing to SQLite MATCH
• Content Security Policy — CSP headers on all HTML responses restrict script/style/frame sources
• XSS escaping — All user-supplied content is HTML-escaped before rendering
• Bookmark authentication — The bookmarklet endpoint requires a secret token
• Identity file protection — The Reticulum identity key is restricted to owner-only permissions (0600)
• Forum caveats — See tinyweb-forum Security for forum-specific risks (voluntary retractions, block gossip manipulation, no rate limiting)

Not hardened

• No HTTPS
• No authentication
• No encryption-at-rest
• No rate limiting
• Bookmarklet token sent as a plain URL parameter
• Forum moderation is gossip-based — block lists can be manipulated

Maintenance

Database Vacuum

Over time, deleted pages leave empty space in the database. Run the vacuum tool periodically to reclaim space:

1. Go to /style in your browser
2. Click "vacuum database" at the bottom of the page

Optional Compression

To reduce storage for semantic search embeddings (~50% savings):

1. Go to /style > Search > AI
2. Enable "compress embeddings"
3. Re-index your existing pages for the compression to apply to existing embeddings

Dependencies

• requests — HTTP fetching
• beautifulsoup4 — HTML parsing and link extraction
• rns — Reticulum mesh networking