174 tests covering URL normalization, FTS5 query sanitization, SSRF/CSRF
guards, sharing-mode logic, DB schema and upsert paths, handler
end-to-end flows, and gateway body-size / mesh-whitelist guards. Each
recent bug-fix commit (6ffd38d, 1bc695f, 8dffd8c) has an explicit
regression test in test_regressions.py. One xfail documents a minor
latent bug in clean_url where port 80 is not stripped from upgraded
https URLs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Bulk delete now routes through a server-rendered confirmation page
listing the selected titles; a `confirmed=1` form field is required
before pages are actually deleted. Mirrors the single-delete flow.
- Reset-template button gains a JS confirm() so stray clicks don't wipe
the custom template.
- Homepage shows a short, neutral empty-state block when the index has
zero pages and no query — just names what tinyweb is and links to
/add, /style, and /subscriptions as equal options.
- /about gains a "your data" section explaining what lives in
~/.tinyweb/ (identity file, index.db), what losing each costs, and
how /export differs from a full backup.
- README gains a "Backups" subsection mirroring the /about copy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Security:
- Bind HTTP gateway to 127.0.0.1 by default; add --bind for LAN opt-in
- Restrict Reticulum mesh surface to GET /api/sites only (CSRF cannot
authenticate mesh callers, so gate by whitelist)
- Cap request body size at 16 MiB to prevent memory DoS
- Redact /bookmark query strings from request logs so the bookmark token
and URLs do not land in stdout / docker / journal logs
- Tighten FTS5 sanitizer: strip colon, drop AND/OR/NOT/NEAR operator words
- Expand .dockerignore; document trust model in README
Features:
- Add sharing mode toggle (share everything except private vs share only
public-tagged) with /share/preview so users can see what subscribers
would receive before enabling sharing
Bugs:
- handle_export() crashed on every call (missing query kwarg)
- Dead float16 decompression branch in embeddings.py silently corrupted
the HNSW index when compress_embeddings was on
- GATEWAY_PORT staleness: --port and find_available_port had no effect
on the actual bind
- semantic_search default mismatched between db.py ("1") and the rest of
the app ("0"), causing embeddings to be generated when the UI said off
- Connection pool returned connections with uncommitted transactions to
the next consumer
- Gateway POST body decode 502'd on non-UTF-8 input
- ensure_rns_config clobbered user-edited ~/.reticulum/config; now only
rewrites files it authored (sentinel-tagged)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Progressive retry in rns_client.py: fast timeout (15s) then slow (60s+)
for LoRa/multi-hop links, with automatic fallback
- Background sync threads so subscriptions page returns immediately
with syncing/error status indicators per subscription
- LoRa RNode configuration in settings page with serial port and
expandable advanced radio settings (frequency, bandwidth, etc.)
- Internet transport now toggleable alongside LoRa — users can
enable one, the other, or both
- Reticulum config auto-generated from settings on startup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bulk delete and retag from browse page with checkboxes
- Select all / deselect all toggle
- Delete confirmation shows count of selected pages
- Auto-cleanup orphaned tags on delete, edit, and bulk actions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace Google Fonts with system font stacks across all themes
- Add Referrer-Policy, X-Content-Type-Options, X-Frame-Options, CSP headers
- Add rel="noreferrer noopener" on all outbound links
- Add no-referrer and dns-prefetch-control meta tags to all themes
- Clean tracking params on outbound links from trusted/remote sources
- Remove Google domains from CSP whitelists
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds pagination, meta, and success message styles, plus input
selectors for new form fields (edit page, manual entry, transport node).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add pyinstaller.spec and GitHub/Forgejo CI workflows for cross-platform builds
- Add AGPLv3 license
- Move data storage to ~/.tinyweb/
- Add --version and --port CLI flags
- Add transport node selection in /style (smart regeneration preserves Reticulum config)
- Add discover more nodes link to rmap.world
- Add semantic_search setting to toggle AI-powered search on/off
- Skip embedding generation, hybrid search, and model preloading when disabled
- Use site owner's meta description as snippet instead of heuristic extraction
- Remove _generate_summary() and snippet() - no more generated snippets
- Show reranker/reindex controls grayed out when semantic search is off
- AI dependencies (onnxruntime, hnswlib, etc.) are now fully optional
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Case-insensitive meta description extraction (fixes sites like Lemmy
with capitalized "Description" meta name)
- Strip aside and noscript tags for cleaner body text
- Extract paragraph text separately for better sentence quality
- Prefer sentences mentioning the site name, then first quality
paragraph, then title as fallback
- Skip meta descriptions under 20 chars (e.g. just "Lemmy")
- Remove embedding/centroid dependency from summary generation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Lemmy and other JS-heavy sites include noscript fallback text like
"Javascript is disabled" that pollutes the stored body text and
generated snippets/summaries.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously reindex skipped pages that already had chunks, leaving stale
embeddings in place. It also overwrote good meta description summaries
with auto-generated ones. Now it clears all chunks first so everything
is re-embedded, and only generates summaries for pages missing one.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements a three-stage search pipeline:
1. BM25 keyword search via FTS5 with column weights
2. Semantic search via Snowflake arctic-embed-s bi-encoder + HNSW index
3. Optional cross-encoder reranking (on by default, toggleable in settings)
Top 20 results are reranked for precision, next 10 appended from RRF
for coverage, giving 30 total results across 3 pages.
- New embeddings.py with ONNX Runtime inference, text chunking, HNSW
index management, RRF fusion, and cross-encoder reranking
- Meta description extraction for authentic page snippets with centroid
extractive fallback
- Stopword filtering in FTS5 queries to avoid overly strict matching
- /reindex page for batch embedding of existing pages
- Semantic embedding of remote pages during subscription sync
- ~125MB dependency footprint (onnxruntime, tokenizers, hnswlib, numpy)
- Models: 34MB bi-encoder + 22MB cross-encoder (downloaded on first use)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Browser textarea submissions convert \n to \r\n, causing the template
comparison against DEFAULT_TEMPLATE to always fail. This saved the bare
skeleton as a custom template, overriding the default navbar.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace cramped table layout with card-based design that works
better in narrow viewports and across different themes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With share_instance = Yes, announces weren't being sent over TCP
in Docker environments. Setting it to No ensures each TinyWeb
instance manages its own Reticulum interfaces directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The announce was firing before the TCP transport connection was fully
established, causing Docker instances to never announce over the mesh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New TinyWeb instances now auto-connect to reticulum.derickphan.com:4242
so users get internet mesh connectivity out of the box without any
manual Reticulum configuration. Env var overrides still supported.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces static CMD with an entrypoint that generates RNS config from
environment variables (RNS_TCP_HOST/PORT), enabling TCP transport for
environments without LAN auto-discovery (e.g. Docker on macOS).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WAL + pooling:
- Enable WAL journal mode for concurrent read/write support
- Add connection pool (size 4) with return_db() to reuse connections
instead of opening/closing on every request
Pagination:
- Search results, /pages, and /tags/<name> now paginate at 50 per page
- Prev/next navigation links appear when results exceed one page
Delta sync:
- Pages table gains last_modified timestamp, set on insert/update
- /api/sites accepts ?since= param to return only changed pages
- Subscription sync uses last_sync timestamp for incremental fetches
- Remote pages upserted instead of delete-all/re-insert
- Full sync includes all_urls list for detecting remote deletions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
clean_url() now canonicalizes: http→https, strips www., removes
trailing slashes, drops default ports, and sorts query params.
Prevents the same page from being indexed multiple times under
different URL variations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lastrowid returns 0 when ON CONFLICT DO UPDATE fires on an existing
row, causing links to not be cleaned up or associated correctly on
re-index. Now fetches the actual row ID with a SELECT after upsert.
Also adds try/finally for connection safety.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SSRF: disable automatic redirects, manually follow up to 5 hops with
IP re-validation at each step to prevent redirect-to-localhost bypass
- Identity file: enforce 0600 permissions on tinyweb_identity at load
and creation to prevent other users from reading the private key
- Error messages: replace raw exception strings with generic messages
to avoid leaking internal paths/hostnames to the UI
- DB connections: wrap all get_db() usage in try/finally to guarantee
close() even when handlers throw mid-operation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bookmark endpoint now requires a secret token (stored in settings)
- Style reset moved from GET to POST with CSRF protection
- Open redirect prevention in _redirect() helper
- Import capped at 100 URLs to prevent abuse
- page_tags cleaned up on delete + PRAGMA foreign_keys enabled
- CSP, X-Frame-Options, X-Content-Type-Options on all responses
- CSRF tokens now per-session via double-submit cookie pattern
- Tag names URL-decoded for special characters
- Gateway forwards cookies in request data
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CSRF: Generate random token at startup, include as hidden field in
all 11 POST forms, validate at top of POST dispatch (returns 403)
- SSRF: Block private/internal IP ranges (127/8, 10/8, 172.16/12,
192.168/16, 169.254/16, ::1, fc00::/7) by resolving hostname before
fetch. Remove verify=False from requests.get().
- DELETE: Change /delete/<id> from GET (instant delete) to GET
(confirmation page) + POST (actual delete) to prevent accidental
deletion from prefetchers/crawlers.
- FTS5: Wrap search input in double quotes to neutralize FTS5
operators (AND, OR, NOT, *, column:). Add try/except fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Save the custom kodama template to themes/kodama.html so it's
version-controlled as a file rather than only living in the database.
Stop tracking index.db since it's runtime data, not source code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
