chore: remove pre-release schema migrations

- /rns/<hash>/<path> proxies pages over RNS with link rewriting
- fetch_remote_page() in rns_client.py for generic RNS page fetching
- mesh_sites table for persisting saved hashes
- standalone site_server.py with its own RNS identity
- tinyweb-site/index.html: SPA with RSS-aware nav (/rns/<hash>/ prefix)
- unified /add form: single input accepts URL or 32-char RNS hash
- RNS add indexes into pages table (fetch root page, extract title/desc)
- rns:<hash> URLs displayed in browse/search, linked to /rns/<hash>/
- expand RNS whitelist: allow GET /, /about, /pages, /tags, /share

.dockerignore

@@ -1,5 +1,15 @@
 __pycache__/
+**/__pycache__/
+*.pyc
 index.db*
+index.hnsw
 tinyweb_identity
 .git/
+.gitignore
 *.md
+.env
+.env.*
+.venv/
+venv/
+models/
+.DS_Store

.env.example

@@ -0,0 +1,7 @@
+# Reticulum configuration directory (default: ~/.tinyweb/.reticulum)
+# RNS_CONFIG_DIR=/path/to/reticulum
+
+# Connect to an internet-accessible TCP transport node for mesh connectivity
+# over the public internet (optional — leave unset for local-only mesh)
+# RNS_TCP_HOST=rnode.bre.land
+# RNS_TCP_PORT=4242

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,7 @@
+**Describe the issue**
+
+**Steps to reproduce**
+
+**Expected vs actual behavior**
+
+**Environment (OS, Python version, install method)**

.gitignore

@@ -1,3 +1,12 @@
 __pycache__/
+*.pyc
+.env
 tinyweb_identity
 index.db
+index.db-shm
+index.db-wal
+models/
+index.hnsw
+*.db
+*.db-shm
+*.db-wal

CONTRIBUTING.md

@@ -0,0 +1,74 @@
+# Contributing to TinyWeb
+
+Thanks for helping. TinyWeb is a decentralized search engine built for the
+Reticulum mesh. Contributions that respect its architecture and constraints
+are welcome.
+
+## Before you start
+
+- Open an issue first for anything beyond a small bug fix to discuss the approach.
+- One change per pull request. Keep it focused and reviewable.
+- TinyWeb has no authentication by design — PRs adding login or user accounts
+  are outside scope unless the architecture changes.
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+python app.py
+```
+
+Open `http://127.0.0.1:8080`. Run with `--bind 0.0.0.0` for LAN access
+(see [SECURITY.md](SECURITY.md) about the risks of exposing an unauthenticated app).
+
+## Running checks
+
+Run the test suite before submitting:
+
+```bash
+python -m pytest
+```
+
+## What fits TinyWeb
+
+- Bug fixes, performance improvements, and mesh reliability work
+- New handlers or integrations that respect the no-JS, no-CSS-framework constraint
+- Documentation and test coverage
+- Edge case handling and error recovery
+
+## What doesn't fit
+
+- Authentication, user accounts, or session systems (by design)
+- JavaScript or CSS frameworks (Tailwind, Bootstrap, React, etc.)
+- Dependencies that assume internet connectivity or cloud infrastructure
+- Database backends other than SQLite
+- Features that break local-first or mesh-native operation
+
+## Pull requests
+
+- Target `main`.
+- Describe what changed and why.
+- Include test output or manual test steps.
+- Screenshots for anything visual.
+- Use [Conventional Commits](https://www.conventionalcommits.org):
+  `fix(search): ...`, `feat(rns): ...`, `docs(readme): ...`
+
+LLM-generated PRs should be preceded by an issue. Unannounced bulk PRs may
+be closed without review.
+
+## Code conventions
+
+- No inline `style=""` except for functional margins/padding/alignment
+- No JavaScript or CSS frameworks
+- No Unicode emoji in UI — use HTML entities or plain text
+- No hardcoded paths — use `DATA_DIR` or `TINYWEB_DATA_DIR` env var
+- No hardcoded domains, IPs, or destination hashes
+
+## Issue reports
+
+Include: install method, OS, steps to reproduce, expected vs actual.
+For mesh issues: interface type, RNS destination hash if relevant.
+
+## Security
+
+See [SECURITY.md](SECURITY.md). Report privately — don't post exploits in issues.

Dockerfile

@@ -2,15 +2,23 @@ FROM python:3.12-slim
 
 WORKDIR /app
 
+# Install build tools for packages like hnswlib
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    g++ \
+    gcc \
+    && rm -rf /var/lib/apt/lists/*
+
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
 
 COPY . .
 
-RUN mkdir -p /data \
+RUN pip install -e /app
-    && ln -sf /data/index.db index.db \
+
-    && ln -sf /data/tinyweb_identity tinyweb_identity
+RUN mkdir -p /data
+
+ENV PYTHONUNBUFFERED=1
 
 EXPOSE 8080
 
-CMD ["python", "app.py"]
+ENTRYPOINT ["./entrypoint.sh"]

LICENSE

@@ -0,0 +1,574 @@
+GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License giving you legal permission to copy, distribute
+and/or modify the software.
+
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+are made publicly available of their being a derivative work, need not
+be distributed to others.
+
+  For example, if you modify a part of a free program, you are
+not required to distribute the object code for the modified version
+itself; however, the GNU Affero General Public License requires you
+to provide source code for any version of the program that you use
+or modify.  This requirement is similar to the requirement that
+the user can receive the source code if they distribute a copy.
+
+  Also, if you link or combine the program with any other software
+that contains code covered by this License (or any work based on the
+program), you must provide the source code for that combined work
+as well.  The GNU Affero General Public License normally requires
+that any work that you distribute or publish that in whole or in
+part contains or is derived from the program or any part thereof,
+to be licensed as a whole at no charge to all third parties under
+the terms of this License.  This is known as "providing source code"
+or "making available" the work.
+
+  An "aggregated" or "combined" work is not covered by this License
+if you do not meet these conditions, and you must provide the source
+code as above.  Additionally, aggregating works does not exempt you
+from the requirements of this License.
+
+  Specifically, if you make an "aggregate" or "combined" work by
+combining this program with other software (or any work based on this
+program) on a volume of a storage or distribution medium, you must
+provide the source code for the combined work as above.  This
+requirement is intended to ensure that any user of the combined work
+gets the source code that you made available, and can exercise the
+right to modify and re-distribute the combined work.
+
+  This License is specifically intended to limit any attempt to
+place your modifications under a license that would restrict re-use
+or further modification by others.  This is to ensure that any
+derivative work you create will be available under the same license
+as the original, so that any derivative work can be re-distributed
+under the same conditions as the original.
+
+  Finally, this License is not intended to limit your rights under
+fair use or other limitations on exclusive rights, such as patents
+or trademarks.  This License does not grant you any rights to the
+names of the authors or copyright holders, nor to trade names,
+trademarks, or service marks, except as needed for the normal and
+customary use in describing the origin of the work and reproducing
+the content of the notice file.
+
+  The source code for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided in copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must contain prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must contain prominent notices stating that it is
+    released under this License and any conditions added under section 7.
+    This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the legal rights of the compilation's users beyond
+what the individual works permit.  Inclusion of a covered work in an
+aggregate does not cause this License to apply to the other parts of
+the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in
+    accord with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A
+product is a consumer product regardless of whether the product has
+substantial commercial, industrial or non-consumer uses, unless such
+uses represent the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions applicable to the entire Program shall be treated
+as though they were included in this License, to the extent that they
+are valid under applicable law.  If additional permissions apply only to
+part of the Program, then that part may be used separately under those
+permissions, but the entire Program remains governed by this License
+without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as
+you received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, or selling the work,
+or by making, using, or selling the work.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available to anyone
+in the United States, you may not convey the work under this License.
+This is done by providing access to copy the corresponding source code
+from a network server at no charge.
+
+  If, during the execution of the Program, the Program is transmitted
+to a user or a computer, either the source code or object code, you
+must meet the requirements of this License regarding the
+Corresponding Source of the work.  You must make sure that the source
+code or object code (as applicable) is available for such users to
+copy and modify, and to run, for their own use, the corresponding
+source in accordance with this License.  This requirement applies
+both to the work as stand-alone and to the work as part of an
+aggregate.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user
+through a computer network, with no transfer of a copy, is not
+conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  12. No Warranty
+
+  THE PROGRAM IS PROVIDED WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK
+AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD
+THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY
+SERVICING, REPAIR OR CORRECTION.
+
+  13. Disclaimer of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR
+THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+  14. Interpretation of Sections 12 and 13.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU Affero General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    <program>  Copyright (C) <year>  <name of author>
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+  The hypothetical commands `show w' and `show c' should show the appropriate
+  parts of the General Public License.  Of course, your program's commands
+  might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+  if any, to sign a "copyright disclaimer" for the program, if necessary.
+  For more information on this, and how to apply and follow the GNU AGPL,
+  see <https://www.gnu.org/licenses/>.

README.md

@@ -1,34 +1,171 @@
 # TinyWeb
 
-A personal, decentralized search engine built on the [Reticulum](https://reticulum.network/) mesh network. Curate your own index of web pages, search it locally, and share collections with friends over an encrypted mesh. No algorithms, no ads, no tracking.
+A personal, decentralized search engine built on the [Reticulum](https://reticulum.network/) mesh network. You save pages you find. They are stored locally and shared over a mesh network so other people can find them too.
+
+## Contents
+
+- [About this project](#about-this-project)
+- [Features](#features)
+- [Performance & Scale](#performance--scale)
+- [Getting started](#getting-started)
+- [Data storage](#data-storage)
+- [Remote gateway](#remote-gateway)
+- [How it works](#how-it-works)
+- [Known rough edges](#known-rough-edges)
+- [Forum plugin](#forum-plugin)
+- [Project structure](#project-structure)
+- [Security](#security)
+- [Maintenance](#maintenance)
+- [Dependencies](#dependencies)
+
+## About this project
+
+Code generated by LLMs. Built by one person.
 
 ## Features
 
 - **Personal search index** — Save pages you find valuable, search them with full-text search (SQLite FTS5)
+- **RNS live browsing** — Browse any RNS site through your instance via `<base>` tag injection, with LRU caching
+- **Unified add form** — Add HTTP URLs or RNS destination hashes through a single form; auto-detection handles both
 - **Tagging** — Organize saved pages with comma-separated tags
 - **Bookmarklet** — One-click indexing from any browser tab
 - **Subscriptions** — Subscribe to friends' TinyWeb instances over Reticulum and search their indexes alongside yours
 - **Custom templates** — Full HTML/CSS/JS template editor to personalize your instance
 - **Import/export** — JSON-based backup and restore
 - **Mesh-native** — Works over Reticulum without the internet; encrypted and decentralized by default
+- **Forum plugin** — Optional link-sharing discussion board over the mesh (see Forum section below)
+
+### What sharing means
+
+Once a subscriber syncs your pages, you have no control over their copy.
+No revocation, no DRM, no expiry. If you shared a page, assume it's out
+there permanently.
+
+Tag-based sharing (`private`, `public`) is advisory. The software
+respects these tags in its API response, but there is no technical
+mechanism preventing a subscriber from re-sharing your data.
+
+Deletion is local only. Removing a page from your index does not
+propagate to subscribers.
+
+## Performance & Scale
+
+### Search Speed
+
+| Pages indexed | Search speed | Notes |
+|--------------|-------------|-------|
+| 1,000 | ~50ms | Fast local FTS5 |
+| 10,000 | ~50-100ms | Full-text search |
+| 100,000 | ~100-200ms | Combined BM25 + semantic |
+| 500,000 | ~200-400ms | With semantic enabled |
+| 1,000,000 | ~300-500ms | Hybrid search |
+
+*Times are estimates for combined BM25 + semantic search. Actual performance varies by hardware, storage type (SSD/HDD), and search complexity.*
+
+### Concurrent Connections
+
+- Database pool: 16 simultaneous connections
+- Suitable for single-user + a few subscriptions
+
+### Export
+
+- Paginated at 10,000 pages per request
+- Use `?batch=N` to export in chunks: `/export?batch=0`, `/export?batch=1`, etc.
 
 ## Getting started
 
+**Requirements:** Python 3.10+ and pip.
+
 ```bash
+git clone https://codeberg.org/tinyweb/tinyweb.git
+cd tinyweb
 pip install -r requirements.txt
 python app.py
 ```
 
-This starts the Reticulum server and an HTTP gateway on `http://localhost:8080`. Open it in your browser.
+Open `http://127.0.0.1:8080` in your browser. The web UI is localhost-only by default.
 
 Your destination hash is printed on startup — share it with friends so they can subscribe to your index.
 
+### Command line options
+
+```bash
+python app.py -p 9000        # Use port 9000 instead of default 8080
+python app.py --bind 0.0.0.0 # Expose to your LAN (no auth — see Security)
+```
+
+### Keeping it running
+
+```bash
+# Terminal session (closes when you log out):
+python app.py
+
+# Background with tmux:
+tmux new-session -d -s tinyweb 'python app.py'
+
+# Background with nohup:
+nohup python app.py &
+```
+
+### Docker
+
+A `docker-compose.yml` is included for containerized setups. Build and run:
+
+```bash
+docker compose up -d
+```
+
+Data persists in the `tinyweb-data` named volume. On Linux with LAN auto-discovery it works as-is; on macOS or remote setups, see `docker-compose.yml` comments for TCP transport config.
+
+## Storage Estimates
+
+Pages are stored as cleaned text (HTML tags stripped, boilerplate removed) — typically 5-15 KB per page across both HTTP and RNS sources:
+
+| Pages | Database | Embeddings* | Total |
+|-------|----------|------------|-------|
+| 10,000 | ~100MB | 80MB | ~180MB |
+| 100,000 | ~1GB | 800MB | ~1.8GB |
+| 500,000 | ~5GB | 4GB | ~9GB |
+| 1,000,000 | ~10GB | 8GB | ~18GB |
+
+*Embeddings require semantic search to be enabled. With compression enabled (Settings > Search > AI), embeddings use ~50% less storage.
+
+Enable optional compression in Settings > Search > AI to reduce embedding storage by ~50%.
+
+## Data storage
+
+### Local (Python/binary)
+
+Your data is stored in `~/.tinyweb/`:
+
+| File | Description |
+|------|-------------|
+| `index.db` | SQLite database with your indexed pages |
+| `tinyweb_identity` | Your Reticulum identity (keep safe!) |
+| `forum.db` | Forum plugin database (only if forum is enabled) |
+| `models/` | Downloaded AI models for semantic search |
+| `index.hnsw` | Semantic search index |
+
+This allows your data to persist between upgrades and stay separate from the application.
+
+### Backups
+
+Back up the whole `~/.tinyweb/` directory periodically. The two files that matter:
+
+- **`tinyweb_identity`** is your permanent mesh identity. If you lose it, your destination hash changes and every subscriber has to re-subscribe to the new one. Keep it somewhere you trust; the file is `0600` by default.
+- **`index.db`** is your full reading history — every page, note, tag, and synced remote page. Losing it loses everything you've curated.
+- **`forum.db`** (if the forum plugin is enabled) — all threads, posts, upvotes, and moderation settings. Losing it loses your forum data.
+
+`models/` and `index.hnsw` are re-derivable (the model will re-download, and the HNSW index rebuilds from the database on next startup with semantic search enabled) so they don't need to be backed up.
+
+The `/export` page produces a JSON dump of your pages. It's a migration aid — it doesn't preserve your identity file, your custom template, or subscription state. A full restore needs a copy of `~/.tinyweb/`.
+
 ## Remote gateway
 
 To browse a remote TinyWeb instance without running your own index:
 
 ```bash
-python gateway.py <destination_hash>
+python -m tinyweb.gateway <destination_hash>
 ```
 
 This connects over Reticulum and serves the remote instance at `http://localhost:8080`.
@@ -40,21 +177,75 @@ This connects over Reticulum and serves the remote instance at `http://localhost
 3. **Subscribe** — Add a friend's destination hash on `/subscriptions` to sync their shared index
 4. **Customize** — Edit your site name, HTML template, and sharing settings on `/style`
 
+## Known rough edges
+
+- Single-user UI
+- All-or-nothing sharing per mode
+- Manual sync (except optional forum auto-sync)
+- No recrawling
+- No browser extension — bookmarklet only
+- Desktop-oriented
+- JSON-only import
+- Forum threads prune after 30 days by default
+- Forum discovery is through the subscription graph rather than topic-based blooms
+- Best-effort maintenance
+
+## Forum plugin
+
+TinyWeb ships with an optional [tinyweb-forum](https://codeberg.org/tinyweb/tinyweb-forum) plugin — a decentralized link-sharing discussion board that runs in-process alongside TinyWeb.
+
+### Install
+
+```bash
+pip install tinyweb-forum
+```
+
+Enable it on the `/style` page under "Forum". A "Forum" link will appear in the navigation bar.
+
+### How it works
+
+- Threads and posts are stored in `~/.tinyweb/forum.db` (separate from your search index)
+- Instances are discovered automatically via mesh announces — no manual setup needed
+- Sync is manual by default: click "sync now" on the forum page. Auto-sync every 5 minutes is optional (toggle on moderation page)
+- At scale, sync uses epidemic gossip: 20 random peers per cycle, converging globally within ~O(log N) cycles
+- Authors are identified by a short pseudonymous identity hash (no accounts, no sign-up)
+- Auto-discovery can be disabled in the moderation page
+- Threads are auto-pruned after 30 days (configurable, or set to 0 to keep everything)
+- Moderation is local: block authors, mute threads, keyword filters, and gossip block lists with peers (auto-block after 3 peer reports)
+- **Trust circle** — Content is gated by a trust graph derived from your TinyWeb subscriptions. Each subscribed peer's forum content (and their transitive trust network) enters your view automatically. Blocking a peer cascades: their downstream trust network is removed from your view.
+
+For full feature docs, see the [tinyweb-forum README](https://codeberg.org/tinyweb/tinyweb-forum).
+
 ## Project structure
 
 ```
-app.py          — Entry point: boots Reticulum, starts HTTP gateway
+app.py                  — Entry point (shim, imports from tinyweb.app)
-gateway.py      — HTTP-to-RNS bridge (local or remote dispatch)
+pyproject.toml          — Package configuration (src layout)
-handlers.py     — Route dispatcher and all request handlers
+src/tinyweb/
-db.py           — SQLite database, FTS5, URL fetching, SSRF protection
+  __init__.py           — Package marker
-templates.py    — HTML template rendering and escaping
+  app.py                — Boots Reticulum, starts HTTP gateway
-rns_client.py   — Reticulum client for fetching remote site lists
+  db.py                 — SQLite database, FTS5, URL fetching, SSRF protection
-themes/         — Saved HTML templates (e.g. kodama.html)
+  gateway.py            — HTTP-to-RNS bridge (local or remote dispatch)
+  templates.py          — HTML template rendering and escaping
+  embeddings.py         — Semantic search: ONNX, HNSW, reranking
+  rns_client.py         — Reticulum client for fetching remote site lists
+  handlers/
+    __init__.py         — Dispatch logic + re-exports
+    _helpers.py         — CSRF, FTS sanitizer, pagination, response builders
+    search.py           — Search (BM25, hybrid, trusted/remote results)
+    pages.py            — Add/edit/delete/bulk/bookmark handlers
+    subscriptions.py    — Sync, sharing, API, subscription CRUD
+    customize.py        — Settings form, about page
+    tags.py             — Tag list and browse
+    data.py             — Export, import, semantic reindex
+themes/                 — Saved HTML templates (e.g. default.html, junimo.html)
 ```
 
 ## Security
 
-TinyWeb includes several hardening measures:
+**The web UI has no authentication.** It is bound to `127.0.0.1` by default, so only processes on the local machine can reach it. If you pass `--bind 0.0.0.0` (or run inside a container with a published port), anyone who can reach that address can fully control your instance — reading private entries, changing settings, and modifying the HTML template (which runs in your browser). Put TinyWeb behind a reverse proxy with auth before exposing it beyond localhost.
+
+Other hardening measures:
 
 - **CSRF protection** — All POST forms use per-session tokens via double-submit cookies
 - **SSRF prevention** — URL fetching validates hostnames against private IP ranges, with redirect re-validation
@@ -63,6 +254,33 @@ TinyWeb includes several hardening measures:
 - **XSS escaping** — All user-supplied content is HTML-escaped before rendering
 - **Bookmark authentication** — The bookmarklet endpoint requires a secret token
 - **Identity file protection** — The Reticulum identity key is restricted to owner-only permissions (0600)
+- **Forum caveats** — See [tinyweb-forum Security](https://codeberg.org/tinyweb/tinyweb-forum#security) for forum-specific risks (voluntary retractions, block gossip manipulation, no rate limiting)
+
+#### Not hardened
+
+- No HTTPS
+- No authentication
+- No encryption-at-rest
+- No rate limiting
+- Bookmarklet token sent as a plain URL parameter
+- Forum moderation is gossip-based — block lists can be manipulated
+
+## Maintenance
+
+### Database Vacuum
+
+Over time, deleted pages leave empty space in the database. Run the vacuum tool periodically to reclaim space:
+
+1. Go to `/style` in your browser
+2. Click "vacuum database" at the bottom of the page
+
+### Optional Compression
+
+To reduce storage for semantic search embeddings (~50% savings):
+
+1. Go to `/style` > Search > AI
+2. Enable "compress embeddings"
+3. Re-index your existing pages for the compression to apply to existing embeddings
 
 ## Dependencies
 
@@ -70,6 +288,4 @@ TinyWeb includes several hardening measures:
 - [beautifulsoup4](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing and link extraction
 - [rns](https://reticulum.network/) — Reticulum mesh networking
 
-## Philosophy
 
-TinyWeb is built for the slow web — intentionality over speed, human curation over algorithmic feeds, privacy over surveillance, and community over corporations. Every page in your index was saved because you found it valuable, not because an algorithm told you to click.

SECURITY.md

@@ -0,0 +1,26 @@
+# Security Policy
+
+TinyWeb is a self-hosted personal search engine with no authentication. It is
+bound to localhost by default. Do not expose it to the internet without a
+reverse proxy with authentication.
+
+## Supported Versions
+
+Security fixes are handled on the default branch.
+
+## Deployment Guidance
+
+- Keep TinyWeb bound to `127.0.0.1` unless you need LAN access
+- If binding to `0.0.0.0`, put a reverse proxy with auth in front (Caddy, nginx)
+- Always use HTTPS when exposing beyond localhost
+- Protect `~/.tinyweb/tinyweb_identity` — losing it changes your destination hash
+- Protect `~/.tinyweb/index.db` — contains your full browsing history
+- Never commit `*.db`, `tinyweb_identity`, `.env`, or `models/` to version control
+- Keep `--bind 0.0.0.0` usage to trusted networks only
+- The bookmarklet token is sent as a plain URL parameter — treat it as a secret
+- Forum plugin: moderation is gossip-based; block lists can be manipulated
+
+## Reporting
+
+Report vulnerabilities privately by creating a confidential issue on the
+repository. Do not file a public issue until the report has been acknowledged.

app.py

@@ -1,75 +1,5 @@
-import os
+import sys
-import time
+from pathlib import Path
-import threading
+sys.path.insert(0, str(Path(__file__).parent / "src"))
-import RNS
+from tinyweb.app import main
-from http.server import HTTPServer
+main()
-
-from db import init_db, set_setting
-from handlers import dispatch_request
-from gateway import GatewayState, GatewayHandler, GATEWAY_PORT
-
-APP_NAME = "tinyweb"
-ASPECTS = ["server"]
-IDENTITY_FILE = "tinyweb_identity"
-
-
-def load_or_create_identity():
-    if os.path.isfile(IDENTITY_FILE):
-        # Ensure identity file is only readable by owner
-        current = os.stat(IDENTITY_FILE).st_mode & 0o777
-        if current != 0o600:
-            os.chmod(IDENTITY_FILE, 0o600)
-        return RNS.Identity.from_file(IDENTITY_FILE)
-    identity = RNS.Identity()
-    identity.to_file(IDENTITY_FILE)
-    os.chmod(IDENTITY_FILE, 0o600)
-    return identity
-
-
-def rns_request_handler(path, data, request_id, link_id, remote_identity, requested_at):
-    if data is None:
-        data = {"method": "GET", "path": "/", "query": {}, "body": {}, "gateway_host": ""}
-    return dispatch_request(data)
-
-
-def start_gateway(reticulum):
-    GatewayState.reticulum = reticulum
-    GatewayState.local_dispatch = dispatch_request
-    server = HTTPServer(("0.0.0.0", GATEWAY_PORT), GatewayHandler)
-    thread = threading.Thread(target=server.serve_forever, daemon=True)
-    thread.start()
-
-
-def main():
-    init_db()
-    reticulum = RNS.Reticulum()
-    identity = load_or_create_identity()
-
-    destination = RNS.Destination(
-        identity,
-        RNS.Destination.IN,
-        RNS.Destination.SINGLE,
-        APP_NAME,
-        *ASPECTS,
-    )
-
-    destination.register_request_handler(
-        "/tinyweb",
-        response_generator=rns_request_handler,
-        allow=RNS.Destination.ALLOW_ALL,
-    )
-
-    destination.announce()
-    set_setting("dest_hash", destination.hash.hex())
-    start_gateway(reticulum)
-
-    print(f"TinyWeb running!")
-    print(f"Open http://localhost:{GATEWAY_PORT} in your browser")
-    print(f"Destination hash: {RNS.prettyhexrep(destination.hash)} (share this so friends can subscribe)")
-
-    while True:
-        time.sleep(1)
-
-
-if __name__ == "__main__":
-    main()

conftest.py

@@ -0,0 +1,124 @@
+"""Shared pytest fixtures for TinyWeb tests.
+
+Three fixtures cover most tests: `temp_db` swaps the SQLite path to a
+per-test tempfile, `seeded_db` layers sample rows on top, and `csrf_session`
+primes the thread-local CSRF token that handlers read.
+"""
+import socket
+
+import pytest
+
+import tinyweb.db as db_module
+import tinyweb.handlers as handlers_module
+
+
+@pytest.fixture
+def temp_db(tmp_path, monkeypatch):
+    """Isolated SQLite DB per test.
+
+    Swaps `db.DATABASE` and `db.DATA_DIR` to a tempdir, clears the connection
+    pool before and after so state doesn't leak across tests, and calls
+    `init_db()` so every schema object exists.
+    """
+    data_dir = tmp_path / "tinyweb"
+    data_dir.mkdir()
+    db_path = data_dir / "index.db"
+
+    monkeypatch.setattr(db_module, "DATA_DIR", str(data_dir))
+    monkeypatch.setattr(db_module, "DATABASE", str(db_path))
+
+    with db_module._pool_lock:
+        for conn in db_module._pool:
+            try:
+                conn.close()
+            except Exception:
+                pass
+        db_module._pool.clear()
+
+    db_module.init_db()
+    yield db_path
+
+    with db_module._pool_lock:
+        for conn in db_module._pool:
+            try:
+                conn.close()
+            except Exception:
+                pass
+        db_module._pool.clear()
+
+
+@pytest.fixture
+def seeded_db(temp_db):
+    """A temp DB with a small, realistic set of pages/tags/links."""
+    db = db_module.get_db()
+    try:
+        rows = [
+            ("https://example.com/rust-intro", "Rust Intro", "A gentle introduction to rust borrow checker.", "notes on ownership"),
+            ("https://example.com/python-tips", "Python Tips", "Daily python tricks for readable code.", ""),
+            ("https://example.com/ocaml-why", "Why OCaml", "Type systems and inference in ocaml.", "private thoughts"),
+            ("https://news.example.org/mesh", "Mesh Networking", "Reticulum and LoRa for decentralized networks.", ""),
+        ]
+        for url, title, body, note in rows:
+            db.execute(
+                "INSERT INTO pages (url, title, body, note, last_modified) "
+                "VALUES (?, ?, ?, ?, '2026-04-01T00:00:00')",
+                (url, title, body, note),
+            )
+        db.commit()
+        page_ids = {
+            row["url"]: row["id"]
+            for row in db.execute("SELECT id, url FROM pages").fetchall()
+        }
+        tag_rows = [
+            (page_ids["https://example.com/rust-intro"], ["rust", "public"]),
+            (page_ids["https://example.com/python-tips"], ["python"]),
+            (page_ids["https://example.com/ocaml-why"], ["ocaml", "private"]),
+            (page_ids["https://news.example.org/mesh"], ["mesh", "public"]),
+        ]
+        for pid, tags in tag_rows:
+            for name in tags:
+                db.execute("INSERT OR IGNORE INTO tags (name) VALUES (?)", (name,))
+                tid = db.execute("SELECT id FROM tags WHERE name = ?", (name,)).fetchone()[0]
+                db.execute(
+                    "INSERT OR IGNORE INTO page_tags (page_id, tag_id) VALUES (?, ?)",
+                    (pid, tid),
+                )
+        db.execute(
+            "INSERT INTO links (page_id, url, label) VALUES (?, ?, ?)",
+            (page_ids["https://example.com/rust-intro"], "https://example.com/rust-advanced", "advanced rust guide"),
+        )
+        db.commit()
+    finally:
+        db_module.return_db(db)
+    return temp_db
+
+
+@pytest.fixture
+def csrf_session(monkeypatch):
+    """Prime the CSRF thread-local so handler code that calls _get_csrf_token works."""
+    token = "test-csrf-token"
+    handlers_module._request_local.csrf_token = token
+    yield token
+    if hasattr(handlers_module._request_local, "csrf_token"):
+        del handlers_module._request_local.csrf_token
+
+
+def patch_dns_fail(monkeypatch):
+    """Make every socket.getaddrinfo call raise gaierror for the rest of this test."""
+    def boom(*args, **kwargs):
+        raise socket.gaierror("test: DNS disabled")
+    monkeypatch.setattr(socket, "getaddrinfo", boom)
+
+
+def patch_dns_ok(monkeypatch, address="93.184.216.34"):
+    """Make every getaddrinfo return a single public IP for the rest of this test."""
+    def ok(host, port, *args, **kwargs):
+        return [(socket.AF_INET, socket.SOCK_STREAM, 0, "", (address, port or 80))]
+    monkeypatch.setattr(socket, "getaddrinfo", ok)
+
+
+def patch_dns_private(monkeypatch, address="127.0.0.1"):
+    """Make every getaddrinfo return a private/blocked IP for the rest of this test."""
+    def private(host, port, *args, **kwargs):
+        return [(socket.AF_INET, socket.SOCK_STREAM, 0, "", (address, port or 80))]
+    monkeypatch.setattr(socket, "getaddrinfo", private)

docker-compose.yml

@@ -6,6 +6,12 @@ services:
     volumes:
       - tinyweb-data:/data
     restart: unless-stopped
+    # Connect to another Reticulum instance over TCP.
+    # Required on macOS (Docker can't do LAN auto-discovery).
+    # On Linux, auto-discovery works with network_mode: host.
+    # environment:
+    #   - RNS_TCP_HOST=your-peer-ip
+    #   - RNS_TCP_PORT=4242
 
 volumes:
   tinyweb-data:

entrypoint.sh

@@ -0,0 +1,35 @@
+#!/bin/sh
+# Generate Reticulum config with optional TCP peer
+# Set RNS_TCP_HOST and RNS_TCP_PORT env vars to connect to a remote instance
+
+CONFIG_DIR="/data/.reticulum"
+CONFIG_FILE="$CONFIG_DIR/config"
+
+mkdir -p "$CONFIG_DIR"
+
+if [ ! -f "$CONFIG_FILE" ]; then
+  cat > "$CONFIG_FILE" <<EOF
+[reticulum]
+  enable_transport = False
+  share_instance = No
+
+[logging]
+  loglevel = 4
+
+[interfaces]
+  [[Default Interface]]
+    type = AutoInterface
+    enabled = Yes
+
+  [[TCP Transport]]
+    type = TCPClientInterface
+    enabled = yes
+    target_host = ${RNS_TCP_HOST:-rnode.bre.land}
+    target_port = ${RNS_TCP_PORT:-4242}
+EOF
+fi
+
+export TINYWEB_DATA_DIR="/data"
+export RNS_CONFIG_DIR="$CONFIG_DIR"
+# Bind to 0.0.0.0 inside the container; isolation is handled by Docker's port mapping.
+exec python app.py --bind 0.0.0.0 "$@"

pyproject.toml

@@ -0,0 +1,12 @@
+[project]
+name = "tinyweb"
+version = "0.1.0"
+description = "Personal decentralized search engine"
+requires-python = ">=3.10"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.backends._legacy:_Backend"

pytest.ini

@@ -0,0 +1,5 @@
+[pytest]
+testpaths = tests
+python_files = test_*.py
+filterwarnings =
+    ignore::DeprecationWarning

requirements-dev.txt

@@ -0,0 +1,2 @@
+-r requirements.txt
+pytest

requirements.txt

@@ -1,3 +1,9 @@
+# forum: pip install tinyweb-forum (optional, adds URL discussion board)
 requests
 beautifulsoup4
 rns
+onnxruntime
+tokenizers
+hnswlib
+numpy
+huggingface_hub

rns_client.py

@@ -1,79 +0,0 @@
-import time
-import RNS
-
-APP_NAME = "tinyweb"
-ASPECTS = ["server"]
-REQUEST_TIMEOUT = 30
-
-
-def fetch_remote_sites(dest_hash_hex, since=""):
-    """
-    Connect to a remote TinyWeb instance over Reticulum and fetch its
-    shared sites. Returns the response dict from /api/sites, or raises
-    an exception on failure. Pass `since` as ISO timestamp for delta sync.
-    """
-    dest_hash = bytes.fromhex(dest_hash_hex)
-
-    # Resolve path if needed
-    if not RNS.Transport.has_path(dest_hash):
-        RNS.Transport.request_path(dest_hash)
-        elapsed = 0
-        while not RNS.Transport.has_path(dest_hash) and elapsed < 15:
-            time.sleep(0.5)
-            elapsed += 0.5
-        if not RNS.Transport.has_path(dest_hash):
-            raise ConnectionError(f"Could not find path to {dest_hash_hex}")
-
-    server_identity = RNS.Identity.recall(dest_hash)
-    if server_identity is None:
-        raise ConnectionError(f"Could not recall identity for {dest_hash_hex}")
-
-    destination = RNS.Destination(
-        server_identity,
-        RNS.Destination.OUT,
-        RNS.Destination.SINGLE,
-        APP_NAME,
-        *ASPECTS,
-    )
-
-    # Establish link
-    link = RNS.Link(destination)
-    elapsed = 0
-    while link.status == RNS.Link.PENDING and elapsed < 15:
-        time.sleep(0.25)
-        elapsed += 0.25
-
-    if link.status != RNS.Link.ACTIVE:
-        raise ConnectionError(f"Could not establish link to {dest_hash_hex}")
-
-    try:
-        # Request /api/sites
-        query = {"since": [since]} if since else {}
-        request_data = {
-            "method": "GET",
-            "path": "/api/sites",
-            "query": query,
-            "body": {},
-            "gateway_host": "",
-        }
-
-        receipt = link.request("/tinyweb", data=request_data, timeout=REQUEST_TIMEOUT)
-
-        elapsed = 0
-        done = (RNS.RequestReceipt.READY, RNS.RequestReceipt.DELIVERED, RNS.RequestReceipt.FAILED)
-        while receipt.get_status() not in done and elapsed < REQUEST_TIMEOUT:
-            time.sleep(0.5)
-            elapsed += 0.5
-
-        if receipt.get_status() in (RNS.RequestReceipt.READY, RNS.RequestReceipt.DELIVERED):
-            resp = receipt.get_response()
-            if resp["status"] == 403:
-                raise PermissionError("That instance has sharing disabled.")
-            if resp["status"] != 200:
-                raise ConnectionError(f"Remote returned status {resp['status']}")
-            import json
-            return json.loads(resp["body"])
-        else:
-            raise ConnectionError(f"Request failed or timed out")
-    finally:
-        link.teardown()

src/tinyweb/__init__.py

@@ -0,0 +1 @@
+

src/tinyweb/app.py

@@ -0,0 +1,323 @@
+import os
+import sys
+import time
+import threading
+import argparse
+import RNS
+from http.server import HTTPServer, ThreadingHTTPServer
+
+from tinyweb.db import init_db, get_setting, set_setting
+from tinyweb.handlers import dispatch_request
+import tinyweb.handlers as handlers_mod
+import tinyweb.templates as templates_mod
+import tinyweb.gateway as gateway
+from tinyweb.gateway import GatewayState, GatewayHandler
+
+IDENTITY_FILE = "tinyweb_identity"
+DEFAULT_TRANSPORT_HOST = "rnode.bre.land"
+DEFAULT_TRANSPORT_PORT = 4242
+DATA_DIR = os.environ.get("TINYWEB_DATA_DIR") or os.path.expanduser("~/.tinyweb")
+
+
+def get_transport_config():
+    host = get_setting("transport_host", DEFAULT_TRANSPORT_HOST)
+    port = get_setting("transport_port", str(DEFAULT_TRANSPORT_PORT))
+    return host, int(port)
+
+
+def find_available_port(start=8080, max_attempts=20, host="127.0.0.1"):
+    """Find an available port starting from start."""
+    import socket
+    for port in range(start, start + max_attempts):
+        try:
+            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+                s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+                s.bind((host, port))
+            return port
+        except OSError:
+            continue
+    return start
+
+
+def get_version():
+    """Get version from git tag or VERSION file."""
+    try:
+        import subprocess
+        tag = subprocess.check_output(
+            ["git", "describe", "--tags", "--abbrev=0"],
+            stderr=subprocess.DEVNULL,
+            text=True
+        ).strip()
+        if tag.startswith("v"):
+            return tag[1:]
+        return tag
+    except Exception:
+        version_file = os.path.join(os.path.dirname(__file__), "VERSION")
+        if os.path.exists(version_file):
+            with open(version_file) as f:
+                return f.read().strip()
+        return "0.0.0"
+
+
+def load_or_create_identity():
+    os.makedirs(DATA_DIR, exist_ok=True)
+    identity_path = os.path.join(DATA_DIR, IDENTITY_FILE)
+    if os.path.isfile(identity_path):
+        current = os.stat(identity_path).st_mode & 0o777
+        if current != 0o600:
+            os.chmod(identity_path, 0o600)
+        return RNS.Identity.from_file(identity_path)
+    identity = RNS.Identity()
+    identity.to_file(identity_path)
+    os.chmod(identity_path, 0o600)
+    return identity
+
+
+# Remote peers on the Reticulum mesh can reach read-only public pages.
+# Only GET is allowed; POST is blocked because CSRF cannot authenticate
+# mesh callers (the attacker controls both the "cookie" and the "form" side).
+_RNS_ALLOWED_GET = {
+    "/", "/about", "/api/sites", "/share/preview",
+}
+
+_RNS_ALLOWED_PREFIXES = ("/pages", "/tags", "/api/sites", "/rns")
+
+
+def _rns_is_allowed(method, path):
+    if method != "GET":
+        return False
+    if path in _RNS_ALLOWED_GET:
+        return True
+    return any(path.startswith(p) for p in _RNS_ALLOWED_PREFIXES)
+
+
+def rns_request_handler(path, data, request_id, link_id, remote_identity, requested_at):
+    if data is None:
+        data = {"method": "GET", "path": "/", "query": {}, "body": {}, "gateway_host": ""}
+    method = data.get("method", "GET")
+    req_path = data.get("path", "/")
+    if not _rns_is_allowed(method, req_path):
+        return {
+            "status": 403,
+            "content_type": "text/plain; charset=utf-8",
+            "body": "Forbidden: this endpoint is not available over Reticulum.",
+            "headers": {},
+        }
+    return dispatch_request(data)
+
+
+def start_gateway(reticulum, bind_host="127.0.0.1"):
+    GatewayState.reticulum = reticulum
+    GatewayState.local_dispatch = dispatch_request
+    HTTPServer.allow_reuse_address = True
+    server = ThreadingHTTPServer((bind_host, gateway.GATEWAY_PORT), GatewayHandler)
+    thread = threading.Thread(target=server.serve_forever, daemon=True)
+    thread.start()
+
+
+def _config_settings_match(config_file, desired_host, desired_port):
+    """Check if existing config transport and LoRa settings match desired values."""
+    import configparser
+    try:
+        config = configparser.ConfigParser()
+        config.read(config_file)
+        # Check TCP transport
+        tcp_enabled = get_setting("tcp_enabled", "1") == "1"
+        has_tcp = config.has_section("TCP Transport")
+        if tcp_enabled != has_tcp:
+            return False
+        if tcp_enabled and has_tcp:
+            if (config.get("TCP Transport", "target_host") != desired_host or
+                    config.get("TCP Transport", "target_port") != str(desired_port)):
+                return False
+        # Check LoRa
+        lora_enabled = get_setting("lora_enabled", "0") == "1"
+        has_lora = config.has_section("RNode LoRa")
+        if lora_enabled != has_lora:
+            return False
+        if lora_enabled and has_lora:
+            if config.get("RNode LoRa", "port", fallback="") != get_setting("lora_port", ""):
+                return False
+            if config.get("RNode LoRa", "frequency", fallback="") != get_setting("lora_frequency", "867200000"):
+                return False
+        return True
+    except Exception:
+        pass
+    return False
+
+
+def ensure_rns_config(config_dir, transport_host=None, transport_port=None):
+    """Generate a default Reticulum config with internet transport if none exists."""
+    if config_dir is None:
+        config_dir = os.path.expanduser("~/.reticulum")
+    config_file = os.path.join(config_dir, "config")
+    if transport_host is None:
+        transport_host = get_setting("transport_host", DEFAULT_TRANSPORT_HOST)
+    if transport_port is None:
+        transport_port = int(get_setting("transport_port", str(DEFAULT_TRANSPORT_PORT)))
+
+    managed_sentinel = "# managed by tinyweb"
+    if os.path.exists(config_file):
+        try:
+            with open(config_file) as f:
+                existing = f.read()
+        except OSError:
+            existing = ""
+        if managed_sentinel not in existing:
+            # User-authored config — don't clobber it.
+            if not _config_settings_match(config_file, transport_host, transport_port):
+                print(
+                    f"Warning: {config_file} was not created by tinyweb; "
+                    "leaving it alone. Edit it manually to change transport/LoRa settings."
+                )
+            return
+        if _config_settings_match(config_file, transport_host, transport_port):
+            return
+
+    # Build optional interface blocks
+    tcp_block = ""
+    if get_setting("tcp_enabled", "1") == "1":
+        tcp_block = f"""
+  [[TCP Transport]]
+    type = TCPClientInterface
+    enabled = yes
+    target_host = {transport_host}
+    target_port = {transport_port}
+"""
+
+    lora_block = ""
+    if get_setting("lora_enabled", "0") == "1":
+        lora_port = get_setting("lora_port", "")
+        if lora_port:
+            lora_frequency = get_setting("lora_frequency", "867200000")
+            lora_bandwidth = get_setting("lora_bandwidth", "125000")
+            lora_txpower = get_setting("lora_txpower", "7")
+            lora_sf = get_setting("lora_sf", "8")
+            lora_cr = get_setting("lora_cr", "5")
+            lora_block = f"""
+  [[RNode LoRa]]
+    type = RNodeInterface
+    enabled = yes
+    port = {lora_port}
+    frequency = {lora_frequency}
+    bandwidth = {lora_bandwidth}
+    txpower = {lora_txpower}
+    spreadingfactor = {lora_sf}
+    codingrate = {lora_cr}
+"""
+
+    os.makedirs(config_dir, exist_ok=True)
+    with open(config_file, "w") as f:
+        f.write(f"""{managed_sentinel}
+[reticulum]
+  enable_transport = False
+  share_instance = No
+
+[logging]
+  loglevel = 4
+
+[interfaces]
+  [[Default Interface]]
+    type = AutoInterface
+    enabled = Yes
+{tcp_block}{lora_block}""")
+    print(f"Created Reticulum config at {config_file}")
+
+
+def _preload_embeddings():
+    """Pre-load the embedding model and build the HNSW index in background."""
+    if get_setting("semantic_search", "0") != "1":
+        print("Semantic search disabled.")
+        return
+    try:
+        from tinyweb.embeddings import _get_session, _get_reranker, build_index
+        _get_session()
+        build_index()
+        if get_setting("use_reranker", "0") == "1":
+            _get_reranker()
+            print("Semantic search ready (with reranker).")
+        else:
+            print("Semantic search ready.")
+    except Exception as e:
+        print(f"Semantic search unavailable: {e}")
+
+
+def main():
+    parser = argparse.ArgumentParser(prog="tinyweb", description="Personal decentralized search engine")
+    parser.add_argument("--version", "-v", action="store_true", help="Show version")
+    parser.add_argument("--port", "-p", type=int, default=None, help="HTTP gateway port (default: 8080)")
+    parser.add_argument(
+        "--bind", "-b", default="127.0.0.1",
+        help="Address to bind the HTTP gateway to (default: 127.0.0.1). "
+             "Use 0.0.0.0 to expose to the LAN; note that the web UI has no authentication.",
+    )
+    args = parser.parse_args()
+
+    if args.version:
+        print(f"TinyWeb {get_version()}")
+        return
+
+    bind_host = args.bind
+    port = args.port or 8080
+    gateway.GATEWAY_PORT = find_available_port(port, host=bind_host)
+
+    init_db()
+    transport_host = get_setting("transport_host", DEFAULT_TRANSPORT_HOST)
+    transport_port = int(get_setting("transport_port", str(DEFAULT_TRANSPORT_PORT)))
+    threading.Thread(target=_preload_embeddings, daemon=True).start()
+    config_dir = os.environ.get("RNS_CONFIG_DIR")
+    ensure_rns_config(config_dir, transport_host, transport_port)
+    reticulum = RNS.Reticulum(configdir=config_dir)
+    identity = load_or_create_identity()
+
+    destination = RNS.Destination(
+        identity,
+        RNS.Destination.IN,
+        RNS.Destination.SINGLE,
+        gateway.APP_NAME,
+        *gateway.ASPECTS,
+    )
+
+    destination.register_request_handler(
+        "/tinyweb",
+        response_generator=rns_request_handler,
+        allow=RNS.Destination.ALLOW_ALL,
+    )
+
+    # Initialize forum plugin if available
+    forum = None
+    try:
+        from tinyweb_forum import ForumPlugin
+        from tinyweb.db import get_site_name
+        forum = ForumPlugin(DATA_DIR, identity, reticulum, site_name=get_site_name())
+        if get_setting("forum_enabled", "0") == "1":
+            forum.enable()
+            templates_mod.FORUM_ENABLED = True
+        handlers_mod.forum_plugin = forum
+        print(f"Forum plugin: {'enabled' if forum.is_enabled() else 'available (enable in settings)'}")
+    except ImportError:
+        print("Forum plugin not installed (pip install tinyweb[forum])")
+    except Exception as e:
+        print(f"Forum plugin error: {e}")
+
+    # Brief delay to ensure all interfaces (especially TCP) are fully ready
+    time.sleep(2)
+    destination.announce()
+    set_setting("dest_hash", destination.hash.hex())
+    start_gateway(reticulum, bind_host=bind_host)
+
+    print(f"TinyWeb running!")
+    if bind_host in ("0.0.0.0", "::"):
+        print(f"Open http://localhost:{gateway.GATEWAY_PORT} in your browser")
+        print(f"WARNING: listening on {bind_host} — the web UI has no authentication. "
+              "Anyone on your network can control this instance.")
+    else:
+        print(f"Open http://{bind_host}:{gateway.GATEWAY_PORT} in your browser")
+    print(f"Destination hash: {RNS.prettyhexrep(destination.hash)} (share this so friends can subscribe)")
+
+    while True:
+        time.sleep(1)
+
+
+if __name__ == "__main__":
+    main()

src/tinyweb/db.py

@@ -2,10 +2,12 @@ import socket
 import ipaddress
 import sqlite3
 import requests
+import os
 from urllib.parse import urlparse, urljoin, parse_qs, urlencode, urlunparse, quote
 from bs4 import BeautifulSoup
 
-DATABASE = "index.db"
+DATA_DIR = os.environ.get("TINYWEB_DATA_DIR") or os.path.expanduser("~/.tinyweb")
+DATABASE = os.path.join(DATA_DIR, "index.db")
 
 BLOCKED_NETWORKS = [
     ipaddress.ip_network("127.0.0.0/8"),
@@ -20,6 +22,22 @@ BLOCKED_NETWORKS = [
 ]
 
 
+def _is_blocked_response(html, status_code):
+    """Check if response is a CDN challenge/block page."""
+    if status_code == 403:
+        return True
+    html_lower = html.lower()
+    if "just a moment" in html_lower or "cloudflare" in html_lower:
+        return True
+    if "enable javascript and cookies" in html_lower:
+        return True
+    if "request rejected" in html_lower:
+        return True
+    if "access denied" in html_lower:
+        return True
+    return False
+
+
 def _validate_url_target(url):
     """Resolve hostname and block private/internal IPs to prevent SSRF."""
     parsed = urlparse(url)
@@ -52,10 +70,16 @@ def clean_url(url):
     # Prefer https
     scheme = "https" if parsed.scheme in ("http", "https") else parsed.scheme
 
-    # Normalize hostname: lowercase, strip www.
+    # Normalize hostname: lowercase, strip www (only if non-www resolves)
     hostname = (parsed.hostname or "").lower()
+    original_hostname = hostname
     if hostname.startswith("www."):
         hostname = hostname[4:]
+        port = parsed.port or (443 if scheme == "https" else 80)
+        try:
+            socket.getaddrinfo(hostname, port, proto=socket.IPPROTO_TCP)
+        except socket.gaierror:
+            hostname = original_hostname
 
     # Preserve explicit non-default ports
     port = parsed.port
@@ -79,7 +103,7 @@ def clean_url(url):
 
 _pool = []
 _pool_lock = __import__("threading").Lock()
-_POOL_SIZE = 4
+_POOL_SIZE = 16
 
 
 def get_db():
@@ -99,6 +123,14 @@ def get_db():
 
 
 def return_db(db):
+    try:
+        db.rollback()
+    except Exception:
+        try:
+            db.close()
+        except Exception:
+            pass
+        return
     with _pool_lock:
         if len(_pool) < _POOL_SIZE:
             _pool.append(db)
@@ -107,6 +139,7 @@ def return_db(db):
 
 
 def init_db():
+    os.makedirs(DATA_DIR, exist_ok=True)
     db = sqlite3.connect(DATABASE)
     db.execute(
         "CREATE TABLE IF NOT EXISTS pages ("
@@ -115,7 +148,8 @@ def init_db():
         "  title TEXT,"
         "  body TEXT,"
         "  note TEXT DEFAULT '',"
-        "  last_modified TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%S','now'))"
+        "  last_modified TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%S','now')),"
+        "  reticulum_dest TEXT DEFAULT ''"
         ")"
     )
     db.execute(
@@ -207,26 +241,66 @@ def init_db():
             VALUES (new.id, new.title, new.url, new.note);
         END;
     """)
-    # Migrate old subscriptions table if needed
+    db.execute(
-    cols = [row[1] for row in db.execute("PRAGMA table_info(subscriptions)").fetchall()]
+        "CREATE TABLE IF NOT EXISTS mesh_sites ("
-    if "url" in cols and "dest_hash" not in cols:
+        "  hash TEXT PRIMARY KEY,"
-        db.execute("ALTER TABLE subscriptions RENAME COLUMN url TO dest_hash")
+        "  name TEXT DEFAULT '',"
+        "  added_at TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%S','now'))"
+        ")"
+    )
+    # Migrate pages: add reticulum_dest column if missing
+    if "reticulum_dest" not in page_cols:
+        db.execute("ALTER TABLE pages ADD COLUMN reticulum_dest TEXT DEFAULT ''")
         db.commit()
 
-    # Migrate remote_pages: add tags column if missing
+    # Migrate subscriptions: add forum_enabled column
-    rp_cols = [row[1] for row in db.execute("PRAGMA table_info(remote_pages)").fetchall()]
+    sub_cols = [row[1] for row in db.execute("PRAGMA table_info(subscriptions)").fetchall()]
-    if "tags" not in rp_cols:
+    if "forum_enabled" not in sub_cols:
-        db.execute("ALTER TABLE remote_pages ADD COLUMN tags TEXT DEFAULT ''")
+        db.execute("ALTER TABLE subscriptions ADD COLUMN forum_enabled INTEGER DEFAULT 0")
         db.commit()
 
-    # Migrate pages: add last_modified column if missing
+    # Chunks table for semantic search embeddings
-    page_cols = [row[1] for row in db.execute("PRAGMA table_info(pages)").fetchall()]
+    db.execute(
-    if "last_modified" not in page_cols:
+        "CREATE TABLE IF NOT EXISTS chunks ("
-        db.execute("ALTER TABLE pages ADD COLUMN last_modified TEXT DEFAULT ''")
+        "  id INTEGER PRIMARY KEY AUTOINCREMENT,"
-        db.execute("UPDATE pages SET last_modified = strftime('%Y-%m-%dT%H:%M:%S','now') WHERE last_modified = ''")
+        "  page_id INTEGER,"
+        "  remote_page_id INTEGER,"
+        "  chunk_index INTEGER NOT NULL,"
+        "  chunk_text TEXT NOT NULL,"
+        "  embedding BLOB NOT NULL,"
+        "  FOREIGN KEY (page_id) REFERENCES pages(id) ON DELETE CASCADE,"
+        "  FOREIGN KEY (remote_page_id) REFERENCES remote_pages(id) ON DELETE CASCADE"
+        ")"
+    )
+    db.execute("CREATE INDEX IF NOT EXISTS idx_chunks_page ON chunks(page_id)")
+    db.execute("CREATE INDEX IF NOT EXISTS idx_chunks_remote ON chunks(remote_page_id)")
+    db.execute("CREATE INDEX IF NOT EXISTS idx_chunks_page_idx ON chunks(page_id, chunk_index)")
+    db.execute("CREATE INDEX IF NOT EXISTS idx_pages_url ON pages(url)")
+    db.execute("CREATE INDEX IF NOT EXISTS idx_pages_modified ON pages(last_modified)")
+    db.execute("CREATE INDEX IF NOT EXISTS idx_page_tags_page ON page_tags(page_id)")
+    db.execute("CREATE INDEX IF NOT EXISTS idx_page_tags_tag ON page_tags(tag_id)")
+
+    # Migrate custom_template: replace hardcoded forum link with {{forum_link}} placeholder
+    cur = db.execute("SELECT value FROM settings WHERE key='custom_template'")
+    row = cur.fetchone()
+    if row:
+        updated = row[0].replace('<a href="/forum">forum</a>', "{{forum_link}}")
+        if updated != row[0]:
+            db.execute("UPDATE settings SET value=? WHERE key='custom_template'", (updated,))
+            db.commit()
+
+    # Migrate custom_template: replace hardcoded site name with {{site_name}} placeholder
+    cur = db.execute("SELECT value FROM settings WHERE key='custom_template'")
+    row = cur.fetchone()
+    if row and '{{site_name}}' not in row[0]:
+        updated = row[0].replace('href="/">tinyweb</a>', 'href="/">{{site_name}}</a>')
+        if updated != row[0]:
+            db.execute("UPDATE settings SET value=? WHERE key='custom_template'", (updated,))
             db.commit()
 
     db.execute("PRAGMA journal_mode=WAL")
+    db.execute("PRAGMA synchronous=NORMAL")
+    db.execute("PRAGMA cache_size=-64000")
     db.commit()
     db.close()
 
@@ -240,6 +314,16 @@ def get_setting(key, default=""):
         return_db(db)
 
 
+def vacuum_db():
+    """Run VACUUM and WAL checkpoint to reclaim space after deletions."""
+    db = get_db()
+    try:
+        db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
+        db.execute("VACUUM")
+    finally:
+        return_db(db)
+
+
 def set_setting(key, value):
     db = get_db()
     try:
@@ -260,6 +344,10 @@ def get_site_name():
 def fetch_page(url):
     _validate_url_target(url)
     resp = requests.get(url, timeout=10, headers={"User-Agent": "TinyWeb/1.0"}, allow_redirects=False)
+
+    if _is_blocked_response(resp.text, resp.status_code):
+        raise Exception(f"Site blocks automated access: {resp.status_code}")
+
     # Follow redirects manually, re-validating each target
     max_redirects = 5
     while resp.is_redirect and max_redirects > 0:
@@ -296,24 +384,40 @@ def fetch_page(url):
         label = a.get_text(strip=True) or href
         links.append((href, label[:200]))
 
-    for tag in soup(["script", "style", "nav", "footer", "header"]):
+    # Extract meta description before stripping tags (case-insensitive)
+    meta_desc = ""
+    for m in soup.find_all("meta"):
+        name = (m.get("name") or "").lower()
+        prop = (m.get("property") or "").lower()
+        content = (m.get("content") or "").strip()
+        if not content:
+            continue
+        if name == "description" and len(content) > len(meta_desc):
+            meta_desc = content
+        elif prop == "og:description" and not meta_desc:
+            meta_desc = content
+
+    for tag in soup(["script", "style", "nav", "footer", "header", "noscript", "aside"]):
         tag.decompose()
     title = soup.title.string.strip() if soup.title and soup.title.string else url
+
     body = soup.get_text(separator=" ", strip=True)
-    return title, body, links
+    return title, body, links, meta_desc
 
 
-def index_url(url, note=""):
+
+def index_url(url, note="", reticulum_dest=""):
     url = clean_url(url)
-    title, body, links = fetch_page(url)
+    title, body, links, meta_desc = fetch_page(url)
+    summary = meta_desc if meta_desc and len(meta_desc) > 20 else ""
     db = get_db()
     try:
         now = __import__("datetime").datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
         db.execute(
-            "INSERT INTO pages (url, title, body, note, last_modified) VALUES (?, ?, ?, ?, ?) "
+            "INSERT INTO pages (url, title, body, note, last_modified, summary, reticulum_dest) VALUES (?, ?, ?, ?, ?, ?, ?) "
             "ON CONFLICT(url) DO UPDATE SET title=excluded.title, body=excluded.body, "
-            "note=excluded.note, last_modified=excluded.last_modified",
+            "note=excluded.note, last_modified=excluded.last_modified, summary=excluded.summary, reticulum_dest=excluded.reticulum_dest",
-            (url, title, body, note, now),
+            (url, title, body, note, now, summary, reticulum_dest),
         )
         page_id = db.execute("SELECT id FROM pages WHERE url = ?", (url,)).fetchone()[0]
         db.execute("DELETE FROM links WHERE page_id = ?", (page_id,))
@@ -323,6 +427,12 @@ def index_url(url, note=""):
                 (page_id, href, label),
             )
         db.commit()
+        if get_setting("semantic_search", "0") == "1":
+            try:
+                from tinyweb.embeddings import store_embeddings
+                store_embeddings(page_id, title, body, db)
+            except Exception:
+                pass  # embedding generation is best-effort
     finally:
         return_db(db)
     return title

src/tinyweb/embeddings.py

@@ -0,0 +1,586 @@
+"""Semantic search using Snowflake arctic-embed-s via ONNX Runtime + hnswlib."""
+
+import os
+import re
+import threading
+import numpy as np
+
+DATA_DIR = os.path.expanduser("~/.tinyweb")
+MODEL_ID = "Snowflake/snowflake-arctic-embed-s"
+MODEL_DIR = os.path.join(DATA_DIR, "models", "snowflake-arctic-embed-s")
+RERANKER_DIR = os.path.join(DATA_DIR, "models", "cross-encoder")
+HNSW_PATH = os.path.join(DATA_DIR, "index.hnsw")
+DIMS = 384
+MAX_TOKENS = 512
+QUERY_PREFIX = "Represent this sentence for searching relevant passages: "
+
+_session = None
+_tokenizer = None
+_lock = threading.Lock()
+
+_reranker_session = None
+_reranker_tokenizer = None
+_reranker_lock = threading.Lock()
+
+# Live HNSW index and chunk-id mapping
+_hnsw_index = None
+_hnsw_ids = []  # maps internal HNSW label -> chunks.id
+_hnsw_lock = threading.Lock()
+
+
+# ---------------------------------------------------------------------------
+# Model download & loading
+# ---------------------------------------------------------------------------
+
+def _ensure_model():
+    """Download the ONNX model and tokenizer from HuggingFace if not present."""
+    os.makedirs(MODEL_DIR, exist_ok=True)
+    model_path = os.path.join(MODEL_DIR, "model.onnx")
+    tokenizer_path = os.path.join(MODEL_DIR, "tokenizer.json")
+    if os.path.exists(model_path) and os.path.exists(tokenizer_path):
+        return
+    from huggingface_hub import hf_hub_download
+    os.makedirs(MODEL_DIR, exist_ok=True)
+    files = {
+        "onnx/model_quantized.onnx": "model.onnx",
+        "tokenizer.json": "tokenizer.json",
+        "tokenizer_config.json": "tokenizer_config.json",
+    }
+    for remote, local in files.items():
+        target = os.path.join(MODEL_DIR, local)
+        if os.path.exists(target):
+            continue
+        cached = hf_hub_download(repo_id=MODEL_ID, filename=remote)
+        # hf_hub_download returns the cached file path; copy to our model dir
+        import shutil
+        shutil.copy2(cached, target)
+
+
+def _get_session():
+    """Return (onnxruntime.InferenceSession, tokenizers.Tokenizer) singleton."""
+    global _session, _tokenizer
+    if _session is not None:
+        return _session, _tokenizer
+    with _lock:
+        if _session is not None:
+            return _session, _tokenizer
+        _ensure_model()
+        import onnxruntime as ort
+        from tokenizers import Tokenizer
+        _session = ort.InferenceSession(
+            os.path.join(MODEL_DIR, "model.onnx"),
+            providers=["CPUExecutionProvider"],
+        )
+        _tokenizer = Tokenizer.from_file(os.path.join(MODEL_DIR, "tokenizer.json"))
+        _tokenizer.enable_truncation(max_length=MAX_TOKENS)
+        _tokenizer.enable_padding(pad_id=0, pad_token="[PAD]", length=None)
+        return _session, _tokenizer
+
+
+def _get_reranker():
+    """Return (onnxruntime.InferenceSession, tokenizers.Tokenizer) for the cross-encoder reranker."""
+    global _reranker_session, _reranker_tokenizer
+    if _reranker_session is not None:
+        return _reranker_session, _reranker_tokenizer
+    with _reranker_lock:
+        if _reranker_session is not None:
+            return _reranker_session, _reranker_tokenizer
+        model_path = os.path.join(RERANKER_DIR, "model.onnx")
+        tok_path = os.path.join(RERANKER_DIR, "tokenizer.json")
+        if not os.path.exists(model_path) or not os.path.exists(tok_path):
+            return None, None
+        import onnxruntime as ort
+        from tokenizers import Tokenizer
+        _reranker_session = ort.InferenceSession(
+            model_path, providers=["CPUExecutionProvider"],
+        )
+        _reranker_tokenizer = Tokenizer.from_file(tok_path)
+        _reranker_tokenizer.enable_truncation(max_length=512)
+        _reranker_tokenizer.enable_padding(pad_id=0, pad_token="[PAD]", length=None)
+        return _reranker_session, _reranker_tokenizer
+
+
+def rerank(query, documents, limit=10):
+    """Score query-document pairs with the cross-encoder and return reranked indices.
+
+    Args:
+        query: search query string
+        documents: list of document texts to score against the query
+        limit: max results to return
+
+    Returns: list of (original_index, score) sorted by score descending.
+    """
+    session, tokenizer = _get_reranker()
+    if session is None:
+        return [(i, 0.0) for i in range(min(limit, len(documents)))]
+
+    # Cross-encoder takes (query, document) pairs — encode as pair sequences
+    pairs = [[query, doc] for doc in documents]
+    encodings = tokenizer.encode_batch(pairs)
+
+    input_ids = np.array([e.ids for e in encodings], dtype=np.int64)
+    attention_mask = np.array([e.attention_mask for e in encodings], dtype=np.int64)
+    token_type_ids = np.array([e.type_ids for e in encodings], dtype=np.int64)
+
+    outputs = session.run(
+        None,
+        {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "token_type_ids": token_type_ids,
+        },
+    )
+    # Output is logits — higher = more relevant
+    scores = outputs[0].flatten()
+    ranked = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
+    return [(i, float(scores[i])) for i in ranked[:limit]]
+
+
+# ---------------------------------------------------------------------------
+# Text chunking
+# ---------------------------------------------------------------------------
+
+_SENTENCE_RE = re.compile(r'(?<=[.!?])\s+')
+
+
+def chunk_text(title, body):
+    """Split body into chunks, each prefixed with title for context.
+
+    Strategy: split on double newlines (paragraphs). If a paragraph exceeds
+    MAX_TOKENS words, split at sentence boundaries. Each chunk is prefixed
+    with the page title.
+    """
+    if not body or not body.strip():
+        return [f"{title}"] if title else []
+
+    prefix = f"{title}: " if title else ""
+    # Rough word budget for chunk body (leave room for prefix)
+    prefix_words = len(prefix.split())
+    max_words = MAX_TOKENS - prefix_words  # approximate; tokenizer may differ
+
+    paragraphs = re.split(r'\n\s*\n', body.strip())
+    chunks = []
+
+    for para in paragraphs:
+        para = para.strip()
+        if len(para) < 20:
+            continue
+        words = para.split()
+        if len(words) <= max_words:
+            chunks.append(prefix + para)
+        else:
+            # Split paragraph into sentences
+            sentences = _SENTENCE_RE.split(para)
+            current = []
+            current_len = 0
+            for sent in sentences:
+                sent_words = len(sent.split())
+                if current_len + sent_words > max_words and current:
+                    chunks.append(prefix + " ".join(current))
+                    current = []
+                    current_len = 0
+                if sent_words > max_words:
+                    # Sentence too long — use sliding window
+                    s_words = sent.split()
+                    for i in range(0, len(s_words), max_words - 50):
+                        window = s_words[i:i + max_words]
+                        chunks.append(prefix + " ".join(window))
+                else:
+                    current.append(sent)
+                    current_len += sent_words
+            if current:
+                chunks.append(prefix + " ".join(current))
+
+    if not chunks and title:
+        chunks = [title]
+
+    return chunks
+
+
+# ---------------------------------------------------------------------------
+# Embedding
+# ---------------------------------------------------------------------------
+
+def embed(texts, is_query=False):
+    """Encode texts into L2-normalized float32 embeddings (N, 384).
+
+    For queries, prepend the model's query prefix.
+    Processes in batches of 32 to limit memory usage.
+    """
+    if not texts:
+        return np.empty((0, DIMS), dtype=np.float32)
+
+    session, tokenizer = _get_session()
+
+    if is_query:
+        texts = [QUERY_PREFIX + t for t in texts]
+
+    batch_size = 32
+    all_embeddings = []
+
+    for start in range(0, len(texts), batch_size):
+        batch = texts[start:start + batch_size]
+        encodings = tokenizer.encode_batch(batch)
+        input_ids = np.array([e.ids for e in encodings], dtype=np.int64)
+        attention_mask = np.array([e.attention_mask for e in encodings], dtype=np.int64)
+        token_type_ids = np.zeros_like(input_ids)
+
+        outputs = session.run(
+            None,
+            {
+                "input_ids": input_ids,
+                "attention_mask": attention_mask,
+                "token_type_ids": token_type_ids,
+            },
+        )
+        emb = outputs[0][:, 0, :]
+        all_embeddings.append(emb)
+
+    embeddings = np.concatenate(all_embeddings, axis=0)
+    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
+    norms = np.maximum(norms, 1e-12)
+    embeddings = embeddings / norms
+    return _maybe_compress(embeddings.astype(np.float32))
+
+
+def _maybe_compress(embeddings):
+    """Compress embeddings to float16 if compression is enabled."""
+    try:
+        from tinyweb.db import get_setting
+        if get_setting("compress_embeddings", "0") == "1":
+            return embeddings.astype(np.float16)
+    except Exception:
+        pass
+    return embeddings
+
+
+def _decompress(embeddings):
+    """Decompress float16 embeddings to float32 if needed."""
+    if embeddings.dtype == np.float16:
+        return embeddings.astype(np.float32)
+    return embeddings
+
+
+def _blob_to_vec(buf):
+    """Decode a stored embedding blob to a float32 vector, inferring dtype from length."""
+    if len(buf) == DIMS * 2:
+        return np.frombuffer(buf, dtype=np.float16).astype(np.float32)
+    return np.frombuffer(buf, dtype=np.float32)
+
+
+# ---------------------------------------------------------------------------
+# HNSW index management
+# ---------------------------------------------------------------------------
+
+BATCH_SIZE = 50000
+
+def build_index(db=None):
+    """Load all embeddings from chunks table and build HNSW index in batches."""
+    import hnswlib
+    global _hnsw_index, _hnsw_ids
+
+    from tinyweb.db import get_db, return_db
+    own_db = db is None
+    if own_db:
+        db = get_db()
+
+    try:
+        total = db.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
+        if total == 0:
+            with _hnsw_lock:
+                _hnsw_index = None
+                _hnsw_ids = []
+            return
+
+        all_ids = []
+        all_embeddings = []
+
+        for offset in range(0, total, BATCH_SIZE):
+            rows = db.execute(
+                "SELECT id, embedding FROM chunks ORDER BY id LIMIT ? OFFSET ?",
+                (BATCH_SIZE, offset),
+            ).fetchall()
+            for r in rows:
+                emb = _blob_to_vec(r["embedding"])
+                all_ids.append(r["id"])
+                all_embeddings.append(emb)
+    finally:
+        if own_db:
+            return_db(db)
+
+    if not all_ids:
+        with _hnsw_lock:
+            _hnsw_index = None
+            _hnsw_ids = []
+        return
+
+    matrix = np.stack(all_embeddings)
+    n = len(all_ids)
+    ids = all_ids
+
+    index = hnswlib.Index(space="cosine", dim=DIMS)
+    index.init_index(max_elements=max(n, 1024), ef_construction=200, M=16)
+    index.add_items(matrix, list(range(n)))
+    index.set_ef(50)
+
+    with _hnsw_lock:
+        _hnsw_index = index
+        _hnsw_ids = ids
+
+
+def _add_to_index(chunk_ids, embeddings_matrix):
+    """Add new embeddings to the live HNSW index."""
+    import hnswlib
+    global _hnsw_index, _hnsw_ids
+
+    with _hnsw_lock:
+        if _hnsw_index is None:
+            index = hnswlib.Index(space="cosine", dim=DIMS)
+            index.init_index(max_elements=1024, ef_construction=200, M=16)
+            index.set_ef(50)
+            _hnsw_index = index
+            _hnsw_ids = []
+
+        current_max = _hnsw_index.get_max_elements()
+        needed = len(_hnsw_ids) + len(chunk_ids)
+        if needed > current_max:
+            _hnsw_index.resize_index(max(needed * 2, current_max * 2))
+
+        labels = list(range(len(_hnsw_ids), len(_hnsw_ids) + len(chunk_ids)))
+        _hnsw_index.add_items(embeddings_matrix, labels)
+        _hnsw_ids.extend(chunk_ids)
+
+
+# ---------------------------------------------------------------------------
+# Store embeddings for pages
+# ---------------------------------------------------------------------------
+
+def store_embeddings(page_id, title, body, db):
+    """Chunk, embed, and store embeddings for a page. Adds to HNSW index."""
+    chunks = chunk_text(title, body)
+    if not chunks:
+        return
+
+    embeddings_matrix = embed(chunks)
+    embeddings_matrix = _decompress(embeddings_matrix)
+
+    db.execute("DELETE FROM chunks WHERE page_id = ?", (page_id,))
+
+    new_ids = []
+    for i, (text, emb) in enumerate(zip(chunks, embeddings_matrix)):
+        cursor = db.execute(
+            "INSERT INTO chunks (page_id, remote_page_id, chunk_index, chunk_text, embedding) "
+            "VALUES (?, NULL, ?, ?, ?)",
+            (page_id, i, text, emb.tobytes()),
+        )
+        new_ids.append(cursor.lastrowid)
+    db.commit()
+
+    _add_to_index(new_ids, embeddings_matrix)
+
+
+def store_remote_embeddings(remote_page_id, title, note, db):
+    """Store a single embedding for a remote page (title + note)."""
+    text = f"{title}: {note}" if note else (title or "")
+    if not text.strip():
+        return
+
+    embeddings_matrix = embed([text])
+    embeddings_matrix = _decompress(embeddings_matrix)
+
+    db.execute("DELETE FROM chunks WHERE remote_page_id = ?", (remote_page_id,))
+    cursor = db.execute(
+        "INSERT INTO chunks (page_id, remote_page_id, chunk_index, chunk_text, embedding) "
+        "VALUES (NULL, ?, 0, ?, ?)",
+        (remote_page_id, text, embeddings_matrix[0].tobytes()),
+    )
+    db.commit()
+
+    _add_to_index([cursor.lastrowid], embeddings_matrix)
+
+
+# ---------------------------------------------------------------------------
+# Search
+# ---------------------------------------------------------------------------
+
+def semantic_search(query_text, limit=100, db=None):
+    """Search for pages by semantic similarity.
+
+    Returns: [(page_id, score, best_chunk_text), ...] sorted by score desc.
+    Groups by page_id, taking the max chunk score per page.
+    """
+    if _hnsw_index is None or not _hnsw_ids:
+        return []
+
+    query_emb = embed([query_text], is_query=True)
+
+    with _hnsw_lock:
+        if _hnsw_index is None or not _hnsw_ids:
+            return []
+        k = min(limit * 3, len(_hnsw_ids))  # oversample to account for grouping
+        if k == 0:
+            return []
+        labels, distances = _hnsw_index.knn_query(query_emb, k=k)
+
+    # Map HNSW labels back to chunk IDs
+    chunk_ids = [_hnsw_ids[int(lbl)] for lbl in labels[0]]
+    # cosine distance -> similarity: hnswlib returns 1-cosine for "cosine" space
+    scores = [1.0 - float(d) for d in distances[0]]
+
+    # Fetch chunk details from DB
+    from tinyweb.db import get_db, return_db
+    own_db = db is None
+    if own_db:
+        db = get_db()
+    try:
+        placeholders = ",".join("?" * len(chunk_ids))
+        rows = db.execute(
+            f"SELECT id, page_id, chunk_text FROM chunks WHERE id IN ({placeholders})",
+            chunk_ids,
+        ).fetchall()
+    finally:
+        if own_db:
+            return_db(db)
+
+    chunk_map = {r["id"]: r for r in rows}
+
+    # Group by page_id, keep best score and chunk text per page
+    page_best = {}  # page_id -> (score, chunk_text)
+    for cid, score in zip(chunk_ids, scores):
+        chunk = chunk_map.get(cid)
+        if not chunk or chunk["page_id"] is None:
+            continue
+        pid = chunk["page_id"]
+        if pid not in page_best or score > page_best[pid][0]:
+            page_best[pid] = (score, chunk["chunk_text"])
+
+    results = [(pid, score, text) for pid, (score, text) in page_best.items()]
+    results.sort(key=lambda x: x[1], reverse=True)
+    return results[:limit]
+
+
+def hybrid_search(query_text, bm25_ranked_ids, limit=10, db=None, use_reranker=False):
+    """Merge BM25 and semantic results via RRF, optionally rerank with cross-encoder.
+
+    Default (two-stage): BM25 + semantic fused via RRF.
+    With use_reranker=True (three-stage): rerank top 20 with cross-encoder.
+
+    Returns: [(page_id, best_chunk_text), ...] in ranked order.
+    """
+    k = 60  # RRF constant
+
+    sem_results = semantic_search(query_text, limit=100, db=db)
+
+    best_chunks = {}  # page_id -> chunk_text
+    for _rank, (pid, _score, chunk_text) in enumerate(sem_results):
+        if pid not in best_chunks:
+            best_chunks[pid] = chunk_text
+
+    # When BM25 has no hits, use raw semantic similarity scores directly
+    # (RRF rank positions distort nearly-equal scores)
+    if not bm25_ranked_ids:
+        fused_ids = [(pid, score) for pid, score, _ in sem_results]
+    else:
+        rrf_scores = {}
+        for rank, pid in enumerate(bm25_ranked_ids):
+            rrf_scores[pid] = rrf_scores.get(pid, 0) + 1.0 / (k + rank + 1)
+        for rank, (pid, _score, chunk_text) in enumerate(sem_results):
+            rrf_scores[pid] = rrf_scores.get(pid, 0) + 1.0 / (k + rank + 1)
+        fused_ids = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
+
+    fused = fused_ids
+    all_ids = [pid for pid, _ in fused]
+
+    if not all_ids:
+        return []
+
+    if not use_reranker:
+        return [(pid, best_chunks.get(pid, "")) for pid in all_ids[:limit]]
+
+    # --- Rerank top 20, append next 10 from RRF order ---
+    rerank_ids = all_ids[:20]
+    tail_ids = all_ids[20:30]
+
+    from tinyweb.db import get_db, return_db
+    own_db = db is None
+    if own_db:
+        db = get_db()
+    try:
+        placeholders = ",".join("?" * len(rerank_ids))
+        rows = db.execute(
+            f"SELECT id, title, body FROM pages WHERE id IN ({placeholders})",
+            rerank_ids,
+        ).fetchall()
+    finally:
+        if own_db:
+            return_db(db)
+
+    page_map = {r["id"]: r for r in rows}
+
+    doc_texts = []
+    ordered_ids = []
+    for pid in rerank_ids:
+        page = page_map.get(pid)
+        if not page:
+            continue
+        chunk = best_chunks.get(pid, "")
+        body_preview = chunk[:200] if chunk else page["body"][:200]
+        doc = f"{page['title']}. {body_preview}"
+        doc_texts.append(doc)
+        ordered_ids.append(pid)
+
+    if not doc_texts:
+        return []
+
+    try:
+        reranked = rerank(query_text, doc_texts, limit=20)
+        results = [(ordered_ids[idx], best_chunks.get(ordered_ids[idx], "")) for idx, _score in reranked]
+    except Exception:
+        results = [(pid, best_chunks.get(pid, "")) for pid in ordered_ids[:20]]
+
+    # Append next 10 from RRF order (no reranking)
+    reranked_set = {pid for pid, _ in results}
+    for pid in tail_ids:
+        if pid not in reranked_set:
+            results.append((pid, best_chunks.get(pid, "")))
+
+    return results[:30]
+
+
+# ---------------------------------------------------------------------------
+# Reindex
+# ---------------------------------------------------------------------------
+
+def reindex_all(db=None, progress_callback=None):
+    """Re-embed all pages and regenerate all summaries. Rebuilds HNSW index."""
+    from tinyweb.db import get_db, return_db
+    own_db = db is None
+    if own_db:
+        db = get_db()
+    try:
+        # Clear existing chunks so everything is regenerated
+        db.execute("DELETE FROM chunks")
+        db.commit()
+
+        rows = db.execute(
+            "SELECT p.id, p.title, p.body, p.summary FROM pages p"
+        ).fetchall()
+
+        total = len(rows)
+        for i, row in enumerate(rows):
+            store_embeddings(row["id"], row["title"], row["body"], db)
+            if progress_callback:
+                progress_callback(i + 1, total)
+
+        # Also handle remote pages
+        remote_rows = db.execute(
+            "SELECT rp.id, rp.title, rp.note FROM remote_pages rp"
+        ).fetchall()
+
+        for rp in remote_rows:
+            store_remote_embeddings(rp["id"], rp["title"], rp["note"], db)
+    finally:
+        if own_db:
+            return_db(db)
+
+    build_index(db)

src/tinyweb/gateway.py

@@ -1,14 +1,21 @@
+import re
 import sys
 import time
 import threading
+import collections
 import RNS
-from http.server import HTTPServer, BaseHTTPRequestHandler
+from http.server import HTTPServer, ThreadingHTTPServer, BaseHTTPRequestHandler
 from urllib.parse import parse_qs, urlparse
 
 APP_NAME = "tinyweb"
 ASPECTS = ["server"]
 GATEWAY_PORT = 8080
 REQUEST_TIMEOUT = 60
+MAX_BODY_SIZE = 16 * 1024 * 1024  # 16 MiB — covers /import and every other form
+RATE_LIMIT_WINDOW = 60
+RATE_LIMIT_MAX = 30
+_rate_tracker = collections.defaultdict(list)
+_rate_lock = threading.Lock()
 
 
 class GatewayState:
@@ -65,14 +72,40 @@ def ensure_link():
 
 class GatewayHandler(BaseHTTPRequestHandler):
 
+    def _check_rate_limit(self):
+        client = self.client_address[0]
+        now = time.time()
+        with _rate_lock:
+            times = _rate_tracker[client]
+            cutoff = now - RATE_LIMIT_WINDOW
+            while times and times[0] < cutoff:
+                times.pop(0)
+            if len(times) >= RATE_LIMIT_MAX:
+                return False
+            times.append(now)
+        return True
+
     def _forward(self, method):
         parsed = urlparse(self.path)
         query = parse_qs(parsed.query)
 
         body = {}
         if method == "POST":
+            if not self._check_rate_limit():
+                self.send_error(429, "Too many requests — slow down.")
+                return
+            try:
                 length = int(self.headers.get("Content-Length", 0))
-            raw = self.rfile.read(length).decode()
+            except ValueError:
+                self.send_error(400, "Invalid Content-Length")
+                return
+            if length < 0:
+                self.send_error(400, "Invalid Content-Length")
+                return
+            if length > MAX_BODY_SIZE:
+                self.send_error(413, "Request body too large")
+                return
+            raw = self.rfile.read(length).decode("utf-8", errors="replace")
             body = parse_qs(raw)
 
         # Parse cookies
@@ -92,6 +125,7 @@ class GatewayHandler(BaseHTTPRequestHandler):
             "body": body,
             "cookies": cookies,
             "gateway_host": self.headers.get("Host", f"localhost:{GATEWAY_PORT}"),
+            "scheme": self.headers.get("X-Forwarded-Proto", "http"),
         }
 
         try:
@@ -123,12 +157,23 @@ class GatewayHandler(BaseHTTPRequestHandler):
 
             self.send_response(resp["status"])
             self.send_header("Content-Type", resp.get("content_type", "text/html; charset=utf-8"))
+            self.send_header("Referrer-Policy", "no-referrer")
+            self.send_header("X-Content-Type-Options", "nosniff")
+            self.send_header("X-Frame-Options", "DENY")
+            self.send_header("Content-Security-Policy",
+                             "default-src 'self'; "
+                             "style-src 'self' 'unsafe-inline'; "
+                             "script-src 'self' 'unsafe-inline'; "
+                             "img-src 'self' data:")
+            resp_body = resp.get("body", "")
+            encoded = resp_body.encode() if isinstance(resp_body, str) else resp_body
+            if encoded:
+                self.send_header("Content-Length", str(len(encoded)))
             for k, v in resp.get("headers", {}).items():
                 self.send_header(k, v)
             self.end_headers()
-            resp_body = resp.get("body", "")
+            if encoded:
-            if resp_body:
+                self.wfile.write(encoded)
-                self.wfile.write(resp_body.encode() if isinstance(resp_body, str) else resp_body)
 
         except ConnectionError as e:
             GatewayState.link = None
@@ -144,12 +189,19 @@ class GatewayHandler(BaseHTTPRequestHandler):
         self._forward("POST")
 
     def log_message(self, format, *args):
-        print(f"[Gateway] {args[0]}")
+        try:
+            msg = format % args
+        except TypeError:
+            msg = format
+        # /bookmark carries a long-lived token and the URL being indexed —
+        # redact the query so it doesn't end up in stdout, journald, docker logs, etc.
+        msg = re.sub(r'(/bookmark)\?\S*', r'\1?[redacted]', msg)
+        print(f"[Gateway] {msg}")
 
 
 def main():
     if len(sys.argv) < 2:
-        print(f"Usage: python gateway.py <destination_hash>")
+        print(f"Usage: python -m tinyweb.gateway <destination_hash>")
         print(f"  The destination hash is printed by app.py on startup.")
         sys.exit(1)
 
@@ -160,7 +212,7 @@ def main():
 
     print(f"Gateway listening on http://localhost:{GATEWAY_PORT}")
     print(f"Open http://localhost:{GATEWAY_PORT} in your browser")
-    HTTPServer(("127.0.0.1", GATEWAY_PORT), GatewayHandler).serve_forever()
+    ThreadingHTTPServer(("127.0.0.1", GATEWAY_PORT), GatewayHandler).serve_forever()
 
 
 if __name__ == "__main__":

src/tinyweb/handlers/__init__.py

@@ -0,0 +1,217 @@
+import json
+import secrets
+import threading
+from urllib.parse import unquote
+
+from tinyweb.db import get_db, return_db, set_setting
+import tinyweb.templates as templates_mod
+from tinyweb.templates import esc, wrap_page
+from tinyweb.rns_client import fetch_remote_sites, fetch_remote_page
+
+from ._helpers import (
+    _request_local, _get_csrf_token, _csrf_field, _check_csrf,
+    _sanitize_fts_query, _get_bookmark_token,
+    _respond, _redirect, _json_response, _text_response, _error,
+    PER_PAGE, BROWSE_PER_PAGE, _paginate, _page_nav,
+    _get_page_tags, _set_page_tags, _cleanup_orphaned_tags,
+)
+from .search import handle_search
+from .pages import (
+    handle_add_form, handle_add_submit, handle_add_manual_submit,
+    handle_pages, _render_bulk_delete_confirm, handle_bulk_action,
+    handle_edit_form, handle_edit_submit,
+    handle_delete_confirm, handle_delete,
+    handle_bookmark,
+)
+from .subscriptions import (
+    _page_is_shared, _shared_sites, _shared_all_urls, _count_shared_pages,
+    handle_share_preview, handle_api_sites,
+    handle_subscriptions, handle_subscription_add, handle_subscription_browse,
+    handle_subscription_pick, _sync_subscription,
+    handle_subscription_sync, handle_subscription_autosync,
+    handle_subscription_delete, handle_subscription_syncall,
+    _sync_threads,
+)
+from .customize import handle_style_form, handle_style_submit, handle_style_template_submit, handle_field_save, handle_about, _set_flash
+from .tags import handle_tags, handle_tag_browse
+from .data import (
+    handle_export, handle_import_form, handle_import_submit,
+    handle_reindex_form, handle_reindex_submit, _reindex_thread,
+)
+from .rns import (
+    handle_rns_delete_hash, handle_rns_browse,
+)
+
+forum_plugin = None
+
+
+def _dispatch_inner(data):
+    method = data.get("method", "GET")
+    path = data.get("path", "/")
+    query = data.get("query", {})
+    body = data.get("body", {})
+    gateway_host = data.get("gateway_host", "")
+    scheme = data.get("scheme", "http")
+
+    def extract_id(prefix):
+        try:
+            return int(path[len(prefix):])
+        except (ValueError, IndexError):
+            return None
+
+    if method == "GET":
+        if path == "/":
+            return handle_search(query)
+        elif path == "/add":
+            prefill_url = query.get("url", [""])[0].strip()
+            return handle_add_form(prefill_url=prefill_url)
+        elif path == "/pages":
+            return handle_pages(query)
+        elif path.startswith("/edit/"):
+            pid = extract_id("/edit/")
+            return handle_edit_form(pid, page=query.get("p", [""])[0]) if pid is not None else _error(400)
+        elif path.startswith("/delete/"):
+            pid = extract_id("/delete/")
+            return handle_delete_confirm(pid) if pid is not None else _error(400)
+        elif path == "/bookmark":
+            return handle_bookmark(query)
+        elif path == "/style":
+            return handle_style_form(gateway_host=gateway_host, scheme=scheme)
+        elif path == "/share/preview":
+            return handle_share_preview()
+        elif path == "/about":
+            return handle_about()
+        elif path == "/export":
+            return handle_export(query)
+        elif path == "/import":
+            return handle_import_form()
+        elif path == "/tags":
+            return handle_tags()
+        elif path.startswith("/tags/"):
+            tag_name = unquote(path[len("/tags/"):])
+            return handle_tag_browse(tag_name, query) if tag_name else _error(400)
+        elif path.startswith("/rns/"):
+            # /rns/<hash>/<subpath>
+            parts = path[len("/rns/"):].split("/", 1)
+            dest_hash = parts[0]
+            if not dest_hash:
+                return _error(404)
+            return handle_rns_browse(path, dest_hash)
+        elif path == "/reindex":
+            return handle_reindex_form()
+        elif path == "/api/sites":
+            return handle_api_sites(query)
+        elif path == "/subscriptions":
+            return handle_subscriptions()
+        elif path == "/subscriptions/add":
+            return handle_add_form(action_type="subscribe")
+        elif path.startswith("/subscriptions/browse/"):
+            sid = extract_id("/subscriptions/browse/")
+            return handle_subscription_browse(sid) if sid is not None else _error(400)
+        elif path.startswith("/forum"):
+            if forum_plugin and forum_plugin.is_enabled():
+                return forum_plugin.handle(method, path, query, {}, data.get("cookies", {}))
+            return _error(404)
+    elif method == "POST":
+        if path.startswith("/forum"):
+            if forum_plugin and forum_plugin.is_enabled():
+                return forum_plugin.handle(method, path, query, body, data.get("cookies", {}))
+            return _error(404)
+        if not _check_csrf(body):
+            return _respond("<h1>403 Forbidden</h1><p>Invalid or missing CSRF token.</p>", status=403)
+        if path == "/add":
+            return handle_add_submit(body)
+        elif path == "/pages/bulk":
+            return handle_bulk_action(body)
+        elif path == "/add/manual":
+            return handle_add_manual_submit(body)
+        elif path.startswith("/edit/"):
+            pid = extract_id("/edit/")
+            return handle_edit_submit(pid, body) if pid is not None else _error(400)
+        elif path.startswith("/delete/"):
+            pid = extract_id("/delete/")
+            return handle_delete(pid) if pid is not None else _error(400)
+        elif path == "/style":
+            return handle_style_submit(body, gateway_host=gateway_host, scheme=scheme)
+        elif path == "/style/template":
+            return handle_style_template_submit(body, gateway_host=gateway_host, scheme=scheme)
+        elif path == "/style/field":
+            return handle_field_save(body)
+        elif path == "/style/reset":
+            set_setting("custom_template", "")
+            _set_flash("Template reset to default.")
+            return _redirect("/style")
+        elif path == "/style/vacuum":
+            from tinyweb.db import vacuum_db
+            vacuum_db()
+            _set_flash("Database vacuumed.")
+            return _redirect("/style")
+        elif path == "/import":
+            return handle_import_submit(body)
+        elif path == "/reindex":
+            return handle_reindex_submit(body)
+        elif path == "/subscriptions/add":
+            return handle_subscription_add(body)
+        elif path == "/subscriptions/pick":
+            return handle_subscription_pick(body)
+        elif path.startswith("/subscriptions/sync/"):
+            sid = extract_id("/subscriptions/sync/")
+            return handle_subscription_sync(sid) if sid is not None else _error(400)
+        elif path.startswith("/subscriptions/autosync/"):
+            sid = extract_id("/subscriptions/autosync/")
+            return handle_subscription_autosync(sid) if sid is not None else _error(400)
+        elif path.startswith("/subscriptions/delete/"):
+            sid = extract_id("/subscriptions/delete/")
+            return handle_subscription_delete(sid) if sid is not None else _error(400)
+        elif path == "/subscriptions/syncall":
+            return handle_subscription_syncall()
+        elif path == "/rns/delete":
+            return handle_rns_delete_hash(body)
+
+    return _error(404)
+
+
+def dispatch_request(data):
+    path = data.get("path", "/")
+    cookies = data.get("cookies", {})
+
+    if path.startswith("/forum") and forum_plugin and forum_plugin.is_enabled():
+        resp = _dispatch_inner(data)
+        resp.setdefault("headers", {})
+        resp["headers"]["X-Frame-Options"] = "DENY"
+        resp["headers"]["X-Content-Type-Options"] = "nosniff"
+        if resp.get("content_type", "").startswith("text/html"):
+            resp["body"] = wrap_page(resp.get("body", ""))
+            resp["headers"]["Content-Security-Policy"] = (
+                "default-src 'self'; "
+                "script-src 'self' 'unsafe-inline'; "
+                "style-src 'self' 'unsafe-inline'; "
+                "img-src * data:; "
+                "frame-ancestors 'none'; "
+                "form-action 'self'; "
+                "base-uri 'self'"
+            )
+        return resp
+
+    csrf_token = cookies.get("_csrf", "")
+    if not csrf_token:
+        csrf_token = secrets.token_hex(32)
+    _request_local.csrf_token = csrf_token
+
+    resp = _dispatch_inner(data)
+
+    resp.setdefault("headers", {})
+    resp["headers"]["Set-Cookie"] = f"_csrf={csrf_token}; SameSite=Strict; HttpOnly; Path=/"
+    resp["headers"]["X-Frame-Options"] = "DENY"
+    resp["headers"]["X-Content-Type-Options"] = "nosniff"
+    if resp.get("content_type", "").startswith("text/html"):
+        resp["headers"]["Content-Security-Policy"] = (
+            "default-src 'self'; "
+            "script-src 'self' 'unsafe-inline'; "
+            "style-src 'self' 'unsafe-inline'; "
+            "img-src * data:; "
+            "frame-ancestors 'none'; "
+            "form-action 'self'; "
+            "base-uri 'self'"
+        )
+    return resp

src/tinyweb/handlers/_helpers.py

@@ -0,0 +1,167 @@
+import json
+import re
+import secrets
+import threading
+
+from tinyweb.db import get_db, return_db, get_setting, set_setting
+from tinyweb.templates import wrap_page
+
+
+_request_local = threading.local()
+
+
+def _get_csrf_token():
+    return getattr(_request_local, 'csrf_token', '')
+
+
+def _csrf_field():
+    return f'<input type="hidden" name="_csrf" value="{_get_csrf_token()}">'
+
+
+def _check_csrf(body):
+    token = body.get("_csrf", [""])[0]
+    expected = _get_csrf_token()
+    if not expected or not token:
+        return False
+    return secrets.compare_digest(token, expected)
+
+
+_STOPWORDS = frozenset({
+    "a", "an", "the", "and", "or", "but", "is", "are", "was", "were",
+    "in", "on", "at", "to", "for", "of", "with", "by", "from", "as",
+    "into", "about", "how", "what", "which", "who", "where", "when",
+    "do", "does", "did", "be", "been", "being", "have", "has", "had",
+    "it", "its", "this", "that", "not", "no", "so", "if", "can", "will",
+    "my", "your", "i", "me", "we", "you", "he", "she", "they",
+})
+
+
+def _sanitize_fts_query(query):
+    words = query.split()
+    if not words:
+        return '""'
+    tokens = []
+    last_idx = len(words) - 1
+    for i, w in enumerate(words):
+        cleaned = re.sub(r'["\'\(\)\*\+\-\^~:]', '', w).strip()
+        if not cleaned:
+            continue
+        if cleaned.lower() in _STOPWORDS:
+            continue
+        if cleaned.upper() in ("AND", "OR", "NOT", "NEAR"):
+            continue
+        if i == last_idx:
+            tokens.append(f"{cleaned}*")
+        else:
+            tokens.append(f'"{cleaned}"')
+    return " ".join(tokens) if tokens else '""'
+
+
+def _get_bookmark_token():
+    token = get_setting("bookmark_token")
+    if not token:
+        token = secrets.token_hex(16)
+        set_setting("bookmark_token", token)
+    return token
+
+
+def _respond(body_html, status=200, use_default=False, head_html=""):
+    return {
+        "status": status,
+        "content_type": "text/html; charset=utf-8",
+        "body": wrap_page(body_html, use_default=use_default, head_html=head_html),
+        "headers": {},
+    }
+
+
+def _redirect(location):
+    if not location.startswith("/") or location.startswith("//"):
+        location = "/"
+    return {
+        "status": 302,
+        "content_type": "text/html; charset=utf-8",
+        "body": "",
+        "headers": {"Location": location},
+    }
+
+
+def _json_response(data, status=200, headers=None):
+    return {
+        "status": status,
+        "content_type": "application/json",
+        "body": json.dumps(data, indent=2),
+        "headers": headers or {},
+    }
+
+
+def _text_response(text, status=200, headers=None):
+    return {
+        "status": status,
+        "content_type": "text/plain",
+        "body": text,
+        "headers": headers or {},
+    }
+
+
+def _error(status):
+    return _respond(f"<h1>{status}</h1>", status)
+
+
+PER_PAGE = 10
+BROWSE_PER_PAGE = 50
+
+
+def _paginate(query, key="p"):
+    try:
+        page = int(query.get(key, ["1"])[0])
+    except (ValueError, IndexError):
+        page = 1
+    return max(1, page)
+
+
+def _page_nav(page, total, base_url, per_page=None):
+    per_page = per_page or PER_PAGE
+    if total <= per_page:
+        return ""
+    total_pages = (total + per_page - 1) // per_page
+    sep = "&" if "?" in base_url else "?"
+    parts = []
+    if page > 1:
+        parts.append(f'<a href="{base_url}{sep}p={page - 1}">&laquo; prev</a>')
+    parts.append(f"page {page} of {total_pages}")
+    if page < total_pages:
+        parts.append(f'<a href="{base_url}{sep}p={page + 1}">next &raquo;</a>')
+    return f'<p class="pagination">{" | ".join(parts)}</p>'
+
+
+def _get_page_tags(page_id, db=None):
+    close = False
+    if db is None:
+        db = get_db()
+        close = True
+    rows = db.execute(
+        "SELECT t.name FROM tags t JOIN page_tags pt ON t.id = pt.tag_id "
+        "WHERE pt.page_id = ? ORDER BY t.name", (page_id,)
+    ).fetchall()
+    if close:
+        return_db(db)
+    return [r["name"] for r in rows]
+
+
+def _set_page_tags(page_id, tag_string, db=None):
+    close = False
+    if db is None:
+        db = get_db()
+        close = True
+    db.execute("DELETE FROM page_tags WHERE page_id = ?", (page_id,))
+    for name in (t.strip().lower() for t in tag_string.split(",") if t.strip()):
+        db.execute("INSERT OR IGNORE INTO tags (name) VALUES (?)", (name,))
+        tag_id = db.execute("SELECT id FROM tags WHERE name = ?", (name,)).fetchone()["id"]
+        db.execute("INSERT OR IGNORE INTO page_tags (page_id, tag_id) VALUES (?, ?)", (page_id, tag_id))
+    if close:
+        db.commit()
+        return_db(db)
+
+
+def _cleanup_orphaned_tags(db):
+    db.execute("DELETE FROM tags WHERE id NOT IN (SELECT DISTINCT tag_id FROM page_tags)")

src/tinyweb/handlers/customize.py

@@ -0,0 +1,367 @@
+from tinyweb.db import get_db, return_db, get_setting, set_setting, get_site_name
+import tinyweb.templates as templates_mod
+from tinyweb.templates import esc, DEFAULT_TEMPLATE
+from ._helpers import _respond, _redirect, _json_response, _csrf_field, _get_bookmark_token, _request_local
+from .subscriptions import _count_shared_pages
+
+_flash = {}
+
+
+def _set_flash(msg):
+    _flash[_request_local.csrf_token] = msg
+
+
+def _get_flash():
+    return _flash.pop(_request_local.csrf_token, "")
+
+
+def handle_style_form(msg="", gateway_host="", scheme="http"):
+    template = get_setting("custom_template") or DEFAULT_TEMPLATE
+    name = get_site_name()
+    sharing = get_setting("sharing_enabled", "0")
+    checked = " checked" if sharing == "1" else ""
+    sharing_mode = get_setting("sharing_mode", "exclude_private")
+    forum = get_setting("forum_enabled", "0")
+    forum_checked = " checked" if forum == "1" else ""
+    exclude_checked = " checked" if sharing_mode != "require_public" else ""
+    require_checked = " checked" if sharing_mode == "require_public" else ""
+    shared_count = _count_shared_pages()
+    semantic = get_setting("semantic_search", "0")
+    semantic_checked = " checked" if semantic == "1" else ""
+    reranker = get_setting("use_reranker", "0")
+    reranker_checked = " checked" if reranker == "1" else ""
+    disabled = "" if semantic == "1" else " disabled"
+    tcp_enabled = get_setting("tcp_enabled", "1")
+    tcp_checked = " checked" if tcp_enabled == "1" else ""
+    tcp_disabled = "" if tcp_enabled == "1" else " disabled"
+    transport_host = get_setting("transport_host", "rnode.bre.land")
+    transport_port = get_setting("transport_port", "4242")
+    compress = get_setting("compress_embeddings", "0")
+    compress_checked = " checked" if compress == "1" else ""
+    lora_enabled = get_setting("lora_enabled", "0")
+    lora_checked = " checked" if lora_enabled == "1" else ""
+    lora_disabled = "" if lora_enabled == "1" else " disabled"
+    lora_port = get_setting("lora_port", "")
+    lora_frequency = get_setting("lora_frequency", "867200000")
+    lora_bandwidth = get_setting("lora_bandwidth", "125000")
+    lora_txpower = get_setting("lora_txpower", "7")
+    lora_sf = get_setting("lora_sf", "8")
+    lora_cr = get_setting("lora_cr", "5")
+    csrf = _csrf_field()
+    from tinyweb.handlers import forum_plugin as _fp
+    if _fp is not None:
+        forum_body = (
+            f"<section id=\"forum\">"
+            f'<form method="post" action="/style">'
+            f"{csrf}"
+            f'<input type="hidden" name="_action" value="forum">'
+            f"<h2>forum</h2>"
+            f'<label><input type="checkbox" name="forum_enabled" value="1"{forum_checked}>'
+            f" enable forum (shared URL discussion board)</label><br>"
+            f"<small>Share URLs and discuss them with other TinyWeb instances.</small><br><br>"
+            f'<input type="submit" value="save forum">'
+            f"</form>"
+            f"</section>"
+        )
+        forum_nav = ' · <a href="#forum">forum</a>'
+    else:
+        forum_body = ""
+        forum_nav = ""
+
+    msg = _get_flash() or msg
+    msg_html = ""
+    if msg:
+        msg_html = '<p><em>{msg}</em></p>'.format(msg=esc(msg))
+
+    return _respond(
+        f"<h1>customize</h1>"
+        f"{msg_html}"
+        f'<nav>'
+        f'<a href="#site-name">site name</a>'
+        f' · <a href="#sharing">sharing</a>'
+        f'{forum_nav}'
+        f' · <a href="#search">search</a>'
+        f' · <a href="#mesh">mesh</a>'
+        f' · <a href="#template">template</a>'
+        f' · <a href="#tools">tools</a>'
+        f'</nav>'
+        f"<hr>"
+        f"<section id=\"site-name\">"
+        f'<form method="post" action="/style">'
+        f"{csrf}"
+        f'<input type="hidden" name="_action" value="name">'
+        f"<h2>site name</h2>"
+        f'<input name="site_name" value="{esc(name)}" placeholder="tinyweb" size="30">'
+        f' <input type="submit" value="save name">'
+        f"</form>"
+        f"</section>"
+        f"<hr>"
+        f"<section id=\"sharing\">"
+        f'<form method="post" action="/style">'
+        f"{csrf}"
+        f'<input type="hidden" name="_action" value="sharing">'
+        f"<h2>sharing</h2>"
+        f'<label><input type="checkbox" name="sharing_enabled" value="1"{checked}>'
+        f" share your site list publicly at /api/sites</label><br>"
+        f'<div style="margin-top:0.6rem">'
+        f"<small>What to share:</small><br>"
+        f'<label><input type="radio" name="sharing_mode" value="exclude_private"{exclude_checked}>'
+        f' share all pages except those tagged <code>private</code></label><br>'
+        f'<label><input type="radio" name="sharing_mode" value="require_public"{require_checked}>'
+        f' share only pages tagged <code>public</code></label><br>'
+        f"</div>"
+        f'<p style="margin-top:0.6rem">'
+        f'Currently sharing <b>{shared_count}</b> page(s). '
+        f'<a href="/share/preview">preview</a>'
+        f"</p>"
+        f'<input type="submit" value="save sharing">'
+        f"</form>"
+        f"</section>"
+        f"<hr>"
+        f"{forum_body}"
+        f"<hr>"
+        f"<section id=\"search\">"
+        f'<form method="post" action="/style">'
+        f"{csrf}"
+        f'<input type="hidden" name="_action" value="search">'
+        f"<h2>search</h2>"
+        f'<label><input type="checkbox" name="semantic_search" value="1"{semantic_checked}>'
+        f" semantic search</label><br><br>"
+        f'<label><input type="checkbox" name="use_reranker" value="1"{reranker_checked}{disabled}>'
+        f" cross-encoder reranking</label><br><br>"
+        f'<label><input type="checkbox" name="compress_embeddings" value="1"{compress_checked}{disabled}>'
+        f" compress embeddings</label><br><br>"
+        f'<a href="/reindex">manage semantic index</a>'
+        f'<br><input type="submit" value="save search">'
+        f"</form>"
+        f"</section>"
+        f"<hr>"
+        f"<section id=\"mesh\">"
+        f'<form method="post" action="/style">'
+        f"{csrf}"
+        f'<input type="hidden" name="_action" value="mesh">'
+        f"<h2>mesh network</h2>"
+        f"<p>Choose how to connect to the mesh.</p>"
+        f"<h3>internet</h3>"
+        f'<label><input type="checkbox" name="tcp_enabled" value="1"{tcp_checked}>'
+        f" connect via internet transport node</label><br>"
+        f"<small>Default: rnode.bre.land:4242</small><br>"
+        f'<input name="transport_host" value="{esc(transport_host)}" size="30"{tcp_disabled}>'
+        f' <input name="transport_port" value="{esc(transport_port)}" size="6"{tcp_disabled}><br>'
+        f"<h3>LoRa</h3>"
+        f'<label><input type="checkbox" name="lora_enabled" value="1"{lora_checked}>'
+        f" connect via LoRa radio</label><br>"
+        f'<label>Serial port: <input id="lora-port" name="lora_port" value="{esc(lora_port)}" '
+        f'placeholder="/dev/ttyUSB0" size="20"{lora_disabled}></label><br>'
+        f"<details><summary>advanced radio settings</summary>"
+        f'<label>Frequency (Hz): <input name="lora_frequency" value="{esc(lora_frequency)}" size="12"{lora_disabled}></label><br>'
+        f"<small>ISM band frequency. Default: 867200000 (868 MHz EU). US: 915000000.</small><br><br>"
+        f'<label>Bandwidth (Hz): <input name="lora_bandwidth" value="{esc(lora_bandwidth)}" size="8"{lora_disabled}></label><br>'
+        f"<small>Default: 125000</small><br><br>"
+        f'<label>TX Power (dBm): <input name="lora_txpower" value="{esc(lora_txpower)}" size="4"{lora_disabled}></label><br>'
+        f"<small>0-17 typical. Check local regulations.</small><br><br>"
+        f'<label>Spreading Factor: <input name="lora_sf" value="{esc(lora_sf)}" size="4"{lora_disabled}></label><br>'
+        f"<small>5-12. Higher = longer range, slower speed.</small><br><br>"
+        f'<label>Coding Rate: <input name="lora_cr" value="{esc(lora_cr)}" size="4"{lora_disabled}></label><br>'
+        f"<small>5-8. Higher = more error correction.</small>"
+        f"</details>"
+        f'<br><input type="submit" value="save mesh">'
+        f"</form>"
+        f"</section>"
+        f"<hr>"
+        f"<section id=\"template\">"
+        f"<h2>template</h2>"
+        f"<p>Edit the full page template.</p>"
+        f'<form method="post" action="/style/template">'
+        f"{csrf}"
+        f'<textarea name="template" rows="12" cols="60" style="width:100%">{esc(template)}</textarea><br><br>'
+        f'<input type="submit" value="save template">'
+        f"</form>"
+        f"</section>"
+        f"<hr>"
+        f"<section id=\"tools\">"
+        f"<h2>tools</h2>"
+        f"<h3>bookmarklet</h3>"
+        f"<p>Drag this link to your bookmarks bar.</p>"
+        f'<p><a href="javascript:void(fetch(\'{esc(scheme)}://{esc(gateway_host or "localhost:8080")}/bookmark?url=\'+encodeURIComponent(location.href)+\'&token={_get_bookmark_token()}\').then(r=>r.text()).then(t=>alert(t)).catch(()=>alert(\'tinyweb not running\')))">+ save to {esc(name)}</a></p>'
+        f"<h3>reset template</h3>"
+        f'<form method="post" action="/style/reset">'
+        f"{csrf}"
+        f'<input type="submit" value="reset to default">'
+        f"</form>"
+        f"<h3>vacuum database</h3>"
+        f'<form method="post" action="/style/vacuum">'
+        f"{csrf}"
+        f'<input type="submit" value="vacuum">'
+        f"</form>"
+        f"</section>"
+        f'<a href="/">back</a>',
+        use_default=True,
+    )
+
+
+def handle_style_submit(body, gateway_host="", scheme="http"):
+    action = body.get("_action", [""])[0]
+
+    if action == "name":
+        name = body.get("site_name", ["tinyweb"])[0].strip()
+        set_setting("site_name", name or "tinyweb")
+        _set_flash("Saved.")
+        return _redirect("/style")
+
+    if action == "sharing":
+        sharing = "1" if body.get("sharing_enabled") else "0"
+        sharing_mode = body.get("sharing_mode", ["exclude_private"])[0]
+        if sharing_mode not in ("exclude_private", "require_public"):
+            sharing_mode = "exclude_private"
+        set_setting("sharing_mode", sharing_mode)
+        set_setting("sharing_enabled", sharing)
+        _set_flash("Saved.")
+        return _redirect("/style")
+
+    if action == "forum":
+        forum_enabled = "1" if body.get("forum_enabled") else "0"
+        current_forum = get_setting("forum_enabled", "0")
+        if forum_enabled != current_forum:
+            from tinyweb.handlers import forum_plugin
+            if forum_enabled == "1" and forum_plugin is None:
+                _set_flash("Forum plugin not installed. Run: pip install tinyweb-forum")
+                return _redirect("/style")
+            if forum_enabled == "1":
+                forum_plugin.enable()
+                try:
+                    forum_plugin.fdb.set_setting("forum_enabled", "1")
+                except Exception:
+                    pass
+            else:
+                forum_plugin.disable()
+                try:
+                    forum_plugin.fdb.set_setting("forum_enabled", "0")
+                except Exception:
+                    pass
+            set_setting("forum_enabled", forum_enabled)
+            templates_mod.FORUM_ENABLED = (forum_enabled == "1")
+        _set_flash("Saved.")
+        return _redirect("/style")
+
+    if action == "search":
+        semantic = "1" if body.get("semantic_search") else "0"
+        reranker = "1" if body.get("use_reranker") else "0"
+        compress = "1" if body.get("compress_embeddings") else "0"
+        set_setting("semantic_search", semantic)
+        set_setting("use_reranker", reranker)
+        set_setting("compress_embeddings", compress)
+        _set_flash("Saved.")
+        return _redirect("/style")
+
+    if action == "mesh":
+        tcp_enabled = "1" if body.get("tcp_enabled") else "0"
+        transport_host = body.get("transport_host", [""])[0].strip()
+        transport_port = body.get("transport_port", [""])[0].strip()
+        set_setting("tcp_enabled", tcp_enabled)
+        if transport_host:
+            set_setting("transport_host", transport_host)
+        if transport_port:
+            set_setting("transport_port", transport_port)
+        lora_enabled = "1" if body.get("lora_enabled") else "0"
+        set_setting("lora_enabled", lora_enabled)
+        set_setting("lora_port", body.get("lora_port", [""])[0].strip())
+        set_setting("lora_frequency", body.get("lora_frequency", ["867200000"])[0].strip())
+        set_setting("lora_bandwidth", body.get("lora_bandwidth", ["125000"])[0].strip())
+        set_setting("lora_txpower", body.get("lora_txpower", ["7"])[0].strip())
+        set_setting("lora_sf", body.get("lora_sf", ["8"])[0].strip())
+        set_setting("lora_cr", body.get("lora_cr", ["5"])[0].strip())
+        _set_flash("Saved.")
+        return _redirect("/style")
+
+
+def handle_style_template_submit(body, gateway_host="", scheme="http"):
+    template = body.get("template", [""])[0].replace("\r\n", "\n").replace("\r", "\n")
+    set_setting("custom_template", template if template.strip() != DEFAULT_TEMPLATE.strip() else "")
+    _set_flash("Template saved.")
+    return _redirect("/style")
+
+
+def handle_field_save(body):
+    key = body.get("key", [""])[0].strip()
+    value = body.get("value", [""])[0].strip()
+    if not key:
+        return _json_response({"status": "error", "message": "No key provided."}, 400)
+    if key == "forum_enabled":
+        from tinyweb.handlers import forum_plugin
+        if value == "1" and forum_plugin is None:
+            return _json_response({"status": "error", "message": "Forum plugin not installed."}, 400)
+        if value == "1":
+            forum_plugin.enable()
+            try:
+                forum_plugin.fdb.set_setting("forum_enabled", "1")
+            except Exception:
+                pass
+        else:
+            forum_plugin.disable()
+            try:
+                forum_plugin.fdb.set_setting("forum_enabled", "0")
+            except Exception:
+                pass
+        templates_mod.FORUM_ENABLED = (value == "1")
+    set_setting(key, value)
+    return _json_response({"status": "ok", "message": ""})
+
+
+def handle_about():
+    name = get_site_name()
+    dest_hash = get_setting("dest_hash")
+    sharing = get_setting("sharing_enabled", "0") == "1"
+    db = get_db()
+    try:
+        page_count = db.execute("SELECT count(*) FROM pages").fetchone()[0]
+        tag_count = db.execute("SELECT count(DISTINCT tag_id) FROM page_tags").fetchone()[0]
+        sub_count = db.execute("SELECT count(*) FROM subscriptions").fetchone()[0]
+    finally:
+        return_db(db)
+
+    sharing_html = (
+        '<p>This instance shares its index publicly. Subscribe to join the network.</p>'
+        if sharing else
+        '<p>This instance is private.</p>'
+    )
+
+    hash_html = ""
+    if dest_hash:
+        hash_html = (
+            f'<h2>subscribe</h2>'
+            f'<p>To subscribe to this instance, add this destination hash in your TinyWeb:</p>'
+            f'<pre>{esc(dest_hash)}</pre>'
+        )
+
+    return _respond(
+        f'<h1>{esc(name)}</h1>'
+        f'<p>A personal, decentralized search engine.</p>'
+        f'<p>You save pages you find. They are stored locally and shared over a mesh network '
+        f'so other people can find them too.</p>'
+        f'<p>Search results come from your index and the indexes of people you are connected to.</p>'
+        f'<ul>'
+        f'<li><b>{page_count}</b> page(s) indexed</li>'
+        f'<li><b>{tag_count}</b> tag(s)</li>'
+        f'<li><b>{sub_count}</b> subscription(s)</li>'
+        f'</ul>'
+        f'{sharing_html}'
+        f'{hash_html}'
+        f'<h2>your data</h2>'
+        f'<p>Everything is stored locally under <code>~/.tinyweb/</code>:</p>'
+        f'<ul>'
+        f'<li><code>tinyweb_identity</code> — your permanent mesh identity. '
+        f'If you lose this file, your destination hash changes and subscribers '
+        f'have to re-subscribe to the new one.</li>'
+        f'<li><code>index.db</code> — your full reading history: every page, '
+        f'note, tag, and synced remote page.</li>'
+        f'<li><code>models/</code> — the semantic search model if you enabled it '
+        f'(redownloadable, safe to delete).</li>'
+        f'</ul>'
+        f'<p><b>Back up <code>~/.tinyweb/</code> periodically.</b> '
+        f'Copying the whole directory to another device preserves your identity and index together. '
+        f'The <a href="/export">export</a> page gives you a JSON dump of pages only — '
+        f'it does not preserve your identity or subscription state, so it is a migration aid, '
+        f'not a substitute for a full backup.</p>'
+        f'<p><a href="/">search</a> | <a href="/pages">browse</a> | <a href="/tags">tags</a></p>'
+    )

src/tinyweb/handlers/data.py

@@ -0,0 +1,125 @@
+import json
+import threading
+
+from tinyweb.db import get_db, return_db, get_setting, set_setting, index_url
+from tinyweb.templates import esc
+from ._helpers import _respond, _json_response, _redirect, _csrf_field
+
+MAX_EXPORT = 10000
+_reindex_thread = None
+
+
+def handle_export(query=None):
+    try:
+        batch = int((query or {}).get("batch", ["0"])[0])
+    except (TypeError, ValueError):
+        batch = 0
+    db = get_db()
+    try:
+        rows = db.execute(
+            "SELECT url, title, note FROM pages ORDER BY id LIMIT ? OFFSET ?",
+            (MAX_EXPORT, batch * MAX_EXPORT),
+        ).fetchall()
+    finally:
+        return_db(db)
+    data = [{"url": r["url"], "title": r["title"], "note": r["note"]} for r in rows]
+    return _json_response(data, headers={"Content-Disposition": "attachment; filename=tinyweb-export.json"})
+
+
+def handle_import_form(msg=""):
+    return _respond(
+        f"<h1>import</h1>"
+        f"<p>Paste the contents of a tinyweb export file (JSON).</p>"
+        f'<form method="post" action="/import">'
+        f'{_csrf_field()}'
+        f'<textarea name="data" rows="12" cols="60" placeholder=\'[{{"url": "...", "note": "..."}}]\'></textarea><br><br>'
+        f'<button type="submit">import</button>'
+        f"</form>"
+        f"<p>{msg}</p>"
+        f'<a href="/pages">back</a>'
+    )
+
+
+def handle_import_submit(body):
+    raw = body.get("data", [""])[0].strip()
+    if not raw:
+        return handle_import_form("Paste JSON data.")
+    try:
+        data = json.loads(raw)
+    except json.JSONDecodeError:
+        return handle_import_form("Invalid JSON.")
+    if not isinstance(data, list):
+        return handle_import_form("Expected a JSON array.")
+
+    MAX_IMPORT = 100
+    if len(data) > MAX_IMPORT:
+        return handle_import_form(f"Too many entries. Maximum is {MAX_IMPORT}.")
+
+    imported = 0
+    errors = 0
+    for entry in data:
+        url = entry.get("url", "").strip()
+        note = entry.get("note", "").strip()
+        if not url:
+            continue
+        try:
+            index_url(url, note)
+            imported += 1
+        except Exception:
+            errors += 1
+
+    return handle_import_form(f"Imported {imported} page(s). {errors} error(s).")
+
+
+def handle_reindex_form():
+    if get_setting("semantic_search", "0") != "1":
+        return _respond(
+            f"<h2>semantic search index</h2>"
+            f"<p>Semantic search is disabled. Enable it in <a href=\"/style\">settings</a> to use embeddings.</p>"
+            f'<p><a href="/">back to search</a></p>'
+        )
+    db = get_db()
+    try:
+        total_pages = db.execute("SELECT count(*) FROM pages").fetchone()[0]
+        pages_with_chunks = db.execute(
+            "SELECT count(DISTINCT page_id) FROM chunks WHERE page_id IS NOT NULL"
+        ).fetchone()[0]
+    finally:
+        return_db(db)
+    progress = get_setting("reindex_progress", "")
+    status_html = ""
+    if progress:
+        status_html = f'<p class="meta">Reindex in progress: {esc(progress)}</p>'
+    elif _reindex_thread and _reindex_thread.is_alive():
+        status_html = '<p class="meta">Reindex running...</p>'
+    return _respond(
+        f"<h2>semantic search index</h2>"
+        f"<p>{pages_with_chunks} of {total_pages} pages have embeddings.</p>"
+        f'{status_html}'
+        f'<form method="post" action="/reindex">'
+        f'{_csrf_field()}'
+        f'<button type="submit">reindex all pages</button>'
+        f'</form>'
+        f'<p><a href="/">back to search</a></p>'
+    )
+
+
+def handle_reindex_submit(body):
+    global _reindex_thread
+    if _reindex_thread and _reindex_thread.is_alive():
+        return handle_reindex_form()
+
+    def _run():
+        try:
+            from tinyweb.embeddings import reindex_all
+            def progress(current, total):
+                set_setting("reindex_progress", f"{current}/{total}")
+            reindex_all(progress_callback=progress)
+        except Exception:
+            pass
+        finally:
+            set_setting("reindex_progress", "")
+
+    _reindex_thread = threading.Thread(target=_run, daemon=True)
+    _reindex_thread.start()
+    return _redirect("/reindex")

src/tinyweb/handlers/pages.py

@@ -0,0 +1,408 @@
+from pathlib import Path
+import json
+import secrets
+from urllib.parse import unquote
+
+from tinyweb.db import get_db, return_db, get_setting, set_setting, get_site_name, index_url, clean_url
+from tinyweb.templates import esc
+from ._helpers import (
+    _csrf_field, _respond, _redirect, _error,
+    _paginate, _page_nav, _get_page_tags, _set_page_tags, _cleanup_orphaned_tags,
+    _get_bookmark_token, _text_response,
+    BROWSE_PER_PAGE,
+)
+
+
+def handle_add_form(msg="", action_type="index", prefill_url=""):
+    if action_type == "subscribe":
+        return _respond(
+            f"<h1>subscribe</h1>"
+            f"<p>Subscribe to a friend's TinyWeb instance to sync their index</p>"
+            f'<form method="post" action="/subscriptions/add">'
+            f'{_csrf_field()}'
+            f'<input name="dest_hash" placeholder="destination hash (32 hex chars)" size="50"><br><br>'
+            f'<button type="submit">subscribe</button>'
+            f"</form>"
+            f"<p><small>or <a href=\"/add\">add a single site</a></small></p>"
+            f"<p>{msg}</p>"
+            f'<a href="/">back</a>'
+        )
+    url_value = f'value="{esc(prefill_url)}" ' if prefill_url else ""
+    return _respond(
+        f"<h1>add site</h1>"
+        f"<p>Add a site to your index — URL or RNS destination hash</p>"
+        f'<form method="post" action="/add">'
+        f'{_csrf_field()}'
+        f'<input name="url" placeholder="https://example.com or 32-char RNS hash" size="50" {url_value}><br><br>'
+        f'<input name="note" placeholder="why are you saving this? (optional)" size="50"><br><br>'
+        f'<input name="tags" placeholder="tags (comma-separated, e.g. solarpunk, mesh)" size="50"><br>'
+        f'<small>tag: private to exclude from sharing</small><br><br>'
+        f'<button type="submit">index</button>'
+        f"</form>"
+        f"<p>{msg}</p>"
+        f'<a href="/">back</a>'
+    )
+
+
+def handle_add_submit(body):
+    raw = body.get("url", [""])[0].strip().replace("<", "").replace(">", "")
+    note = body.get("note", [""])[0].strip()
+    tags = body.get("tags", [""])[0].strip()
+
+    if not raw:
+        return handle_add_form("URL or RNS hash is required.")
+
+    is_rns = (
+        len(raw) == 32
+        and all(c in "0123456789abcdefABCDEF" for c in raw)
+    )
+    if raw.startswith("rns:") or raw.startswith("RNS:"):
+        raw = raw[4:]
+        is_rns = (
+            len(raw) == 32
+            and all(c in "0123456789abcdefABCDEF" for c in raw)
+        )
+
+    if is_rns:
+        from .rns import handle_rns_add_hash
+        errs = handle_rns_add_hash(raw)
+        if errs:
+            return handle_add_form(f"Hash saved but indexing failed: {'; '.join(errs)}")
+        return _redirect("/")
+
+    url = clean_url(raw)
+    if not url.startswith(("http://", "https://")):
+        return handle_add_form("Enter a URL (http:// or https://) or a 32-char RNS destination hash.")
+
+    try:
+        title = index_url(url, note)
+        if tags:
+            db = get_db()
+            try:
+                row = db.execute("SELECT id FROM pages WHERE url = ?", (url,)).fetchone()
+                if row:
+                    _set_page_tags(row["id"], tags, db)
+                    db.commit()
+            finally:
+                return_db(db)
+        return handle_add_form(f'Indexed: {esc(url)}')
+    except ValueError as e:
+        return handle_add_form(f"Error: {esc(str(e))}")
+    except Exception as e:
+        error_msg = str(e).lower()
+        if any(x in error_msg for x in ("block", "cloudflare", "403", "429", "ssl", "handshake", "max retries", "timeout", "connection")):
+            return _respond(
+                f"<h1>add url (manual entry)</h1>"
+                f"<p><strong>{esc(url)}</strong> blocks automated access. "
+                f"You can still save it manually:</p>"
+                f'<form method="post" action="/add/manual">'
+                f'{_csrf_field()}'
+                f'<input type="hidden" name="url" value="{esc(url)}">'
+                f'<input type="hidden" name="note" value="{esc(note)}">'
+                f'<input type="hidden" name="tags" value="{esc(tags)}">'
+                f'<label>Title:</label><br>'
+                f'<input name="manual_title" size="50" placeholder="page title" required><br><br>'
+                f'<label>Description:</label><br>'
+                f'<textarea name="manual_description" rows="4" cols="50" placeholder="what is this site about? (optional)"></textarea><br><br>'
+                f'<button type="submit">save manually</button>'
+                f"</form>"
+                f'<a href="/">back</a>'
+            )
+        return handle_add_form(f"Error: could not fetch or index that URL. {esc(str(e)[:100])}")
+
+
+def handle_add_manual_submit(body):
+    url = clean_url(body.get("url", [""])[0].strip())
+    note = body.get("note", [""])[0].strip()
+    tags = body.get("tags", [""])[0].strip()
+    manual_title = body.get("manual_title", [""])[0].strip()
+    manual_desc = body.get("manual_description", [""])[0].strip()
+
+    if not url:
+        return handle_add_form("URL is required.")
+
+    if not manual_title:
+        return handle_add_form("Title is required for manual entry.")
+
+    db = get_db()
+    try:
+        now = __import__("datetime").datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
+
+        db.execute(
+            "INSERT INTO pages (url, title, body, note, last_modified, summary) VALUES (?, ?, ?, ?, ?, ?) "
+            "ON CONFLICT(url) DO UPDATE SET title=excluded.title, body=excluded.body, "
+            "note=excluded.note, last_modified=excluded.last_modified, summary=excluded.summary",
+            (url, manual_title, manual_desc, note, now, manual_desc[:200]),
+        )
+
+        page_id = db.execute("SELECT id FROM pages WHERE url = ?", (url,)).fetchone()[0]
+
+        if tags:
+            _set_page_tags(page_id, tags, db)
+
+        db.commit()
+
+        if get_setting("semantic_search", "0") == "1":
+            try:
+                from tinyweb.embeddings import store_embeddings
+                store_embeddings(page_id, manual_title, manual_desc, db)
+                db.commit()
+            except Exception as e:
+                print(f"Error generating embeddings: {e}")
+
+        return handle_add_form(f'Added manually: <a href="{esc(url)}" rel="noreferrer noopener">{esc(manual_title)}</a>')
+    finally:
+        return_db(db)
+
+
+def handle_pages(query=None):
+    msg = query.get("msg", [""])[0] if query else ""
+    msg_html = f'<p class="success">{esc(msg)}</p>' if msg else ""
+    page = _paginate(query or {})
+    offset = (page - 1) * BROWSE_PER_PAGE
+    db = get_db()
+    try:
+        total = db.execute("SELECT count(*) FROM pages").fetchone()[0]
+        rows = db.execute(
+            "SELECT id, url, title, note FROM pages ORDER BY id DESC LIMIT ? OFFSET ?",
+            (BROWSE_PER_PAGE, offset),
+        ).fetchall()
+        items = ""
+        for r in rows:
+            note_html = f' — <em>{esc(r["note"])}</em>' if r["note"] else ""
+            tags = _get_page_tags(r["id"], db)
+            tags_html = ""
+            if tags:
+                tag_links = " ".join(f'<a href="/tags/{esc(t)}">[{esc(t)}]</a>' for t in tags)
+                tags_html = f' {tag_links}'
+            url = r["url"]
+            if url.startswith("rns:"):
+                display_url = url
+                link_url = f"/rns/{esc(url[4:])}/"
+            else:
+                display_url = url
+                link_url = url
+            items += (
+                f'<li><label><input type="checkbox" name="ids" value="{r["id"]}"> '
+                f'{esc(r["title"])}</label>{note_html}{tags_html} '
+                f'<small>(<a href="{esc(link_url)}" rel="noreferrer noopener">{esc(display_url)}</a>)</small> '
+                f'<a href="/edit/{r["id"]}?p={page}">edit</a> '
+                f'<a href="/delete/{r["id"]}">remove</a></li>'
+            )
+    finally:
+        return_db(db)
+    return _respond(
+        f"<h1>indexed pages ({total})</h1>"
+        f"{msg_html}"
+        f'<form method="post" action="/pages/bulk">'
+        f'{_csrf_field()}'
+        f'<p><label><input type="checkbox" id="select-all"> select all</label></p>'
+        f"<ul>{items}</ul>"
+        f'{_page_nav(page, total, "/pages", BROWSE_PER_PAGE)}'
+        f'<details><summary>bulk actions</summary>'
+        f'<p><button type="submit" name="action" value="delete" id="bulk-delete">delete selected</button></p>'
+        f'<p><input name="bulk_tags" placeholder="tags (comma-separated)" size="40"> '
+        f'<select name="tag_mode"><option value="add">add tags</option><option value="replace">replace tags</option></select> '
+        f'<button type="submit" name="action" value="retag">retag selected</button></p>'
+        f'</details>'
+        f'</form>'
+        f'<script>'
+        f'document.getElementById("select-all").addEventListener("change",function(){{'
+        f'document.querySelectorAll("input[name=ids]").forEach(function(c){{c.checked=this.checked}}.bind(this))'
+        f'}});'
+        f'document.getElementById("bulk-delete").addEventListener("click",function(e){{'
+        f'var n=document.querySelectorAll("input[name=ids]:checked").length;'
+        f'if(!n){{e.preventDefault();return}}'
+        f'if(!confirm("Delete "+n+" selected page"+(n===1?"":"s")+"?"))e.preventDefault()'
+        f'}});'
+        f'</script>'
+        f'<p><a href="/export">export</a> | <a href="/import">import</a></p>'
+        f'<a href="/">back</a>'
+    )
+
+
+def _render_bulk_delete_confirm(page_ids):
+    db = get_db()
+    try:
+        placeholders = ",".join("?" * len(page_ids))
+        rows = db.execute(
+            f"SELECT id, url, title FROM pages WHERE id IN ({placeholders})",
+            page_ids,
+        ).fetchall()
+    finally:
+        return_db(db)
+    if not rows:
+        return _redirect("/pages")
+    items = "".join(
+        f'<li><b>{esc(r["title"] or r["url"])}</b><br>'
+        f'<small>{esc(r["url"])}</small></li>'
+        for r in rows
+    )
+    hidden_ids = "".join(
+        f'<input type="hidden" name="ids" value="{int(r["id"])}">' for r in rows
+    )
+    n = len(rows)
+    return _respond(
+        f"<h1>confirm delete</h1>"
+        f"<p>Remove the following {n} page{'' if n == 1 else 's'}?</p>"
+        f"<ul>{items}</ul>"
+        f'<form method="post" action="/pages/bulk">'
+        f'{_csrf_field()}'
+        f'{hidden_ids}'
+        f'<input type="hidden" name="action" value="delete">'
+        f'<input type="hidden" name="confirmed" value="1">'
+        f'<button type="submit">yes, delete {n} page{"" if n == 1 else "s"}</button>'
+        f"</form>"
+        f' <a href="/pages">cancel</a>'
+    )
+
+
+def handle_bulk_action(body):
+    ids = body.get("ids", [])
+    action = body.get("action", [""])[0]
+    if not ids:
+        return _redirect("/pages")
+    try:
+        page_ids = [int(i) for i in ids]
+    except ValueError:
+        return _error(400)
+    if action == "delete" and body.get("confirmed", [""])[0] != "1":
+        return _render_bulk_delete_confirm(page_ids)
+    db = get_db()
+    try:
+        if action == "delete":
+            for pid in page_ids:
+                db.execute("DELETE FROM page_tags WHERE page_id = ?", (pid,))
+                db.execute("DELETE FROM links WHERE page_id = ?", (pid,))
+                db.execute("DELETE FROM pages WHERE id = ?", (pid,))
+            _cleanup_orphaned_tags(db)
+            db.commit()
+        elif action == "retag":
+            bulk_tags = body.get("bulk_tags", [""])[0].strip()
+            tag_mode = body.get("tag_mode", ["add"])[0]
+            if bulk_tags:
+                for pid in page_ids:
+                    if tag_mode == "add":
+                        existing = _get_page_tags(pid, db)
+                        new_tags = [t.strip().lower() for t in bulk_tags.split(",") if t.strip()]
+                        merged = ", ".join(sorted(set(existing + new_tags)))
+                        _set_page_tags(pid, merged, db)
+                    else:
+                        _set_page_tags(pid, bulk_tags, db)
+                _cleanup_orphaned_tags(db)
+                db.commit()
+    finally:
+        return_db(db)
+    return _redirect("/pages")
+
+
+def handle_edit_form(page_id, msg="", page=""):
+    db = get_db()
+    try:
+        row = db.execute("SELECT id, url, title, body, note, summary FROM pages WHERE id = ?", (page_id,)).fetchone()
+        if not row:
+            return _error(404)
+        tags = ", ".join(_get_page_tags(page_id, db))
+    finally:
+        return_db(db)
+
+    back = "/pages" + ("?p=" + page if page else "")
+
+    return _respond(
+        f"<h1>edit page</h1>"
+        f"<p><b>{esc(row['title'])}</b><br>"
+        f"<small>{esc(row['url'])}</small></p>"
+        f'<form method="post" action="/edit/{row["id"]}">'
+        f'{_csrf_field()}'
+        f'<input type="hidden" name="_page" value="{esc(page)}">'
+        f'<label>Title:</label><br>'
+        f'<input name="title" value="{esc(row["title"])}" size="60"><br><br>'
+        f'<label>Summary (shown in search results):</label><br>'
+        f'<textarea name="summary" rows="3" cols="60">{esc(row["summary"] or "")}</textarea><br><br>'
+        f'<label>Note (why you saved this):</label><br>'
+        f'<input name="note" value="{esc(row["note"])}" size="50"><br><br>'
+        f'<label>Tags (comma-separated):</label><br>'
+        f'<input name="tags" value="{esc(tags)}" size="50"> '
+        f'<small>(tag: private to keep private)</small><br><br>'
+        f'<button type="submit">save</button>'
+        f"</form>"
+        f"<p>{msg}</p>"
+        f'<a href="{back}">back</a>'
+    )
+
+
+def handle_edit_submit(page_id, body):
+    title = body.get("title", [""])[0].strip()
+    summary = body.get("summary", [""])[0].strip()
+    note = body.get("note", [""])[0].strip()
+    tags = body.get("tags", [""])[0].strip()
+
+    db = get_db()
+    try:
+        db.execute(
+            "UPDATE pages SET title = ?, summary = ?, note = ? WHERE id = ?",
+            (title, summary, note, page_id)
+        )
+
+        _set_page_tags(page_id, tags, db)
+        _cleanup_orphaned_tags(db)
+
+        db.commit()
+
+    finally:
+        return_db(db)
+
+    page = body.get("_page", [""])[0].strip()
+    target = "/pages"
+    if page:
+        target += "?p=" + page
+    return _redirect(target)
+
+
+def handle_delete_confirm(page_id):
+    db = get_db()
+    try:
+        row = db.execute("SELECT id, url, title FROM pages WHERE id = ?", (page_id,)).fetchone()
+    finally:
+        return_db(db)
+    if not row:
+        return _error(404)
+    return _respond(
+        f"<h1>confirm delete</h1>"
+        f"<p>Remove <b>{esc(row['title'])}</b><br>"
+        f"<small>{esc(row['url'])}</small></p>"
+        f'<form method="post" action="/delete/{row["id"]}">'
+        f'{_csrf_field()}'
+        f'<button type="submit">yes, delete</button>'
+        f"</form>"
+        f' <a href="/pages">cancel</a>'
+    )
+
+
+def handle_delete(page_id):
+    db = get_db()
+    try:
+        db.execute("DELETE FROM page_tags WHERE page_id = ?", (page_id,))
+        db.execute("DELETE FROM links WHERE page_id = ?", (page_id,))
+        db.execute("DELETE FROM pages WHERE id = ?", (page_id,))
+        _cleanup_orphaned_tags(db)
+        db.commit()
+    finally:
+        return_db(db)
+    return _redirect("/pages")
+
+
+def handle_bookmark(query):
+    token = query.get("token", [""])[0]
+    expected = _get_bookmark_token()
+    if not token or not secrets.compare_digest(token, expected):
+        return _text_response("error: invalid or missing token", status=403, headers={"Access-Control-Allow-Origin": "*"})
+    url = clean_url(query.get("url", [""])[0].strip())
+    if not url or not url.startswith(("http://", "https://")):
+        return _text_response("error: invalid url", headers={"Access-Control-Allow-Origin": "*"})
+    try:
+        title = index_url(url)
+        msg = f"ok: {title}"
+    except Exception as e:
+        msg = f"error: {e}"
+    return _text_response(msg, headers={"Access-Control-Allow-Origin": "*"})

src/tinyweb/handlers/rns.py

@@ -0,0 +1,185 @@
+import json
+import time
+import threading
+import traceback
+import datetime
+from bs4 import BeautifulSoup
+from tinyweb.db import get_db, return_db
+from tinyweb.rns_client import fetch_remote_page
+from tinyweb.templates import esc
+
+
+class _PageCache:
+    def __init__(self, maxsize=50, ttl=300):
+        self._maxsize = maxsize
+        self._ttl = ttl
+        self._cache = {}
+        self._lock = threading.Lock()
+
+    def get(self, key):
+        with self._lock:
+            entry = self._cache.get(key)
+            if entry is None:
+                return None
+            if time.time() - entry["time"] > self._ttl:
+                del self._cache[key]
+                return None
+            self._cache.pop(key)
+            self._cache[key] = entry
+            return entry["value"]
+
+    def put(self, key, value):
+        with self._lock:
+            if key in self._cache:
+                self._cache.pop(key)
+            elif len(self._cache) >= self._maxsize:
+                oldest = next(iter(self._cache))
+                del self._cache[oldest]
+            self._cache[key] = {"value": value, "time": time.time()}
+
+
+_page_cache = _PageCache()
+
+
+def _inject_base_tag(html, dest_hash):
+    base = f'<base href="/rns/{dest_hash}/">'
+    head_start = html.find("<head")
+    if head_start >= 0:
+        close = html.find(">", head_start)
+        if close >= 0:
+            return html[:close + 1] + base + html[close + 1:]
+    return f"<head>{base}</head>{html}"
+
+
+def _get_mesh_sites():
+    db = get_db()
+    try:
+        return db.execute("SELECT hash, name, added_at FROM mesh_sites ORDER BY added_at DESC").fetchall()
+    finally:
+        return_db(db)
+
+
+def handle_rns_add_hash(dest_hash, name=""):
+    db = get_db()
+    errors = []
+    try:
+        db.execute(
+            "INSERT OR REPLACE INTO mesh_sites (hash, name) VALUES (?, ?)",
+            (dest_hash, name or ""),
+        )
+
+        try:
+            resp = fetch_remote_page(dest_hash, "/")
+            if resp.get("status") == 200:
+                body_raw = resp.get("body", "")
+                soup = BeautifulSoup(body_raw, 'html.parser')
+                for tag in soup(["script", "style", "nav", "footer", "header", "noscript", "aside"]):
+                    tag.decompose()
+                cleaned = soup.get_text(separator=" ", strip=True)
+
+                title = name or dest_hash[:16]
+                if soup.title and soup.title.string:
+                    title = soup.title.string.strip()
+
+                desc = ""
+                m = soup.find("meta", attrs={"name": "description"})
+                if m and m.get("content"):
+                    desc = m["content"].strip()
+                if not desc:
+                    m = soup.find("meta", attrs={"property": "og:description"})
+                    if m and m.get("content"):
+                        desc = m["content"].strip()
+                if not desc:
+                    desc = cleaned[:200].strip()
+
+                url = f"rns:{dest_hash}"
+                now = datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
+                db.execute(
+                    "INSERT OR REPLACE INTO pages (url, title, body, last_modified, summary) VALUES (?, ?, ?, ?, ?)",
+                    (url, title, cleaned, now, desc),
+                )
+            else:
+                errors.append(f"Remote returned status {resp.get('status')}")
+        except Exception as e:
+            errors.append(str(e))
+            traceback.print_exc()
+
+        db.commit()
+    finally:
+        return_db(db)
+
+    return errors if errors else None
+
+
+def handle_rns_delete_hash(body):
+    dest_hash = body.get("hash", [""])[0].strip() if isinstance(body, dict) else body
+    db = get_db()
+    try:
+        db.execute("DELETE FROM mesh_sites WHERE hash = ?", (dest_hash,))
+        db.commit()
+    finally:
+        return_db(db)
+    from tinyweb.handlers.pages import _redirect
+    return _redirect("/")
+
+
+def handle_rns_browse(path, dest_hash):
+    prefix = f"/rns/{dest_hash}"
+    sub_path = path[len(prefix):] if path.startswith(prefix) else "/"
+    if not sub_path:
+        sub_path = "/"
+
+    cache_key = (dest_hash, sub_path)
+    cached = _page_cache.get(cache_key)
+    if cached is not None:
+        return {
+            "status": 200,
+            "content_type": "text/html; charset=utf-8",
+            "body": cached,
+            "headers": {},
+        }
+
+    try:
+        resp = fetch_remote_page(dest_hash, sub_path)
+    except ConnectionError as e:
+        return {
+            "status": 200,
+            "content_type": "text/html; charset=utf-8",
+            "body": f"<h1>could not connect</h1><p>{esc(str(e))}</p>",
+            "headers": {},
+        }
+    except PermissionError:
+        return {
+            "status": 200,
+            "content_type": "text/html; charset=utf-8",
+            "body": "<h1>forbidden</h1><p>the remote instance blocked this request.</p>",
+            "headers": {},
+        }
+
+    if resp.get("status") != 200:
+        return {
+            "status": 200,
+            "content_type": "text/html; charset=utf-8",
+            "body": f"<h1>error</h1><p>remote returned status {resp['status']}</p>",
+            "headers": {},
+        }
+
+    body = resp.get("body", "")
+
+    if resp.get("content_type", "").startswith("application/json"):
+        try:
+            data = json.loads(body)
+            body = f"<pre>{esc(json.dumps(data, indent=2))}</pre>"
+        except (json.JSONDecodeError, TypeError):
+            body = f"<pre>{esc(body[:2000])}</pre>"
+    else:
+        body = _inject_base_tag(body, dest_hash)
+
+    _page_cache.put(cache_key, body)
+
+    return {
+        "status": 200,
+        "content_type": "text/html; charset=utf-8",
+        "body": body,
+        "headers": {},
+    }

src/tinyweb/handlers/search.py

@@ -0,0 +1,196 @@
+from tinyweb.db import get_db, return_db, get_setting, get_site_name, clean_url
+from tinyweb.templates import esc
+from ._helpers import _sanitize_fts_query, _paginate, _get_page_tags, _respond, _page_nav, PER_PAGE
+
+
+def handle_search(query):
+    q = query.get("q", [""])[0].strip()
+    page = _paginate(query)
+    offset = (page - 1) * PER_PAGE
+    db = get_db()
+    try:
+        count = db.execute("SELECT count(*) FROM pages").fetchone()[0]
+        name = get_site_name()
+
+        result_html = ""
+        trusted_html = ""
+        if q:
+            try:
+                fts_q = _sanitize_fts_query(q)
+                bm25_rows = db.execute(
+                    "SELECT p.id, p.url, p.title, p.body, p.note "
+                    "FROM pages_fts f JOIN pages p ON f.rowid = p.id "
+                    "WHERE pages_fts MATCH ? "
+                    "ORDER BY bm25(pages_fts, 10.0, 1.0, 5.0, 3.0) LIMIT 100",
+                    (fts_q,),
+                ).fetchall()
+            except Exception:
+                bm25_rows = []
+
+            bm25_ids = [r["id"] for r in bm25_rows]
+            chunk_snippets = {}
+            if get_setting("semantic_search", "0") == "1":
+                try:
+                    from tinyweb.embeddings import hybrid_search
+                    use_reranker = get_setting("use_reranker", "1") == "1"
+                    fused = hybrid_search(q, bm25_ids, limit=100, db=db, use_reranker=use_reranker)
+                    fused_ids = [pid for pid, _ in fused]
+                    chunk_snippets = {pid: text for pid, text in fused if text}
+                except Exception:
+                    fused_ids = bm25_ids
+            else:
+                fused_ids = bm25_ids
+
+            # Also match by tag
+            search_terms = [w.lower() for w in q.split() if w]
+            if search_terms:
+                placeholders = ",".join("?" * len(search_terms))
+                tag_rows = db.execute(
+                    f"SELECT DISTINCT pt.page_id FROM page_tags pt "
+                    f"JOIN tags t ON t.id = pt.tag_id "
+                    f"WHERE LOWER(t.name) IN ({placeholders})",
+                    search_terms,
+                ).fetchall()
+                tag_ids = {r["page_id"] for r in tag_rows}
+                seen = set(fused_ids)
+                for pid in tag_ids:
+                    if pid not in seen:
+                        fused_ids.append(pid)
+                        seen.add(pid)
+
+            total_results = len(fused_ids)
+            page_ids = fused_ids[offset:offset + PER_PAGE]
+
+            if page_ids:
+                placeholders = ",".join("?" * len(page_ids))
+                all_rows = db.execute(
+                    f"SELECT id, url, title, body, note, summary FROM pages WHERE id IN ({placeholders})",
+                    page_ids,
+                ).fetchall()
+                row_map = {r["id"]: r for r in all_rows}
+                rows = [row_map[pid] for pid in page_ids if pid in row_map]
+            else:
+                rows = []
+
+            if rows:
+                for r in rows:
+                    note_html = ""
+                    if r["note"]:
+                        note_html = f'<div class="note"><em>{esc(r["note"])}</em></div>'
+                    tags = _get_page_tags(r["id"], db)
+                    tags_html = ""
+                    if tags:
+                        tag_links = " ".join(f'<a href="/tags/{esc(t)}" class="tag">[{esc(t)}]</a>' for t in tags)
+                        tags_html = f'<div class="tags">{tag_links}</div>'
+                    snip_html = f'<br>{esc(r["summary"])}' if r["summary"] else ""
+                    url = r["url"]
+                    if url.startswith("rns:"):
+                        display_url = url
+                        link_url = f"/rns/{esc(url[4:])}/"
+                    else:
+                        display_url = url
+                        link_url = url
+                    result_html += (
+                        f'<div class="result">'
+                        f'<a href="{esc(link_url)}" rel="noreferrer noopener">{esc(r["title"])}</a><br>'
+                        f'<small>{esc(display_url)}</small>'
+                        f'{snip_html}'
+                        f'{note_html}{tags_html}'
+                        f'</div>'
+                    )
+            else:
+                result_html = "<p>No results in your index.</p>"
+
+            words = q.lower().split()
+            all_links = db.execute(
+                "SELECT l.url, l.label, p.title AS source_title "
+                "FROM links l JOIN pages p ON l.page_id = p.id",
+            ).fetchall()
+            indexed_urls = set(r["url"] for r in rows) if rows else set()
+            seen = set()
+            trusted = []
+            for l in all_links:
+                if l["url"] in indexed_urls or l["url"] in seen:
+                    continue
+                if any(w in l["label"].lower() for w in words):
+                    seen.add(l["url"])
+                    trusted.append(l)
+                if len(trusted) >= 20:
+                    break
+
+            if trusted:
+                items = ""
+                for l in trusted:
+                    items += (
+                        f'<li><a href="{esc(clean_url(l["url"]))}" rel="noreferrer noopener">{esc(l["label"])}</a> '
+                        f'<small>— from {esc(l["source_title"])}</small></li>'
+                    )
+                trusted_html = (
+                    f'<details class="trusted">'
+                    f'<summary>from your trusted sites ({len(trusted)})</summary>'
+                    f'<ul>{items}</ul>'
+                    f'</details>'
+                )
+
+            try:
+                remote_rows = db.execute(
+                    "SELECT rp.url, rp.title, rp.note, s.name AS source_name "
+                    "FROM remote_pages_fts rpf "
+                    "JOIN remote_pages rp ON rpf.rowid = rp.id "
+                    "JOIN subscriptions s ON rp.subscription_id = s.id "
+                    "WHERE remote_pages_fts MATCH ? ORDER BY rank LIMIT 50",
+                    (_sanitize_fts_query(q),),
+                ).fetchall()
+            except Exception:
+                remote_rows = []
+
+        remote_html = ""
+        if q and remote_rows:
+            by_source = {}
+            for r in remote_rows:
+                source = r["source_name"] or "unknown"
+                by_source.setdefault(source, []).append(r)
+            for source, items in by_source.items():
+                source_items = ""
+                for r in items:
+                    note_html = f' — <em>{esc(r["note"])}</em>' if r["note"] else ""
+                    source_items += (
+                        f'<li><a href="{esc(clean_url(r["url"]))}" rel="noreferrer noopener">{esc(r["title"])}</a>'
+                        f'{note_html} <small>({esc(clean_url(r["url"]))})</small></li>'
+                    )
+                remote_html += (
+                    f'<details class="remote" open>'
+                    f'<summary>from {esc(source)} ({len(items)})</summary>'
+                    f'<ul>{source_items}</ul>'
+                    f'</details>'
+                )
+    finally:
+        return_db(db)
+    sub_count = ""
+    if q and remote_rows:
+        sub_count = f" + {len(remote_rows)} from subscriptions"
+
+    welcome_html = ""
+    if count == 0 and not q:
+        welcome_html = (
+            '<section style="margin-top:1.5rem;max-width:40em">'
+            '<p>Your index is empty.</p>'
+            '<p>tinyweb is a personal search engine for pages you save. '
+            'The index stays on your machine; so does every search.</p>'
+            '<p>From here: <a href="/add">add a page</a>, '
+            '<a href="/style">get the bookmarklet</a>, or '
+            '<a href="/subscriptions">subscribe to another instance</a>.</p>'
+            '</section>'
+        )
+    return _respond(
+        f'<form method="get" action="/">'
+        f'<input name="q" value="{esc(q)}" placeholder="search your index" size="40">'
+        f' <button type="submit">search</button>'
+        f'</form>'
+        f'<p class="meta">{count} pages indexed'
+        f' · <a href="/add">+ add url</a></p>'
+        f'{welcome_html}'
+        f'{result_html}'
+        f'{_page_nav(page, total_results, f"/?q={esc(q)}") if q else ""}'
+        f'{trusted_html}{remote_html}'
+    )

src/tinyweb/handlers/subscriptions.py

@@ -0,0 +1,479 @@
+import threading
+import time
+from datetime import datetime
+
+from tinyweb.db import get_db, return_db, get_setting, set_setting, get_site_name, index_url, clean_url
+from tinyweb.templates import esc
+from tinyweb.rns_client import fetch_remote_sites
+from ._helpers import (
+    _get_page_tags, _respond, _redirect, _json_response, _error,
+    _csrf_field,
+)
+
+_sync_threads = {}
+_sync_starts = {}
+_SYNC_TIMEOUT = 120
+
+MAX_API_SITES = 5000
+MAX_BROWSE = 5000
+
+
+def _page_is_shared(tags, mode):
+    if "private" in tags:
+        return False
+    if mode == "require_public" and "public" not in tags:
+        return False
+    return True
+
+
+def _shared_sites(db, since=""):
+    mode = get_setting("sharing_mode", "exclude_private")
+    if since:
+        rows = db.execute(
+            "SELECT id, url, title, note, last_modified FROM pages "
+            "WHERE last_modified > ? ORDER BY id DESC LIMIT ?",
+            (since, MAX_API_SITES),
+        ).fetchall()
+    else:
+        rows = db.execute(
+            "SELECT id, url, title, note, last_modified FROM pages ORDER BY id DESC LIMIT ?",
+            (MAX_API_SITES,),
+        ).fetchall()
+    sites = []
+    for r in rows:
+        tags = _get_page_tags(r["id"], db)
+        if not _page_is_shared(tags, mode):
+            continue
+        sites.append({
+            "url": r["url"], "title": r["title"], "note": r["note"],
+            "tags": tags, "last_modified": r["last_modified"] or "",
+        })
+    return sites
+
+
+def _shared_all_urls(db):
+    mode = get_setting("sharing_mode", "exclude_private")
+    rows = db.execute(
+        "SELECT id, url FROM pages ORDER BY id DESC LIMIT ?", (MAX_API_SITES,)
+    ).fetchall()
+    return [r["url"] for r in rows if _page_is_shared(_get_page_tags(r["id"], db), mode)]
+
+
+def _count_shared_pages():
+    db = get_db()
+    try:
+        return len(_shared_all_urls(db))
+    finally:
+        return_db(db)
+
+
+def handle_share_preview():
+    mode = get_setting("sharing_mode", "exclude_private")
+    mode_label = (
+        "only pages tagged <code>public</code>"
+        if mode == "require_public"
+        else "all pages except those tagged <code>private</code>"
+    )
+    sharing_on = get_setting("sharing_enabled", "0") == "1"
+    status = (
+        '<p>Sharing is <b>enabled</b>. Subscribers see the pages listed below.</p>'
+        if sharing_on else
+        '<p>Sharing is <b>disabled</b>. Nothing is actually being shared right now; '
+        'this is the list that would be exposed if you enabled it.</p>'
+    )
+    db = get_db()
+    try:
+        sites = _shared_sites(db)
+    finally:
+        return_db(db)
+    if not sites:
+        body = (
+            "<h1>sharing preview</h1>"
+            f"<p>Rule: {mode_label}.</p>"
+            f"{status}"
+            "<p><em>No pages match the current rule.</em></p>"
+            '<p><a href="/style">back to settings</a></p>'
+        )
+        return _respond(body)
+    rows = ""
+    for s in sites:
+        tags_html = ""
+        if s["tags"]:
+            tags_html = " " + " ".join(f"[{esc(t)}]" for t in s["tags"])
+        note_html = f' — <em>{esc(s["note"])}</em>' if s["note"] else ""
+        rows += (
+            f'<li>'
+            f'<a href="{esc(s["url"])}" rel="noreferrer noopener">{esc(s["title"] or s["url"])}</a>'
+            f'{note_html}{tags_html} '
+            f'<br><small>{esc(s["url"])}</small>'
+            f'</li>'
+        )
+    body = (
+        "<h1>sharing preview</h1>"
+        f"<p>Rule: {mode_label}.</p>"
+        f"{status}"
+        f"<p><b>{len(sites)}</b> page(s) visible to subscribers.</p>"
+        f"<ul>{rows}</ul>"
+        '<p><a href="/style">back to settings</a></p>'
+    )
+    return _respond(body)
+
+
+def handle_api_sites(query=None):
+    if get_setting("sharing_enabled", "0") != "1":
+        return _json_response(
+            {"error": "sharing disabled"},
+            status=403,
+            headers={"Access-Control-Allow-Origin": "*"},
+        )
+    since = (query or {}).get("since", [""])[0].strip()
+    db = get_db()
+    try:
+        sites = _shared_sites(db, since=since)
+        all_urls = _shared_all_urls(db) if not since else None
+    finally:
+        return_db(db)
+    data = {"name": get_site_name(), "sites": sites}
+    if all_urls is not None:
+        data["all_urls"] = all_urls
+    return _json_response(data, headers={"Access-Control-Allow-Origin": "*"})
+
+
+def handle_subscriptions(msg=""):
+    db = get_db()
+    try:
+        subs = db.execute("SELECT * FROM subscriptions ORDER BY id DESC").fetchall()
+    finally:
+        return_db(db)
+    now_t = time.time()
+    for sub_id, start_t in list(_sync_starts.items()):
+        if now_t - start_t > _SYNC_TIMEOUT:
+            set_setting(f"sync_status_{sub_id}", "error:Timed out")
+            _sync_threads.pop(sub_id, None)
+            _sync_starts.pop(sub_id, None)
+
+    cards = ""
+    for s in subs:
+        sub_id = s["id"]
+        auto_label = "on" if s["auto_sync"] else "off"
+        last = s["last_sync"] or "never"
+        sync_status = get_setting(f"sync_status_{sub_id}", "")
+        is_syncing = sub_id in _sync_threads and _sync_threads[sub_id].is_alive()
+
+        if is_syncing:
+            status_html = '<div style="margin-top:0.4rem;font-size:0.85rem;color:#2070c0">syncing...</div>'
+        elif sync_status.startswith("done:"):
+            count = sync_status[5:]
+            status_html = f'<div style="margin-top:0.4rem;font-size:0.85rem;color:#30a030">synced {esc(count)} site(s)</div>'
+        elif sync_status.startswith("error:"):
+            err_msg = sync_status[6:]
+            status_html = f'<div style="margin-top:0.4rem;font-size:0.85rem;color:#c03030">{esc(err_msg)}</div>'
+        else:
+            status_html = ""
+
+        if is_syncing:
+            sync_btn = '<button disabled>syncing...</button>'
+        else:
+            sync_btn = (
+                f'<form method="post" action="/subscriptions/sync/{sub_id}" style="display:inline-block;margin:0">'
+                f'{_csrf_field()}<button>sync now</button></form>'
+            )
+
+        cards += (
+            f'<div style="border:1px solid #ddd;border-radius:4px;padding:0.9rem 1rem;margin-bottom:0.75rem">'
+            f'<div style="margin-bottom:0.4rem"><b>{esc(s["name"] or "unknown")}</b></div>'
+            f'<div><small>{esc(s["dest_hash"])}</small></div>'
+            f'<div style="margin-top:0.4rem;font-size:0.85rem;color:#606060">last sync: {esc(last)}</div>'
+            f'{status_html}'
+            f'<div style="display:flex;gap:0.5rem;align-items:center;flex-wrap:wrap;margin-top:0.7rem">'
+            f'<a href="/subscriptions/browse/{sub_id}" style="display:inline-flex;align-items:center;padding:0.3em 0">browse</a>'
+            f'{sync_btn}'
+            f'<form method="post" action="/subscriptions/autosync/{sub_id}" style="display:inline-block;margin:0">'
+            f'{_csrf_field()}<button>auto-sync: {auto_label}</button></form>'
+            f'<form method="post" action="/subscriptions/delete/{sub_id}" style="display:inline-block;margin:0">'
+            f'{_csrf_field()}<button>remove</button></form>'
+            f'</div>'
+            f'</div>'
+        )
+    any_syncing = any(
+        s["id"] in _sync_threads and _sync_threads[s["id"]].is_alive()
+        for s in subs
+    )
+    head_html = '<meta http-equiv="refresh" content="3">' if any_syncing else ""
+    listing = ""
+    if subs:
+        syncall_btn = '<button disabled>syncing...</button>' if any_syncing else '<button>sync all</button>'
+        listing = (
+            f'{cards}'
+            f'<form method="post" action="/subscriptions/syncall">'
+            f'{_csrf_field()}{syncall_btn}</form>'
+        )
+    return _respond(
+        f"<h1>subscriptions</h1>"
+        f'<form method="post" action="/subscriptions/add">'
+        f'{_csrf_field()}'
+        f'<input name="dest_hash" placeholder="destination hash" size="40"> '
+        f'<button>subscribe</button>'
+        f'</form>'
+        f'<p><small>or <a href="/subscriptions/add">subscribe to an instance</a></small></p>'
+        f'<p>{msg}</p>'
+        f'<hr>{listing}'
+        f'<br><a href="/">back</a>',
+        head_html=head_html,
+    )
+
+
+def handle_subscription_add(body):
+    dest_hash = body.get("dest_hash", [""])[0].strip().replace("<", "").replace(">", "")
+    if not dest_hash or len(dest_hash) != 32:
+        return handle_subscriptions("Enter a valid 32-character destination hash.")
+    try:
+        int(dest_hash, 16)
+    except ValueError:
+        return handle_subscriptions("Invalid destination hash (must be hex).")
+    try:
+        data = fetch_remote_sites(dest_hash)
+        name = data.get("name", "")
+    except PermissionError:
+        return handle_subscriptions("That instance has sharing disabled.")
+    except Exception:
+        return handle_subscriptions("Could not reach that instance.")
+    db = get_db()
+    try:
+        db.execute(
+            "INSERT INTO subscriptions (dest_hash, name) VALUES (?, ?) "
+            "ON CONFLICT(dest_hash) DO UPDATE SET name=excluded.name",
+            (dest_hash, name),
+        )
+        db.commit()
+    finally:
+        return_db(db)
+    return handle_subscriptions(f"Subscribed to {esc(name or dest_hash)}.")
+
+
+def handle_subscription_browse(sub_id):
+    db = get_db()
+    try:
+        sub = db.execute("SELECT * FROM subscriptions WHERE id = ?", (sub_id,)).fetchone()
+        if not sub:
+            return _error(404)
+        local_urls = set(r["url"] for r in db.execute("SELECT url FROM pages LIMIT ?", (MAX_BROWSE,)).fetchall())
+
+        remote_rows = db.execute(
+            "SELECT url, title, note, tags FROM remote_pages WHERE subscription_id = ? LIMIT ?",
+            (sub_id, MAX_BROWSE),
+        ).fetchall()
+    finally:
+        return_db(db)
+
+    if remote_rows:
+        sites = []
+        for r in remote_rows:
+            tags = [t for t in r["tags"].split(",") if t] if r["tags"] else []
+            sites.append({"url": r["url"], "title": r["title"], "note": r["note"], "tags": tags})
+    else:
+        try:
+            data = fetch_remote_sites(sub["dest_hash"])
+            sites = data.get("sites", [])
+        except PermissionError:
+            return handle_subscriptions("That instance has sharing disabled.")
+        except Exception:
+            return handle_subscriptions("Could not fetch sites from that instance.")
+
+    new_items = ""
+    existing_items = ""
+    new_count = 0
+    for s in sites:
+        if s["url"] in local_urls:
+            existing_items += (
+                f'<li style="opacity:0.5">{esc(s["title"])} '
+                f'<small>({esc(s["url"])})</small> — already indexed</li>'
+            )
+        else:
+            new_count += 1
+            note_html = f' — <em>{esc(s["note"])}</em>' if s.get("note") else ""
+            tags_html = ""
+            if s.get("tags"):
+                tags_html = " " + " ".join(f'[{esc(t)}]' for t in s["tags"])
+            new_items += (
+                f'<li><label><input type="checkbox" name="urls" value="{esc(s["url"])}">'
+                f' {esc(s["title"])}{note_html}{tags_html}'
+                f' <small>({esc(s["url"])})</small></label></li>'
+            )
+
+    buttons = ""
+    if new_count:
+        buttons = '<button>import selected</button> <button name="import_all" value="1">import all new</button>'
+    return _respond(
+        f'<h1>browsing: {esc(sub["name"] or sub["dest_hash"])}</h1>'
+        f'<p>{len(sites)} site(s) available, {new_count} new</p>'
+        f'<form method="post" action="/subscriptions/pick">'
+        f'{_csrf_field()}'
+        f'<input type="hidden" name="sub_id" value="{sub_id}">'
+        f'<ul>{new_items}</ul>'
+        f'{buttons}'
+        f'</form>'
+        f'<h3>already indexed</h3><ul>{existing_items}</ul>'
+        f'<a href="/subscriptions">back</a>'
+    )
+
+
+def handle_subscription_pick(body):
+    sub_id = body.get("sub_id", [""])[0]
+    import_all = body.get("import_all", [""])[0]
+
+    db = get_db()
+    try:
+        remote_rows = db.execute(
+            "SELECT url, tags FROM remote_pages WHERE subscription_id = ?", (sub_id,)
+        ).fetchall()
+        remote_tags = {r["url"]: r["tags"] for r in remote_rows}
+
+        if import_all:
+            local_urls = set(r["url"] for r in db.execute("SELECT url FROM pages LIMIT ?", (MAX_BROWSE,)).fetchall())
+            urls = [r["url"] for r in remote_rows if r["url"] not in local_urls]
+        else:
+            urls = body.get("urls", [])
+    finally:
+        return_db(db)
+
+    if not urls:
+        return handle_subscriptions("No sites selected.")
+
+    imported = 0
+    errors = 0
+    for url in urls:
+        try:
+            index_url(url)
+            tags_str = remote_tags.get(url, "")
+            if tags_str:
+                db = get_db()
+                try:
+                    row = db.execute("SELECT id FROM pages WHERE url = ?", (url,)).fetchone()
+                    if row:
+                        _set_page_tags(row["id"], tags_str, db)
+                        db.commit()
+                finally:
+                    return_db(db)
+            imported += 1
+        except Exception:
+            errors += 1
+    return handle_subscriptions(f"Imported {imported} page(s). {errors} error(s).")
+
+
+def _sync_subscription(sub_id):
+    db = None
+    try:
+        set_setting(f"sync_status_{sub_id}", "syncing")
+        db = get_db()
+        sub = db.execute("SELECT * FROM subscriptions WHERE id = ?", (sub_id,)).fetchone()
+        if not sub:
+            set_setting(f"sync_status_{sub_id}", "error:Subscription not found.")
+            return
+        since = sub["last_sync"].replace(" ", "T") if sub["last_sync"] else ""
+        try:
+            data = fetch_remote_sites(sub["dest_hash"], since=since)
+            sites = data.get("sites", [])
+            all_urls = data.get("all_urls")
+            remote_name = data.get("name", sub["name"])
+        except PermissionError:
+            set_setting(f"sync_status_{sub_id}", "error:That instance has sharing disabled.")
+            return
+        except Exception as e:
+            set_setting(f"sync_status_{sub_id}", f"error:Could not sync \u2014 {e}")
+            return
+
+        if all_urls is not None:
+            existing = db.execute(
+                "SELECT id, url FROM remote_pages WHERE subscription_id = ?", (sub_id,)
+            ).fetchall()
+            remote_url_set = set(all_urls)
+            for row in existing:
+                if row["url"] not in remote_url_set:
+                    db.execute("DELETE FROM remote_pages WHERE id = ?", (row["id"],))
+
+        synced = 0
+        for s in sites:
+            try:
+                tags_str = ",".join(s.get("tags", []))
+                db.execute(
+                    "INSERT INTO remote_pages (subscription_id, url, title, note, tags) VALUES (?, ?, ?, ?, ?) "
+                    "ON CONFLICT(subscription_id, url) DO UPDATE SET title=excluded.title, note=excluded.note, tags=excluded.tags",
+                    (sub_id, s["url"], s["title"], s.get("note", ""), tags_str),
+                )
+                if get_setting("semantic_search", "0") == "1":
+                    try:
+                        from tinyweb.embeddings import store_remote_embeddings
+                        rp_id = db.execute(
+                            "SELECT id FROM remote_pages WHERE subscription_id = ? AND url = ?",
+                            (sub_id, s["url"]),
+                        ).fetchone()["id"]
+                        store_remote_embeddings(rp_id, s["title"], s.get("note", ""), db)
+                    except Exception:
+                        pass
+                synced += 1
+            except Exception:
+                pass
+        now = datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
+        db.execute("UPDATE subscriptions SET last_sync = ?, name = ? WHERE id = ?", (now, remote_name, sub_id))
+        db.commit()
+        set_setting(f"sync_status_{sub_id}", f"done:{synced}")
+    except Exception as e:
+        set_setting(f"sync_status_{sub_id}", f"error:{e}")
+    finally:
+        if db:
+            return_db(db)
+        _sync_threads.pop(sub_id, None)
+        _sync_starts.pop(sub_id, None)
+
+
+def handle_subscription_sync(sub_id):
+    if sub_id in _sync_threads and _sync_threads[sub_id].is_alive():
+        return _redirect("/subscriptions")
+    set_setting(f"sync_status_{sub_id}", "syncing")
+    _sync_starts[sub_id] = time.time()
+    t = threading.Thread(target=_sync_subscription, args=(sub_id,), daemon=True)
+    _sync_threads[sub_id] = t
+    t.start()
+    return _redirect("/subscriptions")
+
+
+def handle_subscription_autosync(sub_id):
+    db = get_db()
+    try:
+        db.execute("UPDATE subscriptions SET auto_sync = 1 - auto_sync WHERE id = ?", (sub_id,))
+        db.commit()
+    finally:
+        return_db(db)
+    return _redirect("/subscriptions")
+
+
+def handle_subscription_delete(sub_id):
+    db = get_db()
+    try:
+        db.execute("DELETE FROM remote_pages WHERE subscription_id = ?", (sub_id,))
+        db.execute("DELETE FROM subscriptions WHERE id = ?", (sub_id,))
+        db.commit()
+    finally:
+        return_db(db)
+    return _redirect("/subscriptions")
+
+
+def handle_subscription_syncall():
+    db = get_db()
+    try:
+        subs = db.execute("SELECT * FROM subscriptions WHERE auto_sync = 1").fetchall()
+    finally:
+        return_db(db)
+    if not subs:
+        return handle_subscriptions("No subscriptions have auto-sync enabled.")
+    for sub in subs:
+        sub_id = sub["id"]
+        if sub_id in _sync_threads and _sync_threads[sub_id].is_alive():
+            continue
+        set_setting(f"sync_status_{sub_id}", "syncing")
+        _sync_starts[sub_id] = time.time()
+        t = threading.Thread(target=_sync_subscription, args=(sub_id,), daemon=True)
+        _sync_threads[sub_id] = t
+        t.start()
+    return _redirect("/subscriptions")

src/tinyweb/handlers/tags.py

@@ -0,0 +1,59 @@
+from tinyweb.db import get_db, return_db
+from tinyweb.templates import esc
+from ._helpers import _respond, _paginate, _page_nav, _get_page_tags, BROWSE_PER_PAGE
+
+
+def handle_tags():
+    db = get_db()
+    try:
+        rows = db.execute(
+            "SELECT t.name, COUNT(pt.page_id) AS cnt FROM tags t "
+            "JOIN page_tags pt ON t.id = pt.tag_id "
+            "GROUP BY t.id ORDER BY t.name"
+        ).fetchall()
+    finally:
+        return_db(db)
+    items = ""
+    for r in rows:
+        items += f'<li><a href="/tags/{esc(r["name"])}">{esc(r["name"])}</a> ({r["cnt"]})</li>'
+    return _respond(
+        f"<h1>tags</h1>"
+        f"<ul>{items}</ul>" if items else "<p>No tags yet. Add tags when saving or editing pages.</p>"
+        f'<a href="/">back</a>'
+    )
+
+
+def handle_tag_browse(tag_name, query=None):
+    page = _paginate(query or {})
+    offset = (page - 1) * BROWSE_PER_PAGE
+    db = get_db()
+    try:
+        total = db.execute(
+            "SELECT count(*) FROM page_tags pt JOIN tags t ON t.id = pt.tag_id WHERE t.name = ?",
+            (tag_name,),
+        ).fetchone()[0]
+        rows = db.execute(
+            "SELECT p.id, p.url, p.title, p.note FROM pages p "
+            "JOIN page_tags pt ON p.id = pt.page_id "
+            "JOIN tags t ON t.id = pt.tag_id "
+            "WHERE t.name = ? ORDER BY p.id DESC LIMIT ? OFFSET ?",
+            (tag_name, BROWSE_PER_PAGE, offset),
+        ).fetchall()
+        items = ""
+        for r in rows:
+            note_html = f' — <em>{esc(r["note"])}</em>' if r["note"] else ""
+            tags = _get_page_tags(r["id"], db)
+            tag_links = " ".join(f'<a href="/tags/{esc(t)}">[{esc(t)}]</a>' for t in tags)
+            items += (
+                f'<li>{esc(r["title"])}{note_html} {tag_links} '
+                f'<small>(<a href="{esc(r["url"])}" rel="noreferrer noopener">{esc(r["url"])}</a>)</small></li>'
+            )
+    finally:
+        return_db(db)
+    return _respond(
+        f'<h1>tag: {esc(tag_name)}</h1>'
+        f'<p>{total} page(s)</p>'
+        f'<ul>{items}</ul>'
+        f'{_page_nav(page, total, f"/tags/{esc(tag_name)}", BROWSE_PER_PAGE)}'
+        f'<a href="/tags">all tags</a> | <a href="/">back</a>'
+    )

src/tinyweb/rns_client.py

@@ -0,0 +1,117 @@
+import json
+import time
+import RNS
+
+APP_NAME = "tinyweb"
+ASPECTS = ["server"]
+
+# Two-tier timeout profiles: fast first, then slow for LoRa/multi-hop links
+_TIMEOUT_TIERS = [
+    {"path": 15, "link": 15, "request": 30, "poll": 0.25},
+    {"path": 60, "link": 60, "request": 120, "poll": 1.0},
+]
+
+
+# Request path for "/tinyweb" destination
+_RNS_REQUEST_PATH = "/tinyweb"
+
+
+def fetch_remote_sites(dest_hash_hex, since=""):
+    resp = _rns_request(dest_hash_hex, "/api/sites", {"since": [since]} if since else {})
+    return json.loads(resp.get("body", "{}"))
+
+
+def fetch_remote_page(dest_hash_hex, path, query=None):
+    return _rns_request(dest_hash_hex, path, query or {})
+
+
+def _rns_request(dest_hash_hex, path, query=None):
+    """Generic RNS request to a remote TinyWeb instance.
+
+    Connects over RNS, requests the given path, returns the response dict
+    (status, content_type, body, headers). Raises on failure.
+    Uses progressive timeouts: fast first, then slow for LoRa/multi-hop.
+    """
+    last_error = None
+    for tier in _TIMEOUT_TIERS:
+        try:
+            return _fetch(dest_hash_hex, path, query or {}, tier)
+        except PermissionError:
+            raise
+        except Exception as e:
+            last_error = e
+            continue
+    raise ConnectionError(
+        f"Could not reach {dest_hash_hex} after {len(_TIMEOUT_TIERS)} attempts: {last_error}"
+    )
+
+
+def _fetch(dest_hash_hex, path, query, timeouts):
+    """Single RNS fetch attempt with the given timeout profile."""
+    dest_hash = bytes.fromhex(dest_hash_hex)
+    poll = timeouts["poll"]
+
+    if not RNS.Transport.has_path(dest_hash):
+        RNS.Transport.request_path(dest_hash)
+        elapsed = 0
+        while not RNS.Transport.has_path(dest_hash) and elapsed < timeouts["path"]:
+            time.sleep(poll)
+            elapsed += poll
+        if not RNS.Transport.has_path(dest_hash):
+            raise ConnectionError(
+                f"Could not find path to {dest_hash_hex} ({timeouts['path']}s timeout)"
+            )
+
+    server_identity = RNS.Identity.recall(dest_hash)
+    if server_identity is None:
+        raise ConnectionError(f"Could not recall identity for {dest_hash_hex}")
+
+    destination = RNS.Destination(
+        server_identity,
+        RNS.Destination.OUT,
+        RNS.Destination.SINGLE,
+        APP_NAME,
+        *ASPECTS,
+    )
+
+    link = RNS.Link(destination)
+    elapsed = 0
+    while link.status == RNS.Link.PENDING and elapsed < timeouts["link"]:
+        time.sleep(poll)
+        elapsed += poll
+
+    if link.status != RNS.Link.ACTIVE:
+        raise ConnectionError(
+            f"Could not establish link to {dest_hash_hex} ({timeouts['link']}s timeout)"
+        )
+
+    try:
+        request_data = {
+            "method": "GET",
+            "path": path,
+            "query": query,
+            "body": {},
+            "gateway_host": "",
+        }
+        req_timeout = timeouts["request"]
+        receipt = link.request(_RNS_REQUEST_PATH, data=request_data, timeout=req_timeout)
+
+        elapsed = 0
+        done = (RNS.RequestReceipt.READY, RNS.RequestReceipt.DELIVERED, RNS.RequestReceipt.FAILED)
+        while receipt.get_status() not in done and elapsed < req_timeout:
+            time.sleep(poll)
+            elapsed += poll
+
+        if receipt.get_status() in (RNS.RequestReceipt.READY, RNS.RequestReceipt.DELIVERED):
+            resp = receipt.get_response()
+            if resp["status"] == 403:
+                raise PermissionError("Forbidden")
+            if resp["status"] != 200:
+                raise ConnectionError(f"Remote returned status {resp['status']}")
+            return resp
+        else:
+            raise ConnectionError(
+                f"Request failed or timed out ({req_timeout}s timeout)"
+            )
+    finally:
+        link.teardown()

src/tinyweb/templates.py

@@ -0,0 +1,45 @@
+import html
+from tinyweb.db import get_setting
+
+FORUM_ENABLED = False
+
+def esc(s):
+    return html.escape(str(s))
+
+
+def _nav_html():
+    name = esc(get_setting("site_name", "tinyweb"))
+    forum_link = ' | <a href="/forum">forum</a>' if FORUM_ENABLED else ""
+    return (
+        f'<p><b><a href="/">{name}</a></b>'
+        ' | <a href="/">search</a> | <a href="/pages">browse</a>'
+        ' | <a href="/tags">tags</a> | <a href="/subscriptions">subscriptions</a>'
+        f'{forum_link}'
+        ' | <a href="/style">customize</a> | <a href="/about">about</a></p>\n'
+        "<hr>\n"
+    )
+
+DEFAULT_TEMPLATE = "<html>\n<head>\n<meta name=\"referrer\" content=\"no-referrer\">\n<meta http-equiv=\"x-dns-prefetch-control\" content=\"off\">\n</head>\n<body>\n{{nav}}{{content}}\n</body>\n</html>"
+
+
+def _default_template():
+    return (
+        '<html>\n<head>\n<meta name="referrer" content="no-referrer">\n<meta http-equiv="x-dns-prefetch-control" content="off">\n'
+        '</head>\n<body>\n{{nav}}{{content}}\n</body>\n</html>'
+    )
+
+
+def wrap_page(body_html, use_default=False, head_html=""):
+    if use_default:
+        template = _default_template()
+    else:
+        template = get_setting("custom_template") or _default_template()
+        if "{{content}}" not in template:
+            template = _default_template()
+    forum_link = ' <a href="/forum">forum</a>' if FORUM_ENABLED else ""
+    template = template.replace("{{forum_link}}", forum_link)
+    template = template.replace("{{site_name}}", esc(get_setting("site_name", "tinyweb")))
+    template = template.replace("{{nav}}", _nav_html())
+    if head_html:
+        template = template.replace("</head>", head_html + "</head>")
+    return template.replace("{{content}}", body_html)

templates.py

@@ -1,40 +0,0 @@
-import html
-from db import get_setting
-
-
-def esc(s):
-    return html.escape(str(s))
-
-
-def snippet(text, query, ctx=80):
-    pos = text.lower().find(query.lower())
-    if pos == -1:
-        return text[:200]
-    start = max(0, pos - ctx)
-    end = min(len(text), pos + len(query) + ctx)
-    return ("..." if start > 0 else "") + text[start:end] + ("..." if end < len(text) else "")
-
-
-DEFAULT_TEMPLATE = "<html>\n<head>\n</head>\n<body>\n{{content}}\n</body>\n</html>"
-
-
-def _default_template():
-    name = esc(get_setting("site_name", "tinyweb"))
-    return (
-        "<html>\n<head>\n</head>\n<body>\n"
-        f'<p><b><a href="/">{name}</a></b>'
-        ' | <a href="/">search</a> | <a href="/pages">browse</a>'
-        ' | <a href="/tags">tags</a> | <a href="/subscriptions">subscriptions</a>'
-        ' | <a href="/style">customize</a> | <a href="/about">about</a></p>\n'
-        "<hr>\n{{content}}\n</body>\n</html>"
-    )
-
-
-def wrap_page(body_html, use_default=False):
-    if use_default:
-        template = _default_template()
-    else:
-        template = get_setting("custom_template") or _default_template()
-        if "{{content}}" not in template:
-            template = _default_template()
-    return template.replace("{{content}}", body_html)

tests/test_csrf.py

@@ -0,0 +1,60 @@
+"""Tests for `_check_csrf` — form-submission CSRF protection.
+
+Every POST handler calls this to verify the submitted _csrf field matches
+the token stored in the thread-local (which is seeded from the cookie by
+`dispatch_request`). Missing or mismatched tokens must fail closed.
+"""
+import tinyweb.handlers as handlers_module
+from tinyweb.handlers import _check_csrf, _csrf_field, _get_csrf_token
+
+
+def _set_token(token):
+    handlers_module._request_local.csrf_token = token
+
+
+def _clear_token():
+    if hasattr(handlers_module._request_local, "csrf_token"):
+        del handlers_module._request_local.csrf_token
+
+
+def teardown_function(_):
+    _clear_token()
+
+
+def test_rejects_missing_token_in_body():
+    _set_token("server-side-token")
+    assert _check_csrf({}) is False
+
+
+def test_rejects_empty_token_in_body():
+    _set_token("server-side-token")
+    assert _check_csrf({"_csrf": [""]}) is False
+
+
+def test_rejects_mismatched_token():
+    _set_token("server-side-token")
+    assert _check_csrf({"_csrf": ["attacker-token"]}) is False
+
+
+def test_accepts_matching_token():
+    _set_token("server-side-token")
+    assert _check_csrf({"_csrf": ["server-side-token"]}) is True
+
+
+def test_rejects_when_server_token_missing():
+    """If the server-side token is empty (shouldn't happen after dispatch_request
+    seeds it, but be defensive), the check must fail closed."""
+    _clear_token()
+    assert _check_csrf({"_csrf": ["anything"]}) is False
+
+
+def test_csrf_field_renders_current_token():
+    _set_token("abc123")
+    field = _csrf_field()
+    assert 'name="_csrf"' in field
+    assert 'value="abc123"' in field
+
+
+def test_get_csrf_token_returns_empty_when_unset():
+    _clear_token()
+    assert _get_csrf_token() == ""

tests/test_db_index_url.py

@@ -0,0 +1,155 @@
+"""Tests for `index_url` — the main write path.
+
+Covers UPSERT behavior, links being replaced on re-index, FTS index staying
+in sync via triggers, and the connection pool returning clean connections.
+"""
+from unittest.mock import patch
+
+from conftest import patch_dns_ok
+import tinyweb.db as db_module
+from tinyweb.db import get_db, return_db, index_url
+
+
+def _mock_fetch_page(title="Test Page", body="test body text", links=None, meta=""):
+    """Return a replacement for db.fetch_page that yields canned data."""
+    links = links or []
+    def fake(url):
+        return (title, body, links, meta)
+    return fake
+
+
+def test_insert_creates_page_row_and_fts_entry(temp_db, monkeypatch):
+    patch_dns_ok(monkeypatch)
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(
+        title="Rust Intro", body="ownership and borrowing basics", links=[],
+    ))
+    index_url("https://example.com/rust")
+
+    db = get_db()
+    try:
+        row = db.execute("SELECT id, title, body FROM pages").fetchone()
+        assert row is not None
+        assert row["title"] == "Rust Intro"
+        assert "ownership" in row["body"]
+        # Verify FTS trigger fired.
+        fts_hits = db.execute(
+            "SELECT rowid FROM pages_fts WHERE pages_fts MATCH 'ownership*'"
+        ).fetchall()
+        assert len(fts_hits) == 1
+        assert fts_hits[0]["rowid"] == row["id"]
+    finally:
+        return_db(db)
+
+
+def test_re_indexing_same_url_updates_in_place(temp_db, monkeypatch):
+    patch_dns_ok(monkeypatch)
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(
+        title="First Title", body="first body", links=[],
+    ))
+    index_url("https://example.com/page")
+
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(
+        title="Second Title", body="second body", links=[],
+    ))
+    index_url("https://example.com/page")
+
+    db = get_db()
+    try:
+        rows = db.execute("SELECT title, body FROM pages").fetchall()
+    finally:
+        return_db(db)
+    assert len(rows) == 1, "re-indexing should UPDATE not INSERT"
+    assert rows[0]["title"] == "Second Title"
+
+
+def test_links_replaced_on_reindex(temp_db, monkeypatch):
+    patch_dns_ok(monkeypatch)
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(
+        title="T", body="b",
+        links=[("https://example.com/a", "first"), ("https://example.com/b", "second")],
+    ))
+    index_url("https://example.com/src")
+
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(
+        title="T", body="b",
+        links=[("https://example.com/c", "third-only")],
+    ))
+    index_url("https://example.com/src")
+
+    db = get_db()
+    try:
+        rows = db.execute("SELECT url FROM links").fetchall()
+    finally:
+        return_db(db)
+    urls = {r["url"] for r in rows}
+    assert urls == {"https://example.com/c"}, "old links should be deleted on reindex"
+
+
+def test_url_cleaned_before_insert(temp_db, monkeypatch):
+    """index_url should apply clean_url before touching the DB, so tracking params
+    don't create duplicate rows."""
+    patch_dns_ok(monkeypatch)
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(title="T", body="b"))
+    index_url("https://example.com/page?utm_source=twitter#frag")
+
+    db = get_db()
+    try:
+        rows = db.execute("SELECT url FROM pages").fetchall()
+    finally:
+        return_db(db)
+    assert len(rows) == 1
+    assert rows[0]["url"] == "https://example.com/page"
+
+
+def test_summary_populated_from_meta_description(temp_db, monkeypatch):
+    patch_dns_ok(monkeypatch)
+    long_meta = "A thoughtful description that exceeds twenty chars"
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(
+        title="T", body="b", meta=long_meta,
+    ))
+    index_url("https://example.com/page")
+
+    db = get_db()
+    try:
+        row = db.execute("SELECT summary FROM pages").fetchone()
+    finally:
+        return_db(db)
+    assert row["summary"] == long_meta
+
+
+def test_short_meta_description_not_stored_as_summary(temp_db, monkeypatch):
+    patch_dns_ok(monkeypatch)
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(
+        title="T", body="b", meta="too short",
+    ))
+    index_url("https://example.com/page")
+
+    db = get_db()
+    try:
+        row = db.execute("SELECT summary FROM pages").fetchone()
+    finally:
+        return_db(db)
+    assert row["summary"] == ""
+
+
+def test_pool_returns_clean_connection(temp_db, monkeypatch):
+    """Regression for 1bc695f — `return_db` should roll back uncommitted work
+    so the next consumer doesn't see stale state."""
+    patch_dns_ok(monkeypatch)
+    monkeypatch.setattr(db_module, "fetch_page", _mock_fetch_page(title="T", body="b"))
+    index_url("https://example.com/one")
+
+    # Take a connection, make a dirty uncommitted change, return it.
+    db = get_db()
+    db.execute("INSERT INTO pages (url, title, body) VALUES (?, ?, ?)",
+               ("https://dirty.example.com/", "dirty", "dirty"))
+    # NOTE: no commit here — this is the dirty state we want rolled back.
+    return_db(db)
+
+    # A later consumer must not see the dirty row.
+    db2 = get_db()
+    try:
+        urls = {r["url"] for r in db2.execute("SELECT url FROM pages").fetchall()}
+    finally:
+        return_db(db2)
+    assert "https://dirty.example.com/" not in urls

tests/test_db_schema.py

@@ -0,0 +1,90 @@
+"""Tests for `init_db` and the settings key-value store.
+
+`init_db` is called unconditionally on startup, so it must be idempotent
+and create every table/trigger the rest of the app expects.
+"""
+from tinyweb.db import get_db, return_db, init_db, get_setting, set_setting, get_site_name
+
+
+EXPECTED_TABLES = {
+    "pages", "links", "settings", "subscriptions",
+    "remote_pages", "tags", "page_tags", "chunks",
+    # FTS5 virtual tables:
+    "pages_fts", "remote_pages_fts",
+}
+
+
+def test_all_expected_tables_exist(temp_db):
+    db = get_db()
+    try:
+        rows = db.execute(
+            "SELECT name FROM sqlite_master WHERE type IN ('table') AND name NOT LIKE 'sqlite_%'"
+        ).fetchall()
+        names = {r["name"] for r in rows}
+    finally:
+        return_db(db)
+    missing = EXPECTED_TABLES - names
+    assert not missing, f"tables missing after init_db: {missing}"
+
+
+def test_fts_triggers_exist(temp_db):
+    db = get_db()
+    try:
+        rows = db.execute(
+            "SELECT name FROM sqlite_master WHERE type = 'trigger'"
+        ).fetchall()
+        names = {r["name"] for r in rows}
+    finally:
+        return_db(db)
+    # These triggers keep pages_fts in sync with pages on insert/update/delete.
+    for trigger in ("pages_ai", "pages_ad", "pages_au"):
+        assert trigger in names, f"missing trigger {trigger}"
+
+
+def test_init_db_is_idempotent(temp_db):
+    """Running init_db twice on the same DB must not error or duplicate anything."""
+    init_db()
+    init_db()  # second call should be a no-op
+    db = get_db()
+    try:
+        count = db.execute(
+            "SELECT count(*) FROM sqlite_master WHERE name = 'pages'"
+        ).fetchone()[0]
+    finally:
+        return_db(db)
+    assert count == 1
+
+
+def test_get_setting_returns_default_when_missing(temp_db):
+    assert get_setting("nonexistent", "fallback") == "fallback"
+    assert get_setting("nonexistent") == ""
+
+
+def test_set_setting_then_get(temp_db):
+    set_setting("site_name", "my-personal-index")
+    assert get_setting("site_name") == "my-personal-index"
+
+
+def test_set_setting_updates_existing(temp_db):
+    set_setting("key", "first")
+    set_setting("key", "second")
+    assert get_setting("key") == "second"
+
+
+def test_get_site_name_has_default(temp_db):
+    assert get_site_name() == "tinyweb"
+
+
+def test_get_site_name_reflects_override(temp_db):
+    set_setting("site_name", "custom-site")
+    assert get_site_name() == "custom-site"
+
+
+def test_foreign_keys_pragma_enabled(temp_db):
+    """Pool connections should have foreign_keys=ON so CASCADE deletes work."""
+    db = get_db()
+    try:
+        row = db.execute("PRAGMA foreign_keys").fetchone()
+    finally:
+        return_db(db)
+    assert row[0] == 1

tests/test_fts_sanitizer.py

@@ -0,0 +1,113 @@
+"""Tests for `_sanitize_fts_query`.
+
+The sanitizer is the boundary between user input and FTS5 MATCH syntax.
+Commit 1bc695f tightened it after noticing that colons and operator words
+could escape the quoting. These tests keep that regression dead.
+"""
+import pytest
+
+from tinyweb.handlers import _sanitize_fts_query
+
+
+def test_empty_query_returns_no_match_token():
+    assert _sanitize_fts_query("") == '""'
+    assert _sanitize_fts_query("   ") == '""'
+
+
+def test_single_word_becomes_prefix_match():
+    assert _sanitize_fts_query("rust") == "rust*"
+
+
+def test_multi_word_quotes_all_but_last():
+    result = _sanitize_fts_query("rust borrow checker")
+    assert result == '"rust" "borrow" checker*'
+
+
+def test_stopwords_are_dropped():
+    # "the" and "a" should vanish; only "cat" remains (and gets prefix star).
+    assert _sanitize_fts_query("the a cat") == "cat*"
+
+
+def test_all_stopwords_returns_no_match_token():
+    assert _sanitize_fts_query("the and or") == '""'
+
+
+@pytest.mark.parametrize("bad_char", ["'", "(", ")", "+", "-", "^", "~", ":"])
+def test_fts5_operators_stripped_from_tokens(bad_char):
+    """FTS5 special chars inside user tokens must not survive — regression for 1bc695f.
+
+    The sanitizer legitimately adds `"` around tokens and a trailing `*` for prefix
+    matching; both are excluded from this check.
+    """
+    payload = f"foo{bad_char}bar"
+    out = _sanitize_fts_query(payload)
+    assert bad_char not in out, f"{bad_char!r} leaked into {out!r}"
+
+
+def test_asterisk_only_appears_as_trailing_prefix():
+    """Input `*` should not become an in-token asterisk; the sanitizer's trailing `*` is fine."""
+    out = _sanitize_fts_query("foo*bar")
+    assert out.count("*") <= 1
+    if "*" in out:
+        assert out.endswith("*")
+
+
+def test_quote_in_input_does_not_break_out_of_quoted_token():
+    """A `"` in user input must not close the sanitizer's protective quoting.
+
+    The sanitizer wraps each non-last token in double quotes; if a stray `"` from
+    the user slipped through, the resulting FTS5 expression would be interpreted
+    as broken syntax or, worse, a column filter.
+    """
+    out = _sanitize_fts_query('foo"bar baz"qux')
+    # Each pair of quotes in the output should be balanced and around a clean token.
+    assert out.count('"') % 2 == 0
+    # No embedded quotes inside a quoted region.
+    import re
+    for match in re.findall(r'"[^"]*"', out):
+        inner = match[1:-1]
+        assert '"' not in inner
+
+
+@pytest.mark.parametrize("op", ["AND", "OR", "NOT", "NEAR", "and", "or", "not", "near"])
+def test_fts5_operator_words_dropped(op):
+    """AND/OR/NOT/NEAR would be interpreted as operators on the unquoted last token."""
+    out = _sanitize_fts_query(f"foo {op} bar")
+    # the operator word itself should not appear
+    assert op.upper() not in out.upper().split('"'), f"operator {op!r} survived in {out!r}"
+
+
+def test_injection_payload_produces_valid_fts5():
+    """End-to-end: a realistic injection payload must produce syntactically valid FTS5.
+
+    We run the sanitized output through a throwaway FTS5 table; if the sanitizer
+    leaks operator characters the MATCH either raises or interprets malicious syntax.
+    """
+    import sqlite3
+    conn = sqlite3.connect(":memory:")
+    conn.execute("CREATE VIRTUAL TABLE t USING fts5(body)")
+    conn.execute("INSERT INTO t (body) VALUES ('hello world')")
+
+    for payload in [
+        'foo": OR bar NOT baz AND qux*()',
+        '" OR 1=1 --',
+        "title:secret AND public",
+        "(((",
+        "^^^~~~",
+    ]:
+        q = _sanitize_fts_query(payload)
+        # Must not raise — if operators leaked, FTS5 would error or mis-parse.
+        conn.execute("SELECT * FROM t WHERE t MATCH ?", (q,)).fetchall()
+    conn.close()
+
+
+def test_whitespace_only_tokens_dropped():
+    # tokens that become empty after stripping special chars should not produce bare quotes
+    out = _sanitize_fts_query('""" "" ""')
+    assert out == '""'
+
+
+def test_colon_stripped():
+    """Regression for 1bc695f — colon is an FTS5 column filter and must be stripped."""
+    out = _sanitize_fts_query("title:secret")
+    assert ":" not in out

tests/test_gateway_limits.py

@@ -0,0 +1,164 @@
+"""Tests for gateway-level guards: body-size cap and Reticulum surface whitelist.
+
+Regression targets from commit 1bc695f — a 16 MiB upload limit (DoS guard)
+and a strict GET-/api/sites-only whitelist for requests arriving over the
+Reticulum mesh (CSRF can't protect mesh callers, so gate by whitelist).
+"""
+import io
+
+import pytest
+
+from tinyweb import app as app_module
+from tinyweb.gateway import GatewayHandler, MAX_BODY_SIZE
+
+
+class FakeHeaders:
+    """Minimal replacement for http.server request headers."""
+    def __init__(self, items=None):
+        self._items = dict(items or {})
+
+    def get(self, key, default=None):
+        return self._items.get(key, default)
+
+
+class FakeGatewayHandler(GatewayHandler):
+    """Bypass the socket-bound __init__ and capture response calls in memory."""
+    def __init__(self, path="/", method="POST", headers=None, rfile=None):
+        self.path = path
+        self.command = method
+        self.headers = FakeHeaders(headers or {})
+        self.rfile = rfile or io.BytesIO()
+        self.wfile = io.BytesIO()
+        self._captured = {
+            "error": None, "status": None, "headers": [], "body_written": None,
+        }
+
+    def send_error(self, code, msg=""):
+        self._captured["error"] = (code, msg)
+
+    def send_response(self, code):
+        self._captured["status"] = code
+
+    def send_header(self, k, v):
+        self._captured["headers"].append((k, v))
+
+    def end_headers(self):
+        pass
+
+
+def test_post_over_size_cap_rejected_with_413():
+    """Regression for 1bc695f: request bodies over MAX_BODY_SIZE must be rejected
+    without being read into memory."""
+    oversize = MAX_BODY_SIZE + 1
+    handler = FakeGatewayHandler(
+        path="/add",
+        method="POST",
+        headers={"Content-Length": str(oversize)},
+    )
+    handler._forward("POST")
+    assert handler._captured["error"] is not None
+    code, _msg = handler._captured["error"]
+    assert code == 413
+
+
+def test_post_at_size_cap_accepted():
+    """A body exactly at MAX_BODY_SIZE should not be rejected by the size check."""
+    handler = FakeGatewayHandler(
+        path="/_does_not_matter",
+        method="POST",
+        headers={"Content-Length": str(MAX_BODY_SIZE)},
+        # rfile has no data; handler will try to read; local_dispatch isn't set.
+        # We only care that the 413 check passes, not that the request succeeds.
+        rfile=io.BytesIO(b""),
+    )
+    # Stub out local_dispatch so _forward doesn't try the network path.
+    from tinyweb.gateway import GatewayState
+    original = GatewayState.local_dispatch
+    GatewayState.local_dispatch = lambda data: {
+        "status": 404, "content_type": "text/plain", "body": "nope",
+    }
+    try:
+        handler._forward("POST")
+    finally:
+        GatewayState.local_dispatch = original
+    # Not a 413, because the body is exactly at the cap (cap is inclusive).
+    if handler._captured["error"]:
+        assert handler._captured["error"][0] != 413
+
+
+def test_negative_content_length_rejected():
+    handler = FakeGatewayHandler(
+        path="/add",
+        method="POST",
+        headers={"Content-Length": "-1"},
+    )
+    handler._forward("POST")
+    assert handler._captured["error"] is not None
+    code, _msg = handler._captured["error"]
+    assert code == 400
+
+
+def test_invalid_content_length_rejected():
+    handler = FakeGatewayHandler(
+        path="/add",
+        method="POST",
+        headers={"Content-Length": "abc"},
+    )
+    handler._forward("POST")
+    assert handler._captured["error"] is not None
+    code, _msg = handler._captured["error"]
+    assert code == 400
+
+
+# -------- Reticulum mesh surface whitelist --------
+
+
+def test_mesh_rejects_non_api_sites_get():
+    """Regression for 1bc695f: remote mesh callers can only GET /api/sites."""
+    resp = app_module.rns_request_handler(
+        path="/tinyweb",
+        data={"method": "GET", "path": "/pages", "query": {}, "body": {}, "gateway_host": ""},
+        request_id="x", link_id="y", remote_identity=None, requested_at=0,
+    )
+    assert resp["status"] == 403
+
+
+def test_mesh_rejects_post_to_api_sites():
+    resp = app_module.rns_request_handler(
+        path="/tinyweb",
+        data={"method": "POST", "path": "/api/sites", "query": {}, "body": {}, "gateway_host": ""},
+        request_id="x", link_id="y", remote_identity=None, requested_at=0,
+    )
+    assert resp["status"] == 403
+
+
+def test_mesh_rejects_sensitive_local_endpoints():
+    for path in ("/add", "/delete/1", "/style", "/import", "/export"):
+        resp = app_module.rns_request_handler(
+            path="/tinyweb",
+            data={"method": "GET", "path": path, "query": {}, "body": {}, "gateway_host": ""},
+            request_id="x", link_id="y", remote_identity=None, requested_at=0,
+        )
+        assert resp["status"] == 403, f"path {path!r} leaked through mesh whitelist"
+
+
+def test_mesh_allows_api_sites_get(temp_db, csrf_session):
+    """Sanity check: the one whitelisted combination is accepted."""
+    resp = app_module.rns_request_handler(
+        path="/tinyweb",
+        data={"method": "GET", "path": "/api/sites", "query": {}, "body": {}, "gateway_host": ""},
+        request_id="x", link_id="y", remote_identity=None, requested_at=0,
+    )
+    # Status depends on handler output; 200 is the happy path.
+    assert resp["status"] in (200, 403)  # 403 if sharing is disabled by default
+
+
+def test_mesh_handles_missing_data_payload():
+    """Regression-minded check: a None or malformed data object shouldn't crash."""
+    resp = app_module.rns_request_handler(
+        path="/tinyweb",
+        data=None,
+        request_id="x", link_id="y", remote_identity=None, requested_at=0,
+    )
+    # Default data has method=GET, path=/ which is not in the whitelist.
+    assert resp["status"] == 403

tests/test_handlers_pages.py

@@ -0,0 +1,174 @@
+"""Tests for `handle_bulk_action`, edit flow, and the bulk-delete confirm step.
+
+The bulk-delete confirmation flow is a data-loss guard added in commit
+8dffd8c — a stray POST without `confirmed=1` must render the confirmation
+page instead of actually deleting.
+"""
+from tinyweb.db import get_db, return_db
+from tinyweb.handlers import (
+    handle_bulk_action,
+    handle_edit_form,
+    handle_edit_submit,
+    handle_pages,
+)
+
+
+def _all_urls(seeded_db):
+    db = get_db()
+    try:
+        return {r["url"] for r in db.execute("SELECT url FROM pages").fetchall()}
+    finally:
+        return_db(db)
+
+
+def _page_id(seeded_db, url):
+    db = get_db()
+    try:
+        return db.execute("SELECT id FROM pages WHERE url = ?", (url,)).fetchone()["id"]
+    finally:
+        return_db(db)
+
+
+def test_bulk_delete_without_confirmed_renders_confirm_page(seeded_db, csrf_session):
+    """Regression for 8dffd8c: bulk delete must NOT delete until confirmed=1 is set."""
+    pid = _page_id(seeded_db, "https://example.com/rust-intro")
+    urls_before = _all_urls(seeded_db)
+
+    resp = handle_bulk_action({
+        "ids": [str(pid)],
+        "action": ["delete"],
+    })
+    assert resp["status"] == 200
+    assert "confirm delete" in resp["body"].lower()
+    assert "Rust Intro" in resp["body"]
+    # Must still show a hidden confirmed=1 field in the follow-up form.
+    assert 'name="confirmed" value="1"' in resp["body"]
+
+    # Crucially: nothing should have been deleted.
+    assert _all_urls(seeded_db) == urls_before
+
+
+def test_bulk_delete_with_confirmed_actually_deletes(seeded_db, csrf_session):
+    pid = _page_id(seeded_db, "https://example.com/rust-intro")
+
+    resp = handle_bulk_action({
+        "ids": [str(pid)],
+        "action": ["delete"],
+        "confirmed": ["1"],
+    })
+    # Confirmed delete redirects back to /pages.
+    assert resp["status"] in (302, 303)
+
+    urls = _all_urls(seeded_db)
+    assert "https://example.com/rust-intro" not in urls
+    # Other pages untouched.
+    assert "https://example.com/python-tips" in urls
+
+
+def test_bulk_delete_with_no_ids_redirects(seeded_db, csrf_session):
+    resp = handle_bulk_action({
+        "ids": [],
+        "action": ["delete"],
+        "confirmed": ["1"],
+    })
+    assert resp["status"] in (302, 303)
+    assert _all_urls(seeded_db) == {
+        "https://example.com/rust-intro",
+        "https://example.com/python-tips",
+        "https://example.com/ocaml-why",
+        "https://news.example.org/mesh",
+    }
+
+
+def test_bulk_delete_rejects_non_integer_ids(seeded_db, csrf_session):
+    resp = handle_bulk_action({
+        "ids": ["not-a-number"],
+        "action": ["delete"],
+        "confirmed": ["1"],
+    })
+    assert resp["status"] == 400
+
+
+def test_bulk_retag_add_mode_merges_tags(seeded_db, csrf_session):
+    pid = _page_id(seeded_db, "https://example.com/python-tips")
+
+    handle_bulk_action({
+        "ids": [str(pid)],
+        "action": ["retag"],
+        "bulk_tags": ["scripting, tutorials"],
+        "tag_mode": ["add"],
+    })
+    db = get_db()
+    try:
+        rows = db.execute(
+            "SELECT t.name FROM tags t JOIN page_tags pt ON pt.tag_id = t.id "
+            "WHERE pt.page_id = ? ORDER BY t.name",
+            (pid,),
+        ).fetchall()
+    finally:
+        return_db(db)
+    tags = [r["name"] for r in rows]
+    assert "python" in tags  # existing kept
+    assert "scripting" in tags  # new added
+    assert "tutorials" in tags
+
+
+def test_bulk_retag_replace_mode_overwrites_tags(seeded_db, csrf_session):
+    pid = _page_id(seeded_db, "https://example.com/python-tips")
+
+    handle_bulk_action({
+        "ids": [str(pid)],
+        "action": ["retag"],
+        "bulk_tags": ["one, two"],
+        "tag_mode": ["replace"],
+    })
+    db = get_db()
+    try:
+        rows = db.execute(
+            "SELECT t.name FROM tags t JOIN page_tags pt ON pt.tag_id = t.id "
+            "WHERE pt.page_id = ?",
+            (pid,),
+        ).fetchall()
+    finally:
+        return_db(db)
+    tags = {r["name"] for r in rows}
+    assert tags == {"one", "two"}
+    assert "python" not in tags
+
+
+def test_edit_form_renders_current_values(seeded_db, csrf_session):
+    pid = _page_id(seeded_db, "https://example.com/rust-intro")
+    resp = handle_edit_form(pid)
+    assert resp["status"] == 200
+    assert "Rust Intro" in resp["body"]
+    # Existing tags should appear in the tag field.
+    assert "rust" in resp["body"]
+
+
+def test_edit_form_404_for_unknown_page(temp_db, csrf_session):
+    resp = handle_edit_form(99999)
+    assert resp["status"] == 404
+
+
+def test_edit_submit_updates_title_and_note(seeded_db, csrf_session):
+    pid = _page_id(seeded_db, "https://example.com/rust-intro")
+    handle_edit_submit(pid, {
+        "title": ["New Rust Title"],
+        "note": ["new annotation"],
+        "tags": ["rust, updated"],
+    })
+    db = get_db()
+    try:
+        row = db.execute("SELECT title, note FROM pages WHERE id = ?", (pid,)).fetchone()
+    finally:
+        return_db(db)
+    assert row["title"] == "New Rust Title"
+    assert row["note"] == "new annotation"
+
+
+def test_handle_pages_lists_indexed_pages(seeded_db, csrf_session):
+    resp = handle_pages({})
+    assert resp["status"] == 200
+    # Every seeded page title appears on the list page.
+    for title in ("Rust Intro", "Python Tips", "Why OCaml", "Mesh Networking"):
+        assert title in resp["body"]

tests/test_handlers_search.py

@@ -0,0 +1,63 @@
+"""Tests for `handle_search` — the home page + primary user flow."""
+from tinyweb.handlers import handle_search
+
+
+def test_empty_index_empty_query_shows_welcome(temp_db, csrf_session):
+    resp = handle_search({})
+    assert resp["status"] == 200
+    body = resp["body"]
+    assert "Your index is empty" in body
+    # Links the welcome panel offers as equal-weight starting points.
+    assert "/add" in body
+    assert "/style" in body
+    assert "/subscriptions" in body
+
+
+def test_empty_index_with_query_shows_no_results(temp_db, csrf_session):
+    resp = handle_search({"q": ["rust"]})
+    assert resp["status"] == 200
+    assert "No results in your index" in resp["body"]
+
+
+def test_populated_index_with_matching_query_returns_results(seeded_db, csrf_session):
+    resp = handle_search({"q": ["rust"]})
+    assert resp["status"] == 200
+    assert "Rust Intro" in resp["body"]
+    # Page count shown in meta line.
+    assert "4 pages indexed" in resp["body"]
+
+
+def test_query_only_matches_relevant_pages(seeded_db, csrf_session):
+    resp = handle_search({"q": ["ocaml"]})
+    body = resp["body"]
+    assert "Why OCaml" in body
+    assert "Python Tips" not in body
+    assert "Rust Intro" not in body
+
+
+def test_pagination_query_param_respected(seeded_db, csrf_session):
+    """A high page number should still render without crashing."""
+    resp = handle_search({"q": ["example"], "p": ["99"]})
+    assert resp["status"] == 200
+
+
+def test_trusted_sites_fallback_surfaces_when_query_matches_link_label(seeded_db, csrf_session):
+    """Links extracted from indexed pages act as a fallback when direct results
+    are absent or thin; labels are substring-matched case-insensitively."""
+    resp = handle_search({"q": ["advanced"]})
+    body = resp["body"]
+    # The label "advanced rust guide" is on a link extracted from rust-intro.
+    assert "advanced rust guide" in body
+    assert "trusted sites" in body
+
+
+def test_page_count_in_meta_line(seeded_db, csrf_session):
+    resp = handle_search({})
+    assert "4 pages indexed" in resp["body"]
+
+
+def test_csp_and_security_headers_not_in_handler_but_via_dispatch(seeded_db, csrf_session):
+    """Handler itself returns no security headers; dispatch_request wraps them.
+    This test documents the boundary so future refactors don't break assumptions."""
+    resp = handle_search({})
+    assert "headers" not in resp or "Content-Security-Policy" not in resp.get("headers", {})

tests/test_handlers_subs.py

@@ -0,0 +1,112 @@
+"""Tests for subscription handlers.
+
+Subscription add validates the destination hash (32-char hex) locally
+before calling `fetch_remote_sites`; browse uses cached remote_pages when
+available and falls back to a live fetch otherwise.
+"""
+from unittest.mock import patch
+
+import tinyweb.handlers as handlers_module
+from tinyweb.db import get_db, return_db
+from tinyweb.handlers import handle_subscription_add, handle_subscription_browse
+
+
+VALID_HASH = "a" * 32
+
+
+def _subscription_count():
+    db = get_db()
+    try:
+        return db.execute("SELECT count(*) FROM subscriptions").fetchone()[0]
+    finally:
+        return_db(db)
+
+
+def test_rejects_empty_dest_hash(temp_db, csrf_session):
+    resp = handle_subscription_add({"dest_hash": [""]})
+    assert "32-character" in resp["body"]
+    assert _subscription_count() == 0
+
+
+def test_rejects_wrong_length(temp_db, csrf_session):
+    resp = handle_subscription_add({"dest_hash": ["abc123"]})
+    assert "32-character" in resp["body"]
+    assert _subscription_count() == 0
+
+
+def test_rejects_non_hex(temp_db, csrf_session):
+    resp = handle_subscription_add({"dest_hash": ["z" * 32]})
+    assert "hex" in resp["body"].lower()
+    assert _subscription_count() == 0
+
+
+def test_rejects_unreachable_peer(temp_db, csrf_session):
+    with patch.object(handlers_module, "fetch_remote_sites") as fetch:
+        fetch.side_effect = ConnectionError("unreachable")
+        resp = handle_subscription_add({"dest_hash": [VALID_HASH]})
+    assert "Could not reach" in resp["body"]
+    assert _subscription_count() == 0
+
+
+def test_rejects_peer_with_sharing_disabled(temp_db, csrf_session):
+    with patch.object(handlers_module, "fetch_remote_sites") as fetch:
+        fetch.side_effect = PermissionError("sharing disabled")
+        resp = handle_subscription_add({"dest_hash": [VALID_HASH]})
+    assert "sharing disabled" in resp["body"]
+    assert _subscription_count() == 0
+
+
+def test_successful_add_records_subscription(temp_db, csrf_session):
+    with patch.object(handlers_module, "fetch_remote_sites") as fetch:
+        fetch.return_value = {"name": "alice", "sites": []}
+        resp = handle_subscription_add({"dest_hash": [VALID_HASH]})
+    assert "Subscribed to alice" in resp["body"]
+    assert _subscription_count() == 1
+
+
+def test_dest_hash_strips_angle_brackets(temp_db, csrf_session):
+    """Users often paste hashes as `<aaa...>` from RNS log output; strip them."""
+    with patch.object(handlers_module, "fetch_remote_sites") as fetch:
+        fetch.return_value = {"name": "bob", "sites": []}
+        resp = handle_subscription_add({"dest_hash": [f"<{VALID_HASH}>"]})
+    assert _subscription_count() == 1
+
+
+def test_browse_unknown_subscription_is_404(temp_db, csrf_session):
+    resp = handle_subscription_browse(99999)
+    assert resp["status"] == 404
+
+
+def test_browse_marks_already_indexed_urls(seeded_db, csrf_session):
+    # Insert a subscription + some remote pages (one duplicate of local, one new).
+    db = get_db()
+    try:
+        db.execute(
+            "INSERT INTO subscriptions (dest_hash, name) VALUES (?, ?)",
+            (VALID_HASH, "alice"),
+        )
+        sub_id = db.execute("SELECT id FROM subscriptions").fetchone()["id"]
+        db.execute(
+            "INSERT INTO remote_pages (subscription_id, url, title, note, tags) "
+            "VALUES (?, ?, ?, ?, ?)",
+            (sub_id, "https://example.com/rust-intro", "Alice rust pick", "", ""),
+        )
+        db.execute(
+            "INSERT INTO remote_pages (subscription_id, url, title, note, tags) "
+            "VALUES (?, ?, ?, ?, ?)",
+            (sub_id, "https://new.example.com/shiny", "Shiny New Link", "note", "tag1"),
+        )
+        db.commit()
+    finally:
+        return_db(db)
+
+    resp = handle_subscription_browse(sub_id)
+    body = resp["body"]
+    assert resp["status"] == 200
+    assert "already indexed" in body
+    # The duplicate URL should appear in the "already indexed" section.
+    assert "Alice rust pick" in body
+    # The new URL should be in the selectable section.
+    assert "Shiny New Link" in body
+    # Count summary: "2 site(s) available, 1 new"
+    assert "1 new" in body

tests/test_handlers_tags.py

@@ -0,0 +1,101 @@
+"""Tests for tag helpers and the tag browse handler.
+
+Tags are stored via a join table, so orphaned rows in `tags` can accumulate
+if `_cleanup_orphaned_tags` isn't called after deletion/retagging. Tag
+counts shown in the UI rely on this being right.
+"""
+from tinyweb.db import get_db, return_db
+from tinyweb.handlers import (
+    _cleanup_orphaned_tags,
+    _get_page_tags,
+    _set_page_tags,
+    handle_tag_browse,
+    handle_tags,
+)
+
+
+def _page_id(url):
+    db = get_db()
+    try:
+        row = db.execute("SELECT id FROM pages WHERE url = ?", (url,)).fetchone()
+        return row["id"] if row else None
+    finally:
+        return_db(db)
+
+
+def _tag_names():
+    db = get_db()
+    try:
+        return {r["name"] for r in db.execute("SELECT name FROM tags").fetchall()}
+    finally:
+        return_db(db)
+
+
+def test_get_page_tags_returns_sorted_names(seeded_db):
+    pid = _page_id("https://example.com/rust-intro")
+    tags = _get_page_tags(pid)
+    assert tags == sorted(tags)  # alphabetical
+    assert "rust" in tags
+    assert "public" in tags
+
+
+def test_set_page_tags_replaces_existing(seeded_db):
+    pid = _page_id("https://example.com/rust-intro")
+    db = get_db()
+    try:
+        _set_page_tags(pid, "brand, new, tags", db)
+        db.commit()
+    finally:
+        return_db(db)
+    current = _get_page_tags(pid)
+    assert current == ["brand", "new", "tags"]
+
+
+def test_set_page_tags_splits_on_comma_and_lowercases(seeded_db):
+    pid = _page_id("https://example.com/python-tips")
+    db = get_db()
+    try:
+        _set_page_tags(pid, "Foo, BAR, baz", db)
+        db.commit()
+    finally:
+        return_db(db)
+    assert set(_get_page_tags(pid)) == {"foo", "bar", "baz"}
+
+
+def test_cleanup_orphaned_tags_removes_unreferenced(seeded_db):
+    # Clear all tags on one page; previously-unique tags become orphans.
+    pid = _page_id("https://example.com/rust-intro")
+    db = get_db()
+    try:
+        _set_page_tags(pid, "", db)  # empty string = no tags
+        # `rust` was only on the rust-intro page; `public` is also on mesh.
+        _cleanup_orphaned_tags(db)
+        db.commit()
+    finally:
+        return_db(db)
+    names = _tag_names()
+    assert "rust" not in names  # pruned
+    assert "public" in names  # still on mesh
+
+
+def test_handle_tag_browse_filters_by_tag(seeded_db, csrf_session):
+    resp = handle_tag_browse("rust", {})
+    assert resp["status"] == 200
+    body = resp["body"]
+    assert "Rust Intro" in body
+    assert "Python Tips" not in body
+    assert "Why OCaml" not in body
+
+
+def test_handle_tag_browse_unknown_tag_is_graceful(seeded_db, csrf_session):
+    resp = handle_tag_browse("no-such-tag", {})
+    # Should render a valid page with zero results, not error.
+    assert resp["status"] == 200
+
+
+def test_handle_tags_lists_all_tags_with_counts(seeded_db, csrf_session):
+    resp = handle_tags()
+    assert resp["status"] == 200
+    body = resp["body"]
+    for tag in ("rust", "python", "ocaml", "mesh", "public", "private"):
+        assert tag in body

tests/test_link_extraction.py

@@ -0,0 +1,138 @@
+"""Tests for link extraction inside `fetch_page`.
+
+Link extraction powers the "trusted sites" fallback on empty searches and
+feeds the `links` table. Rules: same-domain only, skip binary extensions,
+skip Wikipedia special pages, resolve relatives via urljoin.
+"""
+from unittest.mock import patch
+
+from conftest import patch_dns_ok
+import tinyweb.db as db_module
+
+
+class FakeResponse:
+    def __init__(self, text, status_code=200):
+        self.text = text
+        self.status_code = status_code
+        self.is_redirect = False
+        self.headers = {}
+
+    def raise_for_status(self):
+        if self.status_code >= 400:
+            raise Exception(f"status {self.status_code}")
+
+
+def _fetch_with_html(monkeypatch, url, html):
+    """Invoke fetch_page against `url` with `html` as the mocked response body."""
+    patch_dns_ok(monkeypatch)
+    with patch.object(db_module, "requests") as mock_requests:
+        mock_requests.get.return_value = FakeResponse(html)
+        return db_module.fetch_page(url)
+
+
+def test_only_same_domain_links_kept(monkeypatch):
+    html = """
+    <html><body>
+      <a href="https://example.com/a">same</a>
+      <a href="https://other.com/b">cross</a>
+      <a href="https://sub.example.com/c">subdomain</a>
+    </body></html>
+    """
+    _, _, links, _ = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    urls = [u for u, _label in links]
+    assert "https://example.com/a" in urls
+    assert "https://other.com/b" not in urls
+    assert "https://sub.example.com/c" not in urls
+
+
+def test_binary_extensions_skipped(monkeypatch):
+    html = """
+    <html><body>
+      <a href="/real-page">keep</a>
+      <a href="/image.png">skip</a>
+      <a href="/doc.pdf">skip</a>
+      <a href="/archive.zip">skip</a>
+      <a href="/song.mp3">skip</a>
+      <a href="/styles.css">skip</a>
+    </body></html>
+    """
+    _, _, links, _ = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    urls = [u for u, _label in links]
+    assert "https://example.com/real-page" in urls
+    for ext in (".png", ".pdf", ".zip", ".mp3", ".css"):
+        assert not any(u.endswith(ext) for u in urls), f"{ext} leaked through"
+
+
+def test_wikipedia_special_pages_skipped(monkeypatch):
+    html = """
+    <html><body>
+      <a href="/wiki/Main_Page">keep</a>
+      <a href="/wiki/Special:Random">skip</a>
+      <a href="/wiki/Talk:Foo">skip</a>
+      <a href="/wiki/User:Jimbo">skip</a>
+      <a href="/wiki/Category:Bar">skip</a>
+    </body></html>
+    """
+    _, _, links, _ = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    urls = [u for u, _label in links]
+    assert "https://example.com/wiki/Main_Page" in urls
+    for skip in ("Special:Random", "Talk:Foo", "User:Jimbo", "Category:Bar"):
+        assert not any(skip in u for u in urls), f"wiki {skip!r} leaked"
+
+
+def test_relative_urls_resolved(monkeypatch):
+    html = """<html><body><a href="/relative/path">r</a></body></html>"""
+    _, _, links, _ = _fetch_with_html(monkeypatch, "https://example.com/start", html)
+    urls = [u for u, _label in links]
+    assert "https://example.com/relative/path" in urls
+
+
+def test_fragment_stripped_from_extracted_links(monkeypatch):
+    html = """<html><body><a href="/page#section">r</a></body></html>"""
+    _, _, links, _ = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    urls = [u for u, _label in links]
+    assert "https://example.com/page" in urls
+    assert not any("#" in u for u in urls)
+
+
+def test_duplicate_links_deduped(monkeypatch):
+    html = """
+    <html><body>
+      <a href="/a">first</a>
+      <a href="/a">second</a>
+      <a href="/a">third</a>
+    </body></html>
+    """
+    _, _, links, _ = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    urls = [u for u, _label in links]
+    assert urls.count("https://example.com/a") == 1
+
+
+def test_label_truncated_to_200(monkeypatch):
+    long_text = "x" * 500
+    html = f'<html><body><a href="/p">{long_text}</a></body></html>'
+    _, _, links, _ = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    assert len(links) == 1
+    _, label = links[0]
+    assert len(label) <= 200
+
+
+def test_meta_description_extracted(monkeypatch):
+    html = """
+    <html><head>
+      <meta name="description" content="the real description">
+    </head><body><p>body content</p></body></html>
+    """
+    title, body, links, meta = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    assert meta == "the real description"
+
+
+def test_og_description_fallback(monkeypatch):
+    """When there's no <meta name=description>, og:description wins."""
+    html = """
+    <html><head>
+      <meta property="og:description" content="open graph fallback">
+    </head><body><p>body</p></body></html>
+    """
+    _, _, _, meta = _fetch_with_html(monkeypatch, "https://example.com/", html)
+    assert meta == "open graph fallback"

tests/test_pagination.py

@@ -0,0 +1,58 @@
+"""Tests for `_paginate` and `_page_nav`."""
+from tinyweb.handlers import _paginate, _page_nav, PER_PAGE
+
+
+def test_paginate_default_is_one():
+    assert _paginate({}) == 1
+
+
+def test_paginate_reads_query_string():
+    assert _paginate({"p": ["3"]}) == 3
+
+
+def test_paginate_clamps_to_one():
+    assert _paginate({"p": ["0"]}) == 1
+    assert _paginate({"p": ["-5"]}) == 1
+
+
+def test_paginate_handles_bad_input():
+    assert _paginate({"p": ["not-a-number"]}) == 1
+    assert _paginate({"p": []}) == 1
+
+
+def test_paginate_custom_key():
+    assert _paginate({"batch": ["7"]}, key="batch") == 7
+
+
+def test_page_nav_empty_when_single_page():
+    assert _page_nav(1, PER_PAGE, "/?q=foo") == ""
+    assert _page_nav(1, 0, "/?q=foo") == ""
+
+
+def test_page_nav_shows_next_on_first_page():
+    out = _page_nav(1, PER_PAGE * 3, "/?q=foo")
+    assert "next" in out
+    assert "prev" not in out
+    assert "page 1 of 3" in out
+
+
+def test_page_nav_shows_both_in_middle():
+    out = _page_nav(2, PER_PAGE * 3, "/?q=foo")
+    assert "next" in out
+    assert "prev" in out
+
+
+def test_page_nav_shows_prev_on_last_page():
+    out = _page_nav(3, PER_PAGE * 3, "/?q=foo")
+    assert "next" not in out
+    assert "prev" in out
+    assert "page 3 of 3" in out
+
+
+def test_page_nav_handles_query_string_separator():
+    # when base_url already has ?, pagination links must use &
+    out = _page_nav(1, PER_PAGE * 2, "/?q=foo")
+    assert "&p=2" in out
+    # when base_url has no ?, pagination links use ?
+    out = _page_nav(1, PER_PAGE * 2, "/pages")
+    assert "?p=2" in out

tests/test_regressions.py

@@ -0,0 +1,107 @@
+"""Aggregator of regression tests tied to specific bug-fix commits.
+
+Each test here guards against a specific bug that was once shipped. Running
+just this file gives a one-line-per-bug audit:
+
+    pytest tests/test_regressions.py -v
+
+The test bodies are intentionally small; for the exhaustive behavior of each
+module, see the topical test files (test_fts_sanitizer.py, test_url_cleanup.py,
+etc.). This file's job is to make the bug catalog scannable.
+"""
+import socket
+from unittest.mock import patch
+
+import pytest
+
+from tinyweb import app as app_module
+import tinyweb.db as db_module
+import tinyweb.handlers as handlers_module
+from conftest import patch_dns_fail, patch_dns_ok
+from tinyweb.db import clean_url
+from tinyweb.handlers import _sanitize_fts_query, handle_bulk_action
+
+
+def test_6ffd38d_clean_url_preserves_www_when_bare_domain_fails(monkeypatch):
+    """6ffd38d: `clean_url` used to strip `www.` unconditionally; for sites that
+    only serve at `www.`, this produced unreachable clean URLs."""
+    patch_dns_fail(monkeypatch)
+    assert clean_url("https://www.example.com/page") == "https://www.example.com/page"
+
+
+def test_1bc695f_fts_sanitizer_strips_colon():
+    """1bc695f: FTS5 colon is a column filter — must not appear in sanitized output."""
+    assert ":" not in _sanitize_fts_query("title:secret body:exposed")
+
+
+@pytest.mark.parametrize("op", ["AND", "OR", "NOT", "NEAR"])
+def test_1bc695f_fts_sanitizer_drops_operator_words(op):
+    """1bc695f: operator words (AND/OR/NOT/NEAR) would be interpreted as FTS5
+    operators if they landed on the unquoted last token."""
+    out = _sanitize_fts_query(f"foo {op} bar")
+    # operator itself should not appear in the output
+    tokens = out.replace('"', '').split()
+    assert op not in [t.rstrip("*") for t in tokens]
+
+
+def test_1bc695f_gateway_rejects_oversize_body():
+    """1bc695f: 16 MiB body-size cap prevents memory-exhaustion DoS."""
+    from tests.test_gateway_limits import FakeGatewayHandler
+    from tinyweb.gateway import MAX_BODY_SIZE
+    h = FakeGatewayHandler(
+        path="/add", method="POST",
+        headers={"Content-Length": str(MAX_BODY_SIZE + 1)},
+    )
+    h._forward("POST")
+    assert h._captured["error"] and h._captured["error"][0] == 413
+
+
+def test_1bc695f_mesh_rejects_non_whitelisted_paths():
+    """1bc695f: Reticulum callers are limited to GET /api/sites; CSRF cannot
+    authenticate mesh callers."""
+    resp = app_module.rns_request_handler(
+        path="/tinyweb",
+        data={"method": "POST", "path": "/add", "query": {}, "body": {}, "gateway_host": ""},
+        request_id="x", link_id="y", remote_identity=None, requested_at=0,
+    )
+    assert resp["status"] == 403
+
+
+def test_1bc695f_pool_returns_clean_connection(temp_db, monkeypatch):
+    """1bc695f: uncommitted transactions on a pooled connection used to leak
+    into the next consumer."""
+    from tinyweb.db import get_db, return_db
+    db = get_db()
+    db.execute(
+        "INSERT INTO pages (url, title, body) VALUES (?, ?, ?)",
+        ("https://leak.example.com/", "should not persist", "body"),
+    )
+    return_db(db)  # no commit
+    db2 = get_db()
+    try:
+        urls = {r["url"] for r in db2.execute("SELECT url FROM pages").fetchall()}
+    finally:
+        return_db(db2)
+    assert "https://leak.example.com/" not in urls
+
+
+def test_8dffd8c_bulk_delete_requires_confirmation(seeded_db, csrf_session):
+    """8dffd8c: bulk delete without confirmed=1 must render a confirm page
+    instead of deleting — the JS confirm on /pages is a first-line filter only."""
+    from tinyweb.db import get_db, return_db
+    db = get_db()
+    try:
+        pid = db.execute("SELECT id FROM pages LIMIT 1").fetchone()["id"]
+        count_before = db.execute("SELECT count(*) FROM pages").fetchone()[0]
+    finally:
+        return_db(db)
+
+    resp = handle_bulk_action({"ids": [str(pid)], "action": ["delete"]})
+    assert "confirm delete" in resp["body"].lower()
+
+    db = get_db()
+    try:
+        count_after = db.execute("SELECT count(*) FROM pages").fetchone()[0]
+    finally:
+        return_db(db)
+    assert count_before == count_after, "bulk delete ran without confirmation"

tests/test_sharing_logic.py

@@ -0,0 +1,38 @@
+"""Tests for `_page_is_shared`.
+
+This function decides whether a page is exposed over Reticulum to
+subscribers. Getting it wrong means either a privacy leak or silently
+hiding pages the user meant to share — both are worth a regression net.
+"""
+import pytest
+
+from tinyweb.handlers import _page_is_shared
+
+
+@pytest.mark.parametrize("mode", ["exclude_private", "require_public"])
+def test_private_tag_always_excludes(mode):
+    """`private` tag overrides every mode — the most important invariant."""
+    assert _page_is_shared(["private"], mode) is False
+    assert _page_is_shared(["public", "private"], mode) is False
+
+
+def test_exclude_private_defaults_to_shared():
+    assert _page_is_shared([], "exclude_private") is True
+    assert _page_is_shared(["random-tag"], "exclude_private") is True
+
+
+def test_require_public_needs_public_tag():
+    assert _page_is_shared([], "require_public") is False
+    assert _page_is_shared(["rust"], "require_public") is False
+    assert _page_is_shared(["public"], "require_public") is True
+
+
+def test_require_public_still_vetoes_private():
+    # public AND private → private wins.
+    assert _page_is_shared(["public", "private"], "require_public") is False
+
+
+def test_unknown_mode_treated_as_exclude_private():
+    """The default mode is 'exclude_private'; unknown modes fall through to it."""
+    assert _page_is_shared([], "totally-bogus-mode") is True
+    assert _page_is_shared(["private"], "totally-bogus-mode") is False

tests/test_ssrf.py

@@ -0,0 +1,64 @@
+"""Tests for `_validate_url_target` — SSRF prevention.
+
+Any URL the app fetches must resolve to a public IP; private/internal/
+loopback addresses must be rejected so attacker-controlled URLs cannot
+reach internal services via our HTTP client.
+"""
+import socket
+from unittest.mock import patch
+
+import pytest
+
+from tinyweb.db import _validate_url_target
+
+
+def _mock_getaddrinfo(address):
+    """Return a function suitable as a socket.getaddrinfo replacement."""
+    def f(host, port, *args, **kwargs):
+        family = socket.AF_INET6 if ":" in address else socket.AF_INET
+        return [(family, socket.SOCK_STREAM, 0, "", (address, port or 80))]
+    return f
+
+
+@pytest.mark.parametrize("blocked_ip", [
+    "127.0.0.1",
+    "127.1.2.3",
+    "10.0.0.1",
+    "10.255.255.255",
+    "172.16.0.1",
+    "172.31.255.255",
+    "192.168.0.1",
+    "192.168.255.255",
+    "169.254.169.254",
+    "0.0.0.0",
+    "::1",
+    "fc00::1",
+    "fe80::1",
+])
+def test_blocks_private_and_loopback(monkeypatch, blocked_ip):
+    monkeypatch.setattr(socket, "getaddrinfo", _mock_getaddrinfo(blocked_ip))
+    with pytest.raises(ValueError, match="blocked"):
+        _validate_url_target("https://evil.example.com/internal")
+
+
+def test_allows_public_ipv4(monkeypatch):
+    monkeypatch.setattr(socket, "getaddrinfo", _mock_getaddrinfo("8.8.8.8"))
+    _validate_url_target("https://dns.example.com/")  # does not raise
+
+
+def test_allows_public_ipv6(monkeypatch):
+    monkeypatch.setattr(socket, "getaddrinfo", _mock_getaddrinfo("2001:4860:4860::8888"))
+    _validate_url_target("https://v6.example.com/")  # does not raise
+
+
+def test_rejects_unresolvable_hostname(monkeypatch):
+    def boom(*args, **kwargs):
+        raise socket.gaierror("no such host")
+    monkeypatch.setattr(socket, "getaddrinfo", boom)
+    with pytest.raises(ValueError, match="Cannot resolve"):
+        _validate_url_target("https://does-not-exist.example.com/")
+
+
+def test_rejects_missing_hostname():
+    with pytest.raises(ValueError, match="No hostname"):
+        _validate_url_target("http:///path-only")

tests/test_url_cleanup.py

@@ -0,0 +1,101 @@
+"""Tests for `clean_url` — URL normalization and tracking-param stripping.
+
+Clean URLs are the deduplication key in the pages table, so any change to
+this function can silently cause duplicate rows or mask legitimate saves.
+"""
+import pytest
+
+from conftest import patch_dns_ok, patch_dns_fail
+from tinyweb.db import clean_url, TRACKING_PARAMS
+
+
+def test_strips_fragment(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://example.com/page#section") == "https://example.com/page"
+
+
+def test_prefers_https(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    assert clean_url("http://example.com/page") == "https://example.com/page"
+
+
+def test_lowercases_hostname(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://EXAMPLE.COM/page") == "https://example.com/page"
+
+
+def test_preserves_path_case(monkeypatch):
+    """Paths are case-sensitive and should not be lowercased."""
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://example.com/Foo/Bar") == "https://example.com/Foo/Bar"
+
+
+def test_strips_default_https_port(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://example.com:443/page") == "https://example.com/page"
+
+
+@pytest.mark.xfail(reason="clean_url upgrades http->https before the port-default check, "
+                          "so port 80 is not stripped. Minor dedup bug — harmless but worth fixing.")
+def test_strips_http_port_80(monkeypatch):
+    """Expected: http://foo:80 → https://foo (both scheme-upgrade and port-strip).
+
+    Currently fails because scheme is upgraded to https *before* the port check,
+    so `scheme == "http" and port == 80` is never true by the time the check runs.
+    """
+    patch_dns_ok(monkeypatch)
+    assert clean_url("http://example.com:80/page") == "https://example.com/page"
+
+
+def test_preserves_non_default_port(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://example.com:8443/page") == "https://example.com:8443/page"
+
+
+def test_strips_trailing_slash(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://example.com/page/") == "https://example.com/page"
+
+
+def test_root_slash_preserved(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://example.com/") == "https://example.com/"
+
+
+@pytest.mark.parametrize("param", sorted(TRACKING_PARAMS))
+def test_tracking_params_stripped(monkeypatch, param):
+    patch_dns_ok(monkeypatch)
+    result = clean_url(f"https://example.com/page?{param}=value&keep=yes")
+    assert param not in result
+    assert "keep=yes" in result
+
+
+def test_strips_www_when_nonwww_resolves(monkeypatch):
+    """Standard case: strip `www.` prefix to canonicalize."""
+    patch_dns_ok(monkeypatch)
+    assert clean_url("https://www.example.com/page") == "https://example.com/page"
+
+
+def test_preserves_www_when_nonwww_does_not_resolve(monkeypatch):
+    """Regression for 6ffd38d.
+
+    Some sites only serve their content at `www.domain.tld`; the bare domain
+    doesn't resolve. Stripping `www.` in that case produced a URL that we could
+    never actually fetch or dedupe against the real one.
+    """
+    patch_dns_fail(monkeypatch)
+    assert clean_url("https://www.example.com/page") == "https://www.example.com/page"
+
+
+def test_query_params_sorted_for_stable_ordering(monkeypatch):
+    """Same URL with different param orderings should produce the same clean URL."""
+    patch_dns_ok(monkeypatch)
+    a = clean_url("https://example.com/page?b=2&a=1")
+    b = clean_url("https://example.com/page?a=1&b=2")
+    assert a == b
+
+
+def test_path_and_query_preserved_through_cleanup(monkeypatch):
+    patch_dns_ok(monkeypatch)
+    result = clean_url("https://example.com/path/to/page?id=42&utm_source=twitter")
+    assert result == "https://example.com/path/to/page?id=42"

themes/default.html

@@ -0,0 +1,307 @@
+<!DOCTYPE html>
+<html>
+    <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <meta name="referrer" content="no-referrer">
+    <meta http-equiv="x-dns-prefetch-control" content="off">
+    <style>
+      * { margin: 0; padding: 0; box-sizing: border-box; }
+      html, body { min-height: 100vh; }
+      body {
+        font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Noto Sans', Helvetica, Arial, sans-serif;
+        font-size: 18px;
+        line-height: 1.6;
+        color: #444;
+        background: #fff;
+        padding: 60px 40px;
+        display: flex;
+        flex-direction: column;
+        align-items: center;
+      }
+      a { color: #444; text-decoration: none; cursor: pointer; }
+      a:hover { color: #222; }
+      input, textarea {
+        font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Noto Sans', Helvetica, Arial, sans-serif;
+      }
+      input[type="text"],
+      input[type="url"],
+      input[type="search"],
+      input:not([type]),
+      textarea, select {
+        border: 1px solid #ccc;
+        padding: 8px 12px;
+        font-size: 15px;
+        border-radius: 0;
+        background: #fff;
+        color: #444;
+      }
+      input:focus, textarea:focus {
+        outline: none;
+        border-color: #999;
+      }
+      button, input[type="submit"] {
+        border: 1px solid #ccc;
+        padding: 8px 18px;
+        font-size: 13px;
+        text-transform: uppercase;
+        background: #fff;
+        color: #444;
+        cursor: pointer;
+        border-radius: 0;
+        transition: background 0.2s;
+      }
+      button:hover, input[type="submit"]:hover {
+        background: #f5f5f5;
+      }
+      .shell {
+        width: 100%;
+        max-width: 650px;
+      }
+      nav {
+        display: flex;
+        align-items: baseline;
+        justify-content: space-between;
+        margin-bottom: 30px;
+        flex-wrap: wrap;
+        gap: 12px;
+      }
+      nav .site {
+        font-size: 28px;
+        font-weight: bold;
+        color: #222;
+        text-decoration: none;
+        border-bottom: none;
+      }
+      nav .site:hover { color: #222; }
+      nav .links { display: flex; gap: 8px; flex-wrap: wrap; }
+      nav .links a {
+        font-size: 13px;
+        text-transform: uppercase;
+        padding: 6px 14px;
+        border: 1px solid #ccc;
+        background: #fff;
+        color: #444;
+        text-decoration: none;
+        transition: background 0.2s;
+      }
+      nav .links a:hover {
+        background: #f5f5f5;
+        color: #444;
+      }
+      .content { width: 100%; animation: fadeIn 0.3s ease; }
+      @keyframes fadeIn { from { opacity: 0; } to { opacity: 1; } }
+      h1 {
+        font-size: 32px;
+        font-weight: bold;
+        color: #222;
+        margin-bottom: 20px;
+        line-height: 1.2;
+      }
+      h1 a { color: #222; text-decoration: none; }
+      h2 {
+        font-size: 22px;
+        font-weight: bold;
+        color: #222;
+        margin: 24px 0 12px;
+      }
+      p { margin: 12px 0; color: #444; }
+      a { color: #444; text-decoration: none; border-bottom: 1px solid #ddd; }
+      a:hover { color: #222; border-bottom-color: #999; }
+      em { color: #666; }
+      .result {
+        padding: 20px 0;
+        border-bottom: 1px solid #eee;
+      }
+      .result:last-child { border-bottom: none; }
+      .result > a:first-child {
+        font-size: 18px;
+        color: #222;
+        border-bottom: none;
+      }
+      .result > a:first-child:hover { color: #222; }
+      .note { margin-top: 4px; font-size: 15px; color: #666; }
+      .meta {
+        font-size: 13px;
+        color: #999;
+      }
+      .pagination {
+        font-size: 13px;
+        color: #999;
+        margin: 24px 0;
+        text-align: center;
+      }
+      .pagination a {
+        color: #444;
+        border-bottom: 1px solid #ccc;
+      }
+      .pagination a:hover { color: #222; }
+      .success {
+        color: #444;
+        background: #f5f5f5;
+        border: 1px solid #ddd;
+        padding: 10px 16px;
+        font-size: 15px;
+      }
+      .tags { margin-top: 4px; }
+      .tag, .tags a {
+        font-size: 12px;
+        color: #999;
+        border: 1px solid #ddd;
+        padding: 2px 8px;
+        margin-right: 4px;
+        text-decoration: none;
+      }
+      .tag:hover, .tags a:hover {
+        color: #444;
+        border-color: #ccc;
+      }
+      details {
+        margin: 16px 0;
+        border: 1px solid #eee;
+        padding: 12px 16px;
+        background: #fafafa;
+      }
+      summary { font-size: 15px; color: #666; cursor: pointer; }
+      summary:hover { color: #444; }
+      details ul { margin-top: 8px; padding-left: 20px; }
+      details li { margin: 6px 0; font-size: 15px; }
+      ul, ol { padding-left: 20px; margin: 8px 0; }
+      li { margin: 6px 0; color: #444; }
+      li a { border-bottom: none; }
+      pre {
+        font-family: ui-monospace, 'SF Mono', 'Cascadia Code', 'Segoe UI Mono', Menlo, Consolas, monospace;
+        font-size: 14px;
+        background: #f5f5f5;
+        border: 1px solid #eee;
+        padding: 16px;
+        overflow-x: auto;
+        color: #444;
+        margin: 12px 0;
+      }
+      code {
+        font-family: ui-monospace, 'SF Mono', 'Cascadia Code', 'Segoe UI Mono', Menlo, Consolas, monospace;
+        font-size: 14px;
+        background: #f5f5f5;
+        padding: 2px 6px;
+        color: #444;
+      }
+      textarea {
+        font-family: ui-monospace, 'SF Mono', 'Cascadia Code', 'Segoe UI Mono', Menlo, Consolas, monospace;
+        font-size: 14px;
+        line-height: 1.6;
+        resize: vertical;
+        width: 100%;
+        border: 1px solid #ccc;
+        padding: 10px 12px;
+        background: #fff;
+        color: #444;
+      }
+      table { width: 100%; border-collapse: collapse; margin: 16px 0; }
+      th {
+        text-align: left;
+        font-size: 12px;
+        font-weight: 600;
+        color: #999;
+        text-transform: uppercase;
+        letter-spacing: 0.05em;
+        padding: 8px 12px;
+        border-bottom: 1px solid #eee;
+      }
+      td {
+        padding: 8px 12px;
+        border-bottom: 1px solid #f5f5f5;
+        font-size: 15px;
+      }
+      label { color: #666; }
+      input[type="checkbox"] { accent-color: #444; }
+      .forum-form input, .forum-form button, .forum-toolbar input { border-radius: 0; }
+      .forum-actions a, a.forum-action, a.forum-action-inline {
+        border: 1px solid #ccc; padding: 6px 14px; text-transform: uppercase; font-size: 13px;
+      }
+      .forum-actions a:hover, a.forum-action:hover, a.forum-action-inline:hover {
+        background: #f5f5f5;
+      }
+      a.forum-action-inline { text-transform: none; font-size: 13px; padding: 2px 6px; border: none; }
+      .section { margin: 1.5rem 0; }
+      .section-title { font-weight: 600; margin-bottom: 0.3rem; }
+      .section-desc { font-size: 0.85rem; color: #999; margin-bottom: 0.5rem; }
+      .section ul { margin: 0.3rem 0; }
+      .forum-form input, .forum-form textarea, .forum-form button { margin-bottom: 8px; }
+      .forum-form small { display: block; margin-bottom: 6px; }
+      .forum-form label { display: block; margin-bottom: 6px; }
+      .forum-form + .forum-form { margin-top: 1rem; }
+      .forum-form + .section-title { margin-top: 1rem; }
+      .section-desc + .forum-form { margin-top: 0.8rem; }
+      ul + .forum-form { margin-top: 1rem; }
+      .checkbox-label { display: flex; align-items: center; gap: 6px; margin-bottom: 8px; }
+      .forum-status { font-size: 0.82rem; color: #999; margin: 0 0 0.8rem 0; }
+      .forum-status span { margin-right: 1.2rem; }
+      .forum-nav { margin: 1rem 0; }
+      hr { border: none; border-top: 1px solid #eee; margin: 16px 0; }
+
+      small {
+        font-size: 13px;
+        color: #999;
+      }
+      footer {
+        width: 100%;
+        margin-top: 40px;
+        padding-top: 16px;
+        border-top: 1px solid #eee;
+        text-align: center;
+        color: #999;
+        font-size: 13px;
+      }
+      footer .clock {
+        font-family: ui-monospace, 'SF Mono', 'Cascadia Code', 'Segoe UI Mono', Menlo, Consolas, monospace;
+        font-size: 12px;
+        color: #ccc;
+        margin-top: 4px;
+      }
+      ::selection { background: #222; color: #fff; }
+      @media (max-width: 600px) {
+        body { padding: 30px 20px; font-size: 16px; }
+        h1 { font-size: 26px; }
+        nav { flex-direction: column; align-items: flex-start; }
+        nav .links { gap: 6px; }
+        nav .links a { padding: 5px 10px; font-size: 12px; }
+      }
+    </style>
+    </head>
+    <body>
+    <div class="shell">
+      <nav>
+        <a class="site" href="/">{{site_name}}</a>
+        <div class="links">
+          <a href="/pages">browse</a>
+          <a href="/tags">tags</a>
+          <a href="/subscriptions">network</a>
+          {{forum_link}}
+          <a href="/style">customize</a>
+          <a href="/about">about</a>
+        </div>
+      </nav>
+      <div class="content">
+        {{content}}
+      </div>
+      <footer>
+        <div>curated by hand &middot; shared over mesh</div>
+        <div class="clock" id="clock"></div>
+      </footer>
+    </div>
+    <script>
+    (function() {
+      var d = document.getElementById('clock');
+      if (d) {
+        function tick() {
+          var n = new Date();
+          d.textContent = n.toLocaleString();
+        }
+        tick();
+        setInterval(tick, 1000);
+      }
+    })();
+    </script>
+    </body>
+</html>