lichenblankie/tinyweb
1
0
Fork 0
Code Issues Pull requests Projects 1 Releases 1 Packages Wiki Activity Actions

stripped noscript tags from pages

Browse source 
Lemmy and other JS-heavy sites include noscript fallback text like
"Javascript is disabled" that pollutes the stored body text and
generated snippets/summaries.
This commit is contained in:
lichenblankie 2026-03-27 14:18:54 -07:00
parent 3f8ebdab1d
commit e8915fa381
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
db.py
View file

@ -328,7 +328,7 @@ def fetch_page(url):
if og_tag and og_tag.get("content"):
meta_desc = og_tag["content"].strip()
for tag in soup(["script", "style", "nav", "footer", "header"]):
for tag in soup(["script", "style", "nav", "footer", "header", "noscript"]):
tag.decompose()
title = soup.title.string.strip() if soup.title and soup.title.string else url
body = soup.get_text(separator=" ", strip=True)