feat: add declarative site parser engine

This commit is contained in:
Your Name
2026-08-11 00:46:11 +05:00
parent 01eea99c88
commit 2d6fa3c99a
13 changed files with 1202 additions and 244 deletions
+12 -14
View File
@@ -1,6 +1,6 @@
# N8 Parser: current state
Last updated: 2026-08-10
Last updated: 2026-08-11
This file is the short handoff state for future Codex threads. Read this first before touching the project, so the whole chat history does not need to be carried forward.
@@ -107,20 +107,18 @@ and returns normalized JSON.
`/raw/media/{media_id}` URLs backed by Telegram storage, not by the original
Cloudflare-protected URL.
Current RAA test source:
Site configs now use the versioned declarative v1 contract. The worker supports
RSS or HTML discovery, exact per-field selectors with ordered fallbacks,
detail-page extraction, media normalization, bounded retries, and per-item
errors. A failed detail does not discard successful siblings; the source is
marked with a visible partial diagnostic.
```json
{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50}
```
- Source: `Popularairsoft`, `https://popularairsoft.com/rss.xml`, id `72`.
- Do not add `follow_links` for the current feed: one of ten entries has a real
body and four photos; the other nine are title-plus-video entries and are
intentionally skipped by the body-length threshold.
- Verified on 2026-08-10: source status `ok`, raw post `102` reached
`storage_ready`, and all four photos were uploaded. Its later rejection is
an AI non-target decision, not a parser/media failure.
- Verified RAA deploy: commit `f866a13a8a83598e26a7a6a3406b8bc1c01c9a80`.
PopularAirsoft source id `72` should use `https://popularairsoft.com/`, not its
mixed-content RSS feed. The complete config is documented in
`site_parser_worker/README.md`. It discovers the `The Latest News` cards and
then extracts each `article.news.full` page. Verified live on 2026-08-11: nine
items, nine full details, zero errors. Twitch embeds return as
`external_video`; YouTube embeds are normalized for the existing downloader.
LXC 108 operations: