feat: add declarative site parser engine
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# N8 Parser: current state
|
||||
|
||||
Last updated: 2026-08-10
|
||||
Last updated: 2026-08-11
|
||||
|
||||
This file is the short handoff state for future Codex threads. Read this first before touching the project, so the whole chat history does not need to be carried forward.
|
||||
|
||||
@@ -107,20 +107,18 @@ and returns normalized JSON.
|
||||
`/raw/media/{media_id}` URLs backed by Telegram storage, not by the original
|
||||
Cloudflare-protected URL.
|
||||
|
||||
Current RAA test source:
|
||||
Site configs now use the versioned declarative v1 contract. The worker supports
|
||||
RSS or HTML discovery, exact per-field selectors with ordered fallbacks,
|
||||
detail-page extraction, media normalization, bounded retries, and per-item
|
||||
errors. A failed detail does not discard successful siblings; the source is
|
||||
marked with a visible partial diagnostic.
|
||||
|
||||
```json
|
||||
{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50}
|
||||
```
|
||||
|
||||
- Source: `Popularairsoft`, `https://popularairsoft.com/rss.xml`, id `72`.
|
||||
- Do not add `follow_links` for the current feed: one of ten entries has a real
|
||||
body and four photos; the other nine are title-plus-video entries and are
|
||||
intentionally skipped by the body-length threshold.
|
||||
- Verified on 2026-08-10: source status `ok`, raw post `102` reached
|
||||
`storage_ready`, and all four photos were uploaded. Its later rejection is
|
||||
an AI non-target decision, not a parser/media failure.
|
||||
- Verified RAA deploy: commit `f866a13a8a83598e26a7a6a3406b8bc1c01c9a80`.
|
||||
PopularAirsoft source id `72` should use `https://popularairsoft.com/`, not its
|
||||
mixed-content RSS feed. The complete config is documented in
|
||||
`site_parser_worker/README.md`. It discovers the `The Latest News` cards and
|
||||
then extracts each `article.news.full` page. Verified live on 2026-08-11: nine
|
||||
items, nine full details, zero errors. Twitch embeds return as
|
||||
`external_video`; YouTube embeds are normalized for the existing downloader.
|
||||
|
||||
LXC 108 operations:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user