feat: add declarative site parser engine

This commit is contained in:
Your Name
2026-08-11 00:46:11 +05:00
parent 01eea99c88
commit 2d6fa3c99a
13 changed files with 1202 additions and 244 deletions
+9 -14
View File
@@ -1,6 +1,6 @@
# RAA deployment
Last updated: 2026-08-10.
Last updated: 2026-08-11.
RAA is not a separate code project anymore. The source of truth for code and
documentation is:
@@ -96,20 +96,15 @@ All source records, JSON configs, schedules, errors, raw posts, and media state
remain in the RAA database. The worker is managed by Docker Compose from
`/opt/site-parser` and should be healthy as `site-parser-worker`.
PopularAirsoft source id `72`:
PopularAirsoft source id `72` uses `https://popularairsoft.com/` and the full
v1 config from `site_parser_worker/README.md`. Its RSS is not used for
discovery because it mixes short video pages with full articles. HTML discovery
selects the `The Latest News` cards and the worker then opens every `/news/...`
detail. Verified live 2026-08-11: nine details parsed, zero errors.
- URL: `https://popularairsoft.com/rss.xml`
- Config: `{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50}`
- `access='cloudflare'` deliberately starts the browser/RuCaptcha flow before
fetching the RSS.
- Keep `follow_links` disabled for this feed. Nine current entries have no
body, only title/video, and are intentionally filtered before media upload.
- Cloudflare cookies and browser User-Agent are reused by `media-uploader` for
protected images.
- Admin image previews use authenticated Telegram-backed
`/raw/media/{media_id}` URLs.
- Verified 2026-08-10: source `ok`; raw post `102` is `storage_ready`; four
images are `uploaded`. The AI later rejected the content as non-target.
Cloudflare session state is returned to the main app and is also reused by
`media-uploader` for protected images. Admin previews remain authenticated
Telegram-backed `/raw/media/{media_id}` URLs.
## RAA Categories