From 01eea99c88412d1aea32b653a534cd5b09f40351 Mon Sep 17 00:00:00 2001 From: Your Name Date: Mon, 10 Aug 2026 23:28:47 +0500 Subject: [PATCH] docs: document external site parser --- PROJECT.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++- RAA.md | 30 +++++++++++++++++++++++++++- STATE.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 142 insertions(+), 3 deletions(-) diff --git a/PROJECT.md b/PROJECT.md index 47b9bb9..37b6d34 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -196,7 +196,8 @@ Источники для парсинга. -Сейчас реально используется только `platform='vk'`, но схема заложена под другие площадки. +Используются `platform='vk'` и `platform='site'`. Для сайта конфигурация +конкретного адаптера хранится вместе с источником в `settings_json`. Важные поля: @@ -215,9 +216,14 @@ - `parse_from` - `priority` - `archived_at` +- `settings_json` для `platform='site'` `tag` нужен для будущих хэштегов и финального оформления. +Для сайта отсутствие или ошибка JSON-конфига блокирует запуск с видимой +ошибкой источника. Конфиг задаёт способ получения именно этого сайта; общее +расписание и секреты остаются в `app_settings`. + ### 5.4. `raw_posts` Главная таблица жизненного цикла поста. @@ -558,6 +564,37 @@ Retention: записи старше 30 дней удаляются при ст 8. Сохраняет raw post и media. 9. Создаёт job `media.storage.copy`. +### 7.1.1. Источники сайтов + +Тот же scheduling loop обрабатывает `platform='site'`, но сетевую работу +делегирует внешнему stateless-модулю Site Parser. Основная коробка хранит БД, +источники, конфиги, расписание, статусы и очередь. Внешний модуль получает один +запрос и возвращает унифицированный JSON с текстом, исходной ссылкой, датой и +медиа. Готовые источники запускаются строго последовательно. + +Настройки основной коробки: + +- `site_parser_url` +- `site_parser_token` +- `site_parser_rucaptcha_token` +- `site_parser_timeout_sec` +- `site_parser_interval_minutes` +- `site_parser_min_text_length` + +Персональный `min_text_length` в JSON источника переопределяет общий порог. +Проверяется только тело материала: заголовок не помогает пройти фильтр. +Записи без достаточного текста, включая video-only, не сохраняются и не +попадают в media uploader. + +`access='cloudflare'` включает Playwright и RuCaptcha для конкретного +источника; обычные RSS и сайты идут без браузера. Полученные `cf_clearance` и +точный User-Agent браузера сохраняются в runtime state источника и повторно +используются media uploader при скачивании защищённых картинок. + +Внешний модуль работает в LXC `108` (`192.168.1.113:8080`) через Docker +Compose `/opt/site-parser/docker-compose.yml`. Репозиторный compose находится +в `site_parser_worker/docker-compose.yml`. + ### 7.2. Политика мусорных постов Настройки: @@ -1091,6 +1128,24 @@ Worker учитывает: - active - parse_from +Для сайта важны: + +- platform `site` +- name +- url +- active +- обязательный JSON-конфиг конкретного источника + +Пример рабочего PopularAirsoft: + +```json +{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50} +``` + +Ошибки конфигурации, внешнего worker или сайта показываются в статусе +источника. После загрузки защищённые изображения отображаются в raw/editor по +авторизованному маршруту `/raw/media/{media_id}` из Telegram storage. + ### 12.3. Raw Raw-страница показывает: diff --git a/RAA.md b/RAA.md index e7f05ef..84c3079 100644 --- a/RAA.md +++ b/RAA.md @@ -1,6 +1,6 @@ # RAA deployment -Last updated: 2026-08-04. +Last updated: 2026-08-10. RAA is not a separate code project anymore. The source of truth for code and documentation is: @@ -26,6 +26,7 @@ continue feature work there. - Destination: `raa-fn8`, CTID `105`, IP `192.168.1.105` - Edge proxy: Caddy in CTID `110` proxies `raa.panel.f-n8.ru` to Coolify Traefik at `192.168.1.105:80` - Health check: `https://raa.panel.f-n8.ru/health` +- Current verified deploy: `f866a13a8a83598e26a7a6a3406b8bc1c01c9a80` ## Per-Project Settings @@ -55,6 +56,10 @@ Database settings: - `vk_poster_access_token` is the RAA community token - `site_poster_enabled=false` - `site_poster_provider=""` until a website adapter is configured +- Site parser settings are stored here as well: + `site_parser_url`, `site_parser_token`, `site_parser_rucaptcha_token`, + `site_parser_timeout_sec`, `site_parser_interval_minutes`, and + `site_parser_min_text_length`. - Optional social text headers/footers: - `tg_poster_header_text` / `tg_poster_footer_text` - `vk_poster_header_text` / `vk_poster_footer_text` @@ -83,6 +88,29 @@ AI qualifier/writer are configured with user-provided token, models, and prompts but both the worker controls and double-safety settings were still off. Turning them on will spend LLM tokens and process the pending RAA raw posts. +## Website Sources + +Website sources use the existing `vk-parser` scheduling loop but are sent one +at a time to a stateless external worker in LXC `108` (`192.168.1.113:8080`). +All source records, JSON configs, schedules, errors, raw posts, and media state +remain in the RAA database. The worker is managed by Docker Compose from +`/opt/site-parser` and should be healthy as `site-parser-worker`. + +PopularAirsoft source id `72`: + +- URL: `https://popularairsoft.com/rss.xml` +- Config: `{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50}` +- `access='cloudflare'` deliberately starts the browser/RuCaptcha flow before + fetching the RSS. +- Keep `follow_links` disabled for this feed. Nine current entries have no + body, only title/video, and are intentionally filtered before media upload. +- Cloudflare cookies and browser User-Agent are reused by `media-uploader` for + protected images. +- Admin image previews use authenticated Telegram-backed + `/raw/media/{media_id}` URLs. +- Verified 2026-08-10: source `ok`; raw post `102` is `storage_ready`; four + images are `uploaded`. The AI later rejected the content as non-target. + ## RAA Categories RAA categories live in the RAA `content_categories` table. The UI has only diff --git a/STATE.md b/STATE.md index fbd6548..3781d2c 100644 --- a/STATE.md +++ b/STATE.md @@ -1,6 +1,6 @@ # N8 Parser: current state -Last updated: 2026-08-03 +Last updated: 2026-08-10 This file is the short handoff state for future Codex threads. Read this first before touching the project, so the whole chat history does not need to be carried forward. @@ -37,6 +37,8 @@ Current runtime is local infrastructure, not the old VPS. 1. Sources are configured in admin. 2. VK parser reads enabled sources and stores raw posts. + The same worker also dispatches `platform='site'` sources to the external + site-parser module one at a time. 3. Media/storage uploader sends raw post copies to Telegram storage and stores media metadata. 4. AI qualifier scores raw posts and marks accepted/rejected/maybe. 5. AI writer rewrites accepted posts into editor-ready drafts. @@ -75,6 +77,60 @@ canonical git repository as FN-8. Code work for both projects must happen in (`raa-sender`) remains a valid legacy VK reposting app and is not the shared RAA parser/writer/poster app. +## External Site Parser + +As of 2026-08-10, website parsing is integrated into the normal raw-post +pipeline. The main application owns sources, per-source JSON configs, +schedules, settings, database state, deduplication, media jobs, and errors. +The external module is stateless: it receives one source request, fetches it, +and returns normalized JSON. + +- Runtime: LXC `108`, IP `192.168.1.113`, port `8080`. +- Management: Docker Compose in `/opt/site-parser`. +- Repository compose file: `site_parser_worker/docker-compose.yml`. +- The module is protected by `WORKER_TOKEN`; the value is stored only in + deployment settings/secrets. +- RuCaptcha is stored in the main app setting + `site_parser_rucaptcha_token`; never copy the key into git documentation. +- Main settings are under `/workers` -> `Site Parser`: worker URL/token, + RuCaptcha token, request timeout, global interval, and default minimum body + length. +- Website sources are added on `/sources` with `platform='site'` and their own + JSON config. A missing or invalid config produces a visible source error and + does not run silently. +- Due website sources are processed sequentially, not launched together. +- `access='cloudflare'` enables the Playwright/RuCaptcha path for that source. + Plain RSS/sites do not use a browser. +- Browser cookies plus the exact browser User-Agent are returned to the main + app and reused by `media-uploader` when downloading protected images. +- Uploaded photos are previewed in the admin through authenticated + `/raw/media/{media_id}` URLs backed by Telegram storage, not by the original + Cloudflare-protected URL. + +Current RAA test source: + +```json +{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50} +``` + +- Source: `Popularairsoft`, `https://popularairsoft.com/rss.xml`, id `72`. +- Do not add `follow_links` for the current feed: one of ten entries has a real + body and four photos; the other nine are title-plus-video entries and are + intentionally skipped by the body-length threshold. +- Verified on 2026-08-10: source status `ok`, raw post `102` reached + `storage_ready`, and all four photos were uploaded. Its later rejection is + an AI non-target decision, not a parser/media failure. +- Verified RAA deploy: commit `f866a13a8a83598e26a7a6a3406b8bc1c01c9a80`. + +LXC 108 operations: + +```bash +cd /opt/site-parser +docker-compose up -d --build +docker-compose ps +docker-compose logs -f +``` + ## Important Text Rules - AI writer output text must be clean: no physical hashtags at the end.