docs: document external site parser

PROJECT.md

@@ -196,7 +196,8 @@
 
 Источники для парсинга.
 
-Сейчас реально используется только `platform='vk'`, но схема заложена под другие площадки.
+Используются `platform='vk'` и `platform='site'`. Для сайта конфигурация
+конкретного адаптера хранится вместе с источником в `settings_json`.
 
 Важные поля:
 
@@ -215,9 +216,14 @@
 - `parse_from`
 - `priority`
 - `archived_at`
+- `settings_json` для `platform='site'`
 
 `tag` нужен для будущих хэштегов и финального оформления.
 
+Для сайта отсутствие или ошибка JSON-конфига блокирует запуск с видимой
+ошибкой источника. Конфиг задаёт способ получения именно этого сайта; общее
+расписание и секреты остаются в `app_settings`.
+
 ### 5.4. `raw_posts`
 
 Главная таблица жизненного цикла поста.
@@ -558,6 +564,37 @@ Retention: записи старше 30 дней удаляются при ст
 8. Сохраняет raw post и media.
 9. Создаёт job `media.storage.copy`.
 
+### 7.1.1. Источники сайтов
+
+Тот же scheduling loop обрабатывает `platform='site'`, но сетевую работу
+делегирует внешнему stateless-модулю Site Parser. Основная коробка хранит БД,
+источники, конфиги, расписание, статусы и очередь. Внешний модуль получает один
+запрос и возвращает унифицированный JSON с текстом, исходной ссылкой, датой и
+медиа. Готовые источники запускаются строго последовательно.
+
+Настройки основной коробки:
+
+- `site_parser_url`
+- `site_parser_token`
+- `site_parser_rucaptcha_token`
+- `site_parser_timeout_sec`
+- `site_parser_interval_minutes`
+- `site_parser_min_text_length`
+
+Персональный `min_text_length` в JSON источника переопределяет общий порог.
+Проверяется только тело материала: заголовок не помогает пройти фильтр.
+Записи без достаточного текста, включая video-only, не сохраняются и не
+попадают в media uploader.
+
+`access='cloudflare'` включает Playwright и RuCaptcha для конкретного
+источника; обычные RSS и сайты идут без браузера. Полученные `cf_clearance` и
+точный User-Agent браузера сохраняются в runtime state источника и повторно
+используются media uploader при скачивании защищённых картинок.
+
+Внешний модуль работает в LXC `108` (`192.168.1.113:8080`) через Docker
+Compose `/opt/site-parser/docker-compose.yml`. Репозиторный compose находится
+в `site_parser_worker/docker-compose.yml`.
+
 ### 7.2. Политика мусорных постов
 
 Настройки:
@@ -1091,6 +1128,24 @@ Worker учитывает:
 - active
 - parse_from
 
+Для сайта важны:
+
+- platform `site`
+- name
+- url
+- active
+- обязательный JSON-конфиг конкретного источника
+
+Пример рабочего PopularAirsoft:
+
+```json
+{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50}
+```
+
+Ошибки конфигурации, внешнего worker или сайта показываются в статусе
+источника. После загрузки защищённые изображения отображаются в raw/editor по
+авторизованному маршруту `/raw/media/{media_id}` из Telegram storage.
+
 ### 12.3. Raw
 
 Raw-страница показывает:

RAA.md

@@ -1,6 +1,6 @@
 # RAA deployment
 
-Last updated: 2026-08-04.
+Last updated: 2026-08-10.
 
 RAA is not a separate code project anymore. The source of truth for code and
 documentation is:
@@ -26,6 +26,7 @@ continue feature work there.
 - Destination: `raa-fn8`, CTID `105`, IP `192.168.1.105`
 - Edge proxy: Caddy in CTID `110` proxies `raa.panel.f-n8.ru` to Coolify Traefik at `192.168.1.105:80`
 - Health check: `https://raa.panel.f-n8.ru/health`
+- Current verified deploy: `f866a13a8a83598e26a7a6a3406b8bc1c01c9a80`
 
 ## Per-Project Settings
 
@@ -55,6 +56,10 @@ Database settings:
 - `vk_poster_access_token` is the RAA community token
 - `site_poster_enabled=false`
 - `site_poster_provider=""` until a website adapter is configured
+- Site parser settings are stored here as well:
+  `site_parser_url`, `site_parser_token`, `site_parser_rucaptcha_token`,
+  `site_parser_timeout_sec`, `site_parser_interval_minutes`, and
+  `site_parser_min_text_length`.
 - Optional social text headers/footers:
   - `tg_poster_header_text` / `tg_poster_footer_text`
   - `vk_poster_header_text` / `vk_poster_footer_text`
@@ -83,6 +88,29 @@ AI qualifier/writer are configured with user-provided token, models, and prompts
 but both the worker controls and double-safety settings were still off. Turning
 them on will spend LLM tokens and process the pending RAA raw posts.
 
+## Website Sources
+
+Website sources use the existing `vk-parser` scheduling loop but are sent one
+at a time to a stateless external worker in LXC `108` (`192.168.1.113:8080`).
+All source records, JSON configs, schedules, errors, raw posts, and media state
+remain in the RAA database. The worker is managed by Docker Compose from
+`/opt/site-parser` and should be healthy as `site-parser-worker`.
+
+PopularAirsoft source id `72`:
+
+- URL: `https://popularairsoft.com/rss.xml`
+- Config: `{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50}`
+- `access='cloudflare'` deliberately starts the browser/RuCaptcha flow before
+  fetching the RSS.
+- Keep `follow_links` disabled for this feed. Nine current entries have no
+  body, only title/video, and are intentionally filtered before media upload.
+- Cloudflare cookies and browser User-Agent are reused by `media-uploader` for
+  protected images.
+- Admin image previews use authenticated Telegram-backed
+  `/raw/media/{media_id}` URLs.
+- Verified 2026-08-10: source `ok`; raw post `102` is `storage_ready`; four
+  images are `uploaded`. The AI later rejected the content as non-target.
+
 ## RAA Categories
 
 RAA categories live in the RAA `content_categories` table. The UI has only

STATE.md

@@ -1,6 +1,6 @@
 # N8 Parser: current state
 
-Last updated: 2026-08-03
+Last updated: 2026-08-10
 
 This file is the short handoff state for future Codex threads. Read this first before touching the project, so the whole chat history does not need to be carried forward.
 
@@ -37,6 +37,8 @@ Current runtime is local infrastructure, not the old VPS.
 
 1. Sources are configured in admin.
 2. VK parser reads enabled sources and stores raw posts.
+   The same worker also dispatches `platform='site'` sources to the external
+   site-parser module one at a time.
 3. Media/storage uploader sends raw post copies to Telegram storage and stores media metadata.
 4. AI qualifier scores raw posts and marks accepted/rejected/maybe.
 5. AI writer rewrites accepted posts into editor-ready drafts.
@@ -75,6 +77,60 @@ canonical git repository as FN-8. Code work for both projects must happen in
   (`raa-sender`) remains a valid legacy VK reposting app and is not the shared
   RAA parser/writer/poster app.
 
+## External Site Parser
+
+As of 2026-08-10, website parsing is integrated into the normal raw-post
+pipeline. The main application owns sources, per-source JSON configs,
+schedules, settings, database state, deduplication, media jobs, and errors.
+The external module is stateless: it receives one source request, fetches it,
+and returns normalized JSON.
+
+- Runtime: LXC `108`, IP `192.168.1.113`, port `8080`.
+- Management: Docker Compose in `/opt/site-parser`.
+- Repository compose file: `site_parser_worker/docker-compose.yml`.
+- The module is protected by `WORKER_TOKEN`; the value is stored only in
+  deployment settings/secrets.
+- RuCaptcha is stored in the main app setting
+  `site_parser_rucaptcha_token`; never copy the key into git documentation.
+- Main settings are under `/workers` -> `Site Parser`: worker URL/token,
+  RuCaptcha token, request timeout, global interval, and default minimum body
+  length.
+- Website sources are added on `/sources` with `platform='site'` and their own
+  JSON config. A missing or invalid config produces a visible source error and
+  does not run silently.
+- Due website sources are processed sequentially, not launched together.
+- `access='cloudflare'` enables the Playwright/RuCaptcha path for that source.
+  Plain RSS/sites do not use a browser.
+- Browser cookies plus the exact browser User-Agent are returned to the main
+  app and reused by `media-uploader` when downloading protected images.
+- Uploaded photos are previewed in the admin through authenticated
+  `/raw/media/{media_id}` URLs backed by Telegram storage, not by the original
+  Cloudflare-protected URL.
+
+Current RAA test source:
+
+```json
+{"format":"rss","access":"cloudflare","max_items":10,"min_text_length":50}
+```
+
+- Source: `Popularairsoft`, `https://popularairsoft.com/rss.xml`, id `72`.
+- Do not add `follow_links` for the current feed: one of ten entries has a real
+  body and four photos; the other nine are title-plus-video entries and are
+  intentionally skipped by the body-length threshold.
+- Verified on 2026-08-10: source status `ok`, raw post `102` reached
+  `storage_ready`, and all four photos were uploaded. Its later rejection is
+  an AI non-target decision, not a parser/media failure.
+- Verified RAA deploy: commit `f866a13a8a83598e26a7a6a3406b8bc1c01c9a80`.
+
+LXC 108 operations:
+
+```bash
+cd /opt/site-parser
+docker-compose up -d --build
+docker-compose ps
+docker-compose logs -f
+```
+
 ## Important Text Rules
 
 - AI writer output text must be clean: no physical hashtags at the end.