new_vk_parser/STATE.md

# N8 Parser: current state

Last updated: 2026-08-03

This file is the short handoff state for future Codex threads. Read this first before touching the project, so the whole chat history does not need to be carried forward.

## Runtime / Deploy

Current runtime is local infrastructure, not the old VPS.

- Infrastructure source of truth: `D:\DEVELOPMENT\.infra.md`.
- Git deploy remote: `gitea` -> `http://192.168.1.135:3000/exostring/new_vk_parser.git`.
- Deploy path: push to `gitea/main` -> Gitea webhook -> Coolify -> Docker build/deploy.
- Coolify + Gitea LXC: `192.168.1.135` (CTID 102).
- Coolify app UUID: `n6cnr60anmruiwkiey0pukye` (`FN-8 admin`). API token is only in `D:\DEVELOPMENT\.infra.md`; do not copy it into repo docs.
- Application LXC: `192.168.1.105` (CTID 105, `raa-fn8`).
- Public admin URL: `https://panel.f-n8.ru` (also `https://admin.f-n8.ru` when DNS is current).
- Coolify repository URL must be reachable from the deployment container; use `http://192.168.1.135:3000/exostring/new_vk_parser.git` or another resolvable Gitea URL, not `http://coolify.lan:3000/...`.
- Do not deploy this project through `sw.exostring.xyz`, `109.120.156.205`, `/opt/vk-parser`, or manual `systemctl` unless the user explicitly says the old VPS is being restored.
- Old `sw.exostring.xyz` / `109.120.156.205` notes are stale for this project.

## Local Project

- Local workspace: `D:\DEVELOPMENT\new_vk_parser`
- Main app package: `src/vk_parser_app`
- Admin routes: `src/vk_parser_app/admin.py`
- Templates: `src/vk_parser_app/templates`
- Workers:
  - VK parser: `src/vk_parser_app/workers/parser.py`
  - Telegram media/storage uploader: `src/vk_parser_app/workers/vk_storage_uploader.py`
  - AI qualifier: `src/vk_parser_app/workers/ai_qualifier.py`
  - AI writer: `src/vk_parser_app/workers/ai_writer.py`
- DB migrations: `db/migrations`
- Extended project documentation: `PROJECT.md`

## Pipeline

1. Sources are configured in admin.
2. VK parser reads enabled sources and stores raw posts.
3. Media/storage uploader sends raw post copies to Telegram storage and stores media metadata.
4. AI qualifier scores raw posts and marks accepted/rejected/maybe.
5. AI writer rewrites accepted posts into editor-ready drafts.
6. Editor page allows review, editing, category/source tag selection, media upload/removal, accept/reject.
7. Site poster automatically publishes accepted media-ready posts to Ghost.
   - As of 2026-08-03, site publishing is provider-based: `site_poster_provider=ghost`
     for FN-8, with `site_poster_base_url` and `site_poster_secret_file` copied from
     legacy Ghost settings by migration `040_site_poster_provider_settings.sql`.
   - New installs default `site-poster` to disabled and must explicitly set provider,
     base URL, secret, and `site_poster_enabled=true`.

## RAA Deployment

As of 2026-08-03, RAA is a second Coolify app/database deployed from the same
canonical git repository as FN-8. Code work for both projects must happen in
`D:\DEVELOPMENT\new_vk_parser`.

- Canonical local workspace: `D:\DEVELOPMENT\new_vk_parser`.
- Canonical Gitea repo: `http://192.168.1.135:3000/exostring/new_vk_parser.git`.
- Old local copy `D:\DEVELOPMENT\raa-parser_poster` is not the source of truth.
- Coolify app UUID: `korokrhpoyqxf0nw2y8vk7lp` (`RAA admin`).
- Public admin URL: `https://raa.panel.f-n8.ru`.
- RAA Postgres container: `m9v14hzfh5mk9kelxzcqvhc6`, DB `raa_parser`, user `raa_parser_user`.
- Docker destination/LXC for both FN-8 and RAA: CTID `105`, hostname `raa-fn8`, IP `192.168.1.105`.
- Raw Telegram media channel setting: `tg_media_channel_id=-1004312015562` (`VK RAW RAA`).
- Final Telegram publication chat setting: `tg_poster_chat_id=-1001303630155` (`Red Airsoft | Страйкбол`).
- VK publication owner: `vk_poster_owner_id=-36860851`, `vk_poster_from_group=true`.
- `TG_BOT_TOKEN`, `TELEGRAM_API_ID`, and `TELEGRAM_API_HASH` were copied from FN-8 Coolify env.
- Local Bot API app setting: `local_bot_api_url=http://127.0.0.1:8081`.
- As last checked, RAA has `vk-parser=true`, `vk-storage-uploader=true`.
  AI and publication workers are intentionally disabled until explicitly started:
  `ai-qualifier=false`, `ai-writer=false`, `tg-poster=false`, `vk-poster=false`,
  `site-poster=false`, `tg_poster_enabled=false`, `vk_poster_enabled=false`.
- Detailed RAA notes live in `RAA.md`.
- CT `107` (`raa-parser-poster`) was removed on 2026-08-04. CT `104`
  (`raa-sender`) remains a valid legacy VK reposting app and is not the shared
  RAA parser/writer/poster app.

## Important Text Rules

- AI writer output text must be clean: no physical hashtags at the end.
- `raw_posts.final_text` must also stay clean.
- Category and source hashtags are separate fields:
  - category tag: `final_category_tag` / `rewrite_category_tag`
  - source tag: `final_source_tag` / `rewrite_source_tag` / `sources.tag`
- Publication preview may show hashtags by composing:
  - clean text
  - blank line
  - optional project-wide common tags first, then `#category #source`
- Project-wide common tags live in DB setting `publication_common_tags`.
  FN-8 value: `#новости_снаряжения`. RAA value: `#raa_news #страйкбол`.
- Optional per-platform header/footer settings:
  `tg_poster_header_text`, `tg_poster_footer_text`,
  `vk_poster_header_text`, `vk_poster_footer_text`,
  `max_poster_header_text`, `max_poster_footer_text`.
- MAX header/footer text is sent as HTML. For clickable text use:
  `<a href="https://example.com">Link text</a>`.
- HTML wrapper settings are normalized before publishing: safe `http/https`
  links and simple formatting stay, unsupported/broken tags are stripped while
  their readable text is kept.
- The editor textarea must show only clean editable text.
- The list preview in `/editor` should show the publication preview, including composed hashtags.
- `build_publication_text()` in `src/vk_parser_app/text_utils.py` is display/composition helper only. Do not use it before writing `final_text` to DB.
- `/raw` has a manual checkbox action to send AI-rejected posts back to
  rewrite. It is intentionally limited to `qualification_status='rejected'`;
  editor-rejected posts that were AI-accepted are not selectable there.
- VK/MAX successful publications also sync `raw_posts.publication_status` and
  `editorial_status` to `published`; this keeps VK-only projects from leaving
  posts forever in the editor's accepted tab.

## Recent Hotfix: Hashtags

On 2026-06-17 the hashtag flow was fixed:

- `ai_writer.py` now stores clean `final_text=$2`, not text with hashtags.
- `admin.py` prepares:
  - `review_text`: clean text for textarea
  - `publication_preview_text`: composed display text with hashtags
- editor accept/save store clean text.
- existing DB rows with trailing `#category #source` in `final_text` were cleaned.
- server check after fix: `remaining_hashtag_tail: 0`.

## Admin Pages

- `/sources`: source management.
- `/raw`: raw post feed.
- `/editor`: editor queue.
- `/workers`: worker controls and settings.
- `/prompt-test`: AI writer prompt test page.
- `/users`: admin users.
- `/logs`: audit/log page if enabled.

## Recent Admin UI State

As of 2026-07-29:

- Admin title comes from env `ADMIN_APP_TITLE`; fallback is `Редакторская`. The old hardcoded `Parser Admin` title and lightning SVG are gone.
- Browser/page title comes from env `ADMIN_SITE_TITLE`; fallback is `Редакторская`.
- Admin sidebar is collapsible and collapsed by default; collapsed state shows icons only.
- Theme toggle supports dark/light themes. Tailwind `app.*` colors use RGB-channel CSS variables (`rgb(var(--app-...)/<alpha-value>)`), so do not revert them to hex variables or opacity utilities will break. Light-theme hover contrast is patched in base CSS.
- `/editor` desktop filter panel is collapsible and collapsed by default; the feed expands into the freed width.
- Mobile editor status tabs scroll horizontally instead of overflowing the viewport. `/sources` and `/raw` have mobile layout fixes to avoid page-level horizontal scroll.
- In editor cards, post meta/status/source info is above the post, not in a side column.
- `Оригинальный текст` is a spoiler directly above `Финальный текст`.
- `AI заметки / Резолюция` is a separate spoiler. It contains AI notes/resolution and an AI rewrite preview below it, so editors can compare their changes with the original AI rewrite.
- `Финальный текст` textarea must autosize to fully fit text without internal scroll. Current implementation uses Alpine `ResizeObserver`, trims trailing whitespace in the browser/template, and keeps `min-h-[150px]`.
- Media hiding is handled by the editor media X button: it adds hidden `delete_media_ids`; backend marks media `editor_hidden=TRUE`, so hidden media is not published.
- Reject action has no confirmation popup.
- Save/accept/return actions use targeted HTMX partial updates for status/actions where possible, so open spoilers and the post body should not collapse after clicking action buttons.
- `/editor/{post_id}/return` returns accepted/rejected posts to review. The return button should exist on both accepted and rejected tabs.

## Recent AI Writer State

- Current live writer model setting checked on 2026-07-29: `ai_writer_model = anthropic/claude-sonnet-4-6`.
- Sonnet 4.6 can return `rewrites` as a JSON string instead of a list. `validate_rewrites()` in `src/vk_parser_app/workers/ai_writer.py` now parses stringified `rewrites` before validation.
- 2026-07-30 incident: post `2348` was retried 983 times by AI writer because automatic claim included `rewrite_status='failed'`; estimated internal cost was `$16.465977`. Live setting `ai_writer_enabled` was switched to `false` manually to stop spend. Automatic AI qualifier/writer claims must not include `failed`; failed posts require an explicit manual reset to `pending`. AI writer/qualifier keep running after a failed batch, show the error on `/workers`, and send a Telegram alert via daily-report/TG-poster bot recipients.
- Sonnet 4.6 may return `category_id=18` (`Не целевой контент`) with empty `text` and the explanation in `notes`. This is valid AI rejection, not a writer error; `validate_rewrites()` allows short/empty text only for non-target category.
- AI qualifier has exactly one automatic retry: after the first failed qualification batch, affected posts are returned to `qualification_status='pending'`; after the second failed batch for the same post, they stay `failed` and are not claimed again automatically.

## Prompt Architecture

- Prompts are split into editable human part and technical contract part.
- User should be able to edit human prompt safely without breaking JSON schema.
- Contracts should contain JSON-only/output-schema requirements.
- Writer should return category ID, not free-form tag, where possible.
- Categories are managed in worker/admin settings and have stable IDs.

## Security Notes

- Secrets should not be committed.
- `.env` should contain only secrets and deploy-specific values.
- Runtime settings that are not secrets should live in DB/admin settings.
- Dangerous FN-8 defaults were removed from runtime defaults on 2026-08-03:
  no default TG publication chat, no default VK storage group, no default Ghost URL/key path.
- Docker image now copies `db/migrations`, and `docker-entrypoint.sh` runs
  `python /app/scripts/apply_migrations.py` before starting the app.
- Admin has login protection and server has firewall/fail2ban hardening from previous iterations.

## Working Rules For Future Codex Runs

- Before any deploy/server operation, read `D:\DEVELOPMENT\.infra.md` first.
- Before pushing or deploying, run `git pull --rebase gitea main`.
- For this project, push deployable changes to `gitea/main`; Coolify handles deployment.
- FN-8 and RAA should deploy from the same `new_vk_parser.git` commit SHA.
  Differences between projects belong in Coolify env variables and each project's DB settings.
- Prefer small targeted file reads with `rg` and narrow ranges.
- Deployment status polling should be sparse. Coolify rebuilds can take 7-11 minutes
  because the Docker build currently runs without cache and reinstalls system deps.
- Avoid dumping long post texts from DB unless explicitly needed.
- For runtime checks, use the current Coolify/Gitea/LXC 105 path from `D:\DEVELOPMENT\.infra.md`.
- Do not `git reset --hard` or revert user/server hotfixes.