new_vk_parser/STATE.md

N8 Parser: current state

Last updated: 2026-08-03

This file is the short handoff state for future Codex threads. Read this first before touching the project, so the whole chat history does not need to be carried forward.

Runtime / Deploy

Current runtime is local infrastructure, not the old VPS.

• Infrastructure source of truth: D:\DEVELOPMENT\.infra.md.
• Git deploy remote: gitea -> http://192.168.1.135:3000/exostring/new_vk_parser.git.
• Deploy path: push to gitea/main -> Gitea webhook -> Coolify -> Docker build/deploy.
• Coolify + Gitea LXC: 192.168.1.135 (CTID 102).
• Coolify app UUID: n6cnr60anmruiwkiey0pukye (FN-8 admin). API token is only in D:\DEVELOPMENT\.infra.md; do not copy it into repo docs.
• Application LXC: 192.168.1.105 (CTID 105, raa-fn8).
• Public admin URL: https://panel.f-n8.ru (also https://admin.f-n8.ru when DNS is current).
• Coolify repository URL must be reachable from the deployment container; use http://192.168.1.135:3000/exostring/new_vk_parser.git or another resolvable Gitea URL, not http://coolify.lan:3000/....
• Do not deploy this project through sw.exostring.xyz, 109.120.156.205, /opt/vk-parser, or manual systemctl unless the user explicitly says the old VPS is being restored.
• Old sw.exostring.xyz / 109.120.156.205 notes are stale for this project.

Local Project

• Local workspace: D:\DEVELOPMENT\new_vk_parser
• Main app package: src/vk_parser_app
• Admin routes: src/vk_parser_app/admin.py
• Templates: src/vk_parser_app/templates
• Workers:
  • VK parser: src/vk_parser_app/workers/parser.py
  • Telegram media/storage uploader: src/vk_parser_app/workers/vk_storage_uploader.py
  • AI qualifier: src/vk_parser_app/workers/ai_qualifier.py
  • AI writer: src/vk_parser_app/workers/ai_writer.py
• DB migrations: db/migrations
• Extended project documentation: PROJECT.md

Pipeline

1. Sources are configured in admin.
2. VK parser reads enabled sources and stores raw posts.
3. Media/storage uploader sends raw post copies to Telegram storage and stores media metadata.
4. AI qualifier scores raw posts and marks accepted/rejected/maybe.
5. AI writer rewrites accepted posts into editor-ready drafts.
6. Editor page allows review, editing, category/source tag selection, media upload/removal, accept/reject.
7. Site poster automatically publishes accepted media-ready posts to Ghost.
   • As of 2026-08-03, site publishing is provider-based: site_poster_provider=ghost for FN-8, with site_poster_base_url and site_poster_secret_file copied from legacy Ghost settings by migration 040_site_poster_provider_settings.sql.
   • New installs default site-poster to disabled and must explicitly set provider, base URL, secret, and site_poster_enabled=true.

RAA Deployment

As of 2026-08-03, RAA is a second Coolify app/database deployed from the same canonical git repository as FN-8. Code work for both projects must happen in D:\DEVELOPMENT\new_vk_parser.

• Canonical local workspace: D:\DEVELOPMENT\new_vk_parser.
• Canonical Gitea repo: http://192.168.1.135:3000/exostring/new_vk_parser.git.
• Old local copy D:\DEVELOPMENT\raa-parser_poster is not the source of truth.
• Coolify app UUID: korokrhpoyqxf0nw2y8vk7lp (RAA admin).
• Public admin URL: https://raa.panel.f-n8.ru.
• RAA Postgres container: m9v14hzfh5mk9kelxzcqvhc6, DB raa_parser, user raa_parser_user.
• Docker destination/LXC for both FN-8 and RAA: CTID 105, hostname raa-fn8, IP 192.168.1.105.
• Raw Telegram media channel setting: tg_media_channel_id=-1004312015562 (VK RAW RAA).
• Final Telegram publication chat setting: tg_poster_chat_id=-1001303630155 (Red Airsoft | Страйкбол).
• VK publication owner: vk_poster_owner_id=-36860851, vk_poster_from_group=true.
• TG_BOT_TOKEN, TELEGRAM_API_ID, and TELEGRAM_API_HASH were copied from FN-8 Coolify env.
• Local Bot API app setting: local_bot_api_url=http://127.0.0.1:8081.
• As last checked, RAA has vk-parser=true, vk-storage-uploader=true. AI and publication workers are intentionally disabled until explicitly started: ai-qualifier=false, ai-writer=false, tg-poster=false, vk-poster=false, site-poster=false, tg_poster_enabled=false, vk_poster_enabled=false.
• Detailed RAA notes live in RAA.md.
• CT 107 (raa-parser-poster) was removed on 2026-08-04. CT 104 (raa-sender) remains a valid legacy VK reposting app and is not the shared RAA parser/writer/poster app.

Important Text Rules

• AI writer output text must be clean: no physical hashtags at the end.
• raw_posts.final_text must also stay clean.
• Category and source hashtags are separate fields:
  • category tag: final_category_tag / rewrite_category_tag
  • source tag: final_source_tag / rewrite_source_tag / sources.tag
• Publication preview may show hashtags by composing:
  • clean text
  • blank line
  • optional project-wide common tags first, then #category #source
• Project-wide common tags live in DB setting publication_common_tags. FN-8 value: #новости_снаряжения. RAA value: #raa_news #страйкбол.
• Optional per-platform header/footer settings: tg_poster_header_text, tg_poster_footer_text, vk_poster_header_text, vk_poster_footer_text, max_poster_header_text, max_poster_footer_text.
• MAX header/footer text is sent as HTML. For clickable text use: <a href="https://example.com">Link text</a>.
• HTML wrapper settings are normalized before publishing: safe http/https links and simple formatting stay, unsupported/broken tags are stripped while their readable text is kept.
• The editor textarea must show only clean editable text.
• The list preview in /editor should show the publication preview, including composed hashtags.
• build_publication_text() in src/vk_parser_app/text_utils.py is display/composition helper only. Do not use it before writing final_text to DB.
• /raw has a manual checkbox action to send AI-rejected posts back to rewrite. It is intentionally limited to qualification_status='rejected'; editor-rejected posts that were AI-accepted are not selectable there.
• VK/MAX successful publications also sync raw_posts.publication_status and editorial_status to published; this keeps VK-only projects from leaving posts forever in the editor's accepted tab.

Recent Hotfix: Hashtags

On 2026-06-17 the hashtag flow was fixed:

• ai_writer.py now stores clean final_text=$2, not text with hashtags.
• admin.py prepares:
  • review_text: clean text for textarea
  • publication_preview_text: composed display text with hashtags
• editor accept/save store clean text.
• existing DB rows with trailing #category #source in final_text were cleaned.
• server check after fix: remaining_hashtag_tail: 0.

Admin Pages

• /sources: source management.
• /raw: raw post feed.
• /editor: editor queue.
• /workers: worker controls and settings.
• /prompt-test: AI writer prompt test page.
• /users: admin users.
• /logs: audit/log page if enabled.

Recent Admin UI State

As of 2026-07-29:

• Admin title comes from env ADMIN_APP_TITLE; fallback is Редакторская. The old hardcoded Parser Admin title and lightning SVG are gone.
• Browser/page title comes from env ADMIN_SITE_TITLE; fallback is Редакторская.
• Admin sidebar is collapsible and collapsed by default; collapsed state shows icons only.
• Theme toggle supports dark/light themes. Tailwind app.* colors use RGB-channel CSS variables (rgb(var(--app-...)/<alpha-value>)), so do not revert them to hex variables or opacity utilities will break. Light-theme hover contrast is patched in base CSS.
• /editor desktop filter panel is collapsible and collapsed by default; the feed expands into the freed width.
• Mobile editor status tabs scroll horizontally instead of overflowing the viewport. /sources and /raw have mobile layout fixes to avoid page-level horizontal scroll.
• In editor cards, post meta/status/source info is above the post, not in a side column.
• Оригинальный текст is a spoiler directly above Финальный текст.
• AI заметки / Резолюция is a separate spoiler. It contains AI notes/resolution and an AI rewrite preview below it, so editors can compare their changes with the original AI rewrite.
• Финальный текст textarea must autosize to fully fit text without internal scroll. Current implementation uses Alpine ResizeObserver, trims trailing whitespace in the browser/template, and keeps min-h-[150px].
• Media hiding is handled by the editor media X button: it adds hidden delete_media_ids; backend marks media editor_hidden=TRUE, so hidden media is not published.
• Reject action has no confirmation popup.
• Save/accept/return actions use targeted HTMX partial updates for status/actions where possible, so open spoilers and the post body should not collapse after clicking action buttons.
• /editor/{post_id}/return returns accepted/rejected posts to review. The return button should exist on both accepted and rejected tabs.

Recent AI Writer State

• Current live writer model setting checked on 2026-07-29: ai_writer_model = anthropic/claude-sonnet-4-6.
• Sonnet 4.6 can return rewrites as a JSON string instead of a list. validate_rewrites() in src/vk_parser_app/workers/ai_writer.py now parses stringified rewrites before validation.
• 2026-07-30 incident: post 2348 was retried 983 times by AI writer because automatic claim included rewrite_status='failed'; estimated internal cost was $16.465977. Live setting ai_writer_enabled was switched to false manually to stop spend. Automatic AI qualifier/writer claims must not include failed; failed posts require an explicit manual reset to pending. AI writer/qualifier keep running after a failed batch, show the error on /workers, and send a Telegram alert via daily-report/TG-poster bot recipients.
• Sonnet 4.6 may return category_id=18 (Не целевой контент) with empty text and the explanation in notes. This is valid AI rejection, not a writer error; validate_rewrites() allows short/empty text only for non-target category.
• AI qualifier has exactly one automatic retry: after the first failed qualification batch, affected posts are returned to qualification_status='pending'; after the second failed batch for the same post, they stay failed and are not claimed again automatically.

Prompt Architecture

• Prompts are split into editable human part and technical contract part.
• User should be able to edit human prompt safely without breaking JSON schema.
• Contracts should contain JSON-only/output-schema requirements.
• Writer should return category ID, not free-form tag, where possible.
• Categories are managed in worker/admin settings and have stable IDs.

Security Notes

• Secrets should not be committed.
• .env should contain only secrets and deploy-specific values.
• Runtime settings that are not secrets should live in DB/admin settings.
• Dangerous FN-8 defaults were removed from runtime defaults on 2026-08-03: no default TG publication chat, no default VK storage group, no default Ghost URL/key path.
• Docker image now copies db/migrations, and docker-entrypoint.sh runs python /app/scripts/apply_migrations.py before starting the app.
• Admin has login protection and server has firewall/fail2ban hardening from previous iterations.

Working Rules For Future Codex Runs

• Before any deploy/server operation, read D:\DEVELOPMENT\.infra.md first.
• Before pushing or deploying, run git pull --rebase gitea main.
• For this project, push deployable changes to gitea/main; Coolify handles deployment.
• FN-8 and RAA should deploy from the same new_vk_parser.git commit SHA. Differences between projects belong in Coolify env variables and each project's DB settings.
• Prefer small targeted file reads with rg and narrow ranges.
• Deployment status polling should be sparse. Coolify rebuilds can take 7-11 minutes because the Docker build currently runs without cache and reinstalls system deps.
• Avoid dumping long post texts from DB unless explicitly needed.
• For runtime checks, use the current Coolify/Gitea/LXC 105 path from D:\DEVELOPMENT\.infra.md.
• Do not git reset --hard or revert user/server hotfixes.