Your Name
147 lines
4.6 KiB
Markdown
147 lines
4.6 KiB
Markdown
# Site Parser Worker
|
|
|
|
The worker is a stateless executor. The main application owns source configs,
|
|
schedules, database state, and queues. Each request gives the worker a source
|
|
URL, a versioned JSON instruction, and optional Cloudflare session state. The
|
|
worker returns normalized items plus per-item diagnostics.
|
|
|
|
## Config v1
|
|
|
|
- `discovery.type`: `rss` or `html`.
|
|
- `discovery.fields`: where list/RSS values are located.
|
|
- `detail.enabled`: open every discovered item when enabled.
|
|
- `detail.root_selector`: limits extraction to the actual article.
|
|
- `detail.fields`: exact field rules. A rule may contain ordered `candidates`.
|
|
- `detail.media`: exact selectors and fallback attributes for media.
|
|
- `detail.transport`: `http`, `fetch`, or `browser`.
|
|
- `access.type`: `auto`, `http`, `browser`, or `cloudflare`.
|
|
- `retry`: bounded attempts, delay, and per-attempt timeout.
|
|
|
|
Field extraction modes are `text`, `html`, `attr`, and `json`. Date rules can
|
|
provide `formats` and `timezone`. Relative links are resolved against the page
|
|
URL. Unknown iframe/video providers are returned as `external_video`; they do
|
|
not fail the item.
|
|
|
|
Errors are isolated to one item. Successful items are returned with
|
|
`status=partial`; the source receives a visible diagnostic in the main app.
|
|
Only discovery/network failure prevents the whole request from returning a
|
|
batch.
|
|
|
|
## PopularAirsoft
|
|
|
|
Use `https://popularairsoft.com/` as the source URL. Its RSS mixes full news
|
|
articles with short video pages, so the reliable discovery surface is the
|
|
`The Latest News` HTML block. The worker opens the main page through
|
|
Cloudflare, collects `/news/...` links, and downloads each article using the
|
|
same clearance and an impersonated browser TLS fingerprint.
|
|
|
|
```json
|
|
{
|
|
"version": 1,
|
|
"discovery": {
|
|
"type": "html",
|
|
"item_selector": "#block-views-block-latest-news-list-block-2 .feature-contents, #block-views-block-latest-news-list-block-1 .lt-teasure",
|
|
"limit": 10,
|
|
"fields": {
|
|
"url": {
|
|
"selector": "a.link-title",
|
|
"extract": "attr",
|
|
"attribute": "href",
|
|
"required": true
|
|
},
|
|
"external_id": {
|
|
"selector": "a.link-title",
|
|
"extract": "attr",
|
|
"attribute": "href",
|
|
"required": true
|
|
},
|
|
"title": {
|
|
"selector": "a.link-title",
|
|
"extract": "text",
|
|
"required": true
|
|
},
|
|
"published_at": {
|
|
"selector": "time[datetime]",
|
|
"extract": "attr",
|
|
"attribute": "datetime",
|
|
"required": true
|
|
}
|
|
},
|
|
"media": [
|
|
{
|
|
"type": "photo",
|
|
"selector": ".site-image img",
|
|
"attributes": ["src", "data-src", "srcset"]
|
|
}
|
|
]
|
|
},
|
|
"detail": {
|
|
"enabled": true,
|
|
"always": true,
|
|
"transport": "http",
|
|
"root_selector": "article.news.full",
|
|
"fields": {
|
|
"title": {
|
|
"candidates": [
|
|
{
|
|
"selector": ".feature-contents > .news-story-texts:first-child h2",
|
|
"extract": "text"
|
|
},
|
|
{"source": "list.title"}
|
|
],
|
|
"required": true
|
|
},
|
|
"published_at": {
|
|
"candidates": [
|
|
{
|
|
"selector": ".feature-contents > .news-story-texts:first-child .news-story-date",
|
|
"extract": "text",
|
|
"formats": ["%d %b %Y"],
|
|
"timezone": "UTC"
|
|
},
|
|
{"source": "list.published_at"}
|
|
],
|
|
"required": true
|
|
},
|
|
"author": {
|
|
"selector": ".feature-contents > .news-story-texts:first-child h4",
|
|
"extract": "text"
|
|
},
|
|
"text": {
|
|
"selector": ".feature-contents > .news-story-texts:last-child .field--name-body",
|
|
"extract": "text",
|
|
"required": true,
|
|
"min_length": 50
|
|
}
|
|
},
|
|
"media": [
|
|
{
|
|
"type": "photo",
|
|
"selector": ".feature-contents > .site-image .field--name-field-image img, .field--name-body img",
|
|
"attributes": ["src", "data-src", "srcset"]
|
|
},
|
|
{
|
|
"type": "video",
|
|
"selector": ".field--name-body iframe, .field--name-body video, .field--name-body source",
|
|
"attributes": ["src", "data-src"]
|
|
}
|
|
]
|
|
},
|
|
"access": {
|
|
"type": "cloudflare",
|
|
"wait_for": "#block-views-block-latest-news-list-block-1"
|
|
},
|
|
"retry": {
|
|
"attempts": 2,
|
|
"delay_seconds": 2,
|
|
"timeout_seconds": 90
|
|
},
|
|
"min_text_length": 50
|
|
}
|
|
```
|
|
|
|
Verified against the live site on 2026-08-11: nine items, nine successful
|
|
details, zero errors. The VFC sample returned 5,041 text characters, one photo,
|
|
and two Twitch links. The Double Bell sample returned 1,293 characters, one
|
|
photo, and one normalized YouTube URL.
|