594 lines
12 KiB
Markdown
594 lines
12 KiB
Markdown
# WCX Index
|
|
|
|
Local SQLite-based index for WCX publications.
|
|
|
|
The project imports metadata produced by an existing site scraper, stores it in a SQLite database, and enriches new records with metadata extracted from thumbnail images using Google Cloud Vision OCR.
|
|
|
|
## Current processing flow
|
|
|
|
```text
|
|
WCX site
|
|
↓
|
|
existing scraper
|
|
↓
|
|
import/wcx_site_index.json
|
|
↓
|
|
scripts/import_site.py
|
|
↓
|
|
database/wcx.db
|
|
↓
|
|
scripts/check_ocr.py
|
|
├── scripts/ocr.sh
|
|
└── scripts/parse_ocr.py
|
|
↓
|
|
OCR metadata stored in SQLite
|
|
```
|
|
|
|
The current implementation supports:
|
|
|
|
* inserting new movies from the scraper JSON
|
|
* updating existing movies based on the site's `update` field
|
|
* storing movie duration as seconds
|
|
* running OCR for an individual movie
|
|
* parsing and validating OCR output
|
|
* comparing the OCR name with the database name
|
|
* storing OCR metadata, raw OCR text, status, and errors
|
|
|
|
## Directory structure
|
|
|
|
```text
|
|
/storage/disk1/WCX/
|
|
├── database/
|
|
│ └── wcx.db
|
|
├── import/
|
|
│ └── wcx_site_index.json
|
|
├── scripts/
|
|
│ ├── schema.sql
|
|
│ ├── import_site.py
|
|
│ ├── ocr.sh
|
|
│ ├── parse_ocr.py
|
|
│ └── check_ocr.py
|
|
├── .gitignore
|
|
└── README.md
|
|
```
|
|
|
|
## Requirements
|
|
|
|
The project currently uses:
|
|
|
|
* Python 3
|
|
* SQLite 3
|
|
* Bash
|
|
* curl
|
|
* jq
|
|
* base64
|
|
* Google Cloud Vision API access
|
|
|
|
Install the local command-line dependencies on Ubuntu or Debian:
|
|
|
|
```bash
|
|
sudo apt update
|
|
sudo apt install sqlite3 jq curl
|
|
```
|
|
|
|
The Python scripts only use modules from the Python standard library.
|
|
|
|
## Google Vision API key
|
|
|
|
The OCR script expects the Google Vision API key in the environment:
|
|
|
|
```bash
|
|
export GOOGLE_VISION_API_KEY="your-api-key"
|
|
```
|
|
|
|
The key must be available in the environment whenever `ocr.sh` or `check_ocr.py` is executed.
|
|
|
|
Do not commit the API key to Git.
|
|
|
|
## Creating the database
|
|
|
|
Create the database directory:
|
|
|
|
```bash
|
|
mkdir -p /storage/disk1/WCX/database
|
|
```
|
|
|
|
Create the database from the schema:
|
|
|
|
```bash
|
|
sqlite3 /storage/disk1/WCX/database/wcx.db \
|
|
< /storage/disk1/WCX/scripts/schema.sql
|
|
```
|
|
|
|
Inspect the resulting table:
|
|
|
|
```bash
|
|
sqlite3 /storage/disk1/WCX/database/wcx.db ".schema movie"
|
|
```
|
|
|
|
## Database model
|
|
|
|
The `movie` table currently contains the following columns:
|
|
|
|
| Column | Description |
|
|
| ------------------ | -------------------------------------------------- |
|
|
| `id` | Stable and unique ID from the site |
|
|
| `name` | Movie title |
|
|
| `aka` | Alternative name |
|
|
| `nationality` | Nationality extracted by OCR or edited manually |
|
|
| `age` | Manually maintained age |
|
|
| `shoot_location` | Shoot location extracted by OCR or edited manually |
|
|
| `shoot_date` | Shoot date stored as `YYYY-MM-DD` |
|
|
| `duration_seconds` | Movie duration stored in seconds |
|
|
| `description` | Manually maintained description |
|
|
| `rating` | Manually maintained rating |
|
|
| `web_url` | URL to the movie detail page |
|
|
| `thumbnail` | URL to the thumbnail image |
|
|
| `published` | Original publication date |
|
|
| `updated` | Latest site update date |
|
|
| `created_at` | Timestamp when the database record was created |
|
|
| `modified_at` | Timestamp when the record was last modified |
|
|
| `ocr_status` | Current OCR processing status |
|
|
| `ocr_raw_text` | Unmodified OCR text returned by Google Vision |
|
|
| `ocr_error` | OCR or validation error |
|
|
| `ocr_processed_at` | Timestamp of the latest OCR attempt |
|
|
|
|
Possible OCR statuses are currently:
|
|
|
|
```text
|
|
pending
|
|
completed
|
|
failed
|
|
manual_review
|
|
```
|
|
|
|
## Site JSON input
|
|
|
|
The site importer reads:
|
|
|
|
```text
|
|
/storage/disk1/WCX/import/wcx_site_index.json
|
|
```
|
|
|
|
The file must contain a JSON array.
|
|
|
|
Example entry:
|
|
|
|
```json
|
|
{
|
|
"id": "sweet-cherry_41678",
|
|
"titel": "Sweet Cherry",
|
|
"details": "https://www.woodmancastingx.com/casting-x/sweet-cherry_41678.html",
|
|
"duration": "1:37:00",
|
|
"thumb": "https://example.com/thumbnail.jpg",
|
|
"published": "2026-07-12"
|
|
}
|
|
```
|
|
|
|
An updated publication may also contain:
|
|
|
|
```json
|
|
{
|
|
"update": "2026-07-15"
|
|
}
|
|
```
|
|
|
|
The importer accepts both `update` and `updated`.
|
|
|
|
## Site field mapping
|
|
|
|
| JSON field | Database column |
|
|
| --------------------- | ------------------ |
|
|
| `id` | `id` |
|
|
| `titel` | `name` |
|
|
| `details` | `web_url` |
|
|
| `duration` | `duration_seconds` |
|
|
| `thumb` | `thumbnail` |
|
|
| `published` | `published` |
|
|
| `update` or `updated` | `updated` |
|
|
|
|
Durations are converted to seconds.
|
|
|
|
Examples:
|
|
|
|
```text
|
|
37:35 → 2255
|
|
1:37:00 → 5820
|
|
```
|
|
|
|
## Importing site data
|
|
|
|
Make the importer executable:
|
|
|
|
```bash
|
|
chmod +x /storage/disk1/WCX/scripts/import_site.py
|
|
```
|
|
|
|
Run the import:
|
|
|
|
```bash
|
|
/storage/disk1/WCX/scripts/import_site.py
|
|
```
|
|
|
|
Example output:
|
|
|
|
```text
|
|
Poster i JSON: 1250
|
|
Tillagda: 3
|
|
Uppdaterade: 1
|
|
Oförändrade: 1246
|
|
```
|
|
|
|
## Site import rules
|
|
|
|
### New movie
|
|
|
|
If the movie ID does not exist in the database, a new row is inserted.
|
|
|
|
New rows receive the default OCR status:
|
|
|
|
```text
|
|
pending
|
|
```
|
|
|
|
### Existing movie without a site update
|
|
|
|
If the movie already exists and the site entry has no `update` date, no database changes are made.
|
|
|
|
### Existing movie with a site update
|
|
|
|
An existing row is updated when:
|
|
|
|
* the database has no `updated` value and the site does, or
|
|
* the site's update date is newer than the database value
|
|
|
|
When an update is triggered, all fields supplied by the site are replaced:
|
|
|
|
* `name`
|
|
* `duration_seconds`
|
|
* `web_url`
|
|
* `thumbnail`
|
|
* `published`
|
|
* `updated`
|
|
|
|
The movie ID and manually maintained metadata fields are left unchanged.
|
|
|
|
Dates use the ISO format `YYYY-MM-DD`, allowing chronological comparison as text.
|
|
|
|
## Inspecting the database
|
|
|
|
Show the total number of movies:
|
|
|
|
```bash
|
|
sqlite3 /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT COUNT(*) FROM movie;"
|
|
```
|
|
|
|
Show some movies:
|
|
|
|
```bash
|
|
sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT id, name, duration_seconds, published, updated
|
|
FROM movie
|
|
LIMIT 10;"
|
|
```
|
|
|
|
Show a specific movie:
|
|
|
|
```bash
|
|
sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT *
|
|
FROM movie
|
|
WHERE id = 'susana-melo_6707';"
|
|
```
|
|
|
|
## OCR processing
|
|
|
|
OCR processing is currently performed for one movie at a time.
|
|
|
|
The complete flow is:
|
|
|
|
```text
|
|
movie ID
|
|
↓
|
|
check_ocr.py
|
|
↓
|
|
thumbnail URL loaded from SQLite
|
|
↓
|
|
ocr.sh
|
|
↓
|
|
Google Cloud Vision raw text
|
|
↓
|
|
parse_ocr.py
|
|
↓
|
|
name validation
|
|
↓
|
|
SQLite update
|
|
```
|
|
|
|
## Raw OCR script
|
|
|
|
Run OCR directly against an image URL:
|
|
|
|
```bash
|
|
/storage/disk1/WCX/scripts/ocr.sh \
|
|
"https://example.com/thumbnail.jpg"
|
|
```
|
|
|
|
Example output:
|
|
|
|
```text
|
|
SUSANA MELO
|
|
PORTUGUESE
|
|
WOODMAN CASTINGX.COM
|
|
Budapest (Hungary), March 9, 2014
|
|
```
|
|
|
|
The raw OCR output is not considered a stable data format and must be parsed and validated before metadata is stored.
|
|
|
|
## Expected OCR structure
|
|
|
|
The parser currently expects exactly four non-empty lines:
|
|
|
|
```text
|
|
Line 1: Performer name
|
|
Line 2: Nationality
|
|
Line 3: Branding text, ignored
|
|
Line 4: Shoot location and shoot date
|
|
```
|
|
|
|
Example final line:
|
|
|
|
```text
|
|
Budapest (Hungary), March 9, 2014
|
|
```
|
|
|
|
The line is split on the first comma:
|
|
|
|
```text
|
|
shoot_location = Budapest (Hungary)
|
|
shoot_date = March 9, 2014
|
|
```
|
|
|
|
The date is normalized before storage:
|
|
|
|
```text
|
|
March 9, 2014 → 2014-03-09
|
|
```
|
|
|
|
## Testing the OCR parser
|
|
|
|
Raw OCR output can be piped directly into the parser:
|
|
|
|
```bash
|
|
/storage/disk1/WCX/scripts/ocr.sh \
|
|
"https://example.com/thumbnail.jpg" \
|
|
| /storage/disk1/WCX/scripts/parse_ocr.py
|
|
```
|
|
|
|
Example result:
|
|
|
|
```text
|
|
ocr_name=SUSANA MELO
|
|
nationality=PORTUGUESE
|
|
shoot_location=Budapest (Hungary)
|
|
shoot_date=2014-03-09
|
|
```
|
|
|
|
## Processing OCR for a movie
|
|
|
|
Run the complete OCR workflow using a movie ID:
|
|
|
|
```bash
|
|
/storage/disk1/WCX/scripts/check_ocr.py susana-melo_6707
|
|
```
|
|
|
|
Example output:
|
|
|
|
```text
|
|
id=susana-melo_6707
|
|
database_name=Susana Melo
|
|
ocr_name=SUSANA MELO
|
|
name_match=yes
|
|
nationality=PORTUGUESE
|
|
shoot_location=Budapest (Hungary)
|
|
shoot_date=2014-03-09
|
|
ocr_status=completed
|
|
```
|
|
|
|
The name comparison is case-insensitive and ignores repeated whitespace.
|
|
|
|
For example:
|
|
|
|
```text
|
|
Susana Melo
|
|
SUSANA MELO
|
|
```
|
|
|
|
are considered equal.
|
|
|
|
## Successful OCR processing
|
|
|
|
When OCR execution and parsing succeed and the OCR name matches the database name, the following fields are updated:
|
|
|
|
```text
|
|
nationality
|
|
shoot_location
|
|
shoot_date
|
|
ocr_raw_text
|
|
ocr_status = completed
|
|
ocr_error = NULL
|
|
ocr_processed_at
|
|
modified_at
|
|
```
|
|
|
|
## OCR failure handling
|
|
|
|
If the Google Vision request or `ocr.sh` execution fails:
|
|
|
|
```text
|
|
ocr_status = failed
|
|
```
|
|
|
|
The error is stored in:
|
|
|
|
```text
|
|
ocr_error
|
|
```
|
|
|
|
## Manual review handling
|
|
|
|
The movie is marked for manual review when:
|
|
|
|
* the OCR output cannot be parsed
|
|
* the expected fields are missing
|
|
* the OCR name does not match the database name
|
|
* the OCR output has an unexpected structure
|
|
|
|
In this case:
|
|
|
|
```text
|
|
ocr_status = manual_review
|
|
```
|
|
|
|
The raw OCR text is retained in `ocr_raw_text`, but the parsed metadata fields are not updated.
|
|
|
|
This prevents uncertain OCR output from silently replacing valid metadata.
|
|
|
|
## Inspecting OCR results
|
|
|
|
Show the structured OCR result:
|
|
|
|
```bash
|
|
sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT
|
|
id,
|
|
name,
|
|
nationality,
|
|
shoot_location,
|
|
shoot_date,
|
|
ocr_status,
|
|
ocr_error,
|
|
ocr_processed_at
|
|
FROM movie
|
|
WHERE id = 'susana-melo_6707';"
|
|
```
|
|
|
|
Show the raw OCR text:
|
|
|
|
```bash
|
|
sqlite3 /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT ocr_raw_text
|
|
FROM movie
|
|
WHERE id = 'susana-melo_6707';"
|
|
```
|
|
|
|
List movies waiting for OCR:
|
|
|
|
```bash
|
|
sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT id, name, ocr_status
|
|
FROM movie
|
|
WHERE ocr_status = 'pending'
|
|
ORDER BY published DESC;"
|
|
```
|
|
|
|
List movies requiring manual review:
|
|
|
|
```bash
|
|
sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT id, name, ocr_error
|
|
FROM movie
|
|
WHERE ocr_status = 'manual_review';"
|
|
```
|
|
|
|
List failed OCR attempts:
|
|
|
|
```bash
|
|
sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \
|
|
"SELECT id, name, ocr_error
|
|
FROM movie
|
|
WHERE ocr_status = 'failed';"
|
|
```
|
|
|
|
## Recreating the database
|
|
|
|
During development, the database can be recreated from the schema and site JSON:
|
|
|
|
```bash
|
|
rm /storage/disk1/WCX/database/wcx.db
|
|
|
|
sqlite3 /storage/disk1/WCX/database/wcx.db \
|
|
< /storage/disk1/WCX/scripts/schema.sql
|
|
|
|
/storage/disk1/WCX/scripts/import_site.py
|
|
```
|
|
|
|
Do not recreate the database this way after it contains manually maintained metadata unless a backup has been created first.
|
|
|
|
OCR data must also be recreated if the database is deleted.
|
|
|
|
## Git and generated data
|
|
|
|
The Git repository contains source code, schema, and documentation.
|
|
|
|
The SQLite database and scraper output are runtime data and are not version-controlled.
|
|
|
|
Current `.gitignore` rules:
|
|
|
|
```gitignore
|
|
# SQLite database files
|
|
database/*.db
|
|
database/*.db-shm
|
|
database/*.db-wal
|
|
|
|
# Imported/generated data
|
|
import/*.json
|
|
|
|
# Python cache
|
|
__pycache__/
|
|
*.pyc
|
|
```
|
|
|
|
Files that should be version-controlled include:
|
|
|
|
```text
|
|
README.md
|
|
.gitignore
|
|
scripts/schema.sql
|
|
scripts/import_site.py
|
|
scripts/ocr.sh
|
|
scripts/parse_ocr.py
|
|
scripts/check_ocr.py
|
|
```
|
|
|
|
Files that should not be version-controlled include:
|
|
|
|
```text
|
|
database/wcx.db
|
|
database/wcx.db-shm
|
|
database/wcx.db-wal
|
|
import/wcx_site_index.json
|
|
```
|
|
|
|
## Current limitations
|
|
|
|
The current OCR parser assumes a four-line OCR result.
|
|
|
|
Google Cloud Vision does not guarantee this exact structure. Unexpected output is therefore stored for manual review rather than automatically accepted.
|
|
|
|
OCR processing is currently started manually for one movie ID at a time.
|
|
|
|
## Planned next steps
|
|
|
|
Potential next steps include:
|
|
|
|
* automatically process movies with `ocr_status = pending`
|
|
* add retry support for failed OCR requests
|
|
* provide commands for manually approving or correcting OCR results
|
|
* add database backup handling
|
|
* export the complete index to JSON or CSV
|
|
* add Emby-compatible metadata export
|
|
* schedule the scraper and import process
|
|
* create an orchestration script for the complete automated workflow
|