From f34f74697bdbc5dcb9d4924fa1f1c36ba1c15536 Mon Sep 17 00:00:00 2001 From: Urban Date: Fri, 17 Jul 2026 18:03:20 +0200 Subject: [PATCH] Document OCR processing workflow --- README.md | 509 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 405 insertions(+), 104 deletions(-) diff --git a/README.md b/README.md index b7d3c79..dedc2da 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,40 @@ # WCX Index -Lokalt index för WCX-publiceringar. +Local SQLite-based index for WCX publications. -Projektet bygger ett SQLite-baserat index från JSON-data som genereras av det befintliga scraper-scriptet. +The project imports metadata produced by an existing site scraper, stores it in a SQLite database, and enriches new records with metadata extracted from thumbnail images using Google Cloud Vision OCR. -## Nuvarande flöde +## Current processing flow ```text -WCX-site +WCX site ↓ -befintligt scraper-script +existing scraper ↓ import/wcx_site_index.json ↓ scripts/import_site.py ↓ database/wcx.db + ↓ +scripts/check_ocr.py + ├── scripts/ocr.sh + └── scripts/parse_ocr.py + ↓ +OCR metadata stored in SQLite ``` -Importscriptet: +The current implementation supports: -* lägger till nya filmer -* uppdaterar befintliga filmer när sitens `update`-datum är nyare -* lämnar oförändrade filmer orörda -* lagrar speltid som antal sekunder +* inserting new movies from the scraper JSON +* updating existing movies based on the site's `update` field +* storing movie duration as seconds +* running OCR for an individual movie +* parsing and validating OCR output +* comparing the OCR name with the database name +* storing OCR metadata, raw OCR text, status, and errors -## Katalogstruktur +## Directory structure ```text /storage/disk1/WCX/ @@ -33,79 +42,117 @@ Importscriptet: │ └── wcx.db ├── import/ │ └── wcx_site_index.json -└── scripts/ - ├── schema.sql - └── import_site.py +├── scripts/ +│ ├── schema.sql +│ ├── import_site.py +│ ├── ocr.sh +│ ├── parse_ocr.py +│ └── check_ocr.py +├── .gitignore +└── README.md ``` -## Förutsättningar +## Requirements -Projektet använder: +The project currently uses: * Python 3 * SQLite 3 -* jq, valfritt för inspektion av JSON +* Bash +* curl +* jq +* base64 +* Google Cloud Vision API access -På Ubuntu/Debian: +Install the local command-line dependencies on Ubuntu or Debian: ```bash sudo apt update -sudo apt install sqlite3 jq +sudo apt install sqlite3 jq curl ``` -Python-modulerna `json`, `sqlite3` och `pathlib` ingår i Pythons standardbibliotek. Inga externa Python-paket behövs. +The Python scripts only use modules from the Python standard library. -## Skapa databasen +## Google Vision API key -Databasen skapas från `schema.sql`. +The OCR script expects the Google Vision API key in the environment: + +```bash +export GOOGLE_VISION_API_KEY="your-api-key" +``` + +The key must be available in the environment whenever `ocr.sh` or `check_ocr.py` is executed. + +Do not commit the API key to Git. + +## Creating the database + +Create the database directory: ```bash mkdir -p /storage/disk1/WCX/database +``` +Create the database from the schema: + +```bash sqlite3 /storage/disk1/WCX/database/wcx.db \ < /storage/disk1/WCX/scripts/schema.sql ``` -Kontrollera tabellen: +Inspect the resulting table: ```bash sqlite3 /storage/disk1/WCX/database/wcx.db ".schema movie" ``` -## Databasmodell +## Database model -Tabellen `movie` innehåller: +The `movie` table currently contains the following columns: -| Kolumn | Beskrivning | -| ------------------ | ------------------------------------------ | -| `id` | Stabilt och unikt ID från siten | -| `name` | Filmens titel | -| `aka` | Alternativt namn | -| `nationality` | Nationalitet | -| `age` | Ålder | -| `shoot_location` | Inspelningsplats | -| `shoot_date` | Inspelningsdatum | -| `duration_seconds` | Speltid i sekunder | -| `description` | Manuell beskrivning | -| `rating` | Manuell rating | -| `web_url` | Länk till detaljsidan | -| `thumbnail` | Länk till thumbnail | -| `published` | Ursprungligt publiceringsdatum | -| `updated` | Senaste uppdateringsdatum från siten | -| `created_at` | När posten skapades i databasen | -| `modified_at` | När posten senast uppdaterades i databasen | +| Column | Description | +| ------------------ | -------------------------------------------------- | +| `id` | Stable and unique ID from the site | +| `name` | Movie title | +| `aka` | Alternative name | +| `nationality` | Nationality extracted by OCR or edited manually | +| `age` | Manually maintained age | +| `shoot_location` | Shoot location extracted by OCR or edited manually | +| `shoot_date` | Shoot date stored as `YYYY-MM-DD` | +| `duration_seconds` | Movie duration stored in seconds | +| `description` | Manually maintained description | +| `rating` | Manually maintained rating | +| `web_url` | URL to the movie detail page | +| `thumbnail` | URL to the thumbnail image | +| `published` | Original publication date | +| `updated` | Latest site update date | +| `created_at` | Timestamp when the database record was created | +| `modified_at` | Timestamp when the record was last modified | +| `ocr_status` | Current OCR processing status | +| `ocr_raw_text` | Unmodified OCR text returned by Google Vision | +| `ocr_error` | OCR or validation error | +| `ocr_processed_at` | Timestamp of the latest OCR attempt | -## JSON-indata +Possible OCR statuses are currently: -Importscriptet läser: +```text +pending +completed +failed +manual_review +``` + +## Site JSON input + +The site importer reads: ```text /storage/disk1/WCX/import/wcx_site_index.json ``` -Filen ska innehålla en JSON-array. +The file must contain a JSON array. -Exempel: +Example entry: ```json { @@ -118,7 +165,7 @@ Exempel: } ``` -Vid en uppdaterad publicering kan även följande fält finnas: +An updated publication may also contain: ```json { @@ -126,44 +173,44 @@ Vid en uppdaterad publicering kan även följande fält finnas: } ``` -Importscriptet accepterar både `update` och `updated`. +The importer accepts both `update` and `updated`. -## Fältmappning +## Site field mapping -| JSON-fält | Databaskolumn | -| ------------------------ | ------------------ | -| `id` | `id` | -| `titel` | `name` | -| `details` | `web_url` | -| `duration` | `duration_seconds` | -| `thumb` | `thumbnail` | -| `published` | `published` | -| `update` eller `updated` | `updated` | +| JSON field | Database column | +| --------------------- | ------------------ | +| `id` | `id` | +| `titel` | `name` | +| `details` | `web_url` | +| `duration` | `duration_seconds` | +| `thumb` | `thumbnail` | +| `published` | `published` | +| `update` or `updated` | `updated` | -Speltiden konverteras till sekunder. +Durations are converted to seconds. -Exempel: +Examples: ```text 37:35 → 2255 1:37:00 → 5820 ``` -## Kör importen +## Importing site data -Gör scriptet körbart om det inte redan är gjort: +Make the importer executable: ```bash chmod +x /storage/disk1/WCX/scripts/import_site.py ``` -Kör importen: +Run the import: ```bash /storage/disk1/WCX/scripts/import_site.py ``` -Exempel på resultat: +Example output: ```text Poster i JSON: 1250 @@ -172,24 +219,30 @@ Uppdaterade: 1 Oförändrade: 1246 ``` -## Importregler +## Site import rules -### Ny film +### New movie -Om filmens ID inte finns i databasen skapas en ny post. +If the movie ID does not exist in the database, a new row is inserted. -### Befintlig film utan `update` +New rows receive the default OCR status: -Om filmen redan finns och site-posten saknar `update` görs ingen ändring. +```text +pending +``` -### Befintlig film med `update` +### Existing movie without a site update -Posten uppdateras när: +If the movie already exists and the site entry has no `update` date, no database changes are made. -* databasen saknar ett `updated`-värde, eller -* sitens `update`-datum är nyare än databasens `updated`-värde +### Existing movie with a site update -Vid en uppdatering skrivs följande sitefält över: +An existing row is updated when: + +* the database has no `updated` value and the site does, or +* the site's update date is newer than the database value + +When an update is triggered, all fields supplied by the site are replaced: * `name` * `duration_seconds` @@ -198,20 +251,20 @@ Vid en uppdatering skrivs följande sitefält över: * `published` * `updated` -ID och manuella metadatafält påverkas inte. +The movie ID and manually maintained metadata fields are left unchanged. -Datumen lagras som `YYYY-MM-DD`, vilket gör att de kan jämföras kronologiskt som text. +Dates use the ISO format `YYYY-MM-DD`, allowing chronological comparison as text. -## Inspektera databasen +## Inspecting the database -Visa antal filmer: +Show the total number of movies: ```bash sqlite3 /storage/disk1/WCX/database/wcx.db \ "SELECT COUNT(*) FROM movie;" ``` -Visa några poster: +Show some movies: ```bash sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ @@ -220,7 +273,7 @@ sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ LIMIT 10;" ``` -Visa en specifik film: +Show a specific movie: ```bash sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ @@ -229,29 +282,238 @@ sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ WHERE id = 'susana-melo_6707';" ``` -## Testa uppdateringslogiken +## OCR processing -Ett enkelt test är att medvetet ändra en post: +OCR processing is currently performed for one movie at a time. + +The complete flow is: + +```text +movie ID + ↓ +check_ocr.py + ↓ +thumbnail URL loaded from SQLite + ↓ +ocr.sh + ↓ +Google Cloud Vision raw text + ↓ +parse_ocr.py + ↓ +name validation + ↓ +SQLite update +``` + +## Raw OCR script + +Run OCR directly against an image URL: ```bash -sqlite3 /storage/disk1/WCX/database/wcx.db \ - "UPDATE movie - SET duration_seconds = 1, - updated = NULL +/storage/disk1/WCX/scripts/ocr.sh \ + "https://example.com/thumbnail.jpg" +``` + +Example output: + +```text +SUSANA MELO +PORTUGUESE +WOODMAN CASTINGX.COM +Budapest (Hungary), March 9, 2014 +``` + +The raw OCR output is not considered a stable data format and must be parsed and validated before metadata is stored. + +## Expected OCR structure + +The parser currently expects exactly four non-empty lines: + +```text +Line 1: Performer name +Line 2: Nationality +Line 3: Branding text, ignored +Line 4: Shoot location and shoot date +``` + +Example final line: + +```text +Budapest (Hungary), March 9, 2014 +``` + +The line is split on the first comma: + +```text +shoot_location = Budapest (Hungary) +shoot_date = March 9, 2014 +``` + +The date is normalized before storage: + +```text +March 9, 2014 → 2014-03-09 +``` + +## Testing the OCR parser + +Raw OCR output can be piped directly into the parser: + +```bash +/storage/disk1/WCX/scripts/ocr.sh \ + "https://example.com/thumbnail.jpg" \ + | /storage/disk1/WCX/scripts/parse_ocr.py +``` + +Example result: + +```text +ocr_name=SUSANA MELO +nationality=PORTUGUESE +shoot_location=Budapest (Hungary) +shoot_date=2014-03-09 +``` + +## Processing OCR for a movie + +Run the complete OCR workflow using a movie ID: + +```bash +/storage/disk1/WCX/scripts/check_ocr.py susana-melo_6707 +``` + +Example output: + +```text +id=susana-melo_6707 +database_name=Susana Melo +ocr_name=SUSANA MELO +name_match=yes +nationality=PORTUGUESE +shoot_location=Budapest (Hungary) +shoot_date=2014-03-09 +ocr_status=completed +``` + +The name comparison is case-insensitive and ignores repeated whitespace. + +For example: + +```text +Susana Melo +SUSANA MELO +``` + +are considered equal. + +## Successful OCR processing + +When OCR execution and parsing succeed and the OCR name matches the database name, the following fields are updated: + +```text +nationality +shoot_location +shoot_date +ocr_raw_text +ocr_status = completed +ocr_error = NULL +ocr_processed_at +modified_at +``` + +## OCR failure handling + +If the Google Vision request or `ocr.sh` execution fails: + +```text +ocr_status = failed +``` + +The error is stored in: + +```text +ocr_error +``` + +## Manual review handling + +The movie is marked for manual review when: + +* the OCR output cannot be parsed +* the expected fields are missing +* the OCR name does not match the database name +* the OCR output has an unexpected structure + +In this case: + +```text +ocr_status = manual_review +``` + +The raw OCR text is retained in `ocr_raw_text`, but the parsed metadata fields are not updated. + +This prevents uncertain OCR output from silently replacing valid metadata. + +## Inspecting OCR results + +Show the structured OCR result: + +```bash +sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ + "SELECT + id, + name, + nationality, + shoot_location, + shoot_date, + ocr_status, + ocr_error, + ocr_processed_at + FROM movie WHERE id = 'susana-melo_6707';" ``` -Kör sedan importen igen: +Show the raw OCR text: ```bash -/storage/disk1/WCX/scripts/import_site.py +sqlite3 /storage/disk1/WCX/database/wcx.db \ + "SELECT ocr_raw_text + FROM movie + WHERE id = 'susana-melo_6707';" ``` -Posten ska då uppdateras från JSON-filen och `duration_seconds` ska återställas till sitens värde. +List movies waiting for OCR: -## Återskapa databasen +```bash +sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ + "SELECT id, name, ocr_status + FROM movie + WHERE ocr_status = 'pending' + ORDER BY published DESC;" +``` -Under utveckling kan databasen enkelt återskapas eftersom all automatisk site-data kan importeras igen. +List movies requiring manual review: + +```bash +sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ + "SELECT id, name, ocr_error + FROM movie + WHERE ocr_status = 'manual_review';" +``` + +List failed OCR attempts: + +```bash +sqlite3 -header -column /storage/disk1/WCX/database/wcx.db \ + "SELECT id, name, ocr_error + FROM movie + WHERE ocr_status = 'failed';" +``` + +## Recreating the database + +During development, the database can be recreated from the schema and site JSON: ```bash rm /storage/disk1/WCX/database/wcx.db @@ -262,31 +524,70 @@ sqlite3 /storage/disk1/WCX/database/wcx.db \ /storage/disk1/WCX/scripts/import_site.py ``` -Radera inte databasen på detta sätt när den senare innehåller manuellt registrerad metadata utan att först ta en backup. +Do not recreate the database this way after it contains manually maintained metadata unless a backup has been created first. -## Versionshantering +OCR data must also be recreated if the database is deleted. -Källkod och dokumentation ska versionshanteras i Git. +## Git and generated data -Databasen och importerade datafiler bör normalt inte checkas in. +The Git repository contains source code, schema, and documentation. -Föreslagen `.gitignore`: +The SQLite database and scraper output are runtime data and are not version-controlled. + +Current `.gitignore` rules: ```gitignore +# SQLite database files database/*.db database/*.db-shm database/*.db-wal + +# Imported/generated data import/*.json + +# Python cache __pycache__/ *.pyc ``` -## Nästa steg +Files that should be version-controlled include: -Planerade kommande delar: +```text +README.md +.gitignore +scripts/schema.sql +scripts/import_site.py +scripts/ocr.sh +scripts/parse_ocr.py +scripts/check_ocr.py +``` -* OCR-behandling av thumbnails för nya filmer -* manuell redigering av metadata -* export av komplett index -* integration med Emby -* schemalagd körning +Files that should not be version-controlled include: + +```text +database/wcx.db +database/wcx.db-shm +database/wcx.db-wal +import/wcx_site_index.json +``` + +## Current limitations + +The current OCR parser assumes a four-line OCR result. + +Google Cloud Vision does not guarantee this exact structure. Unexpected output is therefore stored for manual review rather than automatically accepted. + +OCR processing is currently started manually for one movie ID at a time. + +## Planned next steps + +Potential next steps include: + +* automatically process movies with `ocr_status = pending` +* add retry support for failed OCR requests +* provide commands for manually approving or correcting OCR results +* add database backup handling +* export the complete index to JSON or CSV +* add Emby-compatible metadata export +* schedule the scraper and import process +* create an orchestration script for the complete automated workflow