diff --git a/AGENTS.md b/AGENTS.md index 27d9b2d..46c8fb8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,13 +4,13 @@ This repository maintains a local SQLite index of WCX publications. Production code lives in `scripts/`: `wcx_sync.py` scrapes site metadata, `import_site.py` imports it into SQLite, and `update_wcx.sh` coordinates synchronization, import, and OCR. OCR is split across `process_pending_ocr.py`, `check_ocr.py`, `ocr.sh`, and `parse_ocr.py`. -`scripts/schema.sql` defines the database model. Generated databases belong in `database/`, while generated scraper output belongs in `import/`; both are ignored by Git. Historical source data is stored in `migration/`. Matching requirements and design notes live in `docs/`. The main executable test/diagnostic script is `scripts/test_duration_matching.py`. +`scripts/schema.sql` defines the database model. Generated databases belong in `database/`, while generated scraper output belongs in `import/`; both are ignored by Git. Historical source data is stored in `migration/`. Database design notes live in `docs/`. The main executable test/diagnostic script is `scripts/test_duration_matching.py`. ## Architectural boundaries -The repository is currently combined, but it is intended to be split into a reference-data project and a file-management project. The reference-data project owns the database schema, metadata, OCR, imports, history, and aliases, and it is the sole writer to the reference database. File-management code may open that database only in strict read-only mode using SQLite `mode=ro`. +This repository is the reference-data project. Filename matching and `scripts/match_filenames.py` have moved to the separate `/storage/disk1/WCX-collection` repository; this repository no longer contains or runs filename matching. The reference-data project owns the database schema, metadata, OCR, imports, history, and aliases, and it is the sole writer to the reference database. -Local file status, paths, matching decisions, and collection status must never be stored in the reference database. `scripts/match_filenames.py` belongs to the future file-management part and should later move to the separate project. Any future file-management registry or database must be owned separately and may refer to `movie.id` as the stable external ID. Do not introduce a shared Python library or API without a concrete need. +WCX-collection is an external read-only consumer of the reference database and may open it only in strict read-only mode using SQLite `mode=ro`. Local file status, paths, matching decisions, and collection status must never be stored in the reference database. Any future file-management registry or database must be owned separately and may refer to `movie.id` as the stable external ID. Do not introduce a shared Python library or API without a concrete need. ## Build, Test, and Development Commands @@ -21,7 +21,6 @@ There is no build step or third-party Python package installation; scripts use P - `scripts/migrate_csv.py --dry-run` validates legacy CSV data without modifying SQLite. - `scripts/test_duration_matching.py` exercises duration-matching behavior against the configured database and media paths. - `python3 -m py_compile scripts/*.py` performs a quick syntax check. -- `scripts/match_filenames.py /path/to/videos --recursive --debug` diagnoses filename matches; it requires `ffprobe`. Use a copied test database for commands that can write data. Never recreate or overwrite `database/wcx.db` casually because it contains manually maintained metadata. diff --git a/docs/filename-matching-requirements.md b/docs/filename-matching-requirements.md deleted file mode 100644 index d34501d..0000000 --- a/docs/filename-matching-requirements.md +++ /dev/null @@ -1,892 +0,0 @@ -# Kravspecifikation: Matchning av lokala videofiler mot WCX-databasen - -## 1. Syfte - -Funktionen ska identifiera vilken post i WCX-databasen en lokal videofil sannolikt motsvarar. - -Databasen innehåller filmens kanoniska namn enligt siten. Namnet består alltid av ett eller flera personnamn och inga andra beskrivande ord. - -Lokala filnamn är däremot ofta inkonsekventa och kan innehålla: - -* sammanfogade namn -* understreck eller andra separatorer -* upprepade namn -* felstavningar -* extra beskrivande ord -* tekniska eller innehållsrelaterade suffix -* flera personnamn -* inkonsekvent användning av stora och små bokstäver - -Exempel: - -```text -Alyn_Borav_AlynBorav.mp4 -AngeliqueLapiedra_ScarlettLapiedra.mp4 -BrendaBoop_Interwiev_Fisting.mp4 -Busty_Slovakian_Ilona_On_Bed.mp4 -Eden_Ivy_EdenIvy_DP.mp4 -Version_AngelikaFyres_DP.mp4 -``` - -Dessa ska kunna matchas mot databasnamn som: - -```text -Alyn Borav -Angelique Lapiedra + Scarlett Lapiedra -Brenda Boop -Ilona -Eden Ivy -Angelika Fyres -``` - -Funktionen ska prioritera korrekthet framför täckning. En fil som inte kan identifieras med tillräcklig säkerhet ska lämnas för manuell inspektion. - ---- - -## 2. Grundprinciper - -Matchningen ska vara konservativ. - -Funktionen ska inte försöka automatcha varje fil. Den ska klassificera resultatet i tre huvudkategorier: - -```text -matched -ambiguous -unmatched -``` - -Betydelse: - -* `matched`: en kandidat är tillräckligt stark och tydligt bättre än övriga kandidater -* `ambiguous`: en eller flera rimliga kandidater finns, men säkerheten är inte tillräcklig -* `unmatched`: ingen rimlig kandidat kunde identifieras - -En felaktig automatisk match är allvarligare än att en fil lämnas omatchad. - ---- - -## 3. Datakällor - -Funktionen ska använda följande datakällor. - -### 3.1 Aktuella filmposter - -Från tabellen `movie`: - -```text -id -name -duration_seconds -``` - -Övriga fält kan senare användas, men krävs inte för den första versionen. - -### 3.2 Historiska filmposter - -Från tabellen `movie_history`: - -```text -movie_id -name -duration_seconds -archived_at -change_summary -``` - -Historiska längder kan användas för att identifiera att en lokal fil motsvarar en äldre version av samma film. - -### 3.3 Lokala videofiler - -För varje fil ska minst följande information läsas: - -```text -path -filename -file extension -actual duration -``` - -Den faktiska längden hämtas med `ffprobe`. - ---- - -## 4. Namnmodell - -Databasfältet `movie.name` innehåller alltid ett eller flera personnamn. - -Exempel med en person: - -```text -Alyn Borav -Ilona -Eden Ivy -``` - -Exempel med flera personer: - -```text -Angelique Lapiedra + Scarlett Lapiedra -Susanna Melo + Cherry Sweet -``` - -Plustecknet ska betraktas som separator mellan personer. - -Varje databaspost ska därför kunna representeras som: - -```text -movie name -list of person names -normalized full name -normalized person names -``` - -Exempel: - -```text -Movie name: -Angelique Lapiedra + Scarlett Lapiedra - -Persons: -- Angelique Lapiedra -- Scarlett Lapiedra - -Normalized persons: -- angeliquelapiedra -- scarlettlapiedra - -Normalized full name: -angeliquelapiedrascarlettlapiedra -``` - ---- - -## 5. Normalisering av namn - -Filnamn och databasnamn ska normaliseras innan jämförelse. - -Normaliseringen ska minst: - -1. ta bort filändelsen -2. konvertera till gemener -3. ta bort mellanslag -4. ta bort understreck -5. ta bort bindestreck -6. ta bort plustecken -7. ta bort punkter och annan interpunktion -8. normalisera diakritiska tecken -9. behålla bokstäver och siffror - -Exempel: - -```text -Alyn_Borav_AlynBorav.mp4 -→ alynboravalynborav -``` - -```text -Angelique Lapiedra + Scarlett Lapiedra -→ angeliquelapiedrascarlettlapiedra -``` - -```text -CatherineBoss.mp4 -→ catherineboss -``` - -```text -Catherine Boss -→ catherineboss -``` - ---- - -## 6. Matchning av filnamn - -### 6.1 Exakt normaliserad delsträng - -Om ett normaliserat databasnamn förekommer exakt i det normaliserade filnamnet ska detta räknas som en stark signal. - -Exempel: - -```text -Fil: -Eden_Ivy_EdenIvy_DP.mp4 - -Normaliserad fil: -edenivyedenivydp - -Databasnamn: -Eden Ivy - -Normaliserat namn: -edenivy -``` - -`edenivy` förekommer två gånger och ska ge en stark matchning. - -### 6.2 Upprepade namn - -Upprepade personnamn i filnamnet ska inte tolkas som flera personer. - -Exempel: - -```text -Vanessa_Rodriguez_VanessaRodriguez.mp4 -``` - -ska fortfarande motsvara personen: - -```text -Vanessa Rodriguez -``` - -Upprepning kan däremot stärka matchningen. - -### 6.3 Extra ord i filnamnet - -Extra ord behöver inte tas bort för att exakt delsträngsmatchning ska fungera. - -Exempel: - -```text -BrendaBoop_Interwiev_Fisting.mp4 -``` - -kan matchas mot: - -```text -Brenda Boop -``` - -eftersom det normaliserade namnet förekommer intakt i filnamnet. - -Exempel på vanliga extra ord: - -```text -DP -Group -Interview -Fisting -OralSex -TruFun67 -Version -OnBed -PissingInMouth -``` - -Dessa ska initialt behandlas som okänd extrainformation och inte som del av databasnamnet. - -### 6.4 Korta namn - -Korta namn som exempelvis: - -```text -Ilona -Mia -Eva -Ana -``` - -kräver större försiktighet eftersom de kan förekomma som delar av andra ord. - -För korta namn ska tokenbaserad matchning väga tyngre än ren delsträngsmatchning. - -### 6.5 Flera personer - -Om en databaspost innehåller flera personer ska samtliga namn kunna sökas oberoende av ordning. - -Exempel: - -```text -Angelique Lapiedra + Scarlett Lapiedra -``` - -ska kunna matcha både: - -```text -AngeliqueLapiedra_ScarlettLapiedra.mp4 -``` - -och: - -```text -ScarlettLapiedra_AngeliqueLapiedra.mp4 -``` - -Matchningen ska därför inte kräva att personerna förekommer i samma ordning som i databasen. - -### 6.6 Längsta och mest specifika kandidat - -Om flera databasnamn matchar ska den mest specifika kandidaten prioriteras. - -Exempel: - -```text -Scarlett Knight -Scarlett Knight + Anya Shidlerova -``` - -Fil: - -```text -ScarlettKnight_AnyaShidlerova.mp4 -``` - -Posten med båda personerna ska rankas högre. - -### 6.7 Exakt personmängd - -När flera kända personer identifieras i filnamnet ska databaskandidater med exakt samma personmängd prioriteras. - -Exempel: - -```text -Filnamn: -Susanna_Melo_Cherry_Sweet.mp4 - -Identifierade personer: -- Susanna Melo -- Cherry Sweet -``` - -Följande kandidat ska rankas högre: - -```text -Susanna Melo + Cherry Sweet -``` - -än: - -```text -Susanna Melo -``` - ---- - -## 7. Stavfel och fuzzy matching - -Filnamn kan innehålla stavfel. - -Exempel: - -```text -SusannaMello -Susamma_Melo -``` - -kan sannolikt motsvara: - -```text -Susanna Melo -``` - -Fuzzy matching får användas för att skapa kandidater, men ska vara konservativt. - -I den första versionen gäller: - -* fuzzy matching får inte ensam leda till automatisk match -* fuzzy-resultat ska normalt klassificeras som `ambiguous` -* kandidat och orsak ska visas för manuell bedömning - -Exempel: - -```text -Filename: -SusannaMello.mp4 - -Candidate: -Susanna Melo - -Status: -ambiguous - -Reason: -fuzzy name match -``` - -Verifierade stavningsvarianter kan senare sparas som alias. - -Ett alias ska endast registreras efter manuell bekräftelse. - ---- - -## 8. Längd som sekundär signal - -Videons faktiska längd ska användas för att stärka eller försvaga redan rimliga namnkandidater. - -Längden ska inte ensam kunna identifiera en film. - -Exempel: - -```text -File: -Alyn_Borav_AlynBorav.mp4 - -File duration: -43:00 -``` - -Kandidater: - -```text -Alyn Borav -Site duration: 43:05 -Difference: 5 seconds -``` - -```text -Alyn Borav + Cherry Sweet -Site duration: 1:54:00 -Difference: 71 minutes -``` - -Den första kandidaten ska då få betydligt högre poäng. - -### 8.1 Absolut och relativ skillnad - -Både absolut skillnad och procentuell skillnad ska beräknas. - -```text -absolute difference = abs(file duration - site duration) -relative difference = absolute difference / site duration -``` - -### 8.2 Initial poängmodell för längd - -En första approximation kan vara: - -```text -≤ 15 sekunder eller ≤ 1 % mycket stark signal -≤ 60 sekunder eller ≤ 2 % stark signal -≤ 3 minuter eller ≤ 5 % rimlig signal -≤ 10 minuter eller ≤ 10 % svag signal -> 10 minuter och > 10 % negativ signal -``` - -Exakt poängsättning ska kunna justeras efter testning mot verkliga filer. - -### 8.3 Längd får inte rädda en dålig namnmatch - -Exempel: - -```text -unknown_video.mp4 -``` - -ska inte automatchas till en film bara för att längden råkar vara nästan identisk. - -Längd används endast för att diskriminera mellan kandidater som redan har en rimlig namnmatchning. - -### 8.4 Klippta och förlängda versioner - -En större längdskillnad ska inte automatiskt innebära att kandidaten är fel. - -Filen kan vara: - -```text -truncated -extended -edited -missing intro -missing ending -``` - -En stark namnmatchning kombinerad med tydlig längdavvikelse ska normalt ge: - -```text -ambiguous -``` - -eller en särskild versionsklassificering, inte automatiskt `unmatched`. - ---- - -## 9. Historiska längder - -Matchningen ska kunna jämföra filens längd mot både aktuell och historisk sitelängd. - -Exempel: - -```text -Current site duration: 63:05 -Historical duration: 43:05 -Local file duration: 43:00 -``` - -Möjlig slutsats: - -```text -Filmen matchar sannolikt rätt databaspost men motsvarar en äldre version. -``` - -Möjliga versionsklassificeringar: - -```text -current_version_match -historical_version_match -possible_truncated_version -possible_extended_version -duration_mismatch -unknown_version -``` - -Historisk längd ska vara en stödjande signal och inte ensam avgöra filmidentiteten. - ---- - -## 10. Kandidatpoäng - -Varje databaspost ska kunna få en sammanlagd poäng baserad på flera signaler. - -Exempel på positiva signaler: - -```text -exakt normaliserat fullständigt namn -samtliga personnamn hittade -exakt personmängd -namnet förekommer flera gånger -längden ligger mycket nära aktuell sitelängd -längden ligger mycket nära historisk sitelängd -stor marginal till kandidat nummer två -``` - -Exempel på negativa signaler: - -```text -bara en del av namnet hittades -fuzzy matching krävdes -filnamnet innehåller ytterligare kända personer -databasposten förväntar flera personer men endast en hittas -stor skillnad i längd -kort och osäker namnträff -flera kandidater har nästan samma poäng -``` - -Poängmodellen ska vara förklarbar. Resultatet ska visa vilka signaler som påverkade poängen. - ---- - -## 11. Säkerhetskrav för automatisk matchning - -En fil får klassificeras som `matched` endast om: - -1. bästa kandidaten uppnår ett fastställt minimikrav -2. kandidaten bygger på en tillräckligt stark namnmatchning -3. kandidaten ligger tydligt före näst bästa kandidat -4. inga motstridiga personsignaler finns -5. eventuell fuzzy matching inte är den enda avgörande signalen - -Exempel på säker match: - -```text -Candidate 1: 170 -Candidate 2: 45 -Margin: 125 -``` - -Exempel på osäker match: - -```text -Candidate 1: 92 -Candidate 2: 87 -Margin: 5 -``` - -Det senare ska klassificeras som: - -```text -ambiguous -``` - -även om förstaplatsen har relativt hög poäng. - ---- - -## 12. Manuell inspektion - -Filer som klassificeras som `ambiguous` eller `unmatched` ska kunna granskas manuellt. - -För `ambiguous` ska rapporten visa de bästa kandidaterna. - -Exempel: - -```text -Filename: -SusannaMello_Cherry.mp4 - -Status: -ambiguous - -Candidate 1: -Susanna Melo + Cherry Sweet -Score: 91 - -Candidate 2: -Susanna Melo -Score: 86 - -Reasons: -- fuzzy match for Susanna Melo -- partial match for Cherry Sweet -- candidates have similar scores -``` - -Manuell granskning ska kunna resultera i: - -```text -confirmed match -rejected candidates -verified alias -ignored file -``` - -Manuella beslut ska kunna sparas senare, men första versionen behöver endast rapportera kandidater. - ---- - -## 13. Första versionens omfattning - -Den första versionen ska vara helt skrivskyddad. - -Den får: - -* läsa filer från en katalog -* läsa poster från SQLite-databasen -* läsa historiska längder -* köra `ffprobe` -* normalisera filnamn -* rangordna kandidater -* skriva rapporter - -Den får inte: - -* byta namn på filer -* flytta filer -* radera filer -* ändra databasen -* automatiskt registrera alias -* permanent koppla filer till filmer - ---- - -## 14. Förväntat kommando - -Exempel: - -```bash -scripts/match_filenames.py /path/to/video/files -``` - -Alternativt stöd för en textfil med filnamn: - -```bash -scripts/match_filenames.py \ - --file-list filenames.txt -``` - -Möjliga framtida argument: - -```text ---database ---recursive ---limit ---output-directory ---min-score ---min-margin ---debug ---no-ffprobe -``` - ---- - -## 15. Rapportformat - -Resultatet bör delas upp i minst tre rapporter: - -```text -matched.csv -ambiguous.csv -unmatched.csv -``` - -### 15.1 matched.csv - -Föreslagna kolumner: - -```text -path -filename -file_duration_seconds -movie_id -movie_name -site_duration_seconds -matched_duration_seconds -version_classification -score -score_margin -match_reason -``` - -### 15.2 ambiguous.csv - -Föreslagna kolumner: - -```text -path -filename -file_duration_seconds -candidate_1_id -candidate_1_name -candidate_1_score -candidate_2_id -candidate_2_name -candidate_2_score -reason -``` - -### 15.3 unmatched.csv - -Föreslagna kolumner: - -```text -path -filename -file_duration_seconds -best_candidate -best_score -reason -``` - ---- - -## 16. Konsolutskrift - -En läsbar konsolrapport ska också kunna visas. - -Exempel: - -```text -Alyn_Borav_AlynBorav.mp4 -File duration: 43:00 - -1. Alyn Borav - Name score: 120 - Duration score: 50 - Total: 170 - Duration diff: 5 sec - -2. Alyn Borav + Cherry Sweet - Name score: 65 - Duration score: -30 - Total: 35 - Duration diff: 71 min - -Result: matched -Version: current_version_match -``` - -Exempel på osäker fil: - -```text -SusannaMello_Cherry.mp4 -File duration: 42:58 - -1. Susanna Melo + Cherry Sweet - Total: 91 - -2. Susanna Melo - Total: 86 - -Result: ambiguous -Reason: fuzzy match and insufficient score margin -``` - ---- - -## 17. Prestanda - -Filnamnsmatchningen ska kunna genomföras utan att läsa videofilernas innehåll. - -`ffprobe` ska endast läsa metadata och ska inte omkoda eller spela upp videon. - -Första versionen behöver inte använda: - -* fullständiga filhashar -* perceptuella videofingeravtryck -* bildextraktion -* maskininlärning -* OCR från videobilder - -Sådana funktioner kan utvärderas senare för svåra fall. - ---- - -## 18. Framtida utökningar - -Möjliga senare förbättringar: - -### 18.1 Alias - -Verifierade stavningsvarianter kan sparas: - -```text -susannamello → Susanna Melo -susammamelo → Susanna Melo -``` - -Alias ska endast skapas efter manuell bekräftelse. - -### 18.2 Filtabell - -Lokala filer kan senare lagras i en tabell, exempelvis: - -```text -media_file -``` - -En film ska kunna ha flera lokala filer eftersom olika omkodningar och versioner kan förekomma. - -### 18.3 Manuella beslut - -Manuellt bekräftade matchningar ska inte behöva lösas igen. - -### 18.4 Visuella fingeravtryck - -För svåra fall kan ett perceptuellt videofingeravtryck skapas genom att: - -* extrahera bilder med intervall -* skala ned bilderna -* beräkna perceptuella hashvärden -* jämföra sekvenser av hashvärden - -Detta ligger utanför den första versionen. - ---- - -## 19. Acceptanskriterier för första versionen - -Den första versionen är godkänd när den kan: - -1. läsa alla namn och längder från databasen -2. dela flerpersonsnamn på `+` -3. normalisera databasnamn och filnamn -4. hitta exakta normaliserade namn i smutsiga filnamn -5. hantera upprepade namn -6. hantera namn utan separatorer -7. prioritera den mest specifika flerpersonsposten -8. läsa faktisk videolängd med `ffprobe` -9. jämföra mot aktuell sitelängd -10. jämföra mot historiska sitelängder -11. rangordna kandidater -12. klassificera filer som `matched`, `ambiguous` eller `unmatched` -13. lämna osäkra filer för manuell inspektion -14. skriva separata rapporter -15. inte ändra filer eller databas - -Det viktigaste kvalitetsmåttet är: - -```text -precision bland automatiskt matchade filer -``` - -Målet är nära noll felaktiga automatiska matchningar, även om det innebär att en betydande andel filer lämnas för manuell granskning. diff --git a/scripts/match_filenames.py b/scripts/match_filenames.py deleted file mode 100755 index 65ff7c7..0000000 --- a/scripts/match_filenames.py +++ /dev/null @@ -1,1371 +0,0 @@ -#!/usr/bin/env python3 - -""" -Match local video filenames against canonical WCX movie names. - -The script is deliberately conservative and read-only. - -Matching sources: - -- Current canonical name from movie.name -- Verified aliases from movie_name_alias -- Current duration from movie.duration_seconds -- Historical durations from movie_history.duration_seconds -- Actual file duration read with ffprobe - -Important principles: - -- Duration never creates a candidate without a name or alias match. -- A close duration may resolve an otherwise ambiguous name match. -- The least-wrong duration does not win when all candidates have poor matches. -- The current canonical movie name is always displayed. -- Files and database contents are never modified. - -Examples: - - match_filenames.py /storage/disk1/X \ - --database /path/to/wcx.db - - match_filenames.py /storage/disk1/X \ - --database /path/to/wcx.db \ - --recursive - - match_filenames.py /storage/disk1/X \ - --database /path/to/wcx.db \ - --ending mp4,avi - - match_filenames.py /storage/disk1/X \ - --database /path/to/wcx.db \ - --ending mp4 \ - --ending avi \ - --debug -""" - -import argparse -import shutil -import sqlite3 -import subprocess -import sys -import unicodedata -from dataclasses import dataclass, replace -from pathlib import Path - - -VIDEO_EXTENSIONS = { - ".avi", - ".m4v", - ".mkv", - ".mov", - ".mp4", - ".mpeg", - ".mpg", - ".ts", - ".webm", - ".wmv", -} - -MIN_NAME_SCORE = 100 -MIN_NAME_MARGIN = 15 - -# At least this score is considered credible duration support. -MIN_CREDIBLE_DURATION_SCORE = 10 - -# Difference between duration scores required to resolve several name matches. -MIN_DURATION_SCORE_MARGIN = 15 - - -@dataclass(frozen=True) -class MatchName: - """ - One searchable name belonging to a movie. - - source is normally: - current - manual - history - csv - """ - - display_name: str - normalized_name: str - normalized_persons: tuple[str, ...] - source: str - - -@dataclass(frozen=True) -class DurationVersion: - duration_seconds: int - source: str - archived_at: str | None = None - - -@dataclass(frozen=True) -class Movie: - movie_id: str - name: str - match_names: tuple[MatchName, ...] - duration_versions: tuple[DurationVersion, ...] - - -@dataclass(frozen=True) -class DurationMatch: - score: int - classification: str - matched_duration: int | None - difference_seconds: int | None - source: str | None - archived_at: str | None - - -@dataclass(frozen=True) -class MatchCandidate: - movie: Movie - matched_name: MatchName - - name_score: int - duration_score: int - total_score: int - - name_reason: str - duration_reason: str - - full_name_occurrences: int - all_persons_matched: bool - duration_match: DurationMatch - - -@dataclass(frozen=True) -class MatchResult: - status: str - best: MatchCandidate | None - margin: int - reason: str - detected_movies: tuple[str, ...] - - -def parse_arguments() -> argparse.Namespace: - parser = argparse.ArgumentParser( - description=( - "Match local video filenames against canonical WCX movie names." - ) - ) - - parser.add_argument( - "directory", - type=Path, - help="Directory containing video files.", - ) - - parser.add_argument( - "--database", - type=Path, - required=True, - help="SQLite database", - ) - - parser.add_argument( - "--recursive", - action="store_true", - help="Search recursively below the input directory.", - ) - - parser.add_argument( - "--ending", - action="append", - help=( - "Video file endings to include. Examples: " - "'--ending mp4,avi' or '--ending mp4 --ending avi'. " - "A leading dot is optional. " - "Default: all supported video endings." - ), - ) - - parser.add_argument( - "--limit", - type=int, - help="Analyze at most N files.", - ) - - parser.add_argument( - "--debug", - action="store_true", - help="Show candidates, scores, durations and classification details.", - ) - - return parser.parse_args() - - -def normalize_text(value: str) -> str: - decomposed = unicodedata.normalize("NFKD", value) - - without_diacritics = "".join( - character - for character in decomposed - if not unicodedata.combining(character) - ) - - return "".join( - character.casefold() - for character in without_diacritics - if character.isalnum() - ) - - -def split_person_names(value: str) -> tuple[str, ...]: - persons = [] - - for part in value.split("+"): - normalized = normalize_text(part) - - if normalized: - persons.append(normalized) - - return tuple(persons) - - -def normalize_duration(value: object) -> int | None: - if value is None: - return None - - try: - duration = int(value) - except (TypeError, ValueError): - return None - - if duration <= 0: - return None - - return duration - - -def parse_video_extensions( - ending_arguments: list[str] | None, -) -> set[str]: - """ - Parse --ending arguments. - - Supported examples: - - --ending mp4,avi - --ending .mp4,.avi - --ending mp4 --ending avi - - Without --ending, all extensions in VIDEO_EXTENSIONS are used. - """ - if not ending_arguments: - return set(VIDEO_EXTENSIONS) - - extensions: set[str] = set() - - for argument in ending_arguments: - for value in argument.split(","): - extension = value.strip().casefold() - - if not extension: - continue - - if not extension.startswith("."): - extension = f".{extension}" - - extensions.add(extension) - - if not extensions: - raise ValueError( - "--ending did not contain any valid file endings." - ) - - return extensions - - -def load_movies(database_file: Path) -> list[Movie]: - if not database_file.is_file(): - raise FileNotFoundError( - f"Database file does not exist: {database_file}" - ) - - database_uri = f"{database_file.resolve().as_uri()}?mode=ro" - - with sqlite3.connect(database_uri, uri=True) as connection: - connection.row_factory = sqlite3.Row - - movie_rows = connection.execute( - """ - SELECT - id, - name, - duration_seconds - FROM movie - WHERE name IS NOT NULL - AND TRIM(name) <> '' - ORDER BY name, id - """ - ).fetchall() - - alias_rows = connection.execute( - """ - SELECT - movie_id, - alias, - normalized_alias, - source - FROM movie_name_alias - WHERE alias IS NOT NULL - AND TRIM(alias) <> '' - ORDER BY movie_id, alias - """ - ).fetchall() - - history_rows = connection.execute( - """ - SELECT - movie_id, - duration_seconds, - archived_at - FROM movie_history - WHERE duration_seconds IS NOT NULL - AND duration_seconds > 0 - ORDER BY movie_id, archived_at - """ - ).fetchall() - - aliases_by_movie: dict[str, list[MatchName]] = {} - - for row in alias_rows: - movie_id = str(row["movie_id"]) - alias = str(row["alias"]) - - normalized_alias = ( - str(row["normalized_alias"]).strip() - if row["normalized_alias"] - else normalize_text(alias) - ) - - if not normalized_alias: - continue - - aliases_by_movie.setdefault( - movie_id, - [], - ).append( - MatchName( - display_name=alias, - normalized_name=normalized_alias, - normalized_persons=split_person_names(alias), - source=str(row["source"] or "alias"), - ) - ) - - historical_durations_by_movie: dict[ - str, - list[DurationVersion], - ] = {} - - for row in history_rows: - duration = normalize_duration( - row["duration_seconds"] - ) - - if duration is None: - continue - - movie_id = str(row["movie_id"]) - - historical_durations_by_movie.setdefault( - movie_id, - [], - ).append( - DurationVersion( - duration_seconds=duration, - source="historical", - archived_at=( - str(row["archived_at"]) - if row["archived_at"] is not None - else None - ), - ) - ) - - movies = [] - - for row in movie_rows: - movie_id = str(row["id"]) - current_name = str(row["name"]) - - current_match_name = MatchName( - display_name=current_name, - normalized_name=normalize_text(current_name), - normalized_persons=split_person_names(current_name), - source="current", - ) - - match_names = [current_match_name] - - seen_normalized_names = { - current_match_name.normalized_name - } - - for alias in aliases_by_movie.get(movie_id, []): - if alias.normalized_name in seen_normalized_names: - continue - - seen_normalized_names.add( - alias.normalized_name - ) - - match_names.append(alias) - - duration_versions: list[DurationVersion] = [] - seen_durations: set[int] = set() - - current_duration = normalize_duration( - row["duration_seconds"] - ) - - if current_duration is not None: - duration_versions.append( - DurationVersion( - duration_seconds=current_duration, - source="current", - ) - ) - - seen_durations.add(current_duration) - - for historical in historical_durations_by_movie.get( - movie_id, - [], - ): - if historical.duration_seconds in seen_durations: - continue - - seen_durations.add( - historical.duration_seconds - ) - - duration_versions.append(historical) - - movies.append( - Movie( - movie_id=movie_id, - name=current_name, - match_names=tuple(match_names), - duration_versions=tuple(duration_versions), - ) - ) - - return movies - - -def should_ignore_file(path: Path) -> bool: - """ - Ignore common operating-system metadata and resource-fork files. - - macOS commonly creates files such as: - - ._Video.mp4 - .DS_Store - - AppleDouble files may have a valid video extension but are not videos. - """ - name = path.name - - if name.startswith("._"): - return True - - if name in { - ".DS_Store", - "Thumbs.db", - "desktop.ini", - }: - return True - - return False - - -def find_video_files( - directory: Path, - recursive: bool, - extensions: set[str], -) -> list[Path]: - if not directory.is_dir(): - raise NotADirectoryError( - f"Input directory does not exist: {directory}" - ) - - iterator = ( - directory.rglob("*") - if recursive - else directory.glob("*") - ) - - files = [] - - for path in iterator: - if not path.is_file(): - continue - - if should_ignore_file(path): - continue - - if path.suffix.casefold() not in extensions: - continue - - files.append(path) - - return sorted( - files, - key=lambda path: str(path).casefold(), - ) - - -def read_file_duration(file_path: Path) -> int | None: - result = subprocess.run( - [ - "ffprobe", - "-v", - "error", - "-show_entries", - "format=duration", - "-of", - "default=noprint_wrappers=1:nokey=1", - str(file_path), - ], - capture_output=True, - text=True, - check=False, - ) - - if result.returncode != 0: - return None - - output = result.stdout.strip() - - if not output: - return None - - try: - duration = round(float(output)) - except ValueError: - return None - - return duration if duration > 0 else None - - -def calculate_duration_score( - difference_seconds: int, - reference_duration: int, -) -> int: - relative_difference = ( - difference_seconds / reference_duration - if reference_duration > 0 - else 1.0 - ) - - if difference_seconds <= 15: - return 50 - - if difference_seconds <= 60: - return 40 - - if difference_seconds <= 180: - return 25 - - if relative_difference <= 0.05: - return 20 - - if difference_seconds <= 600: - return 10 - - return 0 - - -def classify_poor_duration( - file_duration: int, - reference_duration: int, -) -> str: - if file_duration < reference_duration: - return "shorter_than_known_version" - - if file_duration > reference_duration: - return "longer_than_known_version" - - return "duration_mismatch" - - -def match_duration( - file_duration: int | None, - movie: Movie, -) -> DurationMatch: - if file_duration is None: - return DurationMatch( - score=0, - classification="file_duration_unknown", - matched_duration=None, - difference_seconds=None, - source=None, - archived_at=None, - ) - - if not movie.duration_versions: - return DurationMatch( - score=0, - classification="database_duration_unknown", - matched_duration=None, - difference_seconds=None, - source=None, - archived_at=None, - ) - - scored_versions = [] - - for version in movie.duration_versions: - difference = abs( - file_duration - version.duration_seconds - ) - - score = calculate_duration_score( - difference_seconds=difference, - reference_duration=version.duration_seconds, - ) - - scored_versions.append( - ( - score, - -difference, - version.source == "current", - version, - difference, - ) - ) - - ( - score, - _negative_difference, - _prefer_current, - best_version, - difference, - ) = max(scored_versions) - - if score > 0: - classification = ( - "current_version_match" - if best_version.source == "current" - else "historical_version_match" - ) - else: - classification = classify_poor_duration( - file_duration=file_duration, - reference_duration=best_version.duration_seconds, - ) - - return DurationMatch( - score=score, - classification=classification, - matched_duration=best_version.duration_seconds, - difference_seconds=difference, - source=best_version.source, - archived_at=best_version.archived_at, - ) - - -def score_match_name( - normalized_filename: str, - movie: Movie, - match_name: MatchName, -) -> MatchCandidate | None: - full_name_occurrences = normalized_filename.count( - match_name.normalized_name - ) - - matched_persons = [ - person - for person in match_name.normalized_persons - if person in normalized_filename - ] - - all_persons_matched = ( - bool(match_name.normalized_persons) - and len(matched_persons) - == len(match_name.normalized_persons) - ) - - if full_name_occurrences == 0 and not all_persons_matched: - return None - - name_score = 0 - reasons = [] - - if full_name_occurrences: - name_score += 80 - - if match_name.source == "current": - reasons.append("exact current-name match") - else: - reasons.append( - f"exact alias match: {match_name.display_name}" - ) - - if full_name_occurrences > 1: - repetition_bonus = min( - 15, - (full_name_occurrences - 1) * 5, - ) - - name_score += repetition_bonus - - reasons.append( - f"matched name occurs " - f"{full_name_occurrences} times" - ) - - if all_persons_matched: - name_score += 20 - reasons.append("all persons matched") - - if len(match_name.normalized_persons) > 1: - name_score += 10 - reasons.append("multi-person name matched") - - specificity_bonus = min( - 10, - len(match_name.normalized_name) // 5, - ) - - name_score += specificity_bonus - - reasons.append( - f"specificity bonus {specificity_bonus}" - ) - - empty_duration_match = DurationMatch( - score=0, - classification="not_evaluated", - matched_duration=None, - difference_seconds=None, - source=None, - archived_at=None, - ) - - return MatchCandidate( - movie=movie, - matched_name=match_name, - name_score=name_score, - duration_score=0, - total_score=name_score, - name_reason=", ".join(reasons), - duration_reason="duration not evaluated", - full_name_occurrences=full_name_occurrences, - all_persons_matched=all_persons_matched, - duration_match=empty_duration_match, - ) - - -def best_name_candidate_for_movie( - normalized_filename: str, - movie: Movie, -) -> MatchCandidate | None: - candidates = [] - - for match_name in movie.match_names: - candidate = score_match_name( - normalized_filename=normalized_filename, - movie=movie, - match_name=match_name, - ) - - if candidate is not None: - candidates.append(candidate) - - if not candidates: - return None - - return max( - candidates, - key=lambda candidate: ( - candidate.name_score, - candidate.matched_name.source == "current", - len(candidate.matched_name.normalized_persons), - len(candidate.matched_name.normalized_name), - ), - ) - - -def describe_duration_match( - duration_match: DurationMatch, -) -> str: - classification = duration_match.classification - - if classification == "file_duration_unknown": - return "file duration unavailable" - - if classification == "database_duration_unknown": - return "database duration unavailable" - - if duration_match.matched_duration is None: - return classification - - difference = duration_match.difference_seconds or 0 - - if classification == "current_version_match": - return ( - f"current duration match, difference {difference}s" - ) - - if classification == "historical_version_match": - archived = ( - f", archived {duration_match.archived_at}" - if duration_match.archived_at - else "" - ) - - return ( - f"historical duration match, " - f"difference {difference}s{archived}" - ) - - return ( - f"{classification}, difference {difference}s" - ) - - -def apply_duration_score( - candidate: MatchCandidate, - file_duration: int | None, -) -> MatchCandidate: - duration_match = match_duration( - file_duration=file_duration, - movie=candidate.movie, - ) - - return replace( - candidate, - duration_score=duration_match.score, - total_score=( - candidate.name_score - + duration_match.score - ), - duration_reason=describe_duration_match( - duration_match - ), - duration_match=duration_match, - ) - - -def find_candidates( - filename: str, - movies: list[Movie], - file_duration: int | None, -) -> list[MatchCandidate]: - filename_stem = Path(filename).stem - normalized_filename = normalize_text(filename_stem) - - candidates = [] - - for movie in movies: - candidate = best_name_candidate_for_movie( - normalized_filename=normalized_filename, - movie=movie, - ) - - if candidate is None: - continue - - candidates.append( - apply_duration_score( - candidate=candidate, - file_duration=file_duration, - ) - ) - - return sorted( - candidates, - key=lambda candidate: ( - candidate.total_score, - candidate.duration_score, - candidate.name_score, - len(candidate.matched_name.normalized_persons), - len(candidate.matched_name.normalized_name), - ), - reverse=True, - ) - - -def get_detected_movies( - candidates: list[MatchCandidate], -) -> dict[str, str]: - detected: dict[str, str] = {} - - for candidate in candidates: - if candidate.full_name_occurrences == 0: - continue - - detected[candidate.movie.movie_id] = ( - candidate.movie.name - ) - - return detected - - -def candidate_covers_detected_movies( - candidate: MatchCandidate, - candidates: list[MatchCandidate], - detected_movie_ids: set[str], -) -> bool: - candidate_persons = set( - candidate.matched_name.normalized_persons - ) - - if len(candidate_persons) <= 1: - return False - - detected_names: set[str] = set() - - for other in candidates: - if other.movie.movie_id not in detected_movie_ids: - continue - - if other.full_name_occurrences == 0: - continue - - if len(other.matched_name.normalized_persons) != 1: - continue - - detected_names.add( - other.matched_name.normalized_persons[0] - ) - - return ( - bool(detected_names) - and detected_names.issubset(candidate_persons) - ) - - -def calculate_total_margin( - candidates: list[MatchCandidate], -) -> int: - if not candidates: - return 0 - - if len(candidates) == 1: - return candidates[0].total_score - - return ( - candidates[0].total_score - - candidates[1].total_score - ) - - -def resolve_with_duration( - candidates: list[MatchCandidate], -) -> MatchCandidate | None: - """ - Resolve several exact name candidates using duration. - - Exactly one candidate must have credible duration support, or the best - duration-supported candidate must have a clear duration-score margin. - """ - credible = [ - candidate - for candidate in candidates - if candidate.duration_score - >= MIN_CREDIBLE_DURATION_SCORE - ] - - if not credible: - return None - - credible = sorted( - credible, - key=lambda candidate: ( - candidate.duration_score, - candidate.total_score, - candidate.name_score, - ), - reverse=True, - ) - - if len(credible) == 1: - return credible[0] - - duration_margin = ( - credible[0].duration_score - - credible[1].duration_score - ) - - if duration_margin >= MIN_DURATION_SCORE_MARGIN: - return credible[0] - - return None - - -def classify_candidates( - candidates: list[MatchCandidate], -) -> MatchResult: - if not candidates: - return MatchResult( - status="unmatched", - best=None, - margin=0, - reason="no exact database name or alias found", - detected_movies=(), - ) - - best = candidates[0] - margin = calculate_total_margin(candidates) - - detected = get_detected_movies(candidates) - detected_movie_ids = set(detected) - - if len(detected_movie_ids) > 1: - covering_candidates = [ - candidate - for candidate in candidates - if candidate_covers_detected_movies( - candidate=candidate, - candidates=candidates, - detected_movie_ids=detected_movie_ids, - ) - ] - - if covering_candidates: - covering_candidates.sort( - key=lambda candidate: ( - candidate.total_score, - candidate.duration_score, - candidate.name_score, - ), - reverse=True, - ) - - covering_best = covering_candidates[0] - - other_scores = [ - candidate.total_score - for candidate in candidates - if candidate.movie.movie_id - != covering_best.movie.movie_id - ] - - covering_margin = ( - covering_best.total_score - - max(other_scores, default=0) - ) - - if ( - covering_best.name_score >= MIN_NAME_SCORE - and covering_margin >= MIN_NAME_MARGIN - ): - return MatchResult( - status="matched", - best=covering_best, - margin=covering_margin, - reason=( - "multi-person database entry covers all " - "detected movie names" - ), - detected_movies=tuple( - sorted( - detected.values(), - key=str.casefold, - ) - ), - ) - - duration_winner = resolve_with_duration( - candidates - ) - - if duration_winner is not None: - other_scores = [ - candidate.total_score - for candidate in candidates - if candidate.movie.movie_id - != duration_winner.movie.movie_id - ] - - winner_margin = ( - duration_winner.total_score - - max(other_scores, default=0) - ) - - return MatchResult( - status="matched", - best=duration_winner, - margin=winner_margin, - reason=( - "multiple names matched, but duration " - "clearly supports one candidate" - ), - detected_movies=tuple( - sorted( - detected.values(), - key=str.casefold, - ) - ), - ) - - return MatchResult( - status="ambiguous", - best=best, - margin=margin, - reason=( - "multiple distinct database movies matched; " - "duration does not clearly resolve them" - ), - detected_movies=tuple( - sorted( - detected.values(), - key=str.casefold, - ) - ), - ) - - if ( - best.name_score >= MIN_NAME_SCORE - and ( - len(candidates) == 1 - or margin >= MIN_NAME_MARGIN - ) - ): - return MatchResult( - status="matched", - best=best, - margin=margin, - reason=( - "best candidate exceeds name and margin thresholds" - ), - detected_movies=tuple( - sorted( - detected.values(), - key=str.casefold, - ) - ), - ) - - return MatchResult( - status="ambiguous", - best=best, - margin=margin, - reason="score or margin is insufficient", - detected_movies=tuple( - sorted( - detected.values(), - key=str.casefold, - ) - ), - ) - - -def format_duration(seconds: int | None) -> str: - if seconds is None: - return "unknown" - - hours, remainder = divmod(seconds, 3600) - minutes, seconds = divmod(remainder, 60) - - if hours: - return f"{hours}:{minutes:02d}:{seconds:02d}" - - return f"{minutes}:{seconds:02d}" - - -def format_match_source( - candidate: MatchCandidate, -) -> str: - if candidate.matched_name.source == "current": - return "current name" - - return ( - f"{candidate.matched_name.source} alias " - f"{candidate.matched_name.display_name!r}" - ) - - -def print_result( - path: Path, - file_duration: int | None, - result: MatchResult, - debug: bool, - candidates: list[MatchCandidate], -) -> None: - if result.best is None: - name = "" - score = 0 - version = "-" - else: - name = result.best.movie.name - score = result.best.total_score - version = result.best.duration_match.classification - - print( - f"{path.name:<55} " - f"{name:<35} " - f"{score:>5} " - f"{result.status:<10} " - f"{version}" - ) - - if not debug: - return - - print( - f" file duration: " - f"{format_duration(file_duration)}" - ) - - if not candidates: - print(" no candidates") - print(f" reason: {result.reason}") - return - - if result.detected_movies: - print( - " detected movies: " - + ", ".join(result.detected_movies) - ) - - for position, candidate in enumerate( - candidates[:5], - start=1, - ): - matched_duration = ( - candidate.duration_match.matched_duration - ) - - print( - f" {position}. " - f"{candidate.movie.name} " - f"[name={candidate.name_score}, " - f"duration={candidate.duration_score}, " - f"total={candidate.total_score}]" - ) - - print( - f" via {format_match_source(candidate)}" - ) - - print( - f" name: {candidate.name_reason}" - ) - - print( - f" duration: " - f"{candidate.duration_reason}; " - f"reference=" - f"{format_duration(matched_duration)}" - ) - - print(f" margin: {result.margin}") - print(f" reason: {result.reason}") - - -def main() -> None: - args = parse_arguments() - - if args.limit is not None and args.limit <= 0: - raise ValueError( - "--limit must be greater than zero." - ) - - if shutil.which("ffprobe") is None: - raise FileNotFoundError( - "ffprobe was not found in PATH." - ) - - extensions = parse_video_extensions( - args.ending - ) - - movies = load_movies(args.database) - - video_files = find_video_files( - directory=args.directory, - recursive=args.recursive, - extensions=extensions, - ) - - if args.limit is not None: - video_files = video_files[:args.limit] - - print( - f"{'Filename':<55} " - f"{'Likely database name':<35} " - f"{'Score':>5} " - f"{'Status':<10} " - f"Version" - ) - - print( - f"{'-' * 55} " - f"{'-' * 35} " - f"{'-' * 5} " - f"{'-' * 10} " - f"{'-' * 26}" - ) - - matched = 0 - ambiguous = 0 - unmatched = 0 - ffprobe_failures = 0 - historical_matches = 0 - - for video_file in video_files: - file_duration = read_file_duration( - video_file - ) - - if file_duration is None: - ffprobe_failures += 1 - - candidates = find_candidates( - filename=video_file.name, - movies=movies, - file_duration=file_duration, - ) - - result = classify_candidates(candidates) - - if result.status == "matched": - matched += 1 - elif result.status == "ambiguous": - ambiguous += 1 - else: - unmatched += 1 - - if ( - result.best is not None - and result.best.duration_match.classification - == "historical_version_match" - ): - historical_matches += 1 - - print_result( - path=video_file, - file_duration=file_duration, - result=result, - debug=args.debug, - candidates=candidates, - ) - - print() - print("Summary") - print("-------") - print(f"Files analyzed: {len(video_files)}") - print(f"Matched: {matched}") - print(f"Ambiguous: {ambiguous}") - print(f"Unmatched: {unmatched}") - print(f"Historical versions: {historical_matches}") - print(f"ffprobe failures: {ffprobe_failures}") - - -if __name__ == "__main__": - try: - main() - except ( - FileNotFoundError, - NotADirectoryError, - ValueError, - sqlite3.Error, - ) as error: - print(f"Error: {error}", file=sys.stderr) - sys.exit(1)