Problem
Many AI problems cannot be solved well enough with a single data source. Information can sit in text, images, speech, screenshots, video, or in conflicts between what is said and what is shown. A text-only model can therefore miss important visual or multimodal evidence.
Solution
The project compared a text-based baseline with a multimodal LLM pipeline. The baseline used TF-IDF and logistic regression on text from descriptions, OCR and ASR. The advanced solution added video keyframes and used a multimodal LLM to analyze both textual and visual information.
The system required structured JSON output from the model, including label, confidence, rationale, claim summary, text evidence strength, visual evidence strength, modality conflict and final probability. This made the output more machine-readable, easier to evaluate and easier to use in a larger pipeline.
Result
The multimodal solution improved macro F1 from 0.679 to 0.700 and recall for the target class from 66.7% to 75.0%. The improvement was moderate but useful, because it showed that visual evidence added real signal beyond the text baseline.
The project demonstrates an important AI engineering principle: don’t blindly trust model output. The system used baseline measurement, prompt testing, structured outputs, ablation testing, error analysis and explicit uncertainty handling.