06.1
Modalities as Attributes Nodes or Observational Evidence
A multimodal knowledge graph organizes information in different forms, but inclusion can happen at several depths. The simplest approach attaches an image to an entity. A richer approach makes the image an independent node with capture time, provenance, and licensing information. A finer representation makes image regions, text boxes, audio intervals, or video segments into locatable evidence objects. Each has a role: a display catalogue may need only attributes, identification across documents requires independent resources, and fine grained questions usually require local evidence. Surveys cover different construction and learning tasks; attaching pictures to a database does not by itself establish semantic alignment.[R15][R16]
Design should work backward from questions. Asking where a dish ingredient list appears requires a resource and document location. Asking what the green leaves in the upper left of an image are requires region coordinates and identification evidence. Asking when seasoning enters a cooking video requires a time interval. Finer granularity raises annotation, indexing, and maintenance costs. A progressive strategy therefore preserves complete original resources first and adds local structure when queries justify it, instead of densely annotating every image without a defined purpose. Local structures must also preserve transformations back to the original resource. Otherwise, a correct bounding box can point to the wrong location after resizing or cropping.
Original design
06.2
Shared Embedding Spaces Supply Similarity Clues
CLIP demonstrates transferable visual representation learning from paired images and text and provides an important foundation for cross modal retrieval. A shared space makes similarity between a text query and image candidates computable, but it does not automatically give that similarity factual meaning. A picture close to the phrase kung pao chicken may match through colour, presentation, or training cooccurrence. It does not independently establish that the image shows a particular restaurant product, and it cannot establish the absence of invisible ingredients. Embedding scores should first be treated as candidate ranking signals and combined with product identifiers, context, and provenance.[R17]
Applications should distinguish similarity, matching, and confirmation. Similarity means a model recommends inspection. Matching means a candidate passes specified business conditions. Confirmation means it satisfies an explicit review rule. Recording evidence for each transition prevents visual outputs from silently becoming knowledge facts. Ambiguous details should permit multiple candidates or an unknown answer, accompanied by an explanation of the missing evidence. Evaluation should include more than overall retrieval success. Hard negatives include visually similar but distinct items, different restaurant versions of a dish with the same name, different products in similar packaging, and photographs with matching backgrounds but different subjects. Such cases expose the shortcuts on which a model relies.
Original design
06.3
Entity Linking Is a Cross Modal Identity Decision
Entity linking asks which existing entity a mention, image region, or product record refers to. This differs from recognizing a category. Identifying a chicken dish is classification; linking it to a restaurant menu item on a particular date requires identity conditions. Multimodal entity linking research combines visual, textual, and graph clues, but sparse evidence, ambiguous names, and domain changes require testing on the actual data. This report uses KGMEL as an entry point into that direction without inferring complete implementation details or deployment performance from material beyond the inspected abstract.[R22]
A practical linking process needs candidate generation, condition checking, and rejection. Aliases, identifiers, and vectors first retrieve candidates. Restaurant, time, packaging text, and neighboring relations then help distinguish them. If two candidates remain plausible, preserve an unresolved observation instead of forcing a merge. Separate relations for same entity and same category allow the system to say that an image appears to show a category of dish while the specific product remains unidentified. Incorrect merges are often harder to repair than missed links because they mix evidence from different objects and contaminate downstream summaries. High impact entities therefore justify prioritizing a low false merge rate and providing efficient human inspection. Correct rejection is a valid linking outcome.
06.4
Preserve Local Evidence and Document Layout
Figures and document layout carry relations. Which column header governs a number, which object a caption describes, and which steps an arrow connects may all disappear during plain text conversion. ColPali studies document retrieval using vision language representations, offering a route that uses visual document information directly. This approach can complement an explicit knowledge graph but is not equivalent to one. Preserving page images may recover information lost by text parsing; explaining an answer still requires identifying the relevant page, region, and text.[R23]
Research reports and equipment manuals should preserve both logical structure and visual coordinates. Table cells should link back to row and column headers, figure labels to legends, and continued tables to a stable table identity. Answers can state a conclusion in text and attach a local image for verification. A new failure needs attention: cropping away units, axes, or a qualifying note may make a local image look clearer while changing its meaning. Local evidence should therefore include necessary context and provide access to the full page. Image contribution can be tested by comparing conditions with and without images. A more attractive answer page alone does not demonstrate a substantive benefit from multimodal retrieval.
06.5
Missing Conflicting and Unreliable Modalities
Multimodal fusion should not assume that every modality is complete and reliable. A product may have text but no image, a photograph may be outdated, audio may be misaligned with video, and a review may describe an earlier version. Treating missing modalities as ordinary zero vectors can encourage patterns unrelated to content. Forcing agreement across modalities can conceal real conflicts. Each observation should therefore retain availability, quality, time, and scope, and retrieval should use these explicitly. Missing information should change what the system can answer instead of inviting the model to silently fill factual gaps.
Conflict diagnosis must identify the relevant level. Text stating that peanuts are present and an image detector finding none are not necessarily contradictory because peanuts may be invisible. An explicit claim of no peanuts and an ingredient record asserting peanuts for the same version create a statement conflict. Systems can distinguish contradictory content, uncertain identity, temporal mismatch, and limited observability, then request appropriate additional evidence. Training and testing should deliberately mask modalities, substitute outdated images, and introduce incorrect pairings to check whether confidence decreases appropriately. Mature fusion is not the ability to turn every input into one fluent answer. It is the ability to recognize which inputs cannot jointly support that answer.
06.6
From Multimodal Graphs to Multimodal GraphRAG
A multimodal graph addresses representation, association, and updating. Multimodal GraphRAG must additionally select evidence for a question and generate a verifiable answer. MuRAG offers an early example of external memory containing images and text. MMGraphRAG, mKG-RAG, and RAG-Anything illustrate later attempts to combine multimodal information with structured organization. These names refer to different research systems, not interchangeable components of a standardized architecture. They are treated here as evidence of a design space rather than entries in one performance ranking without matched reproduction.[R18][R19][R20][R24]
A transferable principle is to share evidence identifiers across the graph, retrieval, and generation. An image region should connect to the observation it supports, the entity linked by that observation, the reason it was retrieved, and the answer sentence relying on it. If the graph stores only concepts, retrieval returns only image filenames, and generation produces only a summary, all three components may run successfully without jointly supporting traceability. A minimum output can instead contain an answer claim, evidence location, relation path, and uncertainty statement. Questions that do not need visual information should not incur visual processing automatically. Questions that require images should not depend entirely on automatically generated captions. Evidence needs should determine modality selection.