Knowledge-driven Encode, Retrieve, Paraphrase for MedicalImageReport.pdf下载

Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical report generation. Specifically, KERP decomposes medical reportgenerationintoexplicitmedicalabnormalitygraphlearning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medicalknowledge;thenaRetrievemodulethatretrievestext templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specificcases.ThecoreofKERPisaproposedgenericimplementation unit—Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images andsequences.Experimentsshowthattheproposedapproach generates structured and robust reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.