We aim to establish a systematic approach to narrative detection, providing a machine learning framework covering a multifaceted array of narrative elements, derived from interdisciplinary theory and empirical research. We believe, detecting and tracking adversarial narrative elements along a systematic array of narrative dimensions will achieve the most (i) versatile, (ii) robust, (iii) performant, (iv) and interpretable solution. This approach will be designed to allow continual human-in-the-loop involvement, for example through approaches such as active learning for labeling, prioritization, and classification improvement.
Specifically, we aim to establish and deliver:
- A systematic and structured overview of narrative definitions across disciplines such as misinformation detection, fake news identification, financial fraud and financial narrative detection, and more general (computational) linguistics. For each definition, we specify its key elements and how to measure and quantify them. This will allow us to translate each element into a classification, regression, or representation task, which can be approached with modern machine learning methods.
- A benchmark dataset and task suite designed to test and improve our machine learning models for adversarial narrative detection and tracking
- A machine learning model benchmark framework to detect and later track the systematically defined key elements in the working definitions for adversarial narrative.
- A user-friendly dashboard designed to:
- Visualize network structures of interest (e.g. author network, dissemination network)
- Provide insights and interpretability by visualizing classification boundaries and modeling uncertainties for each narrative element detection task. Where applicable, language model’s attention matrices can be leveraged to highlight which parts in a document are considered as particularly relevant for the machine learning decision mechanism.
Overall, we aim to systematically implement an array of machine learning methods per detection task, guided by our research on relevant narrative definitions and sub-elements. We then aim to use model ensembling approaches across the various machine learning and deep learning methods, which can often improve model performance and strengthen robustness. Furthermore, the confluence of language models and knowledge graphs is a promising research area, in particular for the extraction and verification of information. Finally, our modular setup will allow us to easily adjust and extend the established framework to different narrative definitions and detection tasks as the project develops.
This project is funded by Turing Innovations Limited
