Multimodal Content Analysis for Enhanced User Experience
About this article
Keywords:
Improving the Quality of the Platform; Interaction of Participants; Interactive Systems; Multimodal Review; Study of MaterialsAbstract
The aim of the study was to develop methods of multimodal content analysis to improve the user experience in interactive systems. The study reviewed existing approaches to multimodal content analysis, established criteria for developing new methods, and provided specific examples of practical application of the developed methods in various fields, demonstrating their effectiveness and potential in real-world conditions. The main results consisted of the development of methods, in particular, integration and synchronization of modalities, which can demonstrate high efficiency in medical diagnostics. In turn, the personalization of user experience in streaming services increases user satisfaction through relevant recommendations, while ensuring data privacy and security meet modern regulatory requirements in the healthcare sector. The paper also examines the integration of modalities, which includes convolutional neural networks for image and video processing, recurrent neural networks for text and audio processing, and attention mechanisms for highlighting important parts of the data. Multimodal analysis involves the processing and integration of data from different sources, which provides a complete picture to improve the analysis. The developed methods outperform most existing approaches, providing higher accuracy, speed, robustness to noise, and incomplete data.
References
Ahmad N, 2024. Revolutionizing ERP integration: AI-powered solutions for effortless usability and enhanced user experience. http://dx.doi.org/10.13140/RG.2.2.33419.30247
Alzubi TM, Alzubi JA, Singh A & Alzubi OA 2023. A multimodal human-computer interaction for smart learning system. International Journal of Human-Computer Interaction. https://doi.org/10.1080/10447318.2023.2206758
Androshchuk A 2023. Cyber protection in programming languages Java and C#. Technologies and Engineering, 24(3):9–14. https://doi.org/10.30857/2786-5371.2023.3.1
Aviv I, Gafni R, Sherman S, Aviv B, Sterkin A & Bega E 2023. Cloud infrastructure from python code–breaking the barriers of cloud deployment. In: European Conference on Software Architecture, ECSA (pp.1–8). Available at: https://www.researchgate.net/profile/Itzhak-Aviv/publication/373897534_Cloud_Infrastructure_from_Python_Code_-breaking_the_Barriers_of_Cloud_Deployment/links/6501edd2808f9268d573dea5/Cloud-Infrastructure-from-Python-Code-breaking-the-Barriers-of-Cloud-Deployment.pdf
Awada IA, Mocanu I & Florea AM 2018. Exploiting multimodal interfaces in eLearning systems. International Scientific Conference: eLearning and Software for Education Conference, 2:174–181. https://doi.org/10.12753/2066-026x-21-094
View more references (63)
Azieva G, Kerimkhulle S, Turusbekova U, Alimagambetova A & Niyazbekova S 2021. Analysis of access to the electricity transmission network using information technologies in some countries. E3S Web of Conferences, 258:11003. https://doi.org/10.1051/e3sconf/202125811003
Bezshyyko O, Dolinskii A, Bezshyyko K, Kadenko I, Yermolenko R & Ziemann V 2008. PETAG01: A program for the direct simulation of a pel-let target. Computer Physics Communications, 178(2):144–155. https://doi.org/10.1016/j.cpc.2007.07.013
Bisenovna KA, Ashatuly SA, Beibutovna LZ, Yesilbayuly KS, Zagievna AA, Galymbekovna MZ & Oralkhanuly OB 2024. Improving the effi-ciency of food supplies for a trading company based on an artificial neural network. International Journal of Electrical and Computer Engineering, 14(4):4407–4417. https://doi.org/10.11591/ijece.v14i4.pp4407-4417
Bouchey B, Castek J & Thygeson J 2021. Multimodal learning. In: Learning Environments in STEM Higher Education (pp.35–54). Cham: Springer. https://doi.org/10.1007/978-3-030-58948-6_3
Breitschaft SJ, Pastukhov A & Carbon C-C 2021. Where's my button? Evaluating the user experience of surface haptics in featureless automotive user interfaces. IEEE Transactions on Haptics, 15(2):292–303. https://doi.org/10.1109/TOH.2021.3131058
Chaiyaphum T 2022. Developing supply chain innovation model for operation and adaptation of logistics service providers in the disruptive era. Available at: http://www.dspace.spu.ac.th/handle/123456789/8697
Chiu WHK, Ko WSK, Cho WCS, Hui SYJ, Chan WCL & Kuo MD 2024. Evaluating the diagnostic performance of large language models on complex multimodal medical cases. Journal of Medical Internet Research, 26:e53724. https://doi.org/10.2196/53724
Choudhary M, Singh S & Rathore SS 2024. Beyond text: Multimodal credibility assessment approaches for online user-generated content. ACM Transactions on Intelligent Systems and Technology. https://doi.org/10.1145/3673236
Dewangan O & Mishra M 2023. TLMAEMS: Design of an efficient transfer learning model with auto encoders for multimodal sentiment analysis via deep sentiment networks. European Chemical Bulletin, 12(Special Issue 5):1453–1467. http://dx.doi.org/10.48047/ecb/2023.12.si5.1362023.30/05/2023
Fang H, Liang J & Sha L 2024. Enhanced multimodal recommendation systems through reviews integration. Research Square. https://doi.org/10.21203/rs.3.rs-4333408/v1
Henrique BM, Sobreiro VA & Kimura H 2023. Practical machine learning: Forecasting daily financial markets directions. Expert Systems with Ap-plications, 233:120840. https://doi.org/10.1016/j.eswa.2023.120840
Heraz A, Chen L & Bhyravabhottla KKA 2024. Enhancing user receptivity in e-learning environments through gesture-based emotion tracking as a pathway to mental well-being and health equity. JMIR Formative Research. https://doi.org/10.2196/preprints.57293
Iklassova K, Aitymova A, Kopnova O, Shaporeva A, Abildinova G, Nurbekova Z, Almagambetova L, Gorokhov A & Aitymov Z 2024. Ontology modeling for the automation of questionnaire data processing. Eastern-European Journal of Enterprise Technologies, 5(2-131):36–52. https://doi.org/10.15587/1729-4061.2024.314129
Ince EB, Cha K & Cho J 2024. Towards a conceptual model of users’ expectations of an autonomous in-vehicle multimodal experience. Human Behavior and Emerging Technologies, 2024(1):7418597. https://doi.org/10.1155/2024/7418597
Jagnade G, Sable S & Ikar M 2023. Advancing multimodal fusion in human-computer interaction: Integrating eye tracking, lips detection, speech recognition, and voice synthesis for intelligent cursor control and auditory feedback. International Conference on Computing Communication and Networking Technologies. https://doi.org/10.1109/ICCCNT56998.2023.10306457
Kamil MAF 2023. User Experience analysis of linkedin social media using usability metric for user experience (UMUX). Journal of Information Engineering and Educational Technology, 7(2):78–82. https://doi.org/10.26740/jieet.v7n2.p78-82
Kamtab P & Wiriyanon T 2022. Design a digital multimodal user interface for senior citizens. Journal of Education Studies, 50(1):6. https://doi.org/10.58837/chula.educu.50.1.5
Kerimkhulle S & Aitkozha Z 2017. A criterion for correct solvability of a first order difference equation. AIP Conference Proceedings, 1880:040016. https://doi.org/10.1063/1.5000632
Kerimkhulle S, Kerimkulov Z, Bakhtiyarov D, Turtayeva N & Kim J 2021. In-Field Crop-Weed Classification Using Remote Sensing and Neural Network. In: SIST 2021 - 2021 IEEE International Conference on Smart Information Systems and Technologies (9465970). https://doi.org/10.1109/SIST50301.2021.9465970.
Kerimkhulle S, Obrosova N, Shananin A & Tokhmetov A 2023. Young Duality for Variational Inequalities and Nonparametric Method of Demand Analysis in Input–Output Models with Inputs Substitution: Application for Kazakhstan Economy. Mathematics, 11(19):4216. https://doi.org/10.3390/math11194216
Khoda V, Leshchuk N, Topalov A, Robotko S, Klymenko O & Nekrasov S 2024. Computerized Lathe Control System based on Internet of Things Technology. In: Proceedings - International Conference on Advanced Computer Information Technologies, ACIT (pp.674–677). Ceske Budejovice: Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ACIT62333.2024.10712548
Khrulov M 2025. Analysis of the current state of computer systems in the field of virtual healthcare. Technologies and Engineering, 26(1): 55–66. https://doi.org/10.30857/2786-5371.2025.1.5
Kolodiziev O, Krupka M, Shulga N, Kulchytskyy M & Lozynska O 2021. The level of digital transformation affecting the competitiveness of banks. Banks and Bank Systems, 16(1):81–91. https://doi.org/10.21511/bbs.16(1).2021.08
Konurbayeva ZT, Madiyarova ES & Rakhimberdinova MU 2015. Algorithm for generating competitive potential of engineering within the regional economy. Actual Problems of Economics, 168(6):236–247.
Kovalchuk D 2025. Utilising large language models for automated real-time cyber threat analysis. Bulletin of Cherkasy State Technological Univer-sity, 30(1):48–58. https://doi.org/10.62660/bcstu/1.2025.48
Maksymov A & Tryus Yu 2025. Information technology for solving the multi-criteria decision-making problem using the modified Fuzzy TOPSIS method. Bulletin of Cherkasy State Technological University, 30(1):91–106. https://doi.org/10.62660/bcstu/1.2025.91
Manmothe SR & Jadhav JR 2024. Integrating multimodal data for enhanced analysis and understanding: Techniques for sentiment analysis and cross-modal retrieval. Journal Of Advanced Zoology, 45(S4):22–28. https://doi.org/10.53555/jaz.v45iS4.4144
Mao X & Zhang J 2021. On the power of SVD in the stochastic block model. https://doi.org/10.48550/arXiv.2309.15322
Marques J & Marques RP 2023. Digital transformation of the hotel industry. Cham: Springer.
Meier S 2022. Digital storytelling: A didactic approach to multimodal coherence. Frontiers in Communication, 7:906268. https://doi.org/10.3389/fcomm.2022.906268
Moradi P 2022. Ageing with technology – An extended multimodal design study of active ageing users’ emotional experience with social robots. Available at: https://hdl.handle.net/10292/15512
Mullick T, Shaaban S, Radovic A & Doryab A 2024. Framework for ranking machine learning predictions of limited, multimodal, and longitudinal behavioral passive sensing data: Combining user-agnostic and personalized modeling. JMIR AI, 3:e47805. https://doi.org/10.2196/47805
Naegelin M, Weibel RP, Kerr JI, Schinazi VR, La Marca R, Von Wangenheim F, Hoelscher C & Ferrario A 2023. An interpretable machine learn-ing approach to multimodal stress detection in a simulated office environment. Journal of Biomedical Informatics, 139:104299. https://doi.org/10.1016/j.jbi.2023.104299
Orazbayev B, Zhumadillayeva A, Kabibullin M, Crabbe MJC, Orazbayeva K & Yue X 2023. A Systematic Approach to the Model Development of Reactors and Reforming Furnaces With Fuzziness and Optimization of Operating Modes. IEEE Access, 11:74980–74996. https://doi.org/10.1109/ACCESS.2023.3294701
Panwar V 2024. Leveraging progressive web apps (PWAs) for enhanced user experience and performance: A comprehensive analysis. International Journal of Management IT and Engineering, 14(4):31–43.
Pavlova D, Dovramadjiev T, Daskalov D, Mirchev N, Peev I, Radeva J, Dimova R, Kavaldzhieva K, Mrugalska B, Szabo G, Kandioglou A 2024. 3D Design of a Dental Crown with Artificial Intelligence Based in Cloud Space. Lecture Notes in Networks and Systems, 817:437–445. https://doi.org/10.1007/978-981-99-7886-1_37
Porkodi S & Raman AM 2025. Success of cloud computing adoption over an era in human resource management systems: a comprehensive meta-analytic literature review. Management Review Quarterly, 75(2):1041–1075. https://doi.org/10.1007/s11301-023-00401-0
Pradipta RAH, Widiani HN, Tama DA, Ramdani AL, Komariansyah KK, Nurfajri MI, Barus I & Fami A 2024. Analyzing user experience in the M-TIX application using the user experience questionnaire method. International Journal of Multilingual Education and Applied Linguistics, 1(3):61–71. https://doi.org/10.61132/ijmeal.v1i3.62
Ragan LC & Villarin LJR 2021. Emergent guiding principles for STEM education. In: Innovative Learning Environments in STEM Higher Educa-tion (pp.107–119). Cham: Springer. https://doi.org/10.1007/978-3-030-58948-6_6
Ramtohul A & Khedo KK 2023. An engaging user experience framework for mobile augmented reality. Research Square. https://doi.org/10.21203/rs.3.rs-3080680/v1
Rexhepi BR, Kumar A, Gowtham MS, Rajalakshmi R, Paikaray MD & Adhikari PK 2023. An Secured Intrusion Detection System Integrated with the Conditional Random Field For the Manet Network. International Journal of Intelligent Systems and Applications in Engineering, 11(3s):14–21.
Richas F & Kamal I 2024. Comparison analysis of user experience on m-BCA and BRImo mobile banking applications using the user experience questionnaire (UEQ) method. International Journal of Management and Business Economics, 2(3):47–51. https://doi.org/10.58540/ijmebe.v2i3.551
Rust RT 2020. The future of marketing. International Journal of Research in Marketing, 37(1):15–26. https://doi.org/10.1016/j.ijresmar.2019.08.002
Salim F 2023. Identifying user experience (UX) elements and emotional experiences on smartwatch devices. http://dx.doi.org/10.22677/THESIS.200000726145
Santhosh P, Ashweej A, Agarwal G & Raj GD 2024. Personalized product ranking system for enhanced user experience. MATEC Web of Confer-ences, 392:01077. https://doi.org/10.1051/matecconf/202439201077
Saputra FA, Yeza MP, Fahrezy MF, Fami A & Barus IRG 2024. Article comparing user experience maxim and in drive applications using the user experience questionnaire. ILKOMNIKA Journal of Computer Science and Applied Informatics, 6(1):79–90. https://doi.org/10.28926/ilkomnika.v6i1.614
Smith-Harvey J & Aguayo C 2024. Modes of meaning: Multimodal media & 4E+ cognition in tech-enhanced learning. Pacific Journal of Technolo-gy Enhanced Learning, 6(1):10–11. https://doi.org/10.24135/pjtel.v6i1.181
Taylor DL, Yeung M & Bashet AZ 2021. Personalized and adaptive learning. In: Innovative Learning Environments in STEM Higher Education (pp.17–34). Cham: Springer. https://doi.org/10.1007/978-3-030-58948-6_2
Tkachenko O, Chechet A, Chernykh M, Bunas S & Jatkiewicz P 2025. Scalable Front-End Architecture: Building for Growth and Sustainability. Informatica (Slovenia), 49(1):137–150. https://doi.org/10.31449/inf.v49i1.6304
Tokarieva KS, Kovalchuk OY, Kolesnikov AP, Dzyurbel AD, Bodnar-Petrovska OB & Predmestnikov OG 2024. The use of ai-language models in judicial proceedings: information and legal aspects. Revista Juridica, 2(78):520–538. https://doi.org/10.26668/revistajur.2316-753X.v2i78.6928
UI/UX staffing 2019. Available at: https://evergreens.com.ua/ua/articles/ui-and-ux-personalization.html
Varanitskyi D, Rozkolodko O, Liuta M, Zakharova M & Hotunov V 2024. Analysis of data protection mechanisms in cloud environments. Tech-nologies and Engineering, 25(1):9–16. https://doi.org/10.30857/2786-5371.2024.1.1
Vijayakumar T 2024. Enhancing user experience through emotion-aware interfaces: A multimodal approach. Journal of Innovative Image Pro-cessing, 6(1):27–39. https://doi.org/10.36548/jiip.2024.1.003
Wang C, Yu X, Xu L, Wang Z & Wang W 2022. Multimodal semantic communication accelerated bidirectional caching for 6G MEC. Future Gen-eration Computer Systems, 140:225–237. https://doi.org/10.1016/j.future.2022.10.036
What is Data Protection and Privacy? 2024. Available at: https://cloudian.com/guides/data-protection/data-protection-and-privacy-7-ways-to-protect-user-data/#:~:text=By%20protecting%20data%2C%20companies%20can,%2C%20encryption%2C%20and%20endpoint%20protection
Wildfeuer J & Lehmann C 2024. Drawing multimodality's bigger picture: Metalanguages and corpora for multimodal analyses. Lausanne: Frontiers Media SA. https://doi.org/10.3389/978-2-8325-5196-7
Xhafka E, Sinoimeri D & Teta J 2024. Evaluating the Impact of E-Governance on Public Service Improvement in Albania: A Quantitative Analysis. Sustainability (Switzerland), 16(24):10896. https://doi.org/10.3390/su162410896
Xu Y, Lin Y-S, Zhou X & Shan X 2023. Utilizing emotion recognition technology to enhance user experience in real-time. Computing and Artifi-cial Intelligence, 2(1):1388. https://doi.org/10.59400/cai.v2i1.1388
Yasinthorn Y & Celadon S 2023. Security enhancement and performance optimization for WordPress-based Websites, a case study of WorayuthIT website. Available at: http://digital.csmsu.net:8080/library/handle/123456789/218
Yasmin S & Jamal CQ 2023. Multisensory modeling of tabular data for enhanced perception and immersive experience. In: Advances in Visual Computing (pp.452–465). Cham: Springer. https://doi.org/10.1007/978-3-031-47966-3_36
Zhang Y & Tan H 2021. Effects of multimodal warning types on driver’s task performance, physiological data and user experience. In: Cross-Cultural Design (pp.304–315). Cham: Springer. https://doi.org/10.1007/978-3-030-77080-8_25
Zhetenbayev N, Zhauyt A, Balbayev G & Shingissov B 2022. Robot device for ankle joint rehabilitation: A review. Vibroengineering Procedia, 41:96–102. https://doi.org/10.21595/vp.2022.22507
Ziker C, Truman B & Dodds H 2021. Cross Reality (XR): Challenges and opportunities across the spectrum. In: Innovative Learning Environments in STEM Higher Education (pp.55–77). Cham: Springer. https://doi.org/10.1007/978-3-030-58948-6_4