Thesis supervisor: Tamás Ruppert
Location of studies (in Hungarian): University of Pannonia Abbreviation of location of studies: PE
Description of the research topic:
In this study, we aim to use a combination of different methods, including Operator 4.0 and advanced language models, to track and categorize human intentions during human-machine collaboration, based on human current state, the environment and context of the work tasks. By integrating various types of data into a deep learning network, it will improve our ability to recognize and predict human intention from the action and the perceived changes vice versa, thus understand and analyze human behavior with associating factors, triggers, and clues. The utilized methods include:
1. The combination of brain signals, and gesture dynamics can capture individual body gestures and movements that may indicate the person's emotional state.
2. Data on tracking facial expressions and eye movements.
3. Physiological signals, such as galvanic skin response (GSR), skin conductance measurements or Heart Rate Variability (HRV).
4. Utilizing the action and interaction logs during collaboration works, which serve as a backbone to recognize the context change.
5. The record of ambient sensors will provide the environmental changes during work.
The system will combine all the different kinds of abovementioned inputs to accurately classify human intentions and actions. This process will follow the principles of Operator 4.0, including process analysis, data analysis, AR/visualization, monitoring health data, AI assistance, and virtual operator representations. The models for intention recognition will be trained and tested using data from participants in real or simulated production scenarios and possibly from publicly available datasets. The intention recognition models are going to be trained and validated using data from participants doing tasks in real or simulated production settings, as well as utilizing any relevant datasets that are publicly available. Operator 4.0 elements will be combined with a well-designed and detailed human-machine interactions to create a comprehensive solution for smooth collaboration between humans and machines by allowing machines to accurately understand human intentions.
When using data mining and system engineering tools, the applicant's responsibilities would include:
• Creating a structure to track human intention parameters.
• Collecting physiological and vital sign data from human operators.
• Building an AI system to analyze the collected data.
• Evaluating the system's effectiveness with each type of input (such as speech signals, facial expressions, GSR, etc.)
Login
