A Survey of Human-in-the-loop for Machine Learning机器学习中“人在回路”技术综述
Abstract摘要
Machine learning has become the state-of-the-art technique for many tasks including computer vision, natural language processing, speech processing tasks, etc. However, the unique challenges posed by machine learning suggest that incorporating user knowledge into the system can be beneficial. The purpose of integrating human domain knowledge is also to promote the automation of machine learning. Human-in-the-loop is an area that we see as increasingly important in future research due to the knowledge learned by machine learning cannot win human domain knowledge. Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize the major approaches in the field; along with their technical strengths/ weaknesses, we have a simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and to motivate interested readers to consider approaches for designing effective human-in-the-loop solutions.机器学习已成为计算机视觉、自然语言处理、语音处理等众多任务中最先进的技术。然而，机器学习所面临的独特挑战表明，将用户知识融入系统可能会带来益处。整合人类领域知识的目的也是为了推动机器学习的自动化。由于机器学习所习得的知识无法胜过人类领域知识，我们认为“人在回路”（Human-in-the-loop）是未来研究中日益重要的一个领域。“人在回路”旨在通过整合人类的知识和经验，以最低的成本训练出准确的预测模型。人类可以为机器学习应用提供训练数据，并在基于机器的方法辅助下，直接完成流水线中计算机难以处理的任务。在本文中，我们从数据视角对现有的“人在回路”相关工作进行了综述，并按照递进关系将其分为三类：（1）从数据处理角度提升模型性能的工作；（2）通过干预性模型训练提升模型性能的工作；（3）独立于系统的“人在回路”设计。利用上述分类，我们总结了该领域的主要方法，并结合其技术优缺点，对自然语言处理、计算机视觉及其他领域的相关工作进行了简单的分类与探讨。此外，我们还提出了一些尚待解决的挑战和机遇。本综述旨在为“人在回路”提供一个高层面的总结，并激励感兴趣的读者思考设计高效“人在回路”解决方案的方法。
| Xingjiao Wu1,2, Luwei Xiao2, Yixuan Sun3, Junhang Zhang2, Tianlong Ma1,2, Liang He1,2,∗ |
| 1 Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China |
| 2 School of Computer Science and Technology, East China Normal University, Shanghai, China |
| 3 Fudan University, Shanghai, China |
| {wuxingjiao2885, louisshaw008, madmaxkgb, junhangzhang68}@gmail.com, {tlma, lhe}@cs.ecnu.edu.cn |
Index Terms— Human-in-the-loop, machine learning, deep learning.索引词——人在回路，机器学习，深度学习。
1 Introduction1 引言
Deep learning is a frontier for artificial intelligence, aiming to be closer to its primary goal��artificial intelligence. Deep learning has seen great success in a wide variety of applications, such as natural language processing, speech recognition, medical applications, computer vision, and intelligent transportation system [1]. The great success of deep learning is due to the larger models [2]. The scale of these models has included hundreds of millions of parameters. These hundreds of millions of parameters allow the model to have more degrees of freedom enough to awe-inspiring description capability.深度学习是人工智能的前沿领域，旨在更接近其主要目标——人工智能。深度学习在自然语言处理、语音识别、医疗应用、计算机视觉和智能交通系统等多种应用中取得了巨大成功[1]。深度学习的巨大成功归功于更大规模的模型[2]。这些模型的规模已达到数亿个参数。这数亿个参数使模型拥有了足够的自由度，从而具备了令人惊叹的描述能力。
However, the large number of parameters requires a massive amount of training data with labels [3]. Improving model performance by data annotation has two crucial challenges. On the one hand, the data growth rate is far behind the growth rate of model parameters, which has primarily hindered the further development of the model. On the other hand, the emergence of new tasks has far exceeded the speed of data updates, and annotating all samples is labor-intensive and time-consuming. To tackle this challenge, many researchers build new datasets by generating samples, thereby speeding up model iteration and reducing the cost of data annotation [4, 5, 6]. Besides, a group of researchers devise pre-trained models and transfer learning to solve this challenge [7, 8, 9], such as Transformers [10], BERT [11] and GPT [12]. These works have achieved incredible results.然而，大量的参数需要海量的带标签训练数据[3]。通过数据标注来提升模型性能面临两个关键挑战。一方面，数据的增长速度远落后于模型参数的增长速度，这在很大程度上阻碍了模型的进一步发展。另一方面，新任务的出现速度远超数据更新的速度，而对所有样本进行标注既费时又费力。为了应对这一挑战，许多研究人员通过生成样本来构建新数据集，从而加快模型迭代并降低数据标注成本[4, 5, 6]。此外，一些研究人员设计了预训练模型和迁移学习方法来解决这一挑战[7, 8, 9]，例如 Transformers[10]、BERT[11] 和 GPT[12]。这些工作已经取得了令人难以置信的成果。
Unfortunately, the generated data is only used as base data to initialize the model. To obtain a high-precision usable model, labeling and updating specific data is often necessary. So various work based on weak supervision has been proposed [13, 14]. A great many researchers have proposed using few-shot to push the model to learn from fewer samples [15].遗憾的是，生成的数据仅用作初始化模型的基础数据。为了获得高精度、可用的模型，通常需要对特定数据进行标注和更新。因此，各种基于弱监督的工作被提出[13, 14]。许多研究人员提出使用少样本学习（few-shot）来推动模型从更少的样本中进行学习[15]。
1.1 Significance of Human-in-the-loop1.1 “人在回路”的意义
Integrated a priori knowledge in the learning framework is an effective means to deal with sparse data, as the learner does not need to induce the knowledge from the data [16]. More recently, an ever-growing number of researchers have made efforts to incorporate pre-training knowledge into their learning framework [17, 18, 19]. As special agents, humans have rich prior knowledge. If the developer can encourage the machine to engage with learning human wisdom and knowledge, it would help deal with sparse data, especially in medical fields like clinical diagnosis and lack of training data [20, 21, 22, 23]. Furthermore, recent advances in cognitive science and human-machine interaction have suggested that human-related elements, such as emotional state and practical capability, impact human teaching performance and machine learning results on different tasks.在学习框架中整合先验知识是处理稀疏数据的有效手段，因为学习者无需从数据中归纳出这些知识[16]。最近，越来越多的研究人员致力于将预训练知识纳入其学习框架[17, 18, 19]。作为特殊的智能体，人类拥有丰富的先验知识。如果开发者能够鼓励机器参与学习人类的智慧和知识，将有助于处理稀疏数据，特别是在临床诊断等缺乏训练数据的医疗领域[20, 21, 22, 23]。此外，认知科学和人机交互领域的最新进展表明，情绪状态和实践能力等与人相关的要素会影响人类的教学表现以及机器在不同任务上的学习结果。
A multitude of researchers have proposed utilizing a conception called “human-in-the-loop” to tackle the above challenges, mainly addressing these issues by incorporating human knowledge into the modeling process [24]. Human-in-the-loop conception is an extensive area of research that covers the intersection of computer science, cognitive science, and psychology.众多研究人员提出利用“人在回路”这一概念来应对上述挑战，主要通过将人类知识融入建模过程来解决这些问题[24]。“人在回路”概念是一个广泛的研究领域，涵盖了计算机科学、认知科学和心理学的交叉地带。
As illustrated in Fig. 2, human-in-the-loop (namely “human-in-the-loop” and “machine learning”) is an active research topic in machine learning, and there has been a rich publication in the past ten years.如图 2 所示，“人在回路”（即“人在回路”与“机器学习”）是机器学习中一个活跃的研究课题，过去十年中相关出版物非常丰富。
As shown in Fig. 1, a conventional machine-learning algorithm typically consists of three parts [25]. The first is data preprocessing, the second is data modeling, and the last is the developer modifying the existing process to improve performance. Generally speaking, the performance and results of machine learning models are unpredictable, which leads to a large degree of uncertainty in which part of the machine-human interaction is capable of bringing the best learning effect. Different researchers focus on manual intervention in distinguishable parts. In this paper, we investigate existing studies on human-in-the-loop technology via various implementations of human-in-the-loop from different practical perspectives(e.g. Data Processing, Model Training and Inference, and System construction and Application). It is crucial to explore how interaction type interplays with other components of a human-in-the-loop pipeline to affect the intelligent systems’ learning outcomes. In addition, more research focuses on the design of independent systems to help complete the improvement of the model. In this paper, we first discuss the work of improving model performance from data processing. Next, we discuss the work of improving model performance through interventional model training. Finally, we discuss the configuration of the system independent “human-in-the-loop”.如图 1 所示，传统的机器学习算法通常由三个部分组成[25]。第一部分是数据预处理，第二部分是数据建模，最后一部分是开发者修改现有流程以提升性能。总的来说，机器学习模型的性能和结果是不可预测的，这导致在机器与人交互的哪一部分能够带来最佳学习效果方面存在很大的不确定性。不同的研究人员关注不同部分的人工干预。在本文中，我们通过从不同实践视角（如数据处理、模型训练与推理、系统构建与应用）对“人在回路”技术的各种实现方式进行考察，来研究现有的“人在回路”相关研究。探索交互类型如何与“人在回路”流水线的其他组件相互作用，从而影响智能系统的学习结果，这一点至关重要。此外，更多的研究集中在设计独立的系统来帮助完成模型的改进。在本文中，我们首先讨论从数据处理角度提升模型性能的工作。接下来，我们讨论通过干预性模型训练提升模型性能的工作。最后，我们讨论独立于系统的“人在回路”的配置。
We explore the following questions around human-in-the-loop for machine learning:我们围绕机器学习中的“人在回路”探讨了以下问题：
- •
What is the human-in-the-loop for machine learning challenges, and what are the possible solutions to advance this research?机器学习中“人在回路”面临的挑战是什么，有哪些可能的解决方案来推动这项研究？
- •
From a data perspective, what is the research status of human-in-the-loop for machine learning, research challenges, and what are the future directions?从数据视角来看，机器学习中“人在回路”的研究现状、研究挑战以及未来的方向是什么？
- •
From the perspective of model training, what is the research status of human-in-the-loop for machine learning, research challenges, and what are the future directions?从模型训练视角来看，机器学习中“人在回路”的研究现状、研究挑战以及未来的方向是什么？
- •
From an application perspective, what is the research status of human-in-the-loop for machine learning, research challenges, and what are the future directions?从应用视角来看，机器学习中“人在回路”的研究现状、研究挑战以及未来的方向是什么？
1.2 Importance of this survey1.2 本综述的重要性
This paper is a systematic summary and analysis of the research area of human-in-the-loop with a focus on the following essential aspects:本文是对“人在回路”研究领域进行的系统性总结与分析，重点关注以下关键方面：
- •
We had a comprehensive summary of the work of human-in-the-loop so far, and divided these papers into CV, NLP and other application areas. We connected these papers in series according to data preprocessing, data annotation, and model training and inference from the perspective of data flow; and finally, we focused on the application based on human-in-the-loop;我们对迄今为止“人在回路”的相关工作进行了全面总结，并将这些论文划分为计算机视觉（CV）、自然语言处理（NLP）和其他应用领域。我们从数据流的视角，按照数据预处理、数据标注以及模型训练与推理将这些论文串联起来；最后，我们聚焦于基于“人在回路”的应用；
- •
We have classified and compared various methods of human-in-the-loop. Through classification and comparison, we have summarized the challenges currently encountered by human-in-the-loop and put forward some discussions on solving these challenges;我们对各种“人在回路”的方法进行了分类和比较。通过分类和比较，我们总结了“人在回路”目前遇到的挑战，并就解决这些挑战提出了一些探讨；
- •
We also conducted qualitative evaluations and comparisons between different methods to evaluate them consistently. This would assist readers in deciding which method is appropriate for the problem at hand;我们还对不同方法进行了定性评估和比较，以进行一致性评价。这将有助于读者判断哪种方法适合当前所面临的问题；
- •
We have also identified a series of important milestones achieved by methods in this area;我们还确定了该领域方法所取得的一系列重要里程碑；
- •
In addition to the methodological summary and analysis of human-in-the-loop’s work, we also discussed the system construction and application of human-in-the-loop. We analyze the human-in-the-loop on system construction divided by system components and applications following the baseline of engineering requirements.除了对“人在回路”相关工作进行方法论层面的总结与分析外，我们还讨论了“人在回路”的系统构建与应用。我们按照工程需求的基准，根据系统组件和应用对“人在回路”的系统构建进行了分析。
Before our work, there were also quiet a few excellent works reviewing human-in-the-loop [26, 27, 28, 29, 30, 31], but we are the first review of human-in-the-loop for machine learning. Much of the previous work is from the perspective of hardware and robotics, while we are talking about machine learning. In our research process, we also follow on previous experiences and ideas. Certainly, a single article may not cover all methods in this growing field. Nevertheless, we have sought to make this survey as comprehensive as possible. To help with this goal, we researched and analyzed a good many related references and documents. In addition, we also tried a variety of classification methods to divide the article structure, and finally, we chose to connect the whole work from the perspective of data.在我们之前，也有不少优秀的综述文章回顾了“人在回路”[26, 27, 28, 29, 30, 31]，但我们是首篇针对机器学习中“人在回路”的综述。以往的大部分工作是从硬件和机器人学的视角出发，而我们探讨的是机器学习。在研究过程中，我们也遵循了前人的经验和思路。当然，单篇文章可能无法涵盖这一不断发展领域中的所有方法。尽管如此，我们仍力求使本综述尽可能全面。为了实现这一目标，我们研究并分析了大量相关的参考文献和文档。此外，我们还尝试了多种分类方法来划分文章结构，最终我们选择从数据视角将整个工作连接起来。
1.3 Organization of contents1.3 内容组织
Since there has been no similar review before, we explore all papers (since 1990) containing the keywords “human-in-the-loop” and “machine learning” by Google Scholar. Firstly, we screen out relevant articles by reading the titles and abstracts. Secondly, we briefly classify the screened articles. Moreover, we read the screen articles in detail and constantly revised the classification. Finally, we summarize and form the final content according to the classification.由于此前没有类似的综述，我们通过 Google Scholar 检索了所有包含关键词“human-in-the-loop”和“machine learning”的论文（自 1990 年以来）。首先，我们通过阅读标题和摘要筛选出相关文章。其次，我们对筛选出的文章进行简要分类。此外，我们详细阅读了筛选出的文章，并不断修订分类。最后，我们根据分类总结并形成了最终内容。
In Section 2, we investigate the data processing method based on human-in-the-loop and then discuss data preprocessing, data annotation, and iterative labeling. In Section 3, we summarize and analyze research on model training and reasoning based on human-in-the-loop, and discuss human-in-the-loop from natural language processing and computer visual perspectives, respectively. In Section 4, we review the human-in-the-loop on system construction divided by system components and applications and discuss human-in-the-loop from the software and hardware integrated perspective, respectively. In Section 5, we propose a list of challenges based on the results of the survey. Finally, we conclude our work in Section 6.在第 2 节中，我们调查了基于“人在回路”的数据处理方法，并讨论了数据预处理、数据标注和迭代标注。在第 3 节中，我们总结并分析了基于“人在回路”的模型训练与推理研究，并分别从自然语言处理和计算机视觉视角探讨了“人在回路”。在第 4 节中，我们回顾了按系统组件和应用划分的“人在回路”系统构建，并分别从软硬件集成视角探讨了“人在回路”。在第 5 节中，我们根据综述结果提出了一系列挑战。最后，我们在第 6 节中总结了我们的工作。
2 Data Processing2 数据处理
At present, deep learning has played an irreplaceable role in many fields [1, 32]. The great success of deep learning is due to larger-scale models, which include hundreds of millions of parameters [2]. Such a large amount of parameters empower the model with enough degrees of freedom to obtain awe-inspiring description capability. A massive amount of training data with labels are required to deal with a considerable number of parameters [3]. However, making annotations requiring much labor is likely to lag behind the growth in model capacity, and available datasets are quickly becoming outdated in size and density [33]. So the methods of utilizing unlabeled data to improve the model capability have increasingly gained much attention [3, 34, 35, 36]. The most significant difficulty is based on the fact that unlabeled data usually include incorrect samples, such as disturbing images, defective statements, and violations of constraints. Suppose these inaccurate samples are exactly sampled as the key one, the errors brought about will be fatal [37]. To tackle this challenge, numerous researchers focus on exploring the way to generate a more rich sample space [4, 38, 5], trying to develop a universal model such as Transformers [39], BERT [11] or GPT [12] so that the model can learn features more effectively. Based on these successful methods, researchers then consider the further step: adopting little data to obtain more satisfactory results. So these models are employed for more tasks by fine-tuning and performing incredible results [40, 41, 42]. Although these methods still need to annotate a lot of data, which brings unnecessary trouble, we still noticed that interference in the model performance only some critical samples in the new datasets. Here goes to a critical issue that needs to be solved urgently, How do we find out the key samples, and can we annotate key samples more easily?目前，深度学习在许多领域发挥着不可替代的作用[1, 32]。深度学习的巨大成功归功于大规模模型，这些模型包含数亿个参数[2]。如此大量的参数使模型拥有足够的自由度，从而具备了令人惊叹的描述能力。处理如此多的参数需要海量的带标签训练数据[3]。然而，进行标注需要大量人力，且往往滞后于模型容量的增长，现有数据集在规模和密度上正迅速过时[33]。因此，利用未标注数据来提升模型能力的方法日益受到关注[3, 34, 35, 36]。最大的困难在于未标注数据通常包含错误样本，如干扰图像、有缺陷的表述以及违反约束的情况。假设这些不准确的样本恰好被采样为关键样本，所带来的错误将是致命的[37]。为了应对这一挑战，许多研究人员专注于探索生成更丰富样本空间的方法[4, 38, 5]，试图开发像 Transformers[39]、BERT[11] 或 GPT[12] 这样的通用模型，以便模型能更有效地学习特征。基于这些成功的方法，研究人员随后考虑了进一步的步骤：采用少量数据来获得更满意的结果。因此，这些模型通过微调被应用于更多任务，并取得了令人难以置信的成果[40, 41, 42]。尽管这些方法仍然需要标注大量数据，这带来了不必要的麻烦，但我们仍然注意到，对模型性能产生干扰的只是新数据集中的一些关键样本。这里出现了一个亟待解决的关键问题：我们如何找出关键样本，以及能否更轻松地标注关键样本？
| Work | Data Processing | Year | Area | Task | Quantitative results | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| DP | DA | IL | CV | NLP | SP | Other | ||||
| Yu et al. [33] | ✓ | 2015 | ✓ | Scene and Object Categories | ||||||
| He et al. [43] | ✓ | 2016 | ✓ | CCG Parser | ||||||
| Self et al. [44] | ✓ | 2016 | ✓ | Data Analysis | ||||||
| Zhuang et al. [45] | ✓ | 2017 | ✓ | Knowledge Bases Integration | ||||||
| Li et al. [46] | ✓ | 2017 | ✓ | Data Integration | ||||||
| KIM et al. [47] | ✓ | 2018 | ✓ | Finding Sound Events | ||||||
| Doan et al. [48] | ✓ | 2018 | ✓ | Data Analysis | ||||||
| Dong et al. [49] | ✓ | 2018 | ✓ | ✓ | Data Fusion;DOM extraction | |||||
| Gentile et al. [50] | ✓ | ✓ | 2019 | ✓ | Dictionary Expansion | |||||
| Zhang et al. [51] | ✓ | ✓ | 2019 | ✓ | Entity Extraction | |||||
| Laure et al. [52] | ✓ | 2019 | ✓ | ✓ | ✓ | ✓ | Data Preparation | |||
| Gurajada et al. [53] | ✓ | 2019 | ✓ | Entity Resolution | ||||||
| Lou et al. [54] | ✓ | 2019 | ✓ | Knowledge Graph Programming | ||||||
| Liu et al. [55] | ✓ | ✓ | 2019 | ✓ | Person Re-Identification | |||||
| Wallace et al. [56] | ✓ | 2019 | ✓ | Question Answering | ||||||
| Fan et al. [57] | ✓ | 2019 | ✓ | ✓ | Network Anomaly Detection | |||||
| Krokos et al. [58] | ✓ | 2019 | ✓ | ✓ | Knowledge Discovery | |||||
| Klie et al. [59] | ✓ | 2020 | ✓ | Entity Linking | ||||||
| Chai et al. [30] | ✓ | 2020 | ✓ | Outlier Detection | ||||||
| Butler et al. [60] | ✓ | 2020 | ✓ | Facial Expressions | ||||||
| Ristoski et al. [61] | ✓ | 2020 | ✓ | Relation Extraction | ||||||
| Qian et al. [62] | ✓ | 2020 | ✓ | Entity Name Understanding | ||||||
| Le et al. [63] | ✓ | ✓ | 2020 | ✓ | Self-Annotation For Video Object | |||||
| Bartolo et al. [64] | ✓ | 2020 | ✓ | Reading Comprehension | ||||||
| Cutler et al. [65] | ✓ | 2021 | ✓ | Entity Recognition | ||||||
| Meng et al. [66] | ✓ | 2021 | ✓ | 3D Point Cloud Object Detection | ||||||
| Zhang et al. [67] | ✓ | 2021 | ✓ | Screentone and Manga Processing | ||||||
| Adhikari et al. [68] | ✓ | 2021 | ✓ | Object Detection | ||||||
The intuitive idea to solve this concern with a specific method is in three steps: Select a few samples which models can not recognize. (1) Use the particular approach to annotate selected samples. (2) Push the model to learn features from the latest annotated samples. (3) This idea allows the model to make the most of the data information at the least cost.用特定方法解决这一问题的直观思路分为三个步骤：选择模型无法识别的少量样本。（1）使用特定方法标注所选样本。（2）推动模型从最新标注的样本中学习特征。（3）这一思路使模型能够以最低成本充分利用数据信息。
Multiple researchers try to apply human-in-the-loop-based methods to optimize models from the perspective of data. According to surveys, scientists spend about 80% of their time on data processing compared to model building [30]. We investigated the data processing methods based on human-in-the-loop and established a pipeline as shown in Fig. 3. We reviewed the representative works in data processing and showed the classification result as shown in Table 1. This section explores the strengths and deficiencies of data processing with human-in-the-loop by demonstrating data preprocessing, data annotation, and iterative labeling.多位研究人员尝试应用基于“人在回路”的方法从数据视角优化模型。根据调查，科学家们约 80% 的时间花费在数据处理上，而非模型构建上[30]。我们调查了基于“人在回路”的数据处理方法，并建立了一个如图 3 所示的流水线。我们回顾了数据处理方面的代表性工作，并将分类结果展示在表 1 中。本节通过展示数据预处理、数据标注和迭代标注，探讨了“人在回路”在数据处理方面的优势与不足。
2.1 Data Preprocessing2.1 数据预处理
It is well-known that deep learning is a process of modeling data. The success of deep learning largely depends on the quality of data, and data analysis plays an irreplaceable role in building a more effective model. However, it is not easy to find a static method for data analysis, which means data scientists must analyze existing data by employing experts’ experience. The greatest challenge in data analysis lies in the complexity of high-dimensional data, which makes it hard for models to discover data structure. Furthermore, adjusting these parameters dramatically depends on the knowledge of data experts or domain experts. Motivated by this phenomenon, Self et al. [44] proposed a human-model interactive parameter adjustment mode to facilitate user participation by bridging the gaps between a user’s intention and the parameters of a weighted multidimensional scaling model. Therefore, Doan et al. thought establishing a benchmark is an effective means to solve this challenge [48]. Besides, data analysis inevitably involves two considerations: how to carry out automated parameter analysis methods, and the other is how to explore the ability to establish a specific benchmark. Considering these two issues simultaneously, Laure [52] expanded based on Learn2Clean. They developed automated machine learning approaches (AutoML) that can optimize the hyper-parameters of a considered ML model with a list of by-default preprocessing methods. This method is devoted to proposing a principled and adaptive data preparation approach to help and learn from the user to select the optimal sequence of data preparation tasks. With the rapid development of research, researchers are no longer satisfied with solving specific problems in human-in-the-loop data analysis (HILDA). They are more concerned with several “big picture” questions regarding HILDA. Current data analysis technology can correctly obtain the necessary information and knowledge by constructing a knowledge base or graph. However, for the HILDA community or tools, the degree of attention is not enough. Researchers should pay more attention to such issues and make them more popular in the user community to develop data repositories and tools.众所周知，深度学习是一个对数据进行建模的过程。深度学习的成功在很大程度上取决于数据的质量，而数据分析在构建更有效的模型方面发挥着不可替代的作用。然而，要找到一种静态的数据分析方法并不容易，这意味着数据科学家必须利用专家的经验来分析现有数据。数据分析面临的最大挑战在于高维数据的复杂性，这使得模型难以发现数据结构。此外，调整这些参数在很大程度上依赖于数据专家或领域专家的知识。受此现象启发，Self 等人[44] 提出了一种人机交互参数调整模式，通过弥合用户意图与加权多维标度模型参数之间的差距来促进用户参与。因此，Doan 等人认为建立基准是解决这一挑战的有效手段[48]。此外，数据分析不可避免地涉及两个考虑因素：如何执行自动化参数分析方法，以及如何探索建立特定基准的能力。同时考虑这两个问题，Laure[52] 在 Learn2Clean 的基础上进行了扩展。他们开发了自动机器学习方法（AutoML），能够通过一系列默认的预处理方法来优化所考虑 ML 模型的超参数。该方法致力于提出一种原则性且自适应的数据准备方法，以帮助并向用户学习，从而选择最优的数据准备任务序列。随着研究的快速发展，研究人员不再满足于解决“人在回路”数据分析（HILDA）中的具体问题。他们更关心关于 HILDA 的几个“宏观”问题。当前的数据分析技术可以通过构建知识库或图谱正确获取必要的信息和知识。然而，对于 HILDA 社区或工具而言，关注度还不够。研究人员应更多地关注此类问题，并使其在用户社区中更受欢迎，以开发数据存储库和工具。
Utilizing human-in-the-loop methods to deal with natural language data has inherent advantages over other kinds of data (i.e., speech recognition, medical applications, computer vision, and intelligent transportation system). Most of the human-in-the-loop methods are used in the information extraction stage. As Fig. 4, Gentile et al. [50] propose an interactive dictionary expansion tool using two neural language models. Many researchers add humans to NLP tasks(such as entity analysis, knowledge graphs, etc.) by using crowdsourcing [50, 52, 54]. Ristoski et al. [61] introduced a method of extracting instances from various web resources, which dramatically improves the performance of the system by introducing human-recycling components. Besides, this method can integrate the human experience and knowledge to empower machines’ accurate intelligence. Consistent with the method of dictionary expansion proposed before, the core idea of this method is also realized by expanding the existing dictionary. In addition to the direct annotation, since there are semantic disambiguation phenomena in a few NLP tasks, exploiting the model alone cannot accomplish these tasks. However, the performance of these tasks can be significantly improved by adopting the human knowledge intervention accumulated in unconscious learning. Qian et al. [62] put forward a deep learning-based entity name understanding system called PARTNER, which provided a more reasonable way of interaction. PARTNER is based on active learning and a weak supervision method. It is necessary to apply data screening technology for sample selection to find those error-prone samples in the human-in-the-loop process. Cutler et al. [65] presented a method that marked potentially incorrect labels with high sensitivity in the named entity recognition corpus.利用“人在回路”方法处理自然语言数据比处理其他类型数据（即语音识别、医疗应用、计算机视觉和智能交通系统）具有内在优势。大多数“人在回路”方法被用于信息抽取阶段。如图 4 所示，Gentile 等人[50] 提出了一种使用两个神经语言模型的交互式词典扩展工具。许多研究人员通过众包方式将人类加入到 NLP 任务（如实体分析、知识图谱等）中[50, 52, 54]。Ristoski 等人[61] 引入了一种从各种网络资源中提取实例的方法，通过引入人类回收组件，极大地提升了系统性能。此外，该方法可以整合人类的经验和知识，从而赋予机器准确的智能。与之前提出的词典扩展方法一致，该方法的核心思想也是通过扩展现有词典来实现。除了直接标注外，由于在少数 NLP 任务中存在语义消歧现象，仅靠利用模型无法完成这些任务。然而，通过采用在无意识学习中积累的人类知识干预，这些任务的性能可以得到显著提升。Qian 等人[62] 提出了一种名为 PARTNER 的基于深度学习的实体名称理解系统，它提供了一种更合理的交互方式。PARTNER 基于主动学习和弱监督方法。在“人在回路”过程中，有必要应用数据筛选技术进行样本选择，以找出那些易出错的样本。Cutler 等人[65] 提出了一种在命名实体识别语料库中以高灵敏度标记潜在错误标签的方法。
Our summary of the previous papers finds that most of the existing human-in-the-loop preprocessing studies concern extracting and analyzing complex information in the real world. Nevertheless, there is still very little work that utilizes human-in-the-loop technology to perform data preprocessing on CV tasks. We conjuncture that the nature of this phenomenon is that there is a lack of a perfect mode to integrate the human experience into image processing, detailed discussion can be found in Section 5.我们对以往论文的总结发现，现有的“人在回路”预处理研究大多关注提取和分析现实世界中的复杂信息。然而，利用“人在回路”技术对 CV 任务进行数据预处理的工作仍然很少。我们推测，这种现象的本质在于缺乏一种完美的模式将人类经验整合到图像处理中，详细讨论见第 5 节。
2.2 Data Annotation2.2 数据标注
For new tasks, annotating data is a complex but crucial task to realize artificial intelligence. A considerable number of researchers have proposed employing a human-in-the-loop-based method for fast and precise (compared to fragile labeling) operations, especially in NLP and CV domains.对于新任务而言，标注数据是实现人工智能的一项复杂但至关重要的任务。相当多的研究人员提出采用基于“人在回路”的方法进行快速且精确（相比于脆弱的标注）的操作，特别是在 NLP 和 CV 领域。
In NLP tasks, data annotation is bifurcated into two categories. One is the annotation of specific task datasets, such as entity extraction [50, 51], entity linking [59], and the other is more abstract tasks, such as Q&A tasks [56] and reading comprehension tasks [64].在 NLP 任务中，数据标注分为两类。一类是特定任务数据集的标注，如实体抽取[50, 51]、实体链接[59]，另一类是更抽象的任务，如问答任务[56]和阅读理解任务[64]。
The entity processing task is critical in NLP, and its success or failure directly affects the performance of NLP [69]. Currently, there are two main methods for entity extraction, one is to formulate regular expressions for automatic extraction, and the other is the entity mentioned in the manual tagged document. However, neither of these two strategies can extract entities efficiently and accurately. Zhang et al. [51] devised a human-in-the-loop-based entity extraction method to obtain the best return on investment in a limited time. With the deepening of research, we need to handle a growing base of tasks. The emergence of new schemes is beyond our expectations. Regular expressions can help handle common data, but there is no expected magic for new data never seen before. To mitigate this issue, a few studies proposed approaches to solve the cross-domain problem in entity links. They find the entities mentioned in the text and filter and discriminate them according to entities sorting information. This method is especially suitable for semantic disambiguation tasks [59].实体处理任务在 NLP 中至关重要，其成败直接影响 NLP 的性能[69]。目前，实体抽取主要有两种方法，一种是制定正则表达式进行自动抽取，另一种是手动标记文档中的实体提及。然而，这两种策略都无法高效、准确地抽取实体。Zhang 等人[51] 设计了一种基于“人在回路”的实体抽取方法，以便在有限时间内获得最佳投资回报。随着研究的深入，我们需要处理越来越多的任务基础。新方案的出现超出了我们的预期。正则表达式可以帮助处理常见数据，但对于从未见过的新数据，并没有预期的神奇效果。为了缓解这一问题，一些研究提出了解决实体链接中跨领域问题的方法。他们发现文本中提到的实体，并根据实体排序信息对其进行过滤和辨别。这种方法特别适用于语义消歧任务[59]。
How to cope with more complicated tasks is also a focus of research now. Researchers attempt to integrate human experience and knowledge to endow machines with more intelligence. More specifically, to what extent do neural network models understand the natural language, and can they be further enhanced? To explain this problem and explore more interpretability of the neural network model. Wallace et al. [56] developed an open application system that contains an interactive interface to talk with the machine, thereby generating more Q&A language materials to collect more research data and help the researcher explain the model predictions. Bartolo et al. [64] tried three different sets of annotation methods in the reading comprehension task to build a gradually more robust model in the annotation cycle. Significantly, they created a challenging dataset by collecting 36,000 samples. However, with the enhancement of the cyclic model, the performance gradually deteriorates. In contrast, the more robust model can still learn from the data intensively collected by the weaker model in the loop [70].如何应对更复杂的任务也是目前研究的重点。研究人员试图整合人类经验和知识，赋予机器更多的智能。更具体地说，神经网络模型在多大程度上理解自然语言，以及它们能否得到进一步增强？为了解释这个问题并探索神经网络模型的更多可解释性，Wallace 等人[56] 开发了一个开放式应用系统，该系统包含一个与机器对话的交互界面，从而生成更多的问答语言材料，以收集更多的研究数据并帮助研究人员解释模型预测。Bartolo 等人[64] 在阅读理解任务中尝试了三组不同的标注方法，以在标注周期内构建一个逐渐稳健的模型。值得注意的是，他们通过收集 36,000 个样本创建了一个具有挑战性的数据集。然而，随着循环模型的增强，性能逐渐下降。相比之下，更稳健的模型仍然可以从弱模型在回路中密集收集的数据中学习[70]。
As for CV, human-in-the-loop specifically explored how to exploit weak labeling to provide feedback at present. Besides, it also analyzed how to provide users with a unified intervention experience. It is involved in numerous tasks, such as person re-identification, face recognition, 3D point cloud object detection, and object detection. While considerable current pedestrian re-identification (Re-ID) methods can achieve superior results under the training of a large amount of labeled data, these models cannot produce an exceptional performance as in the experiment when deployed in a natural environment. Moreover, so much data is new in a natural environment because these data have not been in the training set. The trickier part is that new data will constantly accumulate over time, which can cause the model to fail to work. To tackle this problem, Liu et al. [55] proposed a human-in-cycle model based on reinforcement learning, which released the limitation of pre-labeling and upgraded the model through continuously collected data. The goal is to minimize human annotation work meanwhile maximizing the performance of Re-ID. In addition to directly using reinforcement learning for dynamic learning, researchers also pay attention to expanding and refining data on a new task. Facial expression recognition is an exciting task in CV, which is of great help to sentiment analysis and behavior analysis tasks. Traditional facial expression recognition can only deal with the seven simplest facial expressions (i.e.happiness, sadness, fear, anger, disgust, surprise, and contempt). In real life, it is additionally important to deal with more micro-expressions. More specifically, it is an interesting task that builds more refined micro expression processing datasets based on existing expression identification. Butler et al. [60] exploited a micro-expression recognition method based on the human-in-the-loop system. This method provides a flexible interface for manual proofreading of automatically processed tags, thereby ensuring the accuracy and usability of the extended dataset. In addition to directly constructing new datasets, it is also of great significance to explore existing datasets, especially for tasks that are difficult to label, such as target detection tasks, the labeling workload is enormous. To reduce the labor and time cost of annotation of the bounding box of video objects, Le et al. [63] applied an efficient and straightforward interactive self-annotation framework based on cyclic self-supervised learning. The entire framework consists of automatic model learning and interactive processes. The automatic learning process makes the model learn faster and speeds up the interaction process. In the interactive recursive annotation, the detector receives feedback from the human annotator to process the human loop annotation scene. Moreover, to save labeling time, they proposed a new level correction module, which strengthens the utility of neighbor frames by CNN by reducing the distance of annotated frames at each time step. Based on the framework of Le et al., Adhikari et al. [68] modified the framework to be completed in one stage, and the most significant work of humans in it has become to correct errors instead of performing full annotations, which further improved the user experience.
Using the above two approaches is ineffective for more intricate image tasks, such as 3D point cloud labeling tasks. Because of the limited effect of employing only one stage for labeling, Meng et al. [66] designed a multi-stage human-in-the-loop labeling method based on predecessors. However, the previous work only started from the perspective of data annotation. It ignored integrating human experience and knowledge into the model to the greatest extent to incorporate human knowledge and intelligence effectively. Zhang et al. [67] considered specific talents and skills of humans in painting, and these skills cannot be fully quantified as rules and knowledge. If the model can learn painting skills, it would undoubtedly help human-in-the-loop application takes a significant step forward. They present a data-driven framework for generating comics from digital illustrations. To further create high-quality comics, these three components are humanely annotated by the artist.使用上述两种方法对于更复杂的图像任务（如 3D 点云标注任务）是无效的。由于仅采用单阶段进行标注效果有限，Meng 等人[66] 在前人基础上设计了一种多阶段“人在回路”标注方法。然而，以往的工作仅从数据标注的角度出发。它忽略了将人类经验和知识最大程度地整合到模型中，以有效结合人类知识和智能。Zhang 等人[67] 考虑了人类在绘画方面的特殊天赋和技能，而这些技能无法完全量化为规则和知识。如果模型能够学习绘画技能，无疑将有助于“人在回路”应用迈出重要一步。他们提出了一个用于从数字插图生成漫画的数据驱动框架。为了进一步创作高质量漫画，这三个组件由艺术家进行人工标注。
2.3 Iterative Labeling2.3 迭代标注
At present, there is still a high degree of coupling between deep learning tasks and data processing, and the performance of deep learning largely depends on the quality of the data. A large amount of high-quality labeled data is needed for a new task to obtain better performance. However, labeling large-scale data requires a lot of labor and takes a long time, while many iterations of tasks cannot afford such a cost and wait such a long time. Unlike weak annotate and automatically annotate, human-in-the-loop-based methods emphasize finding the essential samples that play a decisive factor in new sample data.目前，深度学习任务与数据处理之间仍然存在高度耦合，深度学习的性能在很大程度上取决于数据的质量。新任务需要大量高质量的标注数据才能获得更好的性能。然而，标注大规模数据需要大量人力且耗时较长，而许多任务的迭代无法承受这种成本并等待如此长的时间。与弱标注和自动标注不同，基于“人在回路”的方法强调寻找在新样本数据中起决定性因素的关键样本。
Unlike the data annotation mentioned above in 2.2, data iterative labeling pays more additional attention to user experience, not just directly allowing users to perform data annotation. From annotation to iterative labeling, the goal has been changed in the following two aspects: to focus on adding knowledge and experience to the learning system. The other is to focus on the interaction with users.与 2.2 节中提到的数据标注不同，数据迭代标注更加关注用户体验，而不只是直接允许用户进行数据标注。从标注到迭代标注，目标在以下两个方面发生了转变：一是专注于向学习系统添加知识和经验；二是专注于与用户的交互。
Yu et al. [33] utilized a partially automated labeling scheme for annotation, which free up human labor by using deep learning of human-in-the-loop. This constitutes the basic prototype of simple iterative annotation. Recently, with the proliferation of reinforcement learning, Liu et al. [55] developed a representative human-in-the-loop system based on reinforcement learning, which applied reinforcement learning to carry out iterative labeling. A typical framework of human-in-the-loop for reinforcement learning is shown in Fig. 5. This novel attempt extends the practical usability of human-in-the-loop to the field of reinforcement learning for the first time, which brings a valuable contribution to the human-in-the-loop community. In addition to implementing the simple manual intervention, they take person Re-Identification as a research task and explore how to minimize human annotation work while optimizing the performance of Re-ID. Fan et al. [57] set out to solve the data challenge in the scheme of network anomaly detection to allow users to intervene in data labeling rather than implement simple user labeling. They introduce a new intelligent labeling method, and the method combines active learning and visual interaction to detect network abnormalities through the iterative labeling process of users. The difference is that they began to pay attention to the connection between the algorithm and the visual interface, and the algorithm and the optical interface are tightly integrated.Yu 等人[33] 利用部分自动化标注方案进行标注，通过使用“人在回路”的深度学习释放了人力。这构成了简单迭代标注的基本原型。最近，随着强化学习的普及，Liu 等人[55] 开发了一种基于强化学习的代表性“人在回路”系统，该系统应用强化学习进行迭代标注。强化学习中“人在回路”的典型框架如图 5 所示。这一新颖尝试首次将“人在回路”的实际可用性扩展到了强化学习领域，为“人在回路”社区做出了宝贵贡献。除了实现简单的人工干预外，他们以行人重识别（Re-ID）作为研究任务，探索如何在优化 Re-ID 性能的同时最小化人工标注工作。Fan 等人[57] 着手解决网络异常检测方案中的数据挑战，允许用户干预数据标注，而不是执行简单的用户标注。他们引入了一种新的智能标注方法，该方法结合了主动学习和视觉交互，通过用户的迭代标注过程来检测网络异常。不同之处在于，他们开始关注算法与视觉界面之间的联系，算法与光学界面紧密集成。
3 Model Training and Inference3 模型训练与推理
In many fields of Artificial Intelligence, such as Natural Language Process (NLP) and Computer Vision (CV), there are a variety of approaches that leverage human intelligence to train and infer experimental results. For both NLP and CV, related research spans deep learning [71] techniques and human-machine hybrid methods. These heuristic methods have taken the diverse quality of human creativity into account to achieve high-quality results.在人工智能的许多领域，如自然语言处理（NLP）和计算机视觉（CV），有多种方法利用人类智能来训练和推断实验结果。对于 NLP 和 CV 而言，相关研究跨越了深度学习[71]技术和人机混合方法。这些启发式方法考虑了人类创造力的多样性，以实现高质量的结果。
3.1 Natural Language Process3.1 自然语言处理
Fig. 6 briefly illustrates the cooperation between the individuals and the model training and inferencing process in the Natural Language Processing Loop. The continuous executive loop develops a more reliable human-AI partnership to a certain extent, contributing to higher accuracy and stronger robustness of the NLP system.图 6 简要说明了个人与自然语言处理回路中模型训练与推理过程之间的协作。持续的执行回路在一定程度上发展了更可靠的人机伙伴关系，有助于提高 NLP 系统的准确性和稳健性。
3.1.1 Text Classification3.1.1 文本分类
Text Classification (TC) is a fundamental NLP task that categorizes a sentence/text into its corresponding category. Karmakharm et al.[72] propose a rumor classification system. The core idea of this system is to obtain additional manual feedback from the journalists to retrain a more accurate machine learning model. This framework first exploits a Rumour Classification System to classify collected social media posts and sends these information back to the journalists. As most state-of-the-art text classification approaches are dominated by the deep neural network [73, 74], which is generally considered as “black boxes” by end-users, another motivation for the researcher to construct the human-in-the-loop framework for TC is to overcome the opaqueness of those models, making them more explainable. To achieve this purpose, Arous et al. [75] put forward a hybrid human-AI framework that gives a moral idea to reinforce human reliability in merging human rationales into a deep learning algorithm. Their work presents MARTA, a Bayesian framework that jointly learns and updates the model parameters and human reliability via an iterative way, enabling the learning processes of parameters and human reliability to benefit from each other until the label and rationales reach agreements.文本分类（TC）是一项基础的 NLP 任务，将句子/文本归入其对应的类别。Karmakharm 等人[72] 提出了一种谣言分类系统。该系统的核心思想是从记者那里获取额外的人工反馈，以重新训练更准确的机器学习模型。该框架首先利用谣言分类系统对收集到的社交媒体帖子进行分类，并将这些信息反馈给记者。由于大多数最先进的文本分类方法都由深度神经网络[73, 74]主导，而这些网络通常被最终用户视为“黑盒”，研究人员构建 TC“人在回路”框架的另一个动机是克服这些模型的不透明性，使其更具可解释性。为了达到这个目的，Arous 等人[75] 提出了一种混合人机框架，提供了一个道德理念，通过将人类基本原理融入深度学习算法来加强人类的可靠性。他们的工作提出了 MARTA，这是一个贝叶斯框架，通过迭代方式联合学习和更新模型参数与人类可靠性，使参数和人类可靠性的学习过程能够相互受益，直到标签和基本原理达成一致。
3.1.2 Syntactic and Semantic Parsing3.1.2 句法和语义分析
Besides text classification, human-in-the-loop approaches for syntactic and semantic parsing are also promising. Syntactic parsing is a process to obtain the valid syntactic structure of input sentences. The goal of semantic parsing is to map natural language to formal domain-specific semantic representations. A human-in-the-loop parsing method [43] is proposed to improve the parsing accuracy of CCG parsing by employing non-expert to answer simple what-questions generated from the parser’s output. These answers are treated as soft constraints when re-training the model. This work is the first attempt at introducing human-in-the-loop for syntactic parsing. However, most parsing technologies still face several challenges: (1) the purpose or expression of users can be ambiguous or vague under some circumstances, posing obstacles for them to get the ground truth in one shot, (2) in the real-world scenario, the performance of state-of-the-art parsers are generally not high enough, and (3) since the mainstream neural network-based models are known as “black-box” that indicates the lack of explainability, it is difficult for end-users to verify the parsing results independently. Currently, Yao et al. [76] propose allowing the semantic parsers system to ask end-users clarification questions and produce an If-Then program simultaneously. Although recent works successfully verified the effectiveness of interactive semantic parsing in practice, they are generally restricted to a specific type of formal language. Furthermore, Yao et al. [77] develop a model-based interactive semantic parsing (MISP) as the general principle for interactive semantic parsing.除了文本分类，用于句法和语义分析的“人在回路”方法也很有前景。句法分析是获取输入句子有效句法结构的过程。语义分析的目标是将自然语言映射到形式化的特定领域语义表示。有人提出了一种“人在回路”的分析方法[43]，通过雇佣非专家回答解析器输出生成的简单“是什么”问题，来提高 CCG 解析的准确性。在重新训练模型时，这些答案被视为软约束。这项工作是引入“人在回路”进行句法分析的首次尝试。然而，大多数解析技术仍面临几个挑战：（1）在某些情况下，用户的目的或表达可能含糊不清，给他们一次性获得真值造成了障碍；（2）在现实场景中，最先进解析器的性能通常不够高；（3）由于主流基于神经网络的模型被称为“黑盒”，这意味着缺乏可解释性，最终用户很难独立验证解析结果。目前，Yao 等人[76] 提出允许语义解析系统询问最终用户的澄清问题，并同时生成 If-Then 程序。尽管近期的工作在实践中成功验证了交互式语义分析的有效性，但它们通常局限于特定类型的形式语言。此外，Yao 等人[77] 开发了一种基于模型的交互式语义分析（MISP），作为交互式语义分析的一般原则。
| Task | Motivation | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Work | TC | SSP | TS | QA | SA | Performance | Interpretability | Usability | Quantitative results |
| Arous et al. (2021) [75] | ✓ | ✓ | ✓ | ||||||
| Karmakharm et al. (2019) [72] | ✓ | ✓ | ✓ | ||||||
| Yao et al. (2019) [76] | ✓ | ✓ | ✓ | ✓ | |||||
| Yao ZiYu et al. (2019) [77] | ✓ | ✓ | ✓ | ||||||
| Ziegler et al. (2019) [78] | ✓ | ✓ | |||||||
| Stiennon et al. (2020) [79] | ✓ | ✓ | |||||||
| Hancock et al. (2019) [80] | ✓ | ✓ | |||||||
| Wallace et al. (2019) [56] | ✓ | ✓ | ✓ | ||||||
| Liu et al. (2021) [81] | ✓ | ✓ | |||||||
3.1.3 Text Summarization3.1.3 文本摘要
Besides applying a human-in-the-loop framework to topic modeling, researchers also use it to generate new texts. Text Summarization (TS) generates a shorter version of a given sentence/text while preserving its meaning [82]. In recent years, there have been some significant breakthroughs in this field. For instance, Ziegler et al. [78] fine-tune pre-trained language models with reinforcement learning by exploiting a reward model trained from human preferences. Then the model is used to generate summaries over Reddit TL, DR, and CNN/DM datasets. However, one limitation of their framework is that there are low agreement rates between labelers and researchers. Stiennon et al. [79] propose to gather a dataset composed of human preferences between pairs of summaries as the first step. Then the prediction of the human-preferred summary is generated by a reward model (RM) trained via supervised learning. Lastly, the score produced by the RM is maximized as much as possible by a policy trained via reinforcement learning (RL). Their method ensures a relatively higher labeler-researcher agreement through the above steps and successfully separates the policy and value networks.除了将“人在回路”框架应用于主题建模外，研究人员还使用它来生成新文本。文本摘要（TS）在保留给定句子/文本含义的同时生成其较短版本[82]。近年来，该领域取得了一些重大突破。例如，Ziegler 等人[78] 通过利用从人类偏好中训练出的奖励模型，使用强化学习对预训练语言模型进行微调。然后，该模型被用于在 Reddit TL, DR 和 CNN/DM 数据集上生成摘要。然而，其框架的一个局限性是标注者与研究人员之间的一致率较低。Stiennon 等人[79] 提出将收集人类对摘要对的偏好作为第一步。然后，通过监督学习训练的奖励模型（RM）生成人类偏好摘要的预测。最后，通过强化学习（RL）训练的策略尽可能最大化 RM 产生的分数。他们的方法通过上述步骤确保了相对较高的人工标注者-研究人员一致性，并成功分离了策略网络和价值网络。
3.1.4 Question Answering3.1.4 问答
Recently, various human-in-the-loop related frameworks have been designed to apply dialogue and Question Answering (QA). The purpose of this task is to allow chatbots/agents to have a conversation with users. These human-in-the-loop dialogue intelligent systems can be bifurcated into two main categories: online feedback loop and offline feedback loop [83]. For the online feedback loop, human feedback is utilized to update the model continuously. Compared with traditional approaches that mismatch the training set and online use case for dialogue systems, researchers have demonstrated that the application of online reinforcement learning can improve the model with human feedback. For instance, a lifetime learning framework is proposed by Hancock et al. [80]. The self-feeding mechanism in this framework enables the chatbot to generate new examples when the conversation with users goes well, and these new examples are exploited to re-train itself continuously. For the offline feedback loop, a large set of human feedback needs to be collected as a training set, then this training set is used to update the model. For instance, Wallace et al. [56] employ “trivia enthusiasts” to creatively generate adversarial examples that can confuse their QA system. These examples are finally implemented for negative training. Since some of the end-user feedback can be misleading, offline methods is more appropriate for improving the robustness of the model.最近，各种与“人在回路”相关的框架已被设计用于应用对话和问答（QA）。该任务的目的是允许聊天机器人/智能体与用户进行对话。这些“人在回路”对话智能系统可分为两大类：在线反馈回路和离线反馈回路[83]。对于在线反馈回路，人类反馈被用于持续更新模型。与对话系统训练集与在线用例不匹配的传统方法相比，研究人员已经证明，应用在线强化学习可以利用人类反馈改善模型。例如，Hancock 等人[80] 提出了一个终身学习框架。该框架中的自喂养机制使得聊天机器人在与用户的对话顺利时能够生成新示例，而这些新示例被用于持续重新训练自身。对于离线反馈回路，需要收集大量人类反馈作为训练集，然后使用该训练集更新模型。例如，Wallace 等人[56] 聘请“琐事爱好者”创造性地生成可能混淆其 QA 系统的对抗性示例。这些示例最终被用于负向训练。由于某些最终用户反馈可能具有误导性，离线方法更适合提高模型的稳健性。
3.1.5 Sentiment Analysis3.1.5 情感分析
Sentiment Analysis (SA) is one of the attractive research branches of Opinion Mining (OM). The research scope of SA is the computational study of individuals’ opinions and attitudes toward entities mentioned in a text. The entities generally refer to individuals or events. Recently numerous neural network-based approaches have been widely utilized and demonstrated their effectiveness in solving sentiment analysis tasks [73, 84, 85]. Most deep learning-based methods for SA use accuracy and F1-score as evaluation metrics. Since these metrics can only evaluate the predictive performance, they lack the mechanisms to explain when and why the sentiment models give false predictions in run-time [86]. Liu et al. [81] introduce an explainable human-in-the-loop SA framework for sentiment analysis task. The execution of their framework is segmented into three steps: First, the human-in-the-loop SA model analyzes local feature contributions. This goal is achieved by executing a data perturbation process. Next, local features are aggregated to calculate the explainable global-level features and humans participate in this loop to assess the relevance of the top-ranked global features to the ground truth and report the errors they find in this process. Finally, the system calculates an erroneous score based on global-level and local-level sentimental features for each instance. Scores higher than a specific threshold are indicated as wrong predictions.情感分析（SA）是观点挖掘（OM）中引人注目的研究分支之一。SA 的研究范围是对个人对文本中提到的实体的观点和态度进行计算研究。实体通常指个人或事件。最近，大量基于神经网络的方法已被广泛使用，并证明了其在解决情感分析任务中的有效性[73, 84, 85]。大多数基于深度学习的 SA 方法使用准确率和 F1-score 作为评估指标。由于这些指标只能评估预测性能，它们缺乏在运行时解释情感模型何时以及为何给出错误预测的机制[86]。Liu 等人[81] 引入了一种用于情感分析任务的可解释“人在回路”SA 框架。其框架的执行分为三个步骤：首先，人在回路 SA 模型分析局部特征贡献。这一目标通过执行数据扰动过程来实现。接下来，聚合局部特征以计算可解释的全局级特征，人类参与此回路以评估排名最高的全局特征与真值的相关性，并报告在此过程中发现的错误。最后，系统根据每个实例的全局级和局部级情感特征计算错误分数。高于特定阈值的分数被标记为错误预测。
3.1.6 Summarization for Human-in-the-Loop in NLP3.1.6 NLP 中“人在回路”综述
A brief overview of representative works in human-in-the-loop NLP is shown in Table 2. For most of the surveyed papers above, their original purpose is to apply human-in-the-loop techniques to various NLP tasks for better performance. The effectiveness of the approaches proposed by these surveyed papers is evaluated via multiple metrics. The experimental results in the documents we investigate show that a relatively small set of human feedback can dramatically boost model performance. For instance, the human-in-the-loop technique improves the classification accuracy for text classification [72]. Similar situations occur in dialogue and question answering where the QA systems have higher ranking metric hits [80]. Besides, human-in-the-loop techniques have also enhanced the model’s robustness and generalization [79]. In addition to improving model performance, some studies have demonstrated that human-in-the-loop methods enable models to be more interpretable and usable in solving NLP problems. For instance, Arous et al. [75] incorporate human rationales into an attention-based Bayesian framework reasonably while weighing worker reliability, thus providing a more human-understandable interpretation of classification results and enhancing the model performance at the same time. Liu et al. [81] chose uni-grams as the explainable feature for LIME [87]; thus the proposed system allows the end-users to understand better the overall contribution of each word to the final sentiment classification made by the model. Wallace et al. [56] invite “trivia enthusiasts” to creatively generate specific adversarial questions that confuse the intelligent question answering system. These questions can be treated as probes further to explore the inherent characteristics of the underlying model behaviors.表 2 展示了“人在回路”NLP 代表性工作的简要概览。对于上述调查的大多数论文，其最初目的是将“人在回路”技术应用于各种 NLP 任务以获得更好的性能。这些被调查论文所提方法的有效性通过多种指标进行评估。我们调查的文档中的实验结果表明，相对少量的人类反馈可以显著提升模型性能。例如，“人在回路”技术提高了文本分类的分类准确率[72]。对话和问答中也出现了类似情况，其中 QA 系统具有更高的排名指标命中率[80]。此外，“人在回路”技术还增强了模型的稳健性和泛化能力[79]。除了提升模型性能外，一些研究还表明，“人在回路”方法使模型在解决 NLP 问题时更具可解释性和可用性。例如，Arous 等人[75] 将人类基本原理合理地纳入基于注意力的贝叶斯框架，同时权衡工作者的可靠性，从而提供了更易于人类理解的分类结果解释，并同时提升了模型性能。Liu 等人[81] 选择单字作为 LIME[87] 的可解释特征；因此，所提出的系统允许最终用户更好地理解每个词对模型最终情感分类的整体贡献。Wallace 等人[56] 邀请“琐事爱好者”创造性地生成特定的对抗性问题，这些问题会混淆智能问答系统。这些问题可以被视为进一步探索底层模型行为固有特征的探针。
3.2 Computer Vision3.2 计算机视觉
In recent years, neural network-based Deep Learning methods (DL) have emerged as the state-of-the-art technique for performing many computer vision tasks [88, 89]. To further improve the performance of these methods, feedback from humans has been integrated into the deep learning architecture to make the whole system more intelligent in solving complex cases that can not be handled politely by the model. A typical human-in-the-loop framework for Computer Vision is outlined in Fig. 7.近年来，基于神经网络的深度学习（DL）方法已成为执行许多计算机视觉任务的最先进技术 [88, 89]。为了进一步提升这些方法的性能，人类反馈已被整合到深度学习架构中，使整个系统在解决模型无法妥善处理的复杂案例时更具智能。图 7 概述了一个典型的计算机视觉“人在回路”（human-in-the-loop）框架。
3.2.1 Object Detection3.2.1 目标检测
Object detection, as one of the most fundamental and challenging problems in computer vision [90], has received significant attention in recent years [91]. The goal of object detection is to detect instances of visual objects of a specific class (such as individuals, vehicles, or other creatures) in digital images. Yao et al. [92] point out that iterations between queries may be expensive and time-consuming, making it unrealistic for executing interaction with end-users. They present an interactive object detection architecture to employ individuals to correct a few annotations proposed by a detector for the un-annotated image with the maximum predicted annotation cost. However, it is still difficult to detect some occluded objects, tiny objects, and blurred objects by these approaches. Madono et al. [93] put forward an efficient human-in-the-loop object detection framework composed of bi-directional deep SORT [94] and annotation-free segment identification (AFSID). Humans’ role in this architecture is to verify the object candidates that bi-directional deep SORT can not detect automatically. Then train the model over the supplementary objects annotated by individuals.目标检测作为计算机视觉中最基础且最具挑战性的问题之一 [90]，近年来受到了广泛关注 [91]。目标检测的目标是检测数字图像中特定类别（如个人、车辆或其他生物）的视觉对象实例。 Yao 等人 [92] 指出，查询之间的迭代可能既昂贵又耗时，使得与最终用户进行交互变得不切实际。他们提出了一种交互式目标检测架构，利用人工来纠正检测器针对未标注图像提出的少量标注，并以此作为预测标注成本最高的部分。然而，这些方法对于检测某些被遮挡、微小及模糊的对象仍然存在困难。Madono 等人 [93] 提出了一种高效的“人在回路”目标检测框架，由双向 deep SORT [94] 和无标注片段识别（AFSID）组成。在此架构中，人类的作用是验证双向 deep SORT 无法自动检测到的对象候选，然后利用人工标注的补充对象对模型进行训练。
3.2.2 Image Restoration3.2.2 图像修复
Image restoration (IR) aims to recover the preliminary version of damaged images [95]. The image inpainting frameworks proposed by previous studies can be bifurcated as exemplar-based [96] approaches and deep learning-based [97] methods. Although profound learning-based works are the mainstream and show decent results, neural network-based approaches constantly suffer from over-fitting when only a relatively small training set is available on a large dataset. Besides, in real-world application, the restored images are often filled with unknown artifacts like uneven texture or monotone color due to the missing crucial semantic information in severely corrupted areas. Weber et al. [98] propose an interactive machine learning system for image restoration based on Deep Image Prior (DIP) [99]. Their proposed human-in-the-loop framework embeds human knowledge into the training process by the following steps. Initially, the images from the dataset are sent to the automated DIP for preliminary restoration. Secondly, the operators actively refine the images via a pre-designed user interface. Thirdly, the refined images are sent back to the input of DIP again for further polishing. Finally, the whole loop continues until the restoration reaches the user’s expectation. In the field of Electron Microscopy, one drawback of automation is that it generally ignores the expertise of the microscopy user that comes with manual analysis. To alleviate such a challenging problem, Roels et al. [100] propose a hybrid human-in-the-loop system that incorporates expert microscopy knowledge with the power of large-scale parallel computing to enhance the Electron Microscopy image quality by exploiting image restoration algorithms.图像修复（IR）旨在恢复受损图像的原始版本 [95]。以往研究提出的图像修复框架可分为基于范例的方法 [96] 和基于深度学习的方法 [97]。尽管基于深度学习的研究是主流并显示出不错的效果，但当大型数据集上仅有相对较小的训练集可用时，基于神经网络的方法往往会遭遇过拟合问题。此外，在实际应用中，由于严重受损区域缺失了关键的语义信息，修复后的图像往往会充满未知的伪影，如纹理不均匀或颜色单调。Weber 等人 [98] 提出了一种基于深度图像先验（DIP）[99] 的交互式机器学习图像修复系统。 他们提出的“人在回路”框架通过以下步骤将人类知识嵌入到训练过程中： 首先，数据集中的图像被发送到自动 DIP 进行初步修复。 其次，操作员通过预先设计的用户界面主动优化图像。 第三，优化后的图像被再次发送回 DIP 的输入端进行进一步润色。最后，整个循环持续进行，直到修复结果达到用户的预期。在电子显微镜领域，自动化的一大缺点是它通常忽略了显微镜用户在手动分析中积累的专业知识。为了缓解这一挑战性问题，Roels 等人 [100] 提出了一种混合“人在回路”系统，通过利用图像修复算法，将专家级的显微镜知识与大规模并行计算能力相结合，以提升电子显微镜的图像质量。
3.2.3 Image Segmentation3.2.3 图像分割
Image segmentation (i.e. semantic segmentation) is a crucial step in most image studies. Image segmentation (IS) aims at assigning a class label to each pixel in the image [101]. This field has recently become explosive popularity because it plays a crucial role in a wide range of computer vision applications [102]. However, few works explore how to effectively expose failures of ��top-performing �� semantic segmentation models and rectify the models by utilizing such counter-examples reasonably. Wang et al. [103] present a two-step hybrid system with human efforts for troubleshooting pixel-level image labeling models. The hybrid system first automatically picks up un-labeled images from a large pool. These selected unlabelled images are used to compose an unlabeled set, which is the most informative in exposing the weaknesses of the target model. To reduce the number of false positives, individuals filter the unlabeled set to obtain a smaller straight set. In the second step, they fine-tune and re-train the target model to study from the counter-examples contained in the refining set without ignoring previously seen examples. Data annotation is always complicated and expensive in the medical image process domain [104]. Ravanbakhsh et al. [105] introduce a training protocol based on combining the conditional Generative Adversarial Network (cGAN) and human workers interactively. For complex cases, human experts are responsible for annotating them. These newly annotated images are used to continue the training and inference procedure.图像分割（即语义分割）是大多数图像研究中的关键步骤。图像分割（IS）旨在为图像中的每个像素分配一个类标签 [101]。该领域最近变得非常热门，因为它在广泛的计算机视觉应用中起着至关重要的作用 [102]。然而，很少有研究探讨如何有效地暴露“表现最佳”的语义分割模型的失败案例，并利用这些反例合理地修正模型。Wang 等人 [103] 提出了一种包含人工参与的二步混合系统，用于排除像素级图像标注模型的故障。该混合系统首先从大型库中自动挑选未标注图像。这些选定的未标注图像被用来组成一个未标注集，该集合在暴露目标模型弱点方面信息量最大。为了减少误报，人工会对未标注集进行过滤，从而获得一个较小的精简集。在第二步中，他们对目标模型进行微调和再训练，以从精简集中包含的反例中学习，同时不忽略先前见过的示例。在医学图像处理领域，数据标注总是复杂且昂贵的 [104]。Ravanbakhsh 等人 [105] 引入了一种基于条件生成对抗网络（cGAN）与人工交互相结合的训练协议。对于复杂案例，由人类专家负责进行标注。这些新标注的图像被用于继续训练和推理过程。
| Task | Motivation | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Work | OD | IR | IS | IE | VOS | Performance | Interpretability | Usability | Quantitative results |
| Yao et al. (2012) [92] | ✓ | ✓ | |||||||
| Madono et al. (2020) [93] | ✓ | ✓ | |||||||
| Roels et al. (2019) [100] | ✓ | ✓ | ✓ | ||||||
| Weber et al. (2020) [98] | ✓ | ✓ | |||||||
| Wang et al. (2020) [103] | ✓ | ✓ | |||||||
| Ravanbakhsh et al. (2020) [105] | ✓ | ✓ | |||||||
| Murata et al. (2019) [106] | ✓ | ✓ | |||||||
| Fischer et al. (2020) [107] | ✓ | ✓ | |||||||
| Benard et al. (2017) [108] | ✓ | ✓ | ✓ | ||||||
| Oh et al. (2019) [109] | ✓ | ✓ | ✓ | ||||||
3.2.4 Image Enhancement3.2.4 图像增强
As one of the challenging issues in computer vision, the purpose of Image Enhancement (IE) is to process an image and generate a new and improved one that is more suitable for a specific application [110]. The research field of image enhancement has attracted ample attention from researchers in recent years, especially after the emergence of deep neural network algorithms [111]. However, most current frameworks have ignored the user preferences and experiences, enhancing the image only via a black-box style, which can leave end-users with sub-optimal results that are not suitable for their specific taste. Murata et al. [106] take user preference into account. The user first provides an example to his system; the image enhancement functions in the framework are applied to the example image via randomly selected parameters. Several objective photos are produced, and the end-user needs to score each of the images. Then the RankNet [112] is exploited to learn the user’s preference from these scores. During the learning process, based on the scores given by users, parameters are optimized to make the generated enhanced images suitable for the taste of users. Fischer et al. [107] devise Neural Image Correction and Enhancement Routine (NICER). A component called Image Manipulator in the NICER first exploits a series of learned image operations (e.g. contrast, brightness) with variable magnitude onto the original image provided by users. Another module named Quality Assessor is followed to evaluate the final enhancement quality by generating related scores. This system iteratively optimizes the parameters of the image enhancement functions to maximize the scores given by the Quality Assessor. Users can modify the parameters of the Image Manipulator before, during, and after the optimization process, guiding the optimization procedure towards more satisfying local optima.作为计算机视觉中的挑战性问题之一，图像增强（IE）的目的是处理图像并生成适合特定应用的新型增强图像 [110]。近年来，尤其是随着深度神经网络算法的出现，图像增强研究领域吸引了研究人员的广泛关注 [111]。然而，目前大多数框架忽略了用户的偏好和经验，仅以“黑盒”方式增强图像，这可能导致最终用户得到不符合其特定品味的次优结果。Murata 等人 [106] 将用户偏好考虑在内。用户首先向其系统提供一个示例；框架中的图像增强功能通过随机选择的参数应用于示例图像。系统产生几张客观照片，最终用户需要对每张图像进行评分。然后利用 RankNet [112] 从这些分数中学习用户的偏好。 在学习过程中，根据用户给出的分数，参数被优化以使生成的增强图像符合用户的品味。Fischer 等人 [107] 设计了神经图像校正与增强程序（NICER）。NICER 中一个名为“图像操纵器”（Image Manipulator）的组件首先利用一系列学习到的图像操作（如对比度、亮度）对用户提供的原始图像进行不同幅度的处理。随后，一个名为“质量评估器”（Quality Assessor）的模块通过生成相关分数来评估最终的增强质量。该系统迭代优化图像增强功能的参数，以最大化“质量评估器”给出的分数。用户可以在优化过程之前、期间和之后修改“图像操纵器”的参数，从而引导优化过程向更令人满意的局部最优解发展。
3.2.5 Video Object Segmentation3.2.5 视频对象分割
The goal of Video Object Segmentation (VOS) is to segment a particular object instance in the entire video sequence of the object mask on a manual or automatic first frame [113]. This research area has become popular in the computer vision community [114]. Since videos have intrinsic characteristics such as motion blur, bad composition, occlusion, etc., it is harder for fully automatic approaches to segment more complex sequences accurately. Employing user to input for each frame is unrealistic due to its expensive costs and time consumption. Thus, the human-in-the-loop framework is adopted for solving such problems. Benard et al. [108] introduce a novel interactive video object segmentation method based on [115]. The core idea of their human-in-the-loop framework is to utilize the current segmentation mask as an additional input. A practical framework for interactive segmentation scenario is designed by Oh et al. [109], named Interaction-and-Propagation Networks (IPN). The IPN is composed of two modules and the critical architecture of these two modules is deep convolutional neural networks. The primary operations of these two modules are interaction and propagation, respectively. Individuals are allowed to interact with the proposed model several times; meanwhile, the feedback is provided in scribbles on multiple frames during this interactive procedure.视频对象分割（VOS）的目标是在手动或自动确定第一帧的对象掩码后，分割整个视频序列中的特定对象实例 [113]。该研究领域在计算机视觉界已变得流行 [114]。由于视频具有运动模糊、构图不佳、遮挡等内在特征，全自动方法很难准确分割更复杂的序列。 由于成本高昂且耗时，为每一帧调用人工输入是不现实的。因此，采用“人在回路”框架来解决此类问题。Benard 等人 [108] 引入了一种基于 [115] 的新型交互式视频对象分割方法。其“人在回路”框架的核心思想是将当前的分割掩码作为附加输入。Oh 等人 [109] 设计了一个实用的交互式分割场景框架，命名为交互与传播网络（IPN）。IPN 由两个模块组成，这两个模块的核心架构是深度卷积神经网络。这两个模块的主要操作分别是交互和传播。允许人工与所提出的模型进行多次交互；同时，在此交互过程中，通过在多个帧上涂鸦来提供反馈。
3.2.6 Summarization for Human-in-the-Loop in CV3.2.6 计算机视觉中“人在回路”的总结
A brief overview of representative studies in human-in-the-loop CV is displayed in Table 3. It can be observed from Table 3 that the motivation of all the surveyed human-in-the-loop works for computer vision is to boost the model performance. From the experiment results of all these surveyed papers, although the evaluation criteria are different, the system that incorporates the human-in-the-loop method performs better than without combining it. Taking Madono et al. [93] as an example, they conduct experiments for pedestrian detection and the results have proven at least two advantages for this task: For one thing, the proposed approach boosts the recall rate by 11 at most over deep SORT. For the other, the amount of unlabeled samples that need manual annotation is decreased by 67 at most compared with bi-directional deep SORT without AFSID, which dramatically improves the overall model performance. Associating the contents in Table 2, this phenomenon in CV is similar to NLP, which demonstrates that the core motivation of almost all human-in-the-loop studies in both CV and NLP serves the purpose of boosting model performance. We also notice that in Table 3, only one work [100] tries to bring interpretability for the model. Roels et al. [100] have validated the potential enhancements that DenoisEM can provide in 3D EM image interpretation by denoising SBF-SEM image data of an Arabidopsis thaliana root tip. Besides, human-in-the-loop conception can also improve the usability of CV models. For instance, Madono et al. [93] have proven that their framework is advantageous/practical in scenarios where obtaining annotations is a costly affair. Oh et al. [109] validate the usefulness and robustness of their Interaction-and-Propagation Networks with real interactive cutout use-cases. Hudec et al. [116] propose a new classification of aggregate functions in terms of mixed behavior by the variability of ordinal sums of associative and disjunctive functions, which play an important role for model fusing the knowledge of domain experts.表 3 展示了计算机视觉中“人在回路”代表性研究的简要概述。从表 3 可以观察到，所有被调查的计算机视觉“人在回路”研究的动机都是为了提升模型性能。从所有这些被调查论文的实验结果来看，尽管评估标准不同，但结合了“人在回路”方法的系统表现优于未结合该方法的系统。以 Madono 等人 [93] 为例，他们针对行人检测进行了实验，结果证明了该任务的至少两个优势：一方面，所提出的方法相比 deep SORT 将召回率最高提升了 11%；另一方面，与没有 AFSID 的双向 deep SORT 相比，需要人工标注的未标注样本数量最多减少了 67%，这极大地改善了整体模型性能。结合表 2 的内容，计算机视觉中的这种现象与自然语言处理（NLP）类似，这表明计算机视觉和自然语言处理中几乎所有“人在回路”研究的核心动机都是为了提升模型性能。我们还注意到，在表 3 中，只有一项研究 [100] 尝试为模型引入可解释性。Roels 等人 [100] 通过对拟南芥根尖的 SBF-SEM 图像数据进行去噪，验证了 DenoisEM 在 3D 电镜图像解释方面所能提供的潜在改进。此外，“人在回路”理念还可以提高计算机视觉模型的可用性。例如，Madono 等人 [93] 已证明他们的框架在获取标注成本高昂的场景中具有优势/实用性。Oh 等人 [109] 通过真实的交互式抠图用例验证了其交互与传播网络的有效性和鲁棒性。 Hudec 等人 [116] 提出了一种新的聚合函数分类法，该分类法基于关联函数和析取函数的序和的可变性，在模型融合领域专家的知识方面发挥着重要作用。
4 System construction and Application4 系统构建与应用
| Systems | Application | Year | Role of human | Description | Quantitative results | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SECS | CP | SIMS | SE | Others | Supervisor | Supervisee | Collaborator | User | ||||
| Brostoff & Sasse [117] | ✓ | 2001 | ✓ | ✓ | Security system for traditional tasks | |||||||
| Cranor [118] | ✓ | 2008 | ✓ | ✓ | Security system for traditional tasks | |||||||
| MacHiry et al. [119] | ✓ | 2013 | ✓ | Software testing | ||||||||
| Kovashka et al. [120] | ✓ | 2015 | ✓ | Image-based searching engine | ||||||||
| Louis Rosenberg [121] | ✓ | 2016 | ✓ | Crowdsourcing | ||||||||
| Yan et al. [122] | ✓ | 2017 | ✓ | ✓ | Software testing | |||||||
| Wogalter [123] | ✓ | 2018 | ✓ | ✓ | Security system for traditional tasks | |||||||
| MA [124] | ✓ | 2018 | ✓ | AI model optimizing in training | ||||||||
| Salam et al. [125] | ✓ | 2019 | ✓ | AI model optimizing in testing | ||||||||
| Plummer et al. [126] | ✓ | 2019 | ✓ | Image-based searching engine | ||||||||
| Fredrik Wrede [127] | ✓ | 2019 | ✓ | ✓ | Stochastic gene regulatory | |||||||
| Singh and Mahmoud [128] | ✓ | 2020 | ✓ | ✓ | ✓ | Security system for traditional tasks | ||||||
| Demartiniet al. [129] | ✓ | 2020 | ✓ | ✓ | Security system for modern tasks | |||||||
| ODEKERKEN & BEX [130] | ✓ | 2020 | ✓ | Security system for modern tasks | ||||||||
| Bohme et al. [131] | ✓ | 2020 | ✓ | Program repairing | ||||||||
| Renner [132] | ✓ | 2020 | ✓ | ✓ | AI model optimizing in the whole steps | |||||||
| Davidson et al. [133] | ✓ | 2021 | ✓ | ✓ | Simulation system for decision making | |||||||
| Demirel [134] | ✓ | 2020 | ✓ | Simulation system for process forecast | ||||||||
| Metzner et al. [135] | ✓ | 2020 | ✓ | Simulation system for procedure control | ||||||||
| Polisetty & Avinesh [136] | ✓ | 2020 | ✓ | Searching engine | ||||||||
| Zhu et al. [137] | ✓ | 2020 | ✓ | ✓ | Renal Pathology | |||||||
| Li et al. [138] | ✓ | 2020 | ✓ | Model checking | ||||||||
Previously, we summarized and reviewed the human-in-the-loop workaround of ‘data processing’ and ‘model training and inference’. Moreover, we noticed that some researchers focus on how to build human-centric applications in a broad spectrum of application scenarios. We selected representative studies from four application scenarios of Security Systems, Code Production Tools, Simulation Systems, and Search Engines to summarize, as shown in Table 4.此前，我们总结并回顾了“数据处理”以及“模型训练与推理”方面的“人在回路”解决方案。此外，我们注意到一些研究人员专注于如何在广泛的应用场景中构建以人为中心的应用程序。 我们从安全系统、代码生产工具、仿真系统和搜索引擎这四个应用场景中选择了代表性研究进行总结，如表 4 所示。
4.1 Security System4.1 安全系统
The security system is an application system with solid coupling with people (Fig. 8). With the development of deep learning algorithm, researchers begin to explore more effective human-machine cooperation modes to replace the manual intervention process gradually. Humans usually fail to execute certain events due to some objective reasons (i.e.inattentiveness, increasing age, nonproficiency). At present, many safety systems are highly coupled to people, which would lead to the continuous amplification of errors. To address this challenge, humans should maximize the chances of operators successfully performing their safety-critical functions. To this end, Cranor [118] proposed a human-in-the-loop-based reasoning framework, which provided a systematic approach to identifying potential causes of human failures, thereby helping system designers use the framework to identify problem areas and proactively address defects before building systems. Similarly, in order to stop operator error in time, Singh et al. [128] started by addressing the operational safety of nuclear power plants (NPP) and the commercial aviation industry. They first used the interaction between operator and control panel (HMI) state, where a visual feedback loop is used to predict the near-term expected state. The HMI state sequence prediction is transformed into a language translation problem and finely modeled in a human-in-the-loop fashion. The proposed method is validated by a dataset of natural factory scenes, demonstrating the effectiveness of the method. 安全系统是一个与人紧密耦合的应用系统（图 8）。随着深度学习算法的发展，研究人员开始探索更有效的人机协作模式，以逐步取代人工干预过程。 由于某些客观原因（如注意力不集中、年龄增长、不熟练），人类通常无法执行特定的事件。 目前，许多安全系统与人的耦合度很高，这会导致错误的持续放大。 为了应对这一挑战，人类应最大限度地提高操作员成功执行其安全关键功能的机会。 为此，Cranor [118] 提出了一个基于“人在回路”的推理框架，该框架提供了一种系统的方法来识别导致人为失误的潜在原因，从而帮助系统设计者利用该框架识别问题区域，并在构建系统之前主动解决缺陷。 同样，为了及时制止操作员的错误，Singh 等人 [128] 从核电站（NPP）和商业航空业的运行安全入手。 他们首先利用操作员与控制面板（HMI）状态之间的交互，其中视觉反馈回路被用于预测近期的预期状态。 HMI 状态序列预测被转化为一个语言翻译问题，并以“人在回路”的方式进行精细建模。 所提出的方法通过一个自然工厂场景的数据集进行了验证，证明了该方法的有效性。
The workflow of the human-in-the-loop security system can be summarized as Fig. 9 in which the artificial intelligence algorithm and humans (experts or crowd-sourcing workers) collaborate all together. Demartini et al. [129] devised a solution to combat online misinformation by combining machine learning algorithms, crowd-sourcing workers and experts, and their introduction to combining automatic and manual fact-checking methods to the challenges and opportunities in combating the spread of online misinformation. They pointed out that by addressing the main question of “who should do what”, the cost and performance of the three roles can be balanced, along with the improvement of credibility and safety of the human-in-the-loop system. For the same purpose, ODEKERKEN et al. [130] introduced a proxy framework for transparent human-in-the-loop classification that combines dynamic arguments with legal case-based reasoning to create a system that can elaborate its decisions and adapt to new situations. In this system, human analysts can add new factors to update system results and make recommendations to the classification algorithm as supervisors.“人在回路”安全系统的工作流程可概括为图 9，其中人工智能算法与人类（专家或众包工作者）共同协作。 Demartini 等人 [129] 设计了一种解决方案，通过结合机器学习算法、众包工作者和专家来打击网络虚假信息，并介绍了将自动和手动事实核查方法相结合以应对打击网络虚假信息传播过程中的挑战和机遇。 他们指出，通过解决“谁该做什么”这一主要问题，可以平衡这三种角色的成本和性能，同时提高“人在回路”系统的可信度和安全性。 出于同样的目的，ODEKERKEN 等人 [130] 引入了一个用于透明“人在回路”分类的代理框架，该框架将动态论证与基于法律案例的推理相结合，创建了一个能够阐述其决策并适应新情况的系统。 在该系统中，人类分析师可以作为监督者添加新因素来更新系统结果，并向分类算法提供建议。
Across that span of time, with the further development of Internet Technology, security systems become an urgent need. However, the use of human-in-the-loop is not limited to virus detection and fraudulent information filtering and has also shown remarkable performance in issues such as privacy protection, authentication attack prevention, and spam filtering.在此期间，随着互联网技术的进一步发展，安全系统成为了一种迫切需求。 然而，“人在回路”的使用并不局限于病毒检测和欺诈信息过滤，在隐私保护、身份验证攻击预防和垃圾邮件过滤等问题上也表现出了显著的性能。
4.2 Code Production Tools4.2 代码生产工具
The programming of programs and the training of models are essential subtasks of artificial intelligence, and this two tasks are highly coupled with humans. However, with the establishment of the open-source community, various codes and model resources have been disclosed, making simple program writing and model training a repetitive work and thus leading to an automated basis for the model. The fundamental difficulty is making integration tools learn to splice existing components, which requires integration tools to learn human integration experience. Under the guidance of this idea, there have been some tools for automation. With the help of these practical tools, developers can collaborate on existing projects, rather than writing projects from scratch (Figure 10).程序的编程和模型的训练是人工智能的基本子任务，这两个任务与人类高度耦合。然而，随着开源社区的建立，各种代码和模型资源已被公开，使得简单的程序编写和模型训练成为重复性工作，从而为模型的自动化奠定了基础。其根本困难在于使集成工具学会拼接现有组件，这需要集成工具学习人类的集成经验。在该思想的指导下，已经出现了一些自动化工具。在这些实用工具的帮助下，开发人员可以协作现有项目，而不是从零开始编写项目（图 10）。
Software testing was first addressed. MacHiry et al. [119] designed a test generation system called Dynodroid for fuzzing unmodified Android applications. Dynodroid decomposed an application into a set of event-driven programs. Dynodroid employed the Android framework to generate a series of events and automatically executed these generated events to interact with the surrounding environment. It collected interactive feedback to program functions. The most worth mentioning is that Dynodroid would use manual intervention mode when necessary, and also tried to exploit input feedback to adjust the frame to generate new output dynamically. Through the above approach, Dynodroid implemented fuzz testing of unmodified Android applications in a human-machine hybrid manner. Yan et al. [122] put forward a human& tool-centric vulnerability analysis system that leveraged humans (with varying levels of expertise) to perform well-defined subtasks. The vulnerability analysis system had a larger scale of available programs and higher labor utilization than previous work. Besides, in the practical application situation, the coders pay attention to the test of the code and want to see the application that can make the program correction. Bohme et al. [131] proposed the first human-in-the-loop semi-automatic program repair framework called LEARN2FIX. LEARN2FIX repaired bugs by negotiating with users to observe errors, trained an automatic error oracle through the bug samples marked in the negotiation and finally realized the automatic repair of the program. LEARN2FIX learned a sufficiently accurate automatic oracle with relatively low labeling effort (e.g. 20 queries). Productization tools of human-in-the-loop systems in software engineering bring great convenience to programmers. In the future, the application of human-in-the-loop in this field will expand from debugging and software testing to most coding programs. In particular, with the introduction of pre-trained models such as BERT [11] and GPT [12], human-machine collaborative programming is becoming a new focus.软件测试首先得到了解决。MacHiry 等人 [119] 设计了一个名为 Dynodroid 的测试生成系统，用于对未经修改的 Android 应用程序进行模糊测试。 Dynodroid 将应用程序分解为一组事件驱动的程序。 Dynodroid 利用 Android 框架生成一系列事件，并自动执行这些生成的事件以与周围环境进行交互。它收集交互反馈以用于程序功能。最值得一提的是，Dynodroid 会在必要时使用人工干预模式，并尝试利用输入反馈来动态调整框架以生成新的输出。 通过上述方法，Dynodroid 以人机混合的方式实现了对未经修改的 Android 应用程序的模糊测试。 Yan 等人 [122] 提出了一种以人与工具为中心的漏洞分析系统，该系统利用人类（具备不同专业水平）来执行定义明确的子任务。与以往工作相比，该漏洞分析系统具有更大规模的可用程序和更高的劳动利用率。 此外，在实际应用中，编码人员关注代码测试，并希望看到能够进行程序修正的应用程序。 Bohme 等人 [131] 提出了第一个名为 LEARN2FIX 的“人在回路”半自动程序修复框架。 LEARN2FIX 通过与用户协商观察错误来修复 bug，通过协商中标记的 bug 样本训练自动错误预言机，最终实现了程序的自动修复。LEARN2FIX 以相对较低的标注工作量（例如 20 次查询）学习到了足够准确的自动预言机。 软件工程中“人在回路”系统的产品化工具为程序员带来了极大的便利。未来，该领域中“人在回路”的应用将从调试和软件测试扩展到大多数编码程序。特别是随着 BERT [11] 和 GPT [12] 等预训练模型的引入，人机协作编程正成为一个新的焦点。
With the development of machine learning, the training of deep models has become another prevalent task. With the continuous exploration of model training, more researchers focus on constructing semi-automatic or even fully automatic training tools. The researchers make efforts to incorporate human knowledge during various stages of model training (Fig. 11). To solve the problem that the underlying structure of the model is unknown in the model design process, Salam et al. [125] applied a semi-automated human-in-the-loop attribute design framework to assist human analysts in converting original attributes into classifications of valid derived properties of the question. The framework first provided a human analyst with k buckets containing promising original attribute selections based on a random walk-based heuristic. It then iteratively allowed human analysts to deal with attributes involving attributes based on a scalable and efficient greedy heuristic algorithm. The top-l derived attribute finally got a designed model frame, and the designer completed the detailed design interactively. To address the model training process, the ultimate goal of an ML system becomes the problem of reducing the time it takes to get a deployable model from scratch. MA et al. [124] exploited a human-machine collaboration system called Helix which optimized execution across iterations by appropriately reusing or recomputing intermediate results. It is worth pointing out that Helix contained a workflow management module as well as a visualization module to interact with people, thus enhancing the usability of the tool. To solve the problem that transparency and operability are ignored in the process of model optimization, Renner et al.[132] focused on both transparency (how the system is explained to humans) and operability (how users provide feedback or coach the system) in interactive machine learning. It combined novel topic visualization techniques with a human-centered interactive topic modeling system. Furthermore, it revealed how users understand and interact with machine learning models and guided the further development of human-in-the-loop systems. 随着机器学习的发展，深度模型的训练已成为另一项普遍任务。 随着对模型训练的不断探索，更多的研究人员专注于构建半自动甚至全自动的训练工具。 研究人员致力于在模型训练的各个阶段整合人类知识（图 11）。 为了解决模型设计过程中模型底层结构未知的问题，Salam 等人 [125] 应用了一种半自动的“人在回路”属性设计框架，以协助人类分析师将原始属性转换为问题的有效派生属性分类。该框架首先基于随机游走启发式算法，为人类分析师提供包含有前景的原始属性选择的 k 个桶。然后，它迭代允许人类分析师基于可扩展且高效的贪婪启发式算法处理涉及属性的问题。最终，前 l 个派生属性获得了一个设计好的模型框架，设计者以交互方式完成了详细设计。 为了解决模型训练过程，ML 系统的最终目标变成了减少从零开始获得可部署模型所需时间的问题。 MA 等人 [124] 利用了一个名为 Helix 的人机协作系统，通过适当地重用或重新计算中间结果来优化跨迭代的执行。值得指出的是，Helix 包含一个工作流管理模块以及一个与人交互的可视化模块，从而提高了工具的可用性。 为了解决模型优化过程中透明度和可操作性被忽略的问题， Renner 等人 [132] 在交互式机器学习中同时关注透明度（系统如何向人类解释）和可操作性（用户如何提供反馈或指导系统）。它将新颖的主题可视化技术与以人为中心的交互式主题建模系统相结合。此外，它揭示了用户如何理解和与机器学习模型交互，并指导了“人在回路”系统的进一步发展。
4.3 Simulation System4.3 仿真系统
The simulation system generally utilizes virtual systems to simulate an object or workflow, and is widely applied in the decision-making of system construction, process forecast, and safety control. Due to the feature of these applications, interaction with a human is indispensable. A good many researchers have been working towards the goal of improving the usability of the human-in-the-loop in predicting and optimizing the integration of soldier systems. Davidson et al. [133] presented a semi-structured interview method with key informants, and in the review method, eight critical requirements for improving human-in-the-loop simulation systems were identified. Addressing essential requirements can improve the ability of current human-in-the-loop simulation tools to accommodate the military’s need for human consideration early in the design process. In addition, the system can also be used for other planning problems such as cargo, transportation problems, and traffic management problems. Other researchers are concerned with the simulation of processes, the most representative of integrating human-machine dialogue into a system. Demirel et al. [134] proposed a novel human-computer interaction method to evaluate the safety and performance of human-product interactions. This method modeled humans into a computationally designed environment through digital human modeling technology. With this system, the form and function of the characters in the workflow could be pre-designed, and ergonomic evaluation indicators could be used to implement the work cycle. In terms of program control, Metzner et al. [135] proposed a simulated human-robot collaboration method that combined virtual reality, motion tracking, and standard simulation software for industrial robots. It used a virtual reality system to simulate the workplace of a robot and its collaborators to design a motion tracking system. The system captured human movements by fusing two above-mentioned subjects. Among other things, this approach would evaluate the performance of the human-robot collaboration system safety controls and the fulfillment of the defined requirements.仿真系统通常利用虚拟系统来模拟对象或工作流，并广泛应用于系统构建、过程预测和安全控制的决策中。由于这些应用的特性，与人的交互是必不可少的。 许多研究人员一直致力于提高“人在回路”在预测和优化士兵系统集成方面的可用性。Davidson 等人 [133] 提出了一种与关键信息提供者进行半结构化访谈的方法，在该审查方法中，确定了改进“人在回路”仿真系统的八项关键需求。解决这些基本需求可以提高当前的“人在回路”仿真工具满足军事在设计过程早期对人为因素考量需求的能力。此外，该系统还可用于其他规划问题，如货物、运输问题和交通管理问题。 其他研究人员关注过程仿真，其中最典型的是将人机对话整合到系统中。 Demirel 等人 [134] 提出了一种新颖的人机交互方法，用于评估人机交互的安全性和性能。该方法通过数字人体建模技术将人类建模到计算机设计的环境中。利用该系统，可以预先设计工作流中角色的形式和功能，并使用人机工程学评估指标来实现工作周期。 在程序控制方面，Metzner 等人 [135] 提出了一种模拟的人机协作方法，结合了虚拟现实、运动跟踪和工业机器人标准仿真软件。它使用虚拟现实系统来模拟机器人及其合作者的工作场所，以设计运动跟踪系统。该系统通过融合上述两个主题来捕捉人类运动。除其他事项外，该方法还将评估人机协作系统安全控制的性能以及对既定需求的满足情况。
4.4 Search Engine4.4 搜索引擎
The automation and intelligence of search engines are also important research areas. We discover that current search engine is still positioned as an human assistant, but if the search engine can further model the human work, it can be more practical. At present, the researches on search engines are mainly divided into recommendation system and image search. Polisetty et al. [136] introduced a new joint recommender system that is updated by continuously learning from user feedback. Recommender systems better prepare information from the three main research directions of information aggregation, enrichment, and recommendation by placing humans in the loop. The system features review summarization and rating prediction, with a web-based interface for improving methods through human-computer interaction. With the development of computer vision, searching through images has become an important task and the image search task has been well solved due to the fast development of deep learning. However, the visual-text semantic matching task remains a significant challenge that affects interaction performance and search results. Since humans have an inherent advantage in handling image search tasks, trying to introduce users into the image search loop (Figure 12) has excellent potential. Kovashka et al. [120] devised a new feedback model for image search that allowed users to interact, communicated preferences through visual comparisons, and ultimately determined user searches. First, the user initiates a query; then the system initiates feedback on the user’s query, and the user provides feedback; then the system collects user feedback and responds until the system no longer receives user feedback; finally, the system learns from the user feedback and updates the system. Compared to traditional passive and active methods, the system can provide more accurate search results with less user interaction through an iterative learning process. Building on Kovashka et al., Plummer et al. [126] put forward an attribute-based interactive image search method, which can iteratively refine image search results using human-in-the-loop feedback. Unlike before, the system trains a deep reinforcement model to learn which images are informative, rather than relying on manual measurements commonly exploited in previous work. Besides, this work extends conditional similarity networks to incorporate global similarity into training visual embeddings, resulting in more natural transitions when users explore learned similarity embeddings. The system is self-updating to provide more accurate image search results. With the help of human-in-the-loop, we believe that search engines can continuously adapt to different changes through interactive learning to meet searchers’ personalized needs better.搜索引擎的自动化和智能化也是重要的研究领域。我们发现当前的搜索引擎仍然被定位为人类助手，但如果搜索引擎能够进一步对人类工作进行建模，它可能会更加实用。 目前，对搜索引擎的研究主要分为推荐系统和图像搜索。 Polisetty 等人 [136] 引入了一种新的联合推荐系统，该系统通过不断学习用户反馈进行更新。推荐系统通过将人类置于回路中，更好地从信息聚合、丰富和推荐这三个主要研究方向准备信息。该系统具有评论摘要和评分预测功能，并配有基于 Web 的界面，可通过人机交互改进方法。 随着计算机视觉的发展，通过图像搜索已成为一项重要任务，并且由于深度学习的飞速发展，图像搜索任务已得到很好的解决。然而，视觉-文本语义匹配任务仍然是一个影响交互性能和搜索结果的重大挑战。由于人类在处理图像搜索任务方面具有内在优势，尝试将用户引入图像搜索回路（图 12）具有巨大的潜力。 Kovashka 等人 [120] 为图像搜索设计了一种新的反馈模型，允许用户进行交互，通过视觉比较传达偏好，并最终确定用户搜索。 首先，用户发起查询；然后系统对用户的查询发起反馈，用户提供反馈；接着系统收集用户反馈并做出响应，直到系统不再收到用户反馈；最后，系统从用户反馈中学习并更新系统。与传统的被动和主动方法相比，该系统可以通过迭代学习过程以更少的用户交互提供更准确的搜索结果。 在 Kovashka 等人的基础上，Plummer 等人 [126] 提出了一种基于属性的交互式图像搜索方法，该方法可以使用“人在回路”反馈迭代地细化图像搜索结果。与以往不同，该系统训练深度强化模型来学习哪些图像具有信息量，而不是依赖于以往工作中常用的手动测量。此外，这项工作扩展了条件相似性网络，将全局相似性整合到视觉嵌入的训练中，从而在用户探索学习到的相似性嵌入时产生更自然的转换。 该系统是自更新的，以提供更准确的图像搜索结果。 在“人在回路”的帮助下，我们相信搜索引擎可以通过交互式学习不断适应各种变化，从而更好地满足搜索者的个性化需求。
In addition to the above scenarios, human-in-the-loop systems are also used in other fields [127, 137], such as bioinformatics, supervisory healthcare, and crowd-sourcing work. For example, Fredrik Wrede et al. [127] utilized human-in-the-loop semi-supervised learning for stochastic gene regulation. Rosenberg et al. [121] applied an artificial swarm intelligence scheduling system for crowd-sourcing tasks, Li et al. [138] employed a model checking method for human-in-the-loop system. With the further development of the systems based on human-computer interaction and the expansion of application scenarios, human-computer interaction will be applied on more occasions.除了上述场景外，“人在回路”系统还被用于其他领域 [127, 137]，如生物信息学、监督医疗和众包工作。例如，Fredrik Wrede 等人 [127] 利用“人在回路”半监督学习进行随机基因调控。 Rosenberg 等人 [121] 应用了一种用于众包任务的人工群体智能调度系统， Li 等人 [138] 采用了一种用于“人在回路”系统的模型检查方法。 随着基于人机交互的系统的进一步发展和应用场景的扩展，人机交互将在更多场合得到应用。
5 Discussion and Future Directions5 讨论与未来方向
In this section, we first discuss some existing challenges and key issues of human-in-the-loop for machine learning. Next, we show future directions on CV, NLP, and applications based on human-in-the-loop.在本节中，我们首先讨论“人在回路”机器学习现存的一些挑战和关键问题。 接下来，我们展示基于“人在回路”的计算机视觉（CV）、自然语言处理（NLP）和应用方面的未来方向。
5.1 The Challenges and Discussion5.1 挑战与讨论
How to add human experience and knowledge to computer vision tasks? By reviewing the previous works, we find that most human-in-the-loop researches only focus on natural language processing. Analyzing the reasons, it is not easy to directly allow people to interact with images effectively( except for direct labeling ), and add human experience and knowledge to the model throughout the cycle. With the development of multi-modal technology, utilizing multi-modality for image representation can be an effective way [139]. It is particularly worth pointing out that Holzinger et al. [140] adopted Graph Neural Networks as a method of choice, enabling information fusion for multi-modal capability, which is a milestone. Besides, applying inverse reinforcement learning also seems to be a feasible and practical solution [141].如何将人类经验和知识添加到计算机视觉任务中？ 通过回顾以往的研究，我们发现大多数“人在回路”研究仅关注自然语言处理。分析其原因，直接让人们与图像进行有效交互（直接标注除外），并在整个周期中将人类经验和知识添加到模型中并不容易。 随着多模态技术的发展，利用多模态进行图像表示可能是一种有效途径 [139]。 特别值得指出的是，Holzinger 等人 [140] 采用了图神经网络作为首选方法，实现了多模态能力的信息融合，这是一个里程碑。 此外，应用逆强化学习似乎也是一种可行且实用的解决方案 [141]。
How does the model learn human knowledge and experience from a higher dimension? The goal of human-in-the-loop is to connect humans to the model loop in a specific way, so that the machine can learn human knowledge and experience during the loop. Most current methods achieve this goal through human data annotation which is only the most basic realization process. As the saying goes, it is better to teach a man fishing than to give him fish. Researchers are supposed to take how to help agents acquire this knowledge effectively into account [142]. Language is an experience accumulated in the human learning process. Researchers currently focus on employing human intervention in dialogue to enable machines to learn human knowledge and intelligence from dialogue procedures iteratively [143]. In addition, numerous reasoning tasks contain higher-dimensional knowledge. By integrating humans into the reasoning loop, the machines can also learn more about human experience [118]. Image quality evaluation and design tasks are a higher level of human activity. Although human aesthetics and design inspiration can constitute the theory; however, more inspiration and aesthetics still come from human experience [144]. If we find a productive way to allow the model to learn more expert experience, the model’s improvement can be dramatic.模型如何从更高维度学习人类知识和经验？ “人在回路”的目标是以特定方式将人类连接到模型回路中，以便机器能够在回路期间学习人类的知识和经验。目前大多数方法通过人类数据标注来实现这一目标，这只是最基本的实现过程。俗话说，授人以鱼不如授人以渔。研究人员应该考虑如何帮助代理有效地获取这些知识 [142]。 语言是人类学习过程中积累的经验。研究人员目前专注于在对话中采用人工干预，使机器能够从对话过程中迭代地学习人类的知识和智能 [143]。 此外，许多推理任务包含更高维度的知识。通过将人类整合到推理回路中，机器也可以学习更多关于人类经验的知识 [118]。 图像质量评估和设计任务是更高层次的人类活动。尽管人类美学和设计灵感可以构成理论；然而，更多的灵感和美学仍然来自人类经验 [144]。如果我们找到一种富有成效的方法让模型学习更多的专家经验，模型的改进将是巨大的。
How to select key samples? The critical technology for the human-in-the-loop is obtaining essential samples and labeling them with human intervention. At present, researchers mainly exploit confidence-based methods to obtain critical samples. This method plays an irreplaceable role in classification tasks [145, 146]. However, for other tasks (e.g. semantic segmentation, regression, and target detection tasks), confidence is the evaluation that is not so noticeable and thus cannot reflect the improvement of the system. Active learning aims to train an accurate prediction model with the least cost by marking the examples that provide the most information. There are multiple mature and worthy reference methods in selecting criteria, and perhaps researchers can obtain inspiration from these methods [147].如何选择关键样本？ “人在回路”的关键技术是获取重要样本并进行人工干预标注。目前，研究人员主要利用基于置信度的方法来获取关键样本。该方法在分类任务中起着不可替代的作用 [145, 146]。 然而，对于其他任务（如语义分割、回归和目标检测任务），置信度是不那么明显的评估指标，因此无法反映系统的改进。 主动学习旨在通过标记提供最多信息的示例，以最小的成本训练准确的预测模型。在选择标准方面有多种成熟且值得参考的方法，也许研究人员可以从这些方法中获得灵感 [147]。
How to construct an evaluation benchmark? To develop the entire community, providing an effective test benchmark is important. At present, there is no uniform standard for human-in-the-loop research benchmarks. To better explore this research topic, it is essential to study how to develop evaluation methods and benchmarks for human-in-the-loop systems. Moreover, the formation of a unified benchmark is also conducive to the further refinement of research [30]. The current human-in-the-loop-based research is a more influential direction for exploring ways that are more conducive to human-in-the-loop. In addition to creating the standards for these interaction methods, restricting and theorization are also particularly important.如何构建评估基准？ 为了促进整个社区的发展，提供有效的测试基准非常重要。目前，“人在回路”研究基准尚无统一标准。为了更好地探索这一研究课题，研究如何开发“人在回路”系统的评估方法和基准至关重要。此外，形成统一的基准也有利于研究的进一步细化 [30]。当前基于“人在回路”的研究是探索更有利于“人在回路”方式的一个更具影响力的方向。除了为这些交互方式制定标准外，限制和理论化也特别重要。
How to implement a general multitasking framework by human-in-the-loop? The real-world task is complex and in its current form, so it is not easy to completely solve it with one characterization [148]. With the emergence of a unified large-scale pre-training model [149, 39], we have seen the hope of achieving a universal model through human-in-the-loop fine-tuning. In particular, the current machine learning models are not as intelligent as humans, so it may be the next direction to consider using a suitable way to introduce human knowledge into large models.如何通过“人在回路”实现通用的多任务框架？ 现实世界的任务是复杂的，且以当前的形式存在，因此很难完全用一种表征来解决它 [148]。 随着统一的大规模预训练模型的出现 [149, 39]，我们看到了通过“人在回路”微调实现通用模型的希望。特别是，当前的机器学习模型并不像人类那样智能，因此考虑使用合适的方式将人类知识引入大模型可能是下一个方向。
5.2 Future Directions5.2 未来方向
To facilitate more researchers developing more advanced Human-in-the-loop systems, we summarize the following concrete future directions for Human-in-the-loop NLP, Human-in-the-loop CV, and Human-in-the-loop Real-world Applications.为了促进更多研究人员开发更先进的“人在回路”系统，我们总结了以下针对“人在回路”NLP、“人在回路”CV 以及“人在回路”现实世界应用的具体未来方向。
5.2.1 Future Directions for Human-in-the-loop NLP Systems5.2.1 “人在回路”NLP 系统的未来方向
- •
For systems like a chatbot, automatic summarization tool, or commercial machine translation, when interacting with them, individuals can only give a reward signal to the one output that is sent to them, which leads to the sparsity in feedback concerning the size of the output space [150].对于聊天机器人、自动摘要工具或商业机器翻译等系统，在与它们交互时，个人只能对发送给他们的一个输出给予奖励信号，这导致了反馈相对于输出空间大小而言的稀疏性 [150]。
- •
In syntactic parsing tasks, exploring more intelligent questions about other types of parsing uncertainties based on human-in-the-loop and scaling the method to large unlabelled corpora or other languages are important [43].在句法分析任务中，基于“人在回路”探索关于其他类型分析不确定性的更智能问题，并将该方法扩展到大型未标注语料库或其他语言非常重要 [43]。
- •
In user-centered design and evaluation of a human-in-the-loop topic modeling task, the key is considering trust or confidence [151].在“人在回路”主题建模任务的以用户为中心的设计和评估中，关键在于考虑信任或置信度 [151]。
- •
In terms of AI safety, among existing human-in-the-loop techniques, some of them also allow malicious individuals to efficiently train models that serve their purpose, which may cause damage to all aspects of society. For example, they could exploit human feedback to fine-tune a language model to be more persuasive and manipulate humans’ beliefs, instill radical ideas, commit fraud, etc. [79].在 AI 安全方面，在现有的“人在回路”技术中，其中一些也允许恶意个人高效地训练服务于其目的的模型，这可能会对社会的各个方面造成损害。例如，他们可以利用人类反馈来微调语言模型，使其更具说服力并操纵人类的信仰、灌输激进思想、进行欺诈等 [79]。
5.2.2 Future Directions for Human-in-the-loop CV Systems5.2.2 “人在回路”CV 系统的未来方向
- •
In the image restoration task, it is necessary to pay attention to predictive parameter optimization based on supervised regression models and scientifically analyzing correlations between the parameters of different algorithms based on the human-in-the-loop methods [100].在图像修复任务中，有必要关注基于监督回归模型的预测参数优化，并基于“人在回路”方法科学地分析不同算法参数之间的相关性 [100]。
- •
In image enhancement tasks, using active learning to help users obtain a better estimation of cluster membership with the fewest image enhancements of images is important [152].在图像增强任务中，使用主动学习来帮助用户以最少的图像增强次数获得更好的聚类成员估计非常重要 [152]。
5.2.3 Future Directions both NLP and CV Systems5.2.3 NLP 和 CV 系统的共同未来方向
- •
Human supervision may be preferable due to various levels of expertise and with the increase of work overload, erroneous decisions are potential to occur [153].由于专业水平各异，且随着工作负荷的增加，人工监督可能更可取，但仍有可能出现错误的决策 [153]。
- •
Collect and share more human feedback datasets for different tasks of NLP and CV [30].为 NLP 和 CV 的不同任务收集并共享更多的人类反馈数据集 [30]。
- •
We should consider user credibility to affect the influence of their annotations by analyzing the quality of provided feedback[72].我们应通过分析所提供反馈的质量来考虑用户可信度，从而影响其标注的影响力 [72]。
- •
More rigorous in-depth user studies need to be designed and conducted to evaluate the effectiveness and robustness of human-in-the-loop frameworks in addition to model performance [151].除了模型性能外，还需要设计和开展更严格、深入的用户研究，以评估“人在回路”框架的有效性和鲁棒性 [151]。
- •
For generative tasks, an explicit function can be defined by user feedback to evaluate and collect the generated signals [150].对于生成任务，可以通过用户反馈定义一个显式函数来评估和收集生成的信号 [150]。
- •
It is vital to find an efficient way to dynamically pick up the most representative and valuable feedback to collect [154].找到一种有效的方式来动态挑选最具代表性和最有价值的反馈进行收集至关重要 [154]。
- •
It is crucial to have a more friendly manner of showing what the model has learned from feedback and what kind of feedback. Specifically, it is possible to explore the process of changing the model in an attempt to visualize the results of the manual feedback [155].以更友好的方式展示模型从反馈中学到了什么以及反馈的类型至关重要。具体而言，可以探索改变模型的过程，试图将人工反馈的结果可视化 [155]。
5.2.4 Future Directions in Real-world Applications5.2.4 现实世界应用的未来方向
- •
It is fundamental to choose an appropriate artificial intervention time, especially the tasks with solid demand for reliability and safety [156].选择适当的人工干预时间是基础，特别是对于可靠性和安全性要求极高的任务 [156]。
- •
- •
Modeling the sensor signal and solving the unified coding of abstract and concrete information is an essential problem in the process of human-computer interaction [159].对传感器信号进行建模并解决抽象和具体信息的统一编码是人机交互过程中的一个基本问题 [159]。
- •
Human intervention remains on superficial judgments (such as acceptance/rejection or direction), and exploring more complex feedback is also a critical issue in human-in-the-loop applications [160].人工干预仍停留在表面判断（如接受/拒绝或方向）上，探索更复杂的反馈也是“人在回路”应用中的一个关键问题 [160]。
- •
Human-in-the-loop-based systems should have high robustness and generalization ability to domain changes, disturbances, and “out-of-range” samples [161].基于“人在回路”的系统应具有针对领域变化、干扰和“范围外”样本的高鲁棒性和泛化能力 [161]。
6 Conclusion6 结论
In this paper, we review existing studies in human-in-the-loop techniques for machine learning.
We first discuss the work of improving model performance from data processing.
We divide this section into three parts: Data Preprocessing, Data Annotation, and Iterative Labeling according to the human-in-the-loop data processing pipeline. The core question is how to achieve more significant performance with fewer samples from a data perspective. Through the investigation of human-in-the-loop-based data processing methods, we find that the current human-in-the-loop-based data processing methods focus more on how to use semi-supervised methods for data collection and annotation than on how to select and identify key samples in the dataset.
Then, we discuss the work of improving model performance through interventional model training.
In this part, we divide into sub-module focusing on NLP and CV according to the target task. The core issue is how to promote the breakthrough of the model to crucial problems by adding human high-dimensional knowledge to the model. By investigating the work of human-in-the-loop in model training, we found that most human-in-the-loop exists in model training by participating in simple data increments. Exploring model training methods that effectively integrate human knowledge is the core of solving such problems. Methods such as inverse reinforcement learning and multimodal learning seem to be effective solutions.
Finally, we discuss the design of the system independent “human-in-the-loop”.
In this section, we provide an overview of several human-in-the-loop real-world applications tasks, and we divide the content into Security Systems, Code Production Tools, Simulation Systems, and Search Engines by task type. Through the investigation of human-in-the-loop applications, we find that the coordination of considerable variables in practical applications is more complicated than a single variable in academic research. In actual production, although the system integrating human-in-the-loop has advantages, how to quickly integrate high-level human knowledge in actual production and maintain the robustness of the system is a problem worthy of further consideration.
Besides, we provide open challenges and opportunities and introduce some exciting questions.
We first discuss some challenges and problems of the current human-in-the-loop in machine learning. On this basis, we further clarify the future directions in terms of CV, NLP, and applications. The establishment of artificial intelligence models that can effectively integrate human high-dimensional knowledge has development potential, especially with the research of large-scale pre-training models, the human-machine hybrid method based on few-shot has broad prospects.
在本文中，我们回顾了机器学习中“人在回路”技术的现有研究。
我们首先讨论了从数据处理方面提升模型性能的工作。
根据“人在回路”数据处理流水线，我们将本节分为数据预处理、数据标注和迭代标注三部分。核心问题是从数据角度出发，如何用更少的样本实现更显著的性能。通过对基于“人在回路”的数据处理方法的调查，我们发现当前基于“人在回路”的数据处理方法更多地关注如何使用半监督方法进行数据收集和标注，而不是如何选择和识别数据集中的关键样本。
然后，我们讨论了通过干预式模型训练来提升模型性能的工作。
在这一部分，我们根据目标任务将其划分为关注 NLP 和 CV 的子模块。核心问题是如何通过向模型添加人类高维知识来促进模型在关键问题上的突破。通过调查“人在回路”在模型训练中的工作，我们发现大多数“人在回路”存在于通过参与简单数据增量来辅助模型训练的过程中。探索有效整合人类知识的模型训练方法是解决此类问题的核心。逆强化学习和多模态学习等方法似乎是有效的解决方案。
最后，我们讨论了系统独立的“人在回路”设计。
在本节中，我们概述了几个“人在回路”的现实应用任务，并按任务类型将内容划分为安全系统、代码生产工具、仿真系统和搜索引擎。通过对“人在回路”应用的调查，我们发现实际应用中大量变量的协调比学术研究中的单一变量更复杂。在实际生产中，尽管整合了“人在回路”的系统具有优势，但如何在实际生产中快速整合高级人类知识并保持系统的鲁棒性是一个值得进一步考虑的问题。
此外，我们提供了开放的挑战和机遇，并介绍了一些令人兴奋的问题。
我们首先讨论了当前机器学习中“人在回路”的一些挑战和问题。在此基础上，我们进一步明确了在 CV、NLP 和应用方面的未来方向。建立能够有效整合人类高维知识的人工智能模型具有发展潜力，特别是在大规模预训练模型的研究下，基于小样本的人机混合方法具有广阔的前景。
Acknowledgment. This work was supported in part by the 2020 East China Normal University Outstanding Doctoral Students Academic Innovation Ability Improvement Project (YBNLTS2020-042), the Science and Technology Commission of Shanghai Municipality (19511120200).致谢。 本工作得到 2020 年华东师范大学优秀博士生学术创新能力提升项目（YBNLTS2020-042）和上海市科学技术委员会（19511120200）的部分资助。
References参考文献
- [1] S. Dong, P. Wang, and K. Abbas, “A survey on deep learning and its applications,” Computer Science Review, vol. 40, p. 100379, 2021. [1] S. Dong, P. Wang, and K. Abbas, “A survey on deep learning and its applications,” Computer Science Review, vol. 40, p. 100379, 2021.
- [2] A. Brutzkus and A. Globerson, “Why do larger models generalize better? a theoretical perspective via the xor problem,” in Proceedings of International Conference on Machine Learning (ICML), 2019, pp. 822–830. [2] A. Brutzkus and A. Globerson, “Why do larger models generalize better? a theoretical perspective via the xor problem,” in Proceedings of International Conference on Machine Learning (ICML), 2019, pp. 822–830.
- [3] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” in Advances in Neural Information Processing Systems (NIPS), 2014, pp. 487–495. [3] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” in Advances in Neural Information Processing Systems (NIPS), 2014, pp. 487–495.
- [4] J. Li, J. Yang, A. Hertzmann, J. Zhang, and T. Xu, “LayoutGAN: Synthesizing graphic layouts with vector-wireframe adversarial networks,” IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2388–2399, 2021. [4] J. Li, J. Yang, A. Hertzmann, J. Zhang, and T. Xu, “LayoutGAN: Synthesizing graphic layouts with vector-wireframe adversarial networks,” IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2388–2399, 2021.
- [5] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han, “Differentiable augmentation for data-efficient gan training,” Advances in Neural Information Processing Systems (NIPS), vol. 33, pp. 7559–7570, 2020. [5] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han, “Differentiable augmentation for data-efficient gan training,” Advances in Neural Information Processing Systems (NIPS), vol. 33, pp. 7559–7570, 2020.
- [6] H. T. Shen, X. Zhu, Z. Zhang, S.-H. Wang, Y. Chen, X. Xu, and J. Shao, “Heterogeneous data fusion for predicting mild cognitive impairment conversion,” Information Fusion, vol. 66, pp. 54–63, 2021. [6] H. T. Shen, X. Zhu, Z. Zhang, S.-H. Wang, Y. Chen, X. Xu, and J. Shao, “Heterogeneous data fusion for predicting mild cognitive impairment conversion,” Information Fusion, vol. 66, pp. 54–63, 2021.
- [7] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” Science China Technological Sciences, pp. 1–26, 2020. [7] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” Science China Technological Sciences, pp. 1–26, 2020.
- [8] M. Zaib, Q. Z. Sheng, and W. Emma Zhang, “A short survey of pre-trained language models for conversational ai-a new age in nlp,” in ACSW, 2020, pp. 1–4. [8] M. Zaib, Q. Z. Sheng, and W. Emma Zhang, “A short survey of pre-trained language models for conversational ai-a new age in nlp,” in ACSW, 2020, pp. 1–4.
- [9] S. Bahrami, F. Dornaika, and A. Bosaghzadeh, “Joint auto-weighted graph fusion and scalable semi-supervised learning,” Information Fusion, vol. 66, pp. 213–228, 2021. [9] S. Bahrami, F. Dornaika, and A. Bosaghzadeh, “Joint auto-weighted graph fusion and scalable semi-supervised learning,” Information Fusion, vol. 66, pp. 213–228, 2021.
- [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, 2017. [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, 2017.
- [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in NAACL, 2019, pp. 4171–4186. [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in NAACL, 2019, pp. 4171–4186.
- [12] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018. [12] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018.
- [13] M. Habermann, W. Xu, M. Zollhofer, G. Pons-Moll, and C. Theobalt, “Deepcap: Monocular human performance capture using weak supervision,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5052–5063. [13] M. Habermann, W. Xu, M. Zollhofer, G. Pons-Moll, and C. Theobalt, “Deepcap: Monocular human performance capture using weak supervision,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5052–5063.
- [14] Y. Wang, W. Yang, F. Ma, J. Xu, B. Zhong, Q. Deng, and J. Gao, “Weak supervision for fake news detection via reinforcement learning,” in AAAI, vol. 34, no. 01, 2020, pp. 516–523. [14] Y. Wang, W. Yang, F. Ma, J. Xu, B. Zhong, Q. Deng, and J. Gao, “Weak supervision for fake news detection via reinforcement learning,” in AAAI, vol. 34, no. 01, 2020, pp. 516–523.
- [15] S. Jia, S. Jiang, Z. Lin, N. Li, M. Xu, and S. Yu, “A survey: Deep learning for hyperspectral image classification with few labeled samples,” Neurocomputing, vol. 448, pp. 179–204, 2021. [15] S. Jia, S. Jiang, Z. Lin, N. Li, M. Xu, and S. Yu, “A survey: Deep learning for hyperspectral image classification with few labeled samples,” Neurocomputing, vol. 448, pp. 179–204, 2021.
- [16] M. Diligenti, S. Roychowdhury, and M. Gori, “Integrating prior knowledge into deep learning,” in ICMLA, 2017, pp. 920–923. [16] M. Diligenti, S. Roychowdhury, and M. Gori, “Integrating prior knowledge into deep learning,” in ICMLA, 2017, pp. 920–923.
- [17] S. Chen, Y. Leng, and S. Labi, “A deep learning algorithm for simulating autonomous driving considering prior knowledge and temporal information,” Computer-Aided Civil and Infrastructure Engineering, vol. 35, no. 4, pp. 305–321, 2020. [17] S. Chen, Y. Leng, and S. Labi, “A deep learning algorithm for simulating autonomous driving considering prior knowledge and temporal information,” Computer-Aided Civil and Infrastructure Engineering, vol. 35, no. 4, pp. 305–321, 2020.
- [18] Y. Lin, S. L. Pintea, and J. C. van Gemert, “Deep hough-transform line priors,” in European Conference on Computer Vision (ECCV), 2020, pp. 323–340. [18] Y. Lin, S. L. Pintea, and J. C. van Gemert, “Deep hough-transform line priors,” in European Conference on Computer Vision (ECCV), 2020, pp. 323–340.
- [19] G. Hartmann, Z. Shiller, and A. Azaria, “Deep reinforcement learning for time optimal velocity control using prior knowledge,” in ICTAI, 2019, pp. 186–193. [19] G. Hartmann, Z. Shiller, and A. Azaria, “Deep reinforcement learning for time optimal velocity control using prior knowledge,” in ICTAI, 2019, pp. 186–193.
- [20] X. Zhang, S. Wang, J. Liu, and C. Tao, “Towards improving diagnosis of skin diseases by combining deep neural network and human knowledge,” BMC medical informatics and decision making, vol. 18, no. 2, pp. 69–76, 2018. [20] X. Zhang, S. Wang, J. Liu, and C. Tao, “Towards improving diagnosis of skin diseases by combining deep neural network and human knowledge,” BMC medical informatics and decision making, vol. 18, no. 2, pp. 69–76, 2018.
- [21] R. Zhang, F. Torabi, L. Guan, D. H. Ballard, and P. Stone, “Leveraging human guidance for deep reinforcement learning tasks,” in International Joint Conference on Artificial Intelligence (IJCAI), 2019. [21] R. Zhang, F. Torabi, L. Guan, D. H. Ballard, and P. Stone, “Leveraging human guidance for deep reinforcement learning tasks,” in International Joint Conference on Artificial Intelligence (IJCAI), 2019.
- [22] A. Holzinger, M. Plass, M. Kickmeier-Rust, K. Holzinger, G. C. Crişan, C.-M. Pintea, and V. Palade, “Interactive machine learning: experimental evidence for the human in the algorithmic loop,” Applied Intelligence, vol. 49, no. 7, pp. 2401–2414, 2019. [22] A. Holzinger, M. Plass, M. Kickmeier-Rust, K. Holzinger, G. C. Crişan, C.-M. Pintea, and V. Palade, “Interactive machine learning: experimental evidence for the human in the algorithmic loop,” Applied Intelligence, vol. 49, no. 7, pp. 2401–2414, 2019.
- [23] Y.-t. Zhuang, F. Wu, C. Chen, and Y.-h. Pan, “Challenges and opportunities: from big data to knowledge in ai 2.0,” Frontiers of Information Technology & Electronic Engineering, vol. 18, no. 1, pp. 3–14, 2017. [23] Y.-t. Zhuang, F. Wu, C. Chen, and Y.-h. Pan, “Challenges and opportunities: from big data to knowledge in ai 2.0,” Frontiers of Information Technology & Electronic Engineering, vol. 18, no. 1, pp. 3–14, 2017.
- [24] V. Kumar, A. Smith-Renner, L. Findlater, K. Seppi, and J. Boyd-Graber, “Why didn’t you listen to me? comparing user control of human-in-the-loop topic models,” in ACL, 2019. [24] V. Kumar, A. Smith-Renner, L. Findlater, K. Seppi, and J. Boyd-Graber, “Why didn’t you listen to me? comparing user control of human-in-the-loop topic models,” in ACL, 2019.
- [25] D. Xin, L. Ma, J. Liu, S. Macke, S. Song, and A. Parameswaran, “Accelerating human-in-the-loop machine learning: Challenges and opportunities,” in Proceedings of the second workshop on data management for end-to-end machine learning, 2018, pp. 1–4. [25] D. Xin, L. Ma, J. Liu, S. Macke, S. Song, and A. Parameswaran, “Accelerating human-in-the-loop machine learning: Challenges and opportunities,” in Proceedings of the second workshop on data management for end-to-end machine learning, 2018, pp. 1–4.
- [26] S. Budd, E. C. Robinson, and B. Kainz, “A survey on active learning and human-in-the-loop deep learning for medical image analysis,” Medical Image Analysis, vol. 71, p. 102062, 2021. [26] S. Budd, E. C. Robinson, and B. Kainz, “A survey on active learning and human-in-the-loop deep learning for medical image analysis,” Medical Image Analysis, vol. 71, p. 102062, 2021.
- [27] W. Jung and F. Jazizadeh, “Human-in-the-loop hvac operations: A quantitative review on occupancy, comfort, and energy-efficiency dimensions,” Applied Energy, vol. 239, pp. 1471–1508, 2019. [27] W. Jung and F. Jazizadeh, “Human-in-the-loop hvac operations: A quantitative review on occupancy, comfort, and energy-efficiency dimensions,” Applied Energy, vol. 239, pp. 1471–1508, 2019.
- [28] S. Agnisarman, S. Lopes, K. C. Madathil, K. Piratla, and A. Gramopadhye, “A survey of automation-enabled human-in-the-loop systems for infrastructure visual inspection,” Automation in Construction, vol. 97, pp. 52–76, 2019. [28] S. Agnisarman, S. Lopes, K. C. Madathil, K. Piratla, and A. Gramopadhye, “A survey of automation-enabled human-in-the-loop systems for infrastructure visual inspection,” Automation in Construction, vol. 97, pp. 52–76, 2019.
- [29] L. Benedikt, C. Joshi, L. Nolan, R. Henstra-Hill, L. Shaw, and S. Hook, “Human-in-the-loop ai in government: a case study,” in IUI, 2020, pp. 488–497. [29] L. Benedikt, C. Joshi, L. Nolan, R. Henstra-Hill, L. Shaw, and S. Hook, “Human-in-the-loop ai in government: a case study,” in IUI, 2020, pp. 488–497.
- [30] C. Chai and G. Li, “Human-in-the-loop techniques in machine learning,” Data Engineering, p. 37, 2020. [30] C. Chai and G. Li, “Human-in-the-loop techniques in machine learning,” Data Engineering, p. 37, 2020.
- [31] B. M. Tehrani, J. Wang, and C. Wang, “Review of human-in-the-loop cyber-physical systems (hilcps): The current status from human perspective,” Computing in Civil Engineering 2019: Data, Sensing, and Analytics, pp. 470–478, 2019. [31] B. M. Tehrani, J. Wang, and C. Wang, “Review of human-in-the-loop cyber-physical systems (hilcps): The current status from human perspective,” Computing in Civil Engineering 2019: Data, Sensing, and Analytics, pp. 470–478, 2019.
- [32] Z. Y. Khan, Z. Niu, S. Sandiwarno, and R. Prince, “Deep learning techniques for rating prediction: a survey of the state-of-the-art,” Artificial Intelligence Review, vol. 54, no. 1, pp. 95–135, 2021. [32] Z. Y. Khan, Z. Niu, S. Sandiwarno, and R. Prince, “Deep learning techniques for rating prediction: a survey of the state-of-the-art,” Artificial Intelligence Review, vol. 54, no. 1, pp. 95–135, 2021.
- [33] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” 2015. [33] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” 2015.
- [34] O. Siméoni, M. Budnik, Y. Avrithis, and G. Gravier, “Rethinking deep active learning: Using unlabeled data at model training,” in Proceedings of the International Conference on Pattern Recognition (ICPR), 2021, pp. 1220–1227. [34] O. Siméoni, M. Budnik, Y. Avrithis, and G. Gravier, “Rethinking deep active learning: Using unlabeled data at model training,” in Proceedings of the International Conference on Pattern Recognition (ICPR), 2021, pp. 1220–1227.
- [35] Y. Wang, L. Zhang, Y. Yao, and Y. Fu, “How to trust unlabeled data instance credibility inference for few-shot learning,” IEEE Transaction on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021. [35] Y. Wang, L. Zhang, Y. Yao, and Y. Fu, “How to trust unlabeled data instance credibility inference for few-shot learning,” IEEE Transaction on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021.
- [36] Y. Shi and A. K. Jain, “Boosting unconstrained face recognition with auxiliary unlabeled data,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2795–2804. [36] Y. Shi and A. K. Jain, “Boosting unconstrained face recognition with auxiliary unlabeled data,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2795–2804.
- [37] Z. Ren, R. Yeh, and A. Schwing, “Not all unlabeled data are equal: Learning to weight data in semi-supervised learning,” Advances in Neural Information Processing Systems (NIPS), vol. 33, 2020. [37] Z. Ren, R. Yeh, and A. Schwing, “Not all unlabeled data are equal: Learning to weight data in semi-supervised learning,” Advances in Neural Information Processing Systems (NIPS), vol. 33, 2020.
- [38] S. Niu, B. Li, X. Wang, and H. Lin, “Defect image sample generation with gan for improving defect recognition,” IEEE Transactions on Automation Science and Engineering, vol. 17, no. 3, pp. 1611–1622, 2020. [38] S. Niu, B. Li, X. Wang, and H. Lin, “Defect image sample generation with gan for improving defect recognition,” IEEE Transactions on Automation Science and Engineering, vol. 17, no. 3, pp. 1611–1622, 2020.
- [39] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys (CSUR), 2021. [39] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” ACM Computing Surveys (CSUR), 2021.
- [40] T. D. Pham, “Classification of covid-19 chest x-rays with deep learning: new models or fine tuning?” Health Information Science and Systems, vol. 9, no. 1, pp. 1–11, 2021. [40] T. D. Pham, “Classification of covid-19 chest x-rays with deep learning: new models or fine tuning?” Health Information Science and Systems, vol. 9, no. 1, pp. 1–11, 2021.
- [41] S. Chen, Y. Hou, Y. Cui, W. Che, T. Liu, and X. Yu, “Recall and learn: Fine-tuning deep pretrained language models with less forgetting,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 7870–7881. [41] S. Chen, Y. Hou, Y. Cui, W. Che, T. Liu, and X. Yu, “Recall and learn: Fine-tuning deep pretrained language models with less forgetting,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 7870–7881.
- [42] G. Wang, W. Li, M. A. Zuluaga, R. Pratt, P. A. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest, S. Ourselin et al., “Interactive medical image segmentation using deep learning with image-specific fine tuning,” IEEE transactions on medical imaging, vol. 37, no. 7, pp. 1562–1573, 2018. [42] G. Wang, W. Li, M. A. Zuluaga, R. Pratt, P. A. Patel, M. Aertsen, T. Doel, A. L. David, J. Deprest, S. Ourselin et al., “Interactive medical image segmentation using deep learning with image-specific fine tuning,” IEEE transactions on medical imaging, vol. 37, no. 7, pp. 1562–1573, 2018.
- [43] L. He, J. Michael, M. Lewis, and L. Zettlemoyer, “Human-in-the-loop parsing,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016, pp. 2337–2342. [43] L. He, J. Michael, M. Lewis, and L. Zettlemoyer, “Human-in-the-loop parsing,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016, pp. 2337–2342.
- [44] J. Z. Self, R. K. Vinayagam, J. Fry, and C. North, “Bridging the gap between user intention and model parameters for human-in-the-loop data analytics,” in Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 2016, pp. 1–6. [44] J. Z. Self, R. K. Vinayagam, J. Fry, and C. North, “Bridging the gap between user intention and model parameters for human-in-the-loop data analytics,” in Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 2016, pp. 1–6.
- [45] Y. Zhuang, G. Li, Z. Zhong, and J. Feng, “Hike: A hybrid human-machine method for entity alignment in large-scale knowledge bases,” in CIKM, 2017, pp. 1917–1926. [45] Y. Zhuang, G. Li, Z. Zhong, and J. Feng, “Hike: A hybrid human-machine method for entity alignment in large-scale knowledge bases,” in CIKM, 2017, pp. 1917–1926.
- [46] G. Li, “Human-in-the-loop data integration,” Proceedings of the VLDB Endowment, vol. 10, no. 12, pp. 2006–2017, 2017. [46] G. Li, “Human-in-the-loop data integration,” Proceedings of the VLDB Endowment, vol. 10, no. 12, pp. 2006–2017, 2017.
- [47] B. Kim and B. Pardo, “A human-in-the-loop system for sound event detection and annotation,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 8, no. 2, pp. 1–23, 2018. [47] B. Kim and B. Pardo, “A human-in-the-loop system for sound event detection and annotation,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 8, no. 2, pp. 1–23, 2018.
- [48] A. Doan, “Human-in-the-loop data analysis: a personal perspective,” in Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 2018, pp. 1–6. [48] A. Doan, “Human-in-the-loop data analysis: a personal perspective,” in Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 2018, pp. 1–6.
- [49] X. L. Dong and T. Rekatsinas, “Data integration and machine learning: A natural synergy,” in COMAD, 2018, pp. 1645–1650. [49] X. L. Dong and T. Rekatsinas, “Data integration and machine learning: A natural synergy,” in COMAD, 2018, pp. 1645–1650.
- [50] A. L. Gentile, D. Gruhl, P. Ristoski, and S. Welch, “Explore and exploit. dictionary expansion with human-in-the-loop,” in European Semantic Web Conference, 2019, pp. 131–145. [50] A. L. Gentile, D. Gruhl, P. Ristoski, and S. Welch, “Explore and exploit. dictionary expansion with human-in-the-loop,” in European Semantic Web Conference, 2019, pp. 131–145.
- [51] S. Zhang, L. He, E. Dragut, and S. Vucetic, “How to invest my time: Lessons from human-in-the-loop entity extraction,” in KDD, 2019, pp. 2305–2313. [51] S. Zhang, L. He, E. Dragut, and S. Vucetic, “How to invest my time: Lessons from human-in-the-loop entity extraction,” in KDD, 2019, pp. 2305–2313.
- [52] L. Berti-Equille, “Reinforcement learning for data preparation with active reward learning,” in International Conference on Internet Science, 2019, pp. 121–132. [52] L. Berti-Equille, “Reinforcement learning for data preparation with active reward learning,” in International Conference on Internet Science, 2019, pp. 121–132.
- [53] S. Gurajada, L. Popa, K. Qian, and P. Sen, “Learning-based methods with human-in-the-loop for entity resolution,” in CIKM, 2019, pp. 2969–2970. [53] S. Gurajada, L. Popa, K. Qian, and P. Sen, “Learning-based methods with human-in-the-loop for entity resolution,” in CIKM, 2019, pp. 2969–2970.
- [54] Y. Lou, M. Uddin, N. Brown, and M. Cafarella, “Knowledge graph programming with a human-in-the-loop: Preliminary results,” in Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 2019, pp. 1–7. [54] Y. Lou, M. Uddin, N. Brown, and M. Cafarella, “Knowledge graph programming with a human-in-the-loop: Preliminary results,” in Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 2019, pp. 1–7.
- [55] Z. Liu, J. Wang, S. Gong, H. Lu, and D. Tao, “Deep reinforcement active learning for human-in-the-loop person re-identification,” in Proceedings of the IEEE international conference on computer vision (ICCV), 2019, pp. 6122–6131. [55] Z. Liu, J. Wang, S. Gong, H. Lu, and D. Tao, “Deep reinforcement active learning for human-in-the-loop person re-identification,” in Proceedings of the IEEE international conference on computer vision (ICCV), 2019, pp. 6122–6131.
- [56] E. Wallace, P. Rodriguez, S. Feng, I. Yamada, and J. Boyd-Graber, “Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 387–401, 2019. [56] E. Wallace, P. Rodriguez, S. Feng, I. Yamada, and J. Boyd-Graber, “Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 387–401, 2019.
- [57] X. Fan, C. Li, X. Yuan, X. Dong, and J. Liang, “An interactive visual analytics approach for network anomaly detection through smart labeling,” Journal of Visualization, vol. 22, no. 5, pp. 955–971, 2019. [57] X. Fan, C. Li, X. Yuan, X. Dong, and J. Liang, “An interactive visual analytics approach for network anomaly detection through smart labeling,” Journal of Visualization, vol. 22, no. 5, pp. 955–971, 2019.
- [58] E. Krokos, H.-C. Cheng, J. Chang, B. Nebesh, C. L. Paul, K. Whitley, and A. Varshney, “Enhancing deep learning with visual interactions,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 9, no. 1, pp. 1–27, 2019. [58] E. Krokos, H.-C. Cheng, J. Chang, B. Nebesh, C. L. Paul, K. Whitley, and A. Varshney, “Enhancing deep learning with visual interactions,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 9, no. 1, pp. 1–27, 2019.
- [59] J.-C. Klie, R. E. de Castilho, and I. Gurevych, “From zero to hero: Human-in-the-loop entity linking in low resource domains,” in ACL, 2020, pp. 6982–6993. [59] J.-C. Klie, R. E. de Castilho, and I. Gurevych, “From zero to hero: Human-in-the-loop entity linking in low resource domains,” in ACL, 2020, pp. 6982–6993.
- [60] C. Butler, H. Oster, and J. Togelius, “Human-in-the-loop ai for analysis of free response facial expression label sets,” in IVA, 2020, pp. 1–8. [60] C. Butler, H. Oster, and J. Togelius, “Human-in-the-loop ai for analysis of free response facial expression label sets,” in IVA, 2020, pp. 1–8.
- [61] P. Ristoski, A. L. Gentile, A. Alba, D. Gruhl, and S. Welch, “Large-scale relation extraction from web documents and knowledge graphs with human-in-the-loop,” Journal of Web Semantics, vol. 60, p. 100546, 2020. [61] P. Ristoski, A. L. Gentile, A. Alba, D. Gruhl, and S. Welch, “Large-scale relation extraction from web documents and knowledge graphs with human-in-the-loop,” Journal of Web Semantics, vol. 60, p. 100546, 2020.
- [62] K. Qian, P. C. Raman, Y. Li, and L. Popa, “Partner: Human-in-the-loop entity name understanding with deep learning,” in The AAAI Conference on Artificial Intelligence, vol. 34, no. 09, 2020, pp. 13 634–13 635. [62] K. Qian, P. C. Raman, Y. Li, and L. Popa, “Partner: Human-in-the-loop entity name understanding with deep learning,” in The AAAI Conference on Artificial Intelligence, vol. 34, no. 09, 2020, pp. 13 634–13 635.
- [63] T.-N. Le, A. Sugimoto, S. Ono, and H. Kawasaki, “Toward interactive self-annotation for video object bounding box: Recurrent self-learning and hierarchical annotation based framework,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 3231–3240. [63] T.-N. Le, A. Sugimoto, S. Ono, and H. Kawasaki, “Toward interactive self-annotation for video object bounding box: Recurrent self-learning and hierarchical annotation based framework,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 3231–3240.
- [64] M. Bartolo, A. Roberts, J. Welbl, S. Riedel, and P. Stenetorp, “Beat the AI: Investigating adversarial human annotation for reading comprehension,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 662–678, 2020. [64] M. Bartolo, A. Roberts, J. Welbl, S. Riedel, and P. Stenetorp, “Beat the AI: Investigating adversarial human annotation for reading comprehension,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 662–678, 2020.
- [65] K. Muthuraman, F. Reiss, H. Xu, B. Cutler, and Z. Eichenberger, “Data cleaning tools for token classification tasks,” in Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances, 2021, pp. 59–61. [65] K. Muthuraman, F. Reiss, H. Xu, B. Cutler, and Z. Eichenberger, “Data cleaning tools for token classification tasks,” in Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances, 2021, pp. 59–61.
- [66] Q. Meng, W. Wang, T. Zhou, J. Shen, Y. Jia, and L. Van Gool, “Towards a weakly supervised framework for 3d point cloud object detection and annotation,” IEEE Transaction on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021. [66] Q. Meng, W. Wang, T. Zhou, J. Shen, Y. Jia, and L. Van Gool, “Towards a weakly supervised framework for 3d point cloud object detection and annotation,” IEEE Transaction on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021.
- [67] L. Zhang, X. Wang, Q. Fan, Y. Ji, and C. Liu, “Generating manga from illustrations via mimicking manga creation workflow,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 5642–5651. [67] L. Zhang, X. Wang, Q. Fan, Y. Ji, and C. Liu, “Generating manga from illustrations via mimicking manga creation workflow,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 5642–5651.
- [68] B. Adhikari and H. Huttunen, “Iterative bounding box annotation for object detection,” in Proceedings of the International Conference on Pattern Recognition (ICPR), 2021, pp. 4040–4046. [68] B. Adhikari and H. Huttunen, “Iterative bounding box annotation for object detection,” in Proceedings of the International Conference on Pattern Recognition (ICPR), 2021, pp. 4040–4046.
- [69] J. L. Martinez-Rodriguez, A. Hogan, and I. Lopez-Arevalo, “Information extraction meets the semantic web: a survey,” Semantic Web, vol. 11, no. 2, pp. 255–335, 2020. [69] J. L. Martinez-Rodriguez, A. Hogan, and I. Lopez-Arevalo, “Information extraction meets the semantic web: a survey,” Semantic Web, vol. 11, no. 2, pp. 255–335, 2020.
- [70] H. Ye, W. Shao, H. Wang, J. Ma, L. Wang, Y. Zheng, and X. Xue, “Face recognition via active annotation and learning,” in ACM International Conference on Multimedia, 2016, pp. 1058–1062. [70] H. Ye, W. Shao, H. Wang, J. Ma, L. Wang, Y. Zheng, and X. Xue, “Face recognition via active annotation and learning,” in ACM International Conference on Multimedia, 2016, pp. 1058–1062.
- [71] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015. [71] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015.
- [72] T. Karmakharm, N. Aletras, and K. Bontcheva, “Journalist-in-the-loop: Continuous learning as a service for rumour analysis,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019, pp. 115–120. [72] T. Karmakharm, N. Aletras, and K. Bontcheva, “Journalist-in-the-loop: Continuous learning as a service for rumour analysis,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019, pp. 115–120.
- [73] Y. Song, J. Wang, T. Jiang, Z. Liu, and Y. Rao, “Targeted sentiment classification with attentional encoder network,” in ICANN. Springer, 2019, pp. 93–103. [73] Y. Song, J. Wang, T. Jiang, Z. Liu, and Y. Rao, “Targeted sentiment classification with attentional encoder network,” in ICANN. Springer, 2019, pp. 93–103.
- [74] X. Bai, P. Liu, and Y. Zhang, “Investigating typed syntactic dependencies for targeted sentiment classification using graph attention neural network,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 503–514, 2020. [74] X. Bai, P. Liu, and Y. Zhang, “Investigating typed syntactic dependencies for targeted sentiment classification using graph attention neural network,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 503–514, 2020.
- [75] I. Arous, L. Dolamic, J. Yang, A. Bhardwaj, G. Cuccu, and P. Cudré-Mauroux, “Marta: Leveraging human rationales for explainable text classification,” in The AAAI Conference on Artificial Intelligence, vol. 35, no. 7, 2021, pp. 5868–5876. [75] I. Arous, L. Dolamic, J. Yang, A. Bhardwaj, G. Cuccu, and P. Cudré-Mauroux, “Marta: Leveraging human rationales for explainable text classification,” in The AAAI Conference on Artificial Intelligence, vol. 35, no. 7, 2021, pp. 5868–5876.
- [76] Z. Yao, X. Li, J. Gao, B. Sadler, and H. Sun, “Interactive semantic parsing for if-then recipes via hierarchical reinforcement learning,” in The AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 2547–2554. [76] Z. Yao, X. Li, J. Gao, B. Sadler, and H. Sun, “Interactive semantic parsing for if-then recipes via hierarchical reinforcement learning,” in The AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 2547–2554.
- [77] Z. Yao, Y. Su, H. Sun, and W.-t. Yih, “Model-based interactive semantic parsing: A unified formulation and a text-to-sql case study,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. [77] Z. Yao, Y. Su, H. Sun, and W.-t. Yih, “Model-based interactive semantic parsing: A unified formulation and a text-to-sql case study,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
- [78] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” 2019. [78] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” 2019.
- [79] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” Advances in Neural Information Processing Systems (NIPS), vol. 33, pp. 3008–3021, 2020. [79] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, “Learning to summarize with human feedback,” Advances in Neural Information Processing Systems (NIPS), vol. 33, pp. 3008–3021, 2020.
- [80] B. Hancock, A. Bordes, P.-E. Mazare, and J. Weston, “Learning from dialogue after deployment: Feed yourself, chatbot!” in ACL, 2019, pp. 3667–3684. [80] B. Hancock, A. Bordes, P.-E. Mazare, and J. Weston, “Learning from dialogue after deployment: Feed yourself, chatbot!” in ACL, 2019, pp. 3667–3684.
- [81] Z. Liu, Y. Guo, A. A. AI, and J. Mahmud, “When and why does a model fail? a human-in-the-loop error detection framework for sentiment analysis,” NAACL-HLT 2021, p. 170, 2021. [81] Z. Liu, Y. Guo, A. A. AI, and J. Mahmud, “When and why does a model fail? a human-in-the-loop error detection framework for sentiment analysis,” NAACL-HLT 2021, p. 170, 2021.
- [82] S. Chopra, M. Auli, and A. M. Rush, “Abstractive sentence summarization with attentive recurrent neural networks,” in NAACL, 2016, pp. 93–98. [82] S. Chopra, M. Auli, and A. M. Rush, “Abstractive sentence summarization with attentive recurrent neural networks,” in NAACL, 2016, pp. 93–98.
- [83] Z. J. Wang, D. Choi, S. Xu, and D. Yang, “Putting humans in the natural language processing loop: A survey,” in Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, 2021, pp. 47–52. [83] Z. J. Wang, D. Choi, S. Xu, and D. Yang, “Putting humans in the natural language processing loop: A survey,” in Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, 2021, pp. 47–52.
- [84] L. Xiao, X. Hu, Y. Chen, Y. Xue, D. Gu, B. Chen, and T. Zhang, “Targeted sentiment classification based on attentional encoding and graph convolutional networks,” Applied Sciences, vol. 10, no. 3, p. 957, 2020. [84] L. Xiao, X. Hu, Y. Chen, Y. Xue, D. Gu, B. Chen, and T. Zhang, “Targeted sentiment classification based on attentional encoding and graph convolutional networks,” Applied Sciences, vol. 10, no. 3, p. 957, 2020.
- [85] L. Xiao, X. Hu, Y. Chen, Y. Xue, B. Chen, D. Gu, and B. Tang, “Multi-head self-attention based gated graph convolutional networks for aspect-based sentiment classification,” Multimedia Tools and Applications, pp. 1–20, 2020. [85] L. Xiao, X. Hu, Y. Chen, Y. Xue, B. Chen, D. Gu, and B. Tang, “Multi-head self-attention based gated graph convolutional networks for aspect-based sentiment classification,” Multimedia Tools and Applications, pp. 1–20, 2020.
- [86] B. Nushi, E. Kamar, and E. Horvitz, “Towards accountable ai: Hybrid human-machine analyses for characterizing system failure,” in The AAAI Conference on Artificial Intelligence, vol. 6, no. 1, 2018. [86] B. Nushi, E. Kamar, and E. Horvitz, “Towards accountable ai: Hybrid human-machine analyses for characterizing system failure,” in The AAAI Conference on Artificial Intelligence, vol. 6, no. 1, 2018.
- [87] M. T. Ribeiro, S. Singh, and C. Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in Annual ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2016, pp. 1135–1144. [87] M. T. Ribeiro, S. Singh, and C. Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in Annual ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2016, pp. 1135–1144.
- [88] X. Wu, Y. Zheng, T. Ma, H. Ye, and L. He, “Document image layout analysis via explicit edge embedding network,” Information Sciences, vol. 577, pp. 436–448, 2021. [88] X. Wu, Y. Zheng, T. Ma, H. Ye, and L. He, “Document image layout analysis via explicit edge embedding network,” Information Sciences, vol. 577, pp. 436–448, 2021.
- [89] X. Wu, B. Xu, Y. Zheng, H. Ye, J. Yang, and L. He, “Fast video crowd counting with a temporal aware network,” Neurocomputing, vol. 403, pp. 13–20, 2020. [89] X. Wu, B. Xu, Y. Zheng, H. Ye, J. Yang, and L. He, “Fast video crowd counting with a temporal aware network,” Neurocomputing, vol. 403, pp. 13–20, 2020.
- [90] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international conference on computer vision (ICCV), 2015, pp. 1440–1448. [90] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international conference on computer vision (ICCV), 2015, pp. 1440–1448.
- [91] Z. Zou, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: A survey,” 2019. [91] Z. Zou, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: A survey,” 2019.
- [92] A. Yao, J. Gall, C. Leistner, and L. Van Gool, “Interactive object detection,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 3242–3249. [92] A. Yao, J. Gall, C. Leistner, and L. Van Gool, “Interactive object detection,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 3242–3249.
- [93] K. Madono, T. Nakano, T. Kobayashi, and T. Ogawa, “Efficient human-in-the-loop object detection using bi-directional deep sort and annotation-free segment identification,” in APSIPA ASC, 2020, pp. 1226–1233. [93] K. Madono, T. Nakano, T. Kobayashi, and T. Ogawa, “Efficient human-in-the-loop object detection using bi-directional deep sort and annotation-free segment identification,” in APSIPA ASC, 2020, pp. 1226–1233.
- [94] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in IEEE International Conference on Image Processing (ICIP), 2017, pp. 3645–3649. [94] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in IEEE International Conference on Image Processing (ICIP), 2017, pp. 3645–3649.
- [95] M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE signal processing magazine, vol. 14, no. 2, pp. 24–41, 1997. [95] M. R. Banham and A. K. Katsaggelos, “Digital image restoration,” IEEE signal processing magazine, vol. 14, no. 2, pp. 24–41, 1997.
- [96] A. Criminisi, P. Perez, and K. Toyama, “Object removal by exemplar-based inpainting,” in Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, 2003, pp. II–II. [96] A. Criminisi, P. Perez, and K. Toyama, “Object removal by exemplar-based inpainting,” in Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, 2003, pp. II–II.
- [97] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in European Conference on Computer Vision (ECCV), 2018, pp. 85–100. [97] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in European Conference on Computer Vision (ECCV), 2018, pp. 85–100.
- [98] T. Weber, H. Hußmann, Z. Han, S. Matthes, and Y. Liu, “Draw with me: Human-in-the-loop for image restoration,” in IUI, 2020, pp. 243–253. [98] T. Weber, H. Hußmann, Z. Han, S. Matthes, and Y. Liu, “Draw with me: Human-in-the-loop for image restoration,” in IUI, 2020, pp. 243–253.
- [99] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 9446–9454. [99] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 9446–9454.
- [100] J. Roels, F. Vernaillen, A. Kremer, A. Gonçalves, J. Aelterman, H. Q. Luong, B. Goossens, W. Philips, S. Lippens, and Y. Saeys, “A human-in-the-loop approach for semi-automated image restoration in electron microscopy,” bioRxiv, p. 644146, 2019. [100] J. Roels, F. Vernaillen, A. Kremer, A. Gonçalves, J. Aelterman, H. Q. Luong, B. Goossens, W. Philips, S. Lippens, and Y. Saeys, “A human-in-the-loop approach for semi-automated image restoration in electron microscopy,” bioRxiv, p. 644146, 2019.
- [101] S. Minaee, Y. Y. Boykov, F. Porikli, A. J. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE Transaction on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021. [101] S. Minaee, Y. Y. Boykov, F. Porikli, A. J. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE Transaction on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021.
- [102] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481–2495, 2017. [102] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481–2495, 2017.
- [103] H. Wang, T. Chen, Z. Wang, and K. Ma, “Efficiently troubleshooting image segmentation models with human-in-the-loop,” pp. 1–1, 2020. [103] H. Wang, T. Chen, Z. Wang, and K. Ma, “Efficiently troubleshooting image segmentation models with human-in-the-loop,” pp. 1–1, 2020.
- [104] A. Taleb, C. Lippert, T. Klein, and M. Nabi, “Multimodal self-supervised learning for medical image analysis,” in IPMI, 2021, pp. 661–673. [104] A. Taleb, C. Lippert, T. Klein, and M. Nabi, “Multimodal self-supervised learning for medical image analysis,” in IPMI, 2021, pp. 661–673.
- [105] M. Ravanbakhsh, V. Tschernezki, F. Last, T. Klein, K. Batmanghelich, V. Tresp, and M. Nabi, “Human-machine collaboration for medical image segmentation,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020, pp. 1040–1044. [105] M. Ravanbakhsh, V. Tschernezki, F. Last, T. Klein, K. Batmanghelich, V. Tresp, and M. Nabi, “Human-machine collaboration for medical image segmentation,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020, pp. 1040–1044.
- [106] Y. Murata and Y. Dobashi, “Automatic image enhancement taking into account user preference,” in CW, 2019, pp. 374–377. [106] Y. Murata and Y. Dobashi, “Automatic image enhancement taking into account user preference,” in CW, 2019, pp. 374–377.
- [107] M. Fischer, K. Kobs, and A. Hotho, “Nicer: Aesthetic image enhancement with humans in the loop,” in The Thirteenth International Conference on Advances in Computer-Human Interactions, 2020, pp. 357–362. [107] M. Fischer, K. Kobs, and A. Hotho, “Nicer: Aesthetic image enhancement with humans in the loop,” in The Thirteenth International Conference on Advances in Computer-Human Interactions, 2020, pp. 357–362.
- [108] A. Benard and M. Gygli, “Interactive video object segmentation in the wild,” 2017. [108] A. Benard and M. Gygli, “Interactive video object segmentation in the wild,” 2017.
- [109] S. W. Oh, J.-Y. Lee, N. Xu, and S. J. Kim, “Fast user-guided video object segmentation by interaction-and-propagation networks,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5247–5256. [109] S. W. Oh, J.-Y. Lee, N. Xu, and S. J. Kim, “Fast user-guided video object segmentation by interaction-and-propagation networks,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5247–5256.
- [110] K. N. Shukla, A. Potnis, and P. Dwivedy, “A review on image enhancement techniques,” IJEACS, vol. 2, no. 7, pp. 232–235, 2017. [110] K. N. Shukla, A. Potnis, and P. Dwivedy, “A review on image enhancement techniques,” IJEACS, vol. 2, no. 7, pp. 232–235, 2017.
- [111] X. Fu, J. Yan, and C. Fan, “Image aesthetics assessment using composite features from off-the-shelf deep models,” in IEEE International Conference on Image Processing (ICIP), 2018, pp. 3528–3532. [111] X. Fu, J. Yan, and C. Fan, “Image aesthetics assessment using composite features from off-the-shelf deep models,” in IEEE International Conference on Image Processing (ICIP), 2018, pp. 3528–3532.
- [112] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to rank using gradient descent,” in Proceedings of International Conference on Machine Learning (ICML), 2005, pp. 89–96. [112] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to rank using gradient descent,” in Proceedings of International Conference on Machine Learning (ICML), 2005, pp. 89–96.
- [113] R. Yao, G. Lin, S. Xia, J. Zhao, and Y. Zhou, “Video object segmentation and tracking: A survey,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 11, no. 4, pp. 1–47, 2020. [113] R. Yao, G. Lin, S. Xia, J. Zhao, and Y. Zhou, “Video object segmentation and tracking: A survey,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 11, no. 4, pp. 1–47, 2020.
- [114] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool, “One-shot video object segmentation,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 221–230. [114] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool, “One-shot video object segmentation,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 221–230.
- [115] N. Xu, B. Price, S. Cohen, J. Yang, and T. S. Huang, “Deep interactive object selection,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 373–381. [115] N. Xu, B. Price, S. Cohen, J. Yang, and T. S. Huang, “Deep interactive object selection,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 373–381.
- [116] M. Hudec, E. Mináriková, R. Mesiar, A. Saranti, and A. Holzinger, “Classification by ordinal sums of conjunctive and disjunctive functions for explainable ai and interpretable machine learning solutions,” Knowledge-Based Systems, vol. 220, p. 106916, 2021. [116] M. Hudec, E. Mináriková, R. Mesiar, A. Saranti, and A. Holzinger, “Classification by ordinal sums of conjunctive and disjunctive functions for explainable ai and interpretable machine learning solutions,” Knowledge-Based Systems, vol. 220, p. 106916, 2021.
- [117] S. Brostoff and M. A. Sasse, “Safe and sound: a safety-critical approach to security,” in Proceedings of the 2001 workshop on New security paradigms, 2001, pp. 41–50. [117] S. Brostoff and M. A. Sasse, “Safe and sound: a safety-critical approach to security,” in Proceedings of the 2001 workshop on New security paradigms, 2001, pp. 41–50.
- [118] L. F. Cranor, “A framework for reasoning about the human in the loop,” in Proceedings of the 1st Conference on Usability, Psychology, and Security, 2008, pp. 1–15. [118] L. F. Cranor, “A framework for reasoning about the human in the loop,” in Proceedings of the 1st Conference on Usability, Psychology, and Security, 2008, pp. 1–15.
- [119] A. Machiry, R. Tahiliani, and M. Naik, “Dynodroid: An input generation system for android apps,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, 2013, pp. 224–234. [119] A. Machiry, R. Tahiliani, and M. Naik, “Dynodroid: An input generation system for android apps,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, 2013, pp. 224–234.
- [120] A. Kovashka, D. Parikh, and K. Grauman, “Whittlesearch: Interactive image search with relative attribute feedback,” International Journal of Computer Vision, vol. 115, no. 2, pp. 185–210, 2015. [120] A. Kovashka, D. Parikh, and K. Grauman, “Whittlesearch: Interactive image search with relative attribute feedback,” International Journal of Computer Vision, vol. 115, no. 2, pp. 185–210, 2015.
- [121] L. Rosenberg, “Artificial swarm intelligence, a human-in-the-loop approach to ai,” in The AAAI Conference on Artificial Intelligence, vol. 30, no. 1, 2016. [121] L. Rosenberg, “Artificial swarm intelligence, a human-in-the-loop approach to ai,” in The AAAI Conference on Artificial Intelligence, vol. 30, no. 1, 2016.
- [122] Y. Shoshitaishvili, M. Weissbacher, L. Dresel, C. Salls, R. Wang, C. Kruegel, and G. Vigna, “Rise of the hacrs: Augmenting autonomous cyber reasoning systems with human assistance,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 347–362. [122] Y. Shoshitaishvili, M. Weissbacher, L. Dresel, C. Salls, R. Wang, C. Kruegel, and G. Vigna, “Rise of the hacrs: Augmenting autonomous cyber reasoning systems with human assistance,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 347–362.
- [123] M. S. Wogalter, “Communication-human information processing (c-hip) model,” in Forensic Human Factors and Ergonomics, 2018, pp. 33–49. [123] M. S. Wogalter, “Communication-human information processing (c-hip) model,” in Forensic Human Factors and Ergonomics, 2018, pp. 33–49.
- [124] L. Ma, “Towards understanding and simplifying human-in-the-loop machine learning,” pp. 1–1, 2018. [124] L. Ma, “Towards understanding and simplifying human-in-the-loop machine learning,” pp. 1–1, 2018.
- [125] M. A. Salam, M. E. Koone, S. Thirumuruganathan, G. Das, and S. Basu Roy, “A human-in-the-loop attribute design framework for classification,” 2019, pp. 1612–1622. [125] M. A. Salam, M. E. Koone, S. Thirumuruganathan, G. Das, and S. Basu Roy, “A human-in-the-loop attribute design framework for classification,” 2019, pp. 1612–1622.
- [126] B. A. Plummer, M. H. Kiapour, S. Zheng, and R. Piramuthu, “Give me a hint! navigating image databases using human-in-the-loop feedback,” in IEEE Winter Conference on Applications of Computer Vision (WACV), 2019, pp. 2048–2057. [126] B. A. Plummer, M. H. Kiapour, S. Zheng, and R. Piramuthu, “Give me a hint! navigating image databases using human-in-the-loop feedback,” in IEEE Winter Conference on Applications of Computer Vision (WACV), 2019, pp. 2048–2057.
- [127] F. Wrede and A. Hellander, “Smart computational exploration of stochastic gene regulatory network models using human-in-the-loop semi-supervised learning,” Bioinformatics, vol. 35, no. 24, pp. 5199–5206, 2019. [127] F. Wrede and A. Hellander, “Smart computational exploration of stochastic gene regulatory network models using human-in-the-loop semi-supervised learning,” Bioinformatics, vol. 35, no. 24, pp. 5199–5206, 2019.
- [128] H. V. Singh and Q. H. Mahmoud, “Human-in-the-loop error precursor detection using language translation modeling of hmi states,” in SMC, 2020, pp. 2237–2242. [128] H. V. Singh and Q. H. Mahmoud, “Human-in-the-loop error precursor detection using language translation modeling of hmi states,” in SMC, 2020, pp. 2237–2242.
- [129] G. Demartini, S. Mizzaro, and D. Spina, “Human-in-the-loop artificial intelligence for fighting online misinformation: Challenges and opportunities,” The Bulletin of the Technical Committee on Data Engineering, vol. 43, no. 3, pp. 1–10, 2020. [129] G. Demartini, S. Mizzaro, and D. Spina, “Human-in-the-loop artificial intelligence for fighting online misinformation: Challenges and opportunities,” The Bulletin of the Technical Committee on Data Engineering, vol. 43, no. 3, pp. 1–10, 2020.
- [130] D. Odekerken and F. Bex, “Towards transparent human-in-the-loop classification of fraudulent web shops,” in Legal Knowledge and Information Systems, 2020, pp. 239–242. [130] D. Odekerken and F. Bex, “Towards transparent human-in-the-loop classification of fraudulent web shops,” in Legal Knowledge and Information Systems, 2020, pp. 239–242.
- [131] M. Böhme, C. Geethal, and V.-T. Pham, “Human-in-the-loop automatic program repair,” in ICST, 2020, pp. 274–285. [131] M. Böhme, C. Geethal, and V.-T. Pham, “Human-in-the-loop automatic program repair,” in ICST, 2020, pp. 274–285.
- [132] A. Renner, “Designing for the human in the loop: Transparency and control in interactive machine learning,” Ph.D. dissertation, University of Maryland, College Park, 2020. [132] A. Renner, “Designing for the human in the loop: Transparency and control in interactive machine learning,” Ph.D. dissertation, University of Maryland, College Park, 2020.
- [133] J. B. Davidson, R. B. Graham, S. Beck, R. T. Marler, and S. L. Fischer, “Improving human-in-the-loop simulation to optimize soldier-systems integration,” Applied Ergonomics, vol. 90, p. 103267, 2021. [133] J. B. Davidson, R. B. Graham, S. Beck, R. T. Marler, and S. L. Fischer, “Improving human-in-the-loop simulation to optimize soldier-systems integration,” Applied Ergonomics, vol. 90, p. 103267, 2021.
- [134] H. O. Demirel, “Digital human-in-the-loop framework,” in International Conference on Human-Computer Interaction, 2020, pp. 18–32. [134] H. O. Demirel, “Digital human-in-the-loop framework,” in International Conference on Human-Computer Interaction, 2020, pp. 18–32.
- [135] M. Metzner, D. Utsch, M. Walter, C. Hofstetter, C. Ramer, A. Blank, and J. Franke, “A system for human-in-the-loop simulation of industrial collaborative robot applications,” in CASE, 2020, pp. 1520–1525. [135] M. Metzner, D. Utsch, M. Walter, C. Hofstetter, C. Ramer, A. Blank, and J. Franke, “A system for human-in-the-loop simulation of industrial collaborative robot applications,” in CASE, 2020, pp. 1520–1525.
- [136] A. Polisetty Venkata Sai, “Information preparation with the human in the loop,” Ph.D. dissertation, TU Darmstadt, 2020. [136] A. Polisetty Venkata Sai, “Information preparation with the human in the loop,” Ph.D. dissertation, TU Darmstadt, 2020.
- [137] Z. Zhu, Y. Lu, R. Deng, H. Yang, A. B. Fogo, and Y. Huo, “Easierpath: An open-source tool for human-in-the-loop deep learning of renal pathology,” in Interpretable and Annotation-Efficient Learning for Medical Image Computing, 2020, pp. 214–222. [137] Z. Zhu, Y. Lu, R. Deng, H. Yang, A. B. Fogo, and Y. Huo, “Easierpath: An open-source tool for human-in-the-loop deep learning of renal pathology,” in Interpretable and Annotation-Efficient Learning for Medical Image Computing, 2020, pp. 214–222.
- [138] N. Li, S. Adepu, E. Kang, and D. Garlan, “Explanations for human-on-the-loop: A probabilistic model checking approach,” in Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, 2020, pp. 181–187. [138] N. Li, S. Adepu, E. Kang, and D. Garlan, “Explanations for human-on-the-loop: A probabilistic model checking approach,” in Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, 2020, pp. 181–187.
- [139] P. Wiriyathammabhum, D. Summers-Stay, C. Fermüller, and Y. Aloimonos, “Computer vision and natural language processing: recent approaches in multimedia and robotics,” ACM Computing Surveys (CSUR), vol. 49, no. 4, pp. 1–44, 2016. [139] P. Wiriyathammabhum, D. Summers-Stay, C. Fermüller, and Y. Aloimonos, “Computer vision and natural language processing: recent approaches in multimedia and robotics,” ACM Computing Surveys (CSUR), vol. 49, no. 4, pp. 1–44, 2016.
- [140] A. Holzinger, B. Malle, A. Saranti, and B. Pfeifer, “Towards multi-modal causability with graph neural networks enabling information fusion for explainable ai,” Information Fusion, vol. 71, pp. 28–37, 2021. [140] A. Holzinger, B. Malle, A. Saranti, and B. Pfeifer, “Towards multi-modal causability with graph neural networks enabling information fusion for explainable ai,” Information Fusion, vol. 71, pp. 28–37, 2021.
- [141] S. Arora and P. Doshi, “A survey of inverse reinforcement learning: Challenges, methods and progress,” Artificial Intelligence, vol. 297, p. 103500, 2021. [141] S. Arora and P. Doshi, “A survey of inverse reinforcement learning: Challenges, methods and progress,” Artificial Intelligence, vol. 297, p. 103500, 2021.
- [142] A. Doan, A. Ardalan, J. Ballard, S. Das, Y. Govind, P. Konda, H. Li, S. Mudgal, E. Paulson, G. P. Suganthan et al., “Human-in-the-loop challenges for entity matching: A midterm report,” in Proceedings of the 2nd workshop on human-in-the-loop data analytics, 2017, pp. 1–6. [142] A. Doan, A. Ardalan, J. Ballard, S. Das, Y. Govind, P. Konda, H. Li, S. Mudgal, E. Paulson, G. P. Suganthan et al., “Human-in-the-loop challenges for entity matching: A midterm report,” in Proceedings of the 2nd workshop on human-in-the-loop data analytics, 2017, pp. 1–6.
- [143] J. Li, A. H. Miller, S. Chopra, M. Ranzato, and J. Weston, “Dialogue learning with human-in-the-loop,” International Conference on Learning Representations (ICLR), pp. 1–23, 2016. [143] J. Li, A. H. Miller, S. Chopra, M. Ranzato, and J. Weston, “Dialogue learning with human-in-the-loop,” International Conference on Learning Representations (ICLR), pp. 1–23, 2016.
- [144] H. Amirpourazarian, A. Pinheiro, E. Fonseca, M. Ghanbari, and M. Pereira, “Quality evaluation of holographic images coded with standard codecs,” IEEE Transactions on Multimedia, pp. 1–1, 2021. [144] H. Amirpourazarian, A. Pinheiro, E. Fonseca, M. Ghanbari, and M. Pereira, “Quality evaluation of holographic images coded with standard codecs,” IEEE Transactions on Multimedia, pp. 1–1, 2021.
- [145] S. Wan, Y. Hou, F. Bao, Z. Ren, Y. Dong, Q. Dai, and Y. Deng, “Human-in-the-loop low-shot learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 7, pp. 3287–3292, 2021. [145] S. Wan, Y. Hou, F. Bao, Z. Ren, Y. Dong, Q. Dai, and Y. Deng, “Human-in-the-loop low-shot learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 7, pp. 3287–3292, 2021.
- [146] L. Yang, Q. Sun, N. Zhang, and Z. Liu, “Optimal energy operation strategy for we-energy of energy internet based on hybrid reinforcement learning with human-in-the-loop,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, pp. 1–11, 2020. [146] L. Yang, Q. Sun, N. Zhang, and Z. Liu, “基于人机协同混合强化学习的能源互联网We-Energy最优能源运行策略,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, 第1–11页, 2020.
- [147] Y. Fu, X. Zhu, and B. Li, “A survey on instance selection for active learning,” Knowledge and information systems, vol. 35, no. 2, pp. 249–283, 2013. [147] Y. Fu, X. Zhu, and B. Li, “主动学习中实例选择综述,” Knowledge and information systems, 第35卷, 第2期, 第249–283页, 2013.
- [148] J. Zhang, P. Fiers, K. A. Witte, R. W. Jackson, K. L. Poggensee, C. G. Atkeson, and S. H. Collins, “Human-in-the-loop optimization of exoskeleton assistance during walking,” Science, vol. 356, no. 6344, pp. 1280–1284, 2017. [148] J. Zhang, P. Fiers, K. A. Witte, R. W. Jackson, K. L. Poggensee, C. G. Atkeson, and S. H. Collins, “步行过程中外骨骼辅助的人机协同优化,” Science, 第356卷, 第6344期, 第1280–1284页, 2017.
- [149] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient transformers: A survey,” 2020. [149] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “高效Transformer：综述,” 2020.
- [150] J. Kreutzer, S. Riezler, and C. Lawrence, “Offline reinforcement learning from human feedback in real-world sequence-to-sequence tasks,” in SPNLP, 2021, pp. 37–43. [150] J. Kreutzer, S. Riezler, and C. Lawrence, “现实世界序列到序列任务中基于人类反馈的离线强化学习,” 载于 SPNLP, 2021, 第37–43页.
- [151] A. Smith, V. Kumar, J. Boyd-Graber, K. Seppi, and L. Findlater, “Closing the loop: User-centered design and evaluation of a human-in-the-loop topic modeling system,” in IUI, 2018, pp. 293–304. [151] A. Smith, V. Kumar, J. Boyd-Graber, K. Seppi, and L. Findlater, “闭环：人机协同主题建模系统的用户中心设计与评估,” 载于 IUI, 2018, 第293–304页.
- [152] A. Kapoor, J. C. Caicedo, D. Lischinski, and S. B. Kang, “Collaborative personalization of image enhancement,” International Journal of Computer Vision, vol. 108, no. 1-2, pp. 148–164, 2014. [152] A. Kapoor, J. C. Caicedo, D. Lischinski, and S. B. Kang, “图像增强的协作式个性化,” International Journal of Computer Vision, 第108卷, 第1-2期, 第148–164页, 2014.
- [153] J.-S. Jwo, C.-S. Lin, and C.-H. Lee, “Smart technology–driven aspects for human-in-the-loop smart manufacturing,” The International Journal of Advanced Manufacturing Technology, vol. 114, no. 5, pp. 1741–1752, 2021. [153] J.-S. Jwo, C.-S. Lin, and C.-H. Lee, “人机协同智能制造的智能技术驱动层面,” The International Journal of Advanced Manufacturing Technology, 第114卷, 第5期, 第1741–1752页, 2021.
- [154] B. Settles, “Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2011, pp. 1467–1478. [154] B. Settles, “闭环：通过特征和实例查询实现快速、交互式的半监督标注,” 载于自然语言处理经验方法会议 (EMNLP), 2011, 第1467–1478页.
- [155] T. Y. Lee, A. Smith, K. Seppi, N. Elmqvist, J. Boyd-Graber, and L. Findlater, “The human touch: How non-expert users perceive, interpret, and fix topic models,” International Journal of Human-Computer Studies, vol. 105, pp. 28–42, 2017. [155] T. Y. Lee, A. Smith, K. Seppi, N. Elmqvist, J. Boyd-Graber, and L. Findlater, “人文触点：非专业用户如何感知、解释和修复主题模型,” International Journal of Human-Computer Studies, 第105卷, 第28–42页, 2017.
- [156] N. M. Marquand, “Automated modeling of human-in-the-loop systems,” Ph.D. dissertation, Purdue University Graduate School, 2021. [156] N. M. Marquand, “人机协同系统的自动化建模,” 博士论文, 普渡大学研究生院, 2021.
- [157] J. J. Dudley and P. O. Kristensson, “A review of user interface design for interactive machine learning,” ACM Transactions on Interactive Intelligent Systems (TiiS), vol. 8, no. 2, pp. 1–37, 2018. [157] J. J. Dudley and P. O. Kristensson, “交互式机器学习用户界面设计综述,” ACM Transactions on Interactive Intelligent Systems (TiiS), 第8卷, 第2期, 第1–37页, 2018.
- [158] K. Shilton, “Values and ethics in human-computer interaction,” Foundations and Trends® in Human–Computer Interaction, vol. 12, no. 2, 2018. [158] K. Shilton, “人机交互中的价值观与伦理,” Foundations and Trends® in Human–Computer Interaction, 第12卷, 第2期, 2018.
- [159] A. Jolfaei, M. Usman, M. Roveri, M. Sheng, M. Palaniswami, and K. Kant, “Guest editorial: Computational intelligence for human-in-the-loop cyber physical systems,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 6, no. 1, pp. 2–5, 2022. [159] A. Jolfaei, M. Usman, M. Roveri, M. Sheng, M. Palaniswami, and K. Kant, “客座社论：面向人机协同信息物理系统的计算智能,” IEEE Transactions on Emerging Topics in Computational Intelligence, 第6卷, 第1期, 第2–5页, 2022.
- [160] W. Xu, M. J. Dainoff, L. Ge, and Z. Gao, “Transitioning to human interaction with ai systems: New challenges and opportunities for hci professionals to enable human-centered ai,” International Journal of Human–Computer Interaction, pp. 1–25, 2022. [160] W. Xu, M. J. Dainoff, L. Ge, and Z. Gao, “向人机交互AI系统的转型：HCI专业人员实现以人为本的AI所面临的新挑战与机遇,” International Journal of Human–Computer Interaction, 第1–25页, 2022.
- [161] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. Change Loy, “Domain generalization: A survey,” 2021. [161] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. Change Loy, “域泛化：综述,” 2021.