Reducing Human Annotation with ML Active Learning利用机器学习主动学习减少人工标注

Active learning aims to select the unlabeled samples that are most informative and ask for labels only for those samples. By interactively querying a user (or some other information source) to label new data points, we can train models using less labels. This tutorial/tutorial walks you through the active learning workflow and shows you how to implement three commonly used query strategies: uncertainty sampling, diversity based sampling and query by committee. We provide intuitive explanations of these methods along with Python implementations. Finally, we will discuss the pros and cons of each method and how they can be used together to create powerful human-in-the-loop systems when annotation is expensive/scarcely available.主动学习旨在挑选出信息量最大的未标注样本，并仅针对这些样本请求标注。通过交互式地询问用户（或其他信息源）来标注新数据点，我们可以用更少的标签来训练模型。本教程将引导你了解主动学习的工作流程，并展示如何实现三种常用的查询策略：不确定性采样、基于多样性的采样和委员会查询。我们提供了这些方法的直观解释以及 Python 实现。最后，我们将讨论每种方法的优缺点，以及在标注成本高昂或资源稀缺时，如何将它们结合起来构建强大的人机协同系统。

Table of Contents目录

1. What is Active Learning
   • Learning from Uncertain or Edge-Case Samples从不确定或边缘案例样本中学习
2. Active Learning Techniques
   • Uncertainty Sampling不确定性采样
   • Diversity-based Sampling基于多样性的采样
   • Query By Committee (QBC)委员会查询 (QBC)
3. Pros and Cons of Active Learning
   • Benefits of Active Learning主动学习的优势
   • Limitations of Active Learning主动学习的局限性
   • How to Choose the Right Strategy in Practice如何在实践中选择正确的策略
4. Conclusion结论

1. What Is Active Learning1. 什么是主动学习

Active learning is an approach to machine learning in which a learning algorithm is able to interactively query a user (or some other information source) to obtain the desired outputs at new data points. In active learning, rather than feeding a learning algorithm a massive pool of labeled examples, you allow the model to look at the unlabeled data points first. It will pick the data points it wants to learn from and send only those to the human for labeling. You repeat this process in cycles. Using active learning, you can often achieve comparable performance to normal supervised learning with many fewer annotated training examples. See figure 1 for the active learning cycle.
主动学习是一种机器学习方法，在这种方法中，学习算法能够交互式地查询用户（或其他信息源）以获得新数据点的理想输出。在主动学习中，你无需向学习算法提供海量的标注示例池，而是先让模型查看未标注的数据点。模型会挑选出它想要学习的数据点，并仅将这些点发送给人工进行标注。你可以循环重复此过程。使用主动学习，通常可以用远少于普通监督学习的标注训练示例，达到相当的性能。参见图 1 了解主动学习循环。

Particularly when humans are in the loop, this technique shines. Rather than having the annotator label thousands of simple/redundant samples, the model instead surfaces which examples it’s least certain about or which examples appear odd/outlier, and then have the annotator label those.当有人工参与时，这种技术尤为出色。与其让标注者标注成千上万的简单或冗余样本，不如让模型找出它最不确定的示例，或者看起来奇怪/属于异常值的示例，然后让标注者对这些进行标注。

To understand the significance of Active Learning, think about an image classification task with 500,000 images. At 20 seconds per image, it will take over 2,700 hours to label this dataset. With active learning, you only request human attention to a few of these images that actually need help to improve your model substantially. This drastically cuts costs and the effort needed to label.要理解主动学习的重要性，可以考虑一个包含 50 万张图像的图像分类任务。如果每张图像需要 20 秒，标注整个数据集将耗时超过 2700 小时。通过主动学习，你只需请求人工关注其中少数确实需要帮助的图像，即可显著提升模型性能。这大大降低了成本和标注所需的工作量。

1.1. Learning From Uncertain or Edge-Case Samples1.1. 从不确定或边缘案例样本中学习

One major benefit of active learning is that your model can focus on learning from the most difficult examples. Typically these examples fall around the decision boundary where your model is most uncertain and have the largest potential to improve the model.主动学习的一个主要好处是，你的模型可以专注于从最困难的示例中学习。通常，这些示例位于模型最不确定的决策边界附近，并且具有提升模型性能的最大潜力。

Effectively this allows your dataset to be enriched with targeted samples. The dataset size will not increase with synthetic transformations, but it will gain valuable information with the addition of new samples.实际上，这允许通过有针对性的样本来丰富你的数据集。数据集的大小不会通过合成变换而增加，但会通过添加新样本获得有价值的信息。

Imagine training a model for self-driving cars. There are some rare events that may not be present frequently enough to be captured while collecting raw data. A cyclist coming out from behind a parked car is a great example. With active learning these rare cases can be presented early on and delivered to human annotators for review. The same could be said for a fraud detection model where you may want your model to direct human reviewers to suspicious activity versus transactions that are clearly legitimate or clearly fraud.想象一下为自动驾驶汽车训练模型。有些罕见事件在收集原始数据时可能出现的频率不够高。一个骑自行车的人从停放的汽车后面冲出来就是一个很好的例子。通过主动学习，这些罕见案例可以尽早呈现并提交给人工标注者进行审查。欺诈检测模型也是如此，你可能希望模型将可疑活动引导给人工审查员，而不是处理那些明显合法或明显欺诈的交易。

If you’d like to follow along with the code for each step you can run and download the companion notebook here: https://github. com/lucasbraga461/active-learning/blob/main/active-learning/notebook.ipynb如果你想跟随每一步的代码进行操作，可以在此处运行并下载配套笔记本：https://github.com/lucasbraga461/active-learning/blob/main/active-learning/notebook.ipynb

If you’d like to learn more about the theory behind this implementation, you can access my research paper here, it’s free access: https://doi.org/10.1109/ACCESS.2025.3624650如果你想了解更多关于此实现背后的理论，可以访问我的研究论文，它是免费获取的：https://doi.org/10.1109/ACCESS.2025.3624650

2. Active Learning Techniques2. 主动学习技术

The intuition behind active learning is using a query strategy to determine which unlabeled samples should be labeled first. Rather than randomly picking samples from our pool, we query the model about what samples it would like to look at. Typically these are samples that will help the model make better predictions. Active learning tends to be useful when labels are slow to acquire, require expert knowledge, or are costly.主动学习背后的直觉是使用查询策略来确定应首先标注哪些未标注样本。我们不是从池中随机挑选样本，而是询问模型它想查看哪些样本。通常，这些样本能帮助模型做出更好的预测。当标签获取缓慢、需要专家知识或成本高昂时，主动学习往往非常有用。

There are many strategies that generally trade-off exploration and exploitation of the regions where the model has the largest uncertainties. These methods have been applied in vision-based tasks, document classification, anomaly detection, and even medicine.有许多策略通常在探索和利用模型不确定性最大的区域之间进行权衡。这些方法已应用于视觉任务、文档分类、异常检测，甚至医学领域。

Below we’ll go over three of the most common query strategies in an intuitive manner: uncertainty sampling, diversity-based sampling, query by committee.下面我们将以直观的方式介绍三种最常见的查询策略：不确定性采样、基于多样性的采样和委员会查询。

Dataset note: The examples in this article use a synthetic dataset created only for demonstration purposes.数据集说明：本文中的示例使用仅用于演示目的的合成数据集。

2.1. Uncertainty Sampling2.1. 不确定性采样

Uncertainty sampling is often the first approach practitioners attempt because it requires very little effort and often leads to good initial gains.不确定性采样通常是从业者尝试的第一种方法，因为它需要的工作量非常少，并且通常能带来不错的初步收益。

Uncertainty sampling is conceptually simple. You start off by labeling only enough samples to create your initial training set. Train your initial model on these samples. This initial model doesn’t have to be great! All we need it to do is make rough estimates of the probabilities for our unlabeled samples. Once we’ve trained the model, we identify the samples about which the model is least certain. These are usually the ones for which it predicts probabilities close to the decision threshold. Then we have these uncertain samples labeled by a human and retrain our model on this new data!不确定性采样的概念很简单。你首先只标注足够的样本来创建初始训练集。在这些样本上训练初始模型。这个初始模型不必非常完美！我们只需要它对未标注样本的概率做出粗略估计。模型训练完成后，我们要找出模型最不确定的样本。这些通常是那些预测概率接近决策阈值的样本。然后，我们让人工对这些不确定的样本进行标注，并在这些新数据上重新训练模型！

If we repeat this process, our model incrementally improves upon its understanding where it’s least confident. You should see quicker gains then if you labeled many easy/redundant examples.如果我们重复这个过程，模型会对它最不自信的区域的理解进行增量改进。你应该会看到比标注许多简单/冗余示例更快的收益。

2.1.1. Train the First Model With Cross-Validation2.1.1. 使用交叉验证训练第一个模型

Before querying uncertain samples, we first need a model that performs reasonably well even when there is not much labeled data available. An easy way to accomplish this is to split the labeled data into training and validation folds, then run a small hyperparameter search using cross-validation to prevent overfitting. The result of splitting the dataset in Code Block 1 can be seen in Figure 2.在查询不确定样本之前，我们首先需要一个即使在标注数据不多时也能表现良好的模型。实现这一目标的一个简单方法是将标注数据拆分为训练集和验证集，然后使用交叉验证进行小规模的超参数搜索，以防止过拟合。代码块 1 中拆分数据集的结果如图 2 所示。

Code Block 1.  Split X and y代码块 1. 拆分 X 和 y

X = df_i1.drop(columns=['label'])
y = df_i1['label']
X

Teams usually run lighter grid searches with 3 or 5 folds in practice, so don’t feel that you need this many folds when applying this process to your projects.在实践中，团队通常会使用 3 折或 5 折进行较轻量的网格搜索，因此在将此过程应用于你的项目时，不必觉得一定要用这么多折数。

Code Block 2. Train a model using nested cross-validation.代码块 2. 使用嵌套交叉验证训练模型。

# Hyperparameter grid and CV settings
param_grid = {"C": [0.001, 0.01, 0.1, 1, 10, 100]}
inner_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)

fold_metrics = []
best_params_each_fold = []

for train_idx, test_idx in outer_cv.split(X, y):
    X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
    y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

    grid = GridSearchCV(
        LogisticRegression(max_iter=500),
        param_grid,
        cv=inner_cv,
        scoring="f1",
        n_jobs=-1
    )
    grid.fit(X_train, y_train)

    best_model = grid.best_estimator_
    best_params_each_fold.append(grid.best_params_)

    preds = best_model.predict(X_test)
    fold_metrics.append({
        "precision": precision_score(y_test, preds),
        "recall": recall_score(y_test, preds),
        "f1": f1_score(y_test, preds)
    })

# Use the most frequent best parameters to train the final model
final_params = pd.DataFrame(best_params_each_fold).mode().iloc[0].to_dict()
final_model = LogisticRegression(**final_params, max_iter=500)
final_model.fit(X, y)

Now that we have our cross validated model we can apply it to our unlabeled set to get a measure of uncertainty. Figure 3 below shows what dataset fold_metrics looks like from Code Block 2.现在我们有了交叉验证后的模型，可以将其应用于未标注集以获得不确定性度量。下方的图 3 展示了代码块 2 中的数据集 fold_metrics 的样子。

2.1.2. Score the Unlabeled Data and Inspect the Probabilities2.1.2. 对未标注数据进行评分并检查概率

We now want to calculate predicted probabilities for every sample in our unlabeled pool. Plotting these scores can help us see where our model is least confident.现在我们要计算未标注池中每个样本的预测概率。绘制这些分数可以帮助我们看到模型最不自信的地方。

Code Block 3. Visualize the distribution of the predicted scores代码块 3. 可视化预测分数的分布

import seaborn as sns

def plot_prediction_distribution(df, column_name):
    if column_name not in df.columns:
        raise ValueError(f"Column '{column_name}' not found in dataframe.")

    if "Prediction_Probability" not in df.columns:
        raise ValueError("Column 'Prediction_Probability' not found in dataframe. Ensure predictions were added.")

    # Ensure Predicted_Label is correctly assigned (1 = Valid, 0 = Invalid)
    plt.figure(figsize=(10, 6))
    sns.histplot(df, x=column_name, hue="Predicted_Label", bins=30, kde=True, palette="coolwarm")

    plt.title(f"Distribution of {column_name} by Prediction Probability")
    plt.xlabel(column_name)
    plt.ylabel("Density")

    # Corrected Legend Order (1 = Valid, 0 = Invalid)
    plt.legend(title="Predicted Label", labels=["Valid", "Invalid"])
    plt.show()

from helpers import prediction_plots

# Get probability scores
new_probabilities = final_model.predict_proba(X_unlabeled)[:, 1]
X_unlabeled["Prediction_Probability"] = new_probabilities
prediction_plots.plot_prediction_distribution(X_unlabeled, "Prediction_Probability")

Because this is a binary classification problem our model will often be least certain with probabilities near 0.5. This represents where AL can be the most valuable. Figure 4 below is what the plot looks like from running Code Block 3.因为这是一个二分类问题，模型通常在概率接近 0.5 时最不确定。这正是主动学习（AL）最有价值的地方。下方的图 4 是运行代码块 3 后的绘图结果。

2.1.3. Choose Samples Inside the Uncertainty Window2.1.3. 选择不确定性窗口内的样本

For our next batch to label we will focus on an area around the decision threshold. We can define a small band around 0.5 then sample from that region. Figure 5 below displays the resulting plot from running Code Block 4.对于下一批要标注的样本，我们将聚焦于决策阈值周围的区域。我们可以定义一个 0.5 左右的小区间，然后从该区域进行采样。下方的图 5 显示了运行代码块 4 后的结果图。

center = 0.50
lower = center - 0.05
upper = center + 0.05

mask = (X_unlabeled["Prediction_Probability"] > lower) & \
        (X_unlabeled["Prediction_Probability"] <= upper)

batch_size = 60
selected_batch = X_unlabeled[mask].sample(batch_size, random_state=42)

These samples should represent where our model is least confident.这些样本应该代表了模型最不自信的地方。

2.1.4. Iterate2.1.4. 迭代

Now repeat the cycle:现在重复这个循环：

1. Train the model训练模型
2. Predict probs on unlabeled set在未标注集上预测概率
3. Select samples closest to uncertain region选择最接近不确定区域的样本
4. Label those and标注这些样本并
5. Rinse and repeat重复上述过程

As this cycle continues the model should become more confident and the uncertain region will become smaller. When this process starts to plateau you can add in more exploratory techniques like diversity sampling or committee-based sampling.随着循环的继续，模型应该会变得更加自信，不确定区域也会变小。当此过程开始进入平台期时，你可以加入更多探索性技术，如多样性采样或基于委员会的采样。

2.1.5. Strengths and Weaknesses of Uncertainty Sampling2.1.5. 不确定性采样的优缺点

Strenghts优点

• Easy to implement and low computational cost.易于实现，计算成本低。
• Can use any model that outputs probability estimates.可以使用任何输出概率估计的模型。
• Extremely useful when labeling is costly or when you have limited amounts of data.在标注成本高昂或数据量有限时非常有用。

Weaknesses缺点

• May excessively focus on a small region of the feature space.可能过度关注特征空间的一小部分区域。
• Relies on model’s ability to accurately estimate probabilities.依赖于模型准确估计概率的能力。
• May select highly similar edge cases over and over again without exploring the data-set as a whole.可能会反复选择高度相似的边缘案例，而没有探索整个数据集。

Summary: Uncertainty Sampling总结：不确定性采样

• Most effective when used in the early iterations of modeling when the model has not quite defined the boundary.在建模的早期迭代中，当模型尚未完全定义边界时最为有效。
• Ultra Efficient since it only requires model probability estimates.超高效，因为它只需要模型概率估计。
• May focus too narrowly if used for too many iterations without other forms of exploration.如果迭代次数过多而没有其他探索形式，可能会关注得过于狭窄。

2.2. Diversity-Based Sampling2.2. 基于多样性的采样

Diversity-based sampling can be considered another extension of active learning where labeling choices are made based upon how diverse/uncommon each unlabeled sample is when compared to all other samples within the feature space. Instead of simply choosing where your model is most uncertain you attempt to choose examples that cover more ground. This can allow your model to label more expressive examples and gain better overall insight into the data it is working with.基于多样性的采样可以被视为主动学习的另一种扩展，其标注选择基于每个未标注样本在特征空间中与其他所有样本相比有多么多样/罕见。与其仅仅选择模型最不确定的地方，不如尝试选择覆盖面更广的示例。这可以使你的模型标注出更具表现力的示例，并对它所处理的数据获得更好的整体洞察。

Diversity-based sampling can be beneficial if your model begins to converge and you find yourself repeatedly querying the same types of edge cases with uncertainty sampling. By sampling more sparsely you will build a more representative training set which can lead to greater gains in future iterations.如果你的模型开始收敛，并且你发现自己在使用不确定性采样时反复查询相同类型的边缘案例，那么基于多样性的采样可能会有所帮助。通过更稀疏地采样，你将构建一个更具代表性的训练集，从而在未来的迭代中获得更大的收益。

2.2.1. Practical Example2.2.1. 实践示例

Let’s say you’ve already ran through a few iterations of uncertainty sampling and already have a couple hundred labeled samples. You’re likely beginning to see a sharp drop off in return because you’re bound to keep querying similar samples. Now would be a good time to take a diversity based step.假设你已经进行了几次不确定性采样迭代，并且已经有了几百个标注样本。你可能会开始看到回报急剧下降，因为你注定会不断查询相似的样本。现在是采取基于多样性步骤的好时机。

In order to do this, we will consider each sample in our pool unlabeled and try to estimate how many nearest neighbors it has. Samples that are located within very dense clusters will likely be very similar to samples you’ve already trained on. Alternatively, samples that have very few neighbors are samples that may exist in more sparsely populated regions.为了做到这一点，我们将考虑池中的每个未标注样本，并尝试估计它有多少个最近邻。位于非常密集簇中的样本很可能与你已经训练过的样本非常相似。相反，邻居很少的样本可能存在于人口更稀疏的区域。

A quick and dirty way to approximate this is to use sklearn’s KNN algorithm and find the average distance to your nearest neighbors. Remember to scale your features before computing distances with Euclidean. Refer to Figure 6 for an illustration of KNN.一种快速而粗略的近似方法是使用 sklearn 的 KNN 算法并找到到最近邻的平均距离。在使用欧几里得距离计算距离之前，记得缩放你的特征。参考图 6 了解 KNN 的说明。

Code Block 5 demonstrates an example workflow. Figure 7 illustrates the resulting dataset most_diverse from Code Block 5.代码块 5 展示了一个工作流程示例。图 7 说明了代码块 5 中产生的最多样化数据集 most_diverse。

Code Block 5. Select samples using diverse sampling strategy代码块 5. 使用多样化采样策略选择样本

from sklearn.neighbors import NearestNeighbors

# Fit the KNN model on the features of the unlabeled data
knn = NearestNeighbors(n_neighbors=k, metric="euclidean")
knn.fit(X_unlabeled[feature_cols])

# Compute average neighbor distance as a proxy for sparsity
distances, _ = knn.kneighbors(X_unlabeled[feature_cols])
X_unlabeled["Density"] = distances.mean(axis=1)

# Select the most diverse samples
most_diverse = X_unlabeled.nlargest(n_samples, "Density")
most_diverse.head()

The selected points usually make up a batch of between 50 and 60 samples. You label them, add them to your ever-growing training set, and retrain your model. From this point, you can see how your model’s predictions evolve, plot the probability distribution once more, or determine whether another round of diversity selection is beneficial before proceeding with active learning via another query strategy.选定的点通常组成 50 到 60 个样本的批次。你标注它们，将它们添加到你不断增长的训练集中，并重新训练你的模型。从这一点开始，你可以观察模型预测的演变，再次绘制概率分布，或者在通过另一种查询策略进行主动学习之前，确定是否需要进行另一轮多样性选择。

2.2.2. Advantages and Disadvantages of Diversity Sampling2.2.2. 多样性采样的优缺点

Advantages优点

• helps you build a more representative training set by exploring regions of the data that were previously underrepresented.通过探索之前代表性不足的数据区域，帮助你构建更具代表性的训练集。
• fewer redundant samples when compared to repeatedly querying by uncertainty only.与仅通过不确定性反复查询相比，冗余样本更少。

Disadvantages缺点

• harder to implement than “vanilla” uncertainty sampling.比“原始”不确定性采样更难实现。
• distance metrics, clustering methods, or visual diagnostics needed to intelligently select samples.需要距离度量、聚类方法或视觉诊断来智能地选择样本。

Summary总结

Diversity Sampling promotes exploration of regions in your data which are not covered by your training set. Lowers redundancy when compared to only using uncertainty for sample selection. Depends on distance metrics or clustering, so can be slightly more expensive.多样性采样促进了对数据中未被训练集覆盖区域的探索。与仅使用不确定性进行样本选择相比，降低了冗余度。依赖于距离度量或聚类，因此成本可能略高。

2.3. Query By Committee (QBC)2.3. 委员会查询 (QBC)

Query by Committee maintains a committee of models rather than a single model. The committee members are each trained on the currently labeled data. Since different models may rely on different learning algorithms, or have different internal defaults, they may give slightly different answers. We can use this.委员会查询维护一个模型委员会，而不是单个模型。委员会成员各自在当前标注的数据上进行训练。由于不同的模型可能依赖于不同的学习算法，或具有不同的内部默认设置，它们可能会给出略有不同的答案。我们可以利用这一点。

When the committee votes wildly among its members, we know we’ve found a difficult/ambiguous sample that would do well to be manually labeled.当委员会成员之间投票结果分歧很大时，我们就知道找到了一个困难/模糊的样本，人工标注它会很有用。

Essentially, we search for points where committee members disagree and query those points for labeling. That way, the AL loop will focus on those samples that are most likely to give the biggest bang for your modeling buck.本质上，我们搜索委员会成员意见不一致的点，并查询这些点进行标注。这样，AL 循环将专注于那些最有可能为你建模投入带来最大回报的样本。

2.3.1. Training a Committee of Models2.3.1. 训练模型委员会

For our committee we’ll want models that reason about the data differently. As such, let’s train three different kinds of models: a linear model, a tree-based model, and a kernel model.对于我们的委员会，我们需要以不同方式推理数据的模型。因此，让我们训练三种不同类型的模型：线性模型、基于树的模型和核模型。

Below is a function that will train four models on the currently-labeled data.下面是一个函数，它将在当前标注的数据上训练四个模型。

Code Block 6. Train a committee of models代码块 6. 训练模型委员会

def train_qbc_committee(X_train, y_train):
    """
    Trains a small committee of diverse models for QBC.
    Returns a dictionary of fitted models.
    """
    models = {
        "Logistic Regression": LogisticRegression(C=0.1, max_iter=500),
        "Random Forest": RandomForestClassifier(n_estimators=50, max_depth=5, min_samples_leaf=5),
        "Extra Trees": ExtraTreesClassifier(n_estimators=50, max_depth=5, min_samples_leaf=5),
        "SVM": SVC(kernel="rbf", C=0.1, probability=True)
    }

    trained = {}
    for name, model in models.items():
        print(f"Training {name}...")
        model.fit(X_train, y_train)
        trained[name] = model

    return trained

X = df_i6.drop(columns=["label"])
y = df_i6["label"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=42
)

committee = query_by_committee.train_qbc_committee(X_train, y_train)

Once this code has been run, we have a group of models that we can ask for individual predictions/probability estimates to use in the next step of the pipeline.一旦这段代码运行完毕，我们就拥有了一组模型，可以在流水线的下一步中使用它们来获取单独的预测/概率估计。

2.3.2. Evaluating the Committee and Building a Stacked Model2.3.2. 评估委员会并构建堆叠模型

Each committee member will output a probability for the positive class. We can take all of those probabilities, for each item in our pool, and build a new feature set that we can trivially train a meta-model on top of. This process is called stacking.每个委员会成员将输出正类的概率。我们可以获取池中每个项目的所有这些概率，并构建一个新的特征集，我们可以在其上轻松训练元模型。这个过程称为堆叠 (stacking)。

Since Logistic Regression is trivial to interpret, and works well with this small number of stacking inputs, we will use it for our stacking layer.由于逻辑回归易于解释，并且在处理少量堆叠输入时效果很好，我们将使用它作为我们的堆叠层。

Table 1. Model Performance after QBC表 1. QBC 后的模型性能

Model	Best parameters	F1-Score	Precision	Recall
LogisticRegression	‘C’: 0.001	0.7688	0.6975	0.8564
RandomForestClassifier	‘max_depth’: 7, ‘min_samples_leaf’: 4, ‘n_estimators’: 100	0.8753	0.8718	0.8806
ExtraTreesClassifier	‘max_depth’: 7, ‘min_samples_leaf’: 6, ‘n_estimators’: 100	0.8772	0.8619	0.8977
SVM	‘C’: 1	0.9258	0.9636	0.8936

Code Block 7 below shows how to create these stacked features. Figure 8 is the resulting dataset from running the code.下方的代码块 7 展示了如何创建这些堆叠特征。图 8 是运行代码后产生的数据集。

Code Block 7. Create a train and test set using the scores of the models trained代码块 7. 使用训练好的模型分数创建训练集和测试集

X_train_scores = query_by_committee.stacking_models(X_train, committee)
X_test_scores = query_by_committee.stacking_models(X_test, committee)

X_train_scores[[
    "Logistic Regression_score",
    "Random Forest_score",
    "Extra Trees_score",
    "SVM_score"
]].head()

Training the stacking layer训练堆叠层

Code Block 8. Train different models on top of the scores of the previous models trained.代码块 8. 在之前训练的模型分数之上训练不同的模型。

models_stacking = query_by_committee.train_qbc_committee(
    X_train_scores[[
        "Logistic Regression_score",
        "Random Forest_score",
        "Extra Trees_score",
        "SVM_score"
    ]],
    y_train,
    cv_folds=5
)

stacked_results = model_evaluation.evaluate_model(
    models_stacking["Logistic Regression"],
    X_test_scores[[
        "Logistic Regression_score",
        "Random Forest_score",
        "Extra Trees_score",
        "SVM_score"
    ]],
    y_test,
    threshold=0.50
)

Why stacking helps为什么堆叠有帮助

After stacking has been performed, increases in overall performance are common. One model might have high accuracy in one region of the data and another might excel elsewhere. One tree might capture nonlinear regions and an SVM may have been able to classify difficult boundaries. By stacking these models, you both utilize each of their strengths and limit the weaknesses of each individual learner.执行堆叠后，整体性能通常会提高。一个模型可能在数据的一个区域具有高准确率，而另一个模型可能在其他地方表现出色。一棵树可能捕捉到非线性区域，而 SVM 可能能够对困难的边界进行分类。通过堆叠这些模型，你既利用了它们各自的优势，又限制了每个个体学习者的弱点。

Table 2.  Model Performance including model stacking表 2. 包含模型堆叠的模型性能

Model	Best parameters	F1-Score	Precision	Recall
LogisticRegression	‘C’: 0.001	0.7688	0.6975	0.8564
RandomForestClassifier	‘max_depth’: 7, ‘min_samples_leaf’: 4, ‘n_estimators’: 100	0.8753	0.8718	0.8806
ExtraTreesClassifier	‘max_depth’: 7, ‘min_samples_leaf’: 6, ‘n_estimators’: 100	0.8772	0.8619	0.8977
SVM	‘C’: 1	0.9258	0.9636	0.8936
Stacking the models using LogisticRegression	‘C’: 0.1	0.9360	0.9223	0.9500

Comparing metrics before and after stacking usually show that your stacked model has higher recall with only a slight loss in precision. For many applications, this tradeoff is worthwhile if it means your model rarely misses positive samples.比较堆叠前后的指标通常表明，你的堆叠模型具有更高的召回率，而精度仅有轻微损失。对于许多应用程序，如果这意味着你的模型很少错过正样本，那么这种权衡是值得的。

Taken together, stacking allows you to create a balanced, robust learner that profits from diversity in your committee and improves your active learning loop.总而言之，堆叠允许你创建一个平衡、稳健的学习者，它得益于委员会的多样性并改进了你的主动学习循环。

2.3.3. Selecting Samples Where the Committee Disagrees2.3.3. 选择委员会意见不一致的样本

After you have trained your committee, you can begin to evaluate how much the models in your committee disagree on each unlabeled sample’s prediction. The reasoning behind this method is fairly straightforward. If several models that typically do not agree happen to make the same prediction for a sample, that sample is unlikely to contain much information. However, if the models do disagree, that indicates that the sample probably exists in a region that the committee finds uncertain or unclear.训练完委员会后，你可以开始评估委员会中的模型对每个未标注样本的预测有多大分歧。这种方法背后的推理相当直接。如果几个通常不一致的模型碰巧对一个样本做出了相同的预测，那么该样本不太可能包含太多信息。然而，如果模型确实存在分歧，则表明该样本可能存在于委员会认为不确定或不清楚的区域。

One way to measure this is to calculate the variance of the predicted probabilities between models in the committee. Higher variance indicates more disagreement, which by extension means those samples are likely very useful if annotated.衡量这一点的一种方法是计算委员会中模型之间预测概率的方差。较高的方差意味着更多的分歧，推而广之，这意味着如果进行标注，这些样本很可能非常有用。

Code Block 9 includes an example function for computing these disagreement scores.代码块 9 包含一个用于计算这些分歧分数的示例函数。

Code Block 9. Function to create the disagreement_scores column代码块 9. 创建 disagreement_scores 列的函数

def select_qbc_samples(X_unlabeled, models):
    """
    Computes prediction disagreement for each unlabeled point
    based on the probability outputs from all committee models.
    """
    X_scores = X_unlabeled.copy()
    base_features = X_unlabeled.iloc[:, :models["Logistic Regression"].n_features_in_]

    preds_matrix = np.zeros((base_features.shape[0], len(models)))

    for i, (name, model) in enumerate(models.items()):
        score_column = f"{name}_score"
        X_scores[score_column] = model.predict_proba(base_features)[:, 1]
        preds_matrix[:, i] = X_scores[score_column]

    # Variance across committee predictions
    X_scores["disagreement_score"] = np.var(preds_matrix, axis=1)
    return X_scores

Once you have created these disagreement scores, you can sort your unlabeled data by this metric and select the top samples for labeling, shown in Code Block 10 and Figure 9.一旦创建了这些分歧分数，你就可以按此指标对未标注数据进行排序，并选择前几个样本进行标注，如代码块 10 和图 9 所示。

Code Block 10. Select the samples with the largest disagreement_score value代码块 10. 选择 disagreement_score 值最大的样本

X_unlabeled = query_by_committee.select_qbc_samples(X_unlabeled, models)

num_samples = 60
most_uncertain = X_unlabeled.nlargest(num_samples, "disagreement_score")
most_uncertain.head()

Notice that this final batch of points should contain the samples where your committee members strongly disagreed with each other. These points should cover the areas of the decision boundary that were causing the largest disagreements between your models. By identifying these samples and labeling them, your model will be able to improve on some of these boundary regions when it is retrained.请注意，这最后一批点应该包含你的委员会成员之间强烈分歧的样本。这些点应该覆盖导致模型之间最大分歧的决策边界区域。通过识别这些样本并对其进行标注，你的模型在重新训练时将能够改进其中一些边界区域。

Query by Committee in a nutshell委员会查询简述

• Uses model disagreement as a measure for identifying informative samples.使用模型分歧作为识别信息量大样本的度量。
• Works best when different models learn different things from the data.当不同的模型从数据中学到不同的东西时效果最好。
• Takes time and resources to train and maintain multiple models.训练和维护多个模型需要时间和资源。

3. Pros and Cons of Active Learning3. 主动学习的优缺点

Active learning can become a vital component of your machine learning workflow by allowing you to shift your annotation efforts away from everything to only what matters. Like any technology there are numerous advantages and some real-world constraints. The sections below outline each in brief so you can determine if active learning will suit your use case.主动学习可以通过让你将标注工作从“全部标注”转移到“仅标注重要部分”，从而成为你机器学习工作流程中至关重要的一部分。像任何技术一样，它既有许多优点，也有一些现实世界的约束。以下部分简要概述了每一项，以便你确定主动学习是否适合你的用例。

3.1. Benefits of Active Learning3.1. 主动学习的优势

3.1.1. Reduce Annotation Costs3.1.1. 降低标注成本

For most problems labeling data is one of the largest investments of time and money. Active learning alleviates this pain point by allowing you to focus your annotation efforts only on samples that provide the most new information to your model. Label only these points and your team can achieve robust model performance without having to allocate resources to annotate at scale.对于大多数问题，标注数据是时间成本和金钱成本最大的投入之一。主动学习通过让你将标注工作集中在仅为模型提供最多新信息的样本上，缓解了这一痛点。只需标注这些点，你的团队就可以实现稳健的模型性能，而不必分配资源进行大规模标注。

3.1.2. Fewer Labels for High-Quality Models3.1.2. 高质量模型所需的标签更少

Active learning attempts to query labels for the most difficult/informative examples. This means that every new label you add to your training set provides more value than if it were randomly sampled. You should expect to see much faster gains in accuracy/recall/F1 with an active learning framework, even with a small labeled dataset.主动学习尝试查询最困难/信息量最大的示例的标签。这意味着你添加到训练集中的每个新标签都比随机采样提供的价值更高。即使在标注数据集很小的情况下，你也应该期望在主动学习框架下看到准确率/召回率/F1 值的更快速增长。

3.1.3. Ideal If Labeled Data Is Limited3.1.3. 标注数据有限时的理想选择

Companies and research teams are often faced with problems where labeled data will always be limited. Active learning allows you to create a framework for iteratively expanding that labeled dataset. It’s used commonly in any machine learning system that deals with perception, recommendations, anomaly detection, or other user-generated signals.公司和研究团队经常面临标注数据总是有限的问题。主动学习允许你创建一个迭代扩展该标注数据集的框架。它通常用于任何处理感知、推荐、异常检测或其他用户生成信号的机器学习系统。

3.1.4. Works With Any ML Problem3.1.4. 适用于任何 ML 问题

There is no one model or type of data that active learning has worked with. Customers have used it for fraud detection, NLP problems, image classification, time-series modeling, and more. If you’re looking for a way to leverage unlabeled data that would otherwise go to waste consider using active learning.没有哪种模型或类型的数据是不适用主动学习的。客户已将其用于欺诈检测、NLP 问题、图像分类、时间序列建模等。如果你正在寻找一种利用否则会被浪费的未标注数据的方法，请考虑使用主动学习。

3.2. Limitations of Active Learning3.2. 主动学习的局限性

3.2.1. Sampling Strategy Is Crucial3.2.1. 采样策略至关重要

Active learning is only as good as the samples you use to train your model. If your sampling strategy consistently returns noisy, deceptive, or uninformative samples your model may never converge. There is significant domain expertise required to pick an appropriate strategy. Once chosen, you’ll need to experiment to ensure that it works well in practice.主动学习的效果取决于你用来训练模型的样本。如果你的采样策略持续返回嘈杂、误导或无信息的样本，你的模型可能永远无法收敛。选择合适的策略需要丰富的领域专业知识。一旦选定，你需要进行实验以确保它在实践中运行良好。

3.2.2. Sampling Bias Is Possible3.2.2. 可能存在采样偏差

Uncertainty-based strategies tend to focus too intensely on smaller regions of your feature-space. Without intervention, your training-set will become unbalanced, limiting your ability to generalize. Uncertainty sampling should be paired with some sort of diversity checking to avoid this pitfall.基于不确定性的策略倾向于过度关注特征空间中较小的区域。如果没有干预，你的训练集将变得不平衡，从而限制你的泛化能力。不确定性采样应与某种多样性检查相结合，以避免这一陷阱。

3.2.3. But we still need human experts3.2.3. 我们仍然需要人类专家

One point to note about active learning is that although sample selection is automated, you still need humans in the loop to perform labeling. There are still some domains that are more dependent on human annotation like cybersecurity, clinical annotation, legal text review or certain vision tasks.关于主动学习需要注意的一点是，虽然样本选择是自动化的，但你仍然需要人类参与来执行标注。在网络安全、临床标注、法律文本审查或某些视觉任务等领域，仍然更依赖人工标注。

3.3. How to choose the right Strategy in Practice3.3. 如何在实践中选择正确的策略

• You can start with uncertainty sampling, specially if you have a small labeled dataset and/or you need early wins.你可以从不确定性采样开始，特别是如果你有一个小的标注数据集和/或需要尽早获得初步成果时。
• Then add diversity sampling if you’ve taken already everything from uncertainty sampling and it’s started to not bring much better results.然后，如果你已经从不确定性采样中获得了所有收益，并且它开始不再带来更好的结果，那么可以添加多样性采样。
• Use query by committee if decision boundaries are expected to be noisy OR you already have a committee of reasonable models you’d like to exploit.如果决策边界预计是嘈杂的，或者你已经拥有一个想要利用的合理模型委员会，请使用委员会查询。

4. Conclusion4. 结论

Active learning is straightforward in concept. Rather than labeling everything in your dataset, you label only the samples that your model needs to learn. When you’re operating in organizations where labeled samples are costly or time-consuming to produce, this difference has a significant impact. You’ll see your model improve at a faster rate, and your team only spends time labeling samples that have been vetted by your active learning loop.主动学习的概念很简单。与其标注数据集中的所有内容，不如只标注模型需要学习的样本。当你所在的组织中，标注样本的成本很高或耗时较长时，这种差异会产生重大影响。你会看到模型以更快的速度改进，而且你的团队只会花费时间标注那些经过主动学习循环审查过的样本。

As with all design decisions, there are trade-offs. The loop introduces additional compute, and your results will change a lot based on your query strategy. Similarly prepared datasets will end-up with very different models if your query strategy produces redundant samples.与所有设计决策一样，这里存在权衡。循环引入了额外的计算，并且你的结果会根据你的查询策略发生很大变化。如果你的查询策略产生冗余样本，那么准备好的类似数据集最终会得到非常不同的模型。

Hence, it’s good practice to mix strategies. Uncertainty sampling works well at the start of a project to learn quickly, while diversity sampling helps you cover more of your feature space. Committee-based sampling allows you to focus on difficult-to-boundary samples.因此，混合使用策略是一种很好的做法。不确定性采样在项目开始时效果很好，可以快速学习，而多样性采样可以帮助你覆盖更多的特征空间。基于委员会的采样允许你专注于边界困难的样本。

Combining them allows you to develop a more robust solution. Concluding, Active Learning helps you prioritize what samples to request human annotation, and it’s specially useful when you have many unlabeled samples and limited annotation resources.将它们结合起来可以让你开发出更稳健的解决方案。总之，主动学习可以帮助你优先考虑请求哪些样本进行人工标注，这在你有大量未标注样本且标注资源有限时特别有用。

Image Note: All images in this article were created by the author unless otherwise stated.图片说明：除非另有说明，本文中的所有图片均由作者创作。