Detecting scraper bots through scroll behaviour通过滚动行为检测爬虫机器人

Ever since I first read "Burstiness and Memory in Complex Systems" by Kwang-Il Goh, I have been obsessed with the two formulas showcased in the paper. Burstiness (B), and its just-as-important counterpart Memory (M), let us understand the dynamics of event-based systems. 自从我第一次读到 Kwang-Il Goh 的《复杂系统中的突发性与记忆》以来，我一直对论文中展示的两个公式着迷。突发性（B）及其同样重要的对应物记忆性（M），让我们能够理解基于事件的系统的动态。

We can use them to analyse the behaviour of sent emails, texts or even heartbeats when only the time at which those events happened is known. This allows us to clearly establish which patterns are human-like and which aren't based on a dataset of already classified data. We know humans reply to texts in a bursty manner (the time they take to answer is not uniform) while simple bots respond as fast as possible, thus they have different B and M coefficients.我们可以用它们来分析已发送电子邮件、短信甚至心跳的行为，只要知道这些事件发生的时间。这使我们能够基于已经分类的数据集，清晰地判断哪些模式像人类，哪些不像。我们知道，人类回复短信的方式具有突发性（回复所需的时间不均匀），而简单的机器人会尽快回复，因此它们具有不同的 B 和 M 系数。

Goh, K.-I., & Barabási, A.-L. (2008), Figure 4Goh, K.-I., & Barabási, A.-L. (2008)，图 4

In previous personal research, I used these two values to differentiate between human and bot sessions based on the timings of their requests, which usually worked, nonetheless, this approach identified sessions as human if they were sent from any browser that loaded CSS and JS files, allowing more advanced bots like ClaudeBot to appear human by using a headless browser. It essentially distinguished between crafted requests versus requests sent from a real browser, instead of bot and human.在之前的个人研究中，我使用这两个值根据请求的时间来区分人类会话和机器人会话，这通常有效。然而，这种方法将会话识别为人类的，只要它们来自于加载了 CSS 和 JS 文件的任何浏览器，这使得像 ClaudeBot 这样的更高级机器人可以通过使用无头浏览器来伪装成人类。它本质上区分的是精心构造的请求与来自真实浏览器的请求，而不是机器人和人类之间的区别。

The scroll wheel鼠标滚轮

Some time ago I noticed a distinctly human behaviour that I believe cannot be easily imitated by bots, scrolling. When I am scrolling a page looking for information using the scroll wheel, I usually don't scroll down linearly until I find what I am looking for, instead, I do so with a bursty pattern. For example, that pattern might more closely resemble time series a or e rather than c.不久前，我注意到一种明显的人类行为，我相信这种行为很难被机器人模仿——滚动。当我使用滚轮在页面上滚动寻找信息时，我通常不会线性地向下滚动直到找到目标，而是以一种突发性的模式进行。例如，这种模式可能更接近时间序列 a 或 e，而不是 c。

Goh, K.-I., & Barabási, A.-L. (2008), Figure 1Goh, K.-I., & Barabási, A.-L. (2008)，图 1

I think most scraping bot developers still haven't honed their scrolling pattern, so I decided to test if burstiness and memory applied to the inter-event times of scroll events were variables that could have predictive power in a machine learning model, in order to distinguish between humans and bots.我认为大多数爬虫机器人开发者还没有完善他们的滚动模式，因此我决定测试一下，如果将突发性和记忆性应用于滚动事件的时间间隔，这些变量是否在机器学习模型中具有预测能力，从而区分人类和机器人。

To do so, I used the dataset provided by the paper "FP-Agent: Fingerprinting AI Browsing Agents" by Ethan Wang, et al., in which the authors explained the process of how they created a machine learning model that could distinguish between different AI browsing agents in a controlled environment (a website made specifically for the purpose). They recorded data of different agents and humans navigating through their website. Their model gave good results, but I don't think it would be effective in a real-world setting when applied to different websites, which involves lots of variation. Nevertheless, they made the dataset available for download on OSF.io.为此，我使用了 Ethan Wang 等人论文《FP-Agent: Fingerprinting AI Browsing Agents》中提供的数据集。作者在论文中解释了如何创建一个机器学习模型，以在受控环境（一个专门为此目的构建的网站）中区分不同的人工智能浏览代理。他们记录了不同代理和人类在其网站上导航的数据。他们的模型结果不错，但我认为在现实环境中应用于不同网站时不会有效，因为这会涉及大量差异。不过，他们将数据集提供下载，位于 OSF.io。

I calculated the burstiness and memory values for the JavaScript "scroll" events of each agent in each page (one burstiness and memory value per page visited). These are the results:我计算了每个代理在每个页面上的 JavaScript “scroll”事件的突发性和记忆性值（每个访问页面得到一个突发性和记忆性值）。结果如下：

We can observe that the human distribution is clearly different from the other agents', presenting a higher Burstiness coefficient and almost no Memory; the only agent that sometimes resembles humans is ChatGPT Agent. This data seems promising, but to further prove if the two values have predictive value I trained a LightGBM (decision tree gradient boosting) model to classify each single-page interaction.我们可以观察到，人类的分布明显区别于其他代理，呈现出更高的突发性系数，几乎没有任何记忆性；唯一有时接近人类的是 ChatGPT Agent。这些数据看起来很有希望，但为了进一步证明这两个值是否具有预测价值，我训练了一个 LightGBM（决策树梯度提升）模型来对每个单页交互进行分类。

The model only had two features available, B and M, and it had to predict which agent performed the interactions. The result was an accuracy of 73.4%, which is not that bad. As expected, the model mainly confused Humans and ChatGPT Agent, most likely due to low feature count or a small dataset (~150 data points per agent is too small), the model would need other features to be able to distinguish them further.该模型只有两个可用特征，B 和 M，它需要预测是哪个代理执行的交互。结果准确率为 73.4%，还不算太差。正如预期的那样，模型主要混淆了人类和 ChatGPT Agent，这很可能是因为特征数量少或数据集较小（每个代理约 150 个数据点太少），模型需要其他特征才能进一步区分它们。

Conclusion结论

Scroll behaviour seems to be a relevant data point for distinguishing bots from humans, and Burstiness and Memory have proven to have predictive power in a controlled environment. This approach will obviously not be effective on websites that don't require scrolling; on the other hand, blogging or information sites like Wikipedia are the ones most able to take advantage of this method.滚动行为似乎是区分机器人和人类的一个相关数据点，突发性和记忆性已被证明在受控环境中具有预测能力。显然，这种方法在不需要滚动的网站上不会有效；另一方面，博客或维基百科等信息类网站最适合利用这种方法。

The model shown in this blog post is not accurate enough, though, if combined with more features extracted from other actions such as mouse movement or typing, I believe it is possible to create a general model capable of protecting websites from scraper bots.不过，这篇博客文章中展示的模型还不够准确。如果结合从其他行为（如鼠标移动或打字）中提取的更多特征，我相信有可能创建一个能够保护网站免受爬虫机器人侵害的通用模型。

I will keep posting my research in this website. Next up, I will be taking a look at mouse movement patterns.我将继续在这个网站上发布我的研究。接下来，我将研究鼠标移动模式。