Fuzzy Matching in Snowflake: A Non-Exhaustive GuideSnowflake 模糊匹配之道:一份精要指南
Aaron MendelsonAaron Mendelson
Introduction引言
If you’ve ever wrangled data, you know how painful it can be to match messy records with typos and non-standard spellings to a canonical set of “golden names.” Typically, it’s a labor-intensive process requiring multiple platforms and methods.凡与数据打过交道的人,想必都尝过其中的苦头:将那些错字连篇、写法各异的杂乱记录,归拢到一套标准的“金名录”下,实非易事。过往处理此类繁杂琐事,往往需动用多方平台,费时费力,极是磨人。
In the following article, I will show how you can build a complete fuzzy matching pipeline inside Snowflake that prioritizes speed and cost without sacrificing accuracy. We will build a 4-stage fuzzy matching pipeline: string matching using regular expressions, hybrid search with Cortex Search, algorithmic matching using Jaro-Winkler, and finally an LLM invocation using the Snowflake AI_COMPLETE function.在这篇文章中,我将演示如何在 Snowflake 内部构建一套完整的模糊匹配流水线。此法讲究速与省,亦不失精准。我们将这套流水线拆解为四重关隘:先以正则表达式初筛,再用 Cortex Search 混合检索,继而辅以 Jaro-Winkler 算法精算,最后请出 Snowflake 的 AI_COMPLETE 函数,调用大模型做终极判别。
This article uses synthetic transactions with non-standard credit card names. See this repo for setup scripts if you’d like to follow along.文中演示所用,皆为模拟的信用卡交易数据,其名目多有不规范之处。若欲亲手演练,可参考此仓库中的脚本进行部署。

Overview of Available Methods方法概览
This table is a non-exhaustive list of Snowflake-native functions and features you can use for fuzzy matching.下表罗列了 Snowflake 原生支持的几种模糊匹配手段,虽非全貌,却也涵盖了核心要义。

String Manipulations字符串处理
Regular expressions are SQL-native tools for pattern matching and string manipulation. They have a famously challenging syntax, but are useful for basic data cleaning, especially when there are known patterns to match on. For example, using REGEXP_REPLACE, we can search for any instances of “AMEX “and replace them with “American Express”. By using regular expressions with TRIM and UPPER, we can normalize the strings to facilitate exact string matching.正则表达式乃 SQL 中处理模式匹配与字符变换的利器。其语法虽晦涩,却极擅处理基础清洗,尤其是在应对有迹可循的模式时。例如,借由 REGEXP_REPLACE,我们可将散落在各处的“AMEX”统一规整为“American Express”。再配合 TRIM 与 UPPER 函数,便能将字符洗练得整齐划一,为后续的精确匹配铺平道路。
Character-level algorithms字符级算法
Snowflake makes both Jaro-Winkler and Levenshtein algorithms available to run on standard warehouse compute. Both algorithms are excellent for fuzzy matching use cases with small typos or inconsistent spellings.Snowflake 将 Jaro-Winkler 与 Levenshtein 两大算法直接植入计算引擎。遇上拼写瑕疵或写法不一的琐碎数据,这两招可谓手到擒来。
Levenshtein assesses the “edit distance” between two strings as measured by the number of single-character edits (deletions, insertions, substitutions) needed to turn one string into the other.Levenshtein 算法旨在计算“编辑距离”,即通过增、删、改单个字符,将一个字符串转化为另一个所需的最小代价。
Jaro-Winkler (JW) measures edit distance with a bias for strings with similar beginnings. This was empirically derived based on Winkler’s work matching names for the US Census Bureau — he noticed that typos are more common at the end of names and that most phonetic errors still preserve the initial sound. It also works well for short strings where there isn’t enough semantic meaning for vectors to capture.Jaro-Winkler (JW) 算法则更进一步,它对字符串开头的相似度更为看重。此法源于美国人口普查局的实践,其发现人们在拼写名字时,末尾常有错漏,而开头往往准确。对于语义贫乏的短字符串,此法尤为灵验。
Python libraries like RapidFuzz use these algorithms too, but with Snowflake you can harness the massive parallelism inherent in Snowflake’s compute engine for much better performance.Python 虽有 RapidFuzz 等库可用,但 Snowflake 胜在能调动底层强大的并行计算引擎,其性能远非单机可比。
Vector-based Methods向量化方法
Snowflake also has a number of vector-based methods that can help with fuzzy matching. The basic idea is that you can turn a string into a vector. Similar words, phrases, and concepts all have similar vectors regardless of how similar the strings are. For example, vectors for “heart attack” and “myocardial infarction” are similar even though their strings have nothing in common. Similarly, if we were matching two credit cards, these methods might do better than JW and Levenshtein in understanding that “BofA Gold” is the same as “Bank of America Gold Card.”Snowflake 亦支持向量化方法。其玄妙之处在于将字符串化作向量,即便字面迥异,只要意蕴相近,向量便会靠拢。譬如,“heart attack”与“myocardial infarction”虽字面毫无关联,但在向量空间里却如出一辙。以此类推,匹配信用卡名时,它比 JW 或 Levenshtein 更能洞察“BofA Gold”与“Bank of America Gold Card”实为同物。
But these methods can struggle with short strings where there isn’t enough context for a meaningful vector. They also have trouble handling strings with typos and abbreviations.然则,若字符串过短,缺乏上下文,向量便难以捕捉深意,遇到错别字或缩写时,亦常有力不从心之感。
Cortex Search Batch InferenceCortex Search 批处理推理
Snowflake also has a search service for hybrid semantic and keyword search called Cortex Search Batch Inference. Like other semantic-based methods, it uses vectors to capture meaning. But it also has a lexical search method similar to BM25. These two methods work in tandem so that Cortex Search does well with semantic meaning (like vector-based solutions), and cases where words are reordered, or there are missing or added words. Importantly, Cortex Search scales much better than JW because it doesn’t require an expensive cross join that can be infeasible with very large datasets. However, the lexical portion of Cortex Search can still struggle with typos and short strings.Cortex Search 乃是结合了语义与关键词的混合搜索服务。它既能像向量法那样捕捉语义,又具备类似 BM25 的词法搜索能力。二者相辅相成,既能应对语序颠倒,也能处理词汇增减。尤为可贵的是,它无需进行昂贵的交叉连接(Cross Join),在大规模数据集上亦能从容应对,远胜 JW 算法的开销。不过,其词法部分在应对短字符串与错别字时,仍需谨慎考量。
LLMs大语言模型 (LLMs)
Snowflake hosts leading LLMs within its own security perimeter. These models allow you to bring human-like intelligence and reasoning to evaluate fuzzy matches. Because they’re integrated into the platform via AI_COMPLETE, you can call them as simple SQL functions. Though highly discerning, they are also the most expensive technique, and most customers will want to use them only on the hardest cases.Snowflake 将顶尖大模型置于安全边界之内。借由 AI_COMPLETE 函数,你只需调用 SQL,便能赋予系统如人般的判断力。虽说此法最为“奢侈”,但面对最棘手的模糊匹配,它往往能给出最令人信服的答案。
Building the Pipeline流水线构建
As mentioned above, the data used in this article is synthetic data representing dirty credit card names. The goal is to match each “dirty” name to one of 100 canonical “golden” names. We’ll build a 4-stage pipeline tuned for speed, accuracy, and cost-effectiveness.如前所述,我们将处理一批杂乱的信用卡名,目标是将其归入 100 个标准的“金名录”中。我们将搭建四重流水线,兼顾速度、精度与成本。
Any fuzzy matching pipeline will begin with string matching. Many names will be similar except for capitalization or spacing. We can easily and cheaply match these since they run on traditional Snowflake warehouses that are charged on a per-second basis.第一关是字符串匹配。许多差异仅在于大小写或空格,这些琐事在 Snowflake 的标准计算仓库中处理,既快又省。
Then we’ll use Cortex Search. This is a good second step because it scales much better than JW, so we can filter out more records and send a smaller number to that stage. We’ll set the precision threshold relatively high to catch matches with high semantic similarity.第二关交给 Cortex Search。它扩展性极佳,能过滤掉大部分记录,让后续步骤更显轻盈。我们将阈值设得较高,以确保语义匹配的可靠性。
In the third step, we’ll use JW. This algorithm is very well suited to our dataset, which has credit card names laden with typos and abbreviations (areas where search struggles). Note that most applications of this algorithm require a CROSS JOIN where all records in a dataset are compared to each other, and only the records with the best JW match score are retained. That can get slow and expensive when your datasets are very large. An added benefit of running JW after Cortex Search is that Cortex Search can return the top 5 matches for a specific dirty record, greatly reducing the number of comparisons to run through JW. Like exact string matching, JW runs on traditional Snowflake compute.第三关动用 JW 算法。对于错字与缩写频发的信用卡名,此法最为对症。我们不直接进行全量交叉连接,而是利用 Cortex Search 筛选出的前 5 个候选对象进行比对,既节省了算力,又保证了效率。
Finally, we’ll use AI_COMPLETE with a Claude model to bring human-level discernment to the problem. It is billed per token and is comparatively more expensive than the other methods.最后一关,请出 Claude 模型,以 AI_COMPLETE 赋予系统人类般的审慎与智慧。此法虽按 Token 计费,开销略高,但用来处理“漏网之鱼”最是妥当。
First, I will assume that if you want to follow along, you’ve completed setup.sql. That script creates the database, warehouse, and synthetic dataset you’ll need for the rest of the demo.若要演练,请先运行 setup.sql 脚本,此脚本将为您备好数据库、仓库及模拟数据集。
Funnel Stage 1: Normalization With String Manipulation第一关:字符串规整
This is the easiest and cheapest step. It scales well and will reduce the number of records you’ll have to send to more expensive methods. This code takes both dirty credit card names and golden records and strips out whitespace, standardizes case, strips out filler words like “credit,” and then expands obvious abbreviations (e.g., it turns AMEX into American Express).此关最为轻便。我们通过代码剔除空格、统一大小写、过滤掉“credit”等冗余词汇,并将 AMEX 等缩写还原为全称,以此减少后续步骤的负担。
CREATE OR REPLACE TABLE stage1_string_match AS
WITH
normalized_payments AS (
SELECT
payment_id,
card_name,
TRIM(REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
UPPER(TRIM(card_name)),
'\\s+', ' '
),
'(^|\\s)AMEX($|\\s)', ' AMERICAN EXPRESS '
),
'(^|\\s)MC($|\\s)', ' MASTERCARD '
),
'(^|\\s)DISC($|\\s)', ' DISCOVER '
),
'\\s+CREDIT\\s+CARD\\s*$', ''
),
'\\s+CREDIT\\s*$', ''
),
'\\s+CARD\\s*$', ''
),
'[.,\\-]', ' '
),
'\\s+', ' '
)) AS norm_name
FROM raw_payments
),
normalized_golden AS (
SELECT
card_id,
canonical_card_name,
TRIM(REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
REGEXP_REPLACE(
UPPER(TRIM(canonical_card_name)),
'\\s+', ' '
),
'(^|\\s)AMEX($|\\s)', ' AMERICAN EXPRESS '
),
'(^|\\s)MC($|\\s)', ' MASTERCARD '
),
'(^|\\s)DISC($|\\s)', ' DISCOVER '
),
'\\s+CREDIT\\s+CARD\\s*$', ''
),
'\\s+CREDIT\\s*$', ''
),
'\\s+CARD\\s*$', ''
),
'[.,\\-]', ' '
),
'\\s+', ' '
)) AS norm_name
FROM golden_cards
)
SELECT
p.payment_id,
p.card_name,
g.canonical_card_name AS matched_golden_name,
g.card_id,
1.0 AS match_score,
'EXACT_NORMALIZED' AS match_method
FROM normalized_payments p
JOIN normalized_golden g ON p.norm_name = g.norm_name;Now that you’ve run that code, you’ll want to see how many matches you actually produced.运行代码后,不妨清点一下已匹配的成果。
SELECT
(SELECT COUNT(*) FROM raw_payments) AS total_payments,
(SELECT COUNT(*) FROM stage1_string_match) AS stage1_matched,
ROUND(stage1_matched / total_payments * 100, 1) AS stage1_pct;Then, we’ll create a new table of all records that were unmatched after stage 1.随后,将那些尚未归位的记录收拢,准备进入下一关。
CREATE OR REPLACE TABLE unmatched_after_stage1 AS
SELECT payment_id, card_name
FROM raw_payments
WHERE payment_id NOT IN (SELECT payment_id FROM stage1_string_match);Stage 2: Batch Cortex Search第二关:Cortex Search 批处理
For the next stage in the pipeline, we’ll use a Cortex Search service. When you create a search service, you specify a target column. The service then vectorizes that column and builds a keyword search index on it. One of the benefits of using Cortex Search is that it lets you find fuzzy matches without doing an expensive cross join where each item in a list of dirty names is compared to every possible canonical name.在此阶段,我们创建搜索服务,指定目标列。服务会自动向量化并建立关键词索引。其妙处在于,无需进行全表交叉比对,便能直接定位模糊匹配的目标。
Here’s how you create a search service.创建搜索服务的代码如下。
CREATE OR REPLACE CORTEX SEARCH SERVICE golden_card_search
ON canonical_card_name
ATTRIBUTES card_id, card_network, card_tier
WAREHOUSE = FUZZY_MATCHING_WH
TARGET_LAG = '1 hour'
AS (
SELECT
canonical_card_name,
card_id,
card_network,
card_tier
FROM golden_cards);Once you create the search service, give it a few minutes to warm up before continuing. The service will need some time to build the index of golden records. Then we’ll create a “candidates table” consisting of the top 5 canonical matches for each dirty credit card name.创建完成后,稍候片刻,待索引构建完毕。接着,我们为每条杂乱记录找出前 5 个最可能的标准名候选。
CREATE OR REPLACE TABLE stage2_candidates AS
SELECT
q.payment_id,
q.card_name,
r.canonical_card_name,
r.card_id,
r.card_network,
r.card_tier,
PARSE_JSON(r.METADATA$RESULT_DETAIL):scores:cosine_similarity::FLOAT AS cosine_score,
ROW_NUMBER() OVER (PARTITION BY q.payment_id ORDER BY cosine_score DESC) AS candidate_rank
FROM unmatched_after_stage1 q,
LATERAL CORTEX_SEARCH_BATCH(
service_name => 'FUZZY_MATCHING_DEMO.PUBLIC.GOLDEN_CARD_SEARCH',
query => q.card_name,
limit => 5
) AS r;Next, we’ll filter for the top candidates that also have a cosine score of at least 0.6. This is important because we are setting a relatively high threshold for semantic similarity. That way, we can match records where we have a high confidence in semantic similarity and pass along the rest of the records to parts of the funnel that can handle them better.随后,我们将余弦相似度阈值设为 0.6,只取高置信度的候选。如此一来,既能消化掉大部分匹配,又能将真正难解的记录留给后续关卡。
CREATE OR REPLACE TABLE stage2_cortex_search AS
SELECT
payment_id,
card_name,
canonical_card_name AS matched_golden_name,
card_id,
cosine_score AS match_score,
'CORTEX_SEARCH' AS match_method
FROM stage2_candidates
WHERE candidate_rank = 1
AND cosine_score >= 0.60;And lastly, let’s understand how many more records we matched at this stage.再来看看这一关又消化了多少记录。
SELECT
(SELECT COUNT(*) FROM unmatched_after_stage1) AS stage2_input,
(SELECT COUNT(*) FROM stage2_cortex_search) AS stage2_matched,
ROUND(stage2_matched / NULLIF(stage2_input, 0) * 100, 1) AS stage2_pct;Stage 3: Jaro-Winkler第三关:Jaro-Winkler 算法
Now that we’ve reduced the number of records we need to match, we can move on to Jaro-Winkler. As discussed above, this algorithm is well-suited to the specific fuzzy matching scenario where we are matching short names with lots of typos and abbreviations. One thing to note is that the method typically uses a CROSS JOIN: every remaining dirty record is joined with every golden record, and only matches with a high enough score are kept. Since we already got the top 5 matches for each record via Cortex Search, we’ll avoid the CROSS JOIN and just use JW on those 5 records.经过前两关的洗礼,数据量已大幅缩减。现在轮到 JW 算法出场。我们不再进行全量交叉连接,而是仅对 Cortex Search 筛选出的 5 个候选进行 JW 分数计算,既准又快。
First, we’ll create a new candidate table that consists of all records not matched by Cortex Search. For each dirty card name, we’ll calculate the JW similarity score for the top 5 candidates.首先,为未匹配的记录建立候选表,计算 JW 相似度。
CREATE OR REPLACE TABLE stage3_jw_scored AS
SELECT
c.payment_id,
c.card_name,
c.canonical_card_name,
c.card_id,
c.card_network,
c.card_tier,
c.cosine_score,
JAROWINKLER_SIMILARITY(c.card_name, c.canonical_card_name) / 100.0 AS jw_score,
ROW_NUMBER() OVER (PARTITION BY c.payment_id ORDER BY jw_score DESC) AS jw_rank
FROM stage2_candidates c
WHERE c.payment_id NOT IN (SELECT payment_id FROM stage2_cortex_search);Then we’ll filter for records that had JW scores above 0.75.接着,筛选出分数高于 0.75 的记录。
CREATE OR REPLACE TABLE stage3_jaro_winkler AS
SELECT
payment_id,
card_name,
canonical_card_name AS matched_golden_name,
card_id,
jw_score AS match_score,
'JARO_WINKLER' AS match_method
FROM stage3_jw_scored
WHERE jw_rank = 1
AND jw_score >= 0.75;Lets see how many we matched in this round:看看这一轮的战果如何:
SELECT
(SELECT COUNT(*) FROM unmatched_after_stage1) - (SELECT COUNT(*) FROM stage2_cortex_search) AS stage3_input,
(SELECT COUNT(*) FROM stage3_jaro_winkler) AS stage3_matched,
ROUND(stage3_matched / NULLIF(stage3_input, 0) * 100, 1) AS stage3_pct;Stage 4: AI_COMPLETE第四关:AI_COMPLETE 终极裁决
For the final stage of our funnel, we will run the remaining unmatched credit card names through an LLM via the Cortex AI SQL function AI_COMPLETE. Because Snowflake hosts leading LLMs within its security boundaries, when you call AI_COMPLETE not only does the data never leave Snowflake, but you can also use the best model for the job at hand.最后一关,将残余记录交给大模型。由于 Snowflake 确保数据不出边界,我们大可放心地让模型进行深度推理,给出最终判断。
As before, we’ll create a candidate table for this stage:同样,为本阶段建立候选表:
CREATE OR REPLACE TABLE stage4_candidates AS
SELECT
c.payment_id,
c.card_name,
c.canonical_card_name,
c.card_id,
c.card_network,
c.card_tier,
c.cosine_score,
JAROWINKLER_SIMILARITY(c.card_name, c.canonical_card_name) / 100.0 AS jw_score
FROM stage2_candidates c
WHERE c.payment_id NOT IN (SELECT payment_id FROM stage2_cortex_search)
AND c.payment_id NOT IN (SELECT payment_id FROM stage3_jaro_winkler);Then we’ll create a prompt and pass it to AI_COMPLETE and write the results to a table. We’ll also ask the LLM to assign a confidence score to each record it matches.编写提示词,调用 AI_COMPLETE,并将结果写入表内。我们还要求模型给出置信度评分。
CREATE OR REPLACE TABLE stage4_ai_complete AS
WITH candidate_lists AS (
SELECT
payment_id,
card_name,
LISTAGG(
' - "' || canonical_card_name || '" (id=' || card_id || ', network=' || card_network || ', tier=' || card_tier || ', cosine=' || ROUND(cosine_score, 3) || ', jw=' || ROUND(jw_score, 3) || ')',
'\n'
) WITHIN GROUP (ORDER BY cosine_score DESC) AS candidates_text
FROM stage4_candidates
GROUP BY payment_id, card_name
),
prompts AS (
SELECT
payment_id,
card_name,
'You are a credit card product matching expert. Determine whether the card name from a payment record matches one of the candidate golden card products, or if it is genuinely unknown/unidentifiable.\n\n'
|| 'Card name from payment record:\n "' || card_name || '"\n\n'
|| 'Top candidate golden card products (ranked by search relevance):\n'
|| candidates_text || '\n\n'
|| 'Rules:\n'
|| '- Common abbreviations: CSP=Chase Sapphire Preferred, CSR=Chase Sapphire Reserve, CFU=Chase Freedom Unlimited, CFF=Chase Freedom Flex, BCE=Blue Cash Everyday, BCP=Blue Cash Preferred, VX=Venture X, QS=Quicksilver.\n'
|| '- If the card name clearly refers to a candidate (accounting for typos, abbreviations, informal names, slang like "Black Card"=Centurion, wrong network attribution), set is_unknown=false and return the matched canonical name exactly as shown.\n'
|| '- If none of the candidates are a plausible match (e.g., "Store Credit Card", "PayPal Credit", "Affirm"), set is_unknown=true and set matched_name to your best guess at what the card might be.\n'
|| '- Set confidence between 0.0 and 1.0.\n'
|| '- Respond with JSON only: {"matched_name":"...","confidence":0.9,"is_unknown":false,"reasoning":"..."}'
AS prompt
FROM candidate_lists
),
llm_results AS (
SELECT
payment_id,
card_name,
TRY_PARSE_JSON(AI_COMPLETE('claude-sonnet-4-6', prompt)) AS result
FROM prompts
)
SELECT
payment_id,
card_name,
result:matched_name::VARCHAR AS matched_golden_name,
result:confidence::FLOAT AS confidence,
result:is_unknown::BOOLEAN AS is_unknown,
result:reasoning::VARCHAR AS reasoning,
g.card_id,
'AI_COMPLETE' AS match_method
FROM llm_results lr
LEFT JOIN golden_cards g
ON g.canonical_card_name = lr.result:matched_name::VARCHAR
WHERE result IS NOT NULL;Finally, we’ll assign all of the low-confidence matches to a review queue:最后,将置信度不足的记录打入“人工复核队列”:
CREATE OR REPLACE VIEW review_queue AS
SELECT
payment_id,
card_name,
matched_golden_name AS llm_best_guess,
confidence,
is_unknown,
reasoning,
CASE
WHEN is_unknown THEN 'UNKNOWN CARD — confirm identity'
WHEN confidence < 0.7 THEN 'LOW CONFIDENCE — verify match manually'
ELSE 'UNEXPECTED'
END AS review_reason
FROM stage4_ai_complete
WHERE is_unknown = TRUE
OR confidence < 0.7;Then we’ll pull all of the matches together into one table:将四关战果汇总至一张总表:
CREATE OR REPLACE TABLE final_matched_payments AS
-- Stage 1: Normalized exact matches (score = 1.0)
SELECT payment_id, card_name, matched_golden_name, card_id, match_score::FLOAT AS match_score, match_method
FROM stage1_string_match
UNION ALL
-- Stage 2: Cortex Search matches (cosine >= 0.60)
SELECT payment_id, card_name, matched_golden_name, card_id, match_score, match_method
FROM stage2_cortex_search
UNION ALL
-- Stage 3: Jaro-Winkler matches (JW >= 0.75)
SELECT payment_id, card_name, matched_golden_name, card_id, match_score, match_method
FROM stage3_jaro_winkler
UNION ALL
-- Stage 4: AI_COMPLETE confident matches (not unknown, confidence >= 0.7)
SELECT payment_id, card_name, matched_golden_name, card_id, confidence AS match_score, match_method
FROM stage4_ai_complete
WHERE is_unknown = FALSE
AND confidence >= 0.7;Conclusion结语
In this article, I walked through how to build a complete fuzzy matching pipeline entirely within Snowflake. Through building a funnel that uses string matching, Cortex Search, Jaro-Winkler, and AI_COMPLETE, we matched the vast majority of dirty credit card names to their canonical counterparts.至此,我们已在 Snowflake 内部筑起了一道严密的模糊匹配流水线。从字符串规整、Cortex Search 检索、JW 算法精算,到 AI 大模型裁决,层层递进,终于将那些杂乱无章的信用卡名,尽数归于正途。

