Modernizing financial analytics with Amazon SageMaker Unified Studio使用 Amazon SageMaker Unified Studio 现代化金融分析

Avanse Financial Services is one of India’s leading education loan providers. Their Data Engineering Team had built a data lake on AWS using Amazon Simple Storage Service (Amazon S3), Amazon Athena, and AWS Glue for data ingestion and processing. However, their analytics and reporting layer ran on an external analytics application that wasn’t integrated with AWS. Data had to be copied from Amazon S3 into this external application before analysts could run any report, its license consumed a significant portion of their budget despite low utilization, and every integration with AWS services required custom-built pipelines.Avanse 金融服务是印度领先的教育贷款提供商之一。他们的数据工程团队使用 Amazon Simple Storage Service（Amazon S3）、Amazon Athena 和 AWS Glue 构建了基于 AWS 的数据湖，用于数据摄取和处理。然而，他们的分析和报告层运行在一个未与 AWS 集成的外部分析应用上。数据必须先从 Amazon S3 复制到该外部应用，分析师才能生成报告；尽管使用率低，该应用的许可证费用仍占用了大量预算；并且每一次与 AWS 服务的集成都需要定制的管道。

After evaluating their options, Avanse migrated to a cloud-native lakehouse architecture using Amazon SageMaker Unified Studio, which unified their data engineering, analytics, and artificial intelligence (AI) workflows in a single governed environment on AWS. In this post, we walk through their migration journey so you can adapt their approach to your own environment.在评估了各种方案后，Avanse 将系统迁移到使用 Amazon SageMaker Unified Studio 的云原生湖仓架构，在 AWS 上实现了数据工程、分析和人工智能（AI）工作流的统一治理环境。本文将详细介绍他们的迁移历程，帮助您将其方法应用到自己的环境中。

Why Avanse chose to modernizeAvanse 选择现代化的原因

The separation between their AWS data lake and their external analytics application created five problems:他们的 AWS 数据湖与外部分析应用之间的分离导致了五个问题：

1. Daily data synchronization bottleneck. Every report required a 4-hour batch copy from Amazon S3 into the external analytics application before analysts could query it. Business decisions were based on data that was at least a day old.每日数据同步瓶颈。每份报告都需要将 Amazon S3 中的数据批量复制 4 小时到外部分析应用后，分析师才能查询。业务决策基于至少一天前的数据。
2. Fixed licensing costs disconnected from usage. The external analytics application charged an annual fee regardless of how many queries analysts ran. Avanse needed usage-based pricing that matched what they actually consumed, not a fixed fee for capacity they weren’t using.固定许可证费用与使用量脱钩。外部分析应用无论分析师执行多少查询，都收取年度费用。Avanse 需要基于实际使用量的计费模式，而不是为未使用的容量支付固定费用。
3. Limited auditability. The external analytics application ran on a shared server where different business units (risk, collections, portfolio management) shared the same resources. It lacked granular audit trails, making it difficult to trace who accessed what data and when, or to allocate costs per team.审计能力受限。外部分析应用运行在共享服务器上，不同业务单元（风险、催收、投资组合管理）共用资源，缺乏细粒度审计日志，难以追踪谁在何时访问了哪些数据，也难以按团队分配成本。
4. No centralized data discovery. Although AWS Glue Data Catalog managed schema metadata for the data lake, the external analytics application couldn’t access it. Analysts working in that application relied on folder structures and manual documentation to find the right datasets, slowing onboarding and increasing the risk of using outdated data.缺乏集中式数据发现。虽然 AWS Glue Data Catalog 管理了数据湖的模式元数据，但外部分析应用无法访问。使用该应用的分析师只能依赖文件夹结构和手工文档来寻找数据集，导致新员工上手慢且使用过时数据的风险增加。
5. Disconnected from AWS services. The external analytics application couldn’t query data in Amazon S3 or use AWS Glue catalogs natively. Every data flow required connectors and custom-built pipelines, adding maintenance overhead.与 AWS 服务脱节。外部分析应用无法原生查询 Amazon S3 中的数据或使用 AWS Glue 目录。每一次数据流都需要连接器和定制管道，增加了维护负担。

Additionally, some datasets were stored on Network File System (NFS) storage outside of Amazon S3, creating another data silo that needed to be consolidated.此外，一些数据集存放在 Amazon S3 之外的网络文件系统（NFS）存储中，形成了另一个需要合并的数据孤岛。

Avanse chose Amazon SageMaker Unified Studio because it addressed all five challenges: direct querying of data in Amazon S3 avoiding synchronization, usage-based compute through Amazon Athena and Amazon EMR Serverless, project-based isolation with per-project billing, lineage tracking with AWS IAM Identity Center, and native integration with their existing AWS services.Avanse 选择 Amazon SageMaker Unified Studio，因为它解决了上述全部五个挑战：直接查询 Amazon S3 中的数据，避免同步；通过 Amazon Athena 和 Amazon EMR Serverless 实现基于使用量的计算；项目级隔离并支持按项目计费；通过 AWS IAM Identity Center 实现血缘追踪；以及与现有 AWS 服务的原生集成。

Solution overview解决方案概览

The core architectural change was moving from a two-application model to a single integrated stack:核心架构变更是从双应用模型转向单一集成堆栈：

Previous architecture

Avanse’s data ingestion and processing ran on AWS (Amazon S3, AWS Glue, Athena), but analytics and reporting ran on an external analytics application. Data had to be batch-copied from Amazon S3 into this external application daily before analysts could query it. Each system had its own access controls, and there was no shared catalog or lineage tracking between them.Avanse 的数据摄取和处理在 AWS 上运行（Amazon S3、AWS Glue、Athena），但分析和报告仍在外部分析应用上。数据必须每天批量复制到该外部应用后，分析师才能查询。每个系统都有自己的访问控制，且两者之间没有共享目录或血缘追踪。

New architecture

Analytics now run directly against data in Amazon S3 through Amazon SageMaker Unified Studio. There’s no data copy step. Analysts query the same data that the ingestion pipelines produce, using Athena for SQL and EMR Serverless for large-scale processing. Governance, access control, and lineage are centralized through IAM Identity Center and SageMaker Catalog.现在，分析直接在 Amazon S3 中通过 Amazon SageMaker Unified Studio 进行，无需数据复制步骤。分析师查询摄取管道生成的同一数据，使用 Athena 进行 SQL 查询，使用 EMR Serverless 进行大规模处理。治理、访问控制和血缘通过 IAM Identity Center 和 SageMaker Catalog 实现集中化。

The following diagram illustrates the target architecture. It follows a lakehouse pattern, storing data in open formats on Amazon S3 while maintaining ACID transaction support for the consistency financial regulators expect.下图展示了目标架构。它遵循湖仓模式，将数据以开放格式存储在 Amazon S3 上，同时保持 ACID 事务支持，以满足金融监管机构对一致性的要求。

The architecture has three layers:该架构包含三层：

1. Data layer – Amazon S3 stores data in open formats (Parquet, Delta Lake) with S3 Intelligent-Tiering for automatic cost optimization. AWS Glue Data Catalog maintains schema metadata, making data discoverable across tools.数据层 – Amazon S3 以开放格式（Parquet、Delta Lake）存储数据，使用 S3 Intelligent‑Tiering 自动进行成本优化。AWS Glue Data Catalog 维护模式元数据，使数据在各种工具间可被发现。
2. Compute layer – Amazon SageMaker Unified Studio provides project-based workspaces organized by business function. Collections uses the built-in SQL Query Editor powered by Athena, Risk Reporting uses JupyterLab for interactive analysis, and MIS runs large-scale Spark jobs through Amazon EMR Serverless. AWS Glue ETL handles data transformations and AWS Lambda provides event-driven triggers for report generation. For machine learning (ML) workloads, Amazon SageMaker AI supports model training and deployment, with Amazon Bedrock available for generative AI capabilities such as enhancing risk narratives.计算层 – Amazon SageMaker Unified Studio 提供基于项目的工作空间，按业务功能组织。Collections 使用内置的 Athena 驱动 SQL 查询编辑器，Risk Reporting 使用 JupyterLab 进行交互式分析，MIS 通过 Amazon EMR Serverless 运行大规模 Spark 作业。AWS Glue ETL 负责数据转换，AWS Lambda 提供事件驱动的报告生成触发。机器学习工作负载使用 Amazon SageMaker AI 进行模型训练和部署，Amazon Bedrock 可用于生成式 AI（如增强风险叙述）。
3. Governance layer – IAM Identity Center provides SSO and audit logging across workspaces. SageMaker Catalog serves as the business glossary with data lineage tracking and access controls. Amazon DataZone connects components through a common metadata layer.治理层 – IAM Identity Center 提供跨工作空间的单点登录和审计日志。SageMaker Catalog 充当业务词汇表，提供数据血缘追踪和访问控制。Amazon DataZone 通过统一的元数据层连接各组件。

Migration journey迁移历程

Avanse followed a five-phase approach. The timelines can be adapted to your environment, but the systematic progression from validation through production deployment is key.Avanse 采用了五阶段方法。时间表可根据您的环境进行调整，但从验证到生产部署的系统化推进是关键。

Phase 1: Technical validation (72-hour workshop)阶段 1：技术验证（72 小时研讨会）

Avanse started with a focused 72-hour workshop using isolated SageMaker environments where developers could experiment without impacting production. Their team tested SQL analytics against existing Athena tables and validated that Python and PySpark could replicate their existing analytics workflows.Avanse 从一次为期 72 小时的研讨会开始，使用隔离的 SageMaker 环境，让开发者在不影响生产的情况下进行实验。团队在现有 Athena 表上测试了 SQL 分析，并验证了 Python 和 PySpark 能够复现其现有分析工作流。

The team confirmed that querying data directly in Amazon S3 addressed their synchronization bottleneck entirely. The 4-hour daily data copy was no longer necessary, which validated the migration approach.团队确认，直接在 Amazon S3 中查询数据完全消除了同步瓶颈。每日 4 小时的数据复制不再需要，从而验证了迁移方案的可行性。

Phase 2: Data migration and storage optimization阶段 2：数据迁移与存储优化

Avanse migrated datasets from NFS storage and legacy analytics formats into Amazon S3, consolidating the data into a single location. They implemented S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns, optimizing costs without impacting retrieval performance.Avanse 将数据集从 NFS 存储和传统分析格式迁移至 Amazon S3，完成数据的单点集中。他们启用了 S3 Intelligent‑Tiering，根据使用模式自动在访问层之间移动数据，在不影响检索性能的前提下降低成本。

They replaced legacy analytics connectors with native Athena workgroups within SageMaker Unified Studio, avoiding data synchronization entirely. Source data remained in Amazon S3, queryable by both Athena SQL and SageMaker notebooks, establishing a single source of truth.他们用 SageMaker Unified Studio 中原生的 Athena 工作组取代了传统的分析连接器，彻底避免了数据同步。源数据仍保留在 Amazon S3 中，既可供 Athena SQL 查询，也可供 SageMaker 笔记本使用，实现单一真相来源。

Phase 3: Compute modernization阶段 3：计算现代化

Avanse moved from a shared analytics server to project-based isolation in SageMaker Unified Studio. Each business function (Risk Reporting, Collections, MIS) received its own project with dedicated compute spaces running JupyterLab. Project-specific IAM execution roles provided access controls and cost allocation per business unit.Avanse 将共享分析服务器迁移至 SageMaker Unified Studio 的项目级隔离模式。每个业务功能（风险报告、催收、MIS）拥有自己的项目和专用计算空间，运行 JupyterLab。项目专属的 IAM 执行角色提供访问控制并实现按业务单元的成本分配。

A single browser-based URL with multi-factor authentication (MFA) now provides access to SQL analytics using the built-in query editor, ML development in JupyterLab notebooks, and big data processing through Amazon EMR Serverless. This replaced the need for local analytics client installations.单一的基于浏览器的 URL 加上多因素认证（MFA），即可访问内置查询编辑器进行 SQL 分析、使用 JupyterLab 笔记本进行机器学习开发，以及通过 Amazon EMR Serverless 进行大数据处理，取代了本地分析客户端的需求。

Phase 4: Governance implementation阶段 4：治理实施

Avanse deployed SageMaker Catalog as their central business data catalog. Analysts now discover approved datasets through semantic search rather than navigating folder structures or relying on manual documentation. They mapped technical Athena table names to business terms. For example, analysts search for “collection efficiency” and find the relevant tables with descriptions, schemas, and lineage.Avanse 部署了 SageMaker Catalog 作为中心业务数据目录。分析师现在通过语义搜索发现已批准的数据集，而不必浏览文件夹或依赖手工文档。他们将技术 Athena 表名映射为业务术语。例如，分析师搜索“collection efficiency”即可找到带有描述、模式和血缘信息的相关表。

Lineage capture traces each metric in risk reports back to source tables, transformations, and intermediate datasets. Every action (notebook execution, SQL query, data access) is tied to IAM Identity Center users, creating the comprehensive audit trail their compliance team needed.血缘捕获将风险报告中的每个指标追溯到源表、转换过程和中间数据集。每一次操作（笔记本执行、SQL 查询、数据访问）都关联到 IAM Identity Center 用户，形成合规团队所需的完整审计日志。

Phase 5: Use case migration阶段 5：用例迁移

Rather than attempting a big-bang migration, Avanse moved critical workflows one at a time:Avanse 采用逐个迁移关键工作流的方式，而非一次性大迁移：

Portfolio MIS (Monthly/Fortnightly)

Previously required the daily 4-hour data copy from Amazon S3 into the external analytics application before report generation could begin. Avanse avoided the data synchronization step entirely and now generates MIS reports by querying existing Athena tables directly in Amazon S3. Because the source data was already on AWS, there was no need to involve the external application for this activity. Report generation dropped from hours to under 30 minutes.之前必须先将 Amazon S3 中的数据每日批量复制 4 小时到外部分析应用，才能开始生成报告。Avanse 完全省去了该同步步骤，现直接在 Amazon S3 中查询 Athena 表生成 MIS 报告。由于源数据已在 AWS 上，无需再调用外部应用，报告生成时间从数小时降至 30 分钟以内。

Collection Efficiency and Bounce Calculation

Ported complex legacy analytics procedures for calculating metrics like collection efficiency and bounce rates to event-driven processing using AWS Glue ETL, AWS Lambda, and PySpark jobs for high-volume data aggregation. The serverless execution model charges only for compute time consumed.将复杂的传统分析过程（如计算催收效率和跳出率）迁移至基于事件的处理，使用 AWS Glue ETL、AWS Lambda 和 PySpark 作业进行大规模数据聚合。无服务器执行模型仅对实际消耗的计算时间计费。

EDW Risk Reporting

Large-scale regulatory joins of Enterprise Data Warehouse assets previously ran as legacy scheduled procedures. These now run as SQL queries in the SageMaker Unified Studio query editor, where analysts execute them on-demand or schedule them through Athena workgroups. The distributed query engine handles complex multi-table joins spanning millions of rows.以前在企业数据仓库资产上的大规模监管关联作业以传统计划任务方式运行。现在这些作业在 SageMaker Unified Studio 查询编辑器中以 SQL 形式执行，分析师可以按需运行或通过 Athena 工作组调度。分布式查询引擎能够处理跨数百万行的多表关联。

Scorecard Generation

Model building shifted from the external analytics application to SageMaker AI workflows. Data scientists use JupyterLab with Python libraries and deploy models directly to SageMaker endpoints, avoiding data movement between separate environments.模型构建从外部分析应用迁移至 SageMaker AI 工作流。数据科学家使用带有 Python 库的 JupyterLab 开发模型，并直接部署到 SageMaker 端点，避免了在不同环境之间移动数据。

Overcoming technical challenges克服技术挑战

One technical challenge was code migration. Avanse’s analytics code base contained years of accumulated proprietary scripts and procedures. Direct line-by-line translation was not practical. Instead, they took a pragmatic approach: basic data transformations moved to SQL in Athena, complex business logic was rewritten in PySpark for scalability, and statistical procedures were replaced with Python libraries like pandas and scikit-learn. The approach was to focus on what the code accomplishes, then implement it using cloud-native patterns.技术挑战之一是代码迁移。Avanse 的分析代码库中积累了多年的专有脚本和过程，逐行直接迁移不可行。于是他们采取务实方法：将基础数据转换迁移至 Athena SQL，将复杂业务逻辑重写为可伸缩的 PySpark，将统计过程替换为 pandas、scikit‑learn 等 Python 库。核心思路是先明确代码实现的业务目标，再使用云原生模式实现。

The other technical challenge was performance validation. The team needed to confirm that querying data in Amazon S3 would deliver acceptable performance compared to the external analytics application’s in-memory processing. Queries against Parquet-formatted data in Amazon S3 using Athena delivered comparable performance for standard reporting workloads, while avoiding the 4-hour daily data synchronization step entirely. For large-scale regulatory joins spanning millions of rows, Amazon EMR Serverless provided distributed Spark processing that completed in minutes rather than the hours required in the external application.另一个技术挑战是性能验证。团队需要确认在 Amazon S3 上查询数据的性能是否能满足需求。针对 Parquet 格式数据的 Athena 查询在标准报告工作负载下表现与外部应用的内存处理相当，同时彻底消除了每日 4 小时的数据同步。对于跨数百万行的监管大关联，Amazon EMR Serverless 提供的分布式 Spark 处理在几分钟内完成，而外部应用需要数小时。

Key outcomes关键成果

“By adopting SageMaker Unified Studio, we as the Data Team eliminated legacy licensing costs, reduced storage and compute expenses with a serverless, usage-based model, and accelerated our periodic report generation. At the same time, we transformed compliance and collaboration by cutting audit timelines while unifying our teams in a single, efficient data environment.” – Komal Thakkar, AVP – Lead, Data Engineering, Avanse Financial Services“通过采用 SageMaker Unified Studio，我们数据团队消除了传统许可证费用，采用无服务器、按使用量计费的模型降低了存储和计算成本，并加速了周期性报告的生成。同时，通过缩短审计周期并统一数据环境，提升了合规性和协作效率。”——Komal Thakkar，AVP – 数据工程负责人，Avanse Financial Services

Best practices最佳实践

Based on their experience, Avanse recommends:基于他们的经验，Avanse 建议：

• Start with a workshop. Validate your specific use cases in a 72-hour technical validation before committing to full migration.先开展研讨会。在投入完整迁移前，先在 72 小时技术验证中验证您的具体用例。
• Migrate use cases, not code. Focus on what your analytics accomplish, then implement using cloud-native patterns rather than translating legacy scripts line by line.迁移用例，而非迁移代码。先明确分析要实现的业务目标，再使用云原生模式实现，而不是逐行翻译遗留脚本。
• Invest in governance early. Implement the data catalog and lineage tracking from day one.尽早投入治理。第一天就实现数据目录和血缘追踪。
• Embrace project-based isolation. Organize around business functions for clear cost allocation and security boundaries.采用项目级隔离。围绕业务功能组织，以实现清晰的成本分配和安全边界。
• Document business logic. Use migration as an opportunity to capture undocumented knowledge in the business glossary and dataset descriptions.记录业务逻辑。利用迁移契机将未记录的业务知识捕获到业务词汇表和数据集描述中。

Conclusion结论

Avanse’s migration from an external analytics application to Amazon SageMaker Unified Studio consolidated their analytics stack into a single integrated environment on AWS. By querying data directly in Amazon S3 instead of copying it into the external application, they alleviated their biggest operational bottleneck. Project-based isolation replaced a shared server model, giving each business unit independent compute and clear cost visibility. And centralized governance through SageMaker Catalog and IAM Identity Center gave their compliance team the audit trails they had been missing.Avanse 将外部分析应用迁移至 Amazon SageMaker Unified Studio， 将分析堆栈整合为 AWS 上的单一环境。通过直接在 Amazon S3 中查询数据，消除了最大的运营瓶颈。项目级隔离取代了共享服务器模型，为每个业务单元提供独立计算和清晰成本可视化。通过 SageMaker Catalog 与 IAM Identity Center 实现的集中治理，为合规团队提供了此前缺失的审计追踪。

The serverless, usage-based model means Avanse no longer pays for idle capacity. The lakehouse architecture supports new analytics patterns as they emerge, and native integration with AWS services, including generative AI through Amazon Bedrock, positions them to adopt new capabilities as their needs evolve.无服务器、按使用量计费的模型意味着 Avanse 不再为闲置容量付费。湖仓架构支持新兴的分析模式，并且与 AWS 服务的原生集成（包括通过 Amazon Bedrock 的生成式 AI）使其能够随业务需求演进而快速采用新能力。

Next steps后续步骤

Start your analytics modernization journey by scheduling a 72-hour technical validation workshop. Contact your AWS account team to discuss your migration approach.通过安排 72 小时技术验证研讨会，开启您的分析现代化之旅。请联系您的 AWS 客户经理，讨论迁移方案。

For more information, see:欲了解更多信息，请参阅：

• Unify data, analytics, and AI with Amazon SageMaker Unified Studio使用 Amazon SageMaker Unified Studio 统一数据、分析和 AI
• AWS Analytics Reference ArchitectureAWS 分析参考架构
• Build a lake house architecture on AWS在 AWS 上构建湖仓架构
• What is Amazon DataZone?什么是 Amazon DataZone？