Modernizing financial analytics with Amazon SageMaker Unified Studio使用 Amazon SageMaker Unified Studio 实现金融分析现代化

Avanse Financial Services is one of India’s leading education loan providers. Their Data Engineering Team had built a data lake on AWS using Amazon Simple Storage Service (Amazon S3), Amazon Athena, and AWS Glue for data ingestion and processing. However, their analytics and reporting layer ran on an external analytics application that wasn’t integrated with AWS. Data had to be copied from Amazon S3 into this external application before analysts could run any report, its license consumed a significant portion of their budget despite low utilization, and every integration with AWS services required custom-built pipelines.Avanse Financial Services 是印度领先的教育贷款提供商之一。其数据工程团队使用 Amazon Simple Storage Service (Amazon S3)、Amazon Athena 和 AWS Glue 在 AWS 上构建了数据湖，用于数据摄取和处理。然而，其分析和报告层运行在一个与 AWS 未集成的外部分析应用程序上。分析师必须先将数据从 Amazon S3 复制到该外部应用程序，才能运行任何报告；该应用程序的许可证尽管利用率低，却消耗了预算的很大一部分，而且每次与 AWS 服务集成都需要构建自定义管道。

After evaluating their options, Avanse migrated to a cloud-native lakehouse architecture using Amazon SageMaker Unified Studio, which unified their data engineering, analytics, and artificial intelligence (AI) workflows in a single governed environment on AWS. In this post, we walk through their migration journey so you can adapt their approach to your own environment.在评估了各种方案后，Avanse 迁移到了使用 Amazon SageMaker Unified Studio 的云原生湖仓架构，在 AWS 上统一的受管环境中整合了数据工程、分析和人工智能 (AI) 工作流。在本文中，我们将介绍他们的迁移历程，以便您能根据自身环境借鉴他们的方法。

Why Avanse chose to modernizeAvanse 为何选择现代化改造

The separation between their AWS data lake and their external analytics application created five problems:他们的 AWS 数据湖与外部分析应用之间的分离带来了五个问题：

1. Daily data synchronization bottleneck. Every report required a 4-hour batch copy from Amazon S3 into the external analytics application before analysts could query it. Business decisions were based on data that was at least a day old.每日数据同步瓶颈。每份报告都需要先从 Amazon S3 批量复制到外部分析应用（耗时 4 小时），分析师才能查询。业务决策基于至少一天前的数据。
2. Fixed licensing costs disconnected from usage. The external analytics application charged an annual fee regardless of how many queries analysts ran. Avanse needed usage-based pricing that matched what they actually consumed, not a fixed fee for capacity they weren’t using.固定许可证费用与使用情况脱节。外部分析应用无论分析师运行多少查询都收取年度费用。Avanse 需要与实际用量匹配的按用量定价，而不是为未使用的容量支付固定费用。
3. Limited auditability. The external analytics application ran on a shared server where different business units (risk, collections, portfolio management) shared the same resources. It lacked granular audit trails, making it difficult to trace who accessed what data and when, or to allocate costs per team.可审计性有限。外部分析应用运行在共享服务器上，不同业务部门（风险、催收、投资组合管理）共享同一资源。它缺乏细粒度的审计跟踪，难以追踪谁在何时访问了什么数据，也难以按团队分摊成本。
4. No centralized data discovery. Although AWS Glue Data Catalog managed schema metadata for the data lake, the external analytics application couldn’t access it. Analysts working in that application relied on folder structures and manual documentation to find the right datasets, slowing onboarding and increasing the risk of using outdated data.缺乏集中的数据发现机制。尽管 AWS Glue Data Catalog 管理着数据湖的 schema 元数据，但外部分析应用无法访问它。在该应用中工作的分析师只能依赖文件夹结构和手工文档来查找数据集，这拖慢了上手速度，也增加了使用过时数据的风险。
5. Disconnected from AWS services. The external analytics application couldn’t query data in Amazon S3 or use AWS Glue catalogs natively. Every data flow required connectors and custom-built pipelines, adding maintenance overhead.与 AWS 服务脱节。外部分析应用无法直接查询 Amazon S3 中的数据，也无法原生使用 AWS Glue 目录。每条数据流都需要连接器和自定义管道，增加了维护成本。

Additionally, some datasets were stored on Network File System (NFS) storage outside of Amazon S3, creating another data silo that needed to be consolidated.此外，部分数据集存储在 Amazon S3 之外的网络文件系统 (NFS) 存储上，形成了另一个需要整合的数据孤岛。

Avanse chose Amazon SageMaker Unified Studio because it addressed all five challenges: direct querying of data in Amazon S3 avoiding synchronization, usage-based compute through Amazon Athena and Amazon EMR Serverless, project-based isolation with per-project billing, lineage tracking with AWS IAM Identity Center, and native integration with their existing AWS services.Avanse 选择 Amazon SageMaker Unified Studio 是因为它解决了全部五个挑战：直接查询 Amazon S3 中的数据，避免同步；通过 Amazon Athena 和 Amazon EMR Serverless 实现按用量计费的计算；通过项目级隔离和按项目计费；通过 AWS IAM Identity Center 实现血缘追踪；以及与现有 AWS 服务的原生集成。

Solution overview解决方案概览

The core architectural change was moving from a two-application model to a single integrated stack:核心架构变化是从双应用模式迁移到单一集成技术栈：

Previous architecture

Avanse’s data ingestion and processing ran on AWS (Amazon S3, AWS Glue, Athena), but analytics and reporting ran on an external analytics application. Data had to be batch-copied from Amazon S3 into this external application daily before analysts could query it. Each system had its own access controls, and there was no shared catalog or lineage tracking between them.Avanse 的数据摄取和处理在 AWS 上运行（Amazon S3、AWS Glue、Athena），但分析和报告运行在外部分析应用中。分析师查询前，数据必须每天从 Amazon S3 批量复制到该外部应用。每个系统都有自己的访问控制，两者之间没有共享的目录或血缘追踪。

New architecture

Analytics now run directly against data in Amazon S3 through Amazon SageMaker Unified Studio. There’s no data copy step. Analysts query the same data that the ingestion pipelines produce, using Athena for SQL and EMR Serverless for large-scale processing. Governance, access control, and lineage are centralized through IAM Identity Center and SageMaker Catalog.现在，分析通过 Amazon SageMaker Unified Studio 直接针对 Amazon S3 中的数据运行，无需数据复制步骤。分析师直接查询摄取管道产生的同一数据，使用 Athena 进行 SQL 查询，使用 EMR Serverless 进行大规模处理。治理、访问控制和血缘追踪通过 IAM Identity Center 和 SageMaker Catalog 集中管理。

The following diagram illustrates the target architecture. It follows a lakehouse pattern, storing data in open formats on Amazon S3 while maintaining ACID transaction support for the consistency financial regulators expect.下图展示了目标架构。它采用湖仓模式，以开放格式将数据存储在 Amazon S3 上，同时保留对 ACID 事务的支持，以满足金融监管机构期望的一致性。

The architecture has three layers:该架构包含三层：

1. Data layer – Amazon S3 stores data in open formats (Parquet, Delta Lake) with S3 Intelligent-Tiering for automatic cost optimization. AWS Glue Data Catalog maintains schema metadata, making data discoverable across tools.数据层 – Amazon S3 以开放格式（Parquet、Delta Lake）存储数据，并使用 S3 Intelligent-Tiering 实现自动成本优化。AWS Glue Data Catalog 维护 schema 元数据，使数据可在各种工具中发现。
2. Compute layer – Amazon SageMaker Unified Studio provides project-based workspaces organized by business function. Collections uses the built-in SQL Query Editor powered by Athena, Risk Reporting uses JupyterLab for interactive analysis, and MIS runs large-scale Spark jobs through Amazon EMR Serverless. AWS Glue ETL handles data transformations and AWS Lambda provides event-driven triggers for report generation. For machine learning (ML) workloads, Amazon SageMaker AI supports model training and deployment, with Amazon Bedrock available for generative AI capabilities such as enhancing risk narratives.计算层 – Amazon SageMaker Unified Studio 提供按业务职能组织的项目工作空间。催收部门使用由 Athena 驱动的内置 SQL 查询编辑器，风险报告使用 JupyterLab 进行交互式分析，管理信息系统 (MIS) 通过 Amazon EMR Serverless 运行大规模 Spark 作业。AWS Glue ETL 处理数据转换，AWS Lambda 为报告生成提供事件驱动触发器。对于机器学习 (ML) 工作负载，Amazon SageMaker AI 支持模型训练和部署，Amazon Bedrock 可用于生成式 AI 功能，如增强风险报告内容。
3. Governance layer – IAM Identity Center provides SSO and audit logging across workspaces. SageMaker Catalog serves as the business glossary with data lineage tracking and access controls. Amazon DataZone connects components through a common metadata layer.治理层 – IAM Identity Center 提供跨工作空间的 SSO 和审计日志。SageMaker Catalog 作为业务术语表，提供数据血缘追踪和访问控制。Amazon DataZone 通过公共元数据层连接各组件。

Migration journey迁移历程

Avanse followed a five-phase approach. The timelines can be adapted to your environment, but the systematic progression from validation through production deployment is key.Avanse 采用了五阶段方法。您可以根据自身环境调整时间表，但关键是从验证到生产部署的系统性推进。

Phase 1: Technical validation (72-hour workshop)第一阶段：技术验证（72 小时工作坊）

Avanse started with a focused 72-hour workshop using isolated SageMaker environments where developers could experiment without impacting production. Their team tested SQL analytics against existing Athena tables and validated that Python and PySpark could replicate their existing analytics workflows.Avanse 从为期 72 小时的专项工作坊开始，使用隔离的 SageMaker 环境，让开发人员可以在不影响生产的情况下进行试验。团队针对现有 Athena 表测试了 SQL 分析，并验证了 Python 和 PySpark 能够复制现有的分析工作流。

The team confirmed that querying data directly in Amazon S3 addressed their synchronization bottleneck entirely. The 4-hour daily data copy was no longer necessary, which validated the migration approach.团队确认，直接查询 Amazon S3 中的数据完全解决了同步瓶颈问题。每日 4 小时的数据复制不再必要，这验证了迁移方法的可行性。

Phase 2: Data migration and storage optimization第二阶段：数据迁移与存储优化

Avanse migrated datasets from NFS storage and legacy analytics formats into Amazon S3, consolidating the data into a single location. They implemented S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns, optimizing costs without impacting retrieval performance.Avanse 将数据集从 NFS 存储和传统分析格式迁移到 Amazon S3，将数据整合到单一位置。他们还实施了 S3 Intelligent-Tiering，根据使用模式在访问层之间自动移动数据，在不影响检索性能的情况下优化成本。

They replaced legacy analytics connectors with native Athena workgroups within SageMaker Unified Studio, avoiding data synchronization entirely. Source data remained in Amazon S3, queryable by both Athena SQL and SageMaker notebooks, establishing a single source of truth.他们用 SageMaker Unified Studio 中原生的 Athena 工作组取代了旧版分析连接器，完全避免了数据同步。源数据仍保留在 Amazon S3 中，可通过 Athena SQL 和 SageMaker 笔记本进行查询，从而建立了单一事实来源。

Phase 3: Compute modernization第三阶段：计算现代化

Avanse moved from a shared analytics server to project-based isolation in SageMaker Unified Studio. Each business function (Risk Reporting, Collections, MIS) received its own project with dedicated compute spaces running JupyterLab. Project-specific IAM execution roles provided access controls and cost allocation per business unit.Avanse 从共享分析服务器迁移到 SageMaker Unified Studio 中的项目级隔离。每个业务职能（风险报告、催收、MIS）都有自己的项目，配有运行 JupyterLab 的专用计算空间。项目特定的 IAM 执行角色提供了访问控制和按业务单元的成本分配。

A single browser-based URL with multi-factor authentication (MFA) now provides access to SQL analytics using the built-in query editor, ML development in JupyterLab notebooks, and big data processing through Amazon EMR Serverless. This replaced the need for local analytics client installations.现在，通过一个基于浏览器的 URL 和多重身份验证 (MFA) 即可访问内置查询编辑器进行 SQL 分析、在 JupyterLab 笔记本中进行机器学习开发，以及通过 Amazon EMR Serverless 进行大数据处理。这取代了本地安装分析客户端的需要。

Phase 4: Governance implementation第四阶段：治理实施

Avanse deployed SageMaker Catalog as their central business data catalog. Analysts now discover approved datasets through semantic search rather than navigating folder structures or relying on manual documentation. They mapped technical Athena table names to business terms. For example, analysts search for “collection efficiency” and find the relevant tables with descriptions, schemas, and lineage.Avanse 部署 SageMaker Catalog 作为其中心业务数据目录。分析师现在通过语义搜索发现已获批的数据集，而不是浏览文件夹结构或依赖手工文档。他们将 Athena 技术表名映射到业务术语。例如，分析师搜索“催收效率”即可找到相关表及其描述、schema 和血缘。

Lineage capture traces each metric in risk reports back to source tables, transformations, and intermediate datasets. Every action (notebook execution, SQL query, data access) is tied to IAM Identity Center users, creating the comprehensive audit trail their compliance team needed.血缘捕获将风险报告中的每个指标追溯到源表、转换和中间数据集。每个操作（笔记本执行、SQL 查询、数据访问）都与 IAM Identity Center 用户关联，从而形成合规团队所需的全面审计跟踪。

Phase 5: Use case migration第五阶段：用例迁移

Rather than attempting a big-bang migration, Avanse moved critical workflows one at a time:Avanse 没有尝试一次性大规模迁移，而是逐个迁移关键工作流：

Portfolio MIS (Monthly/Fortnightly)

Previously required the daily 4-hour data copy from Amazon S3 into the external analytics application before report generation could begin. Avanse avoided the data synchronization step entirely and now generates MIS reports by querying existing Athena tables directly in Amazon S3. Because the source data was already on AWS, there was no need to involve the external application for this activity. Report generation dropped from hours to under 30 minutes.以前，在生成报告之前需要每天花费 4 小时将数据从 Amazon S3 复制到外部分析应用。Avanse 完全避免了数据同步步骤，现在通过直接查询 Amazon S3 中的现有 Athena 表来生成 MIS 报告。由于源数据已在 AWS 上，因此无需外部应用参与此活动。报告生成时间从数小时缩短到 30 分钟以内。

Collection Efficiency and Bounce Calculation

Ported complex legacy analytics procedures for calculating metrics like collection efficiency and bounce rates to event-driven processing using AWS Glue ETL, AWS Lambda, and PySpark jobs for high-volume data aggregation. The serverless execution model charges only for compute time consumed.将用于计算催收效率和退信率等指标的复杂旧版分析程序迁移到事件驱动处理，使用 AWS Glue ETL、AWS Lambda 和 PySpark 作业进行高容量数据聚合。无服务器执行模型仅按消耗的计算时间收费。

EDW Risk Reporting

Large-scale regulatory joins of Enterprise Data Warehouse assets previously ran as legacy scheduled procedures. These now run as SQL queries in the SageMaker Unified Studio query editor, where analysts execute them on-demand or schedule them through Athena workgroups. The distributed query engine handles complex multi-table joins spanning millions of rows.以前，企业数据仓库资产的大规模监管联接以旧版计划程序的形式运行。现在，这些查询以 SQL 查询的形式在 SageMaker Unified Studio 查询编辑器中运行，分析师可以按需执行或通过 Athena 工作组进行调度。分布式查询引擎可处理涉及数百万行的复杂多表联接。

Scorecard Generation

Model building shifted from the external analytics application to SageMaker AI workflows. Data scientists use JupyterLab with Python libraries and deploy models directly to SageMaker endpoints, avoiding data movement between separate environments.模型构建从外部分析应用转向 SageMaker AI 工作流。数据科学家使用带有 Python 库的 JupyterLab，并直接将模型部署到 SageMaker 端点，避免了不同环境之间的数据移动。

Overcoming technical challenges克服技术挑战

One technical challenge was code migration. Avanse’s analytics code base contained years of accumulated proprietary scripts and procedures. Direct line-by-line translation was not practical. Instead, they took a pragmatic approach: basic data transformations moved to SQL in Athena, complex business logic was rewritten in PySpark for scalability, and statistical procedures were replaced with Python libraries like pandas and scikit-learn. The approach was to focus on what the code accomplishes, then implement it using cloud-native patterns.一个技术挑战是代码迁移。Avanse 的分析代码库中包含多年积累的专有脚本和程序。逐行直接转换并不现实。相反，他们采取了务实的方法：基本数据转换迁移到 Athena 中的 SQL，复杂业务逻辑用 PySpark 重写以实现可扩展性，统计程序则替换为 pandas 和 scikit-learn 等 Python 库。方法是关注代码要实现的功能，然后使用云原生模式加以实现。

The other technical challenge was performance validation. The team needed to confirm that querying data in Amazon S3 would deliver acceptable performance compared to the external analytics application’s in-memory processing. Queries against Parquet-formatted data in Amazon S3 using Athena delivered comparable performance for standard reporting workloads, while avoiding the 4-hour daily data synchronization step entirely. For large-scale regulatory joins spanning millions of rows, Amazon EMR Serverless provided distributed Spark processing that completed in minutes rather than the hours required in the external application.另一个技术挑战是性能验证。团队需要确认，与外部分析应用的内存处理相比，查询 Amazon S3 中的数据能否提供可接受的性能。使用 Athena 查询 Amazon S3 中 Parquet 格式的数据，对于标准报告工作负载而言，性能与外部应用相当，同时完全省去了每日 4 小时的数据同步步骤。对于涉及数百万行的大规模监管联接，Amazon EMR Serverless 提供的分布式 Spark 处理可在几分钟内完成，而外部应用则需要数小时。

Key outcomes关键成果

“By adopting SageMaker Unified Studio, we as the Data Team eliminated legacy licensing costs, reduced storage and compute expenses with a serverless, usage-based model, and accelerated our periodic report generation. At the same time, we transformed compliance and collaboration by cutting audit timelines while unifying our teams in a single, efficient data environment.” – Komal Thakkar, AVP – Lead, Data Engineering, Avanse Financial Services“通过采用 SageMaker Unified Studio，我们数据团队消除了传统许可证成本，借助无服务器、按用量付费的模式降低了存储和计算费用，并加快了定期报告的生成。同时，我们缩短了审计时间，并将团队统一到一个高效的数据环境中，从而改善了合规与协作。” – Komal Thakkar，Avanse Financial Services 数据工程负责人助理副总裁 (AVP)

Best practices最佳实践

Based on their experience, Avanse recommends:根据自身经验，Avanse 建议：

• Start with a workshop. Validate your specific use cases in a 72-hour technical validation before committing to full migration.从工作坊开始。在投入全面迁移之前，先通过 72 小时的技术验证来验证您的具体用例。
• Migrate use cases, not code. Focus on what your analytics accomplish, then implement using cloud-native patterns rather than translating legacy scripts line by line.迁移用例，而非代码。关注分析要实现的目标，然后使用云原生模式实施，而不是逐行转换旧脚本。
• Invest in governance early. Implement the data catalog and lineage tracking from day one.尽早投资治理。从第一天起就实施数据目录和血缘追踪。
• Embrace project-based isolation. Organize around business functions for clear cost allocation and security boundaries.采用项目级隔离。围绕业务职能进行组织，以获得清晰的成本分配和安全边界。
• Document business logic. Use migration as an opportunity to capture undocumented knowledge in the business glossary and dataset descriptions.记录业务逻辑。利用迁移的机会，将未文档化的知识沉淀到业务术语表和数据集描述中。

Conclusion结论

Avanse’s migration from an external analytics application to Amazon SageMaker Unified Studio consolidated their analytics stack into a single integrated environment on AWS. By querying data directly in Amazon S3 instead of copying it into the external application, they alleviated their biggest operational bottleneck. Project-based isolation replaced a shared server model, giving each business unit independent compute and clear cost visibility. And centralized governance through SageMaker Catalog and IAM Identity Center gave their compliance team the audit trails they had been missing.Avanse 从外部分析应用迁移到 Amazon SageMaker Unified Studio，将其分析技术栈整合到 AWS 上的单一集成环境中。通过直接在 Amazon S3 中查询数据，而不是将数据复制到外部应用，他们缓解了最大的运维瓶颈。项目级隔离取代了共享服务器模式，让每个业务部门都拥有独立计算资源和清晰的成本可见性。通过 SageMaker Catalog 和 IAM Identity Center 实现的集中治理，也为合规团队提供了此前缺失的审计跟踪。

The serverless, usage-based model means Avanse no longer pays for idle capacity. The lakehouse architecture supports new analytics patterns as they emerge, and native integration with AWS services, including generative AI through Amazon Bedrock, positions them to adopt new capabilities as their needs evolve.无服务器、按用量付费的模式意味着 Avanse 不再为闲置容量付费。湖仓架构支持新兴的分析模式，与 AWS 服务（包括通过 Amazon Bedrock 提供的生成式 AI）的原生集成，使他们能够在需求演进时采用新功能。

Next steps后续步骤

Start your analytics modernization journey by scheduling a 72-hour technical validation workshop. Contact your AWS account team to discuss your migration approach.安排一次 72 小时的技术验证工作坊，开启您的分析现代化之旅。请联系您的 AWS 客户团队，讨论您的迁移方案。

For more information, see:更多信息，请参阅：

• Unify data, analytics, and AI with Amazon SageMaker Unified StudioAWS Analytics Reference ArchitectureBuild a lake house architecture on AWSWhat is Amazon DataZone?使用 Amazon SageMaker Unified Studio 统一数据、分析和 AI；AWS 分析参考架构；在 AWS 上构建湖仓架构；什么是 Amazon DataZone？