The Mother of AI ProjectAI 项目之母

Phase 1 RAG Systems: arXiv Paper Curator第一阶段 RAG 系统：arXiv 论文策展人

A Learner-Focused Journey into Production RAG Systems面向学习者的生产级 RAG 系统构建之旅

Learn to build modern AI systems from the ground up through hands-on implementation通过动手实践，从零开始学习构建现代 AI 系统

Master the most in-demand AI engineering skills: RAG (Retrieval-Augmented Generation)掌握最热门的 AI 工程技能：RAG（检索增强生成）

📖 About This Course📖 关于本课程

This is a learner-focused project where you'll build a complete research assistant system that automatically fetches academic papers, understands their content, and answers your research questions using advanced RAG techniques.这是一个以学习者为中心的项目，你将构建一个完整的科研助手系统，它能自动获取学术论文、理解其内容，并利用先进的 RAG 技术回答你的研究问题。

The arXiv Paper Curator will teach you to build a production-grade RAG system using industry best practices. Unlike tutorials that jump straight to vector search, we follow the professional path: master keyword search foundations first, then enhance with vectors for hybrid retrieval.“arXiv 论文策展人”项目将教你如何使用行业最佳实践构建生产级 RAG 系统。与那些直接跳到向量搜索的教程不同，我们遵循专业路径：先掌握关键词搜索基础，再通过向量技术增强，实现混合检索。

🎯 The Professional Difference: We build RAG systems the way successful companies do - solid search foundations enhanced with AI, not AI-first approaches that ignore search fundamentals.🎯 专业差异：我们像成功企业那样构建 RAG 系统——以扎实的搜索基础为核心，并辅以 AI 增强，而非忽视搜索基础的“AI 优先”方法。

By the end of this course, you'll have your own AI research assistant and the deep technical skills to build production RAG systems for any domain.课程结束时，你将拥有自己的 AI 科研助手，并具备为任何领域构建生产级 RAG 系统的深厚技术能力。

🎓 What You'll Build🎓 你将构建什么

• Week 1: Complete infrastructure with Docker, FastAPI, PostgreSQL, OpenSearch, and Airflow第 1 周：使用 Docker、FastAPI、PostgreSQL、OpenSearch 和 Airflow 构建完整基础设施
• Week 2: Automated data pipeline fetching and parsing academic papers from arXiv第 2 周：自动从 arXiv 获取并解析学术论文的数据流水线
• Week 3: Production BM25 keyword search with filtering and relevance scoring第 3 周：支持过滤和相关性评分的生产级 BM25 关键词搜索
• Week 4: Intelligent chunking + hybrid search combining keywords with semantic understanding第 4 周：智能分块 + 结合关键词与语义理解的混合搜索
• Week 5: Complete RAG pipeline with local LLM, streaming responses, and Gradio interface第 5 周：包含本地 LLM、流式响应和 Gradio 界面的完整 RAG 流水线
• Week 6: Production monitoring with Langfuse tracing and Redis caching for optimized performance第 6 周：使用 Langfuse 追踪和 Redis 缓存进行生产监控，以优化性能
• Week 7: Agentic RAG with LangGraph and Telegram Bot for mobile access第 7 周：使用 LangGraph 的智能体 RAG 以及用于移动端访问的 Telegram Bot

---

🏗️ System Architecture Evolution🏗️ 系统架构演进

Week 7: Agentic RAG & Telegram Bot Integration第 7 周：智能体 RAG 与 Telegram Bot 集成

Complete Week 7 architecture showing Telegram bot integration with the agentic RAG system完整的第 7 周架构，展示了 Telegram Bot 与智能体 RAG 系统的集成

LangGraph Agentic RAG WorkflowLangGraph 智能体 RAG 工作流

Detailed LangGraph workflow showing decision nodes, document grading, and adaptive retrieval详细的 LangGraph 工作流，展示了决策节点、文档评分和自适应检索

Week 7 Code walkthrough + blog: Agentic RAG with LangGraph and Telegram第 7 周代码导读 + 博客：使用 LangGraph 和 Telegram 构建智能体 RAG

Key Innovations in Week 7:第 7 周的关键创新：

• Intelligent Decision-Making: Agents evaluate and adapt retrieval strategies智能决策：智能体评估并调整检索策略
• Document Grading: Automatic relevance assessment with semantic evaluation文档评分：通过语义评估自动进行相关性判定
• Query Rewriting: Adaptive query refinement when results are insufficient查询重写：在结果不足时自动优化查询
• Guardrails: Out-of-domain detection prevents hallucination护栏机制：检测领域外问题以防止幻觉
• Mobile Access: Telegram bot for conversational AI on any device移动端访问：通过 Telegram Bot 在任何设备上进行对话式 AI 交互
• Transparency: Full reasoning step tracking for debugging and trust透明度：追踪完整的推理步骤，便于调试和建立信任

---

🚀 Quick Start🚀 快速入门

📋 Prerequisites📋 前置要求

• Docker Desktop (with Docker Compose)Docker Desktop（含 Docker Compose）
• Python 3.12+Python 3.12+
• UV Package Manager (Install Guide)UV 包管理器（安装指南）
• 8GB+ RAM and 20GB+ free disk space8GB 以上内存和 20GB 以上可用磁盘空间

⚡ Get Started⚡ 开始使用

# 1. Clone and setup
git clone <repository-url>
cd arxiv-paper-curator

# 2. Configure environment (IMPORTANT!)
cp .env.example .env
# The .env file contains all necessary configuration for OpenSearch, 
# arXiv API, and service connections. Defaults work out of the box.
# You need to add Jina embeddings free api key and langfuse keys (check the blogs)

# 3. Install dependencies
uv sync

# 4. Start all services
docker compose up --build -d

# 5. Verify everything works
curl http://localhost:8000/api/v1/health

📚 Weekly Learning Path📚 每周学习路径

Week	Topic	Blog Post	Code Release
Week 0	The Mother of AI project - 6 phases	The Mother of AI project	-
Week 1	Infrastructure Foundation	The Infrastructure That Powers RAG Systems	week1.0
Week 2	Data Ingestion Pipeline	Building Data Ingestion Pipelines for RAG	week2.0
Week 3	OpenSearch ingestion & BM25 retrieval	The Search Foundation Every RAG System Needs	week3.0
Week 4	Chunking & Hybrid Search	The Chunking Strategy That Makes Hybrid Search Work	week4.0
Week 5	Complete RAG system	The Complete RAG System	week5.0
Week 6	Production monitoring & caching	Production-ready RAG: Monitoring & Caching	week6.0
Week 7	Agentic RAG & Telegram Bot	Agentic RAG with LangGraph and Telegram	week7.0

📥 Clone a specific week's release:📥 克隆特定周的 Releases：

# Clone a specific week's code
git clone --branch <WEEK_TAG> https://github.com/jamwithai/arxiv-paper-curator
cd arxiv-paper-curator
uv sync
docker compose down -v
docker compose up --build -d

# Replace <WEEK_TAG> with: week1.0, week2.0, etc.

📊 Access Your Services📊 访问你的服务

Service	URL	Purpose
API Documentation	http://localhost:8000/docs	Interactive API testing
Gradio RAG Interface	http://localhost:7861	User-friendly chat interface
Langfuse Dashboard	http://localhost:3000	RAG pipeline monitoring & tracing
Airflow Dashboard	http://localhost:8080	Workflow management
OpenSearch Dashboards	http://localhost:5601	Hybrid search engine UI

NOTE: Check airflow/simple_auth_manager_passwords.json.generated for Airflow username and password注意：查看 airflow/simple_auth_manager_passwords.json.generated 获取 Airflow 用户名和密码

---

📚 Week 1: Infrastructure Foundation ✅📚 第 1 周：基础设施基础 ✅

Start here! Master the infrastructure that powers modern RAG systems.从这里开始！掌握驱动现代 RAG 系统运行的基础设施。

🎯 Learning Objectives🎯 学习目标

• Complete infrastructure setup with Docker Compose使用 Docker Compose 完成基础设施搭建
• FastAPI development with automatic documentation and health checks开发具备自动文档和健康检查功能的 FastAPI
• PostgreSQL database configuration and managementPostgreSQL 数据库配置与管理
• OpenSearch hybrid search engine setupOpenSearch 混合搜索引擎设置
• Ollama local LLM service configurationOllama 本地 LLM 服务配置
• Service orchestration and health monitoring服务编排与健康监控
• Professional development environment with code quality tools配备代码质量工具的专业开发环境

🏗️ Architecture Overview🏗️ 架构概览

Infrastructure Components:基础设施组件：

• FastAPI: REST endpoints with async support (Port 8000)FastAPI：支持异步的 REST 接口（端口 8000）
• PostgreSQL 16: Paper metadata storage (Port 5432)PostgreSQL 16：论文元数据存储（端口 5432）
• OpenSearch 2.19: Search engine with dashboards (Ports 9200, 5601)OpenSearch 2.19：带仪表盘的搜索引擎（端口 9200, 5601）
• Apache Airflow 3.0: Workflow orchestration (Port 8080)Apache Airflow 3.0：工作流编排（端口 8080）
• Ollama: Local LLM server (Port 11434)Ollama：本地 LLM 服务器（端口 11434）

📓 Setup Guide📓 设置指南

# Launch the Week 1 notebook
uv run jupyter notebook notebooks/week1/week1_setup.ipynb

Completion Guide: Follow the Week 1 notebook for hands-on setup and verification steps.完成指南：按照第 1 周的 Notebook 进行动手设置和验证步骤。

📖 Deep Dive📖 深入探索

Blog Post: The Infrastructure That Powers RAG Systems - Detailed walkthrough and production insights博客文章：驱动 RAG 系统运行的基础设施——详细导读与生产实践心得

---

📚 Week 2: Data Ingestion Pipeline ✅📚 第 2 周：数据摄入流水线 ✅

Building on Week 1 infrastructure: Learn to fetch, process, and store academic papers automatically.在第 1 周的基础设施之上：学习如何自动获取、处理并存储学术论文。

🎯 Learning Objectives

• arXiv API integration with rate limiting and retry logic具备速率限制和重试逻辑的 arXiv API 集成
• Scientific PDF parsing using Docling使用 Docling 进行科学 PDF 解析
• Automated data ingestion pipelines with Apache Airflow使用 Apache Airflow 构建自动化数据摄入流水线
• Metadata extraction and storage workflows元数据提取与存储工作流
• Complete paper processing from API to database从 API 到数据库的完整论文处理流程

🏗️ Architecture Overview

Data Pipeline Components:数据流水线组件：

• MetadataFetcher: 🎯 Main orchestrator coordinating the entire pipelineMetadataFetcher：🎯 协调整个流水线的主要编排器
• ArxivClient: Rate-limited paper fetching with retry logicArxivClient：带速率限制和重试逻辑的论文获取客户端
• PDFParserService: Docling-powered scientific document processingPDFParserService：基于 Docling 的科学文档处理服务
• Airflow DAGs: Automated daily paper ingestion workflowsAirflow DAGs：自动化每日论文摄入工作流
• PostgreSQL Storage: Structured paper metadata and contentPostgreSQL 存储：结构化论文元数据与内容

📓 Implementation Guide📓 实现指南

# Launch the Week 2 notebook  
uv run jupyter notebook notebooks/week2/week2_arxiv_integration.ipynb

Completion Guide: Follow the Week 2 notebook for hands-on implementation and verification steps.完成指南：按照第 2 周的 Notebook 进行动手实现和验证步骤。

📖 Deep Dive

Blog Post: Building Data Ingestion Pipelines for RAG - arXiv API integration and PDF processing博客文章：为 RAG 构建数据摄入流水线——arXiv API 集成与 PDF 处理

---

📚 Week 3: Keyword Search First - The Critical Foundation📚 第 3 周：关键词搜索优先——关键基础

Building on Weeks 1-2 foundation: Implement the keyword search foundation that professional RAG systems rely on.在第 1-2 周的基础上：实现专业 RAG 系统所依赖的关键词搜索基础。

🎯 Learning Objectives

• Why keyword search is essential for RAG systems (foundation first approach)为什么关键词搜索对 RAG 系统至关重要（基础优先方法）
• OpenSearch index management, mappings, and search optimizationOpenSearch 索引管理、映射和搜索优化
• BM25 algorithm and the math behind effective keyword searchBM25 算法及高效关键词搜索背后的数学原理
• Query DSL for building complex search queries with filters and boosting用于构建复杂搜索查询（带过滤和权重提升）的查询 DSL
• Search analytics for measuring relevance and performance用于衡量相关性和性能的搜索分析
• Production patterns used by real companies真实企业使用的生产模式

🏗️ Architecture Overview

Search Infrastructure Components:搜索基础设施组件：

• OpenSearch Service: src/services/opensearch/ - Professional search service implementationOpenSearch Service：src/services/opensearch/ - 专业搜索服务实现
• Search API: src/routers/search.py - Search API endpoints with BM25 scoringSearch API：src/routers/search.py - 带有 BM25 评分的搜索 API 接口
• Learning Materials: notebooks/week3/ - Complete OpenSearch integration guide学习材料：notebooks/week3/ - 完整的 OpenSearch 集成指南
• Quality Metrics: Precision, recall, and relevance scoring质量指标：准确率、召回率和相关性评分

📓 Setup Guide

# Launch the Week 3 notebook
uv run jupyter notebook notebooks/week3/week3_opensearch.ipynb

Completion Guide: Follow the Week 3 notebook for hands-on OpenSearch setup and BM25 search implementation.完成指南：按照第 3 周的 Notebook 进行动手 OpenSearch 设置和 BM25 搜索实现。

📖 Deep Dive

Blog Post: The Search Foundation Every RAG System Needs - Complete BM25 implementation with OpenSearch博客文章：每个 RAG 系统都需要的搜索基础——基于 OpenSearch 的完整 BM25 实现

---

📚 Week 4: Chunking & Hybrid Search - The Semantic Layer📚 第 4 周：分块与混合搜索——语义层

Building on Week 3 foundation: Add the semantic layer that makes search truly intelligent.在第 3 周的基础上：添加使搜索真正智能化的语义层。

🎯 Learning Objectives

• Section-based chunking with intelligent document segmentation基于章节的智能文档分块与分割
• Production embeddings with Jina AI integration and fallback strategies结合 Jina AI 集成与回退策略的生产级 Embedding
• Hybrid search mastery using RRF fusion for keyword + semantic retrieval使用 RRF 融合技术实现关键词与语义检索的混合搜索
• Unified API design with single endpoint supporting multiple search modes统一的 API 设计，单接口支持多种搜索模式
• Performance analysis and trade-offs between search approaches性能分析与不同搜索方法间的权衡

🏗️ Architecture Overview

Hybrid Search Infrastructure Components:混合搜索基础设施组件：

• Text Chunker: src/services/indexing/text_chunker.py - Section-aware chunking with overlap strategiesText Chunker：src/services/indexing/text_chunker.py - 支持章节感知与重叠策略的分块
• Embeddings Service: src/services/embeddings/ - Production embedding pipeline with Jina AIEmbeddings Service：src/services/embeddings/ - 基于 Jina AI 的生产级 Embedding 流水线
• Hybrid Search API: src/routers/hybrid_search.py - Unified search API supporting all modesHybrid Search API：src/routers/hybrid_search.py - 支持所有模式的统一搜索 API
• Learning Materials: notebooks/week4/ - Complete hybrid search implementation guide学习材料：notebooks/week4/ - 完整的混合搜索实现指南

📓 Setup Guide

# Launch the Week 4 notebook
uv run jupyter notebook notebooks/week4/week4_hybrid_search.ipynb

Completion Guide: Follow the Week 4 notebook for hands-on implementation and verification steps.完成指南：按照第 4 周的 Notebook 进行动手实现和验证步骤。

📖 Deep Dive

Blog Post: The Chunking Strategy That Makes Hybrid Search Work - Production chunking and RRF fusion implementation博客文章：让混合搜索发挥作用的分块策略——生产级分块与 RRF 融合实现

---

📚 Week 5: Complete RAG Pipeline with LLM Integration📚 第 5 周：包含 LLM 集成的完整 RAG 流水线

Building on Week 4 hybrid search: Add the LLM layer that turns search into intelligent conversation.在第 4 周混合搜索的基础上：添加 LLM 层，将搜索转化为智能对话。

🎯 Learning Objectives

• Local LLM integration with Ollama for complete data privacy集成 Ollama 本地 LLM，确保完整的数据隐私
• Performance optimization with 80% prompt reduction (6x speed improvement)通过 80% 的 Prompt 缩减实现性能优化（速度提升 6 倍）
• Streaming implementation using Server-Sent Events for real-time responses使用服务器发送事件（SSE）实现实时响应的流式传输
• Dual API design with standard and streaming endpoints双 API 设计，包含标准接口与流式接口
• Interactive Gradio interface with advanced parameter controls具备高级参数控制的交互式 Gradio 界面

🏗️ Architecture Overview

Complete RAG Infrastructure Components:完整 RAG 基础设施组件：

• RAG Endpoints: src/routers/ask.py - Dual endpoints (/api/v1/ask + /api/v1/stream)RAG Endpoints：src/routers/ask.py - 双接口（/api/v1/ask + /api/v1/stream）
• Ollama Service: src/services/ollama/ - LLM client with optimized promptsOllama Service：src/services/ollama/ - 带有优化 Prompt 的 LLM 客户端
• System Prompt: src/services/ollama/prompts/rag_system.txt - Optimized for academic papersSystem Prompt：src/services/ollama/prompts/rag_system.txt - 针对学术论文优化
• Gradio Interface: src/gradio_app.py - Interactive web UI with streaming supportGradio Interface：src/gradio_app.py - 支持流式传输的交互式 Web UI
• Launcher Script: gradio_launcher.py - Easy-launch script (runs on port 7861)Launcher Script：gradio_launcher.py - 简易启动脚本（运行于端口 7861）

📓 Setup Guide

# Launch the Week 5 notebook
uv run jupyter notebook notebooks/week5/week5_complete_rag_system.ipynb

# Launch Gradio interface
uv run python gradio_launcher.py
# Open http://localhost:7861

Completion Guide: Follow the Week 5 notebook for hands-on LLM integration and RAG pipeline implementation.完成指南：按照第 5 周的 Notebook 进行动手 LLM 集成和 RAG 流水线实现。

📖 Deep Dive

Blog Post: The Complete RAG System - Complete RAG system with local LLM integration and optimization techniques博客文章：完整的 RAG 系统——结合本地 LLM 集成与优化技术的完整 RAG 系统

---

📚 Week 6: Production Monitoring and Caching📚 第 6 周：生产监控与缓存

Building on Week 5 complete RAG system: Add observability, performance optimization, and production-grade monitoring.在第 5 周完整 RAG 系统基础上：添加可观测性、性能优化和生产级监控。

🎯 Learning Objectives

• Langfuse integration for end-to-end RAG pipeline tracing集成 Langfuse 实现端到端 RAG 流水线追踪
• Redis caching strategy with intelligent cache keys and TTL management具有智能缓存键和 TTL 管理的 Redis 缓存策略
• Performance monitoring with real-time dashboards for latency and costs具备实时延迟与成本仪表盘的性能监控
• Production patterns for observability and optimization用于可观测性与优化的生产模式
• Cost analysis and LLM usage optimization (150-400x speedup with caching)成本分析与 LLM 使用优化（通过缓存实现 150-400 倍提速）

🏗️ Architecture Overview

Production Infrastructure Components:生产基础设施组件：

• Langfuse Service: src/services/langfuse/ - Complete tracing integration with RAG-specific metricsLangfuse Service：src/services/langfuse/ - 带有 RAG 特定指标的完整追踪集成
• Cache Service: src/services/cache/ - Redis client with exact-match caching and graceful fallbackCache Service：src/services/cache/ - 具有精确匹配缓存与优雅降级功能的 Redis 客户端
• Updated Endpoints: src/routers/ask.py - Integrated tracing and caching middlewareUpdated Endpoints：src/routers/ask.py - 集成了追踪与缓存中间件
• Docker Config: docker-compose.yml - Added Redis service and Langfuse local instanceDocker Config：docker-compose.yml - 添加了 Redis 服务和 Langfuse 本地实例
• Learning Materials: notebooks/week6/ - Complete monitoring and caching implementation guide学习材料：notebooks/week6/ - 完整的监控与缓存实现指南

📓 Setup Guide

# Launch the Week 6 notebook
uv run jupyter notebook notebooks/week6/week6_cache_testing.ipynb

Completion Guide: Follow the Week 6 notebook for hands-on Langfuse tracing and Redis caching implementation.完成指南：按照第 6 周的 Notebook 进行动手 Langfuse 追踪和 Redis 缓存实现。

📖 Deep Dive

Blog Post: Production-ready RAG: Monitoring & Caching - Production-ready RAG with monitoring and caching博客文章：生产就绪的 RAG：监控与缓存——具备监控与缓存功能的生产级 RAG

---

📚 Week 7: Agentic RAG with LangGraph and Telegram Bot📚 第 7 周：使用 LangGraph 和 Telegram Bot 的智能体 RAG

Building on Week 6 production system: Add intelligent reasoning, multi-step decision-making, and Telegram bot integration for mobile-first AI interactions.在第 6 周生产系统基础上：添加智能推理、多步决策，以及用于移动优先 AI 交互的 Telegram Bot 集成。

🎯 Learning Objectives

• LangGraph workflows for state-based agent orchestration with decision nodes用于基于状态的智能体编排与决策节点的 LangGraph 工作流
• Guardrail implementation for query validation and domain boundary detection用于查询验证与领域边界检测的护栏机制实现
• Document grading with semantic relevance evaluation基于语义相关性评估的文档评分
• Query rewriting for automatic query refinement and better retrieval用于自动查询优化与更好检索的查询重写
• Adaptive retrieval with multi-attempt retrieval and intelligent fallback具备多重尝试检索与智能回退的自适应检索
• Telegram bot integration with async operations and error handling支持异步操作和错误处理的 Telegram Bot 集成
• Reasoning transparency by exposing agent decision-making process通过展示智能体决策过程实现推理透明化

🏗️ Architecture Overview

Agentic RAG Infrastructure Components:智能体 RAG 基础设施组件：

• Agent Nodes: src/services/agents/nodes/ - Guardrail, retrieve, grade, rewrite, and generate nodesAgent Nodes：src/services/agents/nodes/ - 护栏、检索、评分、重写和生成节点
• Workflow Orchestration: src/services/agents/agentic_rag.py - LangGraph workflow coordinationWorkflow Orchestration：src/services/agents/agentic_rag.py - LangGraph 工作流协调
• Telegram Bot: src/services/telegram/ - Command handlers and message processingTelegram Bot：src/services/telegram/ - 命令处理程序与消息处理
• Agentic Endpoint: src/routers/agentic_ask.py - Agentic RAG API endpointAgentic Endpoint：src/routers/agentic_ask.py - 智能体 RAG API 接口
• Learning Materials: notebooks/week7/ - Week 7 learning materials and examples学习材料：notebooks/week7/ - 第 7 周学习材料与示例

📓 Setup Guide

# Launch the Week 7 notebook
uv run jupyter notebook notebooks/week7/week7_agentic_rag.ipynb

Completion Guide: Follow the Week 7 notebook for hands-on LangGraph agentic RAG and Telegram bot implementation.完成指南：按照第 7 周的 Notebook 进行动手 LangGraph 智能体 RAG 和 Telegram Bot 实现。

📖 Deep Dive

Blog Post: Agentic RAG with LangGraph and Telegram - Building intelligent agents with decision-making, adaptive retrieval, and mobile access博客文章：使用 LangGraph 和 Telegram 构建智能体 RAG——构建具备决策、自适应检索和移动访问能力的智能体

---

⚙️ Configuration⚙️ 配置

Setup:设置：

cp .env.example .env
# Edit .env for your environment

Key Variables:关键变量：

• JINA_API_KEY - Required for Week 4+ (hybrid search with embeddings)JINA_API_KEY - 第 4 周及以后需要（带 Embedding 的混合搜索）
• TELEGRAM__BOT_TOKEN - Required for Week 7 (Telegram bot integration)TELEGRAM__BOT_TOKEN - 第 7 周需要（Telegram Bot 集成）
• LANGFUSE__PUBLIC_KEY & LANGFUSE__SECRET_KEY - Optional for Week 6 (monitoring)LANGFUSE__PUBLIC_KEY & LANGFUSE__SECRET_KEY - 第 6 周可选（监控）

Complete Configuration: See .env.example for all available options and detailed documentation.完整配置：查看 .env.example 获取所有可用选项及详细文档。

---

🔧 Reference & Development Guide🔧 参考与开发指南

🛠️ Technology Stack🛠️ 技术栈

Service	Purpose	Status
FastAPI	REST API with automatic docs	✅ Ready
PostgreSQL 16	Paper metadata and content storage	✅ Ready
OpenSearch 2.19	Hybrid search engine (BM25 + Vector)	✅ Ready
Apache Airflow 3.0	Workflow automation	✅ Ready
Jina AI	Embedding generation (Week 4)	✅ Ready
Ollama	Local LLM serving (Week 5)	✅ Ready
Redis	High-performance caching (Week 6)	✅ Ready
Langfuse	RAG pipeline observability (Week 6)	✅ Ready

Development Tools: UV, Ruff, MyPy, Pytest, Docker Compose开发工具：UV, Ruff, MyPy, Pytest, Docker Compose

🏗️ Project Structure🏗️ 项目结构

arxiv-paper-curator/
├── src/                    # Main application code
│   ├── routers/            # API endpoints (search, ask, papers)
│   ├── services/           # Business logic (opensearch, ollama, agents, cache)
│   ├── models/             # Database models (SQLAlchemy)
│   ├── schemas/            # Pydantic validation schemas
│   └── config.py           # Environment configuration
├── notebooks/              # Weekly learning materials (week1-7)
├── airflow/                # Workflow orchestration (DAGs)
├── tests/                  # Test suite
└── compose.yml             # Docker service orchestration

📡 API Endpoints Reference📡 API 接口参考

Endpoint	Method	Description	Week
/health	GET	Service health check	Week 1
/api/v1/papers	GET	List stored papers	Week 2
/api/v1/papers/{id}	GET	Get specific paper	Week 2
/api/v1/search	POST	BM25 keyword search	Week 3
/api/v1/hybrid-search/	POST	Hybrid search (BM25 + Vector)	Week 4

API Documentation: Visit http://localhost:8000/docs for interactive API explorerAPI 文档：访问 http://localhost:8000/docs 进行交互式 API 探索

🔧 Essential Commands🔧 基础命令

Using the Makefile (Recommended)使用 Makefile（推荐）

# View all available commands
make help

# Quick workflow
make start         # Start all services
make health        # Check all services health
make test          # Run tests
make stop          # Stop services

All Available Commands所有可用命令

Command	Description
make start	Start all services
make stop	Stop all services
make restart	Restart all services
make status	Show service status
make logs	Show service logs
make health	Check all services health
make setup	Install Python dependencies
make format	Format code
make lint	Lint and type check
make test	Run tests
make test-cov	Run tests with coverage
make clean	Clean up everything

Direct Commands (Alternative)直接命令（替代方案）

# If you prefer using commands directly
docker compose up --build -d    # Start services
docker compose ps               # Check status
docker compose logs            # View logs
uv run pytest                 # Run tests

🎓 Target Audience🎓 目标受众

Who	Why
AI/ML Engineers	Learn production RAG architecture beyond tutorials
Software Engineers	Build end-to-end AI applications with best practices
Data Scientists	Implement production AI systems using modern tools

---

🛠️ Troubleshooting🛠️ 故障排除

Common Issues:常见问题：

• Services not starting? Wait 2-3 minutes, check docker compose logs服务无法启动？等待 2-3 分钟，检查 docker compose 日志
• Port conflicts? Stop other services using ports 8000, 8080, 5432, 9200端口冲突？停止占用 8000, 8080, 5432, 9200 端口的其他服务
• Memory issues? Increase Docker Desktop memory allocation内存问题？增加 Docker Desktop 的内存分配

Get Help:获取帮助：

• Check the comprehensive Week 1 notebook troubleshooting section查看第 1 周 Notebook 中全面的故障排除章节
• Review service logs: docker compose logs [service-name]查看服务日志：docker compose logs [service-name]
• Complete reset: docker compose down --volumes && docker compose up --build -d完全重置：docker compose down --volumes && docker compose up --build -d

---

💰 Cost Structure💰 成本结构

This course is completely free! You'll only need minimal costs for optional services:本课程完全免费！你只需为可选服务承担极少费用：

• Local Development: $0 (everything runs locally)本地开发：$0（所有内容均在本地运行）
• Optional Cloud APIs: ~$2-5 for external LLM services (if chosen)可选云端 API：约 $2-5 用于外部 LLM 服务（如果选择使用）

---

🎉 Ready to Start Your AI Engineering Journey?🎉 准备好开启你的 AI 工程之旅了吗？

Begin with the Week 1 setup notebook and build your first production RAG system!从第 1 周的设置 Notebook 开始，构建你的第一个生产级 RAG 系统！

For learners who want to master modern AI engineering专为希望掌握现代 AI 工程的学习者准备

Built with love by Shirin Khosravi Jam & Shantanu Ladhwe由 Shirin Khosravi Jam 和 Shantanu Ladhwe 用心打造

---

Star HistoryStar 历史

---

📄 License📄 license

MIT License - see LICENSE file for details.MIT license - 详情请参阅 LICENSE 文件。