| ¶Ô±ÈÐÂÎļþ |
| | |
| | | # ç¥è¯åºRAGåéæ£ç´¢åè½å®ç°ææ¡£ |
| | | |
| | | ## ä¸ãåè½æ¦è¿° |
| | | |
| | | åºäº RAGï¼Retrieval-Augmented Generationï¼ææ¯å®ç°ç¥è¯åºé®çåè½ï¼æ¯æï¼ |
| | | - ç¥è¯åºç®¡çï¼CRUDï¼ |
| | | - æä»¶ä¸ä¼ ä¸åéåå¤ç |
| | | - åºäºåéæ£ç´¢çæºè½é®ç |
| | | - å¤ç§æä»¶æ ¼å¼æ¯æï¼txtãmdãdocxãxlsxãxlsãpdfï¼ |
| | | |
| | | ## äºãææ¯æ¶æ |
| | | |
| | | ### 2.1 ææ¯æ |
| | | | ç»ä»¶ | ææ¯ | |
| | | |------|------| |
| | | | åéæ°æ®åº | Pinecone | |
| | | | Embedding模å | é¿éäº DashScope text-embedding-v3 | |
| | | | LLM | é¿éäºéä¹åé® qwen-max | |
| | | | æ¡æ¶ | langchain4j | |
| | | | ORM | MyBatis-Plus | |
| | | |
| | | ### 2.2 æ¶æå¾ |
| | | |
| | | ``` |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | â å端åºç¨ â |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | â |
| | | â¼ |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | â Controller Layer â |
| | | â âââââââââââââââââââââââ âââââââââââââââââââââââââââââââ â |
| | | â â KnowledgeBaseCtrl â â KnowledgeChatController â â |
| | | â â (ç¥è¯åºç®¡ç) â â (ç¥è¯åºé®ç) â â |
| | | â âââââââââââââââââââââââ âââââââââââââââââââââââââââââââ â |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | â |
| | | â¼ |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | â Service Layer â |
| | | â âââââââââââââââââââââââ âââââââââââââââââââââââââââââââ â |
| | | â âKnowledgeBaseService â â KnowledgeRagService â â |
| | | â â (ç¥è¯åºCRUD) â â (åéå/æ£ç´¢) â â |
| | | â âââââââââââââââââââââââ âââââââââââââââââââââââââââââââ â |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | â |
| | | â¼ |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | â AI Layer â |
| | | â âââââââââââââââââââââââ âââââââââââââââââââââââââââââââ â |
| | | â â KnowledgeChatAgent â â EmbeddingStore (Pinecone) â â |
| | | â â (é®çAgent) â â (åéåå¨) â â |
| | | â âââââââââââââââââââââââ âââââââââââââââââââââââââââââââ â |
| | | âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ## ä¸ãå端å®ç° |
| | | |
| | | ### 3.1 æ°æ®åºè®¾è®¡ |
| | | |
| | | #### 3.1.1 ç¥è¯åºè¡¨ï¼knowledge_baseï¼ |
| | | |
| | | ```sql |
| | | CREATE TABLE knowledge_base ( |
| | | id BIGINT AUTO_INCREMENT PRIMARY KEY, |
| | | title VARCHAR(255) COMMENT 'ç¥è¯æ é¢', |
| | | type VARCHAR(50) COMMENT 'ç¥è¯ç±»å', |
| | | scenario VARCHAR(255) COMMENT 'éç¨åºæ¯', |
| | | efficiency VARCHAR(20) COMMENT 'è§£å³æç', |
| | | problem TEXT COMMENT 'é®é¢æè¿°', |
| | | solution TEXT COMMENT 'è§£å³æ¹æ¡', |
| | | key_points TEXT COMMENT 'å
³é®è¦ç¹', |
| | | creator VARCHAR(100) COMMENT 'å建人', |
| | | usage_count INT DEFAULT 0 COMMENT 'ä½¿ç¨æ¬¡æ°', |
| | | file_count INT DEFAULT 0 COMMENT 'æä»¶æ°é', |
| | | total_chunk_count INT DEFAULT 0 COMMENT 'æ»åçæ°é', |
| | | description VARCHAR(500) COMMENT 'ç¥è¯åºæè¿°', |
| | | create_time DATETIME DEFAULT CURRENT_TIMESTAMP, |
| | | update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, |
| | | create_user INT, |
| | | update_user INT, |
| | | tenant_id BIGINT, |
| | | dept_id BIGINT |
| | | ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='ç¥è¯åºè¡¨'; |
| | | ``` |
| | | |
| | | #### 3.1.2 ç¥è¯åºåéè®°å½è¡¨ï¼knowledge_base_vectorï¼ |
| | | |
| | | ```sql |
| | | CREATE TABLE knowledge_base_vector ( |
| | | id BIGINT AUTO_INCREMENT PRIMARY KEY COMMENT '主é®ID', |
| | | knowledge_base_id BIGINT NOT NULL COMMENT 'å
³èç¥è¯åºID', |
| | | storage_blob_id BIGINT NOT NULL COMMENT 'å
³èæä»¶blob ID', |
| | | file_name VARCHAR(255) NOT NULL COMMENT 'æä»¶åç§°', |
| | | file_type VARCHAR(50) NOT NULL COMMENT 'æä»¶ç±»å', |
| | | vector_status TINYINT DEFAULT 0 COMMENT 'åéåç¶æ: 0-å¾
å¤ç, 1-å¤çä¸, 2-已宿, 3-失败', |
| | | vector_error VARCHAR(500) COMMENT 'åéå失败åå ', |
| | | chunk_count INT DEFAULT 0 COMMENT 'åçæ°é', |
| | | namespace VARCHAR(100) COMMENT 'åéå½å空é´', |
| | | create_time DATETIME DEFAULT CURRENT_TIMESTAMP, |
| | | create_user INT, |
| | | update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, |
| | | update_user INT, |
| | | tenant_id BIGINT, |
| | | dept_id BIGINT, |
| | | INDEX idx_knowledge_base_id (knowledge_base_id), |
| | | INDEX idx_storage_blob_id (storage_blob_id), |
| | | INDEX idx_vector_status (vector_status) |
| | | ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='ç¥è¯åºæä»¶åéè®°å½è¡¨'; |
| | | ``` |
| | | |
| | | ### 3.2 Mavenä¾èµ |
| | | |
| | | ```xml |
| | | <!-- langchain4j BOM --> |
| | | <dependencyManagement> |
| | | <dependencies> |
| | | <dependency> |
| | | <groupId>dev.langchain4j</groupId> |
| | | <artifactId>langchain4j-bom</artifactId> |
| | | <version>1.0.0-beta3</version> |
| | | <type>pom</type> |
| | | <scope>import</scope> |
| | | </dependency> |
| | | </dependencies> |
| | | </dependencyManagement> |
| | | |
| | | <dependencies> |
| | | <!-- langchain4j æ ¸å¿ --> |
| | | <dependency> |
| | | <groupId>dev.langchain4j</groupId> |
| | | <artifactId>langchain4j-spring-boot-starter</artifactId> |
| | | </dependency> |
| | | |
| | | <!-- Pinecone åéæ°æ®åº --> |
| | | <dependency> |
| | | <groupId>dev.langchain4j</groupId> |
| | | <artifactId>langchain4j-pinecone</artifactId> |
| | | </dependency> |
| | | |
| | | <!-- é¿éäº DashScope --> |
| | | <dependency> |
| | | <groupId>dev.langchain4j</groupId> |
| | | <artifactId>langchain4j-community-dashscope-spring-boot-starter</artifactId> |
| | | </dependency> |
| | | </dependencies> |
| | | ``` |
| | | |
| | | ### 3.3 é
ç½®æä»¶ï¼application.ymlï¼ |
| | | |
| | | ```yaml |
| | | # Pinecone åéæ°æ®åºé
ç½® |
| | | pinecone: |
| | | api-key: your-pinecone-api-key |
| | | index: your-index-name |
| | | namespace: knowledge-base |
| | | |
| | | # langchain4j é
ç½® |
| | | langchain4j: |
| | | community: |
| | | dashscope: |
| | | streaming-chat-model: |
| | | api-key: your-dashscope-api-key |
| | | model-name: "qwen-max" |
| | | embedding-model: |
| | | api-key: your-dashscope-api-key |
| | | model-name: "text-embedding-v3" |
| | | ``` |
| | | |
| | | ### 3.4 æ ¸å¿ä»£ç å®ç° |
| | | |
| | | #### 3.4.1 å®ä½ç±» |
| | | |
| | | **KnowledgeBase.java** |
| | | ```java |
| | | @Data |
| | | @TableName("knowledge_base") |
| | | public class KnowledgeBase implements Serializable { |
| | | @TableId(type = IdType.AUTO) |
| | | private Long id; |
| | | private String title; |
| | | private String type; |
| | | private String scenario; |
| | | private String efficiency; |
| | | private String problem; |
| | | private String solution; |
| | | private String keyPoints; |
| | | private String creator; |
| | | private Integer usageCount; |
| | | private Integer fileCount; |
| | | private Integer totalChunkCount; |
| | | private String description; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private LocalDateTime createTime; |
| | | @TableField(fill = FieldFill.INSERT_UPDATE) |
| | | private LocalDateTime updateTime; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private Integer createUser; |
| | | @TableField(fill = FieldFill.INSERT_UPDATE) |
| | | private Integer updateUser; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private Long tenantId; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private Long deptId; |
| | | } |
| | | ``` |
| | | |
| | | **KnowledgeBaseVector.java** |
| | | ```java |
| | | @Data |
| | | @TableName("knowledge_base_vector") |
| | | public class KnowledgeBaseVector implements Serializable { |
| | | @TableId(type = IdType.AUTO) |
| | | private Long id; |
| | | private Long knowledgeBaseId; |
| | | private Long storageBlobId; |
| | | private String fileName; |
| | | private String fileType; |
| | | private Integer vectorStatus; |
| | | private String vectorError; |
| | | private Integer chunkCount; |
| | | private String namespace; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private LocalDateTime createTime; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private Integer createUser; |
| | | @TableField(fill = FieldFill.INSERT_UPDATE) |
| | | private LocalDateTime updateTime; |
| | | @TableField(fill = FieldFill.INSERT_UPDATE) |
| | | private Integer updateUser; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private Long tenantId; |
| | | @TableField(fill = FieldFill.INSERT) |
| | | private Long deptId; |
| | | |
| | | // åéåç¶æå¸¸é |
| | | public static final int STATUS_PENDING = 0; |
| | | public static final int STATUS_PROCESSING = 1; |
| | | public static final int STATUS_COMPLETED = 2; |
| | | public static final int STATUS_FAILED = 3; |
| | | } |
| | | ``` |
| | | |
| | | #### 3.4.2 EmbeddingStoreé
ç½® |
| | | |
| | | **EmbeddingStoreConfig.java** |
| | | ```java |
| | | @Configuration |
| | | public class EmbeddingStoreConfig { |
| | | |
| | | @Value("${pinecone.api-key}") |
| | | private String pineconeApiKey; |
| | | |
| | | @Value("${pinecone.index}") |
| | | private String indexName; |
| | | |
| | | @Value("${pinecone.namespace}") |
| | | private String namespace; |
| | | |
| | | @Bean |
| | | public Pinecone pinecone() { |
| | | return new Pinecone.Builder(pineconeApiKey).build(); |
| | | } |
| | | |
| | | @Bean |
| | | public Index pineconeIndex(Pinecone pinecone) { |
| | | return pinecone.getIndexConnection(indexName); |
| | | } |
| | | |
| | | @Bean |
| | | public EmbeddingStore<TextSegment> embeddingStore(EmbeddingModel embeddingModel) { |
| | | return PineconeEmbeddingStore.builder() |
| | | .apiKey(pineconeApiKey) |
| | | .index(indexName) |
| | | .nameSpace(namespace) |
| | | .createIndex(PineconeServerlessIndexConfig.builder() |
| | | .cloud("AWS") |
| | | .region("us-east-1") |
| | | .dimension(embeddingModel.dimension()) |
| | | .build()) |
| | | .build(); |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | #### 3.4.3 RAGæå¡å®ç° |
| | | |
| | | **KnowledgeRagService.java** |
| | | ```java |
| | | public interface KnowledgeRagService { |
| | | void processVectorAsync(Long vectorId); |
| | | void processVector(Long vectorId); |
| | | List<String> searchRelevantContent(String namespace, String query, int maxResults); |
| | | void deleteEmbeddings(String namespace, Long storageBlobId); |
| | | } |
| | | ``` |
| | | |
| | | **KnowledgeRagServiceImpl.java**ï¼æ ¸å¿å®ç°ï¼ |
| | | ```java |
| | | @Slf4j |
| | | @Service |
| | | public class KnowledgeRagServiceImpl implements KnowledgeRagService { |
| | | |
| | | private final KnowledgeBaseVectorService knowledgeBaseVectorService; |
| | | private final StorageBlobService storageBlobService; |
| | | private final EmbeddingModel embeddingModel; |
| | | private final EmbeddingStore<TextSegment> embeddingStore; |
| | | private final FileProperties fileProperties; |
| | | private final Index pineconeIndex; |
| | | |
| | | @Value("${pinecone.namespace}") |
| | | private String namespace; |
| | | |
| | | private static final int CHUNK_SIZE = 500; |
| | | private static final int CHUNK_OVERLAP = 100; |
| | | private static final long CHUNK_THRESHOLD_BYTES = 80L * 1024 * 1024; |
| | | private static final int EMBEDDING_MAX_LENGTH = 8000; |
| | | |
| | | @Override |
| | | @Async("threadPoolTaskExecutor") |
| | | public void processVectorAsync(Long vectorId) { |
| | | processVector(vectorId); |
| | | } |
| | | |
| | | @Override |
| | | public void processVector(Long vectorId) { |
| | | KnowledgeBaseVector vector = knowledgeBaseVectorService.getById(vectorId); |
| | | if (vector == null) return; |
| | | |
| | | try { |
| | | // æ´æ°ç¶æä¸ºå¤çä¸ |
| | | knowledgeBaseVectorService.updateVectorStatus(vectorId, STATUS_PROCESSING, null, null); |
| | | |
| | | // è·åæä»¶å
容 |
| | | StorageBlob blob = storageBlobService.getById(vector.getStorageBlobId()); |
| | | File file = getFile(blob); |
| | | String content = extractFileContent(file, vector.getFileName()); |
| | | |
| | | if (content == null || content.trim().isEmpty()) { |
| | | throw new RuntimeException("æä»¶å
容为空"); |
| | | } |
| | | |
| | | // ææ¬åç |
| | | List<TextSegment> chunks; |
| | | boolean needChunk = file.length() > CHUNK_THRESHOLD_BYTES || content.length() > EMBEDDING_MAX_LENGTH; |
| | | if (needChunk) { |
| | | chunks = splitText(content, vector); |
| | | } else { |
| | | Map<String, Object> metadata = buildMetadata(vector); |
| | | chunks = List.of(TextSegment.from(content, new Metadata(metadata))); |
| | | } |
| | | |
| | | // çæåµå
¥åéå¹¶åå¨ |
| | | int chunkCount = 0; |
| | | for (TextSegment chunk : chunks) { |
| | | Embedding embedding = embeddingModel.embed(chunk).content(); |
| | | embeddingStore.add(embedding, chunk); |
| | | chunkCount++; |
| | | } |
| | | |
| | | // æ´æ°ç¶æä¸ºå®æ |
| | | knowledgeBaseVectorService.updateVectorStatus(vectorId, STATUS_COMPLETED, chunkCount, null); |
| | | |
| | | } catch (Exception e) { |
| | | log.error("åéåå¤ç失败", e); |
| | | knowledgeBaseVectorService.updateVectorStatus(vectorId, STATUS_FAILED, null, e.getMessage()); |
| | | } |
| | | } |
| | | |
| | | @Override |
| | | public List<String> searchRelevantContent(String namespace, String query, int maxResults) { |
| | | Embedding queryEmbedding = embeddingModel.embed(query).content(); |
| | | EmbeddingSearchRequest searchRequest = EmbeddingSearchRequest.builder() |
| | | .queryEmbedding(queryEmbedding) |
| | | .maxResults(maxResults) |
| | | .minScore(0.7) |
| | | .build(); |
| | | EmbeddingSearchResult<TextSegment> searchResult = embeddingStore.search(searchRequest); |
| | | return searchResult.matches().stream() |
| | | .map(match -> match.embedded().text()) |
| | | .collect(Collectors.toList()); |
| | | } |
| | | |
| | | @Override |
| | | public void deleteEmbeddings(String namespace, Long storageBlobId) { |
| | | Struct filter = Struct.newBuilder() |
| | | .putFields("storageBlobId", Value.newBuilder() |
| | | .setStructValue(Struct.newBuilder() |
| | | .putFields("$eq", Value.newBuilder() |
| | | .setNumberValue(storageBlobId.doubleValue()) |
| | | .build())) |
| | | .build()) |
| | | .build(); |
| | | pineconeIndex.delete(new ArrayList<>(), false, this.namespace, filter); |
| | | } |
| | | |
| | | private String extractFileContent(File file, String fileName) throws Exception { |
| | | String ext = getFileExtension(fileName); |
| | | if (isPlainText(ext)) { |
| | | return readFileWithEncoding(file); |
| | | } |
| | | if ("docx".equals(ext)) { |
| | | return extractDocx(file); |
| | | } |
| | | if ("xlsx".equals(ext) || "xls".equals(ext)) { |
| | | return extractExcel(file); |
| | | } |
| | | return readFileWithEncoding(file); |
| | | } |
| | | |
| | | // ... å
¶ä»è¾
婿¹æ³ |
| | | } |
| | | ``` |
| | | |
| | | #### 3.4.4 ç¥è¯åºé®çAgent |
| | | |
| | | **KnowledgeChatAgent.java** |
| | | ```java |
| | | @AiService( |
| | | wiringMode = EXPLICIT, |
| | | streamingChatModel = "qwenStreamingChatModel", |
| | | chatMemoryProvider = "chatMemoryProvider" |
| | | ) |
| | | public interface KnowledgeChatAgent { |
| | | |
| | | @SystemMessage(""" |
| | | ä½ æ¯ä¼ä¸ç¥è¯åºé®ç婿ã |
| | | ä½ éè¦åºäºæä¾çç¥è¯åºå
容åçç¨æ·é®é¢ã |
| | | éµå¾ªä»¥ä¸è§åï¼ |
| | | 1. ä¸¥æ ¼åºäºç¥è¯åºå
容åçï¼ä¸è¦ç¼é ä¿¡æ¯ |
| | | 2. 妿ç¥è¯åºä¸æ²¡æç¸å
³ä¿¡æ¯ï¼æç¡®åç¥ç¨æ· |
| | | 3. åçè¦åç¡®ãç®æ´ãææ¡ç |
| | | 4. å¼ç¨æ¥æºæ¶æ³¨æ"æ ¹æ®ç¥è¯åºå
容" |
| | | """) |
| | | Flux<String> chat(@MemoryId String memoryId, @UserMessage String userMessage); |
| | | } |
| | | ``` |
| | | |
| | | #### 3.4.5 Controllerå± |
| | | |
| | | **KnowledgeBaseController.java** |
| | | ```java |
| | | @RestController |
| | | @RequestMapping("/knowledgeBase") |
| | | @Tag(name = "ç¥è¯åºç®¡ç") |
| | | public class KnowledgeBaseController { |
| | | |
| | | @GetMapping("/getList") |
| | | public AjaxResult getList(@RequestParam(defaultValue = "1") long current, |
| | | @RequestParam(defaultValue = "10") long size, |
| | | KnowledgeBase knowledgeBase) { |
| | | Page page = new Page(current, size); |
| | | return AjaxResult.success(knowledgeBaseService.listpage(page, knowledgeBase)); |
| | | } |
| | | |
| | | @PostMapping("/add") |
| | | public AjaxResult add(@RequestBody KnowledgeBase knowledgeBase) { |
| | | return AjaxResult.success(knowledgeBaseService.save(knowledgeBase)); |
| | | } |
| | | |
| | | @PostMapping("/update") |
| | | public AjaxResult update(@RequestBody KnowledgeBase knowledgeBase) { |
| | | return AjaxResult.success(knowledgeBaseService.updateById(knowledgeBase)); |
| | | } |
| | | |
| | | @DeleteMapping("/delete") |
| | | public AjaxResult delete(@RequestBody List<Long> ids) { |
| | | return AjaxResult.success(knowledgeBaseService.removeByIds(ids)); |
| | | } |
| | | |
| | | @GetMapping("/vector/status/{knowledgeBaseId}") |
| | | @Operation(summary = "æ¥è¯¢ç¥è¯åºæä»¶åéåç¶æ") |
| | | public AjaxResult getVectorStatus(@PathVariable Long knowledgeBaseId) { |
| | | return AjaxResult.success(knowledgeBaseVectorService.getVectorStatusByKnowledgeBaseId(knowledgeBaseId)); |
| | | } |
| | | |
| | | @PostMapping("/vector/reprocess/{vectorId}") |
| | | @Operation(summary = "éæ°åéåæä»¶") |
| | | public AjaxResult reprocessVector(@PathVariable Long vectorId) { |
| | | knowledgeBaseVectorService.reprocessVector(vectorId); |
| | | return AjaxResult.success("已鿰æäº¤åéåä»»å¡"); |
| | | } |
| | | |
| | | @PostMapping("/file/save") |
| | | @Operation(summary = "ä¿åç¥è¯åºæä»¶å
³è") |
| | | public AjaxResult saveKnowledgeBaseFiles(@RequestBody KnowledgeBaseFileDTO dto) { |
| | | // ä¿åéä»¶å
³è并触ååéå |
| | | // ... |
| | | } |
| | | |
| | | @DeleteMapping("/file/delete") |
| | | @Operation(summary = "å é¤ç¥è¯åºæä»¶") |
| | | public AjaxResult deleteKnowledgeBaseFiles(@RequestBody List<Long> vectorIds) { |
| | | knowledgeBaseVectorService.deleteVectors(vectorIds); |
| | | return AjaxResult.success(); |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | **KnowledgeChatController.java** |
| | | ```java |
| | | @RestController |
| | | @RequestMapping("/ai/knowledge") |
| | | @Tag(name = "ç¥è¯åºé®ç") |
| | | public class KnowledgeChatController { |
| | | |
| | | private final KnowledgeChatAgent knowledgeChatAgent; |
| | | private final KnowledgeRagService knowledgeRagService; |
| | | private final KnowledgeBaseService knowledgeBaseService; |
| | | |
| | | @PostMapping(value = "/chat", produces = "text/stream;charset=utf-8") |
| | | @Operation(summary = "ç¥è¯åºé®ç") |
| | | public Flux<String> chat(@RequestBody KnowledgeChatRequest request) { |
| | | // æ£ç´¢ç¸å
³å
容 |
| | | String namespace = "kb-" + request.getKnowledgeBaseId(); |
| | | List<String> relevantContents = knowledgeRagService.searchRelevantContent( |
| | | namespace, request.getQuestion(), 5); |
| | | |
| | | if (relevantContents.isEmpty()) { |
| | | return Flux.just("ç¥è¯åºä¸æªæ¾å°ç¸å
³å
容"); |
| | | } |
| | | |
| | | // æå»ºä¸ä¸æ |
| | | StringBuilder context = new StringBuilder(); |
| | | context.append("以䏿¯ä»ç¥è¯åºä¸æ£ç´¢å°çç¸å
³å
容ï¼\n\n"); |
| | | for (int i = 0; i < relevantContents.size(); i++) { |
| | | context.append("ãå
容").append(i + 1).append("ã\n"); |
| | | context.append(relevantContents.get(i)).append("\n\n"); |
| | | } |
| | | context.append("---\n请åºäºä»¥ä¸ç¥è¯åºå
容åçï¼\n").append(request.getQuestion()); |
| | | |
| | | return knowledgeChatAgent.chat(request.getMemoryId(), context.toString()); |
| | | } |
| | | |
| | | @GetMapping("/list") |
| | | @Operation(summary = "ç¥è¯åºå表") |
| | | public AjaxResult listKnowledgeBases() { |
| | | return AjaxResult.success(knowledgeBaseService.list()); |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ## åãAPIæ¥å£ææ¡£ |
| | | |
| | | ### 4.1 ç¥è¯åºç®¡çæ¥å£ |
| | | |
| | | | æ¥å£ | æ¹æ³ | è·¯å¾ | 说æ | |
| | | |------|------|------|------| |
| | | | è·åå表 | GET | /knowledgeBase/getList | å页æ¥è¯¢ç¥è¯åºå表 | |
| | | | æ°å¢ç¥è¯åº | POST | /knowledgeBase/add | å建ç¥è¯åº | |
| | | | æ´æ°ç¥è¯åº | POST | /knowledgeBase/update | æ´æ°ç¥è¯åºä¿¡æ¯ | |
| | | | å é¤ç¥è¯åº | DELETE | /knowledgeBase/delete | æ¹éå é¤ç¥è¯åº | |
| | | | æ¥è¯¢åéåç¶æ | GET | /knowledgeBase/vector/status/{id} | æ¥è¯¢æä»¶åéåç¶æ | |
| | | | éæ°åéå | POST | /knowledgeBase/vector/reprocess/{id} | éæ°å¤ç失败çæä»¶ | |
| | | | ä¿åæä»¶å
³è | POST | /knowledgeBase/file/save | ä¸ä¼ æä»¶åå
³èå°ç¥è¯åº | |
| | | | å 餿件 | DELETE | /knowledgeBase/file/delete | å é¤ç¥è¯åºæä»¶ | |
| | | |
| | | ### 4.2 ç¥è¯åºé®çæ¥å£ |
| | | |
| | | | æ¥å£ | æ¹æ³ | è·¯å¾ | 说æ | |
| | | |------|------|------|------| |
| | | | ç¥è¯åºé®ç | POST | /ai/knowledge/chat | æµå¼è¿åé®çç»æ | |
| | | | ç¥è¯åºå表 | GET | /ai/knowledge/list | è·åå¯éç¥è¯åºå表 | |
| | | |
| | | ### 4.3 æ¥å£è¯¦ç»è¯´æ |
| | | |
| | | #### 4.3.1 ä¿åç¥è¯åºæä»¶å
³è |
| | | |
| | | **请æ±** |
| | | ```json |
| | | POST /knowledgeBase/file/save |
| | | { |
| | | "knowledgeBaseId": 1, |
| | | "storageBlobIds": [100, 101, 102] |
| | | } |
| | | ``` |
| | | |
| | | **ååº** |
| | | ```json |
| | | { |
| | | "code": 200, |
| | | "msg": "æä½æå" |
| | | } |
| | | ``` |
| | | |
| | | #### 4.3.2 ç¥è¯åºé®ç |
| | | |
| | | **请æ±** |
| | | ```json |
| | | POST /ai/knowledge/chat |
| | | Content-Type: application/json |
| | | |
| | | { |
| | | "knowledgeBaseId": 1, |
| | | "memoryId": "session-uuid", |
| | | "question": "å¦ä½å¤çåºåçç¹å·®å¼ï¼" |
| | | } |
| | | ``` |
| | | |
| | | **ååº**ï¼SSEæµå¼ï¼ |
| | | ``` |
| | | æ ¹æ®ç¥è¯åºå
容ï¼åºåçç¹å·®å¼çå¤çæµç¨å¦ä¸ï¼ |
| | | |
| | | 1. åç°å·®å¼åï¼é¦å
æ ¸å¯¹çç¹è®°å½... |
| | | 2. æ£æ¥æ¯å¦ææ¼çæéç... |
| | | 3. ... |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ## äºãå端å®ç° |
| | | |
| | | ### 5.1 ç¥è¯åºç®¡çé¡µé¢ |
| | | |
| | | ```vue |
| | | <template> |
| | | <div class="knowledge-base"> |
| | | <!-- å表 --> |
| | | <el-table :data="tableData" border> |
| | | <el-table-column prop="title" label="ç¥è¯æ é¢" /> |
| | | <el-table-column prop="type" label="ç¥è¯ç±»å" /> |
| | | <el-table-column prop="fileCount" label="æä»¶æ°é" /> |
| | | <el-table-column prop="totalChunkCount" label="åçæ°é" /> |
| | | <el-table-column label="æä½"> |
| | | <template #default="{ row }"> |
| | | <el-button @click="handleEdit(row)">ç¼è¾</el-button> |
| | | <el-button @click="handleFiles(row)">æä»¶ç®¡ç</el-button> |
| | | <el-button @click="handleChat(row)">é®ç</el-button> |
| | | <el-button type="danger" @click="handleDelete(row)">å é¤</el-button> |
| | | </template> |
| | | </el-table-column> |
| | | </el-table> |
| | | </div> |
| | | </template> |
| | | |
| | | <script setup> |
| | | import { ref, onMounted } from 'vue' |
| | | import { getKnowledgeBaseList, deleteKnowledgeBase } from '@/api/knowledge' |
| | | |
| | | const tableData = ref([]) |
| | | |
| | | const loadData = async () => { |
| | | const res = await getKnowledgeBaseList({ current: 1, size: 10 }) |
| | | tableData.value = res.data.records |
| | | } |
| | | |
| | | onMounted(loadData) |
| | | </script> |
| | | ``` |
| | | |
| | | ### 5.2 æä»¶ä¸ä¼ ä¸åéåç¶æ |
| | | |
| | | ```vue |
| | | <template> |
| | | <div class="file-manager"> |
| | | <!-- æä»¶ä¸ä¼ --> |
| | | <el-upload |
| | | :action="uploadUrl" |
| | | :on-success="handleUploadSuccess" |
| | | multiple |
| | | > |
| | | <el-button type="primary">ä¸ä¼ æä»¶</el-button> |
| | | </el-upload> |
| | | |
| | | <!-- æä»¶å表ä¸åéåç¶æ --> |
| | | <el-table :data="fileList"> |
| | | <el-table-column prop="fileName" label="æä»¶å" /> |
| | | <el-table-column label="åéåç¶æ"> |
| | | <template #default="{ row }"> |
| | | <el-tag :type="getStatusType(row.vectorStatus)"> |
| | | {{ getStatusText(row.vectorStatus) }} |
| | | </el-tag> |
| | | </template> |
| | | </el-table-column> |
| | | <el-table-column prop="chunkCount" label="åçæ°" /> |
| | | <el-table-column label="æä½"> |
| | | <template #default="{ row }"> |
| | | <el-button v-if="row.vectorStatus === 3" @click="reprocess(row)"> |
| | | éæ°å¤ç |
| | | </el-button> |
| | | <el-button type="danger" @click="deleteFile(row)">å é¤</el-button> |
| | | </template> |
| | | </el-table-column> |
| | | </el-table> |
| | | </div> |
| | | </template> |
| | | |
| | | <script setup> |
| | | const uploadUrl = import.meta.env.VITE_APP_BASE_API + '/common/upload' |
| | | |
| | | // ä¸ä¼ æååä¿åå
³è |
| | | const uploadedBlobIds = ref([]) |
| | | |
| | | const handleUploadSuccess = (response, file) => { |
| | | if (response.code === 200) { |
| | | uploadedBlobIds.value.push(response.data.id) |
| | | } |
| | | } |
| | | |
| | | // ä¿åæä»¶å
³è |
| | | const saveFiles = async () => { |
| | | await saveKnowledgeBaseFiles({ |
| | | knowledgeBaseId: props.knowledgeBaseId, |
| | | storageBlobIds: uploadedBlobIds.value |
| | | }) |
| | | // å·æ°æä»¶å表 |
| | | loadFileList() |
| | | } |
| | | |
| | | // ç¶æææ¬æ å° |
| | | const getStatusText = (status) => { |
| | | const map = { |
| | | 0: 'å¾
å¤ç', |
| | | 1: 'å¤çä¸', |
| | | 2: '已宿', |
| | | 3: '失败' |
| | | } |
| | | return map[status] || 'æªç¥' |
| | | } |
| | | |
| | | const getStatusType = (status) => { |
| | | const map = { |
| | | 0: 'info', |
| | | 1: 'warning', |
| | | 2: 'success', |
| | | 3: 'danger' |
| | | } |
| | | return map[status] || 'info' |
| | | } |
| | | </script> |
| | | ``` |
| | | |
| | | ### 5.3 ç¥è¯åºé®ççé¢ |
| | | |
| | | ```vue |
| | | <template> |
| | | <div class="knowledge-chat"> |
| | | <!-- ç¥è¯åºéæ© --> |
| | | <el-select v-model="selectedKbId" placeholder="éæ©ç¥è¯åº"> |
| | | <el-option |
| | | v-for="kb in knowledgeBases" |
| | | :key="kb.id" |
| | | :label="kb.title" |
| | | :value="kb.id" |
| | | /> |
| | | </el-select> |
| | | |
| | | <!-- 对è¯åºå --> |
| | | <div class="chat-messages"> |
| | | <div |
| | | v-for="(msg, index) in messages" |
| | | :key="index" |
| | | :class="['message', msg.role]" |
| | | > |
| | | <div class="content">{{ msg.content }}</div> |
| | | </div> |
| | | </div> |
| | | |
| | | <!-- è¾å
¥æ¡ --> |
| | | <el-input |
| | | v-model="inputQuestion" |
| | | placeholder="请è¾å
¥é®é¢" |
| | | @keyup.enter="sendMessage" |
| | | > |
| | | <template #append> |
| | | <el-button @click="sendMessage" :loading="loading">åé</el-button> |
| | | </template> |
| | | </el-input> |
| | | </div> |
| | | </template> |
| | | |
| | | <script setup> |
| | | import { ref, onMounted } from 'vue' |
| | | import { getKnowledgeBaseList, knowledgeChat } from '@/api/knowledge' |
| | | |
| | | const knowledgeBases = ref([]) |
| | | const selectedKbId = ref(null) |
| | | const messages = ref([]) |
| | | const inputQuestion = ref('') |
| | | const loading = ref(false) |
| | | const memoryId = ref(crypto.randomUUID()) |
| | | |
| | | const sendMessage = async () => { |
| | | if (!inputQuestion.value.trim()) return |
| | | if (!selectedKbId.value) { |
| | | ElMessage.warning('è¯·éæ©ç¥è¯åº') |
| | | return |
| | | } |
| | | |
| | | // æ·»å ç¨æ·æ¶æ¯ |
| | | messages.value.push({ |
| | | role: 'user', |
| | | content: inputQuestion.value |
| | | }) |
| | | |
| | | loading.value = true |
| | | |
| | | try { |
| | | // æµå¼è¯·æ± |
| | | const response = await fetch('/api/ai/knowledge/chat', { |
| | | method: 'POST', |
| | | headers: { 'Content-Type': 'application/json' }, |
| | | body: JSON.stringify({ |
| | | knowledgeBaseId: selectedKbId.value, |
| | | memoryId: memoryId.value, |
| | | question: inputQuestion.value |
| | | }) |
| | | }) |
| | | |
| | | // å¤çSSEæµå¼ååº |
| | | const reader = response.body.getReader() |
| | | const decoder = new TextDecoder() |
| | | let aiContent = '' |
| | | |
| | | messages.value.push({ role: 'assistant', content: '' }) |
| | | |
| | | while (true) { |
| | | const { done, value } = await reader.read() |
| | | if (done) break |
| | | |
| | | const text = decoder.decode(value) |
| | | aiContent += text |
| | | messages.value[messages.value.length - 1].content = aiContent |
| | | } |
| | | } finally { |
| | | loading.value = false |
| | | inputQuestion.value = '' |
| | | } |
| | | } |
| | | |
| | | onMounted(async () => { |
| | | const res = await getKnowledgeBaseList() |
| | | knowledgeBases.value = res.data |
| | | }) |
| | | </script> |
| | | ``` |
| | | |
| | | ### 5.4 APIå°è£
|
| | | |
| | | ```javascript |
| | | // api/knowledge.js |
| | | import request from '@/utils/request' |
| | | |
| | | // è·åç¥è¯åºå表 |
| | | export function getKnowledgeBaseList(params) { |
| | | return request({ |
| | | url: '/knowledgeBase/getList', |
| | | method: 'get', |
| | | params |
| | | }) |
| | | } |
| | | |
| | | // æ°å¢ç¥è¯åº |
| | | export function addKnowledgeBase(data) { |
| | | return request({ |
| | | url: '/knowledgeBase/add', |
| | | method: 'post', |
| | | data |
| | | }) |
| | | } |
| | | |
| | | // æ´æ°ç¥è¯åº |
| | | export function updateKnowledgeBase(data) { |
| | | return request({ |
| | | url: '/knowledgeBase/update', |
| | | method: 'post', |
| | | data |
| | | }) |
| | | } |
| | | |
| | | // å é¤ç¥è¯åº |
| | | export function deleteKnowledgeBase(ids) { |
| | | return request({ |
| | | url: '/knowledgeBase/delete', |
| | | method: 'delete', |
| | | data: ids |
| | | }) |
| | | } |
| | | |
| | | // è·åæä»¶åéåç¶æ |
| | | export function getVectorStatus(knowledgeBaseId) { |
| | | return request({ |
| | | url: `/knowledgeBase/vector/status/${knowledgeBaseId}`, |
| | | method: 'get' |
| | | }) |
| | | } |
| | | |
| | | // éæ°åéå |
| | | export function reprocessVector(vectorId) { |
| | | return request({ |
| | | url: `/knowledgeBase/vector/reprocess/${vectorId}`, |
| | | method: 'post' |
| | | }) |
| | | } |
| | | |
| | | // ä¿åæä»¶å
³è |
| | | export function saveKnowledgeBaseFiles(data) { |
| | | return request({ |
| | | url: '/knowledgeBase/file/save', |
| | | method: 'post', |
| | | data |
| | | }) |
| | | } |
| | | |
| | | // å 餿件 |
| | | export function deleteKnowledgeBaseFiles(vectorIds) { |
| | | return request({ |
| | | url: '/knowledgeBase/file/delete', |
| | | method: 'delete', |
| | | data: vectorIds |
| | | }) |
| | | } |
| | | |
| | | // ç¥è¯åºé®çï¼æµå¼ï¼ |
| | | export async function knowledgeChat(data) { |
| | | const response = await fetch('/api/ai/knowledge/chat', { |
| | | method: 'POST', |
| | | headers: { 'Content-Type': 'application/json' }, |
| | | body: JSON.stringify(data) |
| | | }) |
| | | return response.body |
| | | } |
| | | |
| | | // è·åç¥è¯åºå表ï¼é®çç¨ï¼ |
| | | export function getKnowledgeBaseListForChat() { |
| | | return request({ |
| | | url: '/ai/knowledge/list', |
| | | method: 'get' |
| | | }) |
| | | } |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ## å
ãæ ¸å¿æµç¨ |
| | | |
| | | ### 6.1 æä»¶ä¸ä¼ ä¸åéåæµç¨ |
| | | |
| | | ``` |
| | | 1. å端è°ç¨ /common/upload ä¸ä¼ æä»¶ â è¿å storageBlobId |
| | | 2. å端è°ç¨ /knowledgeBase/file/save å
³èæä»¶å°ç¥è¯åº |
| | | 3. å端å建 KnowledgeBaseVector è®°å½ï¼ç¶æï¼å¾
å¤çï¼ |
| | | 4. åç«¯å¼æ¥è°ç¨ KnowledgeRagService.processVectorAsync() |
| | | âââ æ´æ°ç¶æä¸º"å¤çä¸" |
| | | âââ æåæä»¶å
å®¹ï¼æ¯æå¤ç§æ ¼å¼ï¼ |
| | | âââ èªå¨æ£æµæä»¶ç¼ç ï¼UTF-8/GBKï¼ |
| | | âââ ææ¬åçï¼å¤§æä»¶æé¿å
容æåçï¼ |
| | | âââ çæ Embedding åé |
| | | âââ åå¨å° Pinecone |
| | | âââ æ´æ°ç¶æä¸º"宿"æ"失败" |
| | | ``` |
| | | |
| | | ### 6.2 ç¥è¯åºé®çæµç¨ |
| | | |
| | | ``` |
| | | 1. ç¨æ·éæ©ç¥è¯åºï¼è¾å
¥é®é¢ |
| | | 2. å端è°ç¨ /ai/knowledge/chatï¼æµå¼æ¥å£ï¼ |
| | | 3. å端å¤çï¼ |
| | | âââ æå»ºå½å空é´ï¼kb-{knowledgeBaseId} |
| | | âââ è°ç¨ Embedding 模åçæé®é¢åé |
| | | âââ ä» Pinecone æ£ç´¢ç¸å
³å
容ï¼minScore=0.7, maxResults=5ï¼ |
| | | âââ æå»ºä¸ä¸æ Prompt |
| | | âââ è°ç¨ LLM çæåç |
| | | âââ æµå¼è¿åç»æ |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ## ä¸ã注æäºé¡¹ |
| | | |
| | | 1. **Pinecone å½å空é´**ï¼ä¸è½ä½¿ç¨ `__default__`ï¼å¿
须使ç¨èªå®ä¹å½åç©ºé´ |
| | | 2. **æä»¶ç¼ç **ï¼èªå¨æ£æµ UTF-8/GBKï¼é¿å
ä¹±ç |
| | | 3. **åççç¥**ï¼ |
| | | - æä»¶ > 80MB æå
容 > 8000 åç¬¦æ¶æåç |
| | | - åçå¤§å° 500 å符ï¼éå 100 å符 |
| | | - ä¼å
å¨å¥åè¾¹çåå |
| | | 4. **Embedding éå¶**ï¼é¿éäº DashScope éå¶å次è¾å
¥æå¤§ 8192 å符 |
| | | 5. **åéå é¤**ï¼ä½¿ç¨ Pinecone åç客æ·ç«¯ï¼éè¿ metadata filter å é¤ |
| | | 6. **弿¥å¤ç**ï¼åéåä½¿ç¨ `@Async` 弿¥æ§è¡ï¼é¿å
é»å¡æ¥å£ |
| | | |
| | | --- |
| | | |
| | | ## å
«ãæä»¶æ¸
å |
| | | |
| | | ### å端æä»¶ |
| | | ``` |
| | | src/main/java/com/ruoyi/ |
| | | âââ approve/ |
| | | â âââ controller/ |
| | | â â âââ KnowledgeBaseController.java |
| | | â âââ pojo/ |
| | | â â âââ KnowledgeBase.java |
| | | â â âââ KnowledgeBaseVector.java |
| | | â âââ service/ |
| | | â â âââ KnowledgeBaseService.java |
| | | â â âââ KnowledgeBaseVectorService.java |
| | | â â âââ impl/ |
| | | â â âââ KnowledgeBaseServiceImpl.java |
| | | â â âââ KnowledgeBaseVectorServiceImpl.java |
| | | â âââ mapper/ |
| | | â â âââ KnowledgeBaseMapper.java |
| | | â â âââ KnowledgeBaseVectorMapper.java |
| | | â âââ dto/ |
| | | â âââ KnowledgeBaseVectorVO.java |
| | | âââ ai/ |
| | | âââ config/ |
| | | â âââ EmbeddingStoreConfig.java |
| | | â âââ XiaozhiAgentConfig.java |
| | | âââ controller/ |
| | | â âââ KnowledgeChatController.java |
| | | âââ assistant/ |
| | | â âââ KnowledgeChatAgent.java |
| | | âââ service/ |
| | | â âââ KnowledgeRagService.java |
| | | â âââ impl/ |
| | | â âââ KnowledgeRagServiceImpl.java |
| | | âââ dto/ |
| | | âââ KnowledgeChatRequest.java |
| | | ``` |
| | | |
| | | ### å端æä»¶ |
| | | ``` |
| | | src/views/knowledge/ |
| | | âââ index.vue # ç¥è¯åºå表 |
| | | âââ form.vue # æ°å¢/ç¼è¾è¡¨å |
| | | âââ files.vue # æä»¶ç®¡ç |
| | | âââ chat.vue # ç¥è¯åºé®ç |
| | | |
| | | src/api/knowledge.js # APIå°è£
|
| | | ``` |