提交
This commit is contained in:
420
docs/plans/2026-03-09-bank-statement-duplicate-check.md
Normal file
420
docs/plans/2026-03-09-bank-statement-duplicate-check.md
Normal file
@@ -0,0 +1,420 @@
|
||||
# 银行流水重复校验 Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** 为 `fetchAndSaveBankStatements(...)` 增加基于 `project_id + LE_ACCOUNT_NO + ACCOUNTING_DATE_ID + AMOUNT_DR + AMOUNT_CR` 的数据库级重复校验，确保重复流水跳过插入且保留原数据不变。
|
||||
|
||||
**Architecture:** 先清理测试库中的异常行和重复行，再为 `ccdi_bank_statement` 增加新唯一键。业务代码只负责对 `LE_ACCOUNT_NO` 做轻量标准化，并将批量插入改为 no-op upsert；真实数据库错误仍按现有异步文件处理链路向上抛出。
|
||||
|
||||
**Tech Stack:** Java 21, Spring Boot 3, MyBatis Plus, MySQL, JUnit 5, Mockito
|
||||
|
||||
---
|
||||
|
||||
### Task 1: 固化新去重键的标准化失败测试
|
||||
|
||||
**Files:**
|
||||
- Modify: `ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java`
|
||||
- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java`
|
||||
|
||||
**Step 1: Write the failing test**
|
||||
|
||||
在现有 `CcdiFileUploadServiceImplTest` 中新增一个聚焦标准化的测试。通过 mock `lsfxClient.getBankStatement(...)` 返回一条带空白账号的流水，并捕获传给 `bankStatementMapper.insertBatch(...)` 的实体。
|
||||
|
||||
```java
|
||||
@Test
|
||||
void fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert() {
|
||||
// arrange
|
||||
// leAccountNo = " 62220001 "
|
||||
// accountingDateId = 20260310
|
||||
// amountDr = new BigDecimal("100.00")
|
||||
// amountCr = new BigDecimal("0.00")
|
||||
// act
|
||||
// assert
|
||||
verify(bankStatementMapper).insertBatch(argThat(list ->
|
||||
"62220001".equals(list.get(0).getLeAccountNo())));
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- FAIL
|
||||
- 失败原因体现当前实现尚未对 `LE_ACCOUNT_NO` 做 `trim`
|
||||
|
||||
**Step 3: Write the minimal implementation**
|
||||
|
||||
在 `CcdiFileUploadServiceImpl` 内增加最小辅助方法：
|
||||
|
||||
```java
|
||||
private String trimAccountNo(String value) {
|
||||
return value == null ? null : value.trim();
|
||||
}
|
||||
|
||||
private void normalizeDedupFields(CcdiBankStatement statement) {
|
||||
statement.setLeAccountNo(trimAccountNo(statement.getLeAccountNo()));
|
||||
}
|
||||
```
|
||||
|
||||
并在 `fromResponse(...)` 结果加入批次列表前调用。
|
||||
|
||||
**Step 4: Run test to verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java
|
||||
git commit -m "test(ccdi-project): cover bank statement account no normalization"
|
||||
```
|
||||
|
||||
### Task 2: 用失败测试锁定“重复不失败”的服务层语义
|
||||
|
||||
**Files:**
|
||||
- Modify: `ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java`
|
||||
- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java`
|
||||
|
||||
**Step 1: Write the failing test**
|
||||
|
||||
新增一个测试，模拟 Mapper 在重复导入场景下不抛异常，并验证上传记录最终不会因为重复而进入 `parsed_failed`。
|
||||
|
||||
```java
|
||||
@Test
|
||||
void processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped() {
|
||||
// mock upload success / parse success
|
||||
// mock bankStatementMapper.insertBatch(...) 返回“未报错的重复跳过结果”
|
||||
// assert recordMapper 不会收到 parsed_failed
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- FAIL
|
||||
- 失败点应体现当前实现还没有为重复跳过设计明确语义或日志
|
||||
|
||||
**Step 3: Write minimal implementation**
|
||||
|
||||
只做本任务所需的最小改动：
|
||||
|
||||
- 在 `fetchAndSaveBankStatements(...)` 中补充接口返回数、尝试写入数的日志
|
||||
- 保持 duplicate 场景不抛异常
|
||||
- 不调整其他与重复无关的异常路径
|
||||
|
||||
示例日志：
|
||||
|
||||
```java
|
||||
log.info("【文件上传】流水入库完成: fetchedCount={}, attemptedCount={}, insertedCount={}, duplicateCount={}",
|
||||
fetchedCount, attemptedCount, insertedCount, duplicateCount);
|
||||
```
|
||||
|
||||
**Step 4: Run the test to verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java
|
||||
git commit -m "test(ccdi-project): keep duplicate bank statements from failing uploads"
|
||||
```
|
||||
|
||||
### Task 3: 为测试库编写数据清洗和唯一键迁移脚本
|
||||
|
||||
**Files:**
|
||||
- Create: `assets/database/2026-03-10-bank-statement-dedup.sql`
|
||||
- Modify: `assets/对接流水分析/ccdi_bank_statement.md`
|
||||
|
||||
**Step 1: Write the migration script**
|
||||
|
||||
创建测试库迁移脚本，包含以下 SQL：
|
||||
|
||||
```sql
|
||||
DELETE FROM ccdi_bank_statement
|
||||
WHERE project_id IS NULL
|
||||
OR LE_ACCOUNT_NO IS NULL
|
||||
OR ACCOUNTING_DATE_ID IS NULL;
|
||||
|
||||
UPDATE ccdi_bank_statement
|
||||
SET LE_ACCOUNT_NO = TRIM(LE_ACCOUNT_NO);
|
||||
|
||||
DELETE t1
|
||||
FROM ccdi_bank_statement t1
|
||||
JOIN ccdi_bank_statement t2
|
||||
ON t1.bank_statement_id > t2.bank_statement_id
|
||||
AND t1.project_id = t2.project_id
|
||||
AND t1.LE_ACCOUNT_NO = t2.LE_ACCOUNT_NO
|
||||
AND t1.ACCOUNTING_DATE_ID = t2.ACCOUNTING_DATE_ID
|
||||
AND t1.AMOUNT_DR = t2.AMOUNT_DR
|
||||
AND t1.AMOUNT_CR = t2.AMOUNT_CR;
|
||||
|
||||
ALTER TABLE ccdi_bank_statement
|
||||
MODIFY COLUMN project_id bigint(20) NOT NULL COMMENT '关联项目ID',
|
||||
MODIFY COLUMN LE_ACCOUNT_NO varchar(240) NOT NULL DEFAULT '' COMMENT '企业银行账号',
|
||||
MODIFY COLUMN ACCOUNTING_DATE_ID int(11) NOT NULL COMMENT '账号日期ID';
|
||||
|
||||
ALTER TABLE ccdi_bank_statement
|
||||
ADD UNIQUE KEY uk_bank_statement_dedup (
|
||||
project_id,
|
||||
LE_ACCOUNT_NO,
|
||||
ACCOUNTING_DATE_ID,
|
||||
AMOUNT_DR,
|
||||
AMOUNT_CR
|
||||
);
|
||||
```
|
||||
|
||||
**Step 2: Manually run the migration in the local test database**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mysql -u <user> -p <database> < assets/database/2026-03-10-bank-statement-dedup.sql
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- SQL 全部执行成功
|
||||
- `SHOW INDEX FROM ccdi_bank_statement;` 能看到 `uk_bank_statement_dedup`
|
||||
|
||||
**Step 3: Update the bank statement schema note**
|
||||
|
||||
同步更新 `assets/对接流水分析/ccdi_bank_statement.md`，补齐：
|
||||
|
||||
- `project_id`
|
||||
- 新唯一键说明
|
||||
- `project_id` / `LE_ACCOUNT_NO` / `ACCOUNTING_DATE_ID` 的非空语义
|
||||
|
||||
**Step 4: Verify migration result**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mysql -u <user> -p -e "SHOW CREATE TABLE ccdi_bank_statement\G"
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- 表结构包含新的唯一键
|
||||
- `project_id`、`LE_ACCOUNT_NO`、`ACCOUNTING_DATE_ID` 为 `NOT NULL`
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add assets/database/2026-03-10-bank-statement-dedup.sql assets/对接流水分析/ccdi_bank_statement.md
|
||||
git commit -m "feat(database): add bank statement dedup unique key"
|
||||
```
|
||||
|
||||
### Task 4: 将 Mapper 批量插入改为 no-op upsert
|
||||
|
||||
**Files:**
|
||||
- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/mapper/CcdiBankStatementMapper.java`
|
||||
- Modify: `ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml`
|
||||
|
||||
**Step 1: Write the failing verification case**
|
||||
|
||||
先在本地 MySQL 中准备两次相同业务键的数据，第二次执行当前批量插入 SQL，确认现状会抛唯一键冲突。
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- FAIL 或本地重复场景仍未被正确跳过
|
||||
|
||||
**Step 2: Change the insert SQL**
|
||||
|
||||
将 XML 中的批量插入改为：
|
||||
|
||||
```xml
|
||||
<insert id="insertBatch" parameterType="java.util.List">
|
||||
insert into ccdi_bank_statement (...)
|
||||
values
|
||||
<foreach collection="list" item="item" separator=",">
|
||||
(...)
|
||||
</foreach>
|
||||
on duplicate key update
|
||||
bank_statement_id = bank_statement_id
|
||||
</insert>
|
||||
```
|
||||
|
||||
**Step 3: Keep mapper signature unchanged**
|
||||
|
||||
保持 `CcdiBankStatementMapper.insertBatch(@Param("list") List<CcdiBankStatement> list)` 不变，避免扩大调用面。
|
||||
|
||||
**Step 4: Run the targeted test suite**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
**Step 5: Manual duplicate verification**
|
||||
|
||||
在本地测试库重复导入同一批数据后执行：
|
||||
|
||||
```bash
|
||||
mysql -u <user> -p -e "SELECT COUNT(*) FROM ccdi_bank_statement WHERE project_id = <projectId> AND LE_ACCOUNT_NO = '62220001' AND ACCOUNTING_DATE_ID = 20260310 AND AMOUNT_DR = 100.00 AND AMOUNT_CR = 0.00;"
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- 结果始终为 `1`
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add ccdi-project/src/main/java/com/ruoyi/ccdi/project/mapper/CcdiBankStatementMapper.java ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml
|
||||
git commit -m "feat(ccdi-project): skip duplicate bank statements on insert"
|
||||
```
|
||||
|
||||
### Task 5: 校准日志计数并验证 MySQL 受影响行数语义
|
||||
|
||||
**Files:**
|
||||
- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java`
|
||||
- Modify: `ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java`
|
||||
|
||||
**Step 1: Add a temporary probe in the service**
|
||||
|
||||
在开发阶段先打印：
|
||||
|
||||
```java
|
||||
log.info("【文件上传】dedup probe: batchSize={}, mapperAffectedRows={}", batchList.size(), affectedRows);
|
||||
```
|
||||
|
||||
**Step 2: Run a duplicate import manually**
|
||||
|
||||
使用相同测试文件连续导入两次，并观察第二次日志。
|
||||
|
||||
Expected:
|
||||
|
||||
- 能确认 duplicate 分支下 MyBatis/JDBC 返回的 `affectedRows` 语义
|
||||
|
||||
**Step 3: Finalize the counting logic**
|
||||
|
||||
如果实测 duplicate 返回 `0`，则直接落正式逻辑：
|
||||
|
||||
```java
|
||||
insertedCount += affectedRows;
|
||||
duplicateCount += batchSize - affectedRows;
|
||||
```
|
||||
|
||||
如果实测不稳定，则不要伪造精确计数，改为保守日志：
|
||||
|
||||
```java
|
||||
log.info("【文件上传】流水入库完成: fetchedCount={}, attemptedCount={}", fetchedCount, attemptedCount);
|
||||
```
|
||||
|
||||
**Step 4: Run the test suite**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
**Step 5: Remove temporary probe and commit**
|
||||
|
||||
```bash
|
||||
git add ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java
|
||||
git commit -m "refactor(ccdi-project): finalize bank statement dedup logging"
|
||||
```
|
||||
|
||||
### Task 6: 做最终回归验证并整理交付
|
||||
|
||||
**Files:**
|
||||
- Review: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java`
|
||||
- Review: `ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml`
|
||||
- Review: `assets/database/2026-03-10-bank-statement-dedup.sql`
|
||||
- Review: `assets/对接流水分析/ccdi_bank_statement.md`
|
||||
|
||||
**Step 1: Run automated tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
**Step 2: Run a focused compile**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn clean compile -pl ccdi-project -am
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- BUILD SUCCESS
|
||||
|
||||
**Step 3: Execute manual acceptance checks**
|
||||
|
||||
验证以下结果：
|
||||
|
||||
- 首次导入：数据写入成功
|
||||
- 第二次导入同一数据：原记录不变，数量不增加
|
||||
- 非重复数据再次导入：仅新增新记录
|
||||
- 制造非唯一键类 SQL 错误：上传记录进入 `parsed_failed`
|
||||
|
||||
**Step 4: Prepare the delivery summary**
|
||||
|
||||
总结：
|
||||
|
||||
- 唯一键是否已生效
|
||||
- 重复导入是否跳过
|
||||
- 原数据是否保持不变
|
||||
- 日志计数是否为精确值还是保守值
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git add <verified files>
|
||||
git commit -m "docs: finalize bank statement dedup verification notes"
|
||||
```
|
||||
Reference in New Issue
Block a user