diff --git a/docs/plans/2026-03-09-bank-statement-duplicate-check.md b/docs/plans/2026-03-09-bank-statement-duplicate-check.md new file mode 100644 index 0000000..f8bdb93 --- /dev/null +++ b/docs/plans/2026-03-09-bank-statement-duplicate-check.md @@ -0,0 +1,420 @@ +# 银行流水重复校验 Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** 为 `fetchAndSaveBankStatements(...)` 增加基于 `project_id + LE_ACCOUNT_NO + ACCOUNTING_DATE_ID + AMOUNT_DR + AMOUNT_CR` 的数据库级重复校验,确保重复流水跳过插入且保留原数据不变。 + +**Architecture:** 先清理测试库中的异常行和重复行,再为 `ccdi_bank_statement` 增加新唯一键。业务代码只负责对 `LE_ACCOUNT_NO` 做轻量标准化,并将批量插入改为 no-op upsert;真实数据库错误仍按现有异步文件处理链路向上抛出。 + +**Tech Stack:** Java 21, Spring Boot 3, MyBatis Plus, MySQL, JUnit 5, Mockito + +--- + +### Task 1: 固化新去重键的标准化失败测试 + +**Files:** +- Modify: `ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java` +- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java` + +**Step 1: Write the failing test** + +在现有 `CcdiFileUploadServiceImplTest` 中新增一个聚焦标准化的测试。通过 mock `lsfxClient.getBankStatement(...)` 返回一条带空白账号的流水,并捕获传给 `bankStatementMapper.insertBatch(...)` 的实体。 + +```java +@Test +void fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert() { + // arrange + // leAccountNo = " 62220001 " + // accountingDateId = 20260310 + // amountDr = new BigDecimal("100.00") + // amountCr = new BigDecimal("0.00") + // act + // assert + verify(bankStatementMapper).insertBatch(argThat(list -> + "62220001".equals(list.get(0).getLeAccountNo()))); +} +``` + +**Step 2: Run test to verify it fails** + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert +``` + +Expected: + +- FAIL +- 失败原因体现当前实现尚未对 `LE_ACCOUNT_NO` 做 `trim` + +**Step 3: Write the minimal implementation** + +在 `CcdiFileUploadServiceImpl` 内增加最小辅助方法: + +```java +private String trimAccountNo(String value) { + return value == null ? null : value.trim(); +} + +private void normalizeDedupFields(CcdiBankStatement statement) { + statement.setLeAccountNo(trimAccountNo(statement.getLeAccountNo())); +} +``` + +并在 `fromResponse(...)` 结果加入批次列表前调用。 + +**Step 4: Run test to verify it passes** + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert +``` + +Expected: + +- PASS + +**Step 5: Commit** + +```bash +git add ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java +git commit -m "test(ccdi-project): cover bank statement account no normalization" +``` + +### Task 2: 用失败测试锁定“重复不失败”的服务层语义 + +**Files:** +- Modify: `ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java` +- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java` + +**Step 1: Write the failing test** + +新增一个测试,模拟 Mapper 在重复导入场景下不抛异常,并验证上传记录最终不会因为重复而进入 `parsed_failed`。 + +```java +@Test +void processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped() { + // mock upload success / parse success + // mock bankStatementMapper.insertBatch(...) 返回“未报错的重复跳过结果” + // assert recordMapper 不会收到 parsed_failed +} +``` + +**Step 2: Run test to verify it fails** + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped +``` + +Expected: + +- FAIL +- 失败点应体现当前实现还没有为重复跳过设计明确语义或日志 + +**Step 3: Write minimal implementation** + +只做本任务所需的最小改动: + +- 在 `fetchAndSaveBankStatements(...)` 中补充接口返回数、尝试写入数的日志 +- 保持 duplicate 场景不抛异常 +- 不调整其他与重复无关的异常路径 + +示例日志: + +```java +log.info("【文件上传】流水入库完成: fetchedCount={}, attemptedCount={}, insertedCount={}, duplicateCount={}", + fetchedCount, attemptedCount, insertedCount, duplicateCount); +``` + +**Step 4: Run the test to verify it passes** + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped +``` + +Expected: + +- PASS + +**Step 5: Commit** + +```bash +git add ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java +git commit -m "test(ccdi-project): keep duplicate bank statements from failing uploads" +``` + +### Task 3: 为测试库编写数据清洗和唯一键迁移脚本 + +**Files:** +- Create: `assets/database/2026-03-10-bank-statement-dedup.sql` +- Modify: `assets/对接流水分析/ccdi_bank_statement.md` + +**Step 1: Write the migration script** + +创建测试库迁移脚本,包含以下 SQL: + +```sql +DELETE FROM ccdi_bank_statement +WHERE project_id IS NULL + OR LE_ACCOUNT_NO IS NULL + OR ACCOUNTING_DATE_ID IS NULL; + +UPDATE ccdi_bank_statement +SET LE_ACCOUNT_NO = TRIM(LE_ACCOUNT_NO); + +DELETE t1 +FROM ccdi_bank_statement t1 +JOIN ccdi_bank_statement t2 + ON t1.bank_statement_id > t2.bank_statement_id + AND t1.project_id = t2.project_id + AND t1.LE_ACCOUNT_NO = t2.LE_ACCOUNT_NO + AND t1.ACCOUNTING_DATE_ID = t2.ACCOUNTING_DATE_ID + AND t1.AMOUNT_DR = t2.AMOUNT_DR + AND t1.AMOUNT_CR = t2.AMOUNT_CR; + +ALTER TABLE ccdi_bank_statement + MODIFY COLUMN project_id bigint(20) NOT NULL COMMENT '关联项目ID', + MODIFY COLUMN LE_ACCOUNT_NO varchar(240) NOT NULL DEFAULT '' COMMENT '企业银行账号', + MODIFY COLUMN ACCOUNTING_DATE_ID int(11) NOT NULL COMMENT '账号日期ID'; + +ALTER TABLE ccdi_bank_statement + ADD UNIQUE KEY uk_bank_statement_dedup ( + project_id, + LE_ACCOUNT_NO, + ACCOUNTING_DATE_ID, + AMOUNT_DR, + AMOUNT_CR + ); +``` + +**Step 2: Manually run the migration in the local test database** + +Run: + +```bash +mysql -u -p < assets/database/2026-03-10-bank-statement-dedup.sql +``` + +Expected: + +- SQL 全部执行成功 +- `SHOW INDEX FROM ccdi_bank_statement;` 能看到 `uk_bank_statement_dedup` + +**Step 3: Update the bank statement schema note** + +同步更新 `assets/对接流水分析/ccdi_bank_statement.md`,补齐: + +- `project_id` +- 新唯一键说明 +- `project_id` / `LE_ACCOUNT_NO` / `ACCOUNTING_DATE_ID` 的非空语义 + +**Step 4: Verify migration result** + +Run: + +```bash +mysql -u -p -e "SHOW CREATE TABLE ccdi_bank_statement\G" +``` + +Expected: + +- 表结构包含新的唯一键 +- `project_id`、`LE_ACCOUNT_NO`、`ACCOUNTING_DATE_ID` 为 `NOT NULL` + +**Step 5: Commit** + +```bash +git add assets/database/2026-03-10-bank-statement-dedup.sql assets/对接流水分析/ccdi_bank_statement.md +git commit -m "feat(database): add bank statement dedup unique key" +``` + +### Task 4: 将 Mapper 批量插入改为 no-op upsert + +**Files:** +- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/mapper/CcdiBankStatementMapper.java` +- Modify: `ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml` + +**Step 1: Write the failing verification case** + +先在本地 MySQL 中准备两次相同业务键的数据,第二次执行当前批量插入 SQL,确认现状会抛唯一键冲突。 + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped +``` + +Expected: + +- FAIL 或本地重复场景仍未被正确跳过 + +**Step 2: Change the insert SQL** + +将 XML 中的批量插入改为: + +```xml + + insert into ccdi_bank_statement (...) + values + + (...) + + on duplicate key update + bank_statement_id = bank_statement_id + +``` + +**Step 3: Keep mapper signature unchanged** + +保持 `CcdiBankStatementMapper.insertBatch(@Param("list") List list)` 不变,避免扩大调用面。 + +**Step 4: Run the targeted test suite** + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest +``` + +Expected: + +- PASS + +**Step 5: Manual duplicate verification** + +在本地测试库重复导入同一批数据后执行: + +```bash +mysql -u -p -e "SELECT COUNT(*) FROM ccdi_bank_statement WHERE project_id = AND LE_ACCOUNT_NO = '62220001' AND ACCOUNTING_DATE_ID = 20260310 AND AMOUNT_DR = 100.00 AND AMOUNT_CR = 0.00;" +``` + +Expected: + +- 结果始终为 `1` + +**Step 6: Commit** + +```bash +git add ccdi-project/src/main/java/com/ruoyi/ccdi/project/mapper/CcdiBankStatementMapper.java ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml +git commit -m "feat(ccdi-project): skip duplicate bank statements on insert" +``` + +### Task 5: 校准日志计数并验证 MySQL 受影响行数语义 + +**Files:** +- Modify: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java` +- Modify: `ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java` + +**Step 1: Add a temporary probe in the service** + +在开发阶段先打印: + +```java +log.info("【文件上传】dedup probe: batchSize={}, mapperAffectedRows={}", batchList.size(), affectedRows); +``` + +**Step 2: Run a duplicate import manually** + +使用相同测试文件连续导入两次,并观察第二次日志。 + +Expected: + +- 能确认 duplicate 分支下 MyBatis/JDBC 返回的 `affectedRows` 语义 + +**Step 3: Finalize the counting logic** + +如果实测 duplicate 返回 `0`,则直接落正式逻辑: + +```java +insertedCount += affectedRows; +duplicateCount += batchSize - affectedRows; +``` + +如果实测不稳定,则不要伪造精确计数,改为保守日志: + +```java +log.info("【文件上传】流水入库完成: fetchedCount={}, attemptedCount={}", fetchedCount, attemptedCount); +``` + +**Step 4: Run the test suite** + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest +``` + +Expected: + +- PASS + +**Step 5: Remove temporary probe and commit** + +```bash +git add ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java +git commit -m "refactor(ccdi-project): finalize bank statement dedup logging" +``` + +### Task 6: 做最终回归验证并整理交付 + +**Files:** +- Review: `ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java` +- Review: `ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml` +- Review: `assets/database/2026-03-10-bank-statement-dedup.sql` +- Review: `assets/对接流水分析/ccdi_bank_statement.md` + +**Step 1: Run automated tests** + +Run: + +```bash +mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest +``` + +Expected: + +- PASS + +**Step 2: Run a focused compile** + +Run: + +```bash +mvn clean compile -pl ccdi-project -am +``` + +Expected: + +- BUILD SUCCESS + +**Step 3: Execute manual acceptance checks** + +验证以下结果: + +- 首次导入:数据写入成功 +- 第二次导入同一数据:原记录不变,数量不增加 +- 非重复数据再次导入:仅新增新记录 +- 制造非唯一键类 SQL 错误:上传记录进入 `parsed_failed` + +**Step 4: Prepare the delivery summary** + +总结: + +- 唯一键是否已生效 +- 重复导入是否跳过 +- 原数据是否保持不变 +- 日志计数是否为精确值还是保守值 + +**Step 5: Commit** + +```bash +git status --short +git add +git commit -m "docs: finalize bank statement dedup verification notes" +``` diff --git a/ruoyi-admin/src/main/resources/application.yml b/ruoyi-admin/src/main/resources/application.yml index 104207f..62bead8 100644 --- a/ruoyi-admin/src/main/resources/application.yml +++ b/ruoyi-admin/src/main/resources/application.yml @@ -18,6 +18,7 @@ logging: level: com.ruoyi: debug org.springframework: warn + "com.ruoyi.ccdi.project.mapper.CcdiBankStatementMapper.insertBatch": info # 用户配置 user: