ccdi/docs/plans/2026-03-09-bank-statement-duplicate-check.md

银行流水重复校验 Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: 为 fetchAndSaveBankStatements(...) 增加基于 project_id + LE_ACCOUNT_NO + ACCOUNTING_DATE_ID + AMOUNT_DR + AMOUNT_CR 的数据库级重复校验，确保重复流水跳过插入且保留原数据不变。

Architecture: 先清理测试库中的异常行和重复行，再为 ccdi_bank_statement 增加新唯一键。业务代码只负责对 LE_ACCOUNT_NO 做轻量标准化，并将批量插入改为 no-op upsert；真实数据库错误仍按现有异步文件处理链路向上抛出。

Tech Stack: Java 21, Spring Boot 3, MyBatis Plus, MySQL, JUnit 5, Mockito

---

Task 1: 固化新去重键的标准化失败测试

Files:

• Modify: ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java
• Modify: ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java

Step 1: Write the failing test

在现有 CcdiFileUploadServiceImplTest 中新增一个聚焦标准化的测试。通过 mock lsfxClient.getBankStatement(...) 返回一条带空白账号的流水，并捕获传给 bankStatementMapper.insertBatch(...) 的实体。

@Test
void fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert() {
    // arrange
    // leAccountNo = " 62220001 "
    // accountingDateId = 20260310
    // amountDr = new BigDecimal("100.00")
    // amountCr = new BigDecimal("0.00")
    // act
    // assert
    verify(bankStatementMapper).insertBatch(argThat(list ->
        "62220001".equals(list.get(0).getLeAccountNo())));
}

Step 2: Run test to verify it fails

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert

Expected:

• FAIL
• 失败原因体现当前实现尚未对 LE_ACCOUNT_NO 做 trim

Step 3: Write the minimal implementation

在 CcdiFileUploadServiceImpl 内增加最小辅助方法：

private String trimAccountNo(String value) {
    return value == null ? null : value.trim();
}

private void normalizeDedupFields(CcdiBankStatement statement) {
    statement.setLeAccountNo(trimAccountNo(statement.getLeAccountNo()));
}

并在 fromResponse(...) 结果加入批次列表前调用。

Step 4: Run test to verify it passes

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#fetchAndSaveBankStatements_shouldTrimLeAccountNoBeforeInsert

Expected:

• PASS

Step 5: Commit

git add ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java
git commit -m "test(ccdi-project): cover bank statement account no normalization"

Task 2: 用失败测试锁定“重复不失败”的服务层语义

Files:

• Modify: ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java
• Modify: ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java

Step 1: Write the failing test

新增一个测试，模拟 Mapper 在重复导入场景下不抛异常，并验证上传记录最终不会因为重复而进入 parsed_failed。

@Test
void processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped() {
    // mock upload success / parse success
    // mock bankStatementMapper.insertBatch(...) 返回“未报错的重复跳过结果”
    // assert recordMapper 不会收到 parsed_failed
}

Step 2: Run test to verify it fails

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped

Expected:

• FAIL
• 失败点应体现当前实现还没有为重复跳过设计明确语义或日志

Step 3: Write minimal implementation

只做本任务所需的最小改动：

• 在 fetchAndSaveBankStatements(...) 中补充接口返回数、尝试写入数的日志
• 保持 duplicate 场景不抛异常
• 不调整其他与重复无关的异常路径

示例日志：

log.info("【文件上传】流水入库完成: fetchedCount={}, attemptedCount={}, insertedCount={}, duplicateCount={}",
    fetchedCount, attemptedCount, insertedCount, duplicateCount);

Step 4: Run the test to verify it passes

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped

Expected:

• PASS

Step 5: Commit

git add ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java
git commit -m "test(ccdi-project): keep duplicate bank statements from failing uploads"

Task 3: 为测试库编写数据清洗和唯一键迁移脚本

Files:

• Create: assets/database/2026-03-10-bank-statement-dedup.sql
• Modify: assets/对接流水分析/ccdi_bank_statement.md

Step 1: Write the migration script

创建测试库迁移脚本，包含以下 SQL：

DELETE FROM ccdi_bank_statement
WHERE project_id IS NULL
   OR LE_ACCOUNT_NO IS NULL
   OR ACCOUNTING_DATE_ID IS NULL;

UPDATE ccdi_bank_statement
SET LE_ACCOUNT_NO = TRIM(LE_ACCOUNT_NO);

DELETE t1
FROM ccdi_bank_statement t1
JOIN ccdi_bank_statement t2
  ON t1.bank_statement_id > t2.bank_statement_id
 AND t1.project_id = t2.project_id
 AND t1.LE_ACCOUNT_NO = t2.LE_ACCOUNT_NO
 AND t1.ACCOUNTING_DATE_ID = t2.ACCOUNTING_DATE_ID
 AND t1.AMOUNT_DR = t2.AMOUNT_DR
 AND t1.AMOUNT_CR = t2.AMOUNT_CR;

ALTER TABLE ccdi_bank_statement
    MODIFY COLUMN project_id bigint(20) NOT NULL COMMENT '关联项目ID',
    MODIFY COLUMN LE_ACCOUNT_NO varchar(240) NOT NULL DEFAULT '' COMMENT '企业银行账号',
    MODIFY COLUMN ACCOUNTING_DATE_ID int(11) NOT NULL COMMENT '账号日期ID';

ALTER TABLE ccdi_bank_statement
    ADD UNIQUE KEY uk_bank_statement_dedup (
        project_id,
        LE_ACCOUNT_NO,
        ACCOUNTING_DATE_ID,
        AMOUNT_DR,
        AMOUNT_CR
    );

Step 2: Manually run the migration in the local test database

Run:

mysql -u <user> -p <database> < assets/database/2026-03-10-bank-statement-dedup.sql

Expected:

• SQL 全部执行成功
• SHOW INDEX FROM ccdi_bank_statement; 能看到 uk_bank_statement_dedup

Step 3: Update the bank statement schema note

同步更新 assets/对接流水分析/ccdi_bank_statement.md，补齐：

• project_id
• 新唯一键说明
• project_id / LE_ACCOUNT_NO / ACCOUNTING_DATE_ID 的非空语义

Step 4: Verify migration result

Run:

mysql -u <user> -p -e "SHOW CREATE TABLE ccdi_bank_statement\G"

Expected:

• 表结构包含新的唯一键
• project_id、LE_ACCOUNT_NO、ACCOUNTING_DATE_ID 为 NOT NULL

Step 5: Commit

git add assets/database/2026-03-10-bank-statement-dedup.sql assets/对接流水分析/ccdi_bank_statement.md
git commit -m "feat(database): add bank statement dedup unique key"

Task 4: 将 Mapper 批量插入改为 no-op upsert

Files:

• Modify: ccdi-project/src/main/java/com/ruoyi/ccdi/project/mapper/CcdiBankStatementMapper.java
• Modify: ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml

Step 1: Write the failing verification case

先在本地 MySQL 中准备两次相同业务键的数据，第二次执行当前批量插入 SQL，确认现状会抛唯一键冲突。

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest#processFileAsync_shouldNotFailWhenDuplicateStatementsAreSkipped

Expected:

• FAIL 或本地重复场景仍未被正确跳过

Step 2: Change the insert SQL

将 XML 中的批量插入改为：

<insert id="insertBatch" parameterType="java.util.List">
    insert into ccdi_bank_statement (...)
    values
    <foreach collection="list" item="item" separator=",">
        (...)
    </foreach>
    on duplicate key update
        bank_statement_id = bank_statement_id
</insert>

Step 3: Keep mapper signature unchanged

保持 CcdiBankStatementMapper.insertBatch(@Param("list") List<CcdiBankStatement> list) 不变，避免扩大调用面。

Step 4: Run the targeted test suite

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest

Expected:

• PASS

Step 5: Manual duplicate verification

在本地测试库重复导入同一批数据后执行：

mysql -u <user> -p -e "SELECT COUNT(*) FROM ccdi_bank_statement WHERE project_id = <projectId> AND LE_ACCOUNT_NO = '62220001' AND ACCOUNTING_DATE_ID = 20260310 AND AMOUNT_DR = 100.00 AND AMOUNT_CR = 0.00;"

Expected:

• 结果始终为 1

Step 6: Commit

git add ccdi-project/src/main/java/com/ruoyi/ccdi/project/mapper/CcdiBankStatementMapper.java ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml
git commit -m "feat(ccdi-project): skip duplicate bank statements on insert"

Task 5: 校准日志计数并验证 MySQL 受影响行数语义

Files:

• Modify: ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java
• Modify: ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java

Step 1: Add a temporary probe in the service

在开发阶段先打印：

log.info("【文件上传】dedup probe: batchSize={}, mapperAffectedRows={}", batchList.size(), affectedRows);

Step 2: Run a duplicate import manually

使用相同测试文件连续导入两次，并观察第二次日志。

Expected:

• 能确认 duplicate 分支下 MyBatis/JDBC 返回的 affectedRows 语义

Step 3: Finalize the counting logic

如果实测 duplicate 返回 0，则直接落正式逻辑：

insertedCount += affectedRows;
duplicateCount += batchSize - affectedRows;

如果实测不稳定，则不要伪造精确计数，改为保守日志：

log.info("【文件上传】流水入库完成: fetchedCount={}, attemptedCount={}", fetchedCount, attemptedCount);

Step 4: Run the test suite

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest

Expected:

• PASS

Step 5: Remove temporary probe and commit

git add ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java ccdi-project/src/test/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImplTest.java
git commit -m "refactor(ccdi-project): finalize bank statement dedup logging"

Task 6: 做最终回归验证并整理交付

Files:

• Review: ccdi-project/src/main/java/com/ruoyi/ccdi/project/service/impl/CcdiFileUploadServiceImpl.java
• Review: ccdi-project/src/main/resources/mapper/ccdi/project/CcdiBankStatementMapper.xml
• Review: assets/database/2026-03-10-bank-statement-dedup.sql
• Review: assets/对接流水分析/ccdi_bank_statement.md

Step 1: Run automated tests

Run:

mvn test -pl ccdi-project -Dtest=CcdiFileUploadServiceImplTest

Expected:

• PASS

Step 2: Run a focused compile

Run:

mvn clean compile -pl ccdi-project -am

Expected:

• BUILD SUCCESS

Step 3: Execute manual acceptance checks

验证以下结果：

• 首次导入：数据写入成功
• 第二次导入同一数据：原记录不变，数量不增加
• 非重复数据再次导入：仅新增新记录
• 制造非唯一键类 SQL 错误：上传记录进入 parsed_failed

Step 4: Prepare the delivery summary

总结：

• 唯一键是否已生效
• 重复导入是否跳过
• 原数据是否保持不变
• 日志计数是否为精确值还是保守值

Step 5: Commit

git status --short
git add <verified files>
git commit -m "docs: finalize bank statement dedup verification notes"