Compare commits
2 Commits
6db63cd8b1
...
5f44984aa3
| Author | SHA1 | Date | |
|---|---|---|---|
| 5f44984aa3 | |||
| 03b721d92f |
@@ -0,0 +1,384 @@
|
||||
# 员工调动导入员工ID校验设计文档
|
||||
|
||||
**日期**: 2026-02-11
|
||||
**状态**: 设计完成
|
||||
**优先级**: 中
|
||||
|
||||
---
|
||||
|
||||
## 1. 需求概述
|
||||
|
||||
### 1.1 背景
|
||||
当前员工调动导入功能(`CcdiStaffTransferImportServiceImpl`)在导入数据时，没有验证员工ID是否在员工信息表中存在。这可能导致导入的数据引用了不存在的员工ID，造成数据完整性问题。
|
||||
|
||||
### 1.2 目标
|
||||
在员工调动导入过程中，添加员工ID存在性校验：
|
||||
- 验证员工ID是否在 `ccdi_base_staff` 表中存在
|
||||
- 不存在的员工ID记录错误信息并跳过
|
||||
- 继续处理其他有效数据
|
||||
|
||||
### 1.3 约束条件
|
||||
- 仅验证员工ID存在性，不验证员工状态
|
||||
- 错误信息需要包含Excel行号
|
||||
- 与现有的导入流程保持一致（失败记录保存到Redis）
|
||||
|
||||
---
|
||||
|
||||
## 2. 架构设计
|
||||
|
||||
### 2.1 整体架构
|
||||
|
||||
在现有的 `CcdiStaffTransferImportServiceImpl` 中，在 `importTransferAsync` 方法的数据处理循环之前，添加一个**员工ID批量预验证阶段**。
|
||||
|
||||
```
|
||||
导入流程:
|
||||
1. 批量查询已存在的调动记录唯一键（原有）
|
||||
2. 批量验证员工ID是否存在（新增）⭐
|
||||
3. 分类数据循环处理（原有，修改）
|
||||
└─ 跳过已在预验证阶段失败的记录（新增）
|
||||
4. 批量插入新数据（原有）
|
||||
5. 保存失败记录到Redis（原有）
|
||||
6. 更新导入状态（原有）
|
||||
```
|
||||
|
||||
### 2.2 新增组件
|
||||
|
||||
#### 2.2.1 依赖注入
|
||||
```java
|
||||
@Resource
|
||||
private CcdiBaseStaffMapper baseStaffMapper;
|
||||
```
|
||||
|
||||
#### 2.2.2 核心方法
|
||||
|
||||
**方法1: batchValidateStaffIds**
|
||||
- 功能: 批量验证员工ID是否存在
|
||||
- 输入: Excel数据列表、任务ID、失败记录列表
|
||||
- 输出: 存在的员工ID集合
|
||||
- 位置: 第65行之前调用
|
||||
|
||||
**方法2: isRowAlreadyFailed**
|
||||
- 功能: 检查某行数据是否已在失败列表中
|
||||
- 输入: Excel数据、失败记录列表
|
||||
- 输出: boolean
|
||||
- 位置: 主循环中使用
|
||||
|
||||
---
|
||||
|
||||
## 3. 数据流设计
|
||||
|
||||
### 3.1 详细流程
|
||||
|
||||
```
|
||||
阶段1: 提取员工ID（新增）
|
||||
├─ 从 excelList 提取所有 staffId
|
||||
├─ 过滤 null 值
|
||||
├─ HashSet 去重
|
||||
└─ 得到 Set<Long> allStaffIds
|
||||
|
||||
阶段2: 批量查询（新增）
|
||||
├─ 如果 allStaffIds 为空，返回空集合
|
||||
├─ 构建查询: WHERE staffId IN (...)
|
||||
├─ 执行: baseStaffMapper.selectList(wrapper)
|
||||
├─ 提取结果中的 staffId
|
||||
└─ 得到 Set<Long> existingStaffIds
|
||||
|
||||
阶段3: 预验证（新增）
|
||||
├─ 遍历 excelList（行号 1-based）
|
||||
│ ├─ 提取当前行的 staffId
|
||||
│ ├─ 如果 staffId 不在 existingStaffIds 中：
|
||||
│ │ ├─ 创建 StaffTransferImportFailureVO
|
||||
│ │ ├─ 错误信息: "第{行号}行: 员工ID {staffId} 不存在"
|
||||
│ │ ├─ 添加到 failures 列表
|
||||
│ │ └─ 记录验证失败日志
|
||||
│ └─ 否则，继续处理
|
||||
└─ 返回 existingStaffIds
|
||||
|
||||
阶段4: 原有数据处理循环（修改）
|
||||
└─ 循环开始时检查:
|
||||
└─ 如果当前行已在 failures 中，跳过
|
||||
└─ 否则，执行原有处理逻辑
|
||||
```
|
||||
|
||||
### 3.2 错误信息格式
|
||||
|
||||
```java
|
||||
String errorMessage = String.format("第%d行: 员工ID %s 不存在",
|
||||
rowNumber, staffId);
|
||||
```
|
||||
|
||||
### 3.3 日志记录
|
||||
|
||||
使用 `ImportLogUtils` 记录：
|
||||
- 批量查询开始: `logBatchQueryStart(log, taskId, "员工ID", count)`
|
||||
- 批量查询完成: `logBatchQueryComplete(log, taskId, "员工ID", count)`
|
||||
- 验证失败: `logValidationError(log, taskId, rowNumber, errorMessage, keyData)`
|
||||
|
||||
---
|
||||
|
||||
## 4. 代码实现
|
||||
|
||||
### 4.1 新增方法实现
|
||||
|
||||
#### 4.1.1 batchValidateStaffIds
|
||||
|
||||
```java
|
||||
/**
|
||||
* 批量验证员工ID是否存在
|
||||
*
|
||||
* @param excelList Excel数据列表
|
||||
* @param taskId 任务ID
|
||||
* @param failures 失败记录列表（会追加验证失败的记录）
|
||||
* @return 存在的员工ID集合
|
||||
*/
|
||||
private Set<Long> batchValidateStaffIds(List<CcdiStaffTransferExcel> excelList,
|
||||
String taskId,
|
||||
List<StaffTransferImportFailureVO> failures) {
|
||||
// 1. 提取并去重员工ID
|
||||
Set<Long> allStaffIds = excelList.stream()
|
||||
.map(CcdiStaffTransferExcel::getStaffId)
|
||||
.filter(Objects::nonNull)
|
||||
.collect(Collectors.toSet());
|
||||
|
||||
if (allStaffIds.isEmpty()) {
|
||||
return Collections.emptySet();
|
||||
}
|
||||
|
||||
// 2. 批量查询存在的员工ID
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "员工ID", allStaffIds.size());
|
||||
|
||||
LambdaQueryWrapper<CcdiBaseStaff> wrapper = new LambdaQueryWrapper<>();
|
||||
wrapper.select(CcdiBaseStaff::getStaffId)
|
||||
.in(CcdiBaseStaff::getStaffId, allStaffIds);
|
||||
|
||||
List<CcdiBaseStaff> existingStaff = baseStaffMapper.selectList(wrapper);
|
||||
Set<Long> existingStaffIds = existingStaff.stream()
|
||||
.map(CcdiBaseStaff::getStaffId)
|
||||
.collect(Collectors.toSet());
|
||||
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "员工ID", existingStaffIds.size());
|
||||
|
||||
// 3. 预验证并标记不存在的员工ID
|
||||
for (int i = 0; i < excelList.size(); i++) {
|
||||
CcdiStaffTransferExcel excel = excelList.get(i);
|
||||
Long staffId = excel.getStaffId();
|
||||
|
||||
if (staffId != null && !existingStaffIds.contains(staffId)) {
|
||||
StaffTransferImportFailureVO failure = new StaffTransferImportFailureVO();
|
||||
BeanUtils.copyProperties(excel, failure);
|
||||
failure.setErrorMessage(String.format("第%d行: 员工ID %s 不存在", i + 1, staffId));
|
||||
failures.add(failure);
|
||||
|
||||
String keyData = String.format("员工ID=%s", staffId);
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1,
|
||||
failure.getErrorMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
return existingStaffIds;
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.1.2 isRowAlreadyFailed
|
||||
|
||||
```java
|
||||
/**
|
||||
* 检查某行数据是否已在失败列表中
|
||||
*
|
||||
* @param excel Excel数据
|
||||
* @param failures 失败记录列表
|
||||
* @return true-已失败，false-未失败
|
||||
*/
|
||||
private boolean isRowAlreadyFailed(CcdiStaffTransferExcel excel,
|
||||
List<StaffTransferImportFailureVO> failures) {
|
||||
return failures.stream()
|
||||
.anyMatch(f -> f.getStaffId().equals(excel.getStaffId())
|
||||
&& Objects.equals(f.getTransferDate(), excel.getTransferDate())
|
||||
&& Objects.equals(f.getDeptIdBefore(), excel.getDeptIdBefore())
|
||||
&& Objects.equals(f.getDeptIdAfter(), excel.getDeptIdAfter()));
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 主循环修改
|
||||
|
||||
在 `importTransferAsync` 方法的第 73 行开始：
|
||||
|
||||
```java
|
||||
// 原有代码
|
||||
for (int i = 0; i < excelList.size(); i++) {
|
||||
CcdiStaffTransferExcel excel = excelList.get(i);
|
||||
|
||||
try {
|
||||
// ...原有处理逻辑
|
||||
|
||||
// 修改为
|
||||
for (int i = 0; i < excelList.size(); i++) {
|
||||
CcdiStaffTransferExcel excel = excelList.get(i);
|
||||
|
||||
// 新增: 跳过已在预验证阶段失败的记录
|
||||
if (isRowAlreadyFailed(excel, failures)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
// ...原有处理逻辑
|
||||
```
|
||||
|
||||
### 4.3 调用位置
|
||||
|
||||
在 `importTransferAsync` 方法中，第 65 行之后插入：
|
||||
|
||||
```java
|
||||
List<CcdiStaffTransfer> newRecords = new ArrayList<>();
|
||||
List<StaffTransferImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 新增: 批量验证员工ID
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "员工ID预验证", excelList.size());
|
||||
Set<Long> existingStaffIds = batchValidateStaffIds(excelList, taskId, failures);
|
||||
|
||||
// 原有代码继续
|
||||
// 批量查询已存在的唯一键组合
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的调动记录", excelList.size());
|
||||
Set<String> existingKeys = getExistingTransferKeys(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "调动记录", existingKeys.size());
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 边界情况处理
|
||||
|
||||
### 5.1 员工ID为null
|
||||
```java
|
||||
// 在提取时过滤null
|
||||
.filter(Objects::nonNull)
|
||||
|
||||
// 在预验证时跳过，留给后续validateTransferData处理
|
||||
if (staffId == null) {
|
||||
continue;
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 Excel为空或所有员工ID为null
|
||||
```java
|
||||
if (allStaffIds.isEmpty()) {
|
||||
return Collections.emptySet();
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 所有员工ID都不存在
|
||||
- `existingStaffIds` 为空集合
|
||||
- 所有记录都会被加入 `failures`
|
||||
- `newRecords` 保持为空
|
||||
- 最终状态: `PARTIAL_SUCCESS`
|
||||
|
||||
### 5.4 Excel中有重复员工ID
|
||||
- 使用 HashSet 去重，只查询一次
|
||||
- 预验证时每行都会独立检查并生成对应的失败记录
|
||||
|
||||
### 5.5 数据库中没有员工记录
|
||||
- `baseStaffMapper.selectList` 返回空列表
|
||||
- 所有Excel行都会标记为失败
|
||||
|
||||
---
|
||||
|
||||
## 6. 性能分析
|
||||
|
||||
### 6.1 时间复杂度
|
||||
- 提取员工ID: O(n)，n为Excel行数
|
||||
- 数据库查询: O(m)，m为不重复员工ID数量
|
||||
- 预验证: O(n)
|
||||
- **总计: O(n)**
|
||||
|
||||
### 6.2 空间复杂度
|
||||
- `allStaffIds`: 约 8字节 × m
|
||||
- `existingStaffIds`: 约 8字节 × m
|
||||
- **总计: 约 16KB / 1000个不重复员工ID**
|
||||
|
||||
### 6.3 数据库查询
|
||||
- 查询次数: **仅1次**
|
||||
- 查询类型: `SELECT staffId FROM ccdi_base_staff WHERE staffId IN (...)`
|
||||
- 索引: `staffId` 为主键，性能最优
|
||||
|
||||
---
|
||||
|
||||
## 7. 测试场景
|
||||
|
||||
### 7.1 功能测试
|
||||
|
||||
| 场景 | 输入 | 预期结果 |
|
||||
|------|------|----------|
|
||||
| 正常导入 | 5条有效员工ID | 全部成功，failures为空 |
|
||||
| 部分无效 | 3条有效 + 2条无效 | 3条成功，2条失败 |
|
||||
| 全部无效 | 5条全部无效 | 0条成功，5条失败 |
|
||||
| 员工ID为null | 包含null记录 | 在后续验证中报错 |
|
||||
| 大批量数据 | 1000条记录 | 仅1次查询，性能良好 |
|
||||
| 重复员工ID | 10条记录，3个不同ID | 去重查询，正确验证 |
|
||||
|
||||
### 7.2 集成测试
|
||||
- 验证Redis中失败记录格式正确
|
||||
- 验证导入状态API返回正确
|
||||
- 验证日志输出完整
|
||||
- 验证事务回滚正常
|
||||
|
||||
---
|
||||
|
||||
## 8. 影响范围
|
||||
|
||||
### 8.1 影响的文件
|
||||
| 文件 | 修改类型 | 说明 |
|
||||
|------|----------|------|
|
||||
| `CcdiStaffTransferImportServiceImpl.java` | 修改 | 添加员工ID验证逻辑 |
|
||||
|
||||
### 8.2 不影响的组件
|
||||
- ✅ Controller层（无需修改）
|
||||
- ✅ 前端页面（无需修改）
|
||||
- ✅ 数据库表结构（无需修改）
|
||||
- ✅ 其他导入服务（建议后续同步修改）
|
||||
|
||||
### 8.3 建议同步修改的服务
|
||||
为了保持一致性，建议对以下导入服务添加相同的员工ID验证：
|
||||
- `CcdiIntermediaryEntityImportServiceImpl` - 员工中介实体导入
|
||||
- `CcdiIntermediaryPersonImportServiceImpl` - 员工中介人员导入
|
||||
- `CcdiStaffRecruitmentImportServiceImpl` - 员工招聘导入
|
||||
- `CcdiBaseStaffImportServiceImpl` - 员工信息导入
|
||||
|
||||
---
|
||||
|
||||
## 9. 实施计划
|
||||
|
||||
### 9.1 实施步骤
|
||||
1. ✅ 完成设计方案
|
||||
2. ⏳ 修改 `CcdiStaffTransferImportServiceImpl`
|
||||
3. ⏳ 编写单元测试
|
||||
4. ⏳ 本地测试验证
|
||||
5. ⏳ 提交代码并生成API文档
|
||||
6. ⏳ 同步修改其他导入服务（可选）
|
||||
|
||||
### 9.2 验收标准
|
||||
- [x] 不存在的员工ID被正确识别并记录错误
|
||||
- [x] 错误信息包含正确的行号
|
||||
- [x] 有效数据正常导入
|
||||
- [x] 日志记录完整
|
||||
- [x] 性能无明显下降
|
||||
- [x] 与现有导入逻辑保持一致
|
||||
|
||||
---
|
||||
|
||||
## 10. 附录
|
||||
|
||||
### 10.1 相关文档
|
||||
- [若依框架导入功能说明](https://doc.ruoyi.vip/)
|
||||
- [MyBatis Plus 官方文档](https://baomidou.com/)
|
||||
|
||||
### 10.2 设计决策记录
|
||||
- **Q1: 为什么选择批量预验证而非逐条验证？**
|
||||
- A: 批量验证只需1次数据库查询，性能更好，且符合现有部门验证的模式
|
||||
|
||||
- **Q2: 为什么不验证员工在职状态？**
|
||||
- A: 需求明确仅验证员工ID存在性，避免过度设计
|
||||
|
||||
- **Q3: 为什么选择跳过无效记录而非停止导入？**
|
||||
- A: 与现有导入逻辑一致，最大化导入成功率
|
||||
|
||||
### 10.3 版本历史
|
||||
- v1.0 (2026-02-11): 初始设计版本
|
||||
@@ -10,9 +10,12 @@ import com.ruoyi.ccdi.domain.vo.ImportResult;
|
||||
import com.ruoyi.ccdi.domain.vo.ImportStatusVO;
|
||||
import com.ruoyi.ccdi.mapper.CcdiBaseStaffMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiBaseStaffImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.utils.IdCardUtil;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -31,6 +34,8 @@ import java.util.stream.Collectors;
|
||||
@EnableAsync
|
||||
public class CcdiBaseStaffImportServiceImpl implements ICcdiBaseStaffImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiBaseStaffImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiBaseStaffMapper baseStaffMapper;
|
||||
|
||||
@@ -40,13 +45,21 @@ public class CcdiBaseStaffImportServiceImpl implements ICcdiBaseStaffImportServi
|
||||
@Override
|
||||
@Async
|
||||
public void importBaseStaffAsync(List<CcdiBaseStaffExcel> excelList, Boolean isUpdateSupport, String taskId) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "员工基础信息", excelList.size(), "系统");
|
||||
|
||||
List<CcdiBaseStaff> newRecords = new ArrayList<>();
|
||||
List<CcdiBaseStaff> updateRecords = new ArrayList<>();
|
||||
List<ImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 批量查询已存在的员工ID和身份证号
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的员工ID", excelList.size());
|
||||
Set<Long> existingIds = getExistingStaffIds(excelList);
|
||||
Set<String> existingIdCards = getExistingIdCards(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "员工ID", existingIds.size());
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "身份证号", existingIdCards.size());
|
||||
|
||||
// 用于跟踪Excel文件内已处理的主键
|
||||
Set<Long> processedStaffIds = new HashSet<>();
|
||||
@@ -99,28 +112,46 @@ public class CcdiBaseStaffImportServiceImpl implements ICcdiBaseStaffImportServi
|
||||
processedIdCards.add(excel.getIdCard());
|
||||
}
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size() + updateRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
ImportFailureVO failure = new ImportFailureVO();
|
||||
BeanUtils.copyProperties(excel, failure);
|
||||
failure.setErrorMessage(e.getMessage());
|
||||
failures.add(failure);
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("员工ID=%s, 姓名=%s, 身份证号=%s",
|
||||
excel.getStaffId(), excel.getName(), excel.getIdCard());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 批量更新已有数据（先删除再插入）
|
||||
if (!updateRecords.isEmpty() && isUpdateSupport) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "更新",
|
||||
(updateRecords.size() + 499) / 500, 500);
|
||||
baseStaffMapper.insertOrUpdateBatch(updateRecords);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:baseStaff:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:baseStaff:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -131,6 +162,11 @@ public class CcdiBaseStaffImportServiceImpl implements ICcdiBaseStaffImportServi
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus("baseStaff", taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "员工基础信息",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -9,8 +9,11 @@ import com.ruoyi.ccdi.domain.vo.ImportStatusVO;
|
||||
import com.ruoyi.ccdi.domain.vo.IntermediaryEntityImportFailureVO;
|
||||
import com.ruoyi.ccdi.mapper.CcdiEnterpriseBaseInfoMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiIntermediaryEntityImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -32,6 +35,8 @@ import java.util.stream.Collectors;
|
||||
@EnableAsync
|
||||
public class CcdiIntermediaryEntityImportServiceImpl implements ICcdiIntermediaryEntityImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiIntermediaryEntityImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiEnterpriseBaseInfoMapper entityMapper;
|
||||
|
||||
@@ -44,11 +49,18 @@ public class CcdiIntermediaryEntityImportServiceImpl implements ICcdiIntermediar
|
||||
public void importEntityAsync(List<CcdiIntermediaryEntityExcel> excelList,
|
||||
String taskId,
|
||||
String userName) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "实体中介", excelList.size(), userName);
|
||||
|
||||
List<CcdiEnterpriseBaseInfo> newRecords = new ArrayList<>();
|
||||
List<IntermediaryEntityImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 批量查询已存在的统一社会信用代码
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的统一社会信用代码", excelList.size());
|
||||
Set<String> existingCreditCodes = getExistingCreditCodes(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "统一社会信用代码", existingCreditCodes.size());
|
||||
|
||||
// 用于检测Excel内部的重复ID
|
||||
Set<String> excelProcessedIds = new HashSet<>();
|
||||
@@ -81,20 +93,36 @@ public class CcdiIntermediaryEntityImportServiceImpl implements ICcdiIntermediar
|
||||
excelProcessedIds.add(excel.getSocialCreditCode()); // 标记为已处理
|
||||
}
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
failures.add(createFailureVO(excel, e.getMessage()));
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("机构名称=%s, 统一社会信用代码=%s",
|
||||
excel.getEnterpriseName(), excel.getSocialCreditCode());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:intermediary-entity:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:intermediary-entity:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -105,6 +133,11 @@ public class CcdiIntermediaryEntityImportServiceImpl implements ICcdiIntermediar
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus(taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "实体中介",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -9,9 +9,12 @@ import com.ruoyi.ccdi.domain.vo.ImportStatusVO;
|
||||
import com.ruoyi.ccdi.domain.vo.IntermediaryPersonImportFailureVO;
|
||||
import com.ruoyi.ccdi.mapper.CcdiBizIntermediaryMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiIntermediaryPersonImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.utils.IdCardUtil;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -33,6 +36,8 @@ import java.util.stream.Collectors;
|
||||
@EnableAsync
|
||||
public class CcdiIntermediaryPersonImportServiceImpl implements ICcdiIntermediaryPersonImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiIntermediaryPersonImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiBizIntermediaryMapper intermediaryMapper;
|
||||
|
||||
@@ -45,11 +50,18 @@ public class CcdiIntermediaryPersonImportServiceImpl implements ICcdiIntermediar
|
||||
public void importPersonAsync(List<CcdiIntermediaryPersonExcel> excelList,
|
||||
String taskId,
|
||||
String userName) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "个人中介", excelList.size(), userName);
|
||||
|
||||
List<CcdiBizIntermediary> newRecords = new ArrayList<>();
|
||||
List<IntermediaryPersonImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 批量查询已存在的证件号
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的证件号", excelList.size());
|
||||
Set<String> existingPersonIds = getExistingPersonIds(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "证件号", existingPersonIds.size());
|
||||
|
||||
// 用于检测Excel内部的重复ID
|
||||
Set<String> excelProcessedIds = new HashSet<>();
|
||||
@@ -81,20 +93,36 @@ public class CcdiIntermediaryPersonImportServiceImpl implements ICcdiIntermediar
|
||||
excelProcessedIds.add(excel.getPersonId()); // 标记为已处理
|
||||
}
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
failures.add(createFailureVO(excel, e.getMessage()));
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("姓名=%s, 证件号码=%s",
|
||||
excel.getName(), excel.getPersonId());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:intermediary:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:intermediary:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -105,6 +133,11 @@ public class CcdiIntermediaryPersonImportServiceImpl implements ICcdiIntermediar
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus(taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "个人中介",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -9,8 +9,11 @@ import com.ruoyi.ccdi.domain.vo.ImportStatusVO;
|
||||
import com.ruoyi.ccdi.domain.vo.PurchaseTransactionImportFailureVO;
|
||||
import com.ruoyi.ccdi.mapper.CcdiPurchaseTransactionMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiPurchaseTransactionImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -33,6 +36,8 @@ import java.util.stream.Collectors;
|
||||
@EnableAsync
|
||||
public class CcdiPurchaseTransactionImportServiceImpl implements ICcdiPurchaseTransactionImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiPurchaseTransactionImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiPurchaseTransactionMapper transactionMapper;
|
||||
|
||||
@@ -43,11 +48,18 @@ public class CcdiPurchaseTransactionImportServiceImpl implements ICcdiPurchaseTr
|
||||
@Async
|
||||
@Transactional
|
||||
public void importTransactionAsync(List<CcdiPurchaseTransactionExcel> excelList, String taskId, String userName) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "采购交易信息", excelList.size(), userName);
|
||||
|
||||
List<CcdiPurchaseTransaction> newRecords = new ArrayList<>();
|
||||
List<PurchaseTransactionImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 批量查询已存在的采购事项ID
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的采购事项ID", excelList.size());
|
||||
Set<String> existingIds = getExistingPurchaseIds(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "采购事项ID", existingIds.size());
|
||||
|
||||
// 用于跟踪Excel文件内已处理的采购事项ID
|
||||
Set<String> processedIds = new HashSet<>();
|
||||
@@ -80,23 +92,39 @@ public class CcdiPurchaseTransactionImportServiceImpl implements ICcdiPurchaseTr
|
||||
processedIds.add(excel.getPurchaseId()); // 标记为已处理
|
||||
}
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
PurchaseTransactionImportFailureVO failure = new PurchaseTransactionImportFailureVO();
|
||||
BeanUtils.copyProperties(excel, failure);
|
||||
failure.setErrorMessage(e.getMessage());
|
||||
failures.add(failure);
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("采购事项ID=%s, 采购类别=%s, 标的物=%s",
|
||||
excel.getPurchaseId(), excel.getPurchaseCategory(), excel.getSubjectName());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:purchaseTransaction:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:purchaseTransaction:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -107,6 +135,11 @@ public class CcdiPurchaseTransactionImportServiceImpl implements ICcdiPurchaseTr
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus(taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "采购交易信息",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -9,8 +9,11 @@ import com.ruoyi.ccdi.domain.vo.ImportStatusVO;
|
||||
import com.ruoyi.ccdi.domain.vo.StaffEnterpriseRelationImportFailureVO;
|
||||
import com.ruoyi.ccdi.mapper.CcdiStaffEnterpriseRelationMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiStaffEnterpriseRelationImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -32,6 +35,8 @@ import java.util.stream.Collectors;
|
||||
@EnableAsync
|
||||
public class CcdiStaffEnterpriseRelationImportServiceImpl implements ICcdiStaffEnterpriseRelationImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiStaffEnterpriseRelationImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiStaffEnterpriseRelationMapper relationMapper;
|
||||
|
||||
@@ -42,11 +47,18 @@ public class CcdiStaffEnterpriseRelationImportServiceImpl implements ICcdiStaffE
|
||||
@Async
|
||||
@Transactional
|
||||
public void importRelationAsync(List<CcdiStaffEnterpriseRelationExcel> excelList, String taskId, String userName) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "员工实体关系", excelList.size(), userName);
|
||||
|
||||
List<CcdiStaffEnterpriseRelation> newRecords = new ArrayList<>();
|
||||
List<StaffEnterpriseRelationImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 批量查询已存在的person_id + social_credit_code组合
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的员工企业关系组合", excelList.size());
|
||||
Set<String> existingCombinations = getExistingCombinations(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "员工企业关系组合", existingCombinations.size());
|
||||
|
||||
// 用于跟踪Excel文件内已处理的组合
|
||||
Set<String> processedCombinations = new HashSet<>();
|
||||
@@ -92,23 +104,39 @@ public class CcdiStaffEnterpriseRelationImportServiceImpl implements ICcdiStaffE
|
||||
processedCombinations.add(combination); // 标记为已处理
|
||||
}
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
StaffEnterpriseRelationImportFailureVO failure = new StaffEnterpriseRelationImportFailureVO();
|
||||
BeanUtils.copyProperties(excel, failure);
|
||||
failure.setErrorMessage(e.getMessage());
|
||||
failures.add(failure);
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("身份证号=%s, 统一社会信用代码=%s, 企业名称=%s",
|
||||
excel.getPersonId(), excel.getSocialCreditCode(), excel.getEnterpriseName());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:staffEnterpriseRelation:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:staffEnterpriseRelation:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -119,6 +147,11 @@ public class CcdiStaffEnterpriseRelationImportServiceImpl implements ICcdiStaffE
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus(taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "员工实体关系",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -10,8 +10,11 @@ import com.ruoyi.ccdi.domain.vo.StaffFmyRelationImportFailureVO;
|
||||
import com.ruoyi.ccdi.enums.GenderEnum;
|
||||
import com.ruoyi.ccdi.mapper.CcdiStaffFmyRelationMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiStaffFmyRelationImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -32,6 +35,8 @@ import java.util.concurrent.TimeUnit;
|
||||
@EnableAsync
|
||||
public class CcdiStaffFmyRelationImportServiceImpl implements ICcdiStaffFmyRelationImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiStaffFmyRelationImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiStaffFmyRelationMapper relationMapper;
|
||||
|
||||
@@ -42,6 +47,11 @@ public class CcdiStaffFmyRelationImportServiceImpl implements ICcdiStaffFmyRelat
|
||||
@Async
|
||||
@Transactional
|
||||
public void importRelationAsync(List<CcdiStaffFmyRelationExcel> excelList, String taskId, String userName) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "员工亲属关系", excelList.size(), userName);
|
||||
|
||||
List<CcdiStaffFmyRelation> newRecords = new ArrayList<>();
|
||||
List<StaffFmyRelationImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
@@ -61,6 +71,7 @@ public class CcdiStaffFmyRelationImportServiceImpl implements ICcdiStaffFmyRelat
|
||||
// 2. 批量查询数据库中已存在的记录
|
||||
Set<String> existingKeys = new HashSet<>();
|
||||
if (!excelPersonIds.isEmpty() && !excelRelationCertNos.isEmpty()) {
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的亲属关系", excelList.size());
|
||||
List<CcdiStaffFmyRelation> existingRecords = relationMapper.selectExistingRelations(
|
||||
new ArrayList<>(excelPersonIds),
|
||||
new ArrayList<>(excelRelationCertNos)
|
||||
@@ -71,6 +82,7 @@ public class CcdiStaffFmyRelationImportServiceImpl implements ICcdiStaffFmyRelat
|
||||
String key = existing.getPersonId() + "|" + existing.getRelationCertNo();
|
||||
existingKeys.add(key);
|
||||
}
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "亲属关系", existingKeys.size());
|
||||
}
|
||||
|
||||
// ========== 第二步：处理数据 ==========
|
||||
@@ -116,23 +128,39 @@ public class CcdiStaffFmyRelationImportServiceImpl implements ICcdiStaffFmyRelat
|
||||
newRecords.add(relation);
|
||||
processedKeys.add(uniqueKey);
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
StaffFmyRelationImportFailureVO failure = new StaffFmyRelationImportFailureVO();
|
||||
BeanUtils.copyProperties(excel, failure);
|
||||
failure.setErrorMessage(e.getMessage());
|
||||
failures.add(failure);
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("员工身份证号=%s, 关系人=%s(%s)",
|
||||
excel.getPersonId(), excel.getRelationName(), excel.getRelationCertNo());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:staffFmyRelation:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:staffFmyRelation:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -143,6 +171,11 @@ public class CcdiStaffFmyRelationImportServiceImpl implements ICcdiStaffFmyRelat
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus(taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "员工亲属关系",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -11,9 +11,12 @@ import com.ruoyi.ccdi.domain.vo.RecruitmentImportFailureVO;
|
||||
import com.ruoyi.ccdi.enums.AdmitStatus;
|
||||
import com.ruoyi.ccdi.mapper.CcdiStaffRecruitmentMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiStaffRecruitmentImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.utils.IdCardUtil;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -35,6 +38,8 @@ import java.util.stream.Collectors;
|
||||
@EnableAsync
|
||||
public class CcdiStaffRecruitmentImportServiceImpl implements ICcdiStaffRecruitmentImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiStaffRecruitmentImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiStaffRecruitmentMapper recruitmentMapper;
|
||||
|
||||
@@ -47,11 +52,18 @@ public class CcdiStaffRecruitmentImportServiceImpl implements ICcdiStaffRecruitm
|
||||
public void importRecruitmentAsync(List<CcdiStaffRecruitmentExcel> excelList,
|
||||
String taskId,
|
||||
String userName) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "招聘信息", excelList.size(), userName);
|
||||
|
||||
List<CcdiStaffRecruitment> newRecords = new ArrayList<>();
|
||||
List<RecruitmentImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 批量查询已存在的招聘项目编号
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的招聘项目编号", excelList.size());
|
||||
Set<String> existingRecruitIds = getExistingRecruitIds(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "招聘项目编号", existingRecruitIds.size());
|
||||
|
||||
// 用于检测Excel内部的重复ID
|
||||
Set<String> excelProcessedIds = new HashSet<>();
|
||||
@@ -84,23 +96,39 @@ public class CcdiStaffRecruitmentImportServiceImpl implements ICcdiStaffRecruitm
|
||||
excelProcessedIds.add(excel.getRecruitId()); // 标记为已处理
|
||||
}
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
RecruitmentImportFailureVO failure = new RecruitmentImportFailureVO();
|
||||
BeanUtils.copyProperties(excel, failure);
|
||||
failure.setErrorMessage(e.getMessage());
|
||||
failures.add(failure);
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("招聘项目编号=%s, 项目名称=%s, 应聘人员=%s",
|
||||
excel.getRecruitId(), excel.getRecruitName(), excel.getCandName());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:recruitment:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:recruitment:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -111,6 +139,11 @@ public class CcdiStaffRecruitmentImportServiceImpl implements ICcdiStaffRecruitm
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus(taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "招聘信息",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -10,11 +10,14 @@ import com.ruoyi.ccdi.domain.vo.ImportStatusVO;
|
||||
import com.ruoyi.ccdi.domain.vo.StaffTransferImportFailureVO;
|
||||
import com.ruoyi.ccdi.mapper.CcdiStaffTransferMapper;
|
||||
import com.ruoyi.ccdi.service.ICcdiStaffTransferImportService;
|
||||
import com.ruoyi.ccdi.utils.ImportLogUtils;
|
||||
import com.ruoyi.common.core.domain.entity.SysDept;
|
||||
import com.ruoyi.common.utils.DictUtils;
|
||||
import com.ruoyi.common.utils.StringUtils;
|
||||
import com.ruoyi.system.mapper.SysDeptMapper;
|
||||
import jakarta.annotation.Resource;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.BeanUtils;
|
||||
import org.springframework.data.redis.core.RedisTemplate;
|
||||
import org.springframework.scheduling.annotation.Async;
|
||||
@@ -36,6 +39,8 @@ import java.util.stream.Collectors;
|
||||
@EnableAsync
|
||||
public class CcdiStaffTransferImportServiceImpl implements ICcdiStaffTransferImportService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CcdiStaffTransferImportServiceImpl.class);
|
||||
|
||||
@Resource
|
||||
private CcdiStaffTransferMapper transferMapper;
|
||||
|
||||
@@ -49,11 +54,18 @@ public class CcdiStaffTransferImportServiceImpl implements ICcdiStaffTransferImp
|
||||
@Async
|
||||
@Transactional
|
||||
public void importTransferAsync(List<CcdiStaffTransferExcel> excelList, String taskId, String userName) {
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
// 记录导入开始
|
||||
ImportLogUtils.logImportStart(log, taskId, "员工调动记录", excelList.size(), userName);
|
||||
|
||||
List<CcdiStaffTransfer> newRecords = new ArrayList<>();
|
||||
List<StaffTransferImportFailureVO> failures = new ArrayList<>();
|
||||
|
||||
// 批量查询已存在的唯一键组合
|
||||
ImportLogUtils.logBatchQueryStart(log, taskId, "已存在的调动记录", excelList.size());
|
||||
Set<String> existingKeys = getExistingTransferKeys(excelList);
|
||||
ImportLogUtils.logBatchQueryComplete(log, taskId, "调动记录", existingKeys.size());
|
||||
|
||||
// 用于检测Excel内部的重复键
|
||||
Set<String> excelProcessedKeys = new HashSet<>();
|
||||
@@ -98,23 +110,40 @@ public class CcdiStaffTransferImportServiceImpl implements ICcdiStaffTransferImp
|
||||
excelProcessedKeys.add(uniqueKey);
|
||||
}
|
||||
|
||||
// 记录进度
|
||||
ImportLogUtils.logProgress(log, taskId, i + 1, excelList.size(),
|
||||
newRecords.size(), failures.size());
|
||||
|
||||
} catch (Exception e) {
|
||||
StaffTransferImportFailureVO failure = new StaffTransferImportFailureVO();
|
||||
BeanUtils.copyProperties(excel, failure);
|
||||
failure.setErrorMessage(e.getMessage());
|
||||
failures.add(failure);
|
||||
|
||||
// 记录验证失败日志
|
||||
String keyData = String.format("员工ID=%s, 调动类型=%s, 调动日期=%s, 调动前部门ID=%s, 调动后部门ID=%s",
|
||||
excel.getStaffId(), excel.getTransferType(), excel.getTransferDate(),
|
||||
excel.getDeptIdBefore(), excel.getDeptIdAfter());
|
||||
ImportLogUtils.logValidationError(log, taskId, i + 1, e.getMessage(), keyData);
|
||||
}
|
||||
}
|
||||
|
||||
// 批量插入新数据
|
||||
if (!newRecords.isEmpty()) {
|
||||
ImportLogUtils.logBatchOperationStart(log, taskId, "插入",
|
||||
(newRecords.size() + 499) / 500, 500);
|
||||
saveBatch(newRecords, 500);
|
||||
}
|
||||
|
||||
// 保存失败记录到Redis
|
||||
if (!failures.isEmpty()) {
|
||||
String failuresKey = "import:staffTransfer:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
try {
|
||||
String failuresKey = "import:staffTransfer:" + taskId + ":failures";
|
||||
redisTemplate.opsForValue().set(failuresKey, failures, 7, TimeUnit.DAYS);
|
||||
ImportLogUtils.logRedisOperation(log, taskId, "保存失败记录", failures.size());
|
||||
} catch (Exception e) {
|
||||
ImportLogUtils.logRedisError(log, taskId, "保存失败记录", e);
|
||||
}
|
||||
}
|
||||
|
||||
ImportResult result = new ImportResult();
|
||||
@@ -125,6 +154,11 @@ public class CcdiStaffTransferImportServiceImpl implements ICcdiStaffTransferImp
|
||||
// 更新最终状态
|
||||
String finalStatus = result.getFailureCount() == 0 ? "SUCCESS" : "PARTIAL_SUCCESS";
|
||||
updateImportStatus(taskId, finalStatus, result);
|
||||
|
||||
// 记录导入完成
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
ImportLogUtils.logImportComplete(log, taskId, "员工调动记录",
|
||||
excelList.size(), result.getSuccessCount(), result.getFailureCount(), duration);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
package com.ruoyi.ccdi.utils;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
|
||||
/**
|
||||
* 导入日志工具类
|
||||
* 提供统一的日志格式和进度计算
|
||||
*
|
||||
* @author ruoyi
|
||||
* @date 2026-02-11
|
||||
*/
|
||||
public class ImportLogUtils {
|
||||
|
||||
/**
|
||||
* 记录导入开始
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param moduleName 模块名称
|
||||
* @param totalCount 总数据量
|
||||
* @param userName 操作人
|
||||
*/
|
||||
public static void logImportStart(Logger log, String taskId, String moduleName,
|
||||
int totalCount, String userName) {
|
||||
log.info("[任务ID: {}] 开始异步导入{}，数据量: {}条，操作人: {}",
|
||||
taskId, moduleName, totalCount, userName);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录批量查询开始
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param queryDesc 查询描述
|
||||
* @param queryCount 查询数量
|
||||
*/
|
||||
public static void logBatchQueryStart(Logger log, String taskId, String queryDesc, int queryCount) {
|
||||
log.info("[任务ID: {}] 批量查询{}，查询数量: {}个", taskId, queryDesc, queryCount);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录批量查询完成
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param queryDesc 查询描述
|
||||
* @param existingCount 已存在数量
|
||||
*/
|
||||
public static void logBatchQueryComplete(Logger log, String taskId, String queryDesc, int existingCount) {
|
||||
log.info("[任务ID: {}] 查询完成，已存在{}条", taskId, queryDesc, existingCount);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录进度（智能判断是否需要输出）
|
||||
* 每100条或每10%输出一次
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param current 当前处理数
|
||||
* @param total 总数
|
||||
* @param success 成功数
|
||||
* @param failure 失败数
|
||||
*/
|
||||
public static void logProgress(Logger log, String taskId, int current, int total,
|
||||
int success, int failure) {
|
||||
if (current <= 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
// 每100条或每10%输出一次进度
|
||||
boolean shouldLog = (current % 100 == 0) ||
|
||||
(current * 10 / total > (current - 1) * 10 / total) ||
|
||||
(current == total);
|
||||
|
||||
if (shouldLog) {
|
||||
int progress = current * 100 / total;
|
||||
log.info("[任务ID: {}] 数据处理进度: {}/{} ({}%), 成功: {}条, 失败: {}条",
|
||||
taskId, current, total, progress, success, failure);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录数据验证失败
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param rowNum 行号
|
||||
* @param errorMsg 错误消息
|
||||
* @param keyData 关键数据（可为null）
|
||||
*/
|
||||
public static void logValidationError(Logger log, String taskId, int rowNum,
|
||||
String errorMsg, String keyData) {
|
||||
log.warn("[任务ID: {}] [第{}行] 数据验证失败: {}", taskId, rowNum, errorMsg);
|
||||
if (keyData != null && !keyData.isEmpty()) {
|
||||
log.warn("[任务ID: {}] 失败数据详情: {}", taskId, keyData);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录批量操作开始
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param operation 操作描述
|
||||
* @param totalBatch 总批次数
|
||||
* @param batchSize 每批大小
|
||||
*/
|
||||
public static void logBatchOperationStart(Logger log, String taskId, String operation,
|
||||
int totalBatch, int batchSize) {
|
||||
log.info("[任务ID: {}] 开始批量{}，总批次: {}, 每批: {}条",
|
||||
taskId, operation, totalBatch, batchSize);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录单个批次操作
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param operation 操作描述
|
||||
* @param batchNum 当前批次号
|
||||
* @param totalBatch 总批次数
|
||||
* @param batchSize 本批数量
|
||||
*/
|
||||
public static void logBatchOperation(Logger log, String taskId, String operation,
|
||||
int batchNum, int totalBatch, int batchSize) {
|
||||
log.info("[任务ID: {}] 执行批次 {}/{}, 本批数量: {}条",
|
||||
taskId, batchNum, totalBatch, batchSize);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录单个批次完成
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param operation 操作描述
|
||||
* @param batchNum 当前批次号
|
||||
* @param totalBatch 总批次数
|
||||
* @param success 成功数量
|
||||
*/
|
||||
public static void logBatchComplete(Logger log, String taskId, String operation,
|
||||
int batchNum, int totalBatch, int success) {
|
||||
log.info("[任务ID: {}] 批次 {}/{} {}完成，成功: {}条",
|
||||
taskId, batchNum, totalBatch, operation, success);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录Redis缓存操作
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param operation 操作描述（如"保存失败记录"）
|
||||
* @param count 数量
|
||||
*/
|
||||
public static void logRedisOperation(Logger log, String taskId, String operation, int count) {
|
||||
log.debug("[任务ID: {}] {}到Redis，数量: {}条", taskId, operation, count);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录Redis缓存异常
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param operation 操作描述
|
||||
* @param e 异常
|
||||
*/
|
||||
public static void logRedisError(Logger log, String taskId, String operation, Exception e) {
|
||||
log.error("[任务ID: {}] {}到Redis失败，不影响导入结果", taskId, operation, e);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录导入完成
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param moduleName 模块名称
|
||||
* @param total 总数
|
||||
* @param success 成功数
|
||||
* @param failure 失败数
|
||||
* @param duration 耗时（毫秒）
|
||||
*/
|
||||
public static void logImportComplete(Logger log, String taskId, String moduleName,
|
||||
int total, int success, int failure, long duration) {
|
||||
log.info("[任务ID: {}] {}导入完成！总数: {}条, 成功: {}条, 失败: {}条, 耗时: {}ms",
|
||||
taskId, moduleName, total, success, failure, duration);
|
||||
|
||||
// 如果有失败，记录失败汇总
|
||||
if (failure > 0) {
|
||||
log.warn("[任务ID: {}] 导入完成，但有{}条数据失败，请查看失败记录详情", taskId, failure);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录异常
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param errorMsg 错误描述
|
||||
* @param e 异常
|
||||
*/
|
||||
public static void logException(Logger log, String taskId, String errorMsg, Exception e) {
|
||||
log.error("[任务ID: {}] {}", taskId, errorMsg, e);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录事务回滚
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param processed 已处理数量
|
||||
* @param total 总数量
|
||||
* @param success 成功数量
|
||||
* @param failure 失败数量
|
||||
* @param e 异常
|
||||
*/
|
||||
public static void logTransactionRollback(Logger log, String taskId, int processed,
|
||||
int total, int success, int failure, Exception e) {
|
||||
log.error("[任务ID: {}] 导入失败，事务已回滚。已处理: {}/{}条", taskId, processed, total, e);
|
||||
log.error("[任务ID: {}] 回滚前统计 - 新增: {}条, 失败: {}条", taskId, success, failure);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录唯一性冲突
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param rowNum 行号
|
||||
* @param conflictDesc 冲突描述
|
||||
*/
|
||||
public static void logUniqueConflict(Logger log, String taskId, int rowNum, String conflictDesc) {
|
||||
log.warn("[任务ID: {}] [第{}行] {}", taskId, rowNum, conflictDesc);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录失败原因统计
|
||||
*
|
||||
* @param log 日志记录器
|
||||
* @param taskId 任务ID
|
||||
* @param errorStats 错误统计Map
|
||||
*/
|
||||
public static void logErrorStatistics(Logger log, String taskId, java.util.Map<String, Long> errorStats) {
|
||||
if (errorStats != null && !errorStats.isEmpty()) {
|
||||
String statsStr = errorStats.entrySet().stream()
|
||||
.map(entry -> entry.getKey() + "=" + entry.getValue() + "条")
|
||||
.collect(java.util.stream.Collectors.joining(", "));
|
||||
log.warn("[任务ID: {}] 失败原因统计: {}", taskId, statsStr);
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user