花费时间读取记录并使用java流和spring引导JPA保存在另一个表中

Question

从下面的代码中，我想从PersonInfoEntity表中保存详细信息，对于每个personInfo，我想将记录存储在researchchinfoentity中。我有大约100,000条记录要从PersonInfoEntity插入到researchchinfoentity。问题是下面的代码，它正在谈论大量的时间来保存在ResearchInfo表中的记录。在将近3小时的时间里，它存储了大约2000条记录。请让我知道在哪里需要时间来执行，或者是否需要任何代码优化来插入批量记录。

示例代码:

List<PersonInfoEntity> PersonInfoEntityList = personInfoRepository.findAll();
Map<Long, List<PersonInfoEntity>> personInfoEntityMap = PersonInfoEntityList.stream().
collect(Collectors.groupingBy(
personInfoResponse ->  personInfoResponse.getPerson().getPersonId()
));
List<ResearchEntity> researchEntityList = researchRepository.findAll();
List<ResearchInfoEntity> researchInfoEntityList = new ArrayList<>();
for(ResearchEntity researchEntity : researchEntityList){
List<PersonInfoEntity> personInfoResponseList1 = personInfoEntityMap.get(researchEntity.getPerson().getPersonId());
if(Objects.nonNull(personInfoResponseList1)) {
for (PersonInfoEntity PersonInfoEntity : personInfoResponseList1) {
ResearchInfoEntity researchInfoEntity = new ResearchInfoEntity();
researchInfoEntity.setRecovery(ResearchEntity);
researchInfoEntity.setMilestoneGroupId(PersonInfoEntity.getMilestoneGroupId());
researchInfoEntity.setMilestoneId(PersonInfoEntity.getMilestoneId());
researchInfoEntity.setMilestoneStepId(PersonInfoEntity.getMilestoneStepId());
researchInfoEntity.setMilestoneStepValue(PersonInfoEntity.getMilestoneStepValue());
researchInfoEntity.setCreateBy(PersonInfoEntity.getCreateBy());
researchInfoEntity.setCreateTime(PersonInfoEntity.getCreateTime());
researchInfoEntity.setUpdateBy(PersonInfoEntity.getUpdateBy());
researchInfoEntity.setUpdateTime(PersonInfoEntity.getUpdateTime());
researchInfoEntityList.add(researchInfoEntity);
// researchInfoEntityRepository.save(recoveryMilestoneStep);
}
researchInfoEntityRepository.saveAll(researchInfoEntityList);
}
}

Answer 1

您的PersonInfoEntity到researchInfoEntity的映射可以异步完成。

您也可以尝试使用parallelStream:

public void yourMethod() {
List<PersonInfoEntity> PersonInfoEntityList = personInfoRepository.findAll();
Map<Long, List<PersonInfoEntity>> personInfoEntityMap = PersonInfoEntityList.stream().
collect(Collectors.groupingBy(
personInfoResponse -> personInfoResponse.getPerson().getPersonId()
));
List<ResearchEntity> researchEntityList = researchRepository.findAll();
List<ResearchInfoEntity> researchInfoEntityList = new ArrayList<>();
for (ResearchEntity researchEntity : researchEntityList) {
List<PersonInfoEntity> personInfoResponseList1 = personInfoEntityMap.get(researchEntity.getPerson().getPersonId());
if (Objects.nonNull(personInfoResponseList1)) {
List<ResearchInfoEntity> researchInfoListFromPerson = personInfoResponseList1
.parallelStream() // <--
.map(this::toResearchInfoEntity)
.collect(Collectors.toList());
researchInfoEntityList.addAll(researchInfoListFromPerson);
}
}
researchInfoEntityRepository.saveAll(researchInfoEntityList);
}
private ResearchInfoEntity toResearchInfoEntity(PersonInfoEntity personInfoEntity) {
ResearchInfoEntity researchInfoEntity = new ResearchInfoEntity();
researchInfoEntity.setRecovery(ResearchEntity);
researchInfoEntity.setMilestoneGroupId(PersonInfoEntity.getMilestoneGroupId());
researchInfoEntity.setMilestoneId(PersonInfoEntity.getMilestoneId());
researchInfoEntity.setMilestoneStepId(PersonInfoEntity.getMilestoneStepId());
researchInfoEntity.setMilestoneStepValue(PersonInfoEntity.getMilestoneStepValue());
researchInfoEntity.setCreateBy(PersonInfoEntity.getCreateBy());
researchInfoEntity.setCreateTime(PersonInfoEntity.getCreateTime());
researchInfoEntity.setUpdateBy(PersonInfoEntity.getUpdateBy());
researchInfoEntity.setUpdateTime(PersonInfoEntity.getUpdateTime());
return researchInfoEntity;
}

同时，尝试一次处理100,000个元素会占用大量内存。你可以尝试批量处理你的元素。

例如:

public void export(int batchSize) {
int numberOfElementFetched;
int pageCount = 0;
do {
// entityManager.clear(); // Needed only if you are in a transactional state, you need to clear the entity manager.
// Otherwise, for every iteration, it will keep previous fetched elements in memory
PageRequest requestByBatch = PageRequest.of(pageCount, batchSize, Sort.by(Sort.Direction.ASC));
numberOfElementFetched = yourMethod(requestByBatch);
pageCount++;
}
while (numberOfElementFetched == batchSize);
}
public void yourMethod(PageRequest pageRequest) {
List<PersonInfoEntity> PersonInfoEntityList = personInfoRepository.findAll();
Map<Long, List<PersonInfoEntity>> personInfoEntityMap = PersonInfoEntityList.stream().
collect(Collectors.groupingBy(
personInfoResponse -> personInfoResponse.getPerson().getPersonId()
));
List<ResearchEntity> researchEntityList = researchRepository.findAll(pageRequest).getContent();
List<ResearchInfoEntity> researchInfoEntityList = new ArrayList<>();
for (ResearchEntity researchEntity : researchEntityList) {
List<PersonInfoEntity> personInfoResponseList1 = personInfoEntityMap.get(researchEntity.getPerson().getPersonId());
if (Objects.nonNull(personInfoResponseList1)) {
for (PersonInfoEntity PersonInfoEntity : personInfoResponseList1) {
ResearchInfoEntity researchInfoEntity = new ResearchInfoEntity();
researchInfoEntity.setRecovery(ResearchEntity);
researchInfoEntity.setMilestoneGroupId(PersonInfoEntity.getMilestoneGroupId());
researchInfoEntity.setMilestoneId(PersonInfoEntity.getMilestoneId());
researchInfoEntity.setMilestoneStepId(PersonInfoEntity.getMilestoneStepId());
researchInfoEntity.setMilestoneStepValue(PersonInfoEntity.getMilestoneStepValue());
researchInfoEntity.setCreateBy(PersonInfoEntity.getCreateBy());
researchInfoEntity.setCreateTime(PersonInfoEntity.getCreateTime());
researchInfoEntity.setUpdateBy(PersonInfoEntity.getUpdateBy());
researchInfoEntity.setUpdateTime(PersonInfoEntity.getUpdateTime());
researchInfoEntityList.add(researchInfoEntity);
}
}
}
researchInfoEntityRepository.saveAll(researchInfoEntityList);
return researchEntityList.getSize();
}