Stream Load 出现Total file size will exceed limit 1073741824

Viewed 40

version:2.1.10
场景:使用Stream Load导入数据做实时分析,每秒会调用10次左右的。每次请求的文件远低于数值
这个limit具体是什么产生的,如何调高配置

2 Answers

我指的是TmpFileMgr里的MAX_TOTAL_FILE_SIZE_BYTES

public synchronized TmpFile upload(UploadFile uploadFile) throws TmpFileException {
    if (uploadFile.file.getSize() > MAX_SINGLE_FILE_SIZE) {
        throw new TmpFileException("File size " + uploadFile.file.getSize()
                + " exceed limit " + MAX_SINGLE_FILE_SIZE);
    }

    if (totalFileSize + uploadFile.file.getSize() > MAX_TOTAL_FILE_SIZE_BYTES) {
        throw new TmpFileException("Total file size will exceed limit " + MAX_TOTAL_FILE_SIZE_BYTES);
    }

    if (fileMap.size() > MAX_TOTAL_FILE_NUM) {
        throw new TmpFileException("Number of temp file " + fileMap.size() + " exceed limit " + MAX_TOTAL_FILE_NUM);
    }

    long fileId = fileIdGenerator.incrementAndGet();
    String fileUUID = UUID.randomUUID().toString();

    TmpFile tmpFile = new TmpFile(fileId, fileUUID, uploadFile.file.getOriginalFilename(),
            uploadFile.file.getSize(), uploadFile.columnSeparator);
    try {
        tmpFile.save(uploadFile.file);
    } catch (IOException e) {
        throw new TmpFileException("Failed to upload file. Reason: " + e.getMessage());
    }
    fileMap.put(tmpFile.id, tmpFile);
    totalFileSize += uploadFile.file.getSize();
    return tmpFile;
}

这里的totalFileSize在实例化后一直在增加,我没看到哪里有清空的方式,比如:UploadAction方法中创建的
private static TmpFileMgr fileMgr = new TmpFileMgr(Config.tmp_dir);
导致这个totalFileSize始终在上升,最终导致Total file size will exceed limit 1073741824