refactor(core): Generalize binary data manager interface (no-changelog) (#7164)

Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)

This PR: 
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.

Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.

Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
This commit is contained in:
Iván Ovejero
2023-09-22 17:22:12 +02:00
committed by GitHub
parent 4614e1e1c9
commit 6d6e2488c6
46 changed files with 602 additions and 552 deletions

View File

@@ -1,9 +1,7 @@
import type { Readable } from 'stream';
import type {
IPollResponse,
ITriggerResponse,
IWorkflowSettings as IWorkflowSettingsWorkflow,
BinaryMetadata,
ValidationResult,
} from 'n8n-workflow';
@@ -34,27 +32,6 @@ export interface IWorkflowData {
triggerResponses?: ITriggerResponse[];
}
export interface IBinaryDataConfig {
mode: 'default' | 'filesystem';
availableModes: string;
localStoragePath: string;
}
export interface IBinaryDataManager {
init(startPurger: boolean): Promise<void>;
getFileSize(filePath: string): Promise<number>;
copyBinaryFile(filePath: string, executionId: string): Promise<string>;
storeBinaryMetadata(identifier: string, metadata: BinaryMetadata): Promise<void>;
getBinaryMetadata(identifier: string): Promise<BinaryMetadata>;
storeBinaryData(binaryData: Buffer | Readable, executionId: string): Promise<string>;
retrieveBinaryDataByIdentifier(identifier: string): Promise<Buffer>;
getBinaryPath(identifier: string): string;
getBinaryStream(identifier: string, chunkSize?: number): Readable;
deleteBinaryDataByIdentifier(identifier: string): Promise<void>;
duplicateBinaryDataByIdentifier(binaryDataId: string, prefix: string): Promise<string>;
deleteBinaryDataByExecutionIds(executionIds: string[]): Promise<string[]>;
}
export namespace n8n {
export interface PackageJson {
name: string;