mirror of
https://github.com/Abdulazizzn/n8n-enterprise-unlocked.git
synced 2025-12-17 18:12:04 +00:00
feat(editor): Implement Resource Mapper component (#6207)
* ⚡ scaffolding * ⚡ finished scaffolding * ⚡ renamed types * ⚡ updated subtitle * ⚡ renamed functions file, UI updates * ⚡ query parameters fixes, ui updates, refactoring * ⚡ fixes for credentials test, setup for error parsing * ⚡ rlc for schema and table, error handling tweaks * ⚡ delete operation, new options * ⚡ columns loader * ⚡ linter fixes * ⚡ where clauses setup * ⚡ logic for processing where clauses * ⚡ select operation * ⚡ refactoring * ⚡ data mode for insert and update, wip * ⚡ data mapping, insert update, skip on conflict option * ⚡ select columns with spaces fix * ⚡ update operation update, wip * ⚡ finished update operation * ⚡ upsert operation * ⚡ ui fixes * Copy updates. * Copy updates. * ⚡ option to convert empty strings to nulls, schema checks * ⚡ UI requested updates * ⚡ ssh setup WIP * ⚡ fixes, ssh WIP * ⚡ ssh fixes, credentials * ⚡ credentials testing update * ⚡ uncaught error fix * ⚡ clean up * ⚡ address in use fix * ⚡ improved error message * ⚡ tests setup * ⚡ unit tests wip * ⚡ config files clean up * ⚡ utils unit tests * ⚡ refactoring * ⚡ setup for testing operations, tests for deleteTable operation * ⚡ executeQuery and insert operations tests * ⚡ select, update, upsert operations tests * ⚡ runQueries tests setup * ⚡ hint to query * Copy updates. * ⚡ ui fixes * ⚡ clean up * ⚡ error message update * ⚡ ui update * Minor tweaks to query params decription. * feat(Google Sheets Node): Implement Resource mapper in Google Sheets node (#5752) * ✨ Added initial resource mapping support in google sheets node * ✨ Wired mapping API endpoint with node-specific logic for fetching mapping fields * ✨ Implementing mapping fields logic for google sheets * ✨ Updating Google Sheets execute methods to support resource mapper fields * 🚧 Added initial version of `ResourceLocator` component * 👌 Added `update` mode to resource mapper modes * 👌 Addressing PR feedback * 👌 Removing leftover const reference * 👕 Fixing lint errors * ⚡ singlton for conections * ⚡ credentials test fix, clean up * feat(Postgres Node): Add resource mapper to new version of Postgres node (#5814) * ⚡ scaffolding * ⚡ finished scaffolding * ⚡ renamed types * ⚡ updated subtitle * ⚡ renamed functions file, UI updates * ⚡ query parameters fixes, ui updates, refactoring * ⚡ fixes for credentials test, setup for error parsing * ⚡ rlc for schema and table, error handling tweaks * ⚡ delete operation, new options * ⚡ columns loader * ⚡ linter fixes * ⚡ where clauses setup * ⚡ logic for processing where clauses * ⚡ select operation * ⚡ refactoring * ⚡ data mode for insert and update, wip * ⚡ data mapping, insert update, skip on conflict option * ⚡ select columns with spaces fix * ⚡ update operation update, wip * ⚡ finished update operation * ⚡ upsert operation * ⚡ ui fixes * Copy updates. * Copy updates. * ⚡ option to convert empty strings to nulls, schema checks * ⚡ UI requested updates * ⚡ ssh setup WIP * ⚡ fixes, ssh WIP * ⚡ ssh fixes, credentials * ⚡ credentials testing update * ⚡ uncaught error fix * ⚡ clean up * ⚡ address in use fix * ⚡ improved error message * ⚡ tests setup * ⚡ unit tests wip * ⚡ config files clean up * ⚡ utils unit tests * ⚡ refactoring * ⚡ setup for testing operations, tests for deleteTable operation * ⚡ executeQuery and insert operations tests * ⚡ select, update, upsert operations tests * ⚡ runQueries tests setup * ⚡ hint to query * Copy updates. * ⚡ ui fixes * ⚡ clean up * ⚡ error message update * ⚡ ui update * Minor tweaks to query params decription. * ✨ Updated Postgres node to use resource mapper component * ✨ Implemented postgres <-> resource mapper type mapping * ✨ Updated Postgres node execution to use resource mapper fields in v3 * 🔥 Removing unused import --------- Co-authored-by: Michael Kret <michael.k@radency.com> Co-authored-by: Giulio Andreini <g.andreini@gmail.com> * feat(core): Resource editor componend P0 (#5970) * ✨ Added inital value of mapping mode dropdown * ✨ Finished mapping mode selector * ✨ Finished implementing mapping mode selector * ✨ Implemented 'Columns to match on' dropdown * ✨ Implemented `loadOptionsDependOn` support in resource mapper * ✨ Implemented initial version of mapping fields * ✨ Implementing dependant fields watcher in new component setup * ✨ Generating correct resource mapper field types. Added `supportAutoMap` to node specification and UI. Not showing fields with `display=false`. Pre-selecting matching columns if it's the only one * ✨ Handling matching columns correctly in UI * ✨ Saving and loading resourceMapper values in component * ✨ Implemented proper data saving and loading * ✨ ResourceMapper component refactor, fixing value save/load * ✨ Refactoring MatchingColumnSelect component. Updating Sheets node to use single key match and Postgres to use multi key * ✨ Updated Google Sheets node to work with the new UI * ✨ Updating Postgres Node to work with new UI * ✨ Additional loading indicator that shown if there is no mapping mode selector * ✨ Removing hard-coded values, fixing matching columns ordering, refactoring * ✨ Updating field names in nodes * ✨ Fixing minor UI issues * ✨ Implemented matching fields filter logic * ✨ Moving loading label outside of fields list * ✅ Added initial unit tests for resource mapper * ✅ Finished default rendering test * ✅ Test refactoring * ✅ Finished unit tests * 🔨 Updating the way i18n is used in resource mapper components * ✔️ Fixing value to match on logic for postgres node * ✨ Hiding mapping fields when auto-map mode is selected * ✨ Syncing selected mapping mode between components * ✨ Fixing dateTime input rendering and adding update check to Postgres node * ✨ Properly handling database connections. Sending null for empty string values. * 💄 Updated wording in the error message for non-existing rows * ✨ Fixing issues with selected matching values * ✔️ Updating unit tests after matching logic update * ✨ Updating matching columns when new fields are loaded * ✨ Defaulting to null for empty parameter values * ✨ Allowing zero as valid value for number imputs * ✨ Updated list of types that use datepicker as widger * ✨ Using text inputs for time types * ✨ Initial mapping field rework * ✨ Added new component for mapping fields, moved bit of logic from root component to matching selector, fixing some lint errors * ✨ Added tooltip for columns that cannot be deleted * ✨ Saving deleted values in parameter value * ✨ Implemented control to add/remove mapping fields * ✨ Syncing field list with add field dropdown when changing dependent values * ✨ Not showing removed fields in matching columns selector. Updating wording in matching columns selector description * ✨ Implementing disabled states for add/remove all fields options * ✨ Saving removed columns separately, updating copy * ✨ Implemented resource mapper values validation * ✨ Updated validation logic and error input styling * ✨ Validating resource mapper fields when new nodes are added * ✨ Using node field words in validation, refactoring resource mapper component * ✨ Implemented schema syncing and add/remove all fields * ✨ Implemented custom parameter actions * ✨ Implemented loading indicator in parameter options * 🔨 Removing unnecessary constants and vue props * ✨ Handling default values properly * ✨ Fixing validation logic * 👕 Fixing lint errors * ⚡ Fixing type issues * ⚡ Not showing fields by default if `addAllFields` is set to `false` * ✨ Implemented field type validation in resource mapper * ✨ Updated casing in copy, removed all/remove all option from bottom menu * ✨ Added auto mapping mode notice * ✨ Added support for more types in validation * ✨ Added support for enumerated values * ✨ Fixing imports after merging * ✨ Not showing removed fields in matching columns selector. Refactoring validation logic. * 👕 Fixing imports * ✔️ Updating unit tests * ✅ Added resource mapper schema tests * ⚡ Removing `match` from resource mapper field definition, fixing matching columns loading * ⚡ Fixed schema merging * ⚡ update operation return data fix * ⚡ review * 🐛 Added missing import * 💄 Updating parameter actions icon based on the ui review * 💄 Updating word capitalisation in tooltips * 💄 Added empty state to mapping fields list * 💄 Removing asterisk from fields, updating tooltips for matching fields * ⚡ Preventing matching fields from being removed by 'Remove All option' * ⚡ Not showing hidden fields in the `Add field` dropdown * ⚡ Added support for custom matching columns labels * ⚡ query optimization * ⚡ fix * ⚡ Optimizing Postgres node enumeration logic * ⚡ Added empty state for matching columns * ⚡ Only fully loading fields if there is no schema fetched * ⚡ Hiding mapping fields if there is no matching columns available in the schema * ✔️ Fixing minor issues * ✨ Implemented runtime type validation * 🔨 Refactoring validation logic * ✨ Implemented required check, added more custom messages * ✨ Skipping boolean type in required check * Type check improvements * ✨ Only reloading fields if dependent values actually change * ✨ Adding item index to validation error title * ✨ Updating Postgres fetching logic, using resource mapper mode to determine if a field can be deleted * ✨ Resetting field values when adding them via the addAll option * ⚡ Using minor version (2.2) for new Postgres node * ⚡ Implemented proper date validation and type casting * 👕 Consolidating typing * ✅ Added unit tests for type validations * 👌 Addressing front-end review comments * ⚡ More refactoring to address review changes * ⚡ Updating leftover props * ⚡ Added fallback for ISO dates with invalid timezones * Added timestamp to datetime test cases * ⚡ Reseting matching columns if operation changes * ⚡ Not forcing auto-increment fields to be filled in in Postgres node. Handling null values * 💄 Added a custom message for invalid dates * ⚡ Better handling of JSON values * ⚡ Updating codemirror readonly stauts based on component property, handling objects in json validation * Deleting leftover console.log * ⚡ Better time validation * ⚡ Fixing build error after merging * 👕 Fixing lint error * ⚡ Updating node configuration values * ⚡ Handling postgres arrays better * ⚡ Handling SQL array syntax * ⚡ Updating time validation rules to include timezone * ⚡ Sending expressions that resolve to `null` or `undefined` by the resource mapper to delete cell content in Google Sheets * ⚡ Allowing removed fields to be selected for match * ⚡ Updated the query for fetching unique columns and primary keys * ⚡ Optimizing the unique query * ⚡ Setting timezone to all parsed dates * ⚡ Addressing PR review feedback * ⚡ Configuring Sheets node for production, minor vue component update * New cases added to the TypeValidation test. * ✅ Tweaking validation rules for arrays/objects and updating test cases --------- Co-authored-by: Michael Kret <michael.k@radency.com> Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
This commit is contained in:
Milorad FIlipović
committed by
GitHub
GitHub
parent
5ae1124106
commit
04cfa548af
@@ -37,11 +37,16 @@ import type {
|
||||
IWorkflowExecuteAdditionalData,
|
||||
NodeParameterValue,
|
||||
WebhookHttpMethod,
|
||||
FieldType,
|
||||
INodePropertyOptions,
|
||||
ResourceMapperValue,
|
||||
ValidationResult,
|
||||
} from './Interfaces';
|
||||
import { isValidResourceLocatorParameterValue } from './type-guards';
|
||||
import { isResourceMapperValue, isValidResourceLocatorParameterValue } from './type-guards';
|
||||
import { deepCopy } from './utils';
|
||||
|
||||
import type { Workflow } from './Workflow';
|
||||
import { DateTime } from 'luxon';
|
||||
|
||||
export const cronNodeOptions: INodePropertyCollection[] = [
|
||||
{
|
||||
@@ -1097,6 +1102,173 @@ export function nodeIssuesToString(issues: INodeIssues, node?: INode): string[]
|
||||
return nodeIssues;
|
||||
}
|
||||
|
||||
// Validates field against the schema and tries to parse it to the correct type
|
||||
export const validateFieldType = (
|
||||
fieldName: string,
|
||||
value: unknown,
|
||||
type: FieldType,
|
||||
options?: INodePropertyOptions[],
|
||||
): ValidationResult => {
|
||||
if (value === null || value === undefined) return { valid: true };
|
||||
const defaultErrorMessage = `'${fieldName}' expects a ${type} but we got '${String(value)}'.`;
|
||||
switch (type.toLowerCase()) {
|
||||
case 'number': {
|
||||
try {
|
||||
return { valid: true, newValue: tryToParseNumber(value) };
|
||||
} catch (e) {
|
||||
return { valid: false, errorMessage: defaultErrorMessage };
|
||||
}
|
||||
}
|
||||
case 'boolean': {
|
||||
try {
|
||||
return { valid: true, newValue: tryToParseBoolean(value) };
|
||||
} catch (e) {
|
||||
return { valid: false, errorMessage: defaultErrorMessage };
|
||||
}
|
||||
}
|
||||
case 'datetime': {
|
||||
try {
|
||||
return { valid: true, newValue: tryToParseDateTime(value) };
|
||||
} catch (e) {
|
||||
const luxonDocsURL =
|
||||
'https://moment.github.io/luxon/api-docs/index.html#datetimefromformat';
|
||||
const errorMessage = `${defaultErrorMessage} <br/><br/> Consider using <a href="${luxonDocsURL}" target="_blank"><code>DateTime.fromFormat</code></a> to work with custom date formats.`;
|
||||
return { valid: false, errorMessage };
|
||||
}
|
||||
}
|
||||
case 'time': {
|
||||
try {
|
||||
return { valid: true, newValue: tryToParseTime(value) };
|
||||
} catch (e) {
|
||||
return {
|
||||
valid: false,
|
||||
errorMessage: `'${fieldName}' expects time (hh:mm:(:ss)) but we got '${String(value)}'.`,
|
||||
};
|
||||
}
|
||||
}
|
||||
case 'object': {
|
||||
try {
|
||||
return { valid: true, newValue: tryToParseObject(value) };
|
||||
} catch (e) {
|
||||
return { valid: false, errorMessage: defaultErrorMessage };
|
||||
}
|
||||
}
|
||||
case 'array': {
|
||||
try {
|
||||
return { valid: true, newValue: tryToParseArray(value) };
|
||||
} catch (e) {
|
||||
return { valid: false, errorMessage: defaultErrorMessage };
|
||||
}
|
||||
}
|
||||
case 'options': {
|
||||
const validOptions = options?.map((option) => option.value).join(', ') || '';
|
||||
const isValidOption = options?.some((option) => option.value === value) || false;
|
||||
|
||||
if (!isValidOption) {
|
||||
return {
|
||||
valid: false,
|
||||
errorMessage: `'${fieldName}' expects one of the following values: [${validOptions}] but we got '${String(
|
||||
value,
|
||||
)}'`,
|
||||
};
|
||||
}
|
||||
return { valid: true, newValue: value };
|
||||
}
|
||||
default: {
|
||||
return { valid: true, newValue: value };
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
export const tryToParseNumber = (value: unknown): number => {
|
||||
const isValidNumber = !isNaN(Number(value));
|
||||
|
||||
if (!isValidNumber) {
|
||||
throw new Error(`Could not parse '${String(value)}' to number.`);
|
||||
}
|
||||
return Number(value);
|
||||
};
|
||||
|
||||
export const tryToParseBoolean = (value: unknown): value is boolean => {
|
||||
if (typeof value === 'boolean') {
|
||||
return value;
|
||||
}
|
||||
|
||||
if (typeof value === 'string' && ['true', 'false'].includes(value.toLowerCase())) {
|
||||
return value.toLowerCase() === 'true';
|
||||
}
|
||||
|
||||
const num = Number(value);
|
||||
if (num === 0) {
|
||||
return false;
|
||||
} else if (num === 1) {
|
||||
return true;
|
||||
}
|
||||
throw new Error(`Could not parse '${String(value)}' to boolean.`);
|
||||
};
|
||||
|
||||
export const tryToParseDateTime = (value: unknown): DateTime => {
|
||||
const dateString = String(value).trim();
|
||||
|
||||
// Rely on luxon to parse different date formats
|
||||
const isoDate = DateTime.fromISO(dateString, { setZone: true });
|
||||
if (isoDate.isValid) {
|
||||
return isoDate;
|
||||
}
|
||||
const httpDate = DateTime.fromHTTP(dateString, { setZone: true });
|
||||
if (httpDate.isValid) {
|
||||
return httpDate;
|
||||
}
|
||||
const rfc2822Date = DateTime.fromRFC2822(dateString, { setZone: true });
|
||||
if (rfc2822Date.isValid) {
|
||||
return rfc2822Date;
|
||||
}
|
||||
const sqlDate = DateTime.fromSQL(dateString, { setZone: true });
|
||||
if (sqlDate.isValid) {
|
||||
return sqlDate;
|
||||
}
|
||||
|
||||
throw new Error(`The value "${dateString}" is not a valid date.`);
|
||||
};
|
||||
|
||||
export const tryToParseTime = (value: unknown): string => {
|
||||
const isTimeInput = /^\d{2}:\d{2}(:\d{2})?((\-|\+)\d{4})?((\-|\+)\d{1,2}(:\d{2})?)?$/s.test(
|
||||
String(value),
|
||||
);
|
||||
if (!isTimeInput) {
|
||||
throw new Error(`The value "${String(value)}" is not a valid time.`);
|
||||
}
|
||||
return String(value);
|
||||
};
|
||||
|
||||
export const tryToParseArray = (value: unknown): unknown[] => {
|
||||
try {
|
||||
const parsed = JSON.parse(String(value));
|
||||
if (!Array.isArray(parsed)) {
|
||||
throw new Error(`The value "${String(value)}" is not a valid array.`);
|
||||
}
|
||||
return parsed;
|
||||
} catch (e) {
|
||||
throw new Error(`The value "${String(value)}" is not a valid array.`);
|
||||
}
|
||||
};
|
||||
|
||||
export const tryToParseObject = (value: unknown): object => {
|
||||
if (value && typeof value === 'object' && !Array.isArray(value)) {
|
||||
return value;
|
||||
}
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
|
||||
const o = JSON.parse(String(value));
|
||||
if (typeof o !== 'object' || Array.isArray(o)) {
|
||||
throw new Error(`The value "${String(value)}" is not a valid object.`);
|
||||
}
|
||||
return o;
|
||||
} catch (e) {
|
||||
throw new Error(`The value "${String(value)}" is not a valid object.`);
|
||||
}
|
||||
};
|
||||
|
||||
/*
|
||||
* Validates resource locator node parameters based on validation ruled defined in each parameter mode
|
||||
*
|
||||
@@ -1128,6 +1300,42 @@ export const validateResourceLocatorParameter = (
|
||||
return validationErrors;
|
||||
};
|
||||
|
||||
/*
|
||||
* Validates resource mapper values based on service schema
|
||||
*
|
||||
*/
|
||||
export const validateResourceMapperParameter = (
|
||||
nodeProperties: INodeProperties,
|
||||
value: ResourceMapperValue,
|
||||
skipRequiredCheck = false,
|
||||
): Record<string, string[]> => {
|
||||
const issues: Record<string, string[]> = {};
|
||||
let fieldWordSingular =
|
||||
nodeProperties.typeOptions?.resourceMapper?.fieldWords?.singular || 'Field';
|
||||
fieldWordSingular = fieldWordSingular.charAt(0).toUpperCase() + fieldWordSingular.slice(1);
|
||||
value.schema.forEach((field) => {
|
||||
const fieldValue = value.value ? value.value[field.id] : null;
|
||||
const key = `${nodeProperties.name}.${field.id}`;
|
||||
const fieldErrors: string[] = [];
|
||||
if (field.required && !skipRequiredCheck) {
|
||||
if (value.value === null || fieldValue === null || fieldValue === undefined) {
|
||||
const error = `${fieldWordSingular} "${field.id}" is required`;
|
||||
fieldErrors.push(error);
|
||||
}
|
||||
}
|
||||
if (!fieldValue?.toString().startsWith('=') && field.type) {
|
||||
const validationResult = validateFieldType(field.id, fieldValue, field.type, field.options);
|
||||
if (!validationResult.valid && validationResult.errorMessage) {
|
||||
fieldErrors.push(validationResult.errorMessage);
|
||||
}
|
||||
}
|
||||
if (fieldErrors.length > 0) {
|
||||
issues[key] = fieldErrors;
|
||||
}
|
||||
});
|
||||
return issues;
|
||||
};
|
||||
|
||||
/**
|
||||
* Adds an issue if the parameter is not defined
|
||||
*
|
||||
@@ -1196,8 +1404,9 @@ export function getParameterIssues(
|
||||
node: INode,
|
||||
): INodeIssues {
|
||||
const foundIssues: INodeIssues = {};
|
||||
const isDisplayed = displayParameterPath(nodeValues, nodeProperties, path, node);
|
||||
if (nodeProperties.required === true) {
|
||||
if (displayParameterPath(nodeValues, nodeProperties, path, node)) {
|
||||
if (isDisplayed) {
|
||||
const value = getParameterValueByPath(nodeValues, nodeProperties.name, path);
|
||||
|
||||
if (
|
||||
@@ -1218,24 +1427,37 @@ export function getParameterIssues(
|
||||
}
|
||||
}
|
||||
|
||||
if (nodeProperties.type === 'resourceLocator') {
|
||||
if (displayParameterPath(nodeValues, nodeProperties, path, node)) {
|
||||
const value = getParameterValueByPath(nodeValues, nodeProperties.name, path);
|
||||
if (isINodeParameterResourceLocator(value)) {
|
||||
const mode = nodeProperties.modes?.find((option) => option.name === value.mode);
|
||||
if (mode) {
|
||||
const errors = validateResourceLocatorParameter(value, mode);
|
||||
errors.forEach((error) => {
|
||||
if (foundIssues.parameters === undefined) {
|
||||
foundIssues.parameters = {};
|
||||
}
|
||||
if (foundIssues.parameters[nodeProperties.name] === undefined) {
|
||||
foundIssues.parameters[nodeProperties.name] = [];
|
||||
}
|
||||
if (nodeProperties.type === 'resourceLocator' && isDisplayed) {
|
||||
const value = getParameterValueByPath(nodeValues, nodeProperties.name, path);
|
||||
if (isINodeParameterResourceLocator(value)) {
|
||||
const mode = nodeProperties.modes?.find((option) => option.name === value.mode);
|
||||
if (mode) {
|
||||
const errors = validateResourceLocatorParameter(value, mode);
|
||||
errors.forEach((error) => {
|
||||
if (foundIssues.parameters === undefined) {
|
||||
foundIssues.parameters = {};
|
||||
}
|
||||
if (foundIssues.parameters[nodeProperties.name] === undefined) {
|
||||
foundIssues.parameters[nodeProperties.name] = [];
|
||||
}
|
||||
|
||||
foundIssues.parameters[nodeProperties.name].push(error);
|
||||
});
|
||||
foundIssues.parameters[nodeProperties.name].push(error);
|
||||
});
|
||||
}
|
||||
}
|
||||
} else if (nodeProperties.type === 'resourceMapper' && isDisplayed) {
|
||||
const skipRequiredCheck = nodeProperties.typeOptions?.resourceMapper?.mode !== 'add';
|
||||
const value = getParameterValueByPath(nodeValues, nodeProperties.name, path);
|
||||
if (isResourceMapperValue(value)) {
|
||||
const issues = validateResourceMapperParameter(nodeProperties, value, skipRequiredCheck);
|
||||
if (Object.keys(issues).length > 0) {
|
||||
if (foundIssues.parameters === undefined) {
|
||||
foundIssues.parameters = {};
|
||||
}
|
||||
if (foundIssues.parameters[nodeProperties.name] === undefined) {
|
||||
foundIssues.parameters[nodeProperties.name] = [];
|
||||
}
|
||||
foundIssues.parameters = { ...foundIssues.parameters, ...issues };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user