feat(editor): Implement Resource Mapper component (#6207)

*  scaffolding
*  finished scaffolding
*  renamed types
*  updated subtitle
*  renamed functions file, UI updates
*  query parameters fixes, ui updates, refactoring
*  fixes for credentials test, setup for error parsing
*  rlc for schema and table, error handling tweaks
*  delete operation, new options
*  columns loader
*  linter fixes
*  where clauses setup
*  logic for processing where clauses
*  select operation
*  refactoring
*  data mode for insert and update, wip
*  data mapping, insert update, skip on conflict option
*  select columns with spaces fix
*  update operation update, wip
*  finished update operation
*  upsert operation
*  ui fixes
* Copy updates.
* Copy updates.
*  option to convert empty strings to nulls, schema checks
*  UI requested updates
*  ssh setup WIP
*  fixes, ssh WIP
*  ssh fixes, credentials
*  credentials testing update
*  uncaught error fix
*  clean up
*  address in use fix
*  improved error message
*  tests setup
*  unit tests wip
*  config files clean up
*  utils unit tests
*  refactoring
*  setup for testing operations, tests for deleteTable operation
*  executeQuery and insert operations tests
*  select, update, upsert operations tests
*  runQueries tests setup
*  hint to query
* Copy updates.
*  ui fixes
*  clean up
*  error message update
*  ui update
* Minor tweaks to query params decription.
* feat(Google Sheets Node): Implement Resource mapper in Google Sheets node (#5752)
*  Added initial resource mapping support in google sheets node
*  Wired mapping API endpoint with node-specific logic for fetching mapping fields
*  Implementing mapping fields logic for google sheets
*  Updating Google Sheets execute methods to support resource mapper fields
* 🚧 Added initial version of `ResourceLocator` component
* 👌 Added `update` mode to resource mapper modes
* 👌 Addressing PR feedback
* 👌 Removing leftover const reference
* 👕 Fixing lint errors
*  singlton for conections
*  credentials test fix, clean up
* feat(Postgres Node): Add resource mapper to new version of Postgres node (#5814)
*  scaffolding
*  finished scaffolding
*  renamed types
*  updated subtitle
*  renamed functions file, UI updates
*  query parameters fixes, ui updates, refactoring
*  fixes for credentials test, setup for error parsing
*  rlc for schema and table, error handling tweaks
*  delete operation, new options
*  columns loader
*  linter fixes
*  where clauses setup
*  logic for processing where clauses
*  select operation
*  refactoring
*  data mode for insert and update, wip
*  data mapping, insert update, skip on conflict option
*  select columns with spaces fix
*  update operation update, wip
*  finished update operation
*  upsert operation
*  ui fixes
* Copy updates.
* Copy updates.
*  option to convert empty strings to nulls, schema checks
*  UI requested updates
*  ssh setup WIP
*  fixes, ssh WIP
*  ssh fixes, credentials
*  credentials testing update
*  uncaught error fix
*  clean up
*  address in use fix
*  improved error message
*  tests setup
*  unit tests wip
*  config files clean up
*  utils unit tests
*  refactoring
*  setup for testing operations, tests for deleteTable operation
*  executeQuery and insert operations tests
*  select, update, upsert operations tests
*  runQueries tests setup
*  hint to query
* Copy updates.
*  ui fixes
*  clean up
*  error message update
*  ui update
* Minor tweaks to query params decription.
*  Updated Postgres node to use resource mapper component
*  Implemented postgres <-> resource mapper type mapping
*  Updated Postgres node execution to use resource mapper fields in v3
* 🔥 Removing unused import
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>

* feat(core): Resource editor componend P0 (#5970)
*  Added inital value of mapping mode dropdown
*  Finished mapping mode selector
*  Finished implementing mapping mode selector
*  Implemented 'Columns to match on' dropdown
*  Implemented `loadOptionsDependOn` support in resource mapper
*  Implemented initial version of mapping fields
*  Implementing dependant fields watcher in new component setup
*  Generating correct resource mapper field types. Added `supportAutoMap` to node specification and UI. Not showing fields with `display=false`. Pre-selecting matching columns if it's the only one
*  Handling matching columns correctly in UI
*  Saving and loading resourceMapper values in component
*  Implemented proper data saving and loading
*  ResourceMapper component refactor, fixing value save/load
*  Refactoring MatchingColumnSelect component. Updating Sheets node to use single key match and Postgres to use multi key
*  Updated Google Sheets node to work with the new UI
*  Updating Postgres Node to work with new UI
*  Additional loading indicator that shown if there is no mapping mode selector
*  Removing hard-coded values, fixing matching columns ordering, refactoring
*  Updating field names in nodes
*  Fixing minor UI issues
*  Implemented matching fields filter logic
*  Moving loading label outside of fields list
*  Added initial unit tests for resource mapper
*  Finished default rendering test
*  Test refactoring
*  Finished unit tests
* 🔨 Updating the way i18n is used in resource mapper components
* ✔️ Fixing value to match on logic for postgres node
*  Hiding mapping fields when auto-map mode is selected
*  Syncing selected mapping mode between components
*  Fixing dateTime input rendering and adding update check to Postgres node
*  Properly handling database connections. Sending null for empty string values.
* 💄 Updated wording in the error message for non-existing rows
*  Fixing issues with selected matching values
* ✔️ Updating unit tests after matching logic update
*  Updating matching columns when new fields are loaded
*  Defaulting to null for empty parameter values
*  Allowing zero as valid value for number imputs
*  Updated list of types that use datepicker as widger
*  Using text inputs for time types
*  Initial mapping field rework
*  Added new component for mapping fields, moved bit of logic from root component to matching selector, fixing some lint errors
*  Added tooltip for columns that cannot be deleted
*  Saving deleted values in parameter value
*  Implemented control to add/remove mapping fields
*  Syncing field list with add field dropdown when changing dependent values
*  Not showing removed fields in matching columns selector. Updating wording in matching columns selector description
*  Implementing disabled states for add/remove all fields options
*  Saving removed columns separately, updating copy
*  Implemented resource mapper values validation
*  Updated validation logic and error input styling
*  Validating resource mapper fields when new nodes are added
*  Using node field words in validation, refactoring resource mapper component
*  Implemented schema syncing and add/remove all fields
*  Implemented custom parameter actions
*  Implemented loading indicator in parameter options
* 🔨 Removing unnecessary constants and vue props
*  Handling default values properly
*  Fixing validation logic
* 👕 Fixing lint errors
*  Fixing type issues
*  Not showing fields by default if `addAllFields` is set to `false`
*  Implemented field type validation in resource mapper
*  Updated casing in copy, removed all/remove all option from bottom menu
*  Added auto mapping mode notice
*  Added support for more types in validation
*  Added support for enumerated values
*  Fixing imports after merging
*  Not showing removed fields in matching columns selector. Refactoring validation logic.
* 👕 Fixing imports
* ✔️ Updating unit tests
*  Added resource mapper schema tests
*  Removing `match` from resource mapper field definition, fixing matching columns loading
*  Fixed schema merging
*  update operation return data fix
*  review
* 🐛 Added missing import
* 💄 Updating parameter actions icon based on the ui review
* 💄 Updating word capitalisation in tooltips
* 💄 Added empty state to mapping fields list
* 💄 Removing asterisk from fields, updating tooltips for matching fields
*  Preventing matching fields from being removed by 'Remove All option'
*  Not showing hidden fields in the `Add field` dropdown
*  Added support for custom matching columns labels
*  query optimization
*  fix
*  Optimizing Postgres node enumeration logic
*  Added empty state for matching columns
*  Only fully loading fields if there is no schema fetched
*  Hiding mapping fields if there is no matching columns available in the schema
* ✔️ Fixing minor issues
*  Implemented runtime type validation
* 🔨 Refactoring validation logic
*  Implemented required check, added more custom messages
*  Skipping boolean type in required check
* Type check improvements
*  Only reloading fields if dependent values actually change
*  Adding item index to validation error title
*  Updating Postgres fetching logic, using resource mapper mode to determine if a field can be deleted
*  Resetting field values when adding them via the addAll option
*  Using minor version (2.2) for new Postgres node
*  Implemented proper date validation and type casting
* 👕 Consolidating typing
*  Added unit tests for type validations
* 👌 Addressing front-end review comments
*  More refactoring to address review changes
*  Updating leftover props
*  Added fallback for ISO dates with invalid timezones
* Added timestamp to datetime test cases
*  Reseting matching columns if operation changes
*  Not forcing auto-increment fields to be filled in in Postgres node. Handling null values
* 💄 Added a custom message for invalid dates
*  Better handling of JSON values
*  Updating codemirror readonly stauts based on component property, handling objects in json validation
* Deleting leftover console.log
*  Better time validation
*  Fixing build error after merging
* 👕 Fixing lint error
*  Updating node configuration values
*  Handling postgres arrays better
*  Handling SQL array syntax
*  Updating time validation rules to include timezone
*  Sending expressions that resolve to `null` or `undefined` by the resource mapper to delete cell content in Google Sheets
*  Allowing removed fields to be selected for match
*  Updated the query for fetching unique columns and primary keys
*  Optimizing the unique query
*  Setting timezone to all parsed dates
*  Addressing PR review feedback
*  Configuring Sheets node for production, minor vue component update
* New cases added to the TypeValidation test.
*  Tweaking validation rules for arrays/objects and updating test cases
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
This commit is contained in:
Milorad FIlipović
2023-05-31 11:56:09 +02:00
committed by GitHub
parent 5ae1124106
commit 04cfa548af
57 changed files with 3436 additions and 183 deletions

View File

@@ -0,0 +1,500 @@
<script setup lang="ts">
import type { IUpdateInformation, ResourceMapperReqParams } from '@/Interface';
import { resolveParameter } from '@/mixins/workflowHelpers';
import { useNodeTypesStore } from '@/stores/nodeTypes.store';
import type {
INode,
INodeParameters,
INodeProperties,
INodeTypeDescription,
ResourceMapperField,
ResourceMapperValue,
} from 'n8n-workflow';
import { NodeHelpers } from 'n8n-workflow';
import { computed, onMounted, reactive, watch } from 'vue';
import MappingModeSelect from './MappingModeSelect.vue';
import MatchingColumnsSelect from './MatchingColumnsSelect.vue';
import MappingFields from './MappingFields.vue';
import { fieldCannotBeDeleted, isResourceMapperValue, parseResourceMapperFieldName } from '@/utils';
import { i18n as locale } from '@/plugins/i18n';
import { useNDVStore } from '@/stores/ndv.store';
interface Props {
parameter: INodeProperties;
node: INode | null;
path: string;
inputSize: string;
labelSize: string;
dependentParametersValues: string | null;
}
const nodeTypesStore = useNodeTypesStore();
const ndvStore = useNDVStore();
const props = defineProps<Props>();
const emit = defineEmits<{
(event: 'valueChanged', value: IUpdateInformation): void;
}>();
const state = reactive({
paramValue: {
mappingMode: 'defineBelow',
value: {},
matchingColumns: [] as string[],
schema: [] as ResourceMapperField[],
} as ResourceMapperValue,
parameterValues: {} as INodeParameters,
loading: false,
refreshInProgress: false, // Shows inline loader when refreshing fields
loadingError: false,
});
// Reload fields to map when dependent parameters change
watch(
() => props.dependentParametersValues,
async (currentValue, oldValue) => {
if (oldValue !== null && currentValue !== null && oldValue !== currentValue) {
state.paramValue = {
...state.paramValue,
value: null,
schema: [],
};
emitValueChanged();
await initFetching();
setDefaultFieldValues(true);
}
},
);
onMounted(async () => {
if (props.node) {
state.parameterValues = {
...state.parameterValues,
parameters: props.node.parameters,
};
}
const params = state.parameterValues.parameters as INodeParameters;
const parameterName = props.parameter.name;
if (!(parameterName in params)) {
return;
}
let hasSchema = false;
const nodeValues = params[parameterName] as unknown as ResourceMapperValue;
state.paramValue = {
...state.paramValue,
...nodeValues,
};
if (!state.paramValue.schema) {
state.paramValue = {
...state.paramValue,
schema: [],
};
} else {
hasSchema = state.paramValue.schema.length > 0;
}
Object.keys(state.paramValue.value || {}).forEach((key) => {
if (state.paramValue.value && state.paramValue.value[key] === '') {
state.paramValue = {
...state.paramValue,
value: {
...state.paramValue.value,
[key]: null,
},
};
}
});
if (nodeValues.matchingColumns) {
state.paramValue = {
...state.paramValue,
matchingColumns: nodeValues.matchingColumns,
};
}
await initFetching(hasSchema);
// Set default values if this is the first time the parameter is being set
if (!state.paramValue.value) {
setDefaultFieldValues();
}
updateNodeIssues();
});
const resourceMapperMode = computed<string | undefined>(() => {
return props.parameter.typeOptions?.resourceMapper?.mode;
});
const nodeType = computed<INodeTypeDescription | null>(() => {
if (props.node) {
return nodeTypesStore.getNodeType(props.node.type, props.node.typeVersion);
}
return null;
});
const showMappingModeSelect = computed<boolean>(() => {
return props.parameter.typeOptions?.resourceMapper?.supportAutoMap !== false;
});
const showMatchingColumnsSelector = computed<boolean>(() => {
return (
!state.loading &&
props.parameter.typeOptions?.resourceMapper?.mode !== 'add' &&
state.paramValue.schema.length > 0
);
});
const showMappingFields = computed<boolean>(() => {
return (
state.paramValue.mappingMode === 'defineBelow' &&
!state.loading &&
!state.loadingError &&
state.paramValue.schema.length > 0 &&
hasAvailableMatchingColumns.value
);
});
const matchingColumns = computed<string[]>(() => {
if (!showMatchingColumnsSelector.value) {
return [];
}
if (state.paramValue.matchingColumns) {
return state.paramValue.matchingColumns;
}
return defaultSelectedMatchingColumns.value;
});
const hasAvailableMatchingColumns = computed<boolean>(() => {
if (resourceMapperMode.value !== 'add') {
return (
state.paramValue.schema.filter(
(field) =>
field.canBeUsedToMatch !== false && field.display !== false && field.removed !== true,
).length > 0
);
}
return true;
});
const defaultSelectedMatchingColumns = computed<string[]>(() => {
return state.paramValue.schema.length === 1
? [state.paramValue.schema[0].id]
: state.paramValue.schema.reduce((acc, field) => {
if (field.defaultMatch && field.canBeUsedToMatch === true) {
acc.push(field.id);
}
return acc;
}, [] as string[]);
});
const pluralFieldWord = computed<string>(() => {
return (
props.parameter.typeOptions?.resourceMapper?.fieldWords?.plural ||
locale.baseText('generic.fields')
);
});
async function initFetching(inlineLading = false): Promise<void> {
state.loadingError = false;
if (inlineLading) {
state.refreshInProgress = true;
} else {
state.loading = true;
}
try {
await loadFieldsToMap();
if (!state.paramValue.matchingColumns || state.paramValue.matchingColumns.length === 0) {
onMatchingColumnsChanged(defaultSelectedMatchingColumns.value);
}
} catch (error) {
state.loadingError = true;
} finally {
state.loading = false;
state.refreshInProgress = false;
}
}
async function loadFieldsToMap(): Promise<void> {
if (!props.node) {
return;
}
const requestParams: ResourceMapperReqParams = {
nodeTypeAndVersion: {
name: props.node?.type,
version: props.node.typeVersion,
},
currentNodeParameters: resolveParameter(props.node.parameters) as INodeParameters,
path: props.path,
methodName: props.parameter.typeOptions?.resourceMapper?.resourceMapperMethod,
credentials: props.node.credentials,
};
const fetchedFields = await nodeTypesStore.getResourceMapperFields(requestParams);
if (fetchedFields !== null) {
const newSchema = fetchedFields.fields.map((field) => {
const existingField = state.paramValue.schema.find((f) => f.id === field.id);
if (existingField) {
field.removed = existingField.removed;
} else if (state.paramValue.value !== null && !(field.id in state.paramValue.value)) {
// New fields are shown by default
field.removed = false;
}
return field;
});
state.paramValue = {
...state.paramValue,
schema: newSchema,
};
emitValueChanged();
}
}
async function onModeChanged(mode: string): Promise<void> {
state.paramValue.mappingMode = mode;
if (mode === 'defineBelow') {
await initFetching();
} else {
state.loadingError = false;
}
emitValueChanged();
}
function setDefaultFieldValues(forceMatchingFieldsUpdate = false): void {
state.paramValue.value = {};
const hideAllFields = props.parameter.typeOptions?.resourceMapper?.addAllFields === false;
state.paramValue.schema.forEach((field) => {
if (state.paramValue.value) {
// Hide all non-required fields if it's configured in node definition
if (hideAllFields) {
field.removed = !field.required;
}
if (field.type === 'boolean') {
state.paramValue.value = {
...state.paramValue.value,
[field.id]: false,
};
} else {
state.paramValue.value = {
...state.paramValue.value,
[field.id]: null,
};
}
}
});
emitValueChanged();
if (!state.paramValue.matchingColumns || forceMatchingFieldsUpdate) {
state.paramValue.matchingColumns = defaultSelectedMatchingColumns.value;
emitValueChanged();
}
}
function updateNodeIssues(): void {
if (props.node) {
const parameterIssues = NodeHelpers.getNodeParametersIssues(
nodeType.value?.properties || [],
props.node,
);
if (parameterIssues) {
ndvStore.updateNodeParameterIssues(parameterIssues);
}
}
}
function onMatchingColumnsChanged(matchingColumns: string[]): void {
state.paramValue = {
...state.paramValue,
matchingColumns,
};
// Set all matching fields to be visible
state.paramValue.schema.forEach((field) => {
if (state.paramValue.matchingColumns?.includes(field.id)) {
field.removed = false;
state.paramValue.schema.splice(state.paramValue.schema.indexOf(field), 1, field);
}
});
if (!state.loading) {
emitValueChanged();
}
}
function fieldValueChanged(updateInfo: IUpdateInformation): void {
let newValue = null;
if (
updateInfo.value !== undefined &&
updateInfo.value !== '' &&
updateInfo.value !== null &&
isResourceMapperValue(updateInfo.value)
) {
newValue = updateInfo.value;
}
const fieldName = parseResourceMapperFieldName(updateInfo.name);
if (fieldName && state.paramValue.value) {
state.paramValue.value = {
...state.paramValue.value,
[fieldName]: newValue,
};
emitValueChanged();
}
}
function removeField(name: string): void {
if (name === 'removeAllFields') {
return removeAllFields();
}
const fieldName = parseResourceMapperFieldName(name);
if (fieldName) {
if (state.paramValue.value) {
delete state.paramValue.value[fieldName];
const field = state.paramValue.schema.find((f) => f.id === fieldName);
if (field) {
field.removed = true;
state.paramValue.schema.splice(state.paramValue.schema.indexOf(field), 1, field);
}
emitValueChanged();
}
}
}
function addField(name: string): void {
if (name === 'addAllFields') {
return addAllFields();
}
if (name === 'removeAllFields') {
return removeAllFields();
}
state.paramValue.value = {
...state.paramValue.value,
[name]: null,
};
const field = state.paramValue.schema.find((f) => f.id === name);
if (field) {
field.removed = false;
state.paramValue.schema.splice(state.paramValue.schema.indexOf(field), 1, field);
}
emitValueChanged();
}
function addAllFields(): void {
const newValues: { [name: string]: null } = {};
state.paramValue.schema.forEach((field) => {
if (field.display && field.removed) {
newValues[field.id] = null;
field.removed = false;
state.paramValue.schema.splice(state.paramValue.schema.indexOf(field), 1, field);
}
});
state.paramValue.value = {
...state.paramValue.value,
...newValues,
};
emitValueChanged();
}
function removeAllFields(): void {
state.paramValue.schema.forEach((field) => {
if (
!fieldCannotBeDeleted(
field,
showMatchingColumnsSelector.value,
resourceMapperMode.value,
matchingColumns.value,
)
) {
field.removed = true;
state.paramValue.schema.splice(state.paramValue.schema.indexOf(field), 1, field);
}
});
emitValueChanged();
}
function emitValueChanged(): void {
pruneParamValues();
emit('valueChanged', {
name: `${props.path}`,
value: state.paramValue,
node: props.node?.name,
});
updateNodeIssues();
}
function pruneParamValues(): void {
if (!state.paramValue.value) {
return;
}
const valueKeys = Object.keys(state.paramValue.value);
valueKeys.forEach((key) => {
if (state.paramValue.value && state.paramValue.value[key] === null) {
delete state.paramValue.value[key];
}
});
}
defineExpose({
state,
});
</script>
<template>
<div class="mt-4xs" data-test-id="resource-mapper-container">
<mapping-mode-select
v-if="showMappingModeSelect"
:inputSize="inputSize"
:labelSize="labelSize"
:initialValue="state.paramValue.mappingMode || 'defineBelow'"
:typeOptions="props.parameter.typeOptions"
:serviceName="nodeType?.displayName || locale.baseText('generic.service')"
:loading="state.loading"
:loadingError="state.loadingError"
:fieldsToMap="state.paramValue.schema"
@modeChanged="onModeChanged"
@retryFetch="initFetching"
/>
<matching-columns-select
v-if="showMatchingColumnsSelector"
:label-size="labelSize"
:fieldsToMap="state.paramValue.schema"
:typeOptions="props.parameter.typeOptions"
:inputSize="inputSize"
:loading="state.loading"
:initialValue="matchingColumns"
:serviceName="nodeType?.displayName || locale.baseText('generic.service')"
@matchingColumnsChanged="onMatchingColumnsChanged"
/>
<n8n-text v-if="!showMappingModeSelect && state.loading" size="small">
<n8n-icon icon="sync-alt" size="xsmall" :spin="true" />
{{
locale.baseText('resourceMapper.fetchingFields.message', {
interpolate: {
fieldWord: pluralFieldWord,
},
})
}}
</n8n-text>
<mapping-fields
v-if="showMappingFields"
:parameter="props.parameter"
:path="props.path"
:nodeValues="state.parameterValues"
:fieldsToMap="state.paramValue.schema"
:paramValue="state.paramValue"
:labelSize="labelSize"
:showMatchingColumnsSelector="showMatchingColumnsSelector"
:showMappingModeSelect="showMappingModeSelect"
:loading="state.loading"
:refreshInProgress="state.refreshInProgress"
@fieldValueChanged="fieldValueChanged"
@removeField="removeField"
@addField="addField"
@refreshFieldList="initFetching(true)"
/>
<n8n-notice
v-if="state.paramValue.mappingMode === 'autoMapInputData' && hasAvailableMatchingColumns"
>
{{
locale.baseText('resourceMapper.autoMappingNotice', {
interpolate: {
fieldWord: pluralFieldWord,
serviceName: nodeType?.displayName || locale.baseText('generic.service'),
},
})
}}
</n8n-notice>
</div>
</template>