feat(cli): [nan-1106] import relative files in syncs/actions #2273

khaliqgant · 2024-06-05T19:51:53Z

Describe your changes

Use tsup to import relative files in syncs. The error reporting with tsup isn't as nice as ts-node so the tsup transpiles only and ts-node is still used to type check and report nice errors. The result is actually faster than previous. tsup doesn't transpile if there are any compilation errors.

Note that an additional vite.cli.config file was added which doesn't use threads because it was interfering with tsup

Edge Cases Handled

If the main file imports another file which in turn imports another file then all imported files should be typed checked and checked to make sure nango calls are used correctly
If a library is imported like crypto we shouldn't try and type check that file
If a tsconfig.json file exists in their repo it isn't used and the one we want to use in the nango cli directory is instead used
Limit possible import files to be in the nango-integrations directory
Update to download the entire nango-integrations directory instead of just the particular file (v2)
Verify that changing the transpiled code doesn't produce different results the ts-node transpilation

Issue ticket number and link

NAN-1106

Checklist before requesting a review (skip if just adding/editing APIs & templates)

I added tests, otherwise the reason is:
I added observability, otherwise the reason is:
I added analytics, otherwise the reason is:

linear · 2024-06-05T19:51:55Z

NAN-1106 Import relative files with cc or rollup

bodinsamuel

I have this error when running node ../nango/packages/cli/dist/index.js compile

Error compiling "/Users/samuelbodin/code/nango-integrations/unauthenticated/syncs/trackDeletes.ts":
SyntaxError: Unexpected token '', "��z��" is not valid JSON
    at JSON.parse (<anonymous>)
    at new AnyMap (/Users/samuelbodin/code/nango/node_modules/@jridgewell/src/any-map.ts:20:37)
    at mapSourcePosition (/Users/samuelbodin/code/nango/node_modules/@cspotcode/source-map-support/source-map-support.js:387:14)
    at wrapCallSite (/Users/samuelbodin/code/nango/node_modules/@cspotcode/source-map-support/source-map-support.js:592:20)
    at Function.prepareStackTrace (/Users/samuelbodin/code/nango/node_modules/@cspotcode/source-map-support/source-map-support.js:671:41)
    at prepareStackTraceCallback (node:internal/errors:145:29)
    at getStackString (node:internal/util/inspect:1240:16)
    at formatError (node:internal/util/inspect:1369:15)
    at formatRaw (node:internal/util/inspect:985:14)
    at formatValue (node:internal/util/inspect:840:10)

packages/cli/lib/tests/helpers.ts

vite.cli.config.ts

khaliqgant · 2024-06-06T18:27:48Z

I have this error when running node ../nango/packages/cli/dist/index.js compile

Error compiling "/Users/samuelbodin/code/nango-integrations/unauthenticated/syncs/trackDeletes.ts":
SyntaxError: Unexpected token '', "��z��" is not valid JSON
    at JSON.parse (<anonymous>)
    at new AnyMap (/Users/samuelbodin/code/nango/node_modules/@jridgewell/src/any-map.ts:20:37)
    at mapSourcePosition (/Users/samuelbodin/code/nango/node_modules/@cspotcode/source-map-support/source-map-support.js:387:14)
    at wrapCallSite (/Users/samuelbodin/code/nango/node_modules/@cspotcode/source-map-support/source-map-support.js:592:20)
    at Function.prepareStackTrace (/Users/samuelbodin/code/nango/node_modules/@cspotcode/source-map-support/source-map-support.js:671:41)
    at prepareStackTraceCallback (node:internal/errors:145:29)
    at getStackString (node:internal/util/inspect:1240:16)
    at formatError (node:internal/util/inspect:1369:15)
    at formatRaw (node:internal/util/inspect:985:14)
    at formatValue (node:internal/util/inspect:840:10)

Can you try this again? Should be resolved.

TBonnin

I think there is an inverted condition with the nango-integrations directory detection but other than that it seems to work as expected. I haven't tested very involved or big import though

Question about your PR notes:

Update to download the entire nango-integrations directory instead of just the particular file (v2)
why will we need to do that?

TBonnin · 2024-06-07T01:16:55Z

packages/cli/lib/services/parser.service.ts

        const code = fs.readFileSync(filePath, 'utf-8');
+        const ast = parser.parse(code, { sourceType: 'module', plugins: ['typescript'] });
+        const importedFiles: string[] = [];
+        const traverseFn = (traverse as any).default || traverse;


I know it is unrelated to this PR but I've always wondered why does this .default thing is doing and why we are not using traverse directly

I think during tests traverse wasn't available for some reason

TBonnin · 2024-06-07T01:21:29Z

packages/cli/lib/services/compile.service.ts

+
+        const cwd = process.cwd().split('/').pop();
+        const fullPath = path.resolve(importedFilePathWithExtension);
+        if (cwd === 'nango-integrations' && fullPath.includes(process.cwd())) {


shouldn't it be the negative? ie: if fullPath doesn't includes nango-integration then show error?

Thanks, this was the last thing I added last night without testing thoroughly. Updated and added a test case to support it + improved the error message

khaliqgant · 2024-06-07T08:26:41Z

I think there is an inverted condition with the nango-integrations directory detection but other than that it seems to work as expected. I haven't tested very involved or big import though

Question about your PR notes:

Update to download the entire nango-integrations directory instead of just the particular file (v2)
why will we need to do that?

So a user could download all the code including helper files.

bodinsamuel

Works great 👏🏻
I have a question about unintended side-effects, it's now possible to load any npm package inside the build, I don't think it's okay even if it's handy?

…6-import-relative-files-with-cc-or-rollup

khaliqgant · 2024-06-07T10:46:07Z

Works great 👏🏻 I have a question about unintended side-effects, it's now possible to load any npm package inside the build, I don't think it's okay even if it's handy?

Thanks for catching this! Should be resolved by 1980321

bodinsamuel

Nice, it now catches the most obvious incorrect import, and execute appropriately.
It's still possible to import node_modules by using relative path (i.e: import {attemptifyAsync} from '../../node_modules/stubborn-fs/src/attemptify';). It's a bit harder to catch, especially that stuff could live outside node_modules.

Side note:

You can't import .js file, I'm sure it's not really an issue but it displays the wrong error message in that case. (and it reduces the possibility to import packages)
Because they can import stuff, they'll need a package.json, so we need to check wether the zod/node version is matching what we are using otherwise the types will be wrong and/or expected execution will be slightly different

packages/cli/lib/services/compile.service.ts

khaliqgant · 2024-06-07T13:43:00Z

Nice, it now catches the most obvious incorrect import, and execute appropriately. It's still possible to import node_modules by using relative path (i.e: import {attemptifyAsync} from '../../node_modules/stubborn-fs/src/attemptify';). It's a bit harder to catch, especially that stuff could live outside node_modules.

Yeah, hard to catch these edge cases.

Side note:

You can't import .js file, I'm sure it's not really an issue but it displays the wrong error message in that case. (and it reduces the possibility to import packages)

Something to revisit in future iterations perhaps.

Because they can import stuff, they'll need a package.json, so we need to check wether the zod/node version is matching what we are using otherwise the types will be wrong and/or expected execution will be slightly different

Yeah, we should publish in our documentation what versions of things we're using to make sure it aligns

bburns · 2024-06-12T12:18:42Z

Nice, it now catches the most obvious incorrect import, and execute appropriately. It's still possible to import node_modules by using relative path (i.e: import {attemptifyAsync} from '../../node_modules/stubborn-fs/src/attemptify';). It's a bit harder to catch, especially that stuff could live outside node_modules.

Yeah, hard to catch these edge cases.

Some example code for avoiding path traversal attacks -

const root = '/var/www/'
exports.validatePath = user_input => {
  if (user_input.indexOf('\0') !== -1) {
    return 'Access denied'
  }
  if (!/^[a-z0-9]+$/.test(user_input)) {
    return 'Access denied'
  }
  const path = require('path')
  const safe_input = path.normalize(user_input).replace(/^(\.\.(\/|\\|$))+/, '')
  const path_string = path.join(root, safe_input)
  if (path_string.indexOf(root) !== 0) {
    return 'Access denied'
  }
  return path_string
}

From https://www.stackhawk.com/blog/node-js-path-traversal-guide-examples-and-prevention/

khaliqgant added 5 commits June 5, 2024 17:58

[nan-1106] initial working version

0f32ad5

[nan-1106] resolve recursive imports and run checks

08c7e26

[nan-1106] fix doubling up logic

2ea73d8

[nan-1106] reference tsconfig properly and fix file paths

6158704

[nan-1106] watch and debug

a4c328a

khaliqgant changed the title ~~feat(cli: [nan-1106] import relative files in syncs/actions~~ feat(cli): [nan-1106] import relative files in syncs/actions Jun 5, 2024

khaliqgant added 7 commits June 5, 2024 23:33

start tests

dfebc24

[nan-1106] fix tests

c2e9d90

[nan-1106] ignore fixtures

3dad331

[nan-1106] more tests

4695d24

[nan-1106] separate test

582cb1f

[nan-1106] fix typo

15d2556

[nan-1106] another test

1fdafc4

khaliqgant requested review from TBonnin and bodinsamuel and removed request for TBonnin June 6, 2024 09:39

bodinsamuel reviewed Jun 6, 2024

View reviewed changes

packages/cli/lib/tests/helpers.ts Outdated Show resolved Hide resolved

vite.cli.config.ts Outdated Show resolved Hide resolved

khaliqgant added 3 commits June 6, 2024 21:14

[nan-1106] target esnext

460bec7

[nan-1106] pr review

1853af1

[nan-1106] bring in latest

df9fb24

khaliqgant added 2 commits June 6, 2024 21:44

[nan-1106] all imports must live within the nango-integrations directory

9bff86d

[nan-1106] check to make sure in nango-integrations

bc524ab

TBonnin reviewed Jun 7, 2024

View reviewed changes

khaliqgant added 3 commits June 7, 2024 10:45

[nan-1106] better check and cross platform

fe1db2b

[nan-1106] add relative import test

515c988

[nan-1106] better error message

316d858

khaliqgant requested review from TBonnin and bodinsamuel June 7, 2024 08:27

bodinsamuel approved these changes Jun 7, 2024

View reviewed changes

khaliqgant added 2 commits June 7, 2024 13:09

Merge branch 'master' of github.com:NangoHQ/nango into khaliq/nan-110…

441e6de

…6-import-relative-files-with-cc-or-rollup

[nan-1106] don't bundle node_modules and add allowed imports

1980321

bodinsamuel approved these changes Jun 7, 2024

View reviewed changes

packages/cli/lib/services/compile.service.ts Outdated Show resolved Hide resolved

[nan-1106] pr review feedback

fd6400e

TBonnin approved these changes Jun 7, 2024

View reviewed changes

khaliqgant merged commit 5a2d16c into master Jun 7, 2024
21 checks passed

khaliqgant deleted the khaliq/nan-1106-import-relative-files-with-cc-or-rollup branch June 7, 2024 13:46

bburns mentioned this pull request Jun 12, 2024

Error compiling a sync .ts file - SyntaxError: Unexpected token '', "��z��" is not valid JSON #2312

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): [nan-1106] import relative files in syncs/actions #2273

feat(cli): [nan-1106] import relative files in syncs/actions #2273

khaliqgant commented Jun 5, 2024 •

edited

Loading

linear bot commented Jun 5, 2024

bodinsamuel left a comment

khaliqgant commented Jun 6, 2024

TBonnin left a comment

TBonnin Jun 7, 2024

khaliqgant Jun 7, 2024

TBonnin Jun 7, 2024

khaliqgant Jun 7, 2024

khaliqgant commented Jun 7, 2024

bodinsamuel left a comment

khaliqgant commented Jun 7, 2024

bodinsamuel left a comment

khaliqgant commented Jun 7, 2024

bburns commented Jun 12, 2024

feat(cli): [nan-1106] import relative files in syncs/actions #2273

feat(cli): [nan-1106] import relative files in syncs/actions #2273

Conversation

khaliqgant commented Jun 5, 2024 • edited Loading

Describe your changes

Edge Cases Handled

Issue ticket number and link

Checklist before requesting a review (skip if just adding/editing APIs & templates)

linear bot commented Jun 5, 2024

bodinsamuel left a comment

Choose a reason for hiding this comment

khaliqgant commented Jun 6, 2024

TBonnin left a comment

Choose a reason for hiding this comment

TBonnin Jun 7, 2024

Choose a reason for hiding this comment

khaliqgant Jun 7, 2024

Choose a reason for hiding this comment

TBonnin Jun 7, 2024

Choose a reason for hiding this comment

khaliqgant Jun 7, 2024

Choose a reason for hiding this comment

khaliqgant commented Jun 7, 2024

bodinsamuel left a comment

Choose a reason for hiding this comment

khaliqgant commented Jun 7, 2024

bodinsamuel left a comment

Choose a reason for hiding this comment

khaliqgant commented Jun 7, 2024

bburns commented Jun 12, 2024

khaliqgant commented Jun 5, 2024 •

edited

Loading