npm v7 does not install linked packages dependencies

Current Behavior:

npm v7 does not install linked local packages dependencies.

Expected Behavior:

In npm v6 the dependencies of a local linked package are installed. This means that if app has my-local-pkg as a dependency, running npm install generates app/node_modules folder and also my-local-pkg/node_modules (with its dependencies).
In npm v7, however, only app/node_modules is generated.

Steps To Reproduce:

I created the following repository to illustrate it. The README.md has the instructions on how to reproduce it.
Simplified, the steps are:

Add the following folder structure: ./app and ./my-local-pkg, each with their own package.json
Include this in the ./app/package.json:

{
  "name": "app",
  "dependencies": {
    "my-local-pkg": "file:../my-local-pkg",
    "prettier": "^2.2.1"
  }
}

Include this in the ./my-local-pkg/package.json:

{
  "name": "my-local-pkg",
  "dependencies": {
    "typescript": "^4.1.3"
  }
}

Go to ./app and run npm install.
npm version 6 produces:

node_modules
	.bin
	my-local-pkg
		node_modules
			typescript
	prettier

while npm version 7 produces:

node_modules
	.bin
	my-local-pkg
	prettier

Compare the two outputs, npm v7 does not install my-local-pkg dependencies, there is no ./my-local-pkg/node_modules folder generated.

Environment:

npm v7.1.2

Same behavior with npm@7.2.0

I'd also add that for me the expected behavior for local installs npm install ../some-package is to result in the same package-lock.json as installs from the registry with the only difference being the "resolved" field of local install pointing to ../some-package instead of to the registry (and other related metadata like "link": true).

This means that the dependencies of the locally installed package should end up in the top level of node_modules unless there is a version conflict. I believe that v6 puts all of the local install package's dependencies in its node_modules and not in the top level node_modules (hopefully v7 will change this).

Having the install behavior be consistent will make it easier to develop npm packages by being able to install the package locally in development to test it out and know that it will act the same as installing it from the registry when it's published.

For example, with Rollpkg:

git clone git@github.com:rafgraph/rollpkg.git
mkdir test && cd test
npm init --yes
# compare package-lock.json after running the two commands
npm i --save-dev ../rollpkg
# vs
npm i --save-dev rollpkg

Note that Rollpkg, a zero config build tool to create packages with Rollup and TypeScript, has Jest and ESLint as dependencies (and provides default configs for these), but let's the user call jest and eslint instead of wrapping those commands, which relies on those bins being in the top level of node_modules. The advantage of not wrapping the jest and eslint commands is that if the user needs to use a specific version of Jest or ESlint all they have to do is install that version and it will override the version provided by Rollpkg.

@rafgraph

@rafgraph for that, jest and eslint should be peer deps - and linking with peer deps means you have to link the peer deps also.

Not exactly peer deps, it's a default with optional override. I know this isn't a standard use case, but regardless a consistent package-lock.json would be nice to have.

@garciaalvaro

@garciaalvaro if the path is set within the root directory we will resolve the dependencies in v7 but, if as you've shown, the path is outside the fs root then we treat it like a linked dep and won't resolve those deps. We understand that this is a change from v6.

@darcyclarke

@darcyclarke Thank you for the information.
If I understood correctly, the behaviour is intended. Is there some way to keep/emulate the previous behaviour and install those dependencies too?

@garciaalvaro

@garciaalvaro I'm going to bring this up at our next Open RFC call & see what the consensus is in terms of supporting the old behaivour; I forgot the exact reasoning behind the change but I can imagine that it would have reduced some complexity & potentially cleaned up a number of issues/edgecases. I'll circle back here after the call on Wednesday.

@darcyclarke

All that said, we will consider making --install-links the default behavior in the next major version (v9) which you should start to see release candidates for in the coming weeks/months.

@darcyclarke --install-links sounds cool, if it wasn't for this bug 🙈: #4863

@nlf

@nlf To me it seems like this bug is not fixed. I've demonstrated my failed attempt to take advantage of this fix. Can you please tell me what I did wrong or why I cannot install? It's a nice self-contained repro, so hopefully someone can show me how to make this --install-links work correctly.

@nlf

@nlf The most upsetting feature here is actually that npm link now REMOVES linked pkg deps that are already installed in the parent's node_modules, forcing me to explicitly add those peerDeps to the parent's package.json if i need to use them in my parent project.
Can this be somehow fixed/reverted/added as a feature?

@jasminen

Facing the same issues as @jasminen

I'm still finding npm link to be an absolute pain to work with. Hours spent chasing non-issues.

@jasminen

Still facing issues with --install-link it seems to make my current locally linked package work but it then messes up with my other locally referenced ( not linked ) components/packages which are present in my parent's project directory. And also npm link is really a pain to work with as mentioned by @jasminen we need to install all local's peerDeps in parent.

Maybe a solution to it could be somehow making my linked package refer to its own node_modules for its deps rather than searching for those in parent's.

npm is still broken as far as I'm concerned.

For what it's worth, I had to switch to using yarn to achieve what I want. The package.json remains the same, but I use yarn install in place of npm install because it handles the linked dependencies of local dependencies according to my expectations.

@wyckster

@wyckster interesting I failed with yarn also what version are you using of yarn?

@alexander-schranz

@alexander-schranz

@wyckster interesting I failed with yarn also what version are you using of yarn?

Yarn will correctly install the example in my repro. I just confirmed this to be true with the node:19 docker image: node v19.9.0 and yarn 1.22.19.

Whereas npm v9.6.3 (which comes with that same docker image) still fails to install local dependencies of local dependencies correctly.

@Jarred-Sumner

I still run into issues with npm >= 7, pnpm and yarn.

But today I tried bun from @oven-sh where example @Jarred-Sumner works on. They released 1.0 and seems correclty handle also dependencies of local dependencies and install them. In my case i was able to build our complex setup with it, which was currently only possible with npm@6. While we still use some kind of symlinks which simulates child paths so not sure if it will work in all cases. In my case bun run preinstall && bun install && bun run build did work.

Fix npm install command (#4)

yarn 1.22.21 makes a full copy (not a symlink) of the local package into the node_modules folder and installs all deps properly into the projects node_modules folder.

This was the last straw that made me switch to Deno two years ago, still not fixed.
Well done.

A locally linked package should NOT be treated any different than a package installed from a remote source. That means its deps need to be installed into the projects node_modules folder and nowhere else.

I thought this issue was causing problems in my scenario, but then I realized the npm behavior makes sense, given the nodejs module resolution behavior.

I think my scenario is pretty common so I'll elaborate in case it helps others.

Scenario

my_repo
├── a
├── b
├── c

a depends on b
b depends on c
a and b are not published externally, they're only used within the context of my_repo.
c could be published externally (or it could also be a remote package, I just made it local here because it was easy for testing, this is irrelevant though).
Key requirements: fast refresh/hot reload/etc features (like nextjs fast refresh), must work for both packages (even if running within a local docker container, which requires some volume mounts/etc).

Relevant Nodejs module resolution behavior.

The key is the phrase "real path" in the following: https://nodejs.org/api/esm.html#resolution-algorithm-specification section ESM_RESOLVE.7.4 which says:

"Set resolved to the real path of resolved, maintaining the same URL querystring and fragment components".

So this means that symlinks are resolved before looking for node_modules directories.

What this means is the following works:

my_repo
├── a
│   └── node_modules
│       └── b -> ../../b
├── b
│   └── node_modules
│       └── c -> ../../c/
└── c

but the following does NOT work:

my_repo
├── a
│   ├── a.mjs
│   └── node_modules
│       ├── b -> ../../b
│       └── c -> ../../c/
├── b
│   └── b.mjs
└── c
    └── c.mjs

Note that this has nothing to do with npm, just nodejs.

That means if you want to symlink from a to b to support "fast refresh", you're also stuck making sure that b has its node_modules setup correctly (i.e. npm install has been run within b).

Thoughts

One could argue that cd a; npm install should setup a/node_modules and a/package-lock.json to look the same as if b was a remote package, I thought this at first too. However, if you say --install-links=false, you're specifically requesting a symlink, which means npm knows nodejs will use b/node_modules when doing lookups for b dependencies.

Setting up a/package-lock.json to include b dependencies would then be odd and problematic. Imagine a2 is just like a, it also depends on b.

You then have 3 different package-lock.json files that all cover b dependencies, and they could potentially conflict, meanwhile you only have 1 node_modules folder containing the dependencies that will actually get used (the one inside of b).
You're working on a, you tweak b, all is good, "fast refresh" is working. Then 4 days later you go to work on a2 and you forgot you tweaked b and changed its dependencies. The symlink a2/node_modules/b -> ../../b is still there, so the code works fine, utilizing the forgotten tweak and dep change, but your a2/package-lock.json is now incorrect. Because a2 is working via the symlink, you may never remember to update the package-lock.json, you might even check it into your repo.

Given this, I can see why npm behaves like it does when requesting symlinking.

encountered this old chestnut again today. the following workaround seems to be working for now:

so my top-level project has installed my own dependency called @benev/toolbox which has its own dependencies (like lit and @dimforge/rapier3d)
first, i install my dependency's latest published version with npm i @benev/toolbox@latest -- thus, now npm properly puts the subdependencies in my node_modules
❌ but if i run npm link ../toolbox to use my local copy which i'm actively working on, npm is deleting the subdependencies -- so i do NOT use npm link
✅ instead, i manually create a symlink:
- rm -rf node_modules/@benev/toolbox && ln -s "$(realpath ../toolbox)" node_modules/@benev/toolbox
- understand, that this works because i still have the subdependencies from the ordinary npm i
- if you put this in a package.json script, don't forget to escape the double quotes like \"
🤔 this works well, up until the point you need to test changes to the subdependencies of the symlinked package, in which case, you'll need to dive down a rabbit hole of doing your own funky linking
🧙‍♂️ you may consider at this point doing all your manual funky local symlinking in an npm script so it's reproducible, ie, i do npm i && npm run devlinks -- maybe even the "prepare" script if you're zany

@ljharb

@ljharb Hi, this is still broken on latest NodeJs version 22.9 / npm 10.8.3.

Navigation Menu

Repository navigation

npm v7 does not install linked packages dependencies #2339

Current Behavior:

Expected Behavior:

Steps To Reproduce:

Environment:

83 remaining items

Scenario

Relevant Nodejs module resolution behavior.

Thoughts

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npm v7 does not install linked packages dependencies #2339

Description

Current Behavior:

Expected Behavior:

Steps To Reproduce:

Environment:

Activity

raphinesse commented on Dec 16, 2020

rafgraph commented on Dec 20, 2020

ljharb commented on Dec 20, 2020

rafgraph commented on Dec 20, 2020

darcyclarke commented on Jan 15, 2021

garciaalvaro commented on Jan 22, 2021

darcyclarke commented on Jan 26, 2021

83 remaining items

fabb commented on Sep 10, 2022

wyckster commented on Sep 23, 2022

jasminen commented on Dec 27, 2022

drik98 commented on Dec 27, 2022

mryellow commented on Apr 19, 2023

shamik-fw commented on Jun 13, 2023

wyckster commented on Jul 6, 2023

alexander-schranz commented on Jul 6, 2023

wyckster commented on Jul 12, 2023

alexander-schranz commented on Sep 11, 2023

caramboleyo commented on Dec 23, 2023

brianbraunstein commented on Jan 11, 2024

Scenario

Relevant Nodejs module resolution behavior.

Thoughts

chase-moskal commented on Jul 4, 2024

alesmenzel commented on Sep 20, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions