javascript: singleton packages

Some packages are intended to only exist within a dependency graph once. often things expecting to act like singletons.

I think it would be good to investigate 2 possible fields for package.json tooling to use.

`"singleton": boolean`

If present, the package must have a name unique within the entire module graph, else it will fail to install. This would prevent things like having 2 different versions of react-intl be able to install in the same graph.

`"singletonDependencies": string[] | "all"`

If present this field ensures that dependencies with the same name are not present multiple times within a graph. This could be useful for preventing things like having multiple configuration packages which might not wish to use "singleton" in their own package.json.

If the value is an array of strings, it does search of the graph to ensure that the name is a singleton. This applies to the entire graph, not only the submodules of the current package.

If the value is "all" then all submodules must be singletons and their submodules recursively must also be singletons.

determination of "package name"

Module names are not found within package.json files. They are determined by the presence of a file with a filename that could be loaded as a the starting segments of a bare specifier like request for import("request") or lodash for import("lodash/chunk").

scoped packages

Installation of packages by tools are given the full name by which the package is to be imported:

npm i @npm/invalid

If the package contained "singleton": true. This would result in singleton checks against all the dependency graph for @npm/invalid before placing the installation such that it could be used via import("@npm/invalid"). It would not check the value of "name" within the package.json that is installed.

bundledDependencies

Bundled dependencies are also checked to be singletons, no exceptions are to be made when searching for collisions. If you wish to use a duplicate package in a bundled fashion: you can rename it, add a scope, or put it into a vendor folder.

root package

When writing applications, they tend to be at the root/entry point of your dependency graph. The directory name of this entry point and package.json "name" field are not used when searching for singletons. This allows modules to link to themselves in such a way that they can be loaded as bare specifiers while still being declared singletons.

realpathing

Due to the nature of various runtime flags like --preserve-symlinks, realpathing should not be performed when searching for singletons.

@bmeck

Thanks for starting this issue @bmeck! We've done some thinking on this for Polymer as well, probably best summarized by @rictic here: Polymer/polymer#4823 (comment)
tl;dr: The current "flat" behavior of Yarn/Bower is really the conflation of two package qualities: "is web loadable" & "requires uniqueness".

Fleshing out how to signal the more general "unique"/"singleton" here could be helpful for making web projects (and hopefully web components!) on npm easier to work with.

@zkat

Instead of defining (& maintaining) a brand new "singletonDependencies" dependencies group, could we leverage the "resolutions" mechanism already used by both Bower & Yarn? This could work across your main dependencies, dev, etc.

Documented here: https://yarnpkg.com/lang/en/docs/cli/install/#toc-yarn-install-flat

Update: After re-reading this comment from @zkat it sounds like support for "resolutions" is already planned for npm as well: Polymer/polymer#4823 (comment)

@FredKSchott

@FredKSchott "resolutions" does not search the entire tree for nested packages to my knowledge in Yarn.

/app
 /node_modules/a
   /node_modules/c # version 2.0.0
 /node_modules/b
   /node_modules/c # version 2.0.0

My personal usage is based upon non-flat installs using webpack. If we could mandate that the "resolutions" field be enforced across nested dependencies that seems sane. However, I am not sure this version pinning is needed when we have the version range already in package.json and resolved versions in the different lockfile/shrinkwrap files.

Without resolution:

$ yarn list supports-color
├─┬ mocha@3.5.3
│ └── supports-color@3.1.2 
└─┬ chalk@1.1.3
  └── supports-color@2.0.0

With `"resolutions": {"supports-color": "2.0.0"}` in `package.json`:

$ yarn list supports-color
warning Resolution field "supports-color@2.0.0" is incompatible with requested version "supports-color@3.1.2"
└─ supports-color@2.0.0

Funny enough, with `"resolutions": {"supports-color": "^2.0.0"}`:

$ yarn list supports-color
└─ supports-color@2.0.0

So it dedupes and hoists to the top level (but isn't perfect, some unclear warnings appear depending on whether it resolves to a semver range or specific version).

Right now it seems unreliable:

resolutions@1.0.0 /Users/bfarias/Documents/tmp/resolutions
├─┬ a@1.0.1
│ └── c@1.0.1
└─┬ b@1.0.1
  └── c@1.0.1

LMDV-BRADLEY:resolutions bfarias$ yarn list c
yarn list v0.21.3
✨  Done in 1.25s.
LMDV-BRADLEY:resolutions bfarias$ cat package.json
{
  "name": "resolutions",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "resolutions": {
    "c": "1.0.0"
  },
  "bundledledDependencies": ["a", "b"],
  "dependencies": {
    "a": "*",
    "b": "*"
  }
}

We should figure out what exactly it is doing today before using it.

@bmeck

@bmeck your lockfile may be getting in the way. If it keeps causing problems consider filing an issue with Yarn since top-level, deduped resolution was the goal for --flat & resolutions (source: I helped with the original implementation).

@FredKSchott

@FredKSchott I've tried several different approaches of installing and none seem to complain about nested dependencies, nor duplicates of the same version.

🙏 ➡️ https://github.com/yarnpkg/yarn/issues/new

@FredKSchott

@FredKSchott yarnpkg/yarn#4585 seems to be the issue I think?

However, it still doesn't enforce that only a single copy of that name/version combination exists in the nested dep graph.

@bmeck

@bmeck both yarn/npm hoist packages to the top level (& dedup) when no version conflicts exist. By resolving all occurrences to a single version, it logically follows that that each occurrence of that dep will be hoisted & deduped to a single, top-level dependency.

@FredKSchott

@FredKSchott things like bundledDependencies can prevent that hoisting without version conflicts.

@bmeck

@bmeck yarn doesn't support bundledDependencies, AFAIK

@iarna

@iarna it at least has documentation for it: https://yarnpkg.com/lang/en/docs/dependency-types/#toc-bundleddependencies

@bmeck

@bmeck I asked first.

@zkat

@zkat ?

@bmeck

@bmeck I want to hear why and how it solves complete problems first. My experience with the last feature we had that had similar behavior, peerDependencies, was very much not a positive one. I want to hear why current solutions don't work, why assets would not work for solving those problems, and how this proposal compensates for the problems with the former manifestation of peerDependencies.

Ah, I took your comment as very derisive versus that explanation which is much less hostile.

@bmeck

@bmeck I'd like to focus on keeping the conversation on a single topic, and on rails, as well as based on concrete problems, real-world examples, and multiple possible solutions focused on those problems.

@zkat

@zkat in that case, I can provide a simple example from work.

We have had situations where react-intl and some polyfills have been loaded with specified versions that cause duplication in the module graph. This behavior has been undesirable for those packages, but we do have other packages that correctly are duplicates such as when we do staged transitions between react versions and have multiple versions on a single page. Having support for discrete multiple versions of react is great in this situation, but being able to ensure that react-intl does not get accidentally duplicated also is great.

[edit/addendum]: This situation has been a minor source of trouble in production in the past with real affects for localization. We don't necessarily control all dependencies per page since other teams do manage their applications outside of my team's control.

@bmeck

@bmeck in this specific example: if singleton: true were declared by React (which expects to be a singleton, so it's reasonable for them to use this flag), then you would not even be able to attempt loading multiple versions the way you are.

This specific example seems pretty corner-casey to me, unless we're talking about the separate feature of allowing multiple versions of a same-name package to live side-by-side, possibly in the same tree layer. Perhaps a different approach, or even manual manipulation of package-lock.json would serve better in this case? That allows you to freely manipulate the tree and gives a good escape hatch when these issues crop up.

@zkat

@zkat I'm not sure React needs to be treated as a singleton since it doesn't use a global store to persist data in an externally mutable/observable way. react-intl does however keep a singleton based store due to the addLocaleData API

@bmeck

@bmeck https://reactjs.org/warnings/refs-must-have-owner.html it does, indeed, require a single, non-conflicting version to be loaded, last I checked.

@zkat

@zkat if you pass Components created from differing versions of React between the opposing version you do see those errors, yes. If you just have multiple React versions that do not pass components to the opposing version it has worked fine for us so far (fingers crossed it keeps working).

@bmeck

@bmeck this definitely sounds to me like, in this concrete example, you're describing adding an extra layer of complexity to a problem that you're already "getting away with" while crossing your fingers, rather than providing a solid foundation of a solution to a much more common, larger problem.

@zkat

@zkat Our needs are complex yes. The finger crossing is more that React doesn't prevent transitions using multiple versions by forcing there to be a single version. However, I also am crossing my fingers for being able to get problematic packages that do not support multiple versions to mark themselves as such. React as it is today doesn't cause problems with multiple versions to my usage.

I'm not sure I would state my alternatives as solid foundations for now, but I will be happy if I can have both use cases above. Removing one of my use cases doesn't seem a way forward for progressive migration strategies that use multiple versions; and manually editing package-lock.json seems error prone and still removes the capability to do progressive migration while 2 versions of a package are in use unless you rename the packages to be something like react and react-16/react-next.

To me, this sounds like what peer dependencies already are. I wonder if perhaps a cleaner solution is a top-level package.json key like "requiredSingletons": ["react"], and then npm can check that in the entire graph for you?

Another companion change could be, like the (thankfully dead) preferGlobal, preferPeer - that way packages can declare themselves as one that prefers to be a peer dependency.

I think re-using peerDependencies would fulfill my use cases if the desire is to avoid introducing a new type of dependency. A way to error on failed preferPeer would be nice, but that should be easy to grab and tool off logs output by npm.

Ping on this issue 😄

On Polymer we've moved to publishing our modules using package names in import specifiers, anticipating movement on support in browsers via something like https://github.com/domenic/package-name-maps

We still have requirements that certain packages are singletons, ie polyfills, frameworks and custom elements. If npm supported this it would help our users considerably. Right now we're still recommending yarn --flat if a project can use it, because flatness guarantees uniqueness.

Navigation Menu

javascript: singleton packages #3

`"singleton": boolean`

`"singletonDependencies": string[] | "all"`

determination of "package name"

scoped packages

bundledDependencies

root package

realpathing

Without resolution:

With `"resolutions": {"supports-color": "2.0.0"}` in `package.json`:

Funny enough, with `"resolutions": {"supports-color": "^2.0.0"}`:

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

javascript: singleton packages #3

Description

"singleton": boolean

"singletonDependencies": string[] | "all"

determination of "package name"

scoped packages

bundledDependencies

root package

realpathing

Activity

FredKSchott commented on Oct 2, 2017

FredKSchott commented on Oct 2, 2017

bmeck commented on Oct 2, 2017

FredKSchott commented on Oct 2, 2017

Without resolution:

With "resolutions": {"supports-color": "2.0.0"} in package.json:

Funny enough, with "resolutions": {"supports-color": "^2.0.0"}:

bmeck commented on Oct 2, 2017

FredKSchott commented on Oct 2, 2017

bmeck commented on Oct 2, 2017

FredKSchott commented on Oct 2, 2017

bmeck commented on Oct 3, 2017

FredKSchott commented on Oct 3, 2017

bmeck commented on Oct 3, 2017

iarna commented on Oct 26, 2017

bmeck commented on Oct 30, 2017

10 remaining items

zkat commented on Dec 6, 2017

bmeck commented on Dec 6, 2017

zkat commented on Dec 6, 2017

bmeck commented on Dec 6, 2017

zkat commented on Dec 6, 2017

bmeck commented on Dec 6, 2017

zkat commented on Dec 6, 2017

bmeck commented on Dec 6, 2017

zkat commented on Dec 6, 2017

bmeck commented on Dec 6, 2017

zkat commented on Dec 6, 2017

bmeck commented on Dec 6, 2017

ljharb commented on Dec 6, 2017

bmeck commented on Dec 6, 2017

justinfagnani commented on Mar 22, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`"singleton": boolean`

`"singletonDependencies": string[] | "all"`

With `"resolutions": {"supports-color": "2.0.0"}` in `package.json`:

Funny enough, with `"resolutions": {"supports-color": "^2.0.0"}`: