Introduction to npm module installation mechanism
npm is Node’s module manager, which is extremely powerful. It is one of the important reasons for Node’s success.
Because of npm, we can install modules written by others with just one line of command.
$ npm install
This article introduces the details of the npm module installation mechanism and how to solve the problem of slow installation speed.
1. Speaking from npm install
npm install
The command is used to install the module to the
node_modules
directory.
$ npm install <packageName>
Before installation, it
npm install
will first check
node_modules
whether the specified module already exists in the directory.
If it exists, it will not be reinstalled, even if a new version of the remote warehouse already exists.
If you want, npm will force a reinstallation of a module regardless of whether it has been installed or not. You can use the
-f
or
--force
parameter.
$ npm install <packageName> --force
Two, npm update
If you want to update the installed modules, you must use
npm update
commands.
$ npm update <packageName>
It will first go to the remote warehouse to query the latest version, and then query the local version. If the local version does not exist, or the remote version is newer, it will be installed.
Three, registry
npm update
How does the command know the latest version of each module?
The answer is that the npm module repository provides a query service called registry.
Take npmjs.org as an example, its query service URL is
https://registry.npmjs.org/
.
This URL is followed by the module name, and you will get a JSON object with information about all versions of the module.
For example, if you visit
https://registry.npmjs.org/react
, you will see information about all versions of the react module.
It has the same effect as the following command.
$ npm view react # npm view 的别名 $ npm info react $ npm show react $ npm v react
After the module name of the registry URL, you can also follow the version number or label to query the information of a specific version. For example, if you visit https://registry.npmjs.org/react/v0.14.6, you can see version 0.14.6 of React.
There is an
dist.tarball
attribute in the
returned JSON object, which
is the URL of the compressed package of the version.
dist: { shasum: '2a57c2cf8747b483759ad8de0fa47fb0c5cf5c6a', tarball: 'http://registry.npmjs.org/react/-/react-0.14.6.tgz' },
Go to this website to download the compressed package, unzip it locally, and get the source code of the module.
npm install
And
npm update
commands are installed in this way.
Fourth, the cache directory
npm install
Or
npm update
commands, after downloading the compressed package from the registry, they are all stored in the local cache directory.
This cache directory is the directory under the user’s home directory by default on Linux or Mac, and by default
.npm
on Windows
%AppData%/npm-cache
.
Through the configuration command, you can view the specific location of this directory.
$ npm config get cache $HOME/.npm
You’d better browse this directory.
$ ls ~/.npm # 或者 $ npm cache ls
You will see a large number of modules stored inside, the storage structure is
{cache}/{name}/{version}
.
$ npm cache ls react ~/.npm/react/react/0.14.6/ ~/.npm/react/react/0.14.6/package.tgz ~/.npm/react/react/0.14.6/package/ ~/.npm/react/react/0.14.6/package/package.json
Each version of each module has its own subdirectory, which is a compressed package
package.tgz
file
of the code
, and a description file
package/package.json
.
In addition, a
{cache}/{hostname}/{path}/.cache.json
file
will be generated
.
For example, when downloading the react module from the npm official repository, a
registry.npmjs.org/react/.cache.json
file
will be generated
.
This file saves all version information, as well as the latest modification time of the module and the ETag returned by the server when the latest request was made.
{ "time":{ "modified":"2016-01-06T23:52:45.571Z", // ... }, "_etag":"\"7S37I0775YLURCFIO8N85FO0F\"" }
For some non-critical operations (such as
npm search
or
npm view
), npm will first check
.cache.json
the latest update time of the modules inside, and whether the gap with the current time is within an acceptable range.
If it is, it will no longer make a request to the remote warehouse, but will directly return
.cache.json
the data.
.npm
The directory contains a large number of files, the command to clear it is as follows.
$ rm -rf ~/.npm/* # 或者 $ npm cache clean
Five, the installation process of the module
To sum up, the installation process of the Node module is like this.
- Give
npm installorders- Npm queries the registry for the URL of the module compression package
- Download the compressed package and store it in the
~/.npmdirectory- Unzip the compressed package to the
node_modulesdirectory of the current project
Note that after a module is installed, two copies are actually saved locally.
One is
~/.npm
the compressed package in the
node_modules
directory
, and the other is
the decompressed code in the directory.
However,
npm install
when
running
, only the
node_modules
directory is
checked
, not the
~/.npm
directory.
That is, if a module
～/.npm
under compression package, but is not installed in the
node_modules
directory, npm will still be downloaded from a remote repository a new archive.
This behavior can ensure that the latest code is always obtained, but sometimes it is not what we want. The biggest problem is that it will greatly affect the installation speed. Even if the compressed package of a certain module is in the cache directory, it has to be downloaded from the remote warehouse. How could it not be slow?
In addition, in some occasions there is no network (such as on an airplane), but the module you want to install is clearly in the cache directory, and it cannot be installed at this time.
Six,
--cache-min
parameters
To solve these problems, npm provides a
--cache-min
parameter to install modules from the cache directory.
--cache-min
The parameter specifies a time (in minutes), and only modules that exceed this time will be downloaded from the registry.
$ npm install --cache-min 9999999 <package-name>
The above command specifies that only modules older than 999999 minutes will be downloaded from the registry. In fact, it is specified that all modules are installed from the cache, which greatly speeds up the download speed.
It also has another way of writing.
$ npm install --cache-min Infinity <package-name>
However, this is not equivalent to offline mode, and a
network connection
is still required at
this time
.
Because the current
--cache-min
implementation has some problems.
(1) If the specified module is not in the cache directory, then npm will connect to the registry and download the latest version. This is no problem, but if the specified module is in the cache directory, npm will also connect to the registry and send out the etag of the specified module. The server returns a status code of 304, indicating that there is no need to download the compressed package again.
(2) If a module is already in the cache, but the version is lower than the required version, npm will report an error directly instead of going to the registry to download the latest version.
The npm team is
aware
of these problems and is
rewriting the cache
.
And, a
--offline
parameter
will be provided in the future
so that npm can be used offline.
However, there is no schedule for these improvements.
Therefore, the current use of
--cache-min
improved installation speed is problematic.
Seven, offline installation solutions
The community has proposed several solutions for offline use of npm. They can greatly speed up the installation of modules.
Solutions are roughly divided into three categories.
The first category is Registry agent.
The usage of the above three modules is very similar. They all start a Registry service on this machine, and all
npm install
commands must pass through this service proxy.
# npm-proxy-cache $ npm --proxy http://localhost:8080 \ --https-proxy http://localhost:8080 \ --strict-ssl false \ install # local-npm $ npm set registry http://127.0.0.1:5080 # npm-lazy $ npm --registry http://localhost:8080/ install socket.io
With the local Registry service, the cache installation can be fully realized, and offline use can be realized.
The second category,
npm install
substitution.
If
npm install
the behavior can
be changed
, the cache installation can be achieved.
npm-cache
The tool is this idea.
Any use of
npm install
local, can be used
npm-cache
instead.
$ npm-cache install
The third category,
node_modules
as the cache directory.
The idea of this solution is not to use the
.npm
cache, but to use the
node_modules
directory of the
project
as the cache.
The above two tools can mark the
node_modules
directory of the
project
into a compressed package, and when you install it later, you will take out the files from this compressed package.
(over)