Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introducing esm.run - A New-Age CDN for JavaScript modules #18263

Open
MartinKolarik opened this issue Oct 30, 2020 · 47 comments
Open

Introducing esm.run - A New-Age CDN for JavaScript modules #18263

MartinKolarik opened this issue Oct 30, 2020 · 47 comments

Comments

@MartinKolarik
Copy link
Member

MartinKolarik commented Oct 30, 2020

We are excited to announce a long-awaited feature - improved support for packages distributed as ES modules 馃帀

It is currently in beta, which means we encourage you to try it out, but not use it in production applications yet. This issue will cover all technical details and can be also used to provide feedback or report bugs if you find any.

TL;DR

  1. Demo and essential information: https://www.jsdelivr.com/esm
  2. You can load any ESM file like this:
    • https://cdn.jsdelivr.net/npm/uuid/+esm (default file)
    • https://cdn.jsdelivr.net/npm/uuid/dist/esm-browser/index.js/+esm (explicit file).
  3. There's a new domain that you can use for shorter and prettier links. It redirects everything to our main domain and is not meant to be used in production:
    • https://esm.run/uuid
    • https://esm.run/uuid/dist/esm-browser/index.js

How this works

Locating entry points

To find the default file, we use module and jsnext:main fields from package.json. These fields are currently used by all existing tooling and by most package authors. It is very likely we'll also add support for the node.js exports field in the future.

Handling imports

Resolving

  1. Absolute URLs like https://example.com/foo are always kept in their original form.
  2. Relative imports are first resolved via the browser field if it exists (this applies to the entry point as well) and then resolved using node's experimental resolution algorithm, which supports extension resolution and importing from directories that include an index file. The supported extensions are: .mjs, .js, and .json.

Bundling

  1. Internal imports (pointing to files in the same package):

    • are bundled with the requested file if they are static, e.g., import foo from './foo.js',
    • are rewritten to their jsDelivr URL if they are dynamic, e.g., import('./foo.js').
  2. External imports (pointing to files in other packages):

    • are always rewritten to their jsDelivr URL, whether they are static or dynamic.

This means that one npm package roughly equals one bundle and one HTTP request, which allows good caching and doesn't require too many HTTP requests.

External URLs are always generated with fully resolved dependency versions based on package.json dependencies (or latest if the version is not specified there). This further improves caching and guarantees stable versions even for transitive dependencies.

Performance

  • The generated files are automatically minified.
  • Source maps are available for easy debugging.
  • Once generated, the files are stored in our permanent storage and served from there.
  • In supported browsers, HTTP preloading is used to start fetching external static dependencies as soon as possible.

What doesn't work (but may in the future)

  • Packages that are written in CJS, or a mix of ESM and CJS.
  • Packages that import core node.js modules.
  • Packages that import files in formats other than JS/JSON.
  • Packages larger than 50 MiB.
  • search on esm.run lists all packages. This will soon be changed to list only the supported ones.

The esm.run domain

This domain is introduced as an easier to remember and type alternative. It doesn't run on our multi-cdn architecture and simply redirects all requests to the main cdn domain, so it's not meant to be used in production.

@Finesse
Copy link

Finesse commented Dec 9, 2020

Is there a URL option to bundle all imports inside a single file? Comments in files produced by jsDeliver say that Rollup is used so it won't be a problem.

@lubomirblazekcz
Copy link

lubomirblazekcz commented Dec 19, 2020

Will the current URL scheme change to me more friendly? Currently what I see in devtools/network is bunch of +esm files

I guess this https://cdn.jsdelivr.net/npm/[email protected]/+esm/d3.js would be better than this https://cdn.jsdelivr.net/npm/[email protected]/+esm

Skypack and esm.sh have similar approach for production urls
https://cdn.skypack.dev/pin/[email protected]/min/react.js
https://cdn.esm.sh/[email protected]/esnext/react.js

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Dec 19, 2020

@evromalarkey I did notice this problem but you can easily configure devtools to show full paths by right-clicking the column:

image

Alternatively, you could also add the filename as a query string, which we ignore: https://cdn.jsdelivr.net/npm/[email protected]/+esm?d3

@FredKSchott
Copy link

FredKSchott commented Jan 26, 2021

Congrats on the release! I've run your benchmarks myself a few times now, and can't reproduce your numbers across a few different devices. Here's what I see on my laptop:

Screen Shot 2021-01-25 at 7 41 36 PM

I don't know if this is out of date or just bad, but the data appears outdated across both Skypack and https://esm.run.

Additionally, your benchmark imports are to https://cdn.jsdelivr.net/npm/d3/+esm, and not https://esm.run. That's fine if you want to measure the "optimized" use-case, but then that should really be compared against Skypack's optimized URL as well: https://docs.skypack.dev/skypack-cdn/api-reference/pinned-urls-optimized

Can you update these stats to better represent things to your users?

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Jan 26, 2021

Hi @FredKSchott, the performance may of course change over time as each of the services develops. We would have preferred showing realtime results but that wasn't possible so we went with numbers that were accurate at the time this was launched. We'll recheck and update if the results are consistently different now.

Regarding the pinned URLs, those are not functionally equivalent because they lock down a specific version while jsDelivr (even on our primary domain) and unpkg resolve to latest.

@vp2177
Copy link

vp2177 commented Jan 30, 2021

I'm not sure this is the appropriate place to report this, I've tried this service with lit-element and while there are no errors logged, their html function doesn't seem to work.

Here is a repro: https://vp2177.github.io/js-utils/?jsdelivr-lit-element-issue

I have also tried loading lit-element from SkyPack and in that case it works, you can try by uncommenting that import.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Jan 30, 2021

Thanks for the report @vp2177. I don't immediately see where's the problem as L.html returns the correct object but we'll check this further later.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Feb 6, 2021

@vp2177 after closer inspection we found the problem but not yet sure how we'll be able to fix this.

lit-element imports main lit-html file here and it also imports another nested file shady-render.js from lit-html here. The problem is shady-render.js imports the main lit-html file as well.

When we get a request for shady-render.js we include all of its imports, including the main lib-html file (this is by design - as explained in the first post here, internal imports are always bundled).

Then when lib-element imports the main lib-html file in a separate request, it's loaded a second time. This would not be a breaking problem in most cases but the library relies on instanceof checks and those don't work if two versions of the code are loaded.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Feb 6, 2021

@FredKSchott it seems Skypack is now consistently faster than it was before so we decided to update the numbers.

@lubomirblazekcz
Copy link

lubomirblazekcz commented Feb 10, 2021

https://esm.run/@fullcalendar/daygrid fails to load, you should check all @fullcalendar plugins though, I think it's because import css (eg. https://cdn.jsdelivr.net/npm/@fullcalendar/timegrid/main.js)

Another problem is https://esm.run/dayjs (works on https://esm.run/dayjs/esm though)

@lubomirblazekcz
Copy link

lubomirblazekcz commented Feb 10, 2021

Also I see a potential problem with resolving subfolders (only with production url) when using importmaps

{
  "imports": {
    "dayjs": "https://cdn.jsdelivr.net/npm/dayjs/esm/+esm",
    "dayjs/": "https://cdn.jsdelivr.net/npm/dayjs/esm/+esm/"
  }
}

import isBetween from "dayjs/plugin/isBetween.js" - this wouldn't work

it would work with esm.run url though

{
  "imports": {
    "dayjs": "https://esm.run/dayjs/esm",
    "dayjs/": "https://esm.run/dayjs/esm/"
  }
}

It would work with https://cdn.jsdelivr.net/npm/dayjs/esm/, but that's only because this library has esm version builded on npm. The same problem could be in many other libraries with subfolders.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Feb 10, 2021

I think it's because import css

Yes, that is not supported yet, as mentioned in the original post.

Another problem is https://esm.run/dayjs (works on https://esm.run/dayjs/esm though)

dayjs removed the "module" package.json entry in the recent versions - I'm not sure why but there isn't much we can do in this case.

Regarding the import maps - will check this closer.

@mikabytes
Copy link

mikabytes commented Apr 22, 2021

Would it be possible to add HTTP2 push support, and an option to disable bundling?

With bundling, we get this issue:

Imagine having A.js imports C.js. We run an SPA. On initial fetch only C.js is imported as the rest aren't needed. Later on, A.js is imported.

Result:

  • C.js has been downloaded twice
  • C.js has been parsed and evaluated twice
  • The two imports of C.js are not the same. They are completely different files, so references are broken

Bundling is a dinosaur performance thing. We better avoid it when possible.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Apr 29, 2021

@mikabytes indeed, that is basically the issue described in #18263 (comment) and we might end up introducing some options to fix it.

@Xeevis
Copy link

Xeevis commented Jun 13, 2021

Hello, I have trouble importing libraries that use process.env.NODE_ENV to differentiate development and production builds. For example

import tippy from "https://cdn.jsdelivr.net/npm/tippy.js/+esm"; 

throws in browser

- ReferenceError: process is not defined
- at https://cdn.jsdelivr.net/npm/tippy.js/+esm:12:2691

In skypack, non-production code (including the expression) is stripped away.

https://cdn.skypack.dev/-/[email protected]/dist=es2020,mode=imports/optimized/tippyjs.js
(no process.env.NODE_ENV)

vs

https://cdn.jsdelivr.net/npm/[email protected]/+esm
(plenty of process.env.NODE_ENV)

Since esm.run is using Rollup, this should be easy to solve with @rollup/plugin-replace plugin (which is also what Tippy suggests).

import replace from '@rollup/plugin-replace';

export default {
  plugins: [
    replace({
      'process.env.NODE_ENV': JSON.stringify('production')     
    })
  ]
};

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Jun 13, 2021

We're aware of this pattern for CJS files but I'd say it's a bad idea to assume process exists in ESM code. One of the key advantages of ESM is that it should work in a browser without bundling so the check is usually written in a safe way as process && process.env && process.env.NODE_ENV

Still, if some modules rely on it, we'll look into replacing NODE_ENV to improve the compatibility.

@MaximKing1
Copy link

MaximKing1 commented Aug 11, 2021

Congrats on the release! I've run your benchmarks myself a few times now, and can't reproduce your numbers across a few different devices. Here's what I see on my laptop:

Screen Shot 2021-01-25 at 7 41 36 PM

I don't know if this is out of date or just bad, but the data appears outdated across both Skypack and https://esm.run.

Additionally, your benchmark imports are to https://cdn.jsdelivr.net/npm/d3/+esm, and not https://esm.run. That's fine if you want to measure the "optimized" use-case, but then that should really be compared against Skypack's optimized URL as well: https://docs.skypack.dev/skypack-cdn/api-reference/pinned-urls-optimized

Can you update these stats to better represent things to your users?

Just a question where did you get that comparison table from? Did you make it using something like photoshop or a service looks nice 馃憤

@zarianec
Copy link
Collaborator

zarianec commented Aug 11, 2021

@MaximKing1 this is regular benchmark results from https://www.jsdelivr.com/esm. You just need to resize your browser's width until the page gets redrawn for smaller devices.

@IgorNovozhilov
Copy link

IgorNovozhilov commented Sep 28, 2021

@MartinKolarik

@vp2177 after closer inspection we found the problem but not yet sure how we'll be able to fix this.

lit-element imports main lit-html file here and it also imports another nested file shady-render.js from lit-html here. The problem is shady-render.js imports the main lit-html file as well.

... This would not be a breaking problem in most cases but the library relies on instanceof checks and those don't work if two versions of the code are loaded.

Found a similar problem in Material Web Components.
mwc-tab-bar-base.ts imports file mwc-tab-base.ts as import {TabBase} from '@material/mwc-tab/mwc-tab-base';
At the same time file mwc-tab-base.ts imported as nested file in mwc-tab.ts, which is also imported from mwc-tab-bar-base.ts as import {Tab} from '@material/mwc-tab';.
And as a result, this check does not work: .filter((e: Node) => e instanceof TabBase, because the class declaration is duplicated

Proposal

There is a suggestion how to solve this problem via package.json#exports

  1. (For unambiguity of import and restoring order) For "+esm" mode, if package.json contains exports block external import from CDN on the path to the file not specified in exports

Like in CommonJS, module files within packages can be accessed by appending a path to the package name unless the package鈥檚 package.json contains an "exports" field, in which case files within packages can only be accessed via the paths defined in "exports".
(褋) https://nodejs.org/dist/latest-v16.x/docs/api/esm.html#esm_terminology

  1. Export paths from exports always provide the independent URI's, and replace the import of nested files to inside a bundled file to external paths, if they are finally resolved in the ways specified in exports

For example, for the above case
Both files (https://cdn.jsdelivr.net/npm/@material/mwc-tab/+esm, https://cdn.jsdelivr.net/npm/@material/mwc-tab-bar/+esm) will contain:
import{TabBase as c}from"/npm/@material/[email protected]/mwc-tab-base/+esm";

Perhaps this will be another headache for modules authors , but it will develop people's desire for a thoughtful declaration of dependencies and exported paths for the module. imho 馃悎

FAQ
Anticipating the question that this will increase the number of downloaded files, I will answer that this problem is solved module authors by generalizing the export to one point. It is their decision whether to allocate independent access points, or leave a single file.

I can notice that even directly in "Node.js", importing from different export points can periodically cause problems with duplicating code downloads with different versions. (Tested by personal bitter experience 馃く)

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Sep 28, 2021

@IgorNovozhilov the problem is definitely fixable by package authors writing the imports in a different way but our goal is to make as many packages as possible work without requiring specific patterns.

@IgorNovozhilov
Copy link

IgorNovozhilov commented Sep 28, 2021

make as many packages as possible work

In this case, if you just add replacement nested file imports to external ones, it will definitely increase the number of packages that will work correctly when importing from your CDN 馃檭

without requiring specific patterns

package.json#exports is gradually becoming more popular, it has much more possibilities for use in combination with ES modules. In my opinion, I would still consider exports as basic definition of available export points, and not as specific pattern for "+esm" mode

@claviska
Copy link

claviska commented Oct 11, 2021

This is fantastic! May I suggest using ?module or ?esm instead of /+esm in the URL? Aside from dev tools showing tons of +esm entries, it feels very strange using a virtual folder that could potentially collide with real folders.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Oct 11, 2021

@claviska we're considering making it a prefix instead of a suffix but the query string is not an option, we ignore query strings globally for performance reasons and can't make an exception for a specific parameter.

@claviska
Copy link

claviska commented Oct 11, 2021

That's fair and would 100% satisfy my concerns. Thanks for clarifying!

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Nov 2, 2021

@Xeevis we shipped the process.env.NODE_ENV replacement today. It doesn't apply to previously-generated files so far, but works with the new ones, e.g. https://cdn.jsdelivr.net/npm/[email protected]/+esm

@ylemkimon
Copy link

ylemkimon commented Dec 2, 2021

If there are no dependencies in the ESM, there should be no difference from the original jsDelivr URLs except minification, right? For instance, are https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.mjs (there is no import statement) and https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.mjs/+esm identical except minification?

@zarianec
Copy link
Collaborator

zarianec commented Dec 2, 2021

@ylemkimon optimizations (like tree-shaking) may apply to the bundled code so the content could vary. But even with optimizations functionality should be identical to original files.

In your particular case, the bundled file is identical to the original one.

@cristiano-belloni
Copy link

cristiano-belloni commented Dec 5, 2021

Is the resolving / bundling code open source?

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Dec 5, 2021

@cristiano-belloni no but it's mostly based on rollup.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Dec 9, 2021

We made many improvements here in the past weeks. For now, all of them are available at our preview domain: https://jsdelivr-origin-dev.herokuapp.com/npm (some are already on the production version too):

  1. Importing CSS should work now, see e.g., https://cdn.jsdelivr.net/npm/@fullcalendar/daygrid/+esm
  2. process.env.NODE_ENV is now replaced with production and dead code elimination is applied.
  3. package.json exports/imports fields are supported.
  4. CJS modules are supported.

We're still considering the following issues:

  1. Friendlier URL scheme.
  2. Import maps prefix resolving.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Feb 15, 2022

All of the above-listed features are now live on the production domain.

@nihgwu
Copy link

nihgwu commented Mar 14, 2022

I wanted to try esm.run over esm.sh in after this PR, it's super fast and most of the examples work fine, expect MUI, you can reproduce the issue here, I'd say SkypackCDN has the same issue as well, but somehow esm.sh fixed it

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Mar 14, 2022

Thanks for reporting @nihgwu! Turns out we didn't properly purge the old files versions in some cases. This should be fixed soon.

@nihgwu
Copy link

nihgwu commented Mar 14, 2022

@MartinKolarik forgot to mention another issue, I got two requests for the same react version, which will cause this error, I double checked the request url is exactly the same, don't know why, unless I explicitly use https://cdn.jsdelivr.net/npm/[email protected]/+esm and https://cdn.jsdelivr.net/npm/[email protected]/+esm

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Mar 14, 2022

@nihgwu please check now.

@nihgwu
Copy link

nihgwu commented Mar 14, 2022

@MartinKolarik thank you for your quick response, really appreciated. It almost works, now I got TypeError: Cannot read properties of null (reading 'createSvgIcon') error, it will work if I don't import the icon, reproduce demo

And this issue is still there, which is the biggest problem for me, I can't use https://cdn.jsdelivr.net/npm/[email protected]/+esm as alias as then jsx-runtime will be resolved to https://cdn.jsdelivr.net/npm/[email protected]/+esm/jsx-runtime which is wrong, and I don't want to change all of my react import to remote version as in that way I won't get typings

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Mar 14, 2022

And #18263 (comment) is still there, which is the biggest problem for me, I can't use https://cdn.jsdelivr.net/npm/[email protected]/+esm as alias as then jsx-runtime will be resolved to https://cdn.jsdelivr.net/npm/[email protected]/+esm/jsx-runtime which is wrong, and I don't want to change all of my react import to remote version as in that way I won't get typings

I don't follow here, I don't see any duplicate requests in the jsFiddle.

It almost works, now I got TypeError: Cannot read properties of null (reading 'createSvgIcon') error, it will work if I don't import the icon, reproduce demo

Indeed, I don't see why yet, if you got a chance to pinpoint what exactly is going wrong and where it would greatly help (there's many nested imports so we need to find which file exactly is getting transformed incorrectly if it's supposed to work).

@nihgwu
Copy link

nihgwu commented Mar 14, 2022

@MartinKolarik I tried to workaround the dup react issue here, and I think probably you should use the same strategy, and in that way we will get better debug experience, right now they are all +esm in the network panel, so I propose the following format

// response for `https://esm.run/react`, instead of redirecting
export * from 'https://cdn.jsdelivr.net/npm/[email protected]/+esm'
export { default } from 'https://cdn.jsdelivr.net/npm/[email protected]/+esm'

in this way we can ensure only one version of React will be imported

You can play with this deployment

the current url format is insane
image

@nihgwu
Copy link

nihgwu commented Mar 15, 2022

@MartinKolarik Regarding the duplicated requests for React, repro demo in jsFiddle

The first and forth requests are importing the same content, and for my case, the request urls are also the same
image

@nihgwu
Copy link

nihgwu commented Mar 15, 2022

@MartinKolarik And for the icon issue, here is the problem, the import result is null

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Mar 15, 2022

@MartinKolarik Regarding the duplicated requests for React, repro demo in jsFiddle

The first and forth requests are importing the same content, and for my case, the request urls are also the same image

I believe here it would help if you loaded https://esm.run/[email protected] in your code but the versions handling is indeed a little problematic, I'll see if we can make it more intuitive.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Mar 15, 2022

@MartinKolarik And for the icon issue, here is the problem, the import result is null

The file doesn't have a default export even in the original version so that seems right: https://cdn.jsdelivr.net/npm/@mui/[email protected]/utils/index.js

Btw we should probably move this to a separate issue, feel free to open one.

@nihgwu
Copy link

nihgwu commented Mar 15, 2022

@MartinKolarik in my case the duplicated urls are both https://cdn.jsdelivr.net/npm/[email protected]/+esm, OK I can reproduce it with your suggestion https://jsfiddle.net/sq0xLmz4/

Seems it's related to react-dom, it will happen only if I import react-dom at the same time, but it's OK with other packages I'm wrong on this, it happens to other packages as well

What about my suggestion to response instead of redirecting? That's also what SkypackCDN and esm.sh do

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Mar 15, 2022

Indeed, we may need to do the reexport thing for this case.

@MartinKolarik
Copy link
Member Author

MartinKolarik commented Mar 15, 2022

Also regarding the paths, there's a reasonable workaround #18263 (comment) but we're still considering other options too.

@nihgwu
Copy link

nihgwu commented Mar 15, 2022

Indeed, we may need to do the reexport thing for this case.

Yes I just tested, it's not only for react but any package with a dependent, the dependency will always be requested twice, we should reexport to solve it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests