WebExtensions 101

If you know the basic web technology stack (HTML, JS, and CSS), then you can indeed build a WebExtension.

It wasn't always this simple. In the past, one might have needed to know C or Objective-C and the internal workings of the browser's own code or even the operating system's idiosyncrasies.

Thankfully, there's been some recent consolidation efforts and some work toward standardization. If you're curious about how the actual standards work, be sure to check out the Browser Extension Community Group at the W3C.

However, this new collaborative world is still being built. In its current state, there are rough spots and differences (and in-progress polyfills).

Let's take a look at what it takes today to actually build a WebExtension for real.

In my post Building WebExtensions Because You Can, you'll find a description of the primary pieces that make up most WebExtensions. We'll be using just a handful of the available options and focus on getting one up and running.

Here's the Anatomy of a WebExtension chart from Mozilla for handy reference:

Let's start by installing some basic tooling and setting up the folder structure for this example project.

To get started, let's install Mozilla's web-ext command-line tool.

The web-ext command makes it possible to run Firefox with an empty profile (no existing cookies, history, or other extensions). This helps when testing your extension as there will be less to get in the way, and command line console debugging output -- if you prefer that.

It also serves as a handy way to package WebExtensions into the necessary .zip file and (potentially) communicate with the Mozilla Add-ons site to help you set things for signed self-hosted distribution. The .zip file it generates also works with the Chrome and Opera add-on stores.

Let's use npm to install it as a global command-line tool: sh $ npm install --global web-ext

Now. I'm assuming some things here. First, that you have Node.js and npm installed and available (if you've got the one, you've got the other, btw).

Second, I'll be setting up the project so that it can be used with npm for installing dependencies (and such). However, this is a fairly lightweight project with no need for preprocessing or complex dependencies, so we'll not be using npm directly today. Setting the project up this way though, means you're ready for it when you need it.

The folder structure below is the one I've (mostly) settled on for my WebExtension projects. It provides a foundation for using preprocessors and npm for installing dependencies or hooking in all the other node.js-style build tooling you might like. Take a look:

example-webextension/
+-- extension/
¶   +-- manifest.json
¶   +-- images/
¶   |   +-- icon32.png
¶   +-- lib/
¶   ¶   +-- dependency.js
¶   +-- dist/
¶   ¶   +-- nifty.js
¶   +-- popup.js
¶   +-- popup.html
¶   +-- options.html
¶   +-- styles.css
+-- src/ (optional; for use with typescript, etc)
¶   |   +-- nifty.js (newfangled typescript, etc)
+-- web-ext-artifacts/
¶   |   +-- example-webextension-0.0.1.zip
+-- package.json
+-- CODE_OF_CONDUCT.md
+-- CONTRIBUTING.md
+-- LICENSE
+-- README.md

The extension/ folder holds the actual "output" WebExtension. This is the directory that we'll package up later to distribute to the various WebExtension stores.

Inside this directory is the all-important manifest.json file. This file is essentially the first thing read by the browser when installing or running the WebExtension. It sets permissions, picks which scripts to inject in pages, defines where the options interface is located, and many other things.

We'll dig into that more in a bit, but for now, here's a look at the other files and folders in that directory:

• images/ - which, shockingly, contains the images for the extension

• lib/ - for storing dependencies -- a.k.a., other people's code

• dist/ - for any built/processed output of your code

• *.html - the HTML files used to build the UX for the WebExtension

• *.css - the CSS file(s) for styling all the things

You can certainly reorganize things later if you need, and as your extension grows, this simpler format may not scale to your liking.

It should be noted, that anything in this extension/ folder will get included in the distribution package -- even if it's not used in the WebExtension. This is why we have the extensions contents in a separate folder. For example, we don't want to accidentally include an entire node_modules/ dependency folder in the packaged WebExtension.

The src/ directory holds the unprocessed raw code for our project. This could be plain old JavaScript, but these days you might be writing in TypeScript or something I've not heard of yet. You could also keep your HTML and CSS in this directory if you like preprocessing those from formats like LESS, Jade, etc.

Those, however, aren't my jam, and I stick with ES6 and (typically) plain old HTML and CSS (as you'll see below). But you do you, `k?

The web-ext-artifacts/ folder (and its contents) are generated by the web-ext command-line tool. The .zip files here are used when distributing your extension via the various extension stores -- which we'll touch on at the end.

The package.json file is there because I use browserify and npm to manage the dependencies and build the scripts for (most of) my projects. If you're using git, then you've likely also got a .gitignore file that's excluding node_modules/ (or whatever) is holding your dependencies. I've ignored them here because I ignore them there -- and you likely already know that bit.

To wrap up the tree, there's a handful of text or markdown files about important things:

• how to contribute to this project: CONTRIBUTING.md

• how to be nice while contributing: CODE_OF_CONDUCT.md

• what permissions you're giving to others: LICENSE

• what this project is and how to use it: README.md

For most of you, the above list may be terribly introductory. However, it's very common to find code laying about the internets without two or three of these important files. They're not all mandatory (per se), but having them will greatly increase your chances of building a successful project with help from other people.

Still with me? Let's get to work.

Let's go ahead and prep this extension for using npm to manage future build tooling, preprocessing, etc. I'm using the name link-extractor for this demo extension, but feel free to name yours whatever you'd like.

Open a console and go to your new WebExtension directory. Then run the following command and fill out the questionnaire to your you liking.

$ npm init

Here's what I filled out (using the defaults for name, entry point, and license; I skipped test and repo for now):

name: (link-extractor)
version: (1.0.0) 0.0.1
description: Extracts linked data from links.
entry point: (index.js)
test command:
git repository:
keywords: WebExtension, Linked Data, HTML
author: BigBlueHat <byoung@bigbluehat.com>
license: (ISC)

Which created this JSON file:

{
  "name": "link-extractor",
  "version": "0.0.1",
  "description": "Extracts linked data from links.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [
    "WebExtension",
    "Linked",
    "Data",
    "HTML"
  ],
  "author": "BigBlueHat <byoung@bigbluehat.com>",
  "license": "ISC"
}

You should now have a package.json file that looks similar.

Chrome was the first browser to use this new style of WebExtension, so the API calls in that browser (and most others) under the chrome.* object. However, since things are moving toward a consistently shared API across browsers, these APIs are moving (gradually) to browser.*. Most of these APIs also started out as callback-passing functions, but are now moving to a Promise-based API.

Mozilla has created webextension-polyfill to normalize these inconsistencies across browsers.

Sadly, though, as of this writing the polyfill is not yet published to npm. So the setup means downloading the code from GitHub, installing dependencies, building it locally, then copying over the polyfill into your project.

That's a bit more than we can cover here, so I'll be using the chrome.* APIs for this extension.

However, given that the intent of the Browser Extension CG is to move the extension APIs in this direction, it's not a bad idea to give that a try if you want to get a jump on the rest of us.

For now, we'll stick chrome.* and callbacks.

Today we'll build a web extension that:

• shows a list of links from the existing page

• the text used in the link

• their link relationship (if any)

To do that, this WebExtension will use the following things:

• a browser_action (initially)

• a "pinned" popup

• an injected "content scripts" which we'll communicate with from the extension

To start things off, create a extension/manifest.json in your project directory and add the following JSON:

{
  "manifest_version": 2,
  "name": "link-extractor",
  "version": "0.0.1",
  "description": "Extracts links",
  "author": "BigBlueHat <byoung@bigbluehat.com>",
  "permissions": [
    "activeTab"
  ],
  "browser_action": {
    "browser_style": true,
    "default_title": "View the Links in this Page",
    "default_popup": "popup.html",
    "default_icon": "images/icon32.png"
  }
}

This manifest tells the browser the following:

• We need the activeTab permission so that we can inject scripts into the active tab in the current window.

• We have a browser_action that adds an icon to the browser UI which opens a pinned popup when clicked.

The activeTab permission is useful for narrowing the necessary permission grant to just the active tab instead of all the tabs in all the windows. Without this option, we would have to use the <all_urls> permission (plus its scary warning message about potentially reading and changing all the pages you visit). Alternatively, we could limit the request to just certain URLs or domains, but we'd still have to ask for the "scary" permission level...but only be able to work in a few places. The activeTab permission avoids all of that.

Also, if you don't absolutely need a certain permission for your extension, don't ask for it. This is both safer and friendlier to the folks using your extension -- which means more people are likely to use it.

The browser_action defines the bits for the pinned popup and the button we will add to the browser's interface. The popup.html file is a simple HTML page. The JavaScript included in that page has access to the special chrome.* (or browser.* if you're using the polyfill) extensions APIs which allow us to communicate from the popup to the page in the open tab.

From the popup, we will:

• inject a script to extract the links from the activeTab's web page

• send that list of links back to the popup.js code, which will in turn...

• load the list of links into the popup.html file

Since we're using the activeTab permission and a browser_action, we'll get some initial UI for "free." Once installed, there will be an icon (typically) in the top right area of the browser UI. Feel free to use this one while developing:

You can save that into your extension/images/ directory as icon32.png or make your own.

When that icon is clicked, it will open a pinned popup.

The popup.html contents will be loaded into the pinned popup in the UI. The popup.js referenced from popup.html (see below) will begin its injection, extraction, and communication work on the contents of the active tab.

Here's what popup.html looks like:

https://gist.github.com/ChrisWolfgang/554682029041ff5b84f9650a89a16069.js Later, we might improve the CSS, but for now, this is plenty. We've set up some empty table space to fill in with the link info found in the page, and we've included the popup.js, which will do the heavy lifting of populating the table.

But first, let's do a quick test of what's there already!

There are several ways to run our in-progress WebExtension. Each browser comes with its own UI for loading a manifest.json out of an "unpacked" extension directory. Here are the URLs to visit or things to click to load the UI in the various (popular) browsers:

• Chrome (also Vivaldi & Opera): <chrome://extensions/>

  • click the "Developer mode" checkbox (to review the buttons)

  • click the "Load unpacked extension..." button

• Firefox: <about:debugging>

  • click the "Load Temporary Add-on" button

• Edge: use the drop-down menu, click "Extensions"

  • click the "Load extension" button

So...yeah. Nearly the same, but not quite. Each of these browsers have increasingly more idiosyncratic ways of debugging, however. Since I can't cover them all here, we'll stick with using the web-ext command-line tool we installed earlier.

From the main project directory, run this on the command line: sh web-ext run -s extension/

That command will open an "empty" (no previous profile) Firefox with "stock" extensions (Pocket, etc.) and the in-progress extension we're building here. If everything's set up correctly, you should see an icon that matches the one here. Click it and you should see the table from popup.html above.

I'm hopeful that someday there will be a single command-line tool (perhaps web-ext) that will load "empty" versions of other browsers for testing extensions. Until then, finding the right menus and buttons for your preferred browser is your best option.

If you've used the command above, you should see that empty Firefox sporting your newly installed WebExtension:

Not much to see yet, but we know that web-ext run is working and the files are all loading where they should.

Now, let's make it work!

To keep this extension's effective code minimal and focused, we're going to add good ol' jQuery as a dependency. I know many of you have perhaps "moved on," but I've found (through trial and error...) that the editorial staff at add-on sites run by Mozilla and Opera greatly prefer libraries they already know. In fact, they've automated much of the detection and approval process at this point, so using something they like helps you get through that gate faster.

Head over to jquery.com/download and grab a copy of jQuery to suit your needs. I downloaded jquery-3.2.0.min.js, put it in the extension/lib folder, and added it to the popup.html file in the usual web page-y way -- via a <script> tag.

So popup.html now ends with these two <script> tags just above the </body>:

https://gist.github.com/ChrisWolfgang/f60b17082b97b100d9982e4b75549cfc.js Now let's flesh out the popup.js file:

You'll notice in the popup.js contents below that it uses fairly modern JavaScript. We're using things like arrow functions, let (in place of var), and template literals/strings. This is possible due to WebExtensions only being supported (in this current format) in fairly recent versions of Firefox, Chrome, etc., which also all support much of ES6/ECMAScript 2015.

Consequently, we get to write "modern" JavaScript without the need for any preprocessing. If you do decide to use something even more newfangled (TypeScript, etc.), then you can use the src/ folder plus some preprocessing scripts that output the .js files into extension/. We'll avoid that here to keep complexity just a bit lower.

extension/popup.js looks like this in all its amazing modernity:

let $links = $('#links');
chrome.tabs.executeScript(null, {
  file: 'extractor.js'
}, (results) => {
  let anchors = results[0];
  anchors.forEach((a) => {
    let tr = `<tr>
      <td><a href="${a.href}">${a.href}</a><br />
      ${a.text}
      </td>
      <td>${a.rel}</td>
    </tr>`;
    $links.append(tr);
  });
});

We start off with a little jQuery for later use to add to the table. Then, we use the chrome.tabs.executeScript() function to inject an extractor.js file into the active tab by passing null as the value of the first parameter (tabId). We then have a callback that receives the results of the extractor.js code. The value of results will be an array from all the places extractor.js has run.

Since we're not running it across multiple frames, it should only have a single value in the array: our list of link info. We then loop through that list of links and populate a table row using a template literal, which we then append (via jQuery) into the tbody of our table in popup.html.

Phew! Still with me?

Let's take a look at the extension/extractor.js:

(function() {
  let anchors = document.querySelectorAll('a[href]');
  let rv = [];
  anchors.forEach((a) => {
    rv.push({
      href: a.href,
      rel: a.rel,
      text: a.textContent
    });
  });
  return rv;
// sends the above `rv` as a return value...basically
})();

The code in extractor.js runs upon injection and should end in a value that is a structured clonable value. I'm using an anonymous function here to provide:

• a private variable namespace

• a single, obvious return value

• an idempotent function, so the result is the same regardless of being injected/run multiple times

You'll note I'm using document.querySelectorAll here rather than jQuery. You could of course inject jQuery also, but that takes longer and costs more in processing time than the value it would get us for this tiny bit of code.

Awesome. Let's make sure your files match mine, and then run this thing one last time.

Here's my directory layout:

.
+-- extension
¶   +-- extractor.js
¶   +-- images
¶   ¶   +-- icon32.png
¶   +-- lib
¶   ¶   +-- jquery-3.2.0.min.js
¶   +-- manifest.json
¶   +-- popup.html
¶   +-- popup.js
+-- package.json
4 directories, 10 files

If yours matches, then let's run this thing!

web-ext run -s extension/

Alternatively, you can load it via your preferred browser's add-on debugging UI (see the "Running WebExtensions" section above).

Here's what it should look like now:

If all went well, you have a working WebExtension that you can run in your own browser. However, distributing it is a completely different story.

The steps are essentially:

• package the WebExtensions into a single (zip-based) file archive

• upload it to one or more add-on stores

It's at point no. 2 where things get weird...or at least varied.

Some examples of publishing your WebExtension:

• Firefox and Opera can take up to a month for their editorial teams to review your code

• publication is completely free (so there's that)

• Chrome's add-on store costs you $5USD for the first "20 items"

• instant publishing (well...within an hour), but no editorial review

• Microsoft Edge is a unique beast and requires a full-on Microsoft Dev Center registration

• currently that's $19USD for individuals or $99USD for companies

• not sure what (if any) editorial process happens does open the door to publishing any kind of Windows-based app (which is likely the point of the requirement)

For now, let's just bundle up this little extension and prep it for the free Mozilla Add-ons site. We'll use web-ext again:

$ web-ext build -s extension/

You should now see the web-ext-artifacts/ folder with a link-extractor-0.0.1.zip in your project directory.

Next, let's visit the Submit a New Add-on page.

On that page, choose the "On this site" option. Self-hosted distribution is possible, but it's a bit more complex so we'll (sadly) have to skip it today.

On the next page, click "Select a file..." and choose the .zip file created earlier. An automated review will run which will generate a report. Likely for something this simple, it will only contain warnings (if anything) and often those will be in other people's code (like jQuery). The editorial staff will also see these warnings (or errors) and may reference them in their review.

On the next page, you'll have the opportunity to add listing information to help people find your extension. At the bottom of this page is the "Submit Version for Review" button. Click that...and hope for the best!

In my case, I hit a small snag. There was already an extension using the link-extractor "slug" in the Mozilla Add-on store, so I changed the slug link-info-extractor and will likely now rename the project to that.

Often, you'll find it's these little annoyances (more than anything) that will slow you down when building WebExtensions. They're not technically "hard," but because of all the not-quite-compatible APIs, stores, and distribution models, there can be a disproportionate amount of time spent on resizing icons and screenshots versus coding the extension itself.

There's some likelihood that over time these wrinkles will get ironed out either by the browsers or by your friendly neighborhood polyfill developer. We'll see.

If you know HTML, CSS, and JS, you have what you need to build WebExtensions that can do all kinds of things. In addition to this tech know-how though, you'll need a daily dose of patience as you jump between documentation and distribution sites for the various browsers and their (still weirdly proprietary) stores.

In the end, you may conclude (as I have) that "WebExtensions" is a bit of a misnomer and the "BrowserExtension" term used by the Community Group is more accurate. The unique APIs, the still-in-progress collaboration, and the very-not-webby distribution models will likely trip you up as you go.

However, if there's an "always on" sort of customization you want, or a piece of futuristic proof-of-concept you'd like to experiment with now, WebExtensions (or BrowserExtensions) are the best available tool with the least amount of overhead. At least today.

I sent this little WebExtension through the Mozilla Add-on editorial process, and here's what I ran into using the code above:

• they didn't like jQuery...this time

• the template literal/string + $.append() === an innerHTML() call (bad)

• and I'd left some junk laying about in extension/

Consequently, I did some housecleaning and code rewriting and resubmitted it. It passed with flying colors that time -- check it out!.

Here's what I changed:

First, I removed the extension/lib/mustache.min.js file I'd been pondering using, but forgot to remove when I decided not to. This was their primary complaint actually and the reason for the rejection of v0.0.1.

Second, they asked me to improve the code to not use jQuery (it being a rather large dependency for such little code) and to avoid using innerHTML()-based DOM manipulation. To fix those, I rewrote the template-based bits to pure-DOM calls (ugly...but effective) and by doing so avoided the innerHTML() usage.

Here's the new popup.js file sans-jQuery:

links = document.getElementById('links');
chrome.tabs.executeScript(null, {
  file: 'extractor.js'
}, (results) => {
  let anchors = results[0];
  anchors.forEach((a) => {
    let tr = document.createElement('tr');
      let td_url = document.createElement('td');
        let link = document.createElement('a');
        link.href = a.href;
        link.target = '_blank';
        link.textContent = a.href;
        let br = document.createElement('br');
      td_url.append(link, br, a.text);
      let td_rel = document.createElement('td');
      td_rel.textContent = a.rel;
    tr.append(td_url, td_rel);
    links.append(tr);
  });
});

It's a bit more code (because native-DOM is a bit wordier than jQuery). It was (in the end) worth it to remove the dependency, the subsequent load time, and to avoid the use of innerHTML().

The review process must have been expedited (it being an update and all), because it was cleared later the same day! Check it out in the Mozilla Add-ons site.

I've also put the code up on GitHub and linked to that as the "Support Website" on the Add-ons site.

There's obviously more to be done in both places to make this a well-promoted extension, but that's all polish (screenshots, a meaningful description) to be added soon -- or as needed or wanted (your needs may vary).

Lastly, I find it prudent to publish to the Mozilla Add-on site first. The additional code review (as seen above) is helpful, and it's likely they'll see something you've missed with regards to performance, security, or code quality. Having this editorial process started earlier can save you lots of time later as you work to ship quality WebExtensions to all the other browser add-on stores.

Your mileage will certainly vary, but the feedback is valuable. The increased understanding of what browsers want will also serve to make you a better Web developer. Enjoy that. I do.