Setting up a Dist Mirror

Posted: Oct 16, 2015 by Bryan Tong

This will show an easy way to setup and maintain a mirror of the dist folder on your server for quicker downloads.

In my scenario I wanted to setup a mirror copying what is located here:

This is for testing with which we are helping with some upgrades. In testing I am getting terrible download rates. Thus, slowing down my development. My way of handling this is to mirror these files on one of our servers.

After doing some research on the best way to handle this. I found a thread on how to compose a reliable wget command that can be used on a cron to reliably mirror the dist folder without any interaction from the upstream.

Create a Mirror Folder

In my case I am going to use /opt/nodejs

$ mkdir -p /opt/nodejs/dist
$ chown -R node. /opt/nodejs
$ cd /opt/nodejs

Initial Sync

I like to do the initial sync in the foreground just to make sure it works the way I am intending.

$ cd /opt/nodejs
$ wget -m -nH -e robots=off -np --convert-links --reject="index.html*"

This will take a while as it will actually download every file in this folder.

For my instance here it was 20GB.

Setup a Cron

Now we want to keep our mirror up-to-date so we need a cron job.

$ echo "0 0 * * * root cd /opt/nodejs; wget -m -nH -e robots=off -np --convert-links --reject=\"index.html*\" >> /var/log/nodejs-dist 2>&1" > /etc/cron.d/nodejs-dist

This will install a cron job that will sync our mirror folder every day at midnight.

Setup a Webserver

Lastly, if we want to use our mirror we are going to need a webserver to handle serving the files.

We are an NGINX shop so I will provide an example configuration for NGINX.


server {
  listen 80;
  location = / {
    return 301;
  location /dist {
    autoindex on;
    alias /opt/nodejs/dist;

Checking the Sync Log

The cron job will automatically setup a log for you to view.

You can do this easily.

$ tail /var/log/nodejs-dist

Or watch it during a sync

$ tail -f /var/log/nodejs-dist

Example Mirror

As I was writing this I setup our mirror for anyone to use. Please see it here: