shredder

Shredder Build Status

Remote video encoding platform

Installation

$ git clone [email protected]:eSited/shredder.git
$ cd shredder
$ npm install

Starting

First, setup a local config file like so.

'use strict';
module.exports = {
  mysql: {
    username: 'foo',
  },
  master: {
    enabled: true,
    username: 'myusername',
    password: 'mypassword'
  },
  worker: {
    enabled: true,
    username: 'myusername',
    password: 'mypassword'
  }
}

Of course any additional overrides can be added to the config here.

Second, start the system.

$ node app

Testing

The default test suite can be ran using npm

$ npm test

For development a more interactive test method might be best

$ mocha -R spec --watch

Debug Logging

All debug logging now uses the https://github.com/visionmedia/debug package.

The following can be used to see all messages (typically used in dev)

$ DEBUG=shredder* node app

From there follow the debug filtering rules defined https://github.com/visionmedia/debug#wildcards

Usage

Example Shredder API interaction using the express webserver.

'use strict';
//this example uses promises. its just easier :)
var P = require('bluebird')
//we use request to send the inital call
var request = require('request')

//we must setup a webserver to receive updates
var bodyParser = require('bodyParser')
var express = require('express')
var http = require('http')
var app = express()
var server = http.createServer(3000,'127.0.0.1',app)

//setup middleware to handle JSONP
app.use(bodyParser.json())

//setup our route to handle the update
app.post('/shredder/update',function(req,res){
  console.log('got a shredder update',req.body)
})

//make some promises
P.promisifyAll(request)
P.promisifyAll(server)

//start our server
var session = {}
var jobHandle = ''
server.startAsync()
  .then(function(){
    //first login
    return request.postAsync({
      url: 'http://localhost:5980/user/login',
      json: {
        username: 'myusername',
        password: 'mypassword'
      }
    })
  })
  .spread(function(res,body){
    if(body && body.error) throw new Error(body.error)
    //save the session
    session = body.session
    //make the actual job creation request
    return request.postAsync({
      url: 'http://localhost:5980/job/create',
      json: {
        description: {
          callback: {
            request: { url: 'http://localhost:3000/shredder/update' }
          },
          resource: [
            {
              name: 'thumbnail',
              request: { url: 'http://localhost/foo.png' }
            },
            {
              name: 'video',
              request: { url: 'http://localhost/bar.mp4' }
            }
          ],
          save: ['thumbnail','video']
        }
      }
    })
    .spread(function(res,body){
      if(body && body.error) throw new Error(body.error)
      //save the handle maybe do some db work before starting the job
      jobHandle = body.handle
      //start the job
      return request.postAsync({
        url: 'http://localhost:5980/job/start',
        json: {
          handle: jobHandle
        }
      })
    })
    .spread(function(req,body){
      if(body && body.error) throw new Error(body.error)
      console.log(jobHandle + ' job started!')
    })

API Reference

Shredder exposes itself through an HTTPs JSONP API.

/user/login

Request

  • username Username
  • password Password

Response

  • success
  • session The resulting session object

/user/logout

Session required

Response

  • success

/job/create

  • description required Object or JSON see below
  • priority optional Number (closer to 0 higher priority)
  • handle optional If a handle is desired place it here. It must be 12 alpha numeric characters and must not already be in use.

/job/start

  • handle Job handle to start

Response

  • success

/job/detail

  • handle

Response

Job details object

/job/list

/job/update

Update a job. This can only be done before it has been started.

Request

  • handle
  • priority
  • description

/job/remove

Remove a job. This can only be done before it has been started.

Request

  • handle

Job Description

A job description can be JSON or an object. It will be stored as JSON internally until the job is processed by a worker.

{
  //setup callback handler that will notify of completion/abort/error
  callback: {
    //these options are passed to node-request
    request: {
      url: 'http://localhost/myapp/jobUpdate'
    }
  }
  //options pertaining to the source file
  resource: [
    {
      //the name is used to write the resource file in the job folder
      name: 'video.mp4',
      //passed to node-request
      request: {
        url: 'http://localhost/tmp/24zsf'
      }
    },
    //if we want to watermark our video lets grab that image
    {
      name: 'watermark.png',
      request: {
        url: 'http://localhost/watermark.png'
      }
    }
    //to make a dubbed version we want to download our dubbed audio track
    {
      name: 'dubbed-audio.mp3',
      request: {
        url: 'http://localhost/myvideo-dubbed.mp3'
      }
    }
    //it may also be useful to use the chain to extract data from intermediate pages
    {
      name: 'protected-video-with-regexp.mp4',
      request: [
        //first make the request to extract content
        {
          //passed to node-request
          request: {
            'http://foo/embed'
          },
          parse: {
            //notice here that 'file' is the name of the parameter the result will be assigned too
            file: '(\w+)' //any regular expression is valid but must only return one selection
          }
        },
        //second we use the content we extracted
        {
          request: {
            url: '#{file}'
          }
        }
      ]
    }
  ],
  //options that are used to control the augmentation of resources
  augment: [
    // programs are ran in the same folder as the job so paths are relative
    {
      program: 'ffmpeg',
      args: ['-i','video.mp4','-metadata','title="Video 1"','-s','hd1080','-y','hdvideo.mp4']
    },
    {
      program: 'mp4box',
      command: ['-inter','1250','-hint','-isma','-noprog','hdvideo.mp4']
    }
  ]
}

Changelog

1.1.0

  • Completion of main features
  • Bring sdk up to 1.1.3
  • Fix dispatch system
  • Add notification callbacks
  • Implement status lookup
  • Implement content existence
  • Implement content download

1.0.0

  • Overhauled to be a self contained cluster
  • API re-imagined to simpler to use
  • Extracted from the OOSE package
Languagejavascript
Version1.1.0
Git URLhttps://github.com/nullivex/shredder
LicenseMIT
DescriptionShredder Cloud Compute System
Keywords