Github built it to solve their

background processing problems

after using various existing tools

https://github.com/blog/542-introducing-resque

Defining a worker is easy, it just needs two things:

• A queue defined
• A class method: perform

gem 'resque', require: 'resque/server'

class MyAwesomeWorker
  @queue= :bunny_rabbit
  
  def self.perform(bunny_name)
    sleep 30
    puts "#{bunny_name} is an awesome bunny!"
  end
end

env QUEUE='*' rake resque:work

(Built in Sinatra)

(Average Case)

• Invest for 30 years
• Rebalance investments after each year
• Simulate 205 times, incrementing the starting quarter
  • Start in Dec 1932, invest for 30 years
  • Start in Mar 1933, invest for 30 years
  • Start in Jun 1933, invest for 30 years
  • ... etc
• Save each simulation (called a "period")
• Collect the summary of each as a "Case Result"

Controller

def create
  client_case.input = filtered_params
  client_case.calculate
  # 3 minutes later...
  client.save
  # Render HTML
end

Timeout Error

Model

def calculate
  periods = 0..(periods_to_run-1)
  summaries = [ ]
  periods.each do |period|
    # Do a bunch of fun math
    years_in_period = calculate_period(input, period)
    self.save_period(period, years_in_period)
    summaries << years_in_period.last
  end
  self.summary = summarize(summaries)
end

Controller

def create
  client_case.input = filtered_params
  client_case.save
  Resque.enqueue(CaseWorker, client_case.id)
  # Render HTML
end

CaseWorker

class CaseWorker
  @queue = :cases

  def self.perform(case_id)
    client_case = Case.find(case_id)
    client_case.calculate
    client_case.save
  end
end

No Timeout!

(But it still takes 3 minutes to calculate)

Model

def calculate
  periods = 0..(periods_to_run-1)
  summaries = [ ]
  periods.each do |period|
    # Do a bunch of fun math
    years_in_period = calculate_period(input, period)
    self.save_period(period, years_in_period)
    summaries << years_in_period.last
  end
  self.summary = summarize(summaries)
end

Controller

def create
  client_case.input = filtered_params
  client_case.save
  (0..client_case.periods_to_run-1).each do |period|
    Resque.enqueue(PeriodWorker, case_id, period)
  end
  # Render HTML
end

CaseWorker

class CaseWorker
  @queue = :cases
  def self.perform(case_id)
    client_case = Case.find(case_id)
    client_case.summary = summarize(client_case.summaries)
    client_case.save
  end
end

PeriodWorker

class PeriodWorker
  @queue = :periods

  def self.perform(case_id, period)
    client_case = Case.find(case_id)
    years_in_period = calculate_period(input, period)
    client_case.save_period(period, years_in_period)
    client_case.summaries << years_in_period.last
    client_case.save
    if client_case.periods == client_case.periods_in_case
      Resque.enqueue(CaseWorker, case_id)
    end
  end 
end

No Timeout!

Plus, I can now throw more machines at it.

With 30 "workers" running, 25 seconds.

• We separated the calculation from our web app itself
• We broke down our code into smaller bits
• We ensured that our code could be run in parallel on multiple machines

Project:

https://github.com/resque/resque

Screencast:

http://railscasts.com/episodes/271-resque

sidekiq.org

• Uses threads instead of forks
• Easily extensible
• Has a lot of extra functionality out of the box
• Only loads a single copy of Rails in memory (if using Rails)
• Smaller footprint

• You code must be thread-safe
• Any gem you use must be thread-safe
• More processor intensive