forked from nabewise/bandicoot
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit bce25b5
Showing
12 changed files
with
358 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
*.gem | ||
.bundle | ||
Gemfile.lock | ||
pkg/* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
source "http://rubygems.org" | ||
|
||
# Specify your gem's dependencies in bandicoot.gemspec | ||
gemspec |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,87 @@ | ||
Bandicoot | ||
========= | ||
|
||
"I get knocked down, but I get up again. You're never gonna keep me down" -- | ||
Chumbawumba | ||
|
||
Wouldn't video games suck without levels and save points? I can't even tell | ||
you the number of times I've been playing Pokemon, forgetting to save | ||
regularly, only to get in a trainer battle right when my mom calls me down to | ||
dinner. I was angry for the rest of the day thinking about how all that | ||
effort getting the Magikarp enough XP to evolve was wasted in that one flick | ||
of the power switch. | ||
|
||
I feel similarly about my long running, data intensive tasks in Ruby. If I'm | ||
loading in a file with 200,000 records, some jackass has probably put a | ||
Windows-1252 character in record 195,095 or so. Restarting this process from | ||
the beginning would throw me into a rage. | ||
|
||
Bandicoot lets you set save points from which future computation can resume. | ||
Have a boss battle with some complicated data 5 hours into a process? No | ||
problem! Bandicoot will let you try and fix the bugs and start again. | ||
|
||
Warning | ||
------- | ||
|
||
Bandicoot is a work in progress. Bandicoot is not currently thread (or Fiber) | ||
safe. I'm working on a way to make it work while maintaining decent | ||
semantics, but it is not there yet. It may destroy your computer and force | ||
you to lightly blow on your cartridges. | ||
|
||
Usage | ||
----- | ||
|
||
Basic usage involves setting Bandicoot up and defining save_points | ||
|
||
Bandicoot.start do | ||
5.times do |i| | ||
the_meaning_of_life = 0 | ||
the_meaning_of_life += Bandicoot.save_point(["outer", i]) do | ||
result = 0 | ||
10.times do |j| | ||
result += Bandicoot.save_point(["inner", j]) do | ||
expensive_computation(j) | ||
end | ||
end | ||
result | ||
end | ||
the_meaning_of_life | ||
end | ||
end | ||
|
||
Save points have a key and take a block. The return value of | ||
Bandicoot.save_point is the return of the block. Save points can be nested | ||
arbitrarily and proper scoping will be maintained. | ||
|
||
This example will always run all of the code, writing its progress out to a | ||
save file. If the program crashes, you can continue from where it left off by | ||
changing the first line to | ||
|
||
Bandicoot.start(:continue => "path_to_save_file.save") do | ||
|
||
It will then load in that file and whenever a save_point is encountered, it | ||
will check whether that save_point was completed. If it has been, the return | ||
value it gave last time will be returned and the block will not be run. If it | ||
has not been, it will be run and upon successful completion that save point | ||
will be recorded as well. In this manner, you can keep trying until you get | ||
it right. | ||
|
||
Caveats and Considerations | ||
-------------------------- | ||
|
||
1) Bandicoot uses msgpack for serialization of keys and return values; | ||
therefore, keys and return values must be primitives where item == | ||
deserialize(serialize(item)). Practically, this means primitives and NO | ||
SYMBOLS. Arbitrarily nested arrays/hashes, numbers, and strings will work | ||
fine. | ||
|
||
2) The whole point of Bandicoot is to skip over already run blocks, so | ||
obviously any side effects in that block are not guaranteed to occur. | ||
Oftentimes this is fine and desired (e.g. inserting a row into a database), | ||
but if later code depends on side effects from earlier code, you may have a | ||
bad day. | ||
|
||
3) Your code could fail at any point in the save point block. Bandicoot will | ||
rerun the entire failing block, a bit of idempotence would probably be a good | ||
idea. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
require 'bundler' | ||
Bundler::GemHelper.install_tasks | ||
|
||
require 'rake/testtask' | ||
Rake::TestTask.new(:test) do |test| | ||
test.libs << 'lib' << 'test' | ||
test.pattern = 'test/**/*_test.rb' | ||
test.verbose = true | ||
end | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
# -*- encoding: utf-8 -*- | ||
$:.push File.expand_path("../lib", __FILE__) | ||
require "bandicoot/version" | ||
|
||
Gem::Specification.new do |s| | ||
s.name = "bandicoot" | ||
s.version = Bandicoot::VERSION | ||
s.platform = Gem::Platform::RUBY | ||
s.authors = ["Ben Hughes"] | ||
s.email = ["ben@nabewise.com"] | ||
s.homepage = "" | ||
s.summary = %q{Easy resume/save point lib} | ||
s.description = %q{Doesn't it suck when a long running task crashes? Wouldn't it be great to resume from more or less where you left off.} | ||
|
||
s.rubyforge_project = "bandicoot" | ||
|
||
s.files = `git ls-files`.split("\n") | ||
s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n") | ||
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) } | ||
s.require_paths = ["lib"] | ||
|
||
s.add_dependency "msgpack" | ||
|
||
s.add_development_dependency "mocha" | ||
s.add_development_dependency "fakefs" | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
require 'bandicoot/save_file' | ||
require 'bandicoot/save_point_hash' | ||
require 'bandicoot/context' | ||
|
||
module Bandicoot | ||
@@current = nil | ||
|
||
def self.start(opts={}) | ||
raise AlreadyStartedError if Bandicoot.current | ||
Bandicoot.push_context(opts) | ||
begin | ||
yield | ||
ensure | ||
Bandicoot.current.save_file.close | ||
Bandicoot.pop_context | ||
end | ||
end | ||
|
||
def self.current | ||
@@current | ||
end | ||
|
||
def self.save_point(key, &blk) | ||
raise NotStartedError unless Bandicoot.current | ||
Bandicoot.push_context(:key => key) | ||
begin | ||
Bandicoot.current.run(blk) | ||
ensure | ||
Bandicoot.pop_context | ||
end | ||
end | ||
|
||
def self.push_context(opts={}) | ||
@@current = Bandicoot::Context.new(opts.merge(:parent => Bandicoot.current)) | ||
end | ||
|
||
def self.pop_context | ||
@@current = Bandicoot.current.parent | ||
end | ||
|
||
class AlreadyStartedError < RuntimeError; end | ||
class NotStartedError < RuntimeError; end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
module Bandicoot | ||
class Context | ||
attr_reader :parent, :save_points | ||
|
||
def initialize(opts={}) | ||
@parent = opts[:parent] | ||
@key = opts[:key] | ||
|
||
# if this is the top level context | ||
if !parent | ||
@key ||= "__main__" | ||
@continuing = !!opts[:continue] | ||
if continuing? | ||
@save_file = SaveFile.continue(opts[:continue]) | ||
@save_points = @save_file.save_points[@key] || SavePointHash.new | ||
else | ||
@save_file = SaveFile.create(opts[:save_file] || default_save_filename) | ||
@save_points = SavePointHash.new | ||
end | ||
else | ||
@save_points = parent.save_points[@key] || SavePointHash.new | ||
end | ||
end | ||
|
||
def key | ||
@m_key ||= ((parent && parent.key) || []) + [@key] | ||
end | ||
|
||
def save_file | ||
@save_file ||= (parent && parent.save_file) | ||
end | ||
|
||
def continuing? | ||
@continuing ||= (parent && parent.continuing?) | ||
end | ||
|
||
# makes things a little prettier | ||
def save_point | ||
@save_points | ||
end | ||
|
||
def run(blk) | ||
if continuing? && save_point.completed? | ||
save_point.ret_val | ||
else | ||
finish! blk.call | ||
end | ||
end | ||
|
||
def finish!(retval=nil) | ||
save_file.write(key, retval) if save_file | ||
retval | ||
end | ||
|
||
protected | ||
def default_save_filename | ||
"bandicoot-#{Time.now.to_i}-#{rand(65535)}.save" | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
require 'msgpack' | ||
|
||
module Bandicoot | ||
class SaveFile | ||
attr_reader :file, :save_points | ||
|
||
def self.create(filename) | ||
new(File.open(filename, "w")) | ||
end | ||
|
||
def self.continue(filename) | ||
file = File.open(filename, "r+") | ||
save_points = read_save_points(file) | ||
file.seek(0, IO::SEEK_END) | ||
new(file, save_points) | ||
end | ||
|
||
def initialize(file, save_points=nil) | ||
@file = file | ||
@save_points = save_points | ||
end | ||
|
||
def write(key, retval) | ||
file.write([key, retval].to_msgpack) | ||
end | ||
|
||
def close | ||
file.close | ||
end | ||
|
||
protected | ||
def self.read_save_points(file) | ||
hash = SavePointHash.new | ||
MessagePack::Unpacker.new(file).each do |key, retval| | ||
c = hash | ||
key.each do |x| | ||
c[x] ||= SavePointHash.new | ||
c = c[x] | ||
end | ||
c.complete = true | ||
c.ret_val = retval | ||
end | ||
p hash | ||
hash | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
module Bandicoot | ||
class SavePointHash < Hash | ||
attr_accessor :ret_val, :complete | ||
|
||
def completed? | ||
!!complete | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
module Bandicoot | ||
VERSION = "0.0.1" | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
require File.join(File.dirname(__FILE__), "helper") | ||
|
||
class BandicootTest < Test::Unit::TestCase | ||
|
||
def teardown | ||
FakeFS::FileSystem.clear | ||
end | ||
|
||
def test_start | ||
started = false | ||
Bandicoot.start do | ||
started = true | ||
assert Bandicoot.current | ||
end | ||
assert started | ||
end | ||
|
||
def test_start_with_custom_path | ||
Bandicoot.start(:save_file => "blah") do | ||
1+1 | ||
end | ||
assert File.exists?("blah") | ||
end | ||
|
||
def test_start_with_continue | ||
File.open("blah", "w").close | ||
Bandicoot.start(:continue => false) do | ||
assert !Bandicoot.current.continuing? | ||
end | ||
|
||
Bandicoot.start(:continue => "blah") do | ||
assert Bandicoot.current.continuing? | ||
end | ||
end | ||
|
||
def test_save_point | ||
run_count = 0 | ||
Bandicoot.start(:save_file => "blah") do | ||
x = Bandicoot.save_point(:incr) do | ||
run_count += 1 | ||
42 | ||
end | ||
assert_equal 42, x | ||
end | ||
|
||
Bandicoot.start(:continue => "blah") do | ||
x = Bandicoot.save_point("incr") do | ||
run_count += 1 | ||
42 | ||
end | ||
assert_equal 42, x | ||
end | ||
|
||
assert_equal 1, run_count | ||
end | ||
end | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
require 'rubygems' | ||
require 'redgreen' | ||
require 'bundler/setup' | ||
|
||
require 'bandicoot' | ||
|
||
require 'test/unit' | ||
require 'fakefs' |