Yehuda Katz - Blog Entries

Tokaido: My Hopes and Dreams

A few weeks ago, I started a kickstarter project to fund work on a project to make a long-term, sustainable binary build of Ruby. The outpouring of support was great, and I have far exceeded my original funding goal. First, I’d like to thank everyone in the community who contributed large and small donations. This kickstarter couldn’t have been as successful as it has been without the hundreds (650 at latest count!) of individual donations by interested Rubyists.
In this post, I want to talk about what my goals are for this project, and why I think it will be a tool that everyone, myself included, will use.
<h2>What is Tokaido</h2>
The name “Tokaido” (東海道 in Japanese) comes from the <a href="http://en.wikipedia.org/wiki/T%C5%8Dkaid%C5%8D_Shinkansen">Tōkaidō Shinkansen</a> bullet train line in Japan.
<h3>Precompiled, Static Ruby</h3>
At its core, Tokaido is a binary distribution of Ruby without any external dependencies on your system. This means that Ruby itself, as well as all of the compiled elements of the standard library, will come in a self-contained directory with no additional requirements.
The binary will also have no hardcoded paths, so it will be possible to move the directory anywhere on the file system and have it continue to work as long as its <code>bin</code> directory is on the <code>$PATH</code>. This is an important goal, as Tokaido will ship as a downloadable <code>.app</code> which should work wherever it is dropped on the file system.
<h3>Precompiled Binary Gems</h3>
Tokaido will come with all of the necessary gems to use Rails and common Rails extensions. Because some of these gems have external dependencies (for example, <code>nokogori</code> depends on <code>libxml2</code>), Tokaido will come with precompiled versions of these gems built for OSX that do not depend on external libraries (they will be statically compiled, not dynamically linked).
As part of this project, I plan to work with people who ship common native extensions to help them build and ship binary versions of their gems for OSX without external dependencies. Luis Lavena has been doing amazing work with <a href="https://github.com/luislavena/rake-compiler">rake-compiler</a> to make this process easy on gem developers, and Wayne E. Seguin and Michał Papis have been doing great work with <a href="https://github.com/sm/sm">sm</a>, which makes it easy to precompile the needed dependencies for inclusion in the precompiled gems. These tools will be essential in the effort to make dependency-free precompiled gems a standard part of the OSX Ruby ecosystem.
I anticipate that gem authors will, in general, start distributing precompiled binary versions of their gems. If, by the time I ship the first version of Tokaido, some important gems do not yet ship as precompiled binaries, Tokaido will bootstrap the process by including the binaries.
<h3>Terminal-Based Workflow</h3>
Tokaido does not aim to replace the Terminal as the main way to work with a Rails app. It will ship an isolated environment with no external dependencies designed for use in the Terminal. The application UI will supplement, rather than replace, the normal application workflow. This is a crucial part of the overall goal to make Tokaido an excellent tool for both experienced Rails developers, intermediate Rails developers, and totally new Rails developers.
Because many gems come with executables, and Tokaido couldn’t abstract every possible executable even if it wanted to, it is essential that new developers get used to using the Terminal as early as possible in their Rails experience, but in a way that minimizes unnecessary errors.
<h3>Extras: Code and App Health</h3>
A number of really great tools exist to help Rails applications remain healthy:
<ul>
<li><code>bundle outdated</code> lets you know when new versions of gems are available</li>
<li><code><a href="https://github.com/railsbp/rails_best_practices">rails_best_practices</a></code> helps find common mistakes in Rails applications</li>
<li><code>reek</code>, <code>flog</code>, <code>flay</code>, <code>roodi</code> and other metrics tools help identify Ruby smells and prioritizes areas where you can reduce your technical debt</li>
<li><code>simplecov</code> does coverage analysis on your test suite to identify areas lacking good coverage</li>
<li><code>bullet</code> identifies N+1 queries and unused eager loading</li>
<li><code>ActiveSupport::Notifications</code> provides introspection information about Rails applications</li>
</ul>
The Tokaido UI will attempt to centralize this useful information into a health check, and help you drill in if you want to do more with the information. Because they are so useful, it will encourage you to use these tools in new application, and provide immediate rewards if you do.
<h3>Extras: yourapp.dev</h3>
The <a href="http://pow.cx/">pow</a> dev server and <a href="https://github.com/Fingertips/passengerpane">PassengerPane</a> make it possible to avoid needing to manually load a Rails server and gives you a friendly alias for your application at <code>yourapp.dev</code> instead of having to type in a port on <code>localhost</code>.
Tokaido will come with integrated support for this, so every app you manage through Tokaido will have a aliased development server out of the box.
<h3>Extras: rails:// (or ruby://) Protocol Handler</h3>
Tokaido will come with a protocol handler that will allow <code>.gem</code> files hosted on rubygems.org or other locations to be added to an app managed by Tokaido. It may also allow web sites to host installation templates that could execute against a maintained application. This will require coordination with rubygems.org, and the specifics of it may change over time, but the basic idea is to enable communication between web sites and the Tokaido app.
This idea came from a number of community members, and the specifics (especially around security and the protocol specification) definitely need fleshing out, but it seems extremely promising.
<h3>Extras: Ruby Toolbox Integration?</h3>
Ruby Toolbox has emerged as an amazing directory of Ruby libraries, categorized by type. It also has up-to-date information on Github activity and other useful indicators when evaluating tools. Several people have asked for integration with Ruby Toolbox, and assuming the author is willing, Tokaido will make it easy to use the information in Ruby Toolbox to bootstrap an app and add new functionality as your app grows.
Finding the right gem for the job is easy if you know what you’re looking for, but even experienced developers find the Ruby Toolbox useful when researching tools in an unfamiliar area.
<h2>Goals of Tokaido</h2>
<h3>Eliminate Failure Scenarios</h3>
The primary goal of Tokaido is to build a distribution of Ruby that eliminates the main failure scenarios that people encounter when trying to install Ruby. In general, things fail because of conflicts with the existing system environment (or the user’s environment). These failures do not happen to everyone, but when they happen, they are extremely difficult to recover from. This has happened to me personally and to people I have tried to get started with Ruby.
This sort of thing does not necessarily happen on every installation, but once you start going down the rabbit hole, it can be very difficult to find your way out. The environment difficulties can be caused by anything from an old MacPorts installation to a mistaken attempt to install something to the system once upon a time to something failing during a previous step in the installation process.
It also may not be enough to install a precompiled Ruby into the system, because the system environment itself may be corrupted with things like erroneous executables on the <code>$PATH</code> or bad dynamic libraries that get linked during native compilation. Also, later installations may do damage to the system-installed Ruby. Tokaido is a standalone environment that is loaded on top of an existing session, and therefore minimizes the possible damage that load order or subsequent installs can do.
<h3>Precompile Everything</h3>
In order to eliminate a certain class of failure scenarios, Tokaido will ship with a precompiled Ruby, which will eliminate the possibility of compilation errors when installing Ruby. This precompiled Ruby will also come with all of the dependencies it needs, like <code>zlib</code>, <code>yaml</code> and others, instead of allowing the system to try to find them at runtime. This will eliminate the possibility that changes to the system environment will cause a normally working version of Ruby to fail in some scenarios.
As above, Tokaido will also use this technique for commonly used native gems. For example, Tokaido will ship with a precompiled version of Nokogiri that comes with <code>libxml</code>, instead of relying on the system’s copy of <code>libxml</code> (incidentally, the system’s <code>libxml</code> has occasionally been subtly broken, necessitating installation via homebrew). I expect that this will happen because gem authors will start shipping precompiled versions of their gems for OSX. If there are still a few common gems straggling by the time Tokaido ships, we’ll bootstrap the process by shipping with binary gems we compile ourselves.
<h3>Use Tokaido Myself</h3>
Since Tokaido does not fundamentally alter a developer’s relationship with Ruby, I expect to be able to start using it for day-to-day Rails development. Some of the additional extras, like app health and built-in server support, are things I already do manually and will enjoy having as part of a larger tool. I’m also extremely interested in ideas for other extras that can add value for experienced developers without altering how people work with Ruby today.
<h3>Integration Testing</h3>
One of the coolest, unheralded things about the Rails project is the <a href="http://intertwingly.net/projects/dashboard.html">integration suite</a> for Rails put together by Sam Ruby. It essentially transcribes the Agile Web Development With Rails book into a series of very complete integration tests for Rails. When refactoring Rails for Rails 3, this suite was extremely useful, and helped keep the number of unintentional changes to Rails to a tiny number. It also kept us honest about intentional changes.
This suite is perfect for Tokaido, as it tests every aspect of Rails, which is itself touches a wide swath of Ruby itself. To start, Tokaido will use this suite for integration testing.
<h3>Collaboration</h3>
There are several other projects, notably <code>rvm</code>, trying to solve problems in a similar space. Especially when it comes to precompilation, there is no reason not to work together on the core infrastructure that powers these efforts. I have already started to work with Michał Papis, who also works on <code>sm</code> and <code>rvm</code> on shared work that can be useful for both projects. He has been teaching me all about the work that folks associated with rvm have done to simplify things like downloading and compiling <code>libz.a</code>, a prerequisite for precompiled Rubies.
I will also work with authors of native gems to start shipping precompiled binary gems for OSX. I have already started working with Aaron Patterson, who works on the <code>sqlite</code> and <code>nokogiri</code> gems, and have started reaching out to several others.
I am very interested in working together with anyone working on a similar area so that the tools we are all working on can be shared between projects. This is a core mission of the Tokaido project, and something that the extra funding I got will allow me to prioritize.
<h3>Migration to “System”</h3>
Also through Michał Papis, I learned about an awesome new (still experimental) feature in <code>rvm</code> called mount that will mount an external Ruby into the rvm system. I will make sure that the <code>Tokaido</code> ruby can be added to an <code>rvm</code> installation. This will mean that if someone wants to migrate from Tokaido to a more advanced rvm setup, they can take their Ruby environment with them with no trouble.
I would be open to other approaches to migrating a Tokaido environment to the system as well. The goal would be to seamlessly migrate an environment to the system without introducing opportunities for failure during that process.
<h2>Looking Forward</h2>
I’m really excited to the work I’ve been doing to prepare for this project, and looking forward to shipping a great environment for OSX Ruby users. I have also really enjoyed reaching out to others working in similar areas and the prospect of collaborating with so many smart people on a shared goal.
Thanks so much!
<script type="text/javascript">
addthis_url = 'http%3A%2F%2Fyehudakatz.com%2F2012%2F04%2F13%2Ftokaido-my-hopes-and-dreams%2F';
addthis_title = 'Tokaido%3A+My+Hopes+and+Dreams';
addthis_pub = '';
</script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12" ></script>
<div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/KatzGotYourTongue?a=nvgFbXOt8tQ:Pa8clZyMJio:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/KatzGotYourTongue?d=yIl2AUoC8zA" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/KatzGotYourTongue/~4/nvgFbXOt8tQ" height="1" width="1"/>

JavaScript Needs Blocks

While reading Hacker News posts about JavaScript, I often come across the misconception that Ruby’s blocks are essentially equivalent to JavaScript’s “first class functions”. Because the ability to pass functions around, especially when you can create them anonymously, is extremely powerful, the fact that both JavaScript and Ruby have a mechanism to do so makes it natural to assume equivalence.
In fact, when people talk about why Ruby’s blocks are different from Python‘s functions, they usually talk about anonymity, something that Ruby and JavaScript share, but Python does not have. At first glance, a Ruby block is an “anonymous function” (or colloquially, a “closure”) just as a JavaScript function is one.
This impression, which I admittedly shared in my early days as a Ruby/JavaScript developer, misses an important subtlety that turns out to have large implications. This subtlety is often referred to as “Tennent’s Correspondence Principle”. In short, Tennent’s Correspondence Principle says:
<blockquote style="border-left: 3px solid #999; padding-left: 1em; margin-left: 0">
“For a given expression <code>expr</code>, <code>lambda expr</code> should be equivalent.”
</blockquote>
This is also known as the principle of abstraction, because it means that it is easy to refactor common code into methods that take a block. For instance, consider the common case of file resource management. Imagine that the block form of <code>File.open</code> didn’t exist in Ruby, and you saw a lot of the following in your code:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">begin
f = File.open(filename, "r")
# do something with f
ensure
f.close
end</pre></div></div>

In general, when you see some code that has the same beginning and end, but a different middle, it is natural to refactor it into a method that takes a block. You would write a method like this:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">def read_file(filename)
f = File.open(filename, "r")
yield f
ensure
f.close
end</pre></div></div>

And you’d refactor instances of the pattern in your code with:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">read_file(filename) do |f|
# do something with f
end</pre></div></div>

In order for this strategy to work, it’s important that the code inside the block look the same after refactoring as before. We can restate the correspondence principle in this case as:
<blockquote style="border-left: 3px solid #999; padding-left: 1em; margin-left: 0">

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;"># do something with f</pre></div></div>

should be equivalent to:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">do
# do something with
end</pre></div></div>

</blockquote>
At first glance, it looks like this is true in Ruby and JavaScript. For instance, let’s say that what you’re doing with the file is printing its mtime. You can easily refactor the equivalent in JavaScript:

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">try {
// imaginary JS file API
var f = File.open(filename, "r");
sys.print(f.mtime);
} finally {
f.close();
}</pre></div></div>

Into this:

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">read_file(function(f) {
sys.print(f.mtime);
});</pre></div></div>

In fact, cases like this, which are in fact quite elegant, give people the mistaken impression that Ruby and JavaScript have a roughly equivalent ability to refactor common functionality into anonymous functions.
However, consider a slightly more complicated example, first in Ruby. We’ll write a simple class that calculates a File’s mtime and retrieves its body:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">class FileInfo
def initialize(filename)
@name = filename
end
 
# calculate the File's +mtime+
def mtime
f = File.open(@name, "r")
mtime = mtime_for(f)
return "too old" if mtime < (Time.now - 1000)
puts "recent!"
mtime
ensure
f.close
end
 
# retrieve that file's +body+
def body
f = File.open(@name, "r")
f.read
ensure
f.close
end
 
# a helper method to retrieve the mtime of a file
def mtime_for(f)
File.mtime(f)
end
end</pre></div></div>

We can easily refactor this code using blocks:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">class FileInfo
def initialize(filename)
@name = filename
end
 
# refactor the common file management code into a method
# that takes a block
def mtime
with_file do |f|
mtime = mtime_for(f)
return "too old" if mtime < (Time.now - 1000)
puts "recent!"
mtime
end
end
 
def body
with_file { |f| f.read }
end
 
def mtime_for(f)
File.mtime(f)
end
 
private
# this method opens a file, calls a block with it, and
# ensures that the file is closed once the block has
# finished executing.
def with_file
f = File.open(@name, "r")
yield f
ensure
f.close
end
end</pre></div></div>

Again, the important thing to note here is that we could move the code into a block without changing it. Unfortunately, this same case does not work in JavaScript. Let’s first write the equivalent <code>FileInfo</code> class in JavaScript.

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">// constructor for the FileInfo class
FileInfo = function(filename) {
this.name = filename;
};
 
FileInfo.prototype = {
// retrieve the file's mtime
mtime: function() {
try {
var f = File.open(this.name, "r");
var mtime = this.mtimeFor(f);
if (mtime < new Date() - 1000) {
return "too old";
}
sys.print(mtime);
} finally {
f.close();
}
},
 
// retrieve the file's body
body: function() {
try {
var f = File.open(this.name, "r");
return f.read();
} finally {
f.close();
}
},
 
// a helper method to retrieve the mtime of a file
mtimeFor: function(f) {
return File.mtime(f);
}
};</pre></div></div>

If we try to convert the repeated code into a method that takes a function, the <code>mtime</code> method will look something like:

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">function() {
// refactor the common file management code into a method
// that takes a block
this.withFile(function(f) {
var mtime = this.mtimeFor(f);
if (mtime < new Date() - 1000) {
return "too old";
}
sys.print(mtime);
});
}</pre></div></div>

There are two very common problems here. First, <code>this</code> has changed contexts. We can fix this by allowing a binding as a second parameter, but it means that we need to make sure that every time we refactor to a lambda we make sure to accept a binding parameter and pass it in. The <code>var self = this</code> pattern emerged in JavaScript primarily because of the lack of correspondence.
This is annoying, but not deadly. More problematic is the fact that <code>return</code> has changed meaning. Instead of returning from the outer function, it returns from the inner one.
This is the right time for JavaScript lovers (and I write this as a sometimes JavaScript lover myself) to argue that <code>return</code> behaves exactly as intended, and this behavior is simpler and more elegant than the Ruby behavior. That may be true, but it doesn’t alter the fact that this behavior breaks the correspondence principle, with very real consequences.
Instead of effortlessly refactoring code with the same start and end into a function taking a function, JavaScript library authors need to consider the fact that consumers of their APIs will often need to perform some gymnastics when dealing with nested functions. In my experience as an author and consumer of JavaScript libraries, this leads to many cases where it’s just too much bother to provide a nice block-based API.
In order to have a language with <code>return</code> (and possibly <code>super</code> and other similar keywords) that satisfies the correspondence principle, the language must, like Ruby and Smalltalk before it, have a function lambda and a block lambda. Keywords like <code>return</code> always return from the function lambda, even inside of block lambdas nested inside. At first glance, this appears a bit inelegant, and language partisans often accuse Ruby of unnecessarily having two types of “callables”, in my experience as an author of large libraries in both Ruby and JavaScript, it results in more elegant abstractions in the end.
<h3>Iterators and Callbacks</h3>
It’s worth noting that block lambdas only make sense for functions that take functions and invoke them immediately. In this context, keywords like <code>return</code>, <code>super</code> and Ruby’s <code>yield</code> make sense. These cases include iterators, mutex synchronization and resource management (like the block form of <code>File.open</code>).
In contrast, when functions are used as callbacks, those keywords no longer make sense. What does it mean to return from a function that has already returned? In these cases, typically involving callbacks, function lambdas make a lot of sense. In my view, this explains why JavaScript feels so elegant for evented code that involves a lot of callbacks, but somewhat clunky for the iterator case, and Ruby feels so elegant for the iterator case and somewhat more clunky for the evented case. In Ruby’s case, (again in my opinion), this clunkiness is more from the massively pervasive use of blocks for synchronous code than a real deficiency in its structures.
Because of these concerns, the ECMA working group responsible for ECMAScript, TC39, <a href="http://wiki.ecmascript.org/doku.php?id=strawman:block_lambda_revival">is considering adding block lambdas</a> to the language. This would mean that the above example could be refactored to:

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;">FileInfo = function(name) {
this.name = name;
};
 
FileInfo.prototype = {
mtime: function() {
// use the proposed block syntax, `{ |args| }`.
this.withFile { |f|
// in block lambdas, +this+ is unchanged
var mtime = this.mtimeFor(f);
if (mtime < new Date() - 1000) {
// block lambdas return from their nearest function
return "too old";
}
sys.print(mtime);
}
},
 
body: function() {
this.withFile { |f| f.read(); }
},
 
mtimeFor: function(f) {
return File.mtime(f);
},
 
withFile: function(block) {
try {
var f = File.open(this.name, "r");
block(f);
} finally {
f.close();
}
}
};</pre></div></div>

Note that a parallel proposal, which replaces function-scoped <code>var</code> with block-scoped <code>let</code>, will almost certainly be accepted by TC39, which would slightly, but not substantively, change this example. Also note block lambdas automatically return their last statement.
Our experience with Smalltalk and Ruby show that people do not need to understand the SCARY correspondence principle for a language that satisfies it to yield the desired results. I love the fact that the concept of “iterator” is not built into the language, but is instead a consequence of natural block semantics. This gives Ruby a rich, broadly useful set of built-in iterators, and language users commonly build custom ones. As a JavaScript practitioner, I often run into situations where using a for loop is significantly more straight-forward than using <code>forEach</code>, always because of the lack of correspondence between the code inside a built-in for loop and the code inside the function passed to <code>forEach</code>.
For the reasons described above, I strongly approve of <a href="http://wiki.ecmascript.org/doku.php?id=strawman:block_lambda_revival">the block lambda proposal</a> and hope it is adopted.
<script type="text/javascript">
addthis_url = 'http%3A%2F%2Fyehudakatz.com%2F2012%2F01%2F10%2Fjavascript-needs-blocks%2F';
addthis_title = 'JavaScript+Needs+Blocks';
addthis_pub = '';
</script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12" ></script>
<div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/KatzGotYourTongue?a=v1ctIJnejg4:YSxx--Twyr4:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/KatzGotYourTongue?d=yIl2AUoC8zA" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/KatzGotYourTongue/~4/v1ctIJnejg4" height="1" width="1"/>

Amber.js (formerly SproutCore 2.0) is now Ember.js

After we announced Amber.js last week, a number of people brought Amber Smalltalk, a Smalltalk implementation written in JavaScript, to our attention. After some communication with the folks behind Amber Smalltalk, we started <a href="http://news.ycombinator.com/item?id=3335916">a discussion on Hacker News</a> about what we should do.
Most people told us to stick with Amber.js, but a sizable minority told us to come up with a different name. After thinking about it, we didn’t feel good about the conflict and decided to choose a new name.
Henceforth, the project formerly known as SproutCore 2.0 will be known as Ember.js. Our new website is up at <a href="http://www.emberjs.com">http://www.emberjs.com</a>
(and yes, we know this is pretty ridiculous)
<script type="text/javascript">
addthis_url = 'http%3A%2F%2Fyehudakatz.com%2F2011%2F12%2F12%2Famber-js-formerly-sproutcore-2-0-is-now-ember-js%2F';
addthis_title = 'Amber.js+%28formerly+SproutCore+2.0%29+is+now+Ember.js';
addthis_pub = '';
</script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12" ></script>
<div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/KatzGotYourTongue?a=5_ye09WM9kU:-rXADmBx3qk:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/KatzGotYourTongue?d=yIl2AUoC8zA" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/KatzGotYourTongue/~4/5_ye09WM9kU" height="1" width="1"/>

Announcing Amber.js

A little over a year ago, I got my first serious glimpse at SproutCore, the JavaScript framework Apple used to build MobileMe (now iCloud). At the time, I had worked extensively with jQuery and Rails on client-side projects, and I had never found the arguments for the “solutions for big apps” very compelling. At the time, most of the arguments (at least within the jQuery community) focused on bringing more object orientation to JavaScript, but I never felt that they offered the layers of abstraction you really want to manage complexity.
When I first started to play with SproutCore, I realized that the bindings and computed properties were what gave it its real power. Bindings and computed properties provide a clean mechanism for building the layers of abstractions that improve the structure of large applications.
But even before I got involved in SproutCore, I had an epiphany one day when playing with Mustache.js. Because Mustache.js was a declarative way of describing a translation from a piece of JSON to HTML, it seemed to me that there was enough information in the template to also update the template when the underlying data changed. Unfortunately, Mustache.js itself lacked the power to implement this idea, and I was still lacking a robust enough observer library.
Not wanting to build an observer library in isolation (and believing that jQuery’s data support would work in a pinch), I started working on the first problem: building a template engine powerful enough to build automatically updating templates. The kernel of the idea for Handlebars (helpers and block helpers as the core primitives) came out of a discussion with Carl Lerche back when we were still at Engine Yard, and I got to work.
When I met SproutCore, I realized that it provided a more powerful observer library than anything I was considering at the time for the data-binding aspect of Handlebars, and that SproutCore’s biggest weakness was the lack of a good templating solution in its view layer. I also rapidly became convinced that bindings and computed properties were a significantly better abstraction, and allowed for hiding much more complexity, than manually binding observers.
After some months of retooling SproutCore with Tom Dale to take advantage of an auto-updating templating solution that fit cleanly into SproutCore’s binding model, we reached a crossroads. SproutCore itself was built from the ground up to provide a desktop-like experience on desktop browsers, and our ultimate plan had started to diverge from the widget-centric focus of many existing users of SproutCore. After a lot of soul-searching, we decided to start from scratch with SproutCore 2.0, taking with us the best, core ideas of SproutCore, but leaving the large, somewhat sprawling codebase behind.
Since early this year, we have worked with several large companies, including ZenDesk, BazaarVoice and LivingSocial, to iterate on the core ideas that we started from to build a powerful framework for building ambitious applications.
Throughout this time, though, we became increasingly convinced that calling what we were building “SproutCore 2.0″ was causing a lot of confusion, because SproutCore 1.x was primarily a native-style widget library, while SproutCore 2.0 was a framework for building web-based applications using HTML and CSS for the presentation layer. This lack of overlap causes serious confusion in the IRC room, mailing list, blog, when searching on Google, etc.
To clear things up, we have decided to name the SproutCore-inspired framework we have been building (so far called “SproutCore 2.0″) “Amber.js”. Amber brings a proven MVC architecture to web applications, as well as features that eliminate common boilerplate. If you played with SproutCore and liked the concepts but felt like it was too heavy, give Amber a try. And if you’re a Backbone fan, I think you’ll love how little code you need to write with Amber.
In the next few days, we’ll be launching a new website with examples, documentation, and download links. Stay tuned for further updates soon.
UPDATE: The code for Amber.js is still, as of December 8, hosted at <a href="http://github.com/sproutcore/sproutcore20">the SproutCore organization</a>. It will be moved and re-namespaced within a few days.
<script type="text/javascript">
addthis_url = 'http%3A%2F%2Fyehudakatz.com%2F2011%2F12%2F08%2Fannouncing-amber-js%2F';
addthis_title = 'Announcing+Amber.js';
addthis_pub = '';
</script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12" ></script>
<div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/KatzGotYourTongue?a=kcjvX4gkC1Q:0UOkl6M4RB8:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/KatzGotYourTongue?d=yIl2AUoC8zA" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/KatzGotYourTongue/~4/kcjvX4gkC1Q" height="1" width="1"/>

How to Marshal Procs Using Rubinius

The primary reason I enjoy working with Rubinius is that it exposes, to Ruby, much of the internal machinery that controls the runtime semantics of the language. Further, it exposes that machinery primarily in order to enable user-facing semantics that are typically implemented in the host language (C for MRI, C and C++ for MacRuby, Java for JRuby) to be implemented in Ruby itself.
There is, of course, quite a bit of low-level functionality in Rubinius implemented in C++, but a surprising number of things are implemented in pure Ruby.
One example is the <code>Binding</code> object. To create a new binding in Rubinius, you call <code>Binding.setup</code>:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">def self.setup(variables, code, static_scope, recv=nil)
bind = allocate()
 
bind.self = recv || variables.self
bind.variables = variables
bind.code = code
bind.static_scope = static_scope
return bind
end</pre></div></div>

This method takes a number of more primitive constructs, which I will explain as this article progresses, but we can describe the constructs that make up the high-level Ruby <code>Binding</code> in pure Ruby.
In fact, Rubinius implements <code>Kernel#binding</code> itself in terms of <code>Binding.setup</code>.

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">def binding
return Binding.setup(
Rubinius::VariableScope.of_sender,
Rubinius::CompiledMethod.of_sender,
Rubinius::StaticScope.of_sender,
self)
end</pre></div></div>

Yes, you’re reading that right. Rubinius exposes the ability to extract the constructs that make up a binding, one at a time, from a caller’s scope. And this is not just a hack (like Binding.of_caller for a short time in MRI). It’s core to how Rubinius manages <code>eval</code>, which of course makes heavy use of bindings.
<h2>Marshalling Procs</h2>
For a while, I have wanted the ability to <code>Marshal.dump</code> a proc in Ruby. MRI has historically disallowed it, but there’s nothing conceptually impossible about it. A proc itself is a blob of executable code, a local variable scope (which is just a bunch of pointers to other objects), and a constant lookup scope. Rubinius exposes each of these constructs to Ruby, so Marshaling a proc simply means figuring out how to Marshal each of these constructs.
Let’s take a quick detour to learn about the constructs in question.
<h2>Rubinius::StaticScope</h2>
Rubinius represents Ruby’s constant lookup scope as a <code>Rubinius::StaticScope</code> object. Perhaps the easiest way to understand it would be to look at Ruby’s built-in <code>Module.nesting</code> function.

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">module Foo
p Module.nesting
 
module Bar
p Module.nesting
end
end
 
module Foo::Bar
p Module.nesting
end
 
# Output:
# [Foo]
# [Foo::Bar, Foo]
# [Foo::Bar]</pre></div></div>

Every execution context in Rubinius has a <code>Rubinius::StaticScope</code>, which may optionally have a parent scope. In general, the top static scope (the static scope with no parent) in any execution context is <code>Object</code>.
Because Rubinius allows us to get the static scope of a calling method, we can implement <code>Module.nesting</code> in Rubinius:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">def nesting
scope = Rubinius::StaticScope.of_sender
nesting = []
while scope and scope.module != Object
nesting << scope.module
scope = scope.parent
end
nesting
end</pre></div></div>

A static scope also has an addition property called <code>current_module</code>, which is used during <code>class_eval</code> to define which module the runtime should add new methods to.
Adding <code>Marshal.dump</code> support to a static scope is therefore quite easy:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">class Rubinius::StaticScope
def marshal_dump
[@module, @current_module, @parent]
end
 
def marshal_load(array)
@module, @current_module, @parent = array
end
end</pre></div></div>

These three instance variables are defined as Rubinius slots, which means that they are fully accessible to Ruby as instance variables, but don’t show up in the <code>instance_variables</code> list. As a result, we need to explicitly dump the instance variables that we care about and reload them later.
<h2>Rubinius::CompiledMethod</h2>
A compiled method holds the information necessary to execute a blob of Ruby code. Some important parts of a compiled method are its instruction sequence (a list of the compiled instructions for the code), a list of any literals it has access to, names of local variables, its method signature, and a number of other important characteristics.
It’s actually quite a complex structure, but Rubinius has already knows how to convert an in-memory <code>CompiledMethod</code> into a String, as it dumps compiled Ruby files into compiled files as part of its normal operation. There is one small caveat: this String form that Rubinius uses for its compiled method does not include its static scope, so we will need to include the static scope separately in the marshaled form. Since we already told Rubinius how to marshal a static scope, this is easy.

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">class Rubinius::CompiledMethod
def _dump(depth)
Marshal.dump([@scope, Rubinius::CompiledFile::Marshal.new.marshal(self)])
end
 
def self._load(string)
scope, dump = Marshal.load(string)
cm = Rubinius::CompiledFile::Marshal.new.unmarshal(dump)
cm.scope = scope
cm
end
end</pre></div></div>

<h2>Rubinius::VariableScope</h2>
A variable scope represents the state of the current execution context. It contains all of the local variables in the current scope, the execution context currently in scope, the current <code>self</code>, and several other characteristics.
I wrote about the variable scope <a href="http://yehudakatz.com/2011/02/18/getting-comfortable-with-rubinius-pure-ruby-internals/">before</a>. It’s one of my favorite Rubinius constructs, because it provides a ton of useful runtime information to Ruby that is usually locked away inside the native implementation.
Dumping and loading the <code>VariableScope</code> is also easy:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">class VariableScope
def _dump(depth)
Marshal.dump([@method, @module, @parent, @self, nil, locals])
end
 
def self._load(string)
VariableScope.synthesize *Marshal.load(string)
end
end</pre></div></div>

The <code>synthesize</code> method is new to Rubinius master; getting a new variable scope previously required synthesizing its locals using <code>class_eval</code>, and the new method is better.
<h2>Rubinius::BlockEnvironment</h2>
A Proc is basically nothing but a wrapper around a <code>Rubinius::BlockEnvironment</code>, which wraps up all of the objects we’ve been working with so far. Its <code>scope</code> attribute is a <code>VariableScope</code> and its <code>code</code> attribute is a <code>CompiledMethod</code>.
Dumping it should be quite familiar by now.

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">class BlockEnvironment
def marshal_dump
[@scope, @code]
end
 
def marshal_load(array)
scope, code = *array
under_context scope, code
end
end</pre></div></div>

The only thing new here is the <code>under_context</code> method, which gives a <code>BlockEnvironment</code> its variable scope and compiled method. Note that we dumped the static scope along with the compiled method above.
<h2>Proc</h2>
Finally, a Proc is just a wrapper around a BlockEnvironment, so dumping it is easy:

<div class="wp_syntax"><div class="code"><pre class="ruby" style="font-family:monospace;">class Proc
def _dump(depth)
Marshal.dump(@block)
end
 
def self._load(string)
block = Marshal.load(string)
self.__from_block__(block)
end
end</pre></div></div>

The <code>__from_block__</code> method constructs a new Proc from a BlockEnvironment.
So there you have it. Dumping and reloading Proc objects in pure Ruby using Rubinius! (the full source is at <a href="https://gist.github.com/1378518">https://gist.github.com/1378518</a>).
<script type="text/javascript">
addthis_url = 'http%3A%2F%2Fyehudakatz.com%2F2011%2F11%2F19%2Fhow-to-marshal-procs-using-rubinius%2F';
addthis_title = 'How+to+Marshal+Procs+Using+Rubinius';
addthis_pub = '';
</script><script type="text/javascript" src="http://s7.addthis.com/js/addthis_widget.php?v=12" ></script>
<div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/KatzGotYourTongue?a=0ROvAEqCwNE:8aNP2Ye9jho:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/KatzGotYourTongue?d=yIl2AUoC8zA" border="0"></img></a>
</div><img src="http://feeds.feedburner.com/~r/KatzGotYourTongue/~4/0ROvAEqCwNE" height="1" width="1"/>

Training / Consulting / Mentoring

Tokaido: My Hopes and Dreams

JavaScript Needs Blocks

Amber.js (formerly SproutCore 2.0) is now Ember.js

Announcing Amber.js

How to Marshal Procs Using Rubinius