Javascript error reporting for fun and profit

Here at PipelineDeals, our app is very Javascript intensive. In fact, for our Jupiter release, we moved much of our logic from rails to the browser and Javascript, using Backbone.js, Coffeescript and a host of other new cutting edge technologies.

During deploys and other edge cases, we want to know if a user has received a javascript error, just like we would with any other type of exception. We deploy very frequently. A few weeks ago we ran into a problem where one of our app servers did not minify and concatenate one of our javascript files correctly, and led to a host of issues that customers saw. In order to prevent this in the future, we decided to search around and see if we could have javascript errors report to one of our favorite tools, NewRelic RPM.

As it turns out, catching and reporting Javascript errors is pretty easy. We ended up hooking into window.onerror, which we define right after we add jQuery, right at the top of the page.

// Determine if the error occurred before or after document ready
jQuery(function() { window.documentIsReady = true; });

// report a maximum of 5 errors
window.MaximumErrorCount = 5;

window.onerror = function(errorMsg, file, lineNumber) {
  window.errorCount || (window.errorCount = 0);

  // this example assumes a ppd object that has various information about 
  // our environment and our logged in user.
  if (ppd.env == 'production' && window.errorCount <= window.MaximumErrorCount) {
    window.errorCount += 1;

    // post the error with all the information we need.
    jQuery.post('/javascript_error', {error: errorMsg, file: file, location:     window.location.href, lineNumber: lineNumber, userId: ppd.User.id, documentReady:     window.documentIsReady, ua: navigator.userAgent});
  }
}

The above code will execute if the user receives a javascript error. It will then do an ajax post to our app, which will handle the error.

class JavascriptErrorController < ApplicationController

  def javascript_error
      # post the error to newrelic.
      # You could also send an email, notify hipchat, whatever.
      NewRelic::Agent.notice_error("Javascript error: #{params[:error]}", {:uri => params[:location], :params => params})
    end
    head :ok
  end
end

Not only has this been invaluable for catching that rare, but painful asset problem, but it will also catch legitimate issues and race cases where sometimes a variable might not be defined.

In NewRelic, javascript errors get report just like any other error, and if we do have an asset problem, our error rate will shoot through the roof and all the devs will be notified.

Improving Rails Performance with Twitter's Kiji Ruby

In March, Twitter unveiled "Kiji", an effort to significantly reduce the impact of running the garbage collector in the Ruby Enterprise Edition (REE) runtime.  Many have criticized their effort because it focused on the older 1.8.7 instead of the newer 1.9.x version of ruby.   But for older, larger apps that still run Rails 2.x or need ruby 1.8,  Kiji may be worth exploring.

MRI relies on a single heap that basically resembles  a slab allocator.   New slabs are created via malloc()  and carved up into a fixed 40-byte slot.  Each slot can hold a variety of ruby objects.   When a slab becomes full,  GC is invoked to clean things up by looking for non-reachable objects.  If the slab is still full after GC, then a new slab is allocated.

MRI’s GC uses a very simple naive mark-and-sweep algorithm.  When MRI GC runs, all code execution stops until GC is finished.   The implementation is such that it does not (easily) leave the door open to use more advanced GC techniques. 

For large applications, and espcially web-based applications, this can become a major issue.    There is a direct relationship between the number of objects and the garbage collection runtime.  More objects equals more time spent garbage collecting and less time serving customer requests. 

This is where Kiji comes in.

Kiji’s GC acts like a type of generational algorithm by splitting objects into heaps based on their age with the notion that younger objects are more likely to become garbage.  While not a “true” implementation of a generational GC, because objects cannot move between heaps , there are still benefits to using it.   Kiji’s GC allows the CPU to continue executing ruby code while GC is run.  For web-based applications, this is a huge win, as your users do not have to wait for GC to perform a full run during the request cycle.

Kiji also changes the way that frequently-used objects are stored in the heap.  For web-based ruby applications, the majority of “live” objects in the heap are RNodes.    RNodes are responsible for holding the actual source code of the application.  Kiji implements a long-life stack that solely stores RNodes which is garbage collected much less frequently than the primary stack.  Because of the primary stack now has less objects, GC runs complete much faster, at the expense of the application consuming more memory.

Twitter’s initial announcement and subsequent followup both included results from running the twitter.com webapp under synthetic load but having run Kiji in production for nearly two months we are excited to share the noticeable difference it has made within our own stack.

In the figure below we have collected several weeks worth of runtime data from our production application servers. Each point in the figure represents the average of one day's worth of data collected in one minute samples. To understand the potential effect Kiji might have our application's runtime we captured both the average time a customer's request spent solely within the Ruby runtime as well as our application's apdex, an open standard that converts response time measurements into the degree to which user expectations are met. An application's apdex can range uniformly from zero (no users satisfied) to one (all users satisfied) .

Days spent running REE or Kiji neatly cluster along expected lines, with REE posting signficantly higher response times and subsquently lower apdexes. Whle utilizing nearly fifty percent less of the CPU than REE, Kiji did, however, bring a nearly three fold memory increase. Overall Kiji shaved off nearly one second from our page load time bringing us to an average of three seconds per page well below today's global average.

If you would like to take Kiji for a spin with your own application you can find it on Github. We would love to hear about your experiences using Kiji (or another alternative Ruby runtime) with Rails.

Technical Notes: 

* Our production application servers use Debian 5.0, we have successfully compiled Kiiji on RPM-based distributions (RedHat/CentOS/Fedora) as well. To date we have been unable to successfullly build Kiji on an Ubuntu based box (Lucid - 10.04 and later tested) due to an bug triggered by libc6. 

* We built Kiji against Google's TCMalloc instead of malloc() producing the same binary as Twitter using the method they provided in their README.