Jimmy Schementi

When John Lam and I were preparing for MIX, there were a lot of crazy JSON-serialization coding going on in our demo apps. And none of them were pretty. My implementation was outright dangerous ... it's a good thing I trust Flickr to not send me dangerous JSON. =) After we got back from MIX, I started working on making Dynamic Silverlight play better in the wild, and this JSON issue was the first on my list to attack. Ok, it got pushed a little further down on the que due to all these crazy "PM" tasks I have to do, but never-the-less I've got something to talk about =) Wait, Dangerous? Yep, dangerous! John's solution was much safer, since it used the JSON serializer in Silverlight, but it required an assembly and, well, uck! More seriously, serialization is not what I'm looking for; it's simply converting the JSON arrays/hashes to Ruby array/hashes, and making sure the key/value pairs are valid JSON. Serializing to an object model isn't as interesting to me, as I may have an object that already supports instantiating itself with JSON (Rails' ActiveRecord models ... more on this in a future post =)). Anyway, you can check out John's solution if you have a C# object model that needs to serialize/deserialize into JSON. My first stab at this was wrong on so many levels:

  def json_str
    @str.
    split(":".to_clr_string).
    join("=>".to_clr_string).
    to_clr_string
  end

Yep, that's right ... replace all occurrences of ":" with "=>" .. eek! That'll work fine until you have time values that look like "2007/03/24 09:00:00". =) Then what do I do with this transformed string?

  def parse
    ScriptRuntime.
    create.get_engine("rb").
    create_script_source_from_string(json_str).
    execute
  end

That's right, I eval it! (Why did I write my own eval? Because we don't have eval working in IronRuby today, so this is how you'd use the Hosting APIs of the DLR to implement it). No parsing/making sure the string doesn't reformat my drive or anything. =P

So, you write crappy code ... got a fix?

Yeah, thanks genius ... that's the point of this spheel. Anyway, the solution is to write a real JSON parser in Ruby. There are JSON libraries for Ruby already (gem install json), but I wanted a simple stand-alone parser to include in a Silverlight application. Luckily, that's not a extremely hard task as the JSON semantics are pretty straight-forward. Though, I'm a lazy bastard, so I'm not going to write my own!

Ruby Quiz #155 talks about parsing JSON, and shows a hand-rolled recursive decent parser. It didn't work as-advertised in IronRuby (to be expected though, IronRuby isn't finished yet), so I adapted it to work in IronRuby. This parser not only gets the semantics right, but makes sure I'm not eval-ing random stuff (it makes sure it parses as JSON before eval-ing). Anyway, here's all 154 lines of glory:

require "Microsoft.Scripting"
include Microsoft::Scripting::Hosting

class JSONParser
  # TODO: remove when eval is implemented
  def evaluate(str)
    ScriptRuntime.
    create.
    get_engine("rb").
    create_script_source_from_string(str).
    execute
  end

  # TODO: remove this when Struct is working properly
  class AST
    def initialize(value)
      @value = value
    end
    def value
      @value
    end
  end

  def parse(input)
    @input = StringScanner.new(input)
    if top_level = parse_object || parse_array
      top_level.value
    else
      error("Illegal top-level JSON object")
    end
  end

  private

  def parse_value
    trim_space
    parse_object or
    parse_array or
    parse_string or
    parse_number or
    parse_keyword or
    error("Illegal JSON value")
  ensure
    trim_space
  end

  def parse_object
    if @input.scan(/\{\s*/)
      object = Hash.new
      more_pairs = false
      while true
        key = parse_string
        break if key == false
        @input.scan(/\s*:\s*/) or error("Expecting object separator")
        object[key.value] = parse_value.value
        more_pairs = @input.scan(/\s*,\s*/) or break
      end
      error("Missing object pair") if more_pairs
      @input.scan(/\s*\}/) or error("Unclosed object")
      AST.new(object)
    else
      false
    end
  end

  def parse_array
    if @input.scan(/\[\s*/)
      array = Array.new
      more_values = false
      while true
        contents = begin
          parse_value
        rescue
          nil
        end
        break if contents == false or contents.nil?
        array << contents.value
        more_values = @input.scan(/\s*,\s*/) or break
      end
      error("Missing value") if more_values
      @input.scan(/\s*\]/) or error("Unclosed array")
      AST.new(array)
    else
      false
    end
  end

  def parse_string
    if @input.scan(/"/)
      string = String.new
      while true
        contents = parse_string_content
        contents = parse_string_escape if contents == false
        break if contents == false
        string << contents.value
      end
      @input.scan(/"/) or error("Unclosed string")
      AST.new(string)
    else
      false
    end
  end

  def parse_string_content
    if @input.scan(/[^\\"]+/)
      AST.new(@input.matched)
    else
      false
    end
  end

  def parse_string_escape
    if @input.scan(%r{\\["\\/]})
      AST.new(@input.matched[-1])
    elsif @input.scan(/\\[bfnrt]/)
      AST.new(evaluate(%Q{"#{@input.matched}"}))
    elsif @input.scan(/\\u[0-9a-fA-F]{4}/)
      AST.new([Integer("0x#{@input.matched[2..-1]}")].pack("U"))
    else
      false
    end
  end

  def parse_number
    if @input.scan(/-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?\b/)
      AST.new(evaluate(@input.matched))
    else
      false
    end
  end

  def parse_keyword
    if @input.scan(/\b(?:true|false|null)\b/)
      matched = @input.matched == "null" ? "nil" : @input.matched
      val = evaluate(matched)
      AST.new(val)
    else
      false
    end
  end

  def trim_space
    @input.scan(/\s+/)
  end

  def error(message)
    if @input.eos?
      raise "Unexpected end of input."
    else
      raise "#{message}: #{@input.peek(@input.string.length)}"
    end
  end
end

Usage:

p = JSONParser.new
r = p.parse YOUR_JSON_DATA

Enjoy!