jimmy hacking (at) Microsoft

03 Dec 2008

Here's the "script" I wrote up for my talk this evening at WPI, entitled "jimmy hacking (at) Microsoft -- open source programming languages."

Here are the slides

I'm back! For those who are fortunate enough to not know me, I'm Jimmy Schementi, and graduated from WPI in 2007 with a Bachelors and Masters in Computer Science from this amazing group of people who had the privilege to go either bald or crazy because of me. I ran the Game Development Club, and dedicated a year of my life to Neil Heffernan’s Assistment project. Now I work at Microsoft on implementations of Python and Ruby for the .NET platform. I got them to buy me a Mac. That’s me.

I’m going to waste an hour or so of your time talking about implementing two dynamic languages at Microsoft, IronRuby and IronPython. I’ll also touch on what it’s like working at Microsoft, including the challenges of integrating these languages with Silverlight, a platform for the browser.

But first ...

How I started with programming

image

LOGO. 6th grade. But the version I used was in English. Of course, I really didn’t know I was "programming", since I was making pretty pictures, but it was still fun. Years later, when I first saw Smalltalk, I could have sword it was LOGO. Anyway, I found a nifty logo environment on the web, and if you want to take a trip down memory lane with me here are some things to try out:

to spiral :size
  if :size > 30 [stop] ; an exit condition
  fd :size rt 15 ; many lines of action
  spiral :size *1.02 ; the tailend recursive call
end

reset
spiral 10

Which will produce a spiral:

image 

... and ...

to tree :arms :length :count
  if :count < 1 [stop]
  repeat :arms [
    lt 360/:arms
    pd fw :length
    tree :arms 2 * :length / :arms :count - 1
    pu bw :length
  ]
end

reset
tree 7 128 4

Will make a snowflake:

image

LOGO is a pretty advanced language, supporting recursion, functions, parenthesis-less method calls ... ah, feels like Ruby.

Anyway, why am I showing you LOGO? Because of the way you make logo programs: type a line, run a line, see the result. It’s an extremely productive way to program, and what most of this talk is going to be about. Let me show you what I mean ...

Taste of Ruby

To introduce Ruby, I made a screen-cast where I build a tiny unit testing framework entirely in a interactive environment. Here's two small environments in the browser that you can use to follow along: http://jimmy.schementi.com/silverlight/repl or http://tryruby.hobix.com.


Ruby Testing in a REPL from Jimmy Schementi on Vimeo.

This exploring lead to the making of this tiny unit testing framework. A majority of the logic is here, and the rest is for formatting and mocking. They are interesting because of how they redefine the methods defined in tester.rb, feeling very much like aspects, but built into the language. This allows Ruby to be very powerful for meta-programming, as this unit-testing framework is really a little DSL, looking very far from traditional Ruby syntax.

Terminology

I assume no one here as developed any software on the Microsoft stack, especially since most classes use Java, C, C++, or at least require it to be run-able on the CCC Linux machines. However, there are a lot of parallels between the two worlds, so let me make those connections:

  • Java langauge >>> C#
  • Java runtime/libraries >>> .NET platform (CLR)
  • (imagine) Flash + Java applets >>> Silverlight

Dynamic Language Runtime

IronPython 1.0 was released in September 2006, a Python 2.4 compatible implementation on Microsoft .NET, which performed on par with CPython. That project taught us that the CLR is great for dynamic languages, but there are things common to all dynamic languages that we had to roll ourselves. So, the “Dynamic Language Runtime” project was created to initially provide a dynamic language compiler infrastructure for the CLR, but can support static compilation as well.

A shout out to Bill Chiles, since these next couple of slides are based off of one of his talks.

image

Before VMs, compiler writers built tokenizers, parsers, semantic analyzers, one or more passes of data flow and optimizations, code generated to machine code, and then produced a equivalent binary file. With VMs, we remove the last few phases, so languages tokenize, parse to their syntax trees ...

image

... then do some semantics analysis, generate MSIL (higher level than machine IS), then write that to disk (an "assembly" in .NET).

image

With the DLR, the front-end is the same: tokenize to a AST, but then map the language's AST to a Expression Tree, which would include binding and semantic information. And that's it. The DLR will take those Expression Trees and generate MSIL ... no code generation phase at all.

The Dynamic Language Runtime is a derivative work of the common functionality needed to build Python, Ruby, and JavaScript, so this method is successful for real languages.

Let's dive a bit into how the DLR actually generates MSIL:

image

There are two ways to compile a method call or operation; statically compile to a known method OR to compile to a dynamic expression that describes what you want to do (call, get member, set item, set member, etc).

At runtime the dynamic expression gets bound to a rule (the method "Handle" in this case), which has the specific code needed to perform the operation when given a set of arguments. The rule has a test for when the implementation of the operation is valid (typically a test on the types of the arguments). The DLR caches these rules.

Take "hello " + name for example, which we can think of as x + y ...

image

A DynamicSite is the runtime object that manages method/operation caches for fast lookup of how to perform an operation. We compile an expression such as “x + y” to a call to a DynamicSite’s invoke method. That method starts out with a body that simply calls UpdateBindingAndInvoke.

The first time through, UpdateBindingAndInvoke asks the language that owns the context for the expression if it can perform the operation given the actual objects x and y. The binder may in turn ask the objects if they know how to perform the operation. If the binder gets a rule or can produce one, it returns it to the site.

The rule has a test for when it should be selected in the cache, and it has a target or implementation of the operation in the site (get member, call object, set item, do addition, etc.).

If we call the DynamicSite with different argument types ...

image

... we fail the one test for strings, then fall through to update and bind again.

if we had to integers the second time, we might find the new rule, and we recompile the method that holds the code from the rules. Thus whenever the DynamicSite is invoked on two integers or two strings now, it is very fast.

Since the DLR basically takes text and turns it into executable code, we can expose an API for application developers to “host” the DLR and run arbitrary scripts in their application. This “embedding scripting” functionality is very powerful.

Tooling

image

In static languages, it’s really easy to make a tool that can do “IntelliSense/code completion” for you, since you know all the type information (methods, properties, variables, etc) at development time/compile time, so a editor can use a background compiler to parse your code and see where there are syntax errors, and even do static analysis to find runtime errors like null reference errors, etc.

image

However, with dynamic languages, the code is compiled at runtime, so when you do foo.something, any tool can’t know what methods “foo” has, unless the entire application is running. As more static languages start picking up dynamic traits, this needs to be fixed. So, let's talk about the REPL:

History of the REPL

Lisp, a 50 year old programming language, has three vital functions: read (read from a command prompt), eval (run as string as lisp code), and print (print to screen). Lisp made popular the environment you saw before, where I type a line of code, and see the result immediately. This pattern maps directly onto those three Lisp functions, so that type of application is called a REPL (Read-Eval-Print Loop).

Today, REPLs are only seen in runtime-compilation languages, aka scripting/dynamic languages. Static languages like Java, C++, C#, etc, do not have the facilities to enable a REPL, outside of shelling out to the compilers. However, C# 5.0 MonoScript, a derivative of C# 3.0, provide APIs for accessing the compiler’s internals, enabling compiling at runtime, which REPLs depend on.

Hit the breaks

Ok. I’m getting bored of this history lesson, so I’m going to abruptly change direction and talk about some cool technology that can be use from dynamic languages, Silverlight.

Silverlight

Silverlight is a 4MB browser-plugin for Windows/Intel Mac OS 10.4+/Linux(through Moonlight) which puts a subset of the .NET framework in the browser. You can use the runtime to manipulate the DOM of the webpage (replace Javascript), or get a vector graphics surface to do fancier graphics/animations. Because it is integrated with the CLR, you can use any .NET language to program in the browser.

To you dynamic languages from Silverlight, you'll need the Silverlight Dynamic Languages SDK: http://codeplex.com/sdlsdk. The public source repository lives here (http://github.com/jschementi/agdlr),  if you just want to get the code or even help develop the SDK.

Here I’ll show Ruby running in the browser.

images

http://jimmy.schementi.com/silverlight/photoviewer

Note: if you don't have Silverlight installed, it'll prompt you to install. This will only work on Windows or Mac OS 10.4+, as the Linux implementation, Moonlight, is not released yet.

This is a simple Flickr client, entirely running on the client (does not require a server). It is mainly Ruby code, with the exception of a JavaScript library called Lightbox which I’m using to do the picture-open animations, since it’s awesome. This shows the interop between Ruby and JavaScript in the browser. It also directly manipulates the Html directly, so a end user wouldn’t know the difference.

I mentioned hosting before … so if I can host the DLR in a Silverlight application, couldn’t I make a little IDE in the browser?

DLRConsole Rocks

http://silverlight.net/samples/sl2/dlrconsole/index.html

DLRConsole is a Python Silverlight application, which hosts the DLR and lets you write Python or Ruby code, and manipulate a Silverlight surface. Here’s an example of rendering a clock and dragging it around.

rb> require 'lib/clock'
rb> require 'lib/drag'
rb> $c = Clock.show
rb> def drag(obj)
rb|   Drag.new(obj).enable
rb| ende
rb> drag($c.canvas)
rb> $t = canvas.children.first.children.first
rb> drag($t)
rb> class UIElement
rb|   def drag
rb|     Drag.new(self).enable
rb|   end
rb| end
rb> $tt = TextBlock.new
rb> $tt.text = "Monkey patched!"
rb> canvas.children.add $tt
rb> $tt.drag

Back to REPLs

Ok, let’s pull up for a second. I was talking about the REPL before, so can’t we have it here? Sure! The first example I showed (Ruby testing) was entirely written in a Silverlight REPL! In fact, it’s functionality built-into the Silverlight+DLR integration, so you can fire up a REPL against your live running app (much like “python –i foo.py”

I recently talked about this at Seattle CodeCamp v4.0, and wrote about it, so here's the post and demo.

Embedding Ruby in a Existing C# application

I also talked/wrote about this previously, so view the post here.

Rails Integration

image

There’s an awesome concept that’s lurking here: you have Ruby in the browser! That is, a language that runs either on a server or a user’s local machine can be pushed from a web server to run on the client, much like JavaScript, but it can be the same language on the server. Therefore, it could be the same exact code. I’ve been playing around with this idea for a while, first talked about it at RailsConf 2008, and it's current a Ruby on Rails plug-in called Silverline.

So, here's what I'm talking about ...

http://silverline.schementi.com/client

Note: this demo is fairly contrived, since it will only work for controllers named "ClientController". Also, it looks like it only works in production-mode once, so don't be surprised if the link above doesn't work for you. If you have Rails installed you can run the demos locally for yourself.

This Rails code ...

class ClientController < ApplicationController
  def index
    render :layout => true, :inline => <<-EOS
      <%= link_to_remote 'Show time', :url => {:action => 'time'} %>
      <br /><div id="time_div"></div>
    EOS
  end

  def time
    @time = Time.now
    render :update do |page|
      page.insert_html :bottom, 'time_div', "#{@time.to_s}<br />"
    end
  end
end

... along with a simple template produce this:

image

Very simple app; hit "Show time" with your forehead and it prints the time. As Firebug shows, each time request goes to the server.

class ClientController < ApplicationController
  client :time

  # ...
end

However, with the silverline plugin, if you add "client :time" to the controller, it will run the "time" action on the client:

image

Note that clicking on "Show time" this time doesn't send any requests to the server; the code is executing in the browser.

This simple demo shows how a developer can decide to run a Rails action either on the the client or server. Silverline also provides some rendering helpers to render XAML through ERb. Check out the site if you want to know more.

Open Source

Cartman vs Hippie

Doing Open Source at Microsoft is hard. Like swimming against the current. Sometimes it's even like being the hippies Cartman is spraying. But that's why the people I work with and I do it ... because it's challenging and potentially that much more rewarding. But enough of the sappy talk ...

IronRuby and IronPython are both Microsoft Public License open-source projects, a fully endorsed Open Source Initiative license. The Ms-Pl is an BSD-like license; very short and to the point. In even shorter terms, any Ms-Pl project can be forked, sold, modified, etc, and you owe nothing to Microsoft. IronRuby accepts contributions to the Ruby standard library, but all of IronRuby and IronPython are on track for accepting contributions to the entire codebases shortly.

Ruby and Python themselves are not Microsoft languages. They exist outside Microsoft and have their own thriving communities. Since Microsoft sees value in supplying Ruby and Python on .NET, I get paid to help make that happen. Since those languages are open-source to begin with, and Microsoft is benefiting from having those languages, it's only right for the Iron* implementation to be open source themselves.

Fat lady, sing it

Whew, that's it! This covered a lot, but I hope you have a better understanding of:

  • why dynamic languages are useful
  • what the DLR actually does for dynamic languages
  • why REPLs are the future of productive IDEs
  • how Silverlight and dynamic languages can blow your freakin' mind
  • and why in the world Microsoft is relevant to the open source community

Enjoy!

comments powered by Disqus