Swift + JSON with code generation

Update 2016-01-17: I gave a talk about this topic at the January 2016 meetup of CocoaHeadsNL. See Swift JsonGen [talk].

There are many JSON parsing libraries written in Swift, and equally as many articles about them. This article is not one of those. This article is about a code generation tool, running on NodeJS, written in TypeScript: JsonGen.

JsonGen generates JSON parsers for immutable Swift structs. Get type safety without manually writing a parser.


When we – at Q42 – started work on our first Swift app (September 2014) we ran into an issue: How to get the JSON from API calls into nice, strongly typed, Swift structs. Chris Eidhof’s post on functional JSON parsing sounded appealing, so we tried the Argo library.

As a Haskell developer, Argo’s style looked interesting, although some colleagues were less enthusiastic about all the custom operators. However, before we got to discuss the merits of custom operators, we ran into a bigger issue: Argo didn’t work. When writing more complex parsers, the Swift type inferencer (or some other part of the compiler) kept crashing. I’m not blaming Argo here, I’m sure it’s swiftc that was to blame, and it probably now works with Swift 1.2. Still, it was a problem, so we looked further.

Getting data from JSON

For me, one of the most appealing aspects of working with Swift is type safety. Being able to precisely model all states of a user interface in a Swift enum is huge! This is the whole reason why we want JSON in structs instead of untyped dictionaries.

The native way of reading JSON in Swift is by using dynamic lookup on the dictionary returned by NSJSONSerialization:

let str = "{ \"title\": \"Hello, World!\", \"published\": true, \"author\": { \"first\": \"Tom\", \"last\": \"Lokhorst\" } }"
let data = str.dataUsingEncoding(NSUTF8StringEncoding)!

let obj = try! NSJSONSerialization.JSONObjectWithData(data, options: [])

Dynamic lookup

The result of NSJSONSerialization.JSONObjectWithData is actually a dictionary containing string values and nested dictionaries. By doing dynamic casts you can get to the desired value;

if let dict = obj as? [String: AnyObject],
    author = dict["author"] as? [String: AnyObject],
    firstname = author["first"] as? String,
    lastname = author["last"] as? String {
  print("Author: \(firstname) \(lastname)")

The reasons why I don’t like this approach are; It is really verbose to deconstruct the dictionary to extract some value. It is very error prone, because all the code is untyped. All the field names are in strings, and the types are just added ad-hoc.

Modeling data as structs

To get more type safety, I model my data with structs:

struct Blog {
  let title: String
  let published: Bool
  let author: Author

struct Author {
  let first: String
  let last: String

With a library like Argo, I can create a parser (or decoder) that can parse the AnyObject from NSJSONSerialization.JSONObjectWithData into the structs. However the downside is: I have to write a parser, where I again have to write all the field names in strings!

Code generation with JsonGen

Instead of manually creating the decoder, let’s generate it! That way there is less manual work to do, and no room for human-error. (For example: our app has over 50 structs for dealing with JSON, I don’t want to manually write and maintain the parsers for those). With a decoder generated by JsonGen, the use-side code becomes:

if let blog = Blog.decodeJson(obj) {
  let author = blog.author
  print("Author: \(author.first) \(author.last)")

The swift-json-gen command line tool will generate new source files for each supplied input file. For each struct an extension is created with a decodeJson method. The tool is smart enough to only generate methods for structs that don’t already have that method. So if you want to customize the behaviour of the decoding for one of your structs, all you have to do is implement the decodeJson method yourself and JsonGen won’t generate a new method.


  1. Put the Blog and Author structs in a file called Blog.swift
  2. Run the command: swift-json-gen Blog.swift
  3. This will generate Blog+JsonGen.swift that contains static decodeJson methods for the Blog and Author structs.

JsonGen ships with a file containing decoders for standard Swift and Foundation types: JsonGen.swift. These are decoders for types like Bool, Double, String, NSURL, but also generic types like Optional<T> and Array<T>. JsonGen also resolves type aliases and can generate decoders for nested types, as well as generic types.

If there is part of your JSON structure you don’t want to model in Swift structs, you can leave that part untyped by using one of three type aliases from JsonGen.swift: AnyJson, JsonObject or JsonArray. For example, in the following struct, the data field is kept as an untyped AnyObject:

struct Item {
  let id: String
  let name: String?
  let data: AnyJson

How it works

JsonGen works by calling swiftc with the -dump-ast flag. With this flag, the Swift compiler parses and type-checks the provided Swift files. But instead of generating code, it then outputs the Abstract Syntax Tree (AST) to stderr in a Lisp-like format. This AST is then parsed, and JsonGen looks for structs. For each struct, all fields and corresponding types are looked up. An extension method is generated that tries to find each field in the dictionary. If found, the value will be decoded using the decoder for the type of the field. If either the lookup or the decoding fails, nil is returned instead of the value being decoded.

The generated decoder returns an optional value. If the AnyObject can’t be decoded to the requested type, nil is returned. In development mode, the debugger is stopped using an assertionFailure so that the developer can figure out what is wrong (either the struct, the JSON, or both).

This is the generated code for the Blog struct from before.

extension Blog {
  static func decodeJson(json: AnyObject) -> Blog? {
    let _dict = json as? [String : AnyObject]
    if _dict == nil { return nil }
    let dict = _dict!

    let title_field: AnyObject? = dict["title"]
    if title_field == nil { assertionFailure("field 'title' is missing"); return nil }
    let title_optional: String? = String.decodeJson(title_field!)
    if title_optional == nil { assertionFailure("field 'title' is not String"); return nil }
    let title: String = title_optional!

    let published_field: AnyObject? = dict["published"]
    if published_field == nil { assertionFailure("field 'published' is missing"); return nil }
    let published_optional: Bool? = Bool.decodeJson(published_field!)
    if published_optional == nil { assertionFailure("field 'published' is not Bool"); return nil }
    let published: Bool = published_optional!

    let author_field: AnyObject? = dict["author"]
    if author_field == nil { assertionFailure("field 'author' is missing"); return nil }
    let author_optional: Author? = Author.decodeJson(author_field!)
    if author_optional == nil { assertionFailure("field 'author' is not Author"); return nil }
    let author: Author = author_optional!

    return Blog(title: title, published: published, author: author)

The generated code is simple albeit not very pretty to look at, but hopefully you’ll never have too. The added benefit of generating such simple, straight forward code is; If you should ever decide to stop using JsonGen, you can just continue on by manually editing and maintaining the generated code.

By the way; JsonGen also generates a encodeJson method. The encode method generates a dictionary for any type of struct. This is useful if you want to post data back to a JSON api.

JSON model vs Domain model

When decoding JSON into a domain model, you quickly run into the issue that the JSON structure doesn’t quite match the structure of the domain model. The generated decoder isn’t good enough and you need to customize. As said before, you can manually specify decoders for certain specific types. But, it can be quite a lot of work to create a decoder for a whole type. However, I think this is often not the approach you will want to take.

In my opinion there are two kinds of JSON in an app; It is either from a source that is completely under my control, or the source is external and the JSON format is out of my control. In the first case; I try to have the JSON format match as closely as possible to my domain model. For the few differences between the JSON and the domain model, I write custom decoders. That way I can directly parse JSON into the structs of my domain model.

In the case that the JSON format is out of my control, the format is often weird and doesn’t match my domain model. It has strange internal rules and custom logic. I don’t want to write all that custom logic and those business rules in the untyped world of decoders. I want to use the type checker to tell me if I’ve forgotten to implement an enum case, or if I assume a optional field is always available.

To achieve this, I create two sets of types; One set of structs that exactly matches the raw JSON, and another set of structs and enums that are my domain model. For the JSON structs, I generate decoders and encoders. But for the domain model, I manually write the mapping from the JSON structs to the domain structs. This may seem tedious, but it is necessitated by the fact that that mapping code is full of custom business logic. That business logic is code I want to write and maintain in the world of types.

This abbreviated example demonstrates this approach. The domain model is the Profile struct. The JSON model is the ProfileJson struct:

/* Domain model */

// A Profile consists of a first name and an optional avatar
struct Profile {
  let firstName: String
  let avatarURL: NSURL?

/* JSON model */

// This is the format of JSON data returned by an API.
// The `profile` property maps this JSON model to the domain model.
// The JSON parser for this type is generated by swift-json-gen.
struct ProfileJson {
  let FirstName: String
  let UseAvatar: Bool
  let AvatarUrl: String?

  var profile: Profile {
    // These are the business rules for handling the JSON data:
    // 1. When UseAvatar == false, ignore the url
    // 2. If AvatarUrl is not a real url, ignore it

    let avatarURL: NSURL?
    if let url = AvatarUrl where UseAvatar {
      avatarURL = NSURL(string: url)
    else {
      avatarURL = nil

    return Profile(firstName: FirstName, avatarURL: avatarURL)

(See this gist for a more slightly more detailed example)

Note that there is no need to completely “duplicate” all structs in the domain model into a JSON model. Sometimes the domain happens to match the JSON, in that case I only use the domain model. This happens a lot at the “leaves” of data structures. It is the roots that differ and contain a lot of custom mapping logic. In that case, the root structs exist in both the JSON model and the domain model and the leaves are shared.


If you want type safety, and don’t want to do dynamic lookup, you don’t have to manually write JSON parsers for your structs.
You can use JsonGen to generate decoders and encoders for your Swift structs. Let the untyped code be generated and only work in the nice, strongly typed world of Swift!

Install and use swift-json-gen like so:

$ npm install -g swift-json-gen
$ swift-json-gen MyDirectory/SubDirectory/

Or get the code from GitHub: tomlokhorst/swift-json-gen

This tool is used in a few projects at Q42, but it hasn’t been battle tested outside of our company. If you use it and find it useful, please let me know: @tomlokhorst.

Easy Storyboard segues in Swift

When working with a storyboard and segues, the usual method of performing a segue can be a bit cumbersome.

To deal with this, we’ve created a tiny helper called SegueManager that allows you to pass a closure to a performSegue call.

Current situation

Say you have a simple storyboard containing two view controllers: a master and a detail controller, with a segue between them:

Storyboard with Segue

Usually at the point where you’re performing the segue, the information that needs to be passed to the destination segue is also available. However we can’t directly pass it to performSegueWithIdentifier, but instead need to store it.

This is the code needed to store a string for later, and then perform the segue:

private var textForDetail: String?

@IBAction func openDetailAction(sender: UIButton) {

  // Save text for later
  textForDetail = "This is the detail screen!"

  // Kick off segue
  self.performSegueWithIdentifier("showDetail", sender: self)

The thing we want to pass to the detail view controller can be as simple as a single string, like it is here. But it can also be a complete view model, specific to the detail view controller.

This is the code needed to pass the stored string along to the destination view controller:

override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject?) {
  // Segue is underway, update destination ViewController with value set earlier
  if let vc = segue.destinationViewController as? DetailViewController {
    vc.displayText = textForDetail;

Note that if you have want to add a second segue, to a different view controller, you need to add code in three different places;

  1. You need an extra private var for the view model of the second controller.
  2. You need to add an extra if let check to the prepareForSegue code where the private var is passed to the destination view controller.
  3. At the place where you want to perform the segue, you have to also store the second view model in the private var.

Using SegueManager

With SegueManager you can immediatly pass a closure that sets the view model. This nicely scales when using multiple segues.

You first have to create a SegueManager in the master view controller:

var segueManager: SegueManager!

required init(coder aDecoder: NSCoder) {
  super.init(coder: aDecoder)

  // Create a segue manager based on the current view controller
  segueManager = SegueManager(viewController: self)

Also, you still need to override prepareForSegue, but it only needs one line of code that never needs to be changed:

override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject?) {

And finally, when you want to perform a segue, you can call performSegue on the SegueManager:

segueManager.performSegue("showDetail") { segue in
  let vc = segue.destinationViewController as DetailViewController
  vc.displayText = "This is the detail screen!"

And that’s it! Now, when you want to add a second segue, you only need to add a second call to performSegue, no other code needs to be changed.


Since the SegueManager is so little code, it’s available as a single gist that you can simply copy-and-paste into your project.

I’m curious, currently CocoaPods isn’t available for Swift, but once it is, would you like a tiny library like this to be available as a pod? Or would you be happy with just the gist as it is now?

2015-03-20: Now that CocoaPods has support for Swift, I’ve turned SegueManager into a cocoapod. OS X support has also been added.

[Dutch] Als Google je ontwerp namaakt, dan moet het wel goed zijn

De 9292.nl websiteBegin dit jaar begonnen we met Q42 en Fabrique samen aan de ontwikkeling van een nieuwe iPhone app voor 9292. Natuurlijk zijn we altijd gefocust op de beste gebruikerservaring. Ook bij de ontwikkeling van de iPhone app probeerden we weer een slag te slaan op UX-gebied, zelfs ten opzichte van de vernieuwde 9292.nl website die we een half jaar daarvoor hadden gebouwd.

Om een route met het OV te plannen zijn er fundamenteel drie gegevens nodig: een vertreklocatie, een aankomstlocatie en een tijdstip. Toen we begin 2011 de nieuwe 9292.nl site maakten hebben we veel tijd besteed aan het versimpelen van adresinvoer. We zijn van drie invoervelden op de oude site (plaats, type, locatie) teruggegaan naar één invoerveld met intelligente contextgevoelige suggesties. Het tijdinvoergedeelte bleef echter hetzelfde als op de vorige site: een datumkiezer, een tijdinvoer en twee radio buttons voor ‘Vertrek’ en ‘Aankomst’.


9292 iPhone app homescreenDe iPhone app begint heel prominent met een vraag “Waar wil je heen?”. We wilden het daadwerkelijk plannen van een reis zo dicht mogelijk brengen bij het antwoord van de gebruiker op die vraag. Dus je vult enkel het ‘Naar’ adres in (bijvoorbeeld ‘Thuis’ uit je favorieten), en dan plannen we de snelste reis naar huis met het OV. Om te kunnen plannen met maar 1 invoer moeten we default waarden kiezen voor overige velden. Dat was niet moeilijk voor de vertreklocatie, de iPhone heeft een GPS sensor, dus we kunnen je huidige locatie als vertrekpunt nemen. Het kiezen van een default voor het tijdstip was iets meer denkwerk.

Er zijn 3 verschillende scenario’s hoe gebruikers een tijdstip invoeren bij 9292:

  • Aankomst + tijdstip: “Ik heb om 18:00 afgesproken op de Korenmarkt in Arnhem, toon mij een reis.”
  • Vertrek + nu: “Ik wil zo snel mogelijk naar huis.”
  • Vertrek + tijdstip: “Om 14:00 is het seminar afgelopen, hoe kom ik weer op mijn werk?”

Het invoeren van een aankomsttijdstip is een grote use case, maar wij weten niet om welke tijd je ergens wilt aankomen, dus daar kunnen we geen default voor invullen. Vandaar dat we als default waarde hebben gekozen voor de tweede use case: Vertrek nu.

Het samenvoegen van de datum- en tijdvelden op de iPhone app was niet zo’n moeilijke keuze. iOS heeft standaard een datum+tijd control die we graag wilden gebruiken omdat iPhone-gebruikers daarmee bekend zijn. De realisatie dat de keuze voor vertrek of aankomst onderdeel van de tijdkeuze is, duurde wat langer. Je kiest niet een tijdstip, en kiest vervolgens of je dan wilt aankomen of vertrekken, nee je kiest een aankomst- of vertrektijd.


Toen we eenmaal door hadden dat ‘tijd’ één veld is, dat bestaat uit drie componenten, werd het grafisch ontwerp ook makkelijker. De iPhone app heeft maar drie velden (in plaats van vijf): Van-locatie (met default waarde “Huidige locatie”), Naar-locatie, en tijdstip (met default waarde “Vertrek nu”).

Het tijdinvoer schermAls je op dat veld tapt verschijnt er een datepicker. Rechtsboven zit een Kies-knop waarmee je je selectie kunt bevestigen. Links boven zit een Nu-knop. Deze stelt ‘Vertrek nu’ in en sluit de ‘datepicker’ ook weer. Tussen die twee knoppen zit de keuze voor ‘Vertrek’ of ‘Aankomst’. Hiervoor wilden we geen toggle button met twee kleuren maken, omdat het dan moeilijk is om te zien welke van de twee geselecteerd is.

De twee knoppen ‘Kies’ en ‘Nu’ hebben we dezelfde zwarte kleur gegeven omdat ze beiden hetzelfe effect hebben: de datepicker wordt gesloten. De ‘Vertrek’ en ‘Aankomst’ knoppen wilden we daarom twee andere kleuren geven: geselecteerd en niet-geselecteerd. Om extra te verduidelijken welke van de twee opties geselecteerd is, hebben we een vinkje bij de geselecteerde optie gezet. Door het vinkje in het midden de plaatsen konden we die gebruiken voor beide opties, en bespaarden we ook horizontale ruimte. Door het blauwe vlak mee te laten bewegen met je vinger lijkt het op een zelfde soort schuifje als door heel iOS gebruikt wordt als ‘on/off’-knop.


Als programmeur ben ik de hele dag door bezig met data-modelering, en ik vraag me dan ook af waarom het zo lang heeft geduurd voordat we ons realiseerden dat ‘vertrektijd’ en ‘aankomsttijd’ één begrip is. Misschien heeft het, voor mij in ieder geval, te maken met de gebruikte technologie. We hebben het server gedeelte van 9292 ontwikkeld in C#, een taal die geen algebraische data-types kent. Daarom hebben we tijd en vertrek/aankomst met twee velden geïmplementeerd, bijvoorbeeld: ‘dateTime=2012-12-14T07:00‘ en ‘searchType=Departure‘. Dit beïnvloede mijn denken erg, waardoor ik de twee velden lang als apart heb gezien.

Als we een programmeertaal met algebraische data-types hadden gebruikt, bijvoorbeeld Scala of Haskell, had ik het waarschijnlijk zoals hieronder gemodelleerd, en had ik mij daarmee gelijk gerealiseerd dat ‘tijdstip’ één ding is.

data TimeDetails = Departure DateTime | Arrival DateTime

Als programmeur moet ik in systemen denken en het helpt erg als die systemen even expressief zijn als het mentale model van gebruikers. Dan hoef ik niet constant die vertaling te doen.

Tot slot

Gisteren kwam Google Maps voor iOS uit. Tot mijn verassing zag ik een bekend scherm daarin. Blijkbaar heeft Google dezelfde realisatie gehad als wij. Ze zij in ieder geval op exact dezelfde UI uitgekomen!

9292 tegenover Google Maps

Zou Google even veel tijd aan dit veld hebben besteed als wij? Ik hoop in ieder geval dat we hiermee duizenden gebruikers misschien een halve seconde aan tijd kunnen besparen. Ik wacht de calculatie van Google wel af waarin zij berekenen hoeveel uur frustratie we gezamenlijk de mensheid bespaard hebben.

Why Libraries are better than Frameworks

TLDR; Use lazy evaluation to prevent high coupling.

There are two ways to design an application: based a particular framework, or using libraries.
Actually, there are a lot more ways to build applications, including combinations of the two, but let’s just focus on those.

First, let’s define the word “framework”. I want to set it apart from what’s known as a platform.
A platform is some lowest level on which to build an application, examples are OS’s or virtual machines, e.g.: Linux, OS X, Java, or .NET.
These platforms are great; They allow application developers to focus on the specific application logic, reusing all the ‘plumbing’ like sockets and file systems.

No, what I mean by frameworks, are the things your application (or part of your application) lives in.
Examples of such a frameworks are Java’s Swing, or ASP.NET MVC.

Differences between a framework and a library

There are, of course, a lot of definitions of frameworks and libraries. I don’t want to get into those, rather I’d like to illustrate my view on the differences by means of this picture:

Framework architecture: Application lives inside framework, atop a platform. Libraries architecture: Application built against libraries, atop a platform.

When building an application using a particular framework, the application lives inside the framework.
This is most noticeable when the application is required to inherit from some framework class, although this is not always the case.
From the viewpoint of the application, the framework is the whole world.
The framework is the all-powerful environment which can do everything the application would ever want (for some particular domain, at least).

Alternatively, an application can be writing using libraries (and it’s always libraries, plural, it’s never just the one).
In that design, a library is just tacked on to the side of the application.
The application stands on its own, it has an identity outside of a particular framework, and it just uses libraries to do some part of the work.

Benefits of a framework

On his blog, Martin Fowler describes “Inversion of Control” as perhaps the “defining characteristic of a framework”.
Wikipedia also mentions inversion of control, along with default behaviour and extensibility, as one of the distinguishing features of a framework.

Inversion of control, also known as the Hollywood Principle “Don’t call us, we’ll call you”, can indeed be a benefit.
An example of this is a graphical (web) application that renders some data.
In this example, the framework is responsible for rendering the data to the end-user. Thus the application might not know how much data is required or when to compute it.
By having the framework call application code, it can ask for data when it is needed, and only ask for as much data as can currently be displayed.

Martin describes this better than I can:

Inversion of Control is a key part of what makes a framework different to a library. A library is essentially a set of functions that you can call, these days usually organized into classes.
Each call does some work and returns control to the client.

A framework embodies some abstract design, with more behavior built in. In order to use it you need to insert your behavior into various places in the framework either by subclassing or by plugging in your own classes.
The framework’s code then calls your code at these points.

Downsides of a framework

I’ve already alluded to some of the downsides I think frameworks have.
First of all, frameworks require a lot from client code. In some OO frameworks the application is required to implement an interface, or worse, to inherit from a framework baseclass.
An example of this is Swing, the Java GUI framework, where you’re supposed to create a subclass of JComponent.
When you do this, you can call the framework (e.g. super.getX()), and the framework can call you back, for example if you override the paint method.
However, by using this design you’ve just giving up the only inheritance mechanism you have in Java!
Say goodbye to all your nice object oriented designs.

Frameworks are also notoriously hard to replace. Precisely because the framework is the application’s whole world.
If you’re using an encryption library you don’t like, you can replace it by some other one.
In frameworks, this is not the case, you can’t just replace a framework.
The best you can hope for is to just add a new encryption library, bypassing the capabilities build-in to the framework.
Granted, replacing a library might be some work, since the API’s for the different libraries can be quite different, but it’s probably doable.

The same, hard-to-replace argument can be used against platforms, but there I think it’s less applicable.
An application usually is better, the more native to the platform it feels (and is).
For example, a native OS X application usually works a lot better than something that barely uses OS X’s capabilities.
An application that uses more of a framework doesn’t feel more native, it just feels more default.

The biggest problem I have with frameworks is the tight coupling between the application and the framework.
The application calls the framework to do certain things, and the framework calls the application back.
This tight coupling goes directly against well known software design principles, and it is the ultimate source of the problems described above.

Alternative to a framework

Ok, so that are some of the benefits and downsides to frameworks, but what is a good alternative to a framework? Well, libraries of course!

Here we come to the issue of programming languages. If your language is powerful enough, you can build libraries with the same capabilities as frameworks.
By using higher-order functions, you can pass in code to the library to execute when it needs it.
Some might say that such a library is now a framework, but I wouldn’t call the map function a whole framework. I certainly don’t live in map.

So, using higher-order functions you can replace some of the capabilities of a framework in a library, but that still doesn’t feel right.
If your library uses a lot of higher-order functions, you might get the same functionalities as a framework, but now your code is all written inside-out.
The solution to this issue is lazy evaluation.

In a lazy-by-default language like Haskell, an application can just pass in a (potentially) giant data structure to a rendering library.
The library will only consume as much of the data structure as it needs, and no further evaluation will be done.
In other words; Lazy evaluation uncouples the data producer from the data consumer.
Sure there’s still the need to communicate about how much data is needed, but this is taken care of by the runtime system.
There doesn’t need to be an intimate dialog between a framework and the application about which data is needed and when.

The first example some people give when explaining lazy evaluation is “now you can create an infinite list of prime numbers!”, but who needs that?
I think lazy evaluation brings something that’s way more important, that is a fundamental part of software design; Lazy evaluation gives you composability.


Lego blocks
“LEGO” photograph, by Jez Page.

The silver bullet in software development is composability. Or, if you don’t belief in silver bullets, let’s just agree its a step in the Right Direction.
The old idea of building software out of separate parts, like Lego blocks is still appealing.

However, the fact that there are so many frameworks out there shows that there is a need for something more than “blocks”.
I think the parts we need to build up larger and more complex applications are not shaped like rectangular blocks.
The “components” of the future will make heavy use of lazy evaluation and higher-order functions to allow for better composability.

By writing libraries from the core up to make extensive use of lazy evaluation and higher-order functions, we get reusable, composable, loosely coupled components.
These libraries are way more reusable than any framework, and they will lead to better designed applications.


In conclusion, I think the benefits frameworks have over traditional libraries do not outweigh the costs.
Using sufficiently powerful programming languages, we can create better libraries that use lazy evaluation and higher-order functions.

The only downside that properly designed libraries have to frameworks is that they are way harder to build.
A framework can basically be created by taking any old application, removing the content, putting in some hooks to call back to client code, and now you have a framework!
(That’s perhaps a bit of an oversimplification, but I like to beat up on frameworks ;-).)
Libraries on the other hand need to be designed, they need a good and clean API, otherwise they’re unusable.

On the upside, while frameworks often need the be “learned”, this is less so the case with libraries.
People read whole books on how to use a particular framework, but they usually just start programming against a library.

Again, I think the costs of a framework are bigger than the benefits it has.
Especially now that more and more libraries begin using lazy evaluation and higher-order functions. Particularly in languages like Haskell.
The future of software components is libraries.

Or, to put this article in other words: Haskell people, please stop building “frameworks”, we’re better than that!

AwesomePrelude presentation (video)

Last monday (February 8, 2010) Sebastiaan Visser and I presented our ongoing work on the AwesomePrelude at the Dutch Haskell User Group meeting.

In a previous post “Deeply embedded DSLs in Haskell”, I introduced the AwesomePrelude. While the ideas behind that post haven’t changed, the implementation of the AwesomePrelude is radically different now.

The AwesomePrelude is reimplementation of the Haskell prelude in which all data types have been replaced with type classes. Embedded Domain Specific Languages (DSLs) can implement these type classes to get access to the functions defined in terms of the type classes. For example: by implementing the BoolC type class, a DSL gets to use the (&&), (||) and not functions.

Here’s a recording of our presentation:

The slides are online, and the code is available on github.
The reactive javascript example Sebastiaan mentions in the talk doesn’t use the AwesomePrelude per se, but is based on the same ideas.

Deeply embedded DSLs in Haskell: overloading everything under the sun

Haskell Symposium 2009Haskell is a great language for building embedded domain specific languages. Using algebraic data types and higher order functions, it’s very easy to work with and reason about an embedded language. This week at the ICFP, several people commented on how Haskell allowed them to build custom control structures and express ideas more clearly. However, while powerful, these embedded languages usually aren’t as expressive as their host language Haskell. The concrete syntax usually isn’t as succinct as the Haskell equivalent of a certain expression.

This article explores the very early stages of an idea by Sebastiaan Visser and myself to more deeply embedding DSLs in Haskell. It has been written up rather quickly, so if (when?) you find errors, please do let me know. Also, when I say “early stages”, I really mean just that, there are lots of open questions and unknowns left. To start off, here’s a little teaser:

*Main> -- A simple function that operates on a generalized boolean
*Main> let f x = not x && false || true

*Main> -- Coerce the function to work on normal Prelude Bools
*Main> let g = f :: Prelude.Bool -> Prelude.Bool

*Main> -- Call the function
*Main> g false

*Main> -- Coerce the exact same function to work on JavaScript booleans
*Main let h = f :: Js JsBool -> Js JsBool

*Main> -- Call the function again
*Main> h false
(((false ? false : true) ? false : (false ? false : true))
 ? ((false ? false : true) ? false : (false ? false : true))
 : true)

The JavaScript string above is the JavaScript equivalent of the f function applied to false. If we evaluate that string, we get a JavaScript boolean true!

> {-# LANGUAGE GADTs, FlexibleInstances, MultiParamTypeClasses, FunctionalDependencies #-}


At its most basic level, a DSL is usually expressed as an ADT. Take for example this simple definition of a mathematical expression language:

> data ArithExpr
>   = Lit Integer
>   | Add ArithExpr ArithExpr
>   | Mul ArithExpr ArithExpr
>   | Sub ArithExpr ArithExpr
>  deriving (Show, Eq)

With this evaluator:

> evalArith :: ArithExpr -> Integer
> evalArith (Lit x)   = x
> evalArith (Add x y) = evalArith x + evalArith y
> evalArith (Mul x y) = evalArith x * evalArith y
> evalArith (Sub x y) = evalArith x - evalArith y

This is a very simple language supporting literal values (Lit), addition (Add), multiplication (Mul) and subtraction (Sub). Given this language, we can now write a mathematical expressions like “6 + 12 * 3” as such:

> a1 :: ArithExpr
> a1 = Add (Lit 6) (Mul (Lit 12) (Lit 3))

This is a bit verbose for something that can be expressed in Haskell with a lot less characters. But luckily, we can fix that in this case, by using the Num type class:

> instance Num ArithExpr where
>   x + y         = Add x y
>   x * y         = Mul x y
>   x - y         = Sub x y
>   abs x         = error "Not implemented"
>   signum x      = error "Not implemented"
>   fromInteger x = Lit x

Now that we have made ArithExpr an instance of the Num type class, we can use Haskell syntax to write down ArithExpr expressions:

> a2 :: ArithExpr
> a2 = 6 + 12 * 3

Of course, while this looks like a Haskell epression, it still generates an ArithExpr expression tree:

ghci> a2
Add (Lit 6) (Mul (Lit 12) (Lit 3))

Getting functions for free

Now that our ArithExpr is an instance of Num we can use functions like +, but we can do even more. We can also reuse all existing functions that are defined in terms of the + operator and other functions in the Num type class. For example the sum function:

sum :: (Num a) => [a] -> a
sum []     = fromInteger 0
sum (x:xs) = x + sum xs

Since this function is defined entirely using functions from the Num type class, it also works on our ArithExpr!

> a3 :: ArithExpr
> a3 = sum [3, 4 * 3, 2 - 7]

When we inspect this in GHCi we get the following:

ghci> a3
Add (Add (Add (Lit 0) (Lit 3)) (Mul (Lit 4) (Lit 3)))
    (Sub (Lit 2) (Lit 7))
ghci> evalArith a3

Obviously we didn’t just add lists to our simple expression language. The lists here are Haskell lists, containing ArithExpr values. Haskell can be used as a meta programming language for the ArithExpr language.

Scaling up: using more Haskell syntax

A few months ago Sebastiaan send a message to the Haskell Cafe mailing list titled “Bool as type class to serve EDSLs” where he asked about using the Eq type class for DSLs. In the same way that we make ArithExpr an instance of Num. Lets take this ADT:

> data Expr a where
>   Num      :: ArithExpr -> Expr Integer
>   LitFalse :: Expr Bool
>   LitTrue  :: Expr Bool
>   Equal    :: Expr Integer -> Expr Integer -> Expr Bool
>   If       :: Expr Bool -> Expr a -> Expr a -> Expr a

The Eq and Show instances for this GADT are provided at the end of this article. The evaluator is defined as such:

> eval :: Expr a -> a
> eval (Num ae)      = evalArith ae
> eval (LitFalse)    = False
> eval (LitTrue)     = True
> eval (Equal e1 e2) = eval e1 == eval e2
> eval (If p e1 e2)  = if eval p then eval e1 else eval e2

We can now write down an expression like this:

> a4 :: Expr Integer
> a4 = If (Equal ((2 + 3)) (5))
>         (Num 1)
>         (Num 0)

And here’s where the Eq type class comes in, we’d like to use the (==) operator instead of the Equal constructor. Unfortunately, this is not possible since the type of (==) is this:

(==) :: Eq a => a -> a -> Bool

This shows us that (==) is polymorphic in both its arguments, but not in its return type. We could make our Expr data type an instance of Eq, but won’t be not able to use Equal as an implementation of (==). Equal is of type Expr Integer -> Expr Integer -> Expr Bool, it returns a Expr Bool instead of Bool.

What we would like to have is a more polymorphic (==) of the following type:

(==) :: (Eq a, BooleanLike b) => a -> a -> b

We have replaced the type Bool with a polymorphic b that is an instance of the BooleanLike type clas. This way (==) is polymorphic in both its arguments (a long as they implement Eq) and its result type (as long as its “BooleanLike“).

In other words: we replace the concrete data type Bool with a polymorphic type b in a type class. Can we take this further?

Going large: no more data types!

With that idea in mind (replace data types with type classes), Sebastiaan and I have been working on systematically rewriting the Haskell Prelude to a more general version. We have dubbed this alternative Prelude the AwesomePrelude, and it’s currently available on github.

Lets have a look at how the AwesomePrelude works. First the normal Prelude Bool data type:

data Bool = False | True

This is rewritten in the AwesomePrelude to the following type class (called Bool in the AwesomePrelude):

> class Boolean f r | f -> r where
>   bool  :: r -> r -> f -> r
>   false :: f
>   true  :: f

This type class consists of these parts:

  • The false and true members. These represent the original two constructors, but they now are more polymorphic, since they are of type f, the first argument to the type class.
  • The bool function. Because we no longer have real constructors (we use true and false), we can’t do pattern matching anymore. With bool we can “fold” over a Bool value. The bool function destructs a f value and it returns either the first (false) argument or the second (true) argument.
  • The f type argument. This is the concrete type we use to represent a boolean. In the case of our Expr data type, this is (Expr Bool), see below.
  • The r type argument and f -> r functional dependency. This is the somewhat strange part of the type class. We’ll discuss this after we’ve looked at the Expr instance.

Giving the type class defined above, we make our Expr data type an instance of the type class like so:

> instance Boolean (Expr Bool) (Expr Integer) where
>   bool f t p = If p f t
>   false      = LitFalse
>   true       = LitTrue

This instance demonstrates the need for the r type argument in the Boolean type class. The r variable is the type to which a boolean is “destructed”, when it is “pattern matched” (that is, it is used). We need a way to ensure that the result of using a boolean in a DSL doesn’t fall outside that DSL.

In the case of our Expr data type, the result of If can only be a Expr Integer. Other languages might be a bit more liberal, but they usually do at restrictions. For example, we provide a JavaScript instance of the AwesomePrelude called JsPrelude. In this JsPrelude, we have this: instance Boolean (Js Bool) (Js a). That means that using a Js Bool will always result in a Js a.

To show how this pattern generalizes, lets look at the Maybe type class.

class Maybe f a r | f -> a, f -> r where
  maybe   :: r -> (a -> r) -> f a -> r
  nothing :: f a
  just    :: a -> f a

This type class again has two “constructors”, nothing and just. Also there is a “destructor” function called maybe, which happens to be the same thing as the maybe function in the normal Haskell Prelude.

The real difference between the Maybe type class and the Bool type class is the extra type variable a. This is introduced because the normal Maybe data type has a type variable a. Indeed, the number of type variables in the “type classification” of a data type is equal to its original number of arguments plus two for the f and the r

Conclusion? Free functions and future work!

We are currently working on the AwesomePrelude on github. The AwesomePrelude module exposes type classes for each data type in the normal Prelude. Note that by “each”, I really mean “the five that we felt like writing”. The AwesomePrelude module also exports common functions like (&&), sum and uncurry, that are entirely defined in terms of the “constructors” and “destructors” in the module.

We have written a JsPrelude that provides instances for the type classes defined in the AwesomePrelude. While this module is far from complete (we’re doing some weird type hackery, that isn’t really working), it can already do some cool things, like the example at the top of this article.

… define a function on “type class values” in the exact same way as on normal “data type values” … there is no difference between using the generalized functions and the original functions.

The Main module shows some examples of uses of the generalized Prelude. The nice point about this examples is this though: We can define a function on “type class values” in the exact same way as we would on normal “data type values”. Except for constructors and pattern matching, there is no difference between using the generalized functions and the original functions.

To sum up: Our understanding of this concept is still currently rather limited, but I have the feeling we can do a lot with this, particularly in the field of embedded domain specific languages. After seeing a couple of interesting talks this week at the ICFP and the Haskell Symposium, I think I need to start reading lots of papers (any pointers would be appreciated). More on this to follow!

A few remaining instance declarations:

> instance (Show a) => Show (Expr a) where
>   show (Num ae)      = "Num (" ++ show ae ++ ")"
>   show (LitFalse)    = "LitFalse"
>   show (LitTrue)     = "LitTrue"
>   show (Equal e1 e2) = "Equal (" ++ show e1 ++ ") (" ++ show e2 ++ ")"
>   show (If p e1 e2)  = "If (" ++ show p  ++ ") (" ++ show e1 ++ ")"
>                                           ++ " (" ++ show e2 ++ ")"

> instance (Eq a) => Eq (Expr a) where
>   (Num ae)      == (Num ae')       = ae == ae'
>   (LitFalse)    == (LitFalse)      = True
>   (LitTrue)     == (LitTrue)       = True
>   (Equal e1 e2) == (Equal e1' e2') = e1 == e1' && e2 == e2'
>   (If p e1 e2)  == (If p' e1' e2') = p  == p'  && e1 == e1'
>                                                && e2 == e2'
>   _             == _               = False

> instance Num (Expr Integer) where
>   (Num x) + (Num y) = Num (x + y)
>   (Num x) * (Num y) = Num (x * y)
>   abs (Num x)       = Num (abs x)
>   signum (Num x)    = Num (signum x)
>   fromInteger x     = Num (Lit x)

Waking on LAN

Computers are really nice machines, they have amazing processing power and can be used for all sorts of noble purposes, such as storing your movie collection family photos.

Unfortunately, they also have the downside of generating heat and noise, so that’s why I keep one of mine at a friend’s house (also because he has a very fast internet connection). Strangely though, it seems he also isn’t to fond of all that heat and noise and turns the machine off when he thinks I’m not using it. So that’s why I needed a way to send a Wake on LAN signal, known as a magic packet, to remotely start the machine.

Yesterday I wrote a small Haskell program called wol (available on Hackage) to send the “magic packet” to his router. We had to flash his router with new firmware to be able to have it memorize the internal ip address/mac address/port combination, but now it’s working!

I must say, it took me an embarrassing amount of time to write this program. For some time, I was stuck on why the socket wasn’t working. Then I found out I shouldn’t use the PortNum constructor to construct PortNumbers, because the endianess doesn’t match on my machine. It turns out your supposed to use the Num instance of PortNumber and use functions like fromIntegral. I’m happy to see other people also noticed how annoying this is, and are discussing a revamped network package.

Anyway, now you can all start starting your machine remotely with wol (which, to be honest, is just a Haskell port of some PHP script I found online…).

Hackage on Twitter

The Haskell community is very active. New packages are released to the central Hackage server each day.

If your like me, and like to keep up on all the different new packages, you can now follow Hackage on Twitter.

But be warned! There’s a lot happening, in the 3 days Hackage has been online, it has already tweeted more than 70 new releases.

The Twitter account is running a tool called hackage2twitter, build on top of the feed2twitter library for tweeting posts from a RSS or Atom feed.

If you want to pipe your own feed to Twitter, you can use the feed2twitter library or executable.

bool-extras released

Ever found yourself in need of a if-then-else replacement that can be partially applied? Something you can pass to higher order functions? In short, ever needed this function?

bool :: a -> a -> Bool -> a
bool x _ False = x
bool _ y True  = y

Well, I did. So I’m very happy to announce my first public Haskell library: bool-extras!