Swift Package Manager Plugins [talk]

Last week, I gave an impromptu talk at the CocoaHeadsNL November meetup.

I’ve recently been working on R.swift version 7, which adds Swift Package Manager Plugin support.

After a brief overview of R.swift itself, I demo’ed the way SPM plugins can be added to an Xcode project, as well as an SPM project. I then showed the two types of Plugins; BuildTool plugins and Command plugins.

In the Q&A afterwards, people asked some great questions about SPM plugins and R.swift.

See the R.swift project on Github for a full example of how to use SPM Plugins.

@CloudStorage property wrapper

I wrote a Swift library to help with prototyping (haven’t used it yet in production apps).

CloudStorage is property wrapper that will save values in iCloud key-value storage. These values are persisted across app restarts and will sync between different devices that are signed into the same iCloud account. This property wrapper is similar to AppStorage and SceneStorage, two new types Apple introduced with iOS 14.

The basic API is the same as for AppStorage; add @CloudStorage("someName") in front of a property, to save the value to iCloud key-value storage.

Usage

Make sure you enable the “key-value storage” service in the iCloud capability. See Apple’s documenation.


@CloudStorage("readyForAction") var readyForAction: Bool = false
@CloudStorage("numberOfItems") var numberOfItems: Int = 0
@CloudStorage("orientation") var orientation: String?

See also the example app in the repository.

For what should this be used?

The same caveats apply as with key-value storage itself:

Key-value storage is for discrete values such as preferences, settings, and simple app state.
Use iCloud key-value storage for small amounts of data: stocks or weather information, locations, bookmarks, a recent documents list, settings and preferences, and simple game state. Every app submitted to the App Store or Mac App Store should take advantage of key-value storage.

From Apple’s documentation on choosing the proper iCloud Storage API.

In general, key-value storage is not meant as a general purpose syncing service. If you need any advanced capabilities to prevent data loss, consider using CloudKit instead.

Check out the CloudStorage repository.

Hacking “Marzipan” – Running iOS apps on Mac [talk]

A recording from my talk at the Do iOS conference last month.

At WWDC 2018 Apple announced that in the future it will become possible to run iOS apps on macOS. Rumours about this first appeared in December 2017 and was believed to be codenamed Marzipan. Although there is no official API or support for it yet in this talk Tom Lokhorst shows how he experimented with Marzipan and what he learned.

The example project used in the talk is available on GitHub: MarzipanDemoApp.

Leaky abstractions in Swift with DispatchQueue

At Q42 we have some apps with very occasional weird multi-threading issues. After a bunch of debugging Mathijs Kadijk and I figured out it had something to do with DispatchSpecificKey. This post details what we found out.

Pop quiz! What do you think this prints?


while true {
  let key = DispatchSpecificKey<Int>()

  if let value = DispatchQueue.main.getSpecific(key: key) {
    print("ERR \(value)")
    exit(1)
  }
  else {
    DispatchQueue.main.setSpecific(key: key, value: 42)
    print("OK")
    sleep(1)
  }
}

Answer: This does not print a infinite stream of OKs every second, as I would have thought. Instead, it stops after three iterations of the loop:


OK
OK
ERR 42
Program ended with exit code: 1

The output changes a bit each time I run the program. Sometimes it makes it to three or four OKs, but often it only prints one OK and then stops with an ERR.

My question after seeing this behaviour: Wut? How can two distinct keys result in the same value?!?

Wrappers around C APIs

It turns out these (woefully underdocumented) Swift APIs are wrappers around C APIs. The code for this is all open source, see Queue.swift. The underlying C APIs are better documented and give some insight into how this all works internally:

The documentation for dispatch_queue_set_specific explains how the key is just a pointer to something, it doesn’t really matter what it’s pointing to.

Keys are only compared as pointers and are never dereferenced. Thus, you can use a pointer to a static variable for a specific subsystem or any other value that allows you to identify the value uniquely.

For this reason, the Swift implementation of DispatchSpecificKey is very simple:


public final class DispatchSpecificKey<T> {
	public init() {}
}

The initialised object is only used to get a pointer value that can be passed to the dispatch_queue_set_specific function. This pointer should of course be unique (hint: it is not).

Debugging the weird behaviour

Suspecting this behaviour has something to do with the pointer, lets add a print statement to see what the pointer value is:


let p = Unmanaged.passUnretained(key).toOpaque()
print("Pointer: \(p)")

This resulted in the following output:


Pointer: 0x000000010120b590
OK
Pointer: 0x000000010104ab00
OK
Pointer: 0x000000010120b590
ERR 42
Program ended with exit code: 1

Finally, this the explains the behaviour we’re seeing; Different instances of DispatchSpecificKey share the same pointer!

Presumably ARC cleans up the key variable after we’re no longer using it and in a next iteration of the loop, the same memory location is reused again to store a new instance of DispatchSpecificKey.

Possible fixes for the bug

There are different solutions to fix this unexpected behaviour. One is to simply keep an array of each DispatchSpecificKey that gets created, that way no two DispatchSpecificKeys get assigned to the same memory location.

In my own code, I happened to have the key be a member of an object. The solution was to add a deinitializer to the object:


class MyObject {
  let key = DispatchSpecificKey<Int>()

  deinit {
    DispatchQueue.main.setSpecific(key: key, value: nil)
  }
}

A more general solution would be if DispatchSpecificKey were to be updated to clean up after itself:


public final class DispatchSpecificKey<T> {
  public init() {}

  // Is it OK to keep strong references to queues?
  internal var queues: [DispatchQueue] = []

  deinit {
    for queue in queues {
      queue.setSpecific(key: self, value: nil)
    }
  }
}

Closing thoughts

It took us a lot of debugging to finally narrow down the source of our bug to this reusing of the same memory address by two distinct objects. Normally in Swift code this wouldn’t be a problem, but because the underlying C API uses just pointers to compare for equality, this suddenly matters.

For the C code, it is arguable that the programmer should be responsible for cleaning up after they’re done with a C “dispatch specific key”. They should call dispatch_queue_set_specific with a nil value themselves.

But for the Swift code; The DispatchSpecificKey class really implies programmers shouldn’t have to know about the pointer internals of the C API. So in my opinion that class should have the cleanup code build-in.

Xcode 9 + iOS 10 – Preserve Superview Margins bug

Update 2017-08-25: This bug appears to be fixed in Xcode 9, beta 6.


On Xcode 9, when compiling to iOS 10; views nested in root view with “Preserve Superview Margins” enabled, have an incorrect margin of 8px.

iOS Simulator showing 8px margin

I’ve filed a radar with Apple:

ID: 33992313

Summary

Xcode 9 (beta 5) generates incorrect margins when building for iOS 10 with “Preserves Superview Margins” enabled in Interface Builder.
The same code works correct on Xcode 9 + iOS 11, and also correct on Xcode 8 + iOS 10.

Steps to Reproduce

In Xcode 9, beta 5 / iOS 10:

  • Create a “Wrapper View” in interface builder directly under root view of view controller
  • Pin edges to superview edges on Wrapper View
  • Enable “Preserve Superview Margins” on Wrapper View
  • Add subview to Wrapper View that is pinned to the Wrapper View layout margins

Expected Results

Left and right margins should be 16 or 20, depending on device.

Observed Results

Left and right margins are always 8.

Notes

Does not occur when building with Xcode 8 (for iOS 10) or Xcode 9 (for iOS 11)

Side note:
Setting explicit layout margins instead of using “Preserve Superview Margins” only works after rotating screen.
But this is probably an unrelated iOS 10 bug, as this also happens in Xcode 8.

Strongly typed identifiers in Swift

In a lot of our code, we have structs that contain some sort of identifier. This is usually a serverside generated id, that uniquely identifies some record in a database.


struct Person {
  let id: String
  var name: String
  var age: Int?
}

struct Building {
  let id: String
  var title: String
  var owner: Person
}

Strings as identifiers

Sometimes these identifiers are UUIDs or Ints, but mostly they are Strings. Opaque to the client, but meaningful on the server.

Most functions in our codebase work on structs directly. But some use the identifiers, since they exist anyway:


func scrollToPerson(_ person: Person) {
}

func scrollToPerson(withId id: String) {
}

The scrollToPerson(_:) function is strongly typed, but the scrollToPerson(withId:) function isn’t.
This leads to accidentally mistakes, where we use the wrong identifier:


scrollToPerson(withId: mainBuilding.id) // Wrong
scrollToPerson(withId: mainBuilding.owner.id) // Correct

Strongly typed identifiers

We’ve recently begon creating strongly typed structs for these identifiers:


struct Person {
  struct Identifier: RawRepresentable, Hashable, Equatable {
    let rawValue: String
    init(rawValue: String) { self.rawValue = rawValue }
  }

  let id: Identifier
  var name: String
  var age: Int?
}

struct Building {
  struct Identifier: RawRepresentable, Hashable, Equatable {
    let rawValue: String
    init(rawValue: String) { self.rawValue = rawValue }
  }

  let id: Identifier
  var title: String
  var owner: Person
}

With this strongly typed Identifier in place, we no longer can accidentally use the wrong identifier.


func scrollToPerson(withId id: Person.Identifier) {
}

// This causes a type error:
scrollToPerson(withId: mainBuilding.id) 
                                    ^
// Cannot convert value of type 'Building.Identifier' to expected argument type 'Person.Identifier'

We implement the Hashable and Equatable protocols so we can use these identifiers in Sets and as Dictionary keys.
We use RawRepresentable so we get the Equatable implementation for free, Hashable is implemented in a protocol extension:


extension RawRepresentable where RawValue : Hashable {
  public var hashValue: Int { return rawValue.hashValue }
}

Wishlist: newtype

Haskell has a language feature that implements this pattern of wrapping an existing type to create a new type. It is called: newtype.
That would also be nice to have in Swift. It looks similar to typealias, but creates a new type, instead of just an alias. Then we could write:


struct Person {
  newtype Identifier = String

  let id: Identifier
  var name: String
  var age: Int?
}

Alternative: phantom types

As an alternative to creating multiple separate identifier types, we could create a single generic type, using a phantom type:


struct Identifier<T>: RawRepresentable, Hashable, Equatable {
  let rawValue: String
  init(rawValue: String) { self.rawValue = rawValue }
}

The generic type argument T here isn’t used in the Identifier type itself, it is only used to distinguish different identifiers.
When using this generic identifier, the previous example becomes:


struct Person {
  let id: Identifier<Person>
  var name: String
  var age: Int?
}

func scrollToPerson(withId id: Identifier<Person>) {
}

This does have the benefit of being shorter, and thus easier to write. But is it better?
I don’t think this code is easier to read than the previous version with Person.Identifier. In fact, I think it’s harder to read. This is a classic case of optimising for writing code, instead of optimising for reading the code.

To keep code maintainable in the long run, I thing we should strive for readability over writability. So I’ll stick with writing multiple separate types, and waiting for a newtype construct.

Alternative 2: A typealias

(Addition 2017-07-13)

A colleague suggested a second alternative; Using a typealias to keep te readability of scrollToPerson(withId id: Person.Identifier), but also using the generic single definition.


struct GenericIdentifier<T>: RawRepresentable, Hashable, Equatable {
  let rawValue: String
  init(rawValue: String) { self.rawValue = rawValue }
}

struct Person {
  typealias Identifier = GenericIdentifier<Person>

  let id: Identifier
  var name: String
  var age: Int?
}

func scrollToPerson(withId id: Person.Identifier) {
}

Maybe this is a nice middle ground. Although I would throw this GenericIdentifier<T> away, once Swift gains a newtype construct.

Swift + JSON with code generation

Update 2016-01-17: I gave a talk about this topic at the January 2016 meetup of CocoaHeadsNL. See Swift JsonGen [talk].


There are many JSON parsing libraries written in Swift, and equally as many articles about them. This article is not one of those. This article is about a code generation tool, running on NodeJS, written in TypeScript: JsonGen.

JsonGen generates JSON parsers for immutable Swift structs. Get type safety without manually writing a parser.

Background

When we – at Q42 – started work on our first Swift app (September 2014) we ran into an issue: How to get the JSON from API calls into nice, strongly typed, Swift structs. Chris Eidhof’s post on functional JSON parsing sounded appealing, so we tried the Argo library.

As a Haskell developer, Argo’s style looked interesting, although some colleagues were less enthusiastic about all the custom operators. However, before we got to discuss the merits of custom operators, we ran into a bigger issue: Argo didn’t work. When writing more complex parsers, the Swift type inferencer (or some other part of the compiler) kept crashing. I’m not blaming Argo here, I’m sure it’s swiftc that was to blame, and it probably now works with Swift 1.2. Still, it was a problem, so we looked further.

Getting data from JSON

For me, one of the most appealing aspects of working with Swift is type safety. Being able to precisely model all states of a user interface in a Swift enum is huge! This is the whole reason why we want JSON in structs instead of untyped dictionaries.

The native way of reading JSON in Swift is by using dynamic lookup on the dictionary returned by NSJSONSerialization:

let str = "{ \"title\": \"Hello, World!\", \"published\": true, \"author\": { \"first\": \"Tom\", \"last\": \"Lokhorst\" } }"
let data = str.dataUsingEncoding(NSUTF8StringEncoding)!

let obj = try! NSJSONSerialization.JSONObjectWithData(data, options: [])

Dynamic lookup

The result of NSJSONSerialization.JSONObjectWithData is actually a dictionary containing string values and nested dictionaries. By doing dynamic casts you can get to the desired value;

if let dict = obj as? [String: AnyObject],
    author = dict["author"] as? [String: AnyObject],
    firstname = author["first"] as? String,
    lastname = author["last"] as? String {
  
  print("Author: \(firstname) \(lastname)")
}

The reasons why I don’t like this approach are; It is really verbose to deconstruct the dictionary to extract some value. It is very error prone, because all the code is untyped. All the field names are in strings, and the types are just added ad-hoc.

Modeling data as structs

To get more type safety, I model my data with structs:

struct Blog {
  let title: String
  let published: Bool
  let author: Author
}

struct Author {
  let first: String
  let last: String
}

With a library like Argo, I can create a parser (or decoder) that can parse the AnyObject from NSJSONSerialization.JSONObjectWithData into the structs. However the downside is: I have to write a parser, where I again have to write all the field names in strings!

Code generation with JsonGen

Instead of manually creating the decoder, let’s generate it! That way there is less manual work to do, and no room for human-error. (For example: our app has over 50 structs for dealing with JSON, I don’t want to manually write and maintain the parsers for those). With a decoder generated by JsonGen, the use-side code becomes:

if let blog = Blog.decodeJson(obj) {
  let author = blog.author
  print("Author: \(author.first) \(author.last)")
}

The swift-json-gen command line tool will generate new source files for each supplied input file. For each struct an extension is created with a decodeJson method. The tool is smart enough to only generate methods for structs that don’t already have that method. So if you want to customize the behaviour of the decoding for one of your structs, all you have to do is implement the decodeJson method yourself and JsonGen won’t generate a new method.

Example:

  1. Put the Blog and Author structs in a file called Blog.swift
  2. Run the command: swift-json-gen Blog.swift
  3. This will generate Blog+JsonGen.swift that contains static decodeJson methods for the Blog and Author structs.

JsonGen ships with a file containing decoders for standard Swift and Foundation types: JsonGen.swift. These are decoders for types like Bool, Double, String, NSURL, but also generic types like Optional<T> and Array<T>. JsonGen also resolves type aliases and can generate decoders for nested types, as well as generic types.

If there is part of your JSON structure you don’t want to model in Swift structs, you can leave that part untyped by using one of three type aliases from JsonGen.swift: AnyJson, JsonObject or JsonArray. For example, in the following struct, the data field is kept as an untyped AnyObject:

struct Item {
  let id: String
  let name: String?
  let data: AnyJson
}

How it works

JsonGen works by calling swiftc with the -dump-ast flag. With this flag, the Swift compiler parses and type-checks the provided Swift files. But instead of generating code, it then outputs the Abstract Syntax Tree (AST) to stderr in a Lisp-like format. This AST is then parsed, and JsonGen looks for structs. For each struct, all fields and corresponding types are looked up. An extension method is generated that tries to find each field in the dictionary. If found, the value will be decoded using the decoder for the type of the field. If either the lookup or the decoding fails, nil is returned instead of the value being decoded.

The generated decoder returns an optional value. If the AnyObject can’t be decoded to the requested type, nil is returned. In development mode, the debugger is stopped using an assertionFailure so that the developer can figure out what is wrong (either the struct, the JSON, or both).

This is the generated code for the Blog struct from before.

extension Blog {
  static func decodeJson(json: AnyObject) -> Blog? {
    let _dict = json as? [String : AnyObject]
    if _dict == nil { return nil }
    let dict = _dict!

    let title_field: AnyObject? = dict["title"]
    if title_field == nil { assertionFailure("field 'title' is missing"); return nil }
    let title_optional: String? = String.decodeJson(title_field!)
    if title_optional == nil { assertionFailure("field 'title' is not String"); return nil }
    let title: String = title_optional!

    let published_field: AnyObject? = dict["published"]
    if published_field == nil { assertionFailure("field 'published' is missing"); return nil }
    let published_optional: Bool? = Bool.decodeJson(published_field!)
    if published_optional == nil { assertionFailure("field 'published' is not Bool"); return nil }
    let published: Bool = published_optional!

    let author_field: AnyObject? = dict["author"]
    if author_field == nil { assertionFailure("field 'author' is missing"); return nil }
    let author_optional: Author? = Author.decodeJson(author_field!)
    if author_optional == nil { assertionFailure("field 'author' is not Author"); return nil }
    let author: Author = author_optional!

    return Blog(title: title, published: published, author: author)
  }
}

The generated code is simple albeit not very pretty to look at, but hopefully you’ll never have too. The added benefit of generating such simple, straight forward code is; If you should ever decide to stop using JsonGen, you can just continue on by manually editing and maintaining the generated code.

By the way; JsonGen also generates a encodeJson method. The encode method generates a dictionary for any type of struct. This is useful if you want to post data back to a JSON api.

JSON model vs Domain model

When decoding JSON into a domain model, you quickly run into the issue that the JSON structure doesn’t quite match the structure of the domain model. The generated decoder isn’t good enough and you need to customize. As said before, you can manually specify decoders for certain specific types. But, it can be quite a lot of work to create a decoder for a whole type. However, I think this is often not the approach you will want to take.

In my opinion there are two kinds of JSON in an app; It is either from a source that is completely under my control, or the source is external and the JSON format is out of my control. In the first case; I try to have the JSON format match as closely as possible to my domain model. For the few differences between the JSON and the domain model, I write custom decoders. That way I can directly parse JSON into the structs of my domain model.

In the case that the JSON format is out of my control, the format is often weird and doesn’t match my domain model. It has strange internal rules and custom logic. I don’t want to write all that custom logic and those business rules in the untyped world of decoders. I want to use the type checker to tell me if I’ve forgotten to implement an enum case, or if I assume a optional field is always available.

To achieve this, I create two sets of types; One set of structs that exactly matches the raw JSON, and another set of structs and enums that are my domain model. For the JSON structs, I generate decoders and encoders. But for the domain model, I manually write the mapping from the JSON structs to the domain structs. This may seem tedious, but it is necessitated by the fact that that mapping code is full of custom business logic. That business logic is code I want to write and maintain in the world of types.

This abbreviated example demonstrates this approach. The domain model is the Profile struct. The JSON model is the ProfileJson struct:

/* Domain model */

// A Profile consists of a first name and an optional avatar
struct Profile {
  let firstName: String
  let avatarURL: NSURL?
}

/* JSON model */

// This is the format of JSON data returned by an API.
// The `profile` property maps this JSON model to the domain model.
// The JSON parser for this type is generated by swift-json-gen.
struct ProfileJson {
  let FirstName: String
  let UseAvatar: Bool
  let AvatarUrl: String?

  var profile: Profile {
    // These are the business rules for handling the JSON data:
    // 1. When UseAvatar == false, ignore the url
    // 2. If AvatarUrl is not a real url, ignore it

    let avatarURL: NSURL?
    if let url = AvatarUrl where UseAvatar {
      avatarURL = NSURL(string: url)
    }
    else {
      avatarURL = nil
    }

    return Profile(firstName: FirstName, avatarURL: avatarURL)
  }
}

(See this gist for a more slightly more detailed example)

Note that there is no need to completely “duplicate” all structs in the domain model into a JSON model. Sometimes the domain happens to match the JSON, in that case I only use the domain model. This happens a lot at the “leaves” of data structures. It is the roots that differ and contain a lot of custom mapping logic. In that case, the root structs exist in both the JSON model and the domain model and the leaves are shared.

Conclusion

If you want type safety, and don’t want to do dynamic lookup, you don’t have to manually write JSON parsers for your structs.
You can use JsonGen to generate decoders and encoders for your Swift structs. Let the untyped code be generated and only work in the nice, strongly typed world of Swift!

Install and use swift-json-gen like so:

$ npm install -g swift-json-gen
$ swift-json-gen MyDirectory/SubDirectory/

Or get the code from GitHub: tomlokhorst/swift-json-gen

This tool is used in a few projects at Q42, but it hasn’t been battle tested outside of our company. If you use it and find it useful, please let me know: @tomlokhorst.