Hacking “Marzipan” – Running iOS apps on Mac [talk]

A recording from my talk at the Do iOS conference last month.

At WWDC 2018 Apple announced that in the future it will become possible to run iOS apps on macOS. Rumours about this first appeared in December 2017 and was believed to be codenamed Marzipan. Although there is no official API or support for it yet in this talk Tom Lokhorst shows how he experimented with Marzipan and what he learned.

The example project used in the talk is available on GitHub: MarzipanDemoApp.

Machine Learning + AR at the Rijksmuseum

For the past four months Daniello, an intern at Q42, has worked on training a machine learning model to recognise artworks in the Rijksmuseum. Once recognised, the paintings are located in the AR camera feed and augmented using ARKit.

This week at WWDC, Apple announced CreateML and ARKit 2. Those two solutions combined may make this process a lot easier, but nevertheless, it’s really cool to see what Daniello build:

Art recogniser Rijksmuseum from Q42.

Rijksmuseum ArtViewer [talk]

A recording from my talk at CocoaHeadsNL last month

Tom Lokhorst from Q42 talks about how they developed the ArtViewer for the Rijksmuseum app. This viewer is a highly optimised image viewer to display very large images. It efficiently uses caching and tiling to save memory and bandwidth.

Rijksmuseum ArtViewer, Tom Lokhorst from CocoaHeadsNL.

See the Rijksmuseum app in the App Store.

Leaky abstractions in Swift with DispatchQueue

At Q42 we have some apps with very occasional weird multi-threading issues. After a bunch of debugging Mathijs Kadijk and I figured out it had something to do with DispatchSpecificKey. This post details what we found out.

Pop quiz! What do you think this prints?


while true {
  let key = DispatchSpecificKey<Int>()

  if let value = DispatchQueue.main.getSpecific(key: key) {
    print("ERR \(value)")
    exit(1)
  }
  else {
    DispatchQueue.main.setSpecific(key: key, value: 42)
    print("OK")
    sleep(1)
  }
}

Answer: This does not print a infinite stream of OKs every second, as I would have thought. Instead, it stops after three iterations of the loop:


OK
OK
ERR 42
Program ended with exit code: 1

The output changes a bit each time I run the program. Sometimes it makes it to three or four OKs, but often it only prints one OK and then stops with an ERR.

My question after seeing this behaviour: Wut? How can two distinct keys result in the same value?!?

Wrappers around C APIs

It turns out these (woefully underdocumented) Swift APIs are wrappers around C APIs. The code for this is all open source, see Queue.swift. The underlying C APIs are better documented and give some insight into how this all works internally:

The documentation for dispatch_queue_set_specific explains how the key is just a pointer to something, it doesn’t really matter what it’s pointing to.

Keys are only compared as pointers and are never dereferenced. Thus, you can use a pointer to a static variable for a specific subsystem or any other value that allows you to identify the value uniquely.

For this reason, the Swift implementation of DispatchSpecificKey is very simple:


public final class DispatchSpecificKey<T> {
	public init() {}
}

The initialised object is only used to get a pointer value that can be passed to the dispatch_queue_set_specific function. This pointer should of course be unique (hint: it is not).

Debugging the weird behaviour

Suspecting this behaviour has something to do with the pointer, lets add a print statement to see what the pointer value is:


let p = Unmanaged.passUnretained(key).toOpaque()
print("Pointer: \(p)")

This resulted in the following output:


Pointer: 0x000000010120b590
OK
Pointer: 0x000000010104ab00
OK
Pointer: 0x000000010120b590
ERR 42
Program ended with exit code: 1

Finally, this the explains the behaviour we’re seeing; Different instances of DispatchSpecificKey share the same pointer!

Presumably ARC cleans up the key variable after we’re no longer using it and in a next iteration of the loop, the same memory location is reused again to store a new instance of DispatchSpecificKey.

Possible fixes for the bug

There are different solutions to fix this unexpected behaviour. One is to simply keep an array of each DispatchSpecificKey that gets created, that way no two DispatchSpecificKeys get assigned to the same memory location.

In my own code, I happened to have the key be a member of an object. The solution was to add a deinitializer to the object:


class MyObject {
  let key = DispatchSpecificKey<Int>()

  deinit {
    DispatchQueue.main.setSpecific(key: key, value: nil)
  }
}

A more general solution would be if DispatchSpecificKey were to be updated to clean up after itself:


public final class DispatchSpecificKey<T> {
  public init() {}

  // Is it OK to keep strong references to queues?
  internal var queues: [DispatchQueue] = []

  deinit {
    for queue in queues {
      queue.setSpecific(key: self, value: nil)
    }
  }
}

Closing thoughts

It took us a lot of debugging to finally narrow down the source of our bug to this reusing of the same memory address by two distinct objects. Normally in Swift code this wouldn’t be a problem, but because the underlying C API uses just pointers to compare for equality, this suddenly matters.

For the C code, it is arguable that the programmer should be responsible for cleaning up after they’re done with a C “dispatch specific key”. They should call dispatch_queue_set_specific with a nil value themselves.

But for the Swift code; The DispatchSpecificKey class really implies programmers shouldn’t have to know about the pointer internals of the C API. So in my opinion that class should have the cleanup code build-in.

Xcode 9 + iOS 10 – Preserve Superview Margins bug

Update 2017-08-25: This bug appears to be fixed in Xcode 9, beta 6.


On Xcode 9, when compiling to iOS 10; views nested in root view with “Preserve Superview Margins” enabled, have an incorrect margin of 8px.

iOS Simulator showing 8px margin

I’ve filed a radar with Apple:

ID: 33992313

Summary

Xcode 9 (beta 5) generates incorrect margins when building for iOS 10 with “Preserves Superview Margins” enabled in Interface Builder.
The same code works correct on Xcode 9 + iOS 11, and also correct on Xcode 8 + iOS 10.

Steps to Reproduce

In Xcode 9, beta 5 / iOS 10:

  • Create a “Wrapper View” in interface builder directly under root view of view controller
  • Pin edges to superview edges on Wrapper View
  • Enable “Preserve Superview Margins” on Wrapper View
  • Add subview to Wrapper View that is pinned to the Wrapper View layout margins

Expected Results

Left and right margins should be 16 or 20, depending on device.

Observed Results

Left and right margins are always 8.

Notes

Does not occur when building with Xcode 8 (for iOS 10) or Xcode 9 (for iOS 11)

Side note:
Setting explicit layout margins instead of using “Preserve Superview Margins” only works after rotating screen.
But this is probably an unrelated iOS 10 bug, as this also happens in Xcode 8.

Strongly typed identifiers in Swift

In a lot of our code, we have structs that contain some sort of identifier. This is usually a serverside generated id, that uniquely identifies some record in a database.


struct Person {
  let id: String
  var name: String
  var age: Int?
}

struct Building {
  let id: String
  var title: String
  var owner: Person
}

Strings as identifiers

Sometimes these identifiers are UUIDs or Ints, but mostly they are Strings. Opaque to the client, but meaningful on the server.

Most functions in our codebase work on structs directly. But some use the identifiers, since they exist anyway:


func scrollToPerson(_ person: Person) {
}

func scrollToPerson(withId id: String) {
}

The scrollToPerson(_:) function is strongly typed, but the scrollToPerson(withId:) function isn’t.
This leads to accidentally mistakes, where we use the wrong identifier:


scrollToPerson(withId: mainBuilding.id) // Wrong
scrollToPerson(withId: mainBuilding.owner.id) // Correct

Strongly typed identifiers

We’ve recently begon creating strongly typed structs for these identifiers:


struct Person {
  struct Identifier: RawRepresentable, Hashable, Equatable {
    let rawValue: String
    init(rawValue: String) { self.rawValue = rawValue }
  }

  let id: Identifier
  var name: String
  var age: Int?
}

struct Building {
  struct Identifier: RawRepresentable, Hashable, Equatable {
    let rawValue: String
    init(rawValue: String) { self.rawValue = rawValue }
  }

  let id: Identifier
  var title: String
  var owner: Person
}

With this strongly typed Identifier in place, we no longer can accidentally use the wrong identifier.


func scrollToPerson(withId id: Person.Identifier) {
}

// This causes a type error:
scrollToPerson(withId: mainBuilding.id) 
                                    ^
// Cannot convert value of type 'Building.Identifier' to expected argument type 'Person.Identifier'

We implement the Hashable and Equatable protocols so we can use these identifiers in Sets and as Dictionary keys.
We use RawRepresentable so we get the Equatable implementation for free, Hashable is implemented in a protocol extension:


extension RawRepresentable where RawValue : Hashable {
  public var hashValue: Int { return rawValue.hashValue }
}

Wishlist: newtype

Haskell has a language feature that implements this pattern of wrapping an existing type to create a new type. It is called: newtype.
That would also be nice to have in Swift. It looks similar to typealias, but creates a new type, instead of just an alias. Then we could write:


struct Person {
  newtype Identifier = String

  let id: Identifier
  var name: String
  var age: Int?
}

Alternative: phantom types

As an alternative to creating multiple separate identifier types, we could create a single generic type, using a phantom type:


struct Identifier<T>: RawRepresentable, Hashable, Equatable {
  let rawValue: String
  init(rawValue: String) { self.rawValue = rawValue }
}

The generic type argument T here isn’t used in the Identifier type itself, it is only used to distinguish different identifiers.
When using this generic identifier, the previous example becomes:


struct Person {
  let id: Identifier<Person>
  var name: String
  var age: Int?
}

func scrollToPerson(withId id: Identifier<Person>) {
}

This does have the benefit of being shorter, and thus easier to write. But is it better?
I don’t think this code is easier to read than the previous version with Person.Identifier. In fact, I think it’s harder to read. This is a classic case of optimising for writing code, instead of optimising for reading the code.

To keep code maintainable in the long run, I thing we should strive for readability over writability. So I’ll stick with writing multiple separate types, and waiting for a newtype construct.

Alternative 2: A typealias

(Addition 2017-07-13)

A colleague suggested a second alternative; Using a typealias to keep te readability of scrollToPerson(withId id: Person.Identifier), but also using the generic single definition.


struct GenericIdentifier<T>: RawRepresentable, Hashable, Equatable {
  let rawValue: String
  init(rawValue: String) { self.rawValue = rawValue }
}

struct Person {
  typealias Identifier = GenericIdentifier<Person>

  let id: Identifier
  var name: String
  var age: Int?
}

func scrollToPerson(withId id: Person.Identifier) {
}

Maybe this is a nice middle ground. Although I would throw this GenericIdentifier<T> away, once Swift gains a newtype construct.

Rewrite code to make it stronger

From The Incomparable podcast episode 354: Sons of Caledonia.
Dan Moren talking about writing a novel (audio):

The amount of time you spent writing that first novel… rewriting it is going to be more, probably.
[…]

It’s like playing Jenga, right? It’s like: “Oh my God, if I take this brick out right here, will this whole thing fall over?”
[…]

It’s trying to get yourself out of this mindset of: “Oh, my story is this delicate spiderweb, and if I break this one strand, the whole thing will fall apart.”
And trying to get yourself more in the mindset of: “My story is like a piece of iron, that is being worked in a forge.”

Where you’re hammering on it to make it tempered, to make it stronger than the thing it was originally.
Like you take a sword, and it’s all about folding the metal, hammering it, and trying to reinforce it. To the point where it’s not going break immediately. You can hammer on something pretty hard. To try to sort of work in the right shape.

That’s what you’re going for, when you’re rewriting. Otherwise, you just sort of get paralysed with fear that anything that you change will break everything. That’s not the case, stories are pretty resilient in that way.
And you can always fix it, is the good news.

This really resonated with me when I heard it. I think this applies equally to writing code as it does to writing in general.

Writing code is like writing a novel. The first iteration of the code that passes the unit tests, is just the very first draft. Now you can start rewriting.

“Jump to Definition”-shortcut not accessible

The “Jump to Definition”-shortcut is not accessible for left-handed users in Xcode 9. There’s no Ctrl key on the right hand side.

Photo of left hand on Macbook trackpad, with right hand crossing above to press ctrl-key on left hand side

I’ve filed a radar with Apple:

ID: 33004913

Summary

In Xcode 9, “Jump to Definition” shortcut is now Ctrl+Cmd+click (was Cmd+click in Xcode 8). On a MacBook keyboard, there’s no Ctrl key on the right hand side. This makes the shortcut not accessible for left-handed users.

Steps to Reproduce

Use MacBook trackpad with left hand.
Hold Ctrl+Cmd down with right hand, while clicking source code token with left hand.

Expected Results

Easy access to “Jump to Definition”-shortcut.

Observed Results

On a MacBook, it is very difficult to hold Ctrl+Cmd with right hand, while clicking with left hand.

This requires crossing hands over each other.

Version

Xcode 9 beta 1

Notes

Suggestion: Use Cmd+Option+click instead.
Option button is available on left and right hand side of keyboard.

Configuration

Relevant to at least US and International Dutch keyboards.