Full screen video in Mac Catalyst

If you’ve tried to make a video full screen in a Mac Catalyst app (for example Apple’s Developer app), you may have noticed that the video only maximises to the size of the window.

As can be seen in the iPad screenshot, it’s not just macOS. On iOS videos are also restricted to the size of the window.

However on macOS, we do want to be able to play video full screen, often because we have multiple monitors and want to dedicate one to displaying video.

Fortunately, with the help of some private APIs we can maximise the window at the same time as the video, so we can have a true “full screen” experience.

Maximising the Window

By maximising the window at the same time as the AVPlayerViewController, we can get the desired behaviour. The video is playing full screen, but you can still switch to other windows using Mission Control.

Switching the video tool full screen triggers the window to resize

As the video shows, this approach is far from perfect, there’s some stuttering and the animations look crap. But it’s a step in the right direction!

Implementation

I’ve implemented the resizing in two ways:

  1. Switching the video to full screen resizes the window
  2. Maximising the window, triggers the video to go full screen

Resizing the window can be done using some documented AppKit APIs, but going full screen with the AVPlayerController requires some private APIs. These are the APIs used:

  1. NSWindowWillEnterFullScreenNotification / NSWindowWillExitFullScreenNotification notifications to monitor window resizes
  2. playerViewController(_:willBeginFullScreenPresentationWithAnimationCoordinator:) to monitor player resizes
  3. NSWindow.toggleFullScreen to resize the window
  4. _transitionToFullScreen / _transitionFromFullScreen private APIs to resize the AVPlayerViewController

There’s multiple ways of implementing this, you can add a new AppKit bundle to your Catalyst UIKit app to call these APIs. Or you can use NSSelector and performSelector with strings to call these APIs. But I prefer using the Dynamic library.

The Dynamic library is a very convenient way of calling AppKit methods from Catalyst. It’s basically a wrapper around the NSSelector and performSelector methods, using Swift’s @dynamicMemberLookup and @dynamicCallable methods.

For all code, see the Github project.

Server Driven UI

This post is based on a talk I gave at CocoaHeadsNL in July 2020. Warning: because this is based on a transcription of a talk, sentences and wording may be weird or incoherent. The original talk, including a live Q&A afterwards, can be seen here:

Over the years, I’ve build many different types of apps. However in one aspect they’ve all been very similar; In the client/server architecture, the server sends domain objects to the client (encoded in JSON), and the client renders these domain objects to some pretty UI.

Server Driven UI is different. The server does not send domain objects, with the client having to decide how to render those. Instead, the server decides what and how to render, and just sends instructions to the client. You know, kinda like HTML…

(Honestly, it’s not HTML… But it sort of is. But really, it isn’t)

Continue reading “Server Driven UI”

@CloudStorage property wrapper

I wrote a Swift library to help with prototyping (haven’t used it yet in production apps).

CloudStorage is property wrapper that will save values in iCloud key-value storage. These values are persisted across app restarts and will sync between different devices that are signed into the same iCloud account. This property wrapper is similar to AppStorage and SceneStorage, two new types Apple introduced with iOS 14.

The basic API is the same as for AppStorage; add @CloudStorage("someName") in front of a property, to save the value to iCloud key-value storage.

Usage

Make sure you enable the “key-value storage” service in the iCloud capability. See Apple’s documenation.


@CloudStorage("readyForAction") var readyForAction: Bool = false
@CloudStorage("numberOfItems") var numberOfItems: Int = 0
@CloudStorage("orientation") var orientation: String?

See also the example app in the repository.

For what should this be used?

The same caveats apply as with key-value storage itself:

Key-value storage is for discrete values such as preferences, settings, and simple app state.
Use iCloud key-value storage for small amounts of data: stocks or weather information, locations, bookmarks, a recent documents list, settings and preferences, and simple game state. Every app submitted to the App Store or Mac App Store should take advantage of key-value storage.

From Apple’s documentation on choosing the proper iCloud Storage API.

In general, key-value storage is not meant as a general purpose syncing service. If you need any advanced capabilities to prevent data loss, consider using CloudKit instead.

Check out the CloudStorage repository.

My wish for watchOS: Dutch-style Cycling support

Screenshot of Outdoor Cycle activity on Apple Watch

Currently, watchOS has a workout called “Outdoor Cycle”, the icon shows a person on a racing bicycle. But that is not what cycling means, for millions of people who use a utility bicycle for daily transportation.

Instead of specific activity, that some people do, some of the time. Cycling can also be a boring part of everyday life, that everyone does, all of the time.

How cycling in The Netherlands is different

Here I’m going to make some assumptions and generalisations about places I don’t live… When I think of cycling in some other places, like say California, I think of something like this:

Here we see people, hunched over on racing or road bikes. They go fast, wear special protective clothing and are are really working out. More importantly, culturally these people are doing something abnormal, in the sense that cycling is not the norm, most people don’t cycle regularly.

In contrast, this is a typical Dutch view of cycling:

In The Netherlands, cycling is a common mode of transport. 27% of all trips are made by bicycle. In Amsterdam, a third of morning commutes are made by bicycle, making it the most popular form of transport (above walking and cars). At first glance you can see the differences: people sit upright, drive slowly, and they are wearing ordinary clothing. The bicycles themselves are also different; They are relatively cheap, have a kickstand, a chain guard and angled-back handlebars. They also include one or two luggage carriers. 

Side note; It’s not that the other type of bicycle doesn’t exist in The Netherlands. We too have people who (in the weekends), put on a helmet, dress up in lycra, and take out their other sporting bicycle. To do competitive racing, or mountain biking in the woods. But those are recreational sports, not day-to-day transport.

My which for Apple Watch: Support for Utility Cycling

I would like it if watchOS 7 added an additional option. To support this normal, everyday, utility cycling. It seems a bit weird to call this a “workout”, but hey “outdoor walking” is also in there, so what the heck. I even designed an icon for this:

Screenshot of Apple Watch Workouts app with hypothetical Utility Cycle activity

In a similar way to Outdoor walking, this activity should be automatically detected and activated. This could be used to more accurately track activity when commuting to work, taking the kids to school or grocery shopping. And while you’re at it Apple, please also add the cycling modality to Apple Maps!


PS: For those unfamiliar, here’s a little mood impression of what cycling in The Netherlands looks like:

(Also, do check out the rest of the rest of BicycleDutch channel on YouTube. It’s great!)

Every two years, Apple increases software support by a year

Over the last 12 years, a pattern has emerged: Every two years Apple has adds an extra year of iOS support to its System-on-chips (SoCs).

The original iPhone and iPhone 3G both had three years of software updates. The iPhone 3GS and iPhone 4 had four years of updates, and so on.

This diagram visualises that pattern:
iOS versions per SoC. 2007 and 2008: 3 years. 2009 and A4: 4 years. A5 and A6: 5 years. A7 and A8: 6 years. A9 and A10: 7 years? A11 and A12: 8 years?

The coloured lines indicate iOS support a specific SoC. The grey line indicates when devices with this SoC were sold.

Note that this diagram specifically plots SoCs, and not iPhone releases. The same A-series SoCs are used across iPhones, iPads and iPod touches. I’ve merged the releases of several of these devices to create this diagram.

Should this trend continue in the future, the A8 will be supported for 6 years. The A9/A10 for 7 years, and the A11/A12 for 8 years.

That means the iPhone XS released in September 2018, which runs the A12 Bionic, would get updates until 2025, with iOS 19.

The iPhone 6 exception

Again, the above diagram refers to SoCs in general, and not iPhones, because if we plot that, there’s an exception for the iPhone 6.
iOS versions per iPhone. iPhone 6 is only supported for 5 years, but it's SoC is supported for 6 years.

The coloured lines indicate support iOS support for an iPhone. The grey line indicates when the iPhone sold.

The iPhone 6 was released in September 2014 and was for sale until March 2017.

In September 2019, when iOS 13 comes out, the iPhone 6 will no longer be supported. This is unfortunate because a lot of users are still using iPhone 6 devices. At Q42 we see it is the 6th most popular iPhone with 7% usage, more popular than the iPhone SE or iPhone XR.

Although support for the iPhone 6 (and iPod touch 6th gen) is being dropped, the iPad Air 2 and iPad mini 4 of the same generation are still supported. The iPad mini 4 uses the same A8 SoC as the iPhone 6.

A possible explanation for why it is being dropped might be the size of the RAM. iPhone 6 uses an A8 SoC, but it only has 1GB of RAM. The iPads both have 2GB of RAM.

Every device supported by iOS 13, has at least 2GB of RAM.

The future?

It is impossible to predict the future, but I don’t think this “RAM filter” will occur again soon.

The next obvious step would be some future version of iOS that only supports 3GB RAM. But that would cut product lines in half. iPhone 7 and 8 both have 2GB or RAM, but the iPhone 7 Plus and 8 Plus use 3 GB.

The step after that is 4GB as a minimum, but that includes some very recent devices. The iPhone XR (September 2018) and iPad Air 3 (March 2019) both ship with 3GB of RAM.

My assumption is that, for the coming years, device deprecations will again be solely based on SoC.

So…

Will this trend of adding a support year continue indefinitely? If not, when will it stop? Will there be another iPhone 6-style hiccup? Who knows!

I can’t wait for WWDC 2020, to see what iOS 14 brings!


All data about SoCs and release years are taken from Wikipedia. This chart lists all iOS-like devices from the past 12 years.

Hacking “Marzipan” – Running iOS apps on Mac [talk]

A recording from my talk at the Do iOS conference last month.

At WWDC 2018 Apple announced that in the future it will become possible to run iOS apps on macOS. Rumours about this first appeared in December 2017 and was believed to be codenamed Marzipan. Although there is no official API or support for it yet in this talk Tom Lokhorst shows how he experimented with Marzipan and what he learned.

The example project used in the talk is available on GitHub: MarzipanDemoApp.

Machine Learning + AR at the Rijksmuseum

For the past four months Daniello, an intern at Q42, has worked on training a machine learning model to recognise artworks in the Rijksmuseum. Once recognised, the paintings are located in the AR camera feed and augmented using ARKit.

This week at WWDC, Apple announced CreateML and ARKit 2. Those two solutions combined may make this process a lot easier, but nevertheless, it’s really cool to see what Daniello build:

Art recogniser Rijksmuseum from Q42.

Rijksmuseum ArtViewer [talk]

A recording from my talk at CocoaHeadsNL last month

Tom Lokhorst from Q42 talks about how they developed the ArtViewer for the Rijksmuseum app. This viewer is a highly optimised image viewer to display very large images. It efficiently uses caching and tiling to save memory and bandwidth.

Rijksmuseum ArtViewer, Tom Lokhorst from CocoaHeadsNL.

See the Rijksmuseum app in the App Store.

Leaky abstractions in Swift with DispatchQueue

At Q42 we have some apps with very occasional weird multi-threading issues. After a bunch of debugging Mathijs Kadijk and I figured out it had something to do with DispatchSpecificKey. This post details what we found out.

Pop quiz! What do you think this prints?


while true {
  let key = DispatchSpecificKey<Int>()

  if let value = DispatchQueue.main.getSpecific(key: key) {
    print("ERR \(value)")
    exit(1)
  }
  else {
    DispatchQueue.main.setSpecific(key: key, value: 42)
    print("OK")
    sleep(1)
  }
}

Answer: This does not print a infinite stream of OKs every second, as I would have thought. Instead, it stops after three iterations of the loop:


OK
OK
ERR 42
Program ended with exit code: 1

The output changes a bit each time I run the program. Sometimes it makes it to three or four OKs, but often it only prints one OK and then stops with an ERR.

My question after seeing this behaviour: Wut? How can two distinct keys result in the same value?!?

Wrappers around C APIs

It turns out these (woefully underdocumented) Swift APIs are wrappers around C APIs. The code for this is all open source, see Queue.swift. The underlying C APIs are better documented and give some insight into how this all works internally:

The documentation for dispatch_queue_set_specific explains how the key is just a pointer to something, it doesn’t really matter what it’s pointing to.

Keys are only compared as pointers and are never dereferenced. Thus, you can use a pointer to a static variable for a specific subsystem or any other value that allows you to identify the value uniquely.

For this reason, the Swift implementation of DispatchSpecificKey is very simple:


public final class DispatchSpecificKey<T> {
	public init() {}
}

The initialised object is only used to get a pointer value that can be passed to the dispatch_queue_set_specific function. This pointer should of course be unique (hint: it is not).

Debugging the weird behaviour

Suspecting this behaviour has something to do with the pointer, lets add a print statement to see what the pointer value is:


let p = Unmanaged.passUnretained(key).toOpaque()
print("Pointer: \(p)")

This resulted in the following output:


Pointer: 0x000000010120b590
OK
Pointer: 0x000000010104ab00
OK
Pointer: 0x000000010120b590
ERR 42
Program ended with exit code: 1

Finally, this the explains the behaviour we’re seeing; Different instances of DispatchSpecificKey share the same pointer!

Presumably ARC cleans up the key variable after we’re no longer using it and in a next iteration of the loop, the same memory location is reused again to store a new instance of DispatchSpecificKey.

Possible fixes for the bug

There are different solutions to fix this unexpected behaviour. One is to simply keep an array of each DispatchSpecificKey that gets created, that way no two DispatchSpecificKeys get assigned to the same memory location.

In my own code, I happened to have the key be a member of an object. The solution was to add a deinitializer to the object:


class MyObject {
  let key = DispatchSpecificKey<Int>()

  deinit {
    DispatchQueue.main.setSpecific(key: key, value: nil)
  }
}

A more general solution would be if DispatchSpecificKey were to be updated to clean up after itself:


public final class DispatchSpecificKey<T> {
  public init() {}

  // Is it OK to keep strong references to queues?
  internal var queues: [DispatchQueue] = []

  deinit {
    for queue in queues {
      queue.setSpecific(key: self, value: nil)
    }
  }
}

Closing thoughts

It took us a lot of debugging to finally narrow down the source of our bug to this reusing of the same memory address by two distinct objects. Normally in Swift code this wouldn’t be a problem, but because the underlying C API uses just pointers to compare for equality, this suddenly matters.

For the C code, it is arguable that the programmer should be responsible for cleaning up after they’re done with a C “dispatch specific key”. They should call dispatch_queue_set_specific with a nil value themselves.

But for the Swift code; The DispatchSpecificKey class really implies programmers shouldn’t have to know about the pointer internals of the C API. So in my opinion that class should have the cleanup code build-in.