Will Richardson

Me Twitter GitHub

Scrutinising a Scalable Programming Language

In my opinion, most developers should know three different programming languages, three different tools that can be used for most of the projects you come across. These should include a scripting language - for automating a task or doing something quickly; an application language - for writing user-facing or client applications, which have to be run on systems that you don’t control; and an “infrastructure” language for building a system that has many components and runs on your own platform.

I don’t really do any systems programming - which I see as basically being dominated by C anyway - so I would lump it in with your application language, as the situation is very similar. If this is not the case then the target hardware and tooling probably dictates using C anyway.

A scripting language is usually dynamically typed and interpreted. It starts up fast and lets you cut as many corners as needed until the problem is solved. It should be good at dealing with files, text, and simple HTTP requests. For this I will usually use Ruby, as it is very quick to write and has useful shortcuts for common things scripts do - like running shell commands. On occasion I will use Python, but I can’t think of any reason that I would use it over Ruby - except for Ruby not being installed.

The application language I use most often is Java (or Kotlin, more recently). This is partly due to having to use it for university, but also if you want to write something that can be run on pretty much anyones computer with minimal effort, the JVM is probably your best bet. It has its problems, of course - but it’s easier to get someone to install a JRE and run a fat JAR than it would be to ensure they’re running the correct version of Python and the dependencies are installed.

The “infrastructure” language is probably closest to what I find most interesting - I first thought of this after finishing the hackathon that resulted in Herbert. Herbert is a Slackbot that does some timesheeting. It includes the proper Slack app integration hooks and stuff, so it needs to be able to join many different teams at once, and dynamically join teams as they add the integration. It’s written entirely in Ruby, because it was easy to get a working prototype (see above) - but because Ruby doesn’t have real threads, running a load of things at the same time is a bit orthogonal to it’s strengths. In subsequent hackathons, Logan and I chose to use Elixir, which is better suited to piecing a selection of bits together and having them communicate (thanks, BEAM!) so it would probably be my choice here.


I expect that most people - like me - would choose three fairly different languages. What if we had a language that could be used easily for all of these situations? What would it take, and what (in my opinion) is stopping existing languages from being used everywhere?

Most of these languages can be used in all situations, but there are some things that get in the way of them being a great choice. If you are an expert in one particular language then the familiarity will likely win out over the suitability of another.

Go

Go excels at building highly concurrent and performant applications. What makes it even more useful is the ability to easily cross compile to pretty much every major platform. Not having exceptions makes it easy to cut corners in error handling - which makes building an initial prototype or experiment easier. For me the verbose syntax and lack of (seemingly simple) language features pushes me away from Go - but I’ll happily reap the benefits of others using it by making use of their applications.

The other aspect of Go that I find frustrating is the enforced package structure and (lack of) dependency management. There is a fair bit of work from just using go run on a single file to setting up a proper Go environment - especially if you have an existing way of organising your projects.

Java/ Other JVM Languages

The main pitfall of using the JVM for quick tasks is that it usually takes longer for the JVM to start than it does for your program to run. In the time that it takes to start a Clojure REPL, I can typically open a few other terminals and launch a Ruby, Python, and Elixir REPL.

Plain Java also lacks a build system - who in their right mind would use javac to compile multiple files? No simple script should start by writing a pom.xml file. Kotlin is a lot closer to being used for scripting, but still has the problem of slow JVM startup.

Python & Ruby

Both are great scripting languages, and often get paired with native extensions that increase performance without you having to step down to a lower-level language yourself. However I find that the native extensions or systems that manage the running of your code (Passenger, etc) hard to understand and deploy. I like the idea of being able to run the application completely in production mode locally, which is often not really the case for these types of systems. For example, Rails is usually run in production through a Rack-based server that lets your application handle concurrent requests easily, however in development it just uses a simple single-threaded server.

This makes deployment more difficult, and Pythons versioning fiasco doesn’t help. I once wrote a simple script to run on my Raspberry Pi, and because I couldn’t get the correct version of Pip installed to load the dependencies onto the Pi I just reimplemented the same script in Go and cross-compiled it.

JavaScript & Node

I don’t have a lot of experience with JavaScript, but it is one of the few languages that is probably being used in almost every scenario. Quick scripts using Node, applications either in Electron or even just as a web page, and potentially back to Node for your infrastructure. However the ecosystem and build tools are not really to my liking (to say the least), so I will rarely use Node for anything unless I really have to.

Swift

Swift has the potential to be a great option for solving all kinds of problems, but the focus on iOS and macOS development hinders it in the areas that I am interested in - backend development on Linux. This fragments libraries and frameworks, some depend on being built by XCode and use Foundation or Cocoa, others use the Swift Package Manager and use third-party implementations of what you would expect from a standard library. For example, I don’t know how to open a file in Swift. I know how to do it using Foundation, but not on Linux - or maybe that’s changed in the latest version?

String handling makes Swift awkward to use for scripting - indexing can only be done using a Index rather than an Int. This makes sense - indexing a UTF-8 string is an O(n) operation as characters can be pretty much any number of bytes. It does mean that you end up with a spaghetti of string.startIndex and string.index(after: idx).

Elixir

I find Elixir great for making systems with many different components - which isn’t a surprise given the concurrent focus of Erlang and the BEAM VM. The tooling for Elixir is also great, which makes it a pleasure to develop with. Creating a new project, pulling in some dependencies, building a system, and deploying it is well supported for a new language.

The tooling is necessary - working with multiple files isn’t really possible outside of a Mix (the Elixir build system) project, as Mix makes a load of bits and bobs that allows BEAM to put all the code together.

I would never imagine building something designed to run on a computer that’s not my own with Elixir - having someone install Erlang or bundling all of it into a platform-specific binary seems a bit flaky. Also the type of code you write for a client-side application usually isn’t suited for Elixir.

Crystal

Crystal is an interesting new language - it’s still in alpha. What makes it interesting is it has global type inference; instead of just inferring the type for variable assignments based on the return type of functions, it takes into account the usage of every function to determine what types the arguments and result are.

What this means is that Crystal code can be written almost like a dynamically typed programming language, but any type errors will get caught at compile time - basically eliminating typos (ha! type-os!). Obviously this makes Crystal a great language for quick development.

Static typing should make Crystal quite well suited to building larger systems. Of course this is dependent on how it continues to evolve and its ecosystem continues expanding and maturing.


So what would make an ideal scalable language? Something that can be used in any scenario, to build any kind of application. This is a stretch - especially if you want it to be any good at any of one of the problems. For me this basically boils down to readable, concise syntax that allows me to write code that fits in with the standard library; often this includes some solid compile-time checks, catching problems before the code is run, but without adding too much extra overhead to writing the code.

The tools in the language need to make it easy to go from a single script, to a few files, to a full on project with dependencies, tests, etc. Of course as soon as dependencies are involved you need a good ecosystem of libraries and frameworks ready to be used on all the platforms you need them on.

On balance the language should probably compile to native code, but what would be neat is the option to run in a lightweight VM that can be bundled into a native binary so that the application can be more easily deployed without installing a runtime environment. Developing Elixir is very easy because you can recompile and interact with the application from the REPL (this is obviously not new to LISP developers - I have only used Clojure and find that IEx in Elixir is far easier to use than the Leiningen REPL).

Due to my habit of jumping between languages I don’t imagine I will settle on one that I would use for everything any time soon. Although I always keep an eye on Swift to see how that’s progressing - just next year is going to be the year of Desktop Linux, next year will be the year of Server-side Swift. Always next year.

A Downgrade

I have decided to downgrade my phone. I don’t feel the need for a large HD screen, a massive battery, and 3GB of RAM. The extra resolution of the 13MP camera pretty much wasted on me. I have used this phone - the OnePlus One - for the last two and a half years and have decided that when I get a new phone, I no longer need something of this calibre.

When the One came out, it boasted top-of-the-line specs - an incredible 401ppi 5.5” screen, massive 3100 mAh battery, a quad-core 2.4GHz processor, and a 13MP camera.

Almost all new phones in the last few years put the specs of the One to shame - boasting 8-core CPUs, over 6GB of RAM, screen densities upwards of 550ppi, and 16MP cameras.

So, what to do?

Thankfully, I found a device that suits my needs a bit better. The screen is a respectable 326ppi, the battery just 1800 mAh, only 2GB of RAM, and a 12MP camera.

This device does incur the cost of running a less common operating system that is proprietary to this particular manufacturer, rather than using Android, the de-facto industry standard. I have used Android for a long time, so this did weigh heavily in my decision to move, but in the end I decided that the more modest, lower-specced device was the right choice.

The phone arrived last week and even though the specs are behind even my One from 2014, thankfully I haven’t noticed any lack of performance or responsiveness in day-to-day use. Even with its battery that is less than two thirds the size the the One’s, I still haven’t dropped below 30% at the end of the day.

So, long story short - I got an iPhone 8 and it’s amazing.

Top Tips for iOS 11

Some top tips from Frederico Viticci’s iOS 11 review:

  1. Press CMD-Option-D to show the dock. No more awkwardly trying to reach for the edge of the screen when your keyboard is in the way.
  2. Grab the top nubbin in split view to re-arrange the windows in split view.

And some things that I wish were top tips but aren’t:

  1. Drag and drop apps in the multitasking view to create enter split view.
  2. Use CMD & arrow keys to move apps from spotlight into split view.

And some things that aren’t mentioned that I would like to see:

  1. Some way to enter 50/50 split view immediately, without going into 70/30 first.
  2. Open-in-place support in all the apps.

Pug: An Abomination of Shell Scripting

Pug started out a few months ago as a slightly silly idea of writing my own plugin manager for Vim and ZSH plugins. There is no shortage of ways of managing Vim plugins - Pathogen or Vundle seem to be the most common. For ZSH everyone swears by Oh My Zsh which includes every bell and whistle you could imagine.

However each of these only work for the one tool. What if (for some reason) I wanted a tmux plugin? I’d have to install some tmux package manager - if there is one. Pug is the one tool to rule all my package managing needs.

Pug can be used to manage packages for any utility - out of the box it has installers for Vim and ZSH, but other installers can be added by writing a simple shell script. I’ll probably write some more builtin ones myself.

To get started with Pug, head over to the Pug repo. My favourite ZSH plugins - syntax highlighting and auto suggestions - can be installed with Pug:

Install Pug:

curl https://raw.githubusercontent.com/javanut13/pug/master/install.sh | bash

Create a deps.pug file somewhere:

vim deps.pug

Add the dependencies (zsh-autosuggestions and zsh-syntax-highlighting) to deps.pug:

#!/usr/local/bin/pug load

zsh github: zsh-users/zsh-autosuggestions
zsh github: zsh-users/zsh-syntax-highlighting

Load the dependencies:

pug load deps.pug

You’ll be prompted to add this to your .zshrc file:

source "$HOME/.pug/source/zsh/pug"

Done. No more submodules.

Learn Enhancer Reaches Version 2.0

When I first started Learn Enhancer at the start of my degree - sometime in early 2014 - it was just a single JavaScript file that would redirect to a certain URL if it found a certain element. Eventually I coughed up the $5 required to submit it to the Chrome Webstore, and it started its slow creep on to students browsers.

This initial version didn’t catch every case, and so just over a year later I grew frustrated enough to dig right into the HTTP requests and alter them to ensure that the file could be previewed - which I outlined on this blog. Since then I branched out into re-styling every page on Learn with gentler colours (with design supervision from Sarang) - which had the delightful side-effect of letting me see whenever someone has Learn Enhancer installed. It is nice to see my work being used by people that have no clue who I am.

One of the features that I wanted to include but never quite worked out was auto-login, so that I didn’t have to two buttons before accessing content on Learn. This never quite worked out, until this morning I saw Patrick working on a script to do it. Thankfully he was receptive to the idea of incorporating this into Learn Enhancer, and soon he had a it working in the extension.

So here we go, Learn Enhancer has hit 2.0. Since this is my last year at UC, I have put Learn Enhancer on GitHub so that (in theory) current students can contribute fixes when Learn inevitably changes after I have graduated.

If you are a current UC student, download Learn Enhancer, you will almost definitely appreciate it.

Using tmux in the Real World

Every now and again I happen across a post outlining how to use tmux. Since I first happened upon tmux in 2015, my use of it has grown from “occasional”, to “frequent”, almost to “continual”. What I find frustating with these posts is that they don’t describe how to actually use tmux in the real world. The post in question that prompted this post tells you how to start a session, create new windows, then how to switch between and resize windows.

The thing that they fail to explain is that the default tmux commands and shortcuts are terrible. Common operations require far too many fiddley keystrokes to be done quickly. Moving between panes is by default Prefix followed by an arrow key, so to move to the right you would enter C-b → - but you can’t press the arrow key while you have control held down, because that will resize the window. So if you have four panes, you have to press C-b, release ctrl, press the arrow key, press C-b, release ctrl, press the arrow key, press C-b, release ctrl, press the arrow key once more and you’re there - unless of course you pressed the arrow key before the ctrl key was released, which means you will have a slightly resized pane instead 1.

This lack of usability is repeated - splits are created with Prefix % and Prefix " but which one does horizontal and which one is vertical? I have no idea, plus having keys that you have to use the shift key to get at just makes them harder to get at. The tmux command also leaves a lot to be desired - it’s not simple to connect back to an existing session given its name.

What irks me the most is that a beginner will read one of these posts and think that they have to remember all these arcane commands and be able to enter them at lighening speed. This is far from the reality of using tmux (or most other command-line tools) - everyone that I know that uses tmux has a config that makes tmux fit to the way they think of things. Each one of them had to learn the defaults and then find out if there was a better way - which is a significant barrier for most people.

The aspect of tmux that redeems these oddities is it’s extensive set of customisation options. Every command can be bound to a new shortcut, and shortcuts can be entered without needing to press the prefix key first. So what I’m going to do is build a set of reasonable defaults, so you can jump ahead and use tmux like a sane person.


This post isn’t a one-stop-shop for all your tmux needs, instead it’s just a quick walkthough of the basic ways that I make tmux more appropriate for daily use 2. All the snippets should be added to your tmux config file, which lines in ~/.tmux.conf by default.

The first thing that most tutorials tell you to do is remap the prefix to something other than C-b, because C-b is a bit too much of a stretch for most people. I use C-z, many people use C-a. Whatever you use is up to you. To remap the prefix, add this to your .tmux.conf:

unbind C-b
set -g prefix C-z
bind C-z send-prefix

This deactivates C-b, sets C-z as the prefix and makes a shortcut C-z C-z that will send C-z to the program inside tmux (so you can still use the shortcut). Replace C-z with another shortcut that tickles your fancy if you so desire. (I’ll use C-z when I’m talking about the prefix in examples, just remember to use yours if it is different).

The next thing is splitting panes. This will depend on how you visualise the panes, but I think of a horizontal split as two panes with a divider that is horizontal, and a vertical split has a vertical divider. This is the opposite to how tmux thinks of it, so depending on how you think, you may want to skip this.

Since tmux 1.9, new windows and panes open in the directory that tmux started in. I prefer the old method where they would open in the same directory as the previous window or pane. I frequently run some command, and if it takes a while I will open a split and continue working in the same location while waiting for the command to complete. I find this behaviour useful, and I think you will too. So:

# Open new windows with the same path (C-z c)
bind c new-window -c "#{pane_current_path}"
# Create a 'vertical' split (divide vertically) using C-z v
bind v split-window -h -c "#{pane_current_path}"
# And a horizontal split (divide horizontally) using C-z h
bind h split-window -v -c  "#{pane_current_path}"

Ok so on to the main event, the thing that makes tmux actually usable - faster pane switching. I use vim so I’m used to using h/j/k/l for left/down/up/right movement, you may prefer the arrow keys. Up to you. The key is to make these shortcuts not require the prefix before them, so you can smush some buttons repeatedly instead of repeating an exact sequence.

# For h/j/k/l movement
bind -n C-h select-pane -L
bind -n C-j select-pane -D
bind -n C-k select-pane -U
bind -n C-l select-pane -R
# For arrow key movement
bind -n C-Left select-pane -L
bind -n C-Down select-pane -D
bind -n C-Up select-pane -U
bind -n C-Right select-pane -R

These lowers the barrier to moving between your panes, which should hopefully encourage you to get crazy and open as many panes as you can fit on your screen. Wait, what if I don’t want to have everything in exact halves? Then you’ll have to resize a pane!

The post that inspired this one instructs you to resize panes by opening the command mode Prefix : and entering resize-pane -L, to move the split to the left. Now that is just super tedious. You can give it number of the amount you want to resize it, but that devolves into guesstimating pretty quickly. Instead I like to leverage the meta (alt/ option) key, so M-l (alt + L) will resize the pane to the left. Again you could make this M-Left if arrow keys are your forté.

# h/j/k/l
bind -n M-h resize-pane -L
bind -n M-j resize-pane -D
bind -n M-k resize-pane -U
bind -n M-l resize-pane -R
# Arrow keys
bind -n M-Left resize-pane -L
bind -n M-Down resize-pane -D
bind -n M-Up resize-pane -U
bind -n M-Right resize-pane -R

And boom, you can resize panes super quickly. One last shortcut that isn’t quite essential, but still useful is a quick window-switching shortcut, I like M-n and M-p to replace C-z n and C-z p. Especially if you’re flicking through a lot of windows.

bind -n M-n next-window
bind -n M-p previous-window

Two more useful things; set the default terminal to be 256 color so that your editor looks good, and set the starting index of windows to be 1 rather than 0 so it follows the order of the keyboard:

set -g base-index 1
set -g default-terminal "screen-256color"

So what to do about managing your sessions? Almost everyone I’ve talked to has made a little wrapper script that basically does this: if no arguments are given, list all the sessions. If an argument is given, connect to that session if it exists, otherwise create a session with that name. This avoids having unnamed sessions and means you don’t have to remember to run tmux ls every time. I’ve made a version of this with more bells and whistles but this is the basic idea:

mux() {
  local name="$1"
  if [ -z "$name" ]; then
    tmux ls
    return
  fi
  tmux attach -t "$name" || tmux new -s "$name"
}

Chuck that in your .bash_profile, .zshrc or whatever, then run mux to view your sessions, or mux my-session to create or connect to a session.

These are the changes that I have made to make tmux usable, but don’t forget that there are a whole load of things that I just do the default way. This post isn’t an exhaustive tutorial on using tmux, but rather an outline of how to make it more useful if you share my sensibilities.

  1. I know that if C-h/j/k/l were the default this would stop those keys being able to be used for other things, but I think the productivity gain is far greater than the loss of some keys (this is probably just because I don’t use anything that needs those shortcuts). 

  2. Other things that irk me are poor window indicators in the status bar - mine has more color to show the current window. The status bar also does a poor job of showing the status info - especially the current host. I change the color of part of the status bar depending on the host I’m on (mostly for aesthetics). And the default green highlight is super gross. 

Metaprogramming and Macros for Server-Side Swift

I have been a fan of the Swift programming language since it was first announced, and especially after it was open sourced. The place that I thought Swift could be the most interesting for me was for server applications - I’m not much of an iOS/ macOS developer. The progress of Swift-on-Linux is slow for someone that doesn’t like digging around Makefiles and linking to C libraries.

However, there are some things about web applications that aren’t currently served by the design of Swift. This can basically be boiled down to one thing - compile-time macros. Having a macro system allows for a lot of really cool syntactic sugar, as well as removing work that would otherwise need to be done on the first request, or at startup. Many of these are taken from my brief time learning Phoenix, a web framework written in Elixir - if I’ve misinterpreted something or ruled out some approach that is actually possible, let me know.

The main use of macros in your typical web framework is the routing configuration. Phoenix and Rails both support a DSL (implemented using the syntax of the language, Elixir or Ruby). Both of these look quite similar, basically allowing you to do this:

# In Phoenix
get "/", MyController, :index
# In Rails
get "/", to: 'my_controller#index'

The DSL gets more complicated when you include resourceful routes and other goodies. But at its core the purpose of the DSL is to allow the developer to use the same tools (i.e: the same editor and highlighting) to define their routes in a succinct manner. Phoenix can go one step further, because Elixir supports macros. The routes are checked when the project is compiled, and can be turned into arbitrary code that responds to web requests following the rules defined.

For example, the get macro can check that the path is valid, that it doesn’t clash with any other routes, and make helper functions for linking to that page (e.g. a my_controller_index_path() function). This is done at compile time, so when the code is run it is no different to running the “hand written” equivalent.

This is not the case in Ruby - because it is a dynamic language these methods can be created at runtime. There is basically no loss in performance because to support this level of metaprogramming (and because it is interpreted) Ruby is super slow compared to compiled languages.

When it comes to compiled languages without macros (like Swift, Go, Java, etc) you can’t pre-calculate information while the code is being compiled. Go lacks the features 1 to implement any kind of usable DSL. Revel (the #1 result when googling for “golang web framework”) has a separate routes file - written in a Revel-specific syntax that is parsed at runtime. This creates complexity in the packaging and distribution of the application - it no longer can be built as a single binary as it relies on this config file.

Swift does allow for creating concise DSLs. Vapor and Perfect are Swift web frameworks. Both of them offer routing DSLs that look something like:

app.get("/:page_id") { request ->
	return Response(.text, request.params["page_id"])
}

But this is processed at runtime, and doesn’t allow for creating helper methods for creating URLs, or grouping methods together into a class-based controller like Rails does. The latter could just be a necessary limitation of Swift, instead of making classes you could create a “controller factory” DSL, which you might use like:

controller("MyController") { app ->
	app.get("/stuff") { request ->
    // do something with stuff
  }
  // etc
}

Although this doesn’t get around the fact that much of your logic is defined in string literals that don’t get looked at until the application is running, or the fact that the routes must be generated when the application starts - if you wanted to make a super-efficient trie or other data structure for better processing requests, you sacrifice startup time in both development and production, even if the structure never changes until a new version is deployed.

Moving code-level information out of strings allows for static analysis to perform more useful checks when validating code. For example, regular expressions are often written as string literals (e.g: in Java) and so don’t get checked for validity until the program reaches them. Other languages have builtin regex literals (JavaScript, Ruby) to fix this problem. Elixir goes one step further thanks to (you guessed it) macros, specifically “sigils”. These are macros that wrap around a special “literal creation” syntax. This is used not only for regexes (written like ~r/abc\w{5}/) but for common “make a list of strings” helpers commonly found in other languages: ~w(foo bar) is equivalent to ["foo", "bar"]. So if you made a cool new type of regex that adds some awesome new feature, you can implement a macro that lets you write it easily and have all the same advantages as the builtin version.

View templates (think ERB, Liquid, Handlebars, etc) can also be parsed and optimised at compile time using macros - Phoenix does this so that when running the application all that needs to be done is string concatenation, no parsing needed.

So where does that leave us with Swift-based web development? It doesn’t seem any worse off than Go in terms of ability to dynamically create methods, etc - and Go appears to be used for web development a wee bit. The other option is code generation - but that’s always going to be a second-class way of doing this, as it relies on other tools and requires the other tool to parse the rest of the codebase to get the advantages you would from a macro system.

There might of course be a time when Swift gets a macro system, which will create a huge opportunity for new syntax and more concise, expressive code. However given the complexity of Swift and decisions so far, I would not hold my breath.

  1. lol no generics. 

Needless complexity: Generalising a Scheme for Aikido Training

It is perhaps a little known fact that I have practiced Aikido for about the last 13 years now.

I’m bad at writing introductions, so let’s jump straight to the problem. When training with more than one other person, you have to have some way of deciding who attacks who - you can’t just alternate. Normally when practicing tachi waza the most senior student goes first and does the technique four times to the uke before the roles are swapped. So what to do when someone else joins your pair?

You could just make a directed triangle - the person A is attacked by B, the B by C, and C by A, before the cycle repeats. This is easy to describe and can easily be extended to any number of people, but person A will never be attacked by person C - they miss out on any feedback that person C may have for them. You want a method that will allow everyone to train with everyone else, as well as allowing each member to do the technique enough times to improve.

At the moment, this is the recommended way of training:

A - B
A - C
B - C
B - A
C - A
C - B

Now this is fine. Apart from the fact that it only applies to exactly three people. The programmer in me wants a method that applies to any number of people. How about:

func train(members: [Person]) {
  for nage in members.sorted(by: .rank) {
    for uke in members.sorted(by: .rank) where uke != nage {
      uke.attack(nage)
    }
  }
}

Basically starting from the highest ranked member, each member should have a turn as nage. They should be attacked by each other member, in the order of their rank. This is how training in a pair works, and works just the same way if the whole class is training together.

This gets slightly more confusing when you doing weapons practice - there is a less clear distinction between the uke and the nage; the uke is often not thrown by the nage, and the uke still has to learn the attack as it not just a single strike or grab.

It’s common with weapons practice for a pair to train with one role, then swap and train before moving on to the next member of the group. This reduces the distraction of changing partners, letting you focus on the technique. This can be generalised in a similar way - this time each member of the group in descending rank order is the ‘key’ member, who practices both sides of the technique with each other member, then the ‘key’ member is changed to the next member in rank.

func train(members: [Person]) {
  for key in members.sorted(by: .rank) {
    for other in members.sorted(by: .rank) where key != member {
      other.attack(key)
      key.attack(other)
    }  
  }
}

Basically I think too much about the efficiency of how I am training, rather than focussing on the training itself. I guess that’s what happens when you spend all day learning about Software Engineering and stuff.

More Fun With Generics in Kotlin

Android now supporting Kotlin means more people playing around with it. Ben Trengrove is one of them - he has made a quite neat way of representing units in a type-safe wrapper. This disallows doing operations on two units of different measurements - for example speed cannot be added to a distance. Adding helper extensions to numeric types allows you to use it like this:

val distance = 21.kilometers

// This is OK because they are both distances
println(distance + 5.miles)

// This fails because you can't add distance to time
println(distance + 9.minutes)

You can have a look at Ben’s code here. Currently Ben’s code allows you to multiply one quantity by another. The result is a quantity with the same unit of the operands passed to the multiplication. This doesn’t follow the rules of dimensional analysis - if a distance is multiplied by another, the result not a distance, it’s a 2D area.

What this means is that you can do code like this:

val width = 10.meters
val height = 5.meters

// Area has unit Distance.Meter, not meters squared
val area = width * height

This piqued my interest - how could you implement units like speed and area, that are composed of multiple units? Of course you could just remove the .div and .times methods and replace them with extension functions that return a Quantity<Speed> or Quantity<Area> for each combination of units that you’re interested in.

But surely we can do better? This is what I set out to do, I wanted to be able to define the base units and derive every other unit from them. If you want to skip the rambling, you can check out the end result here.


The premise of this approach is to make two new subclasses of Unit1 that each have two generic constraints - each of which must also be a type of Unit. These represent a division type and a multiplication type, QuotientUnit and ProductUnit. All units have a suffix attribute that stores the standard identifier for that type (like “m” for meters, “s” for seconds, etc). QuotientUnit and ProductUnit create their suffix from the suffixes of their parts with either “/” or “.” inbetween.

The basic units are still defined in a similar way: 2

abstract class Unit(val suffix: String, val ratio: Double) {
  internal fun convertToBaseUnit(amount: Double) = amount * ratio
  internal fun convertFromBaseUnit(amount: Double) = amount / ratio

  override fun toString() = suffix
}

open class Distance(suffix: String, ratio: Double): Unit(suffix, ratio) {
  companion object {
    val Mile = Distance("mi", 1.60934 * 1000.0)
    val Kilometer = Distance("km", 1000.0)
    val Meter = Distance("m", 1.0)
    val Centimeter = Distance("cm", 0.1)
    val Millimeter = Distance("mm", 0.01)
  }
}

The implementation of the composite units looks like this: 3

class QuotientUnit<A: Unit, B: Unit>(a: A, b: B):
    Unit("$a/$b", a.ratio / b.ratio)
class ProductUnit<A: Unit, B: Unit>(a: A, b: B):
    Unit("$a.$b", a.ratio * b.ratio)

They are really just a placeholder to keep the type system in check. We can then use these to extend our .div and .times methods to return quantities with composite types. So inside the Quantity class we add:

operator fun <R: Unit> div(quantity: Quantity<R>): Quantity<QuotientUnit<T, R>> {
  return Quantity(amount / quantity.amount, QuotientUnit(unit, quantity.unit))
} 
operator fun <R: Unit> times(quantity: Quantity<R>): Quantity<ProductUnit<T, R>> {
  return Quantity(amount * quantity.amount, ProductUnit(unit, quantity.unit))
} 

So now when we divide a distance in kilometers by a time in hours, we get a QuotientUnit<Distance, Time> with a suffix of “km/h”:

val distance = 21.kilometers
val time = 1.5.hours
println("Speed is: ${distance / time}") // Speed is: 14 km/h

And we should be able to do conversions between composite units as well, because the ratio of a composite unit is calculated based on the original units.

val speed = 21.kilometers / 1.5.hours
val milesPerHour = speed.to(QuotientUnit(Mile, Hour))
println("Speed is $milesPerHour") // Speed is 8.7 mi/h

Now that’s quite useful. However typing QuotientUnit(Mile, Hour) is not very elegant. Perhaps we can use some helper function to make this a bit more readable?

We can actually do better than a helper function, instead we can make an extension operator that defines / and * on pairs of units. This lets us spell a composite unit like this: Mile / Hour, which is the same as QuotientUnit(Mile, Hour). You can do this like so:

operator fun <A: Unit, B: Unit> A.div(other: B) = QuotientUnit(this, other)
operator fun <A: Unit, B: Unit> A.times(other: B) = ProductUnit(this, other)

With all this, we can now do the unit conversion problems you get in physics class with almost no effort:

// James Bond is running along the roof of a train
// It takes him 1 minute to run the length of a 20-metre carriage
// The train is moving at 60 miles per hour
// How fast is James moving relative to the ground, in km/h?
val jamesSpeed = 20.meters / 1.minute
val trainSpeed = 60.miles / 1.hour

val totalSpeed = jamesSpeed + trainSpeed

val metricSpeed = totalSpeed.to(Kilometer / Hour)

println("Speed relative to ground is: $metricSpeed")
// Speed relative to ground is: 97.76 km/h

Easy!

The last thing I wanted to clear up was the need to repeat code for all the helper properties (6.minutes, 9.kilometers etc). These have to be repeated for every type of unit, and I wanted a way of creating units without this repitition. In reality you’d probably keep these to make it easier on yourself, but it’s nice to have an alternative.

How about just a simple infix function that operates on a number and a unit? Or how about if you multiply a number by a unit, it creates a quantity with that unit? What about invoking the unit with brackets - like a function call - to create a quantity in that unit? These are all quite straightforward: 4

// 5 into Minute makes a Quantity(5, Minute)
infix fun <T: Unit> Number.into(unit: T) = Quantity(this.toDouble(), unit)
// 79 * Kilometer makes a Quantity(79, Kilometer)
operator fun <T: Unit> Number.times(unit: T) = this into unit
// Second * 3 makes a Quantity(3, Second)
operator fun <T: Unit> T.times(value: Number) = value into this
// Hour(12) makes a Quantity(12, Hour)
operator fun <T: Unit> T.invoke(value: Number) = value into this

All of these make it super easy to create quantities. And of course, you can use them with composite units: 5 * (Kilometer / Hour), Second(8), 9.into(Metre * Metre) each create speed, duration, and area quantities.

Using these we could solve our physics problem from above like so:

// James' target is running twice as fast as him along the train
// How fast is the target moving relative to the ground, in m/s?
val jamesSpeed = 20 * (Meter / Minute)
val targetSpeed = 2 * jamesSpeed
val trainSpeed = 60 * (Mile / Hour)

val totalSpeed = targetSpeed + trainSpeed

val metricSpeed = totalSpeed.into(Meter / Second)

println("Speed relative to ground is: $metricSpeed")
// Speed relative to ground is: 27.5 m/s

Another helper that we can add is a division operator that operates on two quantities of the same type, producing a ratio of the two values rather than another quantity. You can use this to see “how many items of length X fit in space Y?”. This is done like so, inside the Quantity class:

operator fun div(other: Quantity<T>) = unit.convertToBaseUnit(amount) / other.unit.convertToBaseUnit(other.amount)

// To work out the ratio between two speeds:

println("Speed ratio: ${jamesSpeed / targetSpeed}")
// Speed ratio: 0.5

For completeness I also added operators for multiplying values by numbers with no units - letting you do things like “double this distance” with 2 * distance. Quantities are also comparable, so less than and greater than also work.

Hopefully this explanation illuminates some of the magic generics in the code - which you can view here. I’m sure there are operations and helpers that I’m missing, or ways that this code can be cleaned up and simplified. This would make for a kick-ass back-end for a unit conversion app!

  1. When implementing this, my first idea that turned out to be a dud was to make every different measure a Kotlin object. This did mean that the types could be part of generic constraints, so QuotientUnit<Kilometer, Second> is a valid type. This seemed like a good idea initially, but quickly ended when I realised that QuotientUnit<Mile, Second> is a different type to QuotientUnit<Kilometer, Minute> - even though they both represent distance/time

  2. In my examples I omit the prefix of the base type (eg Distance.) for readability. You can import them directly which allows you to use units with no prefix (eg Metre) or just import the class and access the companion variables (eg Distance.Metre). 

  3. I’ve changed my Unit class to have a .toString method that simply returns the suffix, differing slightly from Ben’s original version. 

  4. I decided to rename .to to .into so that it didn’t clash with the built-in .to extension in Kotlin that turns two objects into a Pair 

Templates, Code Generation, and Macros

Macros are a really cool feature that is includes in a few cool languages (Clojure/ Common Lisp/ other Lisps, Elixir, Rust, and Crystal) that allow you to reduce boilerplate code, extend the capabilities of the language, and process data at compile time. There is no shortage of tutorials on macros, but I am going to approach them from a direction that may be more familiar to some people.

It doesn’t take long while programming to come across some kind of template. Whether it’s str.format() in Python, or the moustache-templated bindings in $JS_FRAMEWORK you’ll end up writing something that will be used to generate something else. We can use these to separate the content from how it will be displayed.

This snippet is some Embedded Ruby, the <% and %> denote a start and end of Ruby code. Outside the Ruby is just HTML.

<ul>
  <% for item in items %>
    <li><%= item %></li>
  <% end %>
</ul>

Here we have items which is a list of strings, and we iterate over it creating an <li> with the content of each item. The server will run this code and generate some HTML, which is then sent to the browser. The browser doesn’t know that the HTML was part of a template, it looks exactly the same as hand-written HTML. What we’re doing is using Ruby to generate more code that could have been written by hand. Doing this for a formatting language makes sense - there is no way for HTML to show dynamic content. However because we’re just dealing with text, we can use a template to generate any type of file - even code.

function <%= func_name %>() {
  return "This is <%= func_name %>";
}

This is a silly example, but we’re using a template to create a Javascript function. All it does is returns “This is “ followed my the function name, but if we were making a library that interacted with a database, and we wanted to have an API that was something like <table_name>.get_by_<column_name>() without having to do metaprogramming (for example if the language doesn’t support metaprogramming, or you don’t want the runtime cost of doing reflection).

A common example of something that uses a “template” to generate code is a parser generator (like Cup, Yacc). These read a file that is in their own syntax for defining a grammar and how to store the AST, and produce a file of code in the target language that will parse something according to the specification.

Casey Liss recently wrote about Sourcery, a library for automaticaly generating boilerplate Swift code - it can do things like make a type Equatable by generating an == method that compares every property of the type. This is possible by generating a .swift file with the code needed to define this method.

Sourcery is quite cool - but it means that you have to have a special Sourcery file that defines what it should do, and remember to run the Sourcery script to generate the new files. Ideally you would put the definition for what boilerplate to generate inside the Swift file with all the rest of the code, and the Swift compiler would generate it automatically before running the program.

This is essentially what macros are. They are pieces of code that make more code, and are run when the program is compiled. For example if Swift supported this, it might look something like:

func ==(other: this.class.name) {
  for attr in this.storedVariables {
    quote {
      guard self.{{ attr }} == other.{{ attr }} else { return false }
    }
  }
  quote {
    return true
  }
}

I’m imagining that quote will turn whatever is inside it into code that will be generated (like other languages), and the double curly braces escape a variable - {{ attr }} would be expanded to the name of the attribute.

For macros to be super effective, the language should be represented in its own data structures (languages that are like this are called homoiconic). Clojure is one of these, and provides syntax for denoting which code is to be evaluated, and which code is to be used to generate more code. ` or ' start a “this is for code generation block” - everything after the that is like all the HTML outside the <% %> tags in Ruby. Code after ~ is equivalent to code inside the <% %> tags.

So we could make a macro that prints the code it will evaluate before it runs (like the -x option in Bash):

(defmacro debug [code]
  `(do
    (println '~code)
    ~code))

This can then be used just like a normal function call, but instead of calling the function at runtime, it gets replaced when the code is compiled.

(debug (println (+ 8 6 (* 5 7))))
; Will be replaced with
(do
  (println '(println (+ 8 6 (* 5 7))))
  (+ 8 6 (* 5 7)))

The new code will first print "(+ 8 6 (* 5 7))" then run the maths.

That’s a silly example, but a far more practical example is the Ecto library for Elixir. It is a DSL for running SQL queries, by using a macro it basically adds an SQL-like language right into Elixir, which can be checked for validity at compile time, rather than putting SQL in string literals where errors are only known when the code is run.

Running code at compile time also lets you do some cool tricks that don’t involve creating “new” syntax. For example, resource files can be loaded right into the program, so nothing has to be read from disk when the application is running. Phoenix (an Elixir web framework) loads all the views when the code compiles and turns it into a function that concatenates strings - so no parsing has to be done at runtime.

Of course, many smart compilers let you use lambdas and stuff to create “new syntax” that gets expanded at compile time, but macros allow the developer to have more control over what happens when code is compiled and truly add new contructs to the language.