Think of APIs as Domain-specific languages (DSL)

Today’s blog post is about why you should think of a DSL when building APIs. Don’t be afraid, I am going to explain what a DSLs is if you don’t know it by now and I will give you an overview of examples. You might already know them but most developers don’t recognize them as domain-specific, but just another language.

Definition

First of all, what exactly is a DSL, what does the term domain-specific language stand for? The best way to explain this is by naming the opposite of it, which you already know because you do use it everyday: General-purpose programming languages. Whether it’s PHP, Ruby, C(#), Java, … you can do just about anything with them, that may be developing applications for the web, mobile devices or whatever. But what if you have a specific problem, a problem that has been solved like a thousand times before? For many problems like these, programming languages have been invented that perfectly fit the problem’s domain and make it easy to solve it.

Martin Fowler describes DSLs like that:

Domain-specific language: a computer programming language of limited expressiveness focused on a particular domain.

He further names a distinction between internal and external languages which means:

  1. internal DSLs are built in a specific language and work only for that language
  2. external languages which then need a full parser in the language you want the DSL to use in

Examples

external DSLs

SQL is an external DSL and it’s domain should be clear to anyone that ever used it: (relational) data. Another example you all should know is Cascading Stylesheets and it’s domain is to style the representation of Markup, mostly used for HTML. Have you ever used a build system like make, rake or ant? These are all external Domain-specific languages which all define a set of commands. Redis (an in-memory database system) and the maintainer of the project, Salvatore Sanfilippo even defined the first of the project’s goals in the Redis manifesto as follows:

A DSL for Abstract Data Types. Redis is a DSL (Domain Specific Language) that manipulates abstract data types and implemented as a TCP daemon.

internal Domain-specific languages

I am pretty shure you already used some of the above examples, but what about internal languages? Have you ever come across one? I am sure you have, here are some examples for you:

Sinatra is a Ruby micro HTTP framework (I already blogged a tutorial series about Sinatra) that defines a set of commands that help you to build more or less simple web applications or REST services. It’s domain is HTTP and if you take a look at it, you will find that it’s pretty close to the HTTP specification. Have you ever heard of Erlang?  It once was an internal DSL at Ericsson, where they used it to build highly concurrent telecommunication applications. Later they open sourced it and now you can use it to build just about anything. It evolved from an internal Domain-specific to a general-purpose language. And JavaScript once has been a DSL to develop interactive web application frontends, but node.js took it out of the browser and made it a general-purpose language.

What can we learn from DSLs when building APIs?

Have a look at the above examples and you’ll notice that they all share a few principles:

  • define a rather small set of commands
  • ease of use
  • just fit the domain’s needs – no more, no less

And I think this essence should also apply if you build an API, no matter if it’s external (say for example a REST interface) or internal. You should always keep it simple and easy to use and don’t confuse the user of your API with domain-unrelated things.

PHP dependency and package management with Composer

Recently, I held a talk at Mayflower GmbH about Composer, a fresh and easy new way to manage the dependencies of a PHP project. Actually I did that talk twice, once in Würzburg and once in Munich. The Slides are in german, but I think you’ll get the point even if you’re not speaking german.

test your Ruby app with various versions using rvm

Recently I started to test my highscore library on Travis CI and they’re using rvm on their test machines to easily switch the version and platforms. I only had MRI 1.9.3 on my local machine, but had some problems on 1.8.7. Removing 1.9.3 and installing 1.8.7 was no option, so I installed rvm (Ruby Version Manager) on my machine and I love it! So I am going to talk about it today, the benefits, going through the (easy) installation process and what you should be aware of.

First of all if you don’t know it yet: there are several implementations of Ruby interpreters out there: the best known is MRI (Matz-Ruby-Interpreter), it’s the first implementation and written in C, then there’s Ruby Enterprise Edition, an improved version of the MRI. JRuby runs on the Java Virtual Machine, Rubinius is also written in C, but younger implementation, IronRuby runs on the .NET platform and MacRuby runs, well, only on Macs (who would’ve guessed it?).

Setup rvm

Installing rvm is a simple task, all you have to is that:

bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)

This will install rvm in your $HOME directory and you should add the following line to your .bashrc (or in your .zshrc if you use the awesome zsh)

source $HOME/.rvm/scripts/rvm

Installing rubies

Now you have rvm on your system, but you don’t have any Rubies in it. rvm list known will give you a list of all Rubies that are available. Start with installing the default Ruby, type rvm install 1.9.3 followed by rvm default 1.9.3. Now if you do ruby -v, you should see the 1.9.3 version. If not, start a new shell session, the rvm script must be loaded for the session first.

To switch to another Version or platform, you need the use command: rvm use jruby will switch to the latest JRuby version that you installed.

ruby gems

You already may have noticed that all the Rubies are installed in your $HOME directory, in the folder .rvm. All gems that you install will be put in this directory, there’s a gems folder for every Ruby that you installed via rvm there. So you need to install gems for every version that you want to use.

Rvm has a lot of functions that go beyond the scope of the article, you could run rake tasks on multiple Ruby versions in one step, install a gem on multiple platforms, run your unit tests on 1.8.7 and 1.9.3 in one step (rvm 1.8.7,1.9.3 do rake task) and much more.
You should definitely check out the documentation.

Playing around with different interpreters and building libraries that run on many Ruby platforms is fun with that wonderful piece of software. Have fun!

Developing a daemon in PHP

I’ve already talked about Unix programming here, but today I want to go a step   ahead an really implement a daemon, this time I am going to use PHP. You may have already heard of daemons here or there, but I’ll give you some facts about them anyway:

  1. daemons are long running processes,
  2. they run in the background,
  3. you can’t interact with them via a TTY
  4. originated from the Unix world
  5. a common convention is to put a trailing “d” on their name, e.g. httpd

These are the more obvious attributes of a daemon, but there’s a little more to say especially about 2. Let’s start PHP in the terminal and put the process in the background:

php -a &

Now if you look through the processes on your system with ps -fax | grep php you will see something like that:

  502 8621 480 0 6:33pm ttys000 0:00.00 php -a

The first number is of no interest for us, but the second and the third are. These are the process ID (pid) and the parent process ID (ppid) of our little PHP process. As you can see the parent process is (in my case) 480 and if you do another grep’ing for 480 you can see that this is the shell where you’ve typed php -a & in. What happens if you log off the shell? All child processes of the shell will die, so will your PHP process.

But daemons should not die when a user logs off, they should run forever! If you go the tree of processes up on your system you might find a process that has the ppid 1. It’s the first process that is started when your OS starts and the last one to exit. So we should run our daemon as a child of that process.

Preconditions

Before we can implement our first daemon make sure you have the pcntl extension for PHP installed. Most Unix-based programming languages have the necessary functions implemented in the core, but not PHP. Just search for pcntl in your package manager and install the package.

Our first daemon: mattd

It does nothing, except printing dots to /dev/null, but that shouldn’t matter. It’s a demonstration daemon. Here’s the dump, I’ll go through it and explain what and why we need each line of code in it and what it does:

#!/usr/bin/env php
<?php
$pid = pcntl_fork();

if ($pid === -1) {
die('Could not fork!');
} elseif ($pid) {
// this is the parent process
exit;
} else {
// this is the child process
chdir("/");

// reopen standard file descriptors
        // this is necessary to decouple the daemon from the TTY
fclose(STDIN);
fclose(STDOUT);
fclose(STDERR);

$STDIN = fopen('/dev/null', 'r');
$STDOUT = fopen('/dev/null', 'wb');
$STDERR = fopen('/dev/null', 'wb');

// make it session leader
posix_setsid();

while(1) {
                // just print a dot to /dev/null (STDOUT)
echo ".";
                sleep(10);
}
}
view raw mattd.php This Gist brought to you by GitHub.

These are the steps we need to do in the daemon:

 

  1. fork a new child process
  2. pcntl_fork() returns two times, the PID of the parent process and with a 0 for the forked child process
  3. so there are two ways the code runs from here, one is the parent, the other is the child process (if forking was possible)
  4. the parent immediately exits (it’s ppid is the shell you started the daemon from)
  5. the child goes in the else branch
  6. a daemon has now working directory, so reset it to /
  7. close all the file descriptors that are bound to the shell used, that is STDIN/OUT/ERR
  8. start a forever-running loop that does something useful

As I said before we need to run our daemon as a child of the pid 1. We can do this by forking and the exiting the parent process. The then new process runs as a child of our php process first but as this process exits, it has no parent anymore.
But having no parent is not possible, so it will get the new ppid 1.

Daemon != Cronjob

Typically, PHP developers are more familiar with developing cronjobs than with daemons and although both are used for similar tasks, they don’t share a common idiom. The main difference is that cronjobs are mostly long-running processes whereas daemons are forever-running processes.

The other thing is that cronjobs are always started in the context of a shell. Daemons don’t run in any context, their parent process ID is always 1, the first process started on the system. You can easily test this: create a small PHP script that has an endless loop in it and put it in your crontab. Now wait until the script gets started and do ps -fax | grep you_cronjob and you’ll see that your cronjob has a PPID other than 1.

That’s it for now …

A short disclaimer to the end: I have not tested this daemon in production nor will I. I am just keen to learn what goes on in the background of Unix programs and I love to share my knowledge to my readers. If I am wrong and you know better, please leave a comment here, thanks :)

practices of an agile developer

I often ask myself: “What can I do to improve my daily work, to become more efficient and a better developer in the non-technical sense?” In the end, it’s all about being agile. But what does that mean? I assume you already know about the agile manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The book “Practices of an agile developer” gave me some good tips on getting ‘more agile’. It’s not one of these books that give you just a detailed text about what those four lines mean, it’s merely meant to be a guideline that will guide you through the day if you are working in an agile team.

Each section begins describing the non-agile way and then tells why this is bad and what you should do to get better results.

The best about the book is this, a PDF that summarizes all guidelines from the book with a short description. If you want to read it, you can grab a copy here.

I don’t get money for this link, I just read this book and found it worthy to be mentioned here! In fact, I love the books from Pragmatic Programmers, I read Programming Ruby, Metaprogramming Ruby, The RSpec Book, Programming Erlang – Software for a concurrent world, Agile web development with Rails and I’m currently reading Pragmatic Thinking and Learning.

Unix programming using Ruby

The other day I blogged about an alternative for bash, zsh. But what if you want to write your own shell that behaves exactly like you want it to?

Jesse Storimer (the author of the truly great Working with Unix Processes and already working on his Unix beard ;-) ) has just started a new series of blog posts in which he’ll explain how to write the quintessential of a Unix program – a basic shell in Ruby.

Today most of the web development or software development in general happens on Unix machines, whether it’s Linux, Mac OS or BSD. Personally I don’t use any other systems, so why don’t learn some of the basics and understand the big picture that’s behind those architectures?

Of course, this is just a little project for demonstration and learning purpose but there really is an interactive shell written in Ruby and it also uses Ruby as it’s input language. Say hello to rush.

zsh – a bash alternative that’s easily customizable with oh-my-zsh

I think I now what you are thinking right now: “Why do I even NEED an alternative to bash?”. And I can tell you I thought the same, too. Bash is perfect, there’s auto completion, a command history and all those neat things that help you being productive when using the shell.

And then one day in mid-2011 I discovered the Z shell and once I read about all the nice features it has, I started being even more productive and I fell in love with this tool.

(more…)

using git as a backup tool

I recently had the problem, that I did not do any kind of backup of this site. The blog has grown since I began blogging again about a year ago and especially in the past few weeks since I relased messie and highscore under the MIT license.

I don’t wanted to loose all posts and backup at least the database (it’s a MySQL system). In the past I dumped the complete schema and downloaded that via SFTP onto my local machine, but it’s always the same: You set it up and all is ok and it runs for a few months. Then something unexpected happens and the dump is not downloaded but you don’t have time nor want to do that whole thing again and see what failed and so on. That goes on and on and on.

(more…)

building tiny web apps with Sinatra part 2

It’s been a while since Part 1 but I’m here again to implement the outstanding two features. That is: showing an article and creating new articles.

Feature 2: show a single article per page

We’ll start with showing a single article per page, the routing for this will be /posts/:id, so that the first article will be /posts/1 etc. I started with implementing a method in the Post class that will load one single file, given the id of the post:

class Post

  ...

  def self.load_file(id, directory = 'data/')
    file = directory + id.to_s + '.json'
    post = JSON.load(File.read(file))

    self.new(:id => id, :body => post['body'], :title => post['title'])
  end
end

(more…)