Monads and monad transformers have a wholly undeserved reputation for being somewhat fearful and hard to understand. Unfortunately, even the best books on Haskell tend to further this reputation by choosing to introduce monads by giving a couple of examples, then stating the monad laws and running away very fast, and the treatment of monad transformers, even in the otherwise excellent Real World Haskell, is unclear at best.

As I said, this reputation is undeserved. In this paper I will attempt to give a general motivation for monads as machines which carry around information not all of which is visible at any time, and for which there is a defined recipe to, given two old machines, make a new one

. I then move on to do the same for the closely related topics of monoids, additive monads (MonadPlus in Haskell) and finally monad transformers.

For each of these types I give motivation, an explanation of why the monad laws etc are the way they are, and examples, generally simple ones.  The section on monad transformers is more complex than the rest: I demonstrate a completely general way in which given two monads m and t then Trans t m a := m (t a) is a monad, and Trans t is a monad transformer.  I then show that the classical MaybeT monad transformer is an instance of this construction.  As far as I am aware, this is not written down anywhere easily accessible, so this is a useful contribution not only to pedagogy, but to the general theory of monad transformers.

So here it is:

• What is a monad? (PDF file)

A common problem

Say I am writing some code in Haskell, and I have a value of type a and a list of type [b] that I want to convert into a list of type [(a,b)] where every entry is one of the pair of the a value and one of the values from the list. So I want a function:

pairThem :: a -> [b] -> [(a,b)]

the ‘obvious’ solution is one of

pairThem x ys = map (\y -> (x,y)) ys

or

pairThem x ys = zip (repeat x) ys

Simple enough, but they’re hard to generalise, the use of a lambda to do something so simple is rather ugly, and using infinite lists on such a problem seems rather excessive, I feel.

What I’m going to do here is take a detour via monad which helps motivate the concept of an applicative functor, which is not always well-explained, giving an elementary explanation for where they come from and why. So I’ll discuss monads, then applicative functors, always bearing in mind the example of the list, which is an instance of each.

Using a monad

It’s more illuminating to remember that lists form a monad. If we look for a monadic solution we see beyond the surface problem of manipulating lists and can get an insight into what’s really going on, as well as seeing just how powerful monads can be. So the natural function with the right signature is

pairThem x ys = liftM2 (,) [x] ys

But [x] = return x in the list monad, so in fact we could write this as

pairThem :: (Monad m) => a -> m b -> m (a,b)
pairThem x ys = liftM2 (,) (return x) ys

This works (see the illustration) but the point is that we’re not doing anything list-specific any more. This is a piece of extremely generic monadic code for pairing two monad values to get a monadic tuple value. One of the main goals in writing Haskell code is to make your code as generic as possible. Here I’ve gone from writing code which insists that I use lists to code that requires only a monad. This must be a good thing.

Why does it work?

liftM2 is usually defined as

liftM2 :: (Monad m) => (a -> b -> c) -> m a -> m b -> m c
liftM2 f mx my = do
  x <- mx
  y <- my
  return $ f x y

which can be rewritten in a more functional way as

liftM2 f mx my = mx >>= (\x -> my >>= (\y -> return (f x y)))

(it’s a good exercise to see why these are equivalent). So when m = [] and f = (,) let’s take this apart. First :

(\x -> my >>= (\y -> return (f x y)) = (\x -> concatMap (\y -> [(x,y)]) my)

and so

liftM2 (,) mx my = mx >>= (\x -> (\y -> concatMap [(x,y)]) my)
                 = concatMap (\x -> (\y -> concatMap [(x,y)]) my) mx

which, when mx = [x], reduces to

liftM2 (,) [x] my = concatMap (\y -> [(x,y)]) my = map (\y -> (x,y)) my

which is where we started.

Generalisations

We can see immediately that in fact the function

cartesian :: (Monad m) => m a -> m b -> m (a,b)
cartesian = liftM2 (,)

will, when applied to lists, produce their Cartesian product (all pairs with one value in the pair from each list). You would normally be taught to do that with a list comprehension

[(x,y) | x <- xs, y <- ys]

So the moral is, monads can do an awful lot that you might not expect, and much complex list manipulation can be made into simple monadic operations if you look at it the right way. And, as well as genericity, simplicity has to be our goal: mistakes are far easier in complex code.

Applicative Functors

Functors

We’re used to the idea of the Functor type. If a is a type and f is a parameterised type, then f is a functor if there is an operation

fmap :: (a -> b) -> f a -> f b

that lifts a function from bare types into a function between the f types that they parameterise. So if f = [] then fmap = map. If f is a monad then f = liftM. To see that these two definitions are consistent observe that

liftM f = (\ x -> do
  a <- x
  return $ f a

or, in the list monad,

liftM f $ x = x >>= (return f)
            = concatMap (return f) x
            = map f x

Applicative Functors

Now, functions are types too, so if f is a functor, I can quite easily form the type

f (a -> b)

For example this could be a list of functions. Now what would I do with a list of functions a -> b? Obviously, I could apply them to something in a to get a list of type [b], or I could apply them to a list [a] to get a list [b]. Let’s take these in reverse order.

Mapping a list to a new list

I want something like:

apply :: (Functor f) => f (a -> b) -> f a -> f b

Let’s take this apart. On lists I have to do this

apply gs xs = concatMap (\ g -> map g xs) gs

So I have to unwrap the list of functions fs to get at the individual functions within it, apply them individually to the data list xs and then concatenate the results.

More generally, I need a function something like:

(<*>) :: (Functor f) => f (a -> b) -> f a -> f b

such that

apply = (<*>)

Clearly therefore we must have

gs <*> xs = concatMap (\ g -> map g xs) gs
          = gs >>= (\g -> concatMap (return . g) xs)
          = gs >>= (\g -> xs >>= (return . g))
          = gs `ap` xs

Mapping a list to a new list

Now we want to do

apply :: f (a -> b) -> a -> f b

we already have <*>, so if we had a function

pure  :: (Functor f) => a -> f a

then we could take

apply f x = f <*> (pure x)

On lists we clearly have

pure x = [x]
       = return x

Defining the type

We have defined operations:

pure  :: (Functor f) => a -> f a
(<*>) :: (Functor f) => f (a -> b) -> f a -> f b

A Functor with these two operations is called an Applicative Functor. In general, any monad can be made into an applicative functor by taking

pure = return
<*>  = ap

just as we showed for lists above.

Let us take stock. In general, if we have a functor f and a function

g :: a -> b

Then we can form

fmap . g : f a -> f b

However, there is a much richer class of ‘functions’ related to f, i.e. objects like

h :: f (a -> b)

So if we take f = [] then fmap allows us to lift single functions to the monad, but applying lists of functions requires more machinery than just fmap. Similarly in the State monad then we can form

liftM g :: State s a -> State s b

which evaluates the monad and changes its value, but leaves the state alone. However there is a wider class of transformation

h :: State s (a -> b)

which combines a transformation of the value with a transformation of the state.

Now, I can do this in any monad with ap. The strength of applicative functors lies in the fact that they allow this extra richness in transformations without demanding the full structure of a monad.

A mathematical analogy

For those accustomed to simple algebra, here is an analogy. Say have a functor that takes rings to rank 2 free modules over those rings. So if then is the natural map

This is precisely what fmap would do under the circumstances.

However, the full space of morphisms should allow the first and second components of a morphism to differ, so we want morphisms like

And hence, very naturally, a map

This is precisely what an applicative functor gives us.

Applying this to the problem

So, say f is an applicative functor. Clearly we have equivalents of liftM and liftM2:

liftA  :: (Applicative f) => (a -> b) -> f a -> f b
liftA g x = (pure g) <*> x

liftA2 :: (Applicative f) => (a -> b -> c) -> f a -> f b -> f c
liftA2 g x y = ((pure g) <*> x) <*>  y

If f is a monad then

liftA g x = (pure g) <*>  x
          = (return g) `ap` x
          = (return g) >>= (\g' -> x >>= (\x' -> (return . g') x))
          = x >>= (\x' -> (return . g) x)
          = liftM g x

liftA2 g x y = ((pure g) <*>  x) <*>  y
             = (liftA g x) <*>  y
             = (liftM g x) `ap` y
             = (liftM g x) >>= (\g' -> y >>= (return . g'))
             = x >>= (\x' -> (return . g) x) >>=
                     (\g' -> y >>= (\y' -> (return . g') y'))
             = x >>= (\x' -> y >>= (\y -> return (g x y)))
             = liftM2 x y

So, in conclusion we can generalise our functions above still further to

pairThem :: (Applicative f) => a -> f b -> f (a,b)
pairThem x ys = liftA2 (,) (pure x) <*> ys

cartesian :: (Applicative f) => f a -> f b -> f (a,b)
cartesian = liftA2 (,)

this, it turns out, is the most general way of expressing the operations we started from.

Conclusion

So, what we have seen is first that it can always be worth looking for Monads, Functors and Applicative Functors lurking in apparently innocent problems, as they provide an extremely elegant way of reducing code to the most simple and minimal form.

In addition, I have used this to motivate the class of Applicative Functor, which is a Functor f that lets me take a value of type f (a -> b) and apply it to a value of type f a to get a f b, which is the natural generalisation of applying map across a list of functions. So Functors generalise applying a function to a list with map; Applicative Functors generalise using concatMap to apply a list of functions to a a list of values.

The point of all this is as follows: if f is a parameterised type, then there is every chance that functions f a -> f b are much richer than just a -> b, and so the functorial operator fmap only gives access to part of this structure. The way we access this wider structure is by understanding f (a -> b) and then using an applicative functor to transform this into functions f a -> f b.

Well, it didn’t fit in that paper, but I thought it was worth writing down.  So, without further ado, I have written a very short note that:

1. Describes how any data store can be made into a general monad transformer
2. Shows how, in the case of the trivial monad, this reduces to the simple storage monad described in the other post
3. Shows that if the underlying data is an additive monad, then so is the data store (which is basically just saying that it’s a monoid with >>=).

And here it is: note on storage as a monad transformer.

This introduces some interesting challenges, as we want the type of the data held in the service to change between rounds of processing, which may cause problems within Haskell’s strict type system.  I have found a very simple way around this which is simultaneously a strong enabler for storage services backed by (say) a database engine.

So, I have written a description of how this was done (together with some interesting general observations about the properties of data stores) and made the code available via GIT:

• The paper Distributed Storage with CloudHaskell
• The GIT repository: git://github.com/Julianporter/Distributed-Haskell.git

The paper discusses the type-related problem with message passing between distributed Haskell components at some length, to make it clear why we have to pass messages consisting of concrete types.

I have an old SPARC ULTRA 10  lying around, and I thought to myself that as it wasn’t doing anything, and as there was a Debian distribution for SPARC, I’d give it a go.  After all, installing Debian on Intel boxes was easy, wasn’t it?  Eight hours later I can affirm that no, it is not easy at all.  In fact, there are any number of pitfalls along the way for the unwary, and just following the instructions Debian give you will lead to staring at a blank screen wondering what went wrong.   All this information is somewhere in the internet, but it’s not even remotely in the same place. So, in the interest of all you people who might want to try to run Debian on a SPARC, here are my tips for successful installation.  Not that I promise they’ll work, but they might at least help . . .

The installation

Boot media

I decided to go for a network install CD, which is 140 MB, and sets up the core system, downloading everything else.  So here comes problem number one.  The official Debian SPARC network install image is here, but you don’t want that, because it crashes half way through trying to boot the kernel.  You want the testing version of the image, which you download here:

• http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/sparc/iso-cd/debian-testing-sparc-netinst.iso

Boot parameters

Okay, so now you can do

boot cdrom

to get to the kernel boot prompt.  You might think the obvious thing to do would be just to press enter to go for the default boot.  And if you try that then things go swimmingly right up to the point where the Debian installer  tries to install DHCP, at which point it finds the package doesn’t have a valid PKI signature, and it asks you what to do about it.  Unfortunately, it asks you via the serial port.  So, you could connect up a terminal emulator to the serial port, or you could do this: when you are asked to select a boot type, enter:

expert debian-installer/allow_unauthenticated=true

Selecting a kernel

I chose linux-image-3.0.0-1-sparc64 out of the available options.  It seemed sensible to have the most up-to-date alternative.

Selecting packages

Now everything will go fine right up to the point where the base system s installed and you have to decide which of the major software packages to install.  Things like web server and graphical user interface.  Whatever you do, do not select any of the groups.  If you do, then it will install all of them and then, just as it’s processing the last one, the installer will crash.  Just go with the pre-selected set of utilities, press go, and after a wait, you should end up with a basic text-based Debian system.

I say should: it depends on your choice of network mirror.  Bizarrely, slow mirrors seem to work better.  So if it crashes the first time, try another mirror.  However, in that case the safest option is to skip this step entirely, and jump straight to installing the boot-loader.

The software selection

Now is the time to work out what you actually want running on your SPARC.  the GNOME desktop environment does slow the thing down tremendously, so if you want to use the machine primarily as a server, I recommend just installing the packages you want with aptitude.  However BEWARE:.  If you try to install too many packages at once, aptitude will crash.  So just install a few at a time.  Alternatively, install as much as possible from the command line with apt-get (running as root).

This naturally means installing GNOME is a positive minefield.  If you must do it, invoke the following:

apt-get install xorg gdm gnome gnome-desktop

That should work.  If it crashes, reboot, then do

dpkg --configure -a

and then re-run the apt-get.

A while ago, I wrote a piece describing a simple way of implementing MapReduce as a Monad in Haskell.  As part of my further research I’ve discovered that in fact MapReduce is very naturally a Monad Transformer.  This means that given any monadic type m one can associate to it a MapReduce type MapReduceT m.  If m is an Additive Monad, then so is MapReduceT m.

This is theoretically interesting, as well as providing a nice pedagogical introduction to Monad Transformers and Additive Monads.  Furthermore, I have proved that the Haskell State Monad is in fact the MapReduce type associated to the (rather trivial) monad of maps.  I’ve written this up in a short paper, and would welcome comments and, to be honest, suggestions as to anywhere I could try to publish it.

Next steps

I’m working (slowly) on a distributed version of this framework based on CloudHaskell, at the moment limiting myself to the original, simple MapReduce monad.  So far I have distributed storage working, so when I get the time, it should not take much more effort.  If anyone is interested in joining in with this research programme, I’d be very happy to share the work.

References

Here are the relevant papers:

1. Map Reduce as a Monad: the original paper, published in The Monad Reader
2. The MapReduce type of a Monad: the new paper describing MapReduceT and the relation to the State monad

One of the major problems facing IT providers is that of how we can be both quicker and cheaper.  Quicker in the sense that we need to be able to respond to new requirements from our customers in a timely way.  Cheaper in the sense that we urgently need to reduce whole-life support costs of our systems (Gartner estimate that 80% of the total cost of ownership of any IT system comes from life-time support: thus only 20% constitutes the only aspect of cost that we traditionally concern ourselves with).

It seems that we can actually solve both of these problems at once, the magic bullet in this case being the concept of service catalogue.  The argument goes that if we have a service catalogue of standard offerings to our customers, then we are quicker because most of the time we can meet requirements by something off the catalogue, which we know how to do, so (assuming, always that we have adopted those other magic bullets of capacity management and capacity planning) meeting them should be just a matter of cranking the handle.  Similarly, we are cheaper because if we only support systems based on the catalogue, then we reduce the diversity of the install base, and so make it cheaper to support.

This argument is all very well as far as it goes, but unfortunately, as I will show below, it doesn’t go far enough, and in fact, if the service catalogue is implemented in the rather naïve way indicated above, then it will not do all we want of it; this is not to say that all those who have cried ‘the service catalogue is the answer’ are deluded, but simply that, as is the nature of magic bullets, the concept needs considerable refinement before it can be made to live up to its promise.

This note carries out the initial stage in that refinement, showing how what I will call a layered service catalogue can, by joining customer requirement management to a structured design process, go some way to making the potential benefits of reduced time to market (TTM) and total cost of ownership (TCO) realisable.  The method I will adopt is to analyse the naïve service catalogue view, to understand its benefits and disbenefits.  Then I will examine two antithetical approaches to maintaining the catalogue.  As it will turn out that the benefits and disbenefits of these two approaches are complementary, I will then perform the Hegelian two-step to arrive at a synthesis: the layered service catalogue.

Note that, for simplicity, I will discuss things from the point of view of an IT organisation offering services to its customers.  Of course, my discussion is entirely general, and so it can apply equally well to any delivery-based organisation.

 The naïve service catalogue

Description

This approach is very much as outlined as above, but I will put it more formally.  We have a catalogue of the services that it offers to its customers.  These are well-defined shrink-wrapped offerings, meaning that all design, etc work has been done and they basically consist of a blueprint from which a working copy of the service being offered can be built.

When a customer approaches us with a requirement, appropriate staff work with them to determine which offering from the catalogue is best suited to their needs.  Engineering staff can then be tasked with making an instance of that offering available for the customer (this may involve procurement, or simply reuse of existing capacity: for the purposes of my argument the question of where and the capacity is found is unimportant).

Benefits

TTM is indeed reduced, because all that we ever do is to take a canned solution and put it out on the machine-room floor.  All the nasty, time-consuming, risky requirement analysis, system engineering, scope creep, testing, etc that takes up so much time is relegated to … somewhere else.

Similarly, TCO is reduced, for us at least, in that now instead of having a huge number of specially designed systems, each lovingly tailored to meet a specific requirement, now all it has to support is serried ranks of clones of the systems on the catalogue.  This reduces the diversity of the systems to be supported, making it possible to manage the IT infrastructure not as many separate systems (as at present) but as a largely unified estate of capacity, reducing ongoing support costs.  Thus, life-cycle support is reduced provided that the catalogue does not get too large.

Disbenefits

Now consider the disbenefits.  As hinted in the comment above about TTM, this approach works perfectly provided customers never ask for anything new.  Unfortunately, this is just what they do all the time; the business does not stand still, so it is entirely unrealistic to assume that we can guess in advance what its customers will want from it in the next eighteen months.

What can we do to get around this?  Not much, to be honest.  We could use what I will call the yah-boo-sucks model, and tell our customers that we don’t care if they want something new: they’re not getting it.  That would be easy to do, and probably even fun, but it would not, perhaps, be very corporate-minded of us.  More practically, there are two antithetical approaches in common use today, which I will call the grand vision of the future and the vertical stovepipe.  Though neither is suitable on its own, it turns out that a dialectical synthesis of the two results in the required magic bullet.

Thesis: grand vision of the future

Here a bunch of clever people get together and try to design the shape of the enterprise n years ahead; generally they do this by coming up with an architecture for the enterprise, which defines precisely what services everybody should offer to their customers.  By appropriate crystal-ball gazing, the clever people hope to make the architecture sufficiently general that it will last n years.

There are three major problems here.  First, as we know too well, crystal-ball gazing is not a very effective way of predicting the future, so we can be sure that the day after our shiny new architecture is approved and set in stone, some annoying customer will come up with a requirement that we didn’t think of, that it can’t meet.

Second, and following from this, if the architecture is really general enough to predict n years ahead, it is most likely to be so general that it is of no use in actually trying to define the services on the catalogue.

Third, this has been tried many times, not always with significant success.

Thus, this approach does little to deal with the actual problem of unpredictable requirements, and turns out to be little more than a very fancy equivalent to the yah-boo-sucks model.  However, it does score significantly in terms of TCO: if this approach can be made to work at all then reduced diversity in the install base can be designed in from the start.  This means that we support only a limited range of technologies implementing a limited number of pre-defined services, and so support cost, and hence TCO is reduced.

Antithesis: the vertical stovepipe

Here, we get a new requirement that can’t be serviced from the catalogue and responds with the traditional cry of ‘we need a project!’[1]  A project (or other similar delivery vehicle) is initiated, and it carries out a complete stand-alone requirements capture – design – test – build – TTO cycle, producing a system tailored to the requirement, which is then added to the catalogue.

This is not actually too bad, in that it provides a relatively economical way to get new designs onto the catalogue.  In terms of TTM, requirements that need a new project take longer than straightforward off-the-shelf service requests, but at least the customer will know that in advance, and by virtue of having (hopefully) agreed that they need something new, not something already on the catalogue, they will be aware that they will expect to wait longer to get it, and they will be able to monitor progress.  In other words, they carry out the classic schedule versus quality trade-off.

The problems start when I look at the TCO implications.  If the vertical stovepipe is adopted naively (I will show below how it can be adapted to get around this, so bear with me for now) then as it designs the new system from scratch there is no mechanism in place to force it to make use of pre-existing components, and so the size and diversity of the catalogue increases, meaning that over time we have to support more and more diverse systems, so TCO increases.

Synthesis: the layered service catalogue

An architecture helps to reduce TCO, but is fundamentally unrealistic if it forces us to lock down the catalogue for however long.  Vertical stovepipes avoid lock down, but have a disastrous effect on TCO.  Somehow the two need to be joined together.

Let’s start by thinking about the process followed by the vertical stovepipe.  Basically it does the entire design waterfall: it starts from a requirement, then it designs a system: starting from a conceptual design, then adding on more detail, then choosing technologies to use, then designing components, then integrating them.

However, why does it have to do all of this?  Why can’t it reuse work already done before to design other similar systems?  For example, say I have been asked to produce a web service that has to meet all kinds of special security requirements that none of my existing web services are able to handle.  Do I really need to start again from scratch?  Isn’t there quite a lot I can reuse from my existing designs?

Layers

Say, therefore that we have defined a set of standard points along the path from requirement to completely designed system (where these points are doesn’t matter for now; of course, if a standard process like RUP were being used then obvious points would be at the end of each of the four major phases).  Say that whenever we design a new service offering for the catalogue, the standard design process forces the designers to document their design at each of these points and then enter it into a managed layered service catalogue.

This means that the catalogue consists of a number of designs, each belonging to a specific layer, which corresponds to designs at one of the standard points, and hence to designs existing at a precisely defined level of abstraction, so the higher (i.e. more abstract) the layer a design is in, the more easy it is to extend it to add new capabilities.

Below the top layer, each design has a specific parent, which is the design in the layer above from which it was derived.  Designs in the top (most abstract) layer derive from a corporate architecture defined by the business, which describes what the business proposes its portfolio should look like.

The bottom (most concrete) layer of the catalogue corresponds to deployable systems, so it is equivalent to the naïve service catalogue.  It is the catalogue of things that exist today.

Using the layers

Now say I have received a request from a customer for a particular requirement that can’t be met by one of my existing bottom layer designs.  I need to kick off a project (or whatever) to design and produce a system to meet the requirement.  What is new is that now I don’t tell the project (or whatever) design authority to go away and come up with a design.  Instead, she, and I look for something in the layered catalogue that can be used as a basis for meeting the requirement; in other words, instead of starting from scratch, she uses pre-existing designs in the layered catalogue to minimise the amount of new work she has to do (If she has to start from scratch, right at the top layer, then she has to base it on the corporate architecture; she would, of course, have to persuade me that the benefit to the business outweighed the obvious impact on TCO of doing so).

This carries obvious benefit: if the design authority increases reuse then we get to reduce risk, TTM (by not wasting time reinventing the wheel) and TCO because new systems will most likely be based on partial designs for existing systems, and so will be close to them, hence reducing diversity.  As all designs will be fully documented, we will have a good understanding of what it has to support.  Also, if we take care to include business criteria as well as technical criteria when deciding how much and what to reuse, we can trade-off quality against TCO.

What’s more, this approach gives me an architecture for free; what is the compendium of service designs (existing at various levels of abstraction) within the layered catalogue, but an architecture?[2]  Thus, it appears that this approach (particularly if supplemented by a structured design methodology that encourages good design practices, e.g. RUP, XP) gives another magic bullet: working strategically by thinking tactically.  If we follow it consistently, the result will be a consistent strategic architecture that evolves to meet changing business requirements.

Thus, I suggest that the following process can be adopted for using a layered service catalogue to meet a new requirement:

• Always start at the bottom layer of the catalogue (services ready for implementation, capacity on demand).
• When dealing with a new requirement, search in each layer, starting from the bottom, for the design best able to meet the requirement; if none can be found proceed to the next layer up.
• At each point where the decision is made to proceed to a higher layer, a business decision must be made to trade off quality against impact on TCO of expanding the catalogue; as this impact increases the higher the layer you start from (as higher layers are more generic, and hence starting from them increases diversity in the lower layers) the more stringent this decision process should be.
• Once a suitable design has been located and an estimate of TCO impact and TTM have been received and approved, authorise a project (or whatever) to implement the requirement starting from that design.
• Add all new designs (at each standard point passed) to the layered design catalogue in the appropriate layer.
• As the project progresses, place a decision point at the stage at which its design process hits one of the standard points: this is a good time to re-evaluate the projected TTM and TCO impact, and for us to determine whether to authorise continuing work.

What’s next?

I started off this note by expressing suspicion of magic bullets, and so, as I seem (mixing my metaphors horribly) to have pulled several rabbits out of hats in the course of my argument, it’s worthwhile to note places where more work is needed:

• The ‘standard points’ in the design process need to be defined with sufficient precision that design authorities will know what they have to document and when.
• Following on from this, a standard design process is needed; this must be fully documented, with specifications of the standard points, documentation standards, etc.  This is particularly important, as the magic bullet of ‘working strategically by thinking tactically’ is crucially dependent on this process (for example, it must strongly encourage SDAs to design with reusability in mind).
• The process must be sufficiently rigorously defined and policed to prevent the natural tendency for design authorities to attempt to blur the distinction between the layers.[3]
• We need to join up design processes across the business.
• We need to work out whether ‘we need a project’ is really a suitable response to any new requirement that involves more risk than crossing the road.  A more nuanced, perhaps contract-based, approach may be preferable, particularly as customers will generally be entirely uninterested in how we deliver their requirement, provided that it does, and that they can measure its success in so doing.
• We need an approach to risk analysis suited to thinking in terms of TCO rather than (as in classic PRINCE 2) pure delivery cost.

Say I am writing some code in Haskell, and I have a value of type a and a list of type [b] that I want to convert into a list of type [(a,b)] where every entry is one of the pair of the a value and one of the values from the list. So I want a function:

pairThem :: a -> [b] -> [(a,b)]

the ‘obvious’ solution is one of

pairThem x ys = map (\y -> (x,y)) my

or

pairThem x ys = zip (repeat x) ys

Simple enough, but they’re hard to generalise, the use of a lambda to do something so simple is rather ugly, and using infinite lists on such a problem seems rather excessive, I feel.

Using a monad

It’s more illuminating to remember that lists form a monad.  If we look for a monadic solution we see beyond the surface problem of manipulating lists and can get an insight into what’s really going on, as well as seeing just how powerful monads can be.  So  the natural function with the right signature is

pairThem x ys = liftM2 (,) [x] ys

But  [x] = return x  in the list monad, so in fact we could write this as

pairThem :: (Monad m) => a -> m b -> m (a,b)
pairThem x ys = liftM2 (,) (return x) ys

This works (see the illustration) but the point is that we’re not doing anything list-specific any more.  This is a piece of extremely generic monadic code for pairing two monad values to get a monadic tuple value.  One of the main goals in writing Haskell code is to make your code as generic as possible.  Here I’ve gone from writing code which insists that I use lists to code that requires only a monad.  This must be a good thing.

Why does it work?

liftM2 is usually defined as

liftM2 :: (a -> b -> c) -> m a -> m b -> m c
liftM2 f mx my =
do
x <- mx
y <- my
return (f x y)

which can be rewritten in a more functional way as

liftM2 f mx my = mx >>= (\x -> my >>= (\y -> return (f x y)))

(it’s a good exercise to see why these are equivalent).  So when m = [] and f = (,) let’s take this apart. First :

(\x -> my >>= return (\y -> f x)) = (\x -> concatMap (\y -> [(x,y)]) my)

and so

liftM2 (,) mx my = mx >>= (\x -> (\y -> concatMap [(x,y)]) my)

                 = concatMap (\x -> (\y -> concatMap [(x,y)]) my) mx

which, when mx = [x], reduces to

liftM2 (,) [x] my = concatMap (\y -> [(x,y)]) my

                  = map (\y -> (x,y)) my

which is where we started.

Generalisations

We can see immediately that in fact the function

cartesian :: (Monad m) => m a -> m b -> m (a,b)

cartesian = liftM2 (,)

will, when applied to lists, produce their Cartesian product (all pairs with one value in the pair from each list).  You would normally be taught to do that with a list comprehension

[(x,y) | x <- xs, y <- ys]

So the moral is, monads can do an awful lot that you might not expect, and much complex list manipulation can be made into simple monadic operations if you look at it the right way. And, as well as genericity, simplicity has to be our goal: mistakes are far easier in complex code.

I have a considerable interest in the use of massively parallel cloud systems for handling hard problems, but unfortunately they currently rely on extremely complex infrastructure that is quite intrusive when it comes to writing applications.  My intention is, and has been, to find a simple framework that can be used to provide cloud computing capability for Haskell applications while remaining simple and lightweight.

I have discussed elsewhere my initial implementation of MapReduce as a monadic operation that works in parallel processes on a single machine.  Here I take the first steps towards analysing what else is needed to turn this into a working cloud based system.

The basics

As I showed in my paper on MapReduce, nearly all massively parallel algorithms are some kind of version of the basic scatter-gather approach.  This works as follows:

foreach processing step
    divide data into chunks
    give chunks to processing units
    processing units transform data
    gather results from processing units
    concatenate to form next step's data

It’s useful to reformulate this from the processing unit’s point of view, so we get:

foreach processing step
    request chunk of data
    while chunk allocated
        get chunk
        transform chunk
        set transformed chunk
        request chunk of data

This makes it clear that scatter-gather requires two basic distributed services.

The services

The datastore

First we need a simple central datastore that:

1. Accepts chunks and concatenates them together to form a complete dataset
2. Can return the complete dataset when asked to

This service is, of course, little more than a mildly intelligent front end on a database table.

Note that in standard MapReduce, the partitioning of data into chunks is performed by the central datastore.  In view of our desire to run generalised MapReduce, we have to send the entire dataset to all processing nodes.  In fact, if this is handled sensibly (e.g. via IP multicast) then there is no difference in network loading at all.

The broker

Next we need a broker, whose job is simply to wait for processing nodes to ask for chunks to be assigned to them, and either to do so or to inform them that the step is finished and they should proceed to the next step.  So the broker has to:

1. Keep a track on how many processing nodes there are
2. Decide how many chunks to divide each step into
3. Wait for nodes to request chunks and hand them off one by one
4. Decide when the step is completed

There is also a processing node component to the broker, which in each step accepts the processing function, asks for a chunk and waits for a response.  If a chunk is forthcoming, it gets the data, applies the function, puts back the result and asks for another chunk.  If none is forthcoming it goes on to the next step, accepting a new processing function and proceeding as before.

This structure suggests that in fact what we have here is some kind of distributed monad or related structure. Indeed, as it is clearly stateful, it is more or less inevitable that it should be.

The next step

I have made some observations about implementation, noting that the datastore is a glorified database table, and the broker could be a kind of monad. In my next piece I will follow up on this, and discuss (at a high level) some aspects of how exactly the services could be built.

It turns out that this is basically because nobody has a clue what quality means, and so people go between the extremes of customer satisfaction surveys (totally subjective) and immensely complex requirement to product linkage measures (objective in themselves, but entirely subjective when it comes to deciding what the requirements actually were).  None of these are very satisfactory.

So, I had a think, and I realised that there is one unifying feature that both customer and supplier can agree on: risk.  What happens in any procurement is that the customer had a risk, and they pay the supplier to mitigate it.  So why not measure quality in terms of the amount of risk mitigated?

This is very easy to understand, but there’s something else.  If the customer gives the supplier a 473 page requirement document, they are basically saying ‘we want the solution to look like this’.  If they specify the risk they are facing, the supplier is free to provide the most effective mitigation whatever it may be.  It may well be that the best solution is not technical at all, but a process change.

So: simple, precise, enabling.  All good things.  I’ve put up a draft paper, on which I would be very glad to receive comments.

• Draft paper on measuring quality