Planet Trapexit - Erlang/OTP News

REALITY.SYS: Stierkampf mal anders.

Sat, 04 Jun 2011 19:27:58 +0000

Wie so viele andere schwachsinnige Rituale ist auch der sogenannte “Stierkampf” (besser: langsames Stiereschlachten) eine “Tradition”. Das Bemerkenswerte daran ist aber vor allem die Tatsache, dass angeblich aufgeklärte Westler wohl keine Probleme damit haben, dieser morbiden Veranstaltung beizuwohnen. Menschen (fast) wie du und ich sitzen schön brav in der Menge, oft von ihren minderjährigen Kindern flankiert, und schauen zu, wie ein Stier langsam zu Tode gequält wird, während mehrere Zirkusfritzen, sog. “Torreros”, in ihren glitzernden Schlachterkostümen um ihn herum tänzeln.

Ich könnte jetzt mehrere Millionen Gründe aufzählen, warum der sog. “Stierkampf” eine Tierquälerei ist. Dies wäre aber nicht so “ökonomisch” und schon gar nicht effektiv, wie wenn ich einfach einige Videos anführe, wo das Tier gewinnt.

Solche Videos zeigen den einzig wahren Stierkampf!

Und ich muss zugeben, dass ich es als sehr angenehm empfinde, zu sehen, wenn das Tier gewinnt. Jedem Besucher, jedem “Torrero” und vor allem den Vollhonks, die hinter den Stieren her laufen, kann ich nur die Bekanntschaft mit dem Stierhorn wünschen. Manchmal kann die Vernunft einfach nicht anders vermittelt werden. 200 Jahre Aufklärung sind für einige von uns eben nicht ausreiched.

Click here to view the embedded video.

REALITY.SYS: Immer dieses Händewaschen…

Sat, 04 Jun 2011 08:23:13 +0000

Angesichts der, ähnlich SARS und Schweinepest, herbeigetrommelten Seuche “EHEC”, muss man sich ernsthaft die Frage stellen, ob wir in Sachen Hygiene nicht immer noch im Mittelalter stecken.

Hallo Leute! Sich die Hände richtig zu waschen ist eine Sache, die man normalerweise als Kleinkind gelernt haben muss!

Und die Betonung, dass man nach einem Toilettenbesuch die Hände waschen sollte, ist mehr als lächerlich. Es ist erschreckend, traurig und zeugt nur von unserer sterilen Lebensweise in dieser angeblich fortschrittlichen Welt. Ja, sterile Lebensweise, als Bezeichnung für unser Abgekoppelt-Sein von elementarsten menschlichen Ritualformen wie richtiges Waschen des eigenen Körpers, richtiger Umgang mit fremden Sachen (z.B. indem wir ausgestellte Waren nicht ohne Weiteres betatschen sollten!) etc.

Wie blamabel muss es denn für uns sein, wenn man uns über die Medien darum bitten muss, sich die Hände richtig zu wachen und das Gemüse bzw. Fleisch vor dem Verzehr ebenfalls gewaschen zu haben.

Aber nein, am leichtesten ist es doch, diesen blöden Südeuropäern alles in die Schuhe zu schieben. Ja, genau, die Spanier waren’s, diese gemeinen Kerle. Sie haben uns all das eingebrockt und siehe da: wir müssen uns jetzt die Hände waschen!

ejabberd@jabber.ru: ejabberd 2.1.8 - Fix PubSub

Fri, 03 Jun 2011 11:39:09 +0000

The ejabberd 2.1.7 released yesterday contains a bug that breaks PubSub.

If you use ejabberd 2.1.7 and PubSub, you can find the patch and the fixed mod_pubsub.beam in the page EJAB-1457.

Updated binary installers will be available next monday in http://www.process-one.net/en/ejabberd/downloads

Process-one Blogs: “Google Social”: A paradox of its own

Thu, 02 Jun 2011 12:59:07 +0000

Yesterday was a sad day for the Internet, as an open and decentralized content repository.

At the same moment, Twitter launch its follow button and Google extend its +1 button for websites. They both are attempting to catch up with Facebook success with the "like button".

Of course, it feels ridiculous for content producers and consumers to expect to place more buttons on their pages in the hope to attract more traffic. The escalation in the battle for embedding buttons in web pages and the battle to get the users clicks is leading nowhere.

I would rather however focus on Google move, the paradox and possibly the trap it could become for the company. I have read numerous times that Google initiative in "social" area had always been a failure. Even Eric Schmidt admitted at D9 conference that he had messed up on Google Social initiative.

In my opinion, the main reason for failure is because social, as defined by today's market, is the complete opposite of Google model and view of the online world.

"Like buttons" are replacements for the HTML links that authors used to place on their webpage to talk about topics they liked and point to relevant content. But as HTML links were purely open and available for search engine web spiders, the end result of the "like button" action is closed and proprietary. Facebook and Twitter are one of the biggest threat for Google: they are cutting Google supply of their raw material: the data to analyze, both content and relations (links).

Google reactions yesterday has been to launch its own initiative, its own "like button" to gather its own data. However, doing so Google enter the trap of the closed Internet, in which they cannot prosper.

Google's model has been to compete on algorithm, not data. They have beaten all other search engine using the same data set than anyone could use. They hire smart scientists to produce clever mathematical solution to the world data analysis problem. Algorithm competition is their reason to exist: organizing the world data, not owning it.

Launching a "like button" is admitting that they were wrong in their core vision. Can the world be organized automatically by algorithm, from news to search ? It is an impossible equation to solve for them without deeper changes in Google's vision. In other words, the social graph that Facebook wants to build is actually what the web is about, and should be at the heart of the open web. Copying that closed vision of the web is a mistake.

What happen today looks like Yahoo!'s revenge (even if Yahoo! company is now out of the game): curation is the return of directories of content, with a larger number of curators and ranking depending on the proximity of the curators to yourself.

Google simply cannot win with this definition of the "social web". If social is about building competing subsets of Internet, defined by their user base, Google will decline on search, their core business. They cannot remain the entry point on the Internet for a large number of users, because they could not be able to bring them on every place you might want to go on the web.

The solution for Google requires to change the view of the social web, not to build their own clone of social network. Google social should be about putting people at the heart of an open Internet, where companies keeps on competing on algorithms. They already have valuable tools to rely on: google reader can share publicly what you like to read, blogger to publish your thought. They can build "Google People" around them.

I think that Google should keep on producing tools to make easy to share publicly what people would like to share, from status, to picture, though and links. This should be published in a well defined and documented way that will lead to a more open web. To be frank, such standards are emerging in many initiatives (webfinger, salmon, foaf, ostatus, opensocial and many others). Google promotes or even design several of those web protocols.

However, Google +1, until it leads to publicly available data on user profile, is a move in the wrong direction for Google. Let's hope for the Internet that their next move will open the data and bring the competition back to algorithms.

Credits: Photo by Blaise Alleyne

ejabberd@jabber.ru: ejabberd 2.1.7, 3.0.0-alpha-3 and exmpp 0.9.7 -- security release

Wed, 01 Jun 2011 17:05:55 +0000

ejabberd 2.1.7, and ejabberd 3.0.0-alpha-3, and exmpp 0.9.7 have been released, after a few months of development. They contain a lot of bugfixes, improvements and some new features.

If you have ejabberd running in a public server, please update it immediately: those releases contain a security fix that disables entity expansion completely to prevent billion laughs DoS attack (CVE-2011-1753).

ejabberd 2.1.7

This release contains many bugfixes, improvements and a few new features.

A short list of changes:

erlang.org RSS Feed: R14B03 released

Wed, 25 May 2011 16:00:00 +0000

Erlang/OTP R14B03 has been released as planned on May 25:th 2011. It is the third R14 service release.

See the release notes in the readme file

Download the new release from the download page.

Highlights:

Diameter is a brand new application in this release. The application support the diameter protocol specified in RFC 3588 and is intended to provide an Authentication, Authorization and Accounting (AAA) framework for applications.
The documentation for stdlib and kernel now uses type specifications from the source modules which should guarantee that the documentation and code are consistent with regard to the type information.

Erlang Factory News: Countdown to the Erlang Factory London!

Wed, 25 May 2011 09:01:32 +0000

There are only two weeks left to the Erlang Factory London. If you haven't registered yet, you can still do this by clicking here.

We have confirmed speakers including Robert Virding and Mike Williams (Inventors of Erlang), Tony Falco (COO of Basho Technologies), John Hughes (Inventor of QuickCheck) and Eric Merritt (Author of Erlang and OTP in Action). Many more are listed on the Speakers Page.

You are welcome to follow @erlangfactory on Twitter.

Programming in the 21st Century: Constantly Create

Sun, 22 May 2011 06:00:00 +0000

When I wrote Flickr as a Business Simulator, I was thinking purely about making a product--photos--and getting immediate feedback from a real audience. Seeing how much effort it takes to build-up a following. Learning if what you think people will like and what they actually like are the same thing.

It works just as well for learning what it's like to be in any kind of creative profession, such as an author of fiction or a recording artist.

Go look at music reviews on Amazon, and you'll see people puzzling over why a band's latest release doesn't have the spark of their earlier material, pointing out filler songs on albums, complaining about inconsistency between tracks. Sometimes the criticisms are empty, but there's often a ring of truth. There's an underlying question of why. How could a songwriter or band release material that isn't always at the pinnacle of perfection?

After years of posting photos to Flickr, I get it. I'm just going along, taking my odd photographs, when all of a sudden one resonates and breaks through and I watch the view numbers jump way up. Then I've got pressure: How can I follow that up? Sometimes I do, with a couple of winners in a row, but inevitably I can't stay at that level. Sometimes I take a break, not posting shots for a month or more, and then I lose all momentum.

When I'm at a low point, when I devolve into taking pictures of mundane subjects, pictures I know aren't good, I think about how I'm ever going to get out of that rut. Inevitably I do, though it's often a surprise when I go from a forgettable photo one day to something inspired the next.

The key for me is to keep going, to keep taking and posting photos. If I get all perfectionist then there's too much pressure, and I start second-guessing myself. If I give up when my quality drops off, then that's not solving anything. The steady progress of continual output, whether good or bad output, is part of the overall creative process.

Damien Katz: I'm in Boston next week for a Couchbase Meetup

Thu, 19 May 2011 19:03:26 +0000

Get yo couch on! Sign up here: http://www.meetup.com/Boston-CouchDB/events/17374461/

damien@couchbase.com

Erlang Training and Consulting
- News: 18 May 2011: Radio-Electronics publishes an article on Open Telecommunications Platform, OTP for Open Communications by Francesco Cesarini

Wed, 18 May 2011 16:17:43 +0000

Francesco Cesarini, Technical Director of Erlang Solutions, discusses the benefits of the Erlang Open Telecom Platform and how it can be used in product development for writing software for telecommunications systems.

In the article he looks at Erlang as a programming language, what exactly is OTP, OTP as more than a telecoms platform, and the benefits of OTP. This is an article you can't afford to miss. For the full article check out the Radio-Electronics website.

Erlang Factory News: Early Bird Rate - Book before 22 May to save £100

Wed, 18 May 2011 09:16:37 +0000

Book your place now at the Erlang Factory London 2011 at the Early Bird Rate of £495 for the conference and £1295 for the University and conference. This is a saving of £100 off the standard price of the conference!Only valid until the 22 May!

Speakers so far include , Kostis Sagonas, the leader of the HIPE Team at Uppsala University, Marcus Kern, CTO at MIG, Steve Vinoski, distributed systems expert and Scott Lystig Fritchie, senior software engineer at Basho Technologies. Many more speakers are still to be announced.

Book now to secure this fantastic rate!

Trapexit's Erlang Blog Filter: More Erlang Web Server Benchmarking

Wed, 18 May 2011 05:54:45 +0000

In my previous blog entry I questioned the value of most web server benchmarking, particularly as related to Erlang. Typical benchmarks are misleading, inaccurate, and poorly executed. Perhaps worse, the intent of publishing them seems to be to assert that the fastest web server (at least according to the tests performed) is of course also the best web server. You’d think the flaws of this fallacy would be so obvious that nobody would fall for it, but think again: watching the delicious “erlang” tag over the past few days revealed the benchmarks my blog post referred to to be one of the most bookmarked Erlang-related pages during that timeframe.

Not surprisingly, though, it looks like I’m not the only one bothered by poor benchmarking practices. Over on his blog, Mark Nottingham just published a brilliant set of rules for HTTP load testing. It’s quite instructive to take your favorite set of published web server benchmarks and see just how many of Mark’s rules they violate.

Like I hinted last time, if you want benchmarks, you are best off by far if you run them yourself. That way, their relevance to the problems you’re addressing will be much more likely, and you can run them in a similar, or even the same, environment on which you plan to deploy. You can also gear the benchmarks to much more closely resemble your applications and the loads you require them to handle. Doing the benchmarking work yourself will give you valuable hands-on experience with the servers and frameworks you’re considering, allowing you to get a feel for important factors such as feature completeness and correctness, ease of development, flexibility, and ease of deployment and runtime management/monitoring, none of which can be gauged by someone else’s performance benchmarks. Finally, by doing your own benchmarking you can also help ensure the validity and usefulness of your results by following Mark’s load testing rules.

Erlang Inside: Motivated Reasoning and Erlang vs Python vs Node

Mon, 16 May 2011 15:51:08 +0000

A comparison between Misultin, Mochiweb, Cowboy, Node.JS, and Tornado. Tests such as this one tend to focus on either raw parsing speed or total number of concurrent connections, and which is more important depends on the ultimate application. Or they perfectly tune the tester’s favorite framework and use a stock configuration for the other competitors. [...]

Erlang Inside: Erlang in Haskell

Mon, 16 May 2011 03:49:18 +0000

Haskellers who want to experiment with the concepts of Erlang in Haskell can now do so with an experimental Erlang-like distributed computing framework called ‘remote’. It’s quite basic but covers concurrency, selective receive of messages, local and global registration of processes, and trapping exits. Related Microsoft Research Paper.

Erlang Training and Consulting
- News: 11 May 2011: Juan Puig has been invited to talk at The University of Coruña's Computer Science faculties 25th Anniversary celebrations!

Wed, 11 May 2011 17:32:43 +0000

The University of Coruña's Computer Science faculty is turning 25 years, and in order to celebrate such an important event, this Thursday, 12th May 2011, talks, workshops and expositions will be run in Palexco, the main auditorium in A Coruña. We here at Erlang Solutions are proud that Juan Puig has been invited to talk at this event as one of the graduates from The University of Coruña. He will then give a more technical lecture on the Friday at the Computer Science department.

Location: Palexco auditorium, A Coruña, Spain
Time: 12th May, 12:30h
Program (in Spanish)

Juan's talk will be:
Title: Building fault-tolerant systems.
Summary: An introduction to the current software and fault-tolerant systems will be held, followed by how Erlang/OTP does provide with the right built-in features to build large and scalable fault-tolerant systems.

Friday 13,
Juan will be giving a lecture within the Computer Science faculty, more technical, within the slot of a Software Design course of the Computer Engineering degree.
Location: Room 3.3, Faculty of Computer Science, University of A Coruña, Spain
Time: 13th May, 11:30h
Title: "Design of abstract models with Erlang"
Summary: Introduction to Erlang followed by a quick OTP and its behaviours showing how to design large applications. Second part of the talk will be oriented to property based-testing design.

Scattered Thoughts: mist

Wed, 11 May 2011 09:40:59 +0000

I have a new secret project. It's called Mist. The goal is to make it easy to build, use, distribute and maintain small-scale p2p apps. By small-scale I mean apps where each invocation has at most a few dozen users. You would use Mist to build the next etherpad, not the next bittorrent.

The main inspiration behind Mist is Sugar. Sugar supports seamless sharing of activities (applications) across local mesh networks or the internet, even if some of the peers don't even have the activity installed. Where Sugar is focused on providing a working system here and now, I am more interested in how distributed programming can be made easier in the future. I also want to build applications that I can use myself so my initial target is Ubuntu/Gnome.

Starting from September, I'm going to be doing just enough freelance work to support full time work on Mist. I'm aware that this is a huge project and probably doomed to failure so there a few key principles that I hope will keep me on track. Wherever possible I will reuse existing code and protocols. Mist will be built out of small components each of which are useful by themselves. Where practical Mist will be compatible with Sugar, so that any improvements can be folded back into the Sugar codebase. Finally, Mist is a prototype. The goal will be to get something up and running to experiment with different ways of building distributed apps.

So without further ado here is a misty outline of what Mist might one day look like:

Seamless connectivity, serverless presence and activity discovery
- Mostly borrowed from Sugar
- Apps communicate over Telepathy tubes
- Use Sugar presence service for activity discovery
- LAN / ad-hoc networks supported by Telepathy Salut and Avahi
- Centralized communication supported using Telepathy Gabble to talk to Jabber servers
- Decentralized communication supported using a Telepathy plugin for erl-telehash
  - Use telehash peers for ICE
  - Use link-local XMMP to talk
  - Use telehash taps to implement Avahi compatible mDNS
- Use pgp for authentication and end-to-end encryption
Securely portable applications
- Use an existing system. Some options which will need to be researched:
  - Bitfrost is specific to the XO laptop and probably too hard to port
  - Browsers provide well-tested sandboxes and mobile code but limit access to hardware. Perhaps ChromeOS or Firefox Chromeless?
  - Java apps can possibly be installed securely. Looking at the Android source may be instructive.
Development tools for distributed apps
- Want to experiment with Bloom. I have some ideas for static analysis of bloom programs that might be fun.
- Expose libraries/APIs for common patterns eg leader election, operational transform
- Support serverless syncing of contacts, bookmarks, mail etc using DHT with enforced reciprocal storage (pretty sure I've read a paper on this somewhere)

The immediate plan for the next few months (during which I'm still working full time) is:

Get erl-telehash working (currently has no test suite and doesn't support taps, _ring/_line or rate limiting)
Get ICE working between telehash peers
Write a Telepathy backend using erl-telehash and link-local XMPP
Extend Empathy to use the telehash Telepathy backend

If you are wondering where Mist gets its name from - it's not the cloud.

Trapexit's Erlang Blog Filter: Erlang Web Server Benchmarking

Tue, 10 May 2011 02:01:01 +0000

Over on his blog, Roberto Ostinelli published “A comparison between Misultin, Mochiweb, Cowboy, NodeJS and Tornadoweb.” I was going to write a reply comment there, but it got pretty long so I decided to publish it here instead. I’m going to ignore the non-Erlang web servers it discusses and focus entirely on Erlang. I’m not trying to really pick specifically on Roberto here, but rather I decided to finally write something I’ve been meaning to write for awhile now about Erlang web servers and benchmarking.

First, I second the request made by one commenter for including Yaws in the measurements. Roberto, if you need help with the code or setup, just let me know. If one insists on writing these kinds of benchmarks, which as you’ll learn if you read this whole entry is something I question, the least he or she could do is include Yaws since it’s the granddaddy of all Erlang web servers.

Based on the benchmark code Roberto published, I wrote the following simple Yaws module to conform to the problem statement and registered it in my Yaws configuration as a “/” appmod:

-module(yaws_bench).
-export([out/1]). out(Arg) -> [{status, 200}, {content, "text/xml", case yaws_api:queryvar(Arg, "value") of {ok, Value} -> ["<http_test><value>", Value, "</value></http_test>"]; _ -> "<http_test><error>no value specified</error></http_test>" end}].

I then measured it on my Ubuntu 10.10 two-core system using Roberto’s published httperf command against the misultin and Mochiweb code he published, and found that Yaws definitely holds its own, even though it’s a full-featured web server and does not claim to be just a lightweight library offering (sometimes partial) HTTP support as some frameworks do. For some tests Yaws outperforms misultin, and for others it doesn’t. This is interesting, considering that neither Klacke nor I have made any attempts at performance improvements in Yaws recently.

Second, the benchmarks do not compare apples to apples. Both Mochiweb and Yaws, for example, produce replies that are larger in size than misultin’s replies, primarily because they both include Server and Date headers. As I’ve learned from years of helping maintain Yaws, date calculations can noticeably and surprisingly impact Erlang web server performance, yet simply leaving Date headers out isn’t an option for real-world apps since HTTP 1.1 pretty much requires them (section 13.2.3 of RFC 2616 states, “HTTP/1.1 requires origin servers to send a Date header, if possible, with every response, giving the time at which the response was generated…”). Caches use Date headers for several reasons, for example in the absence of cache-control headers to help heuristically calculate content expiration. Even ignoring the date calculation requirements, just creating and delivering larger replies due to the presence of the Server header will negatively impact any comparisons based on request/second measurements.

Third, the benchmarking approach includes no application “think time.” How many real-world apps just blast request after request down a connection without any intervening time to handle replies? If the goal is to measure something akin to real-world apps, then the benchmarks should at least be using something like httperf’s --wsess option to simulate client think time. And unfortunately doing that is hard to get right for generic benchmarks, since different client apps will have different think times.

On a related note, what exactly is the goal of these benchmarks? To imply that faster is better? That’s unfortunately a commonly-held fallacy. Given that the blog entry states that the target is dynamic applications, then consider the fact that the performance of a real-world dynamic application is often dominated by something other than the web server — perhaps some back-end service from which page data is being fetched, for example. A real-world setup greatly concerned with performance is likely to have nginx out in front, probably with a local cache, to handle fast-path requests, shunting only those requests it can’t fulfill off to the slower back-end server. Such benchmarking games are therefore often misguided as far as real-world dynamic apps are concerned because they end up measuring something that isn’t even in the critical path in a real setup.

I don’t agree with Kyle Drake’s comment on Roberto’s blog about code ugliness, since the Erlang code posted there is very clear and would look like “garbage” only to someone who doesn’t know the language. But I do agree with the sentiment, which is that for dynamic apps, what often matters is what kind of code, and how much, you have to write and maintain to support your app. Given that Erlang web servers tend to make use of the underlying Erlang/OTP facilities for HTTP parsing and socket handling, then all things considered you’re just not going to get a huge variation in performance among them, assuming they’re written halfway decently. What matters for dynamic apps are the stability of the web server/library and the programming model it offers. These are what Roberto should really be benchmarking, but of course that’s basically impossible since stability would take a long time to prove, and programming model is a matter of taste that can’t be conveniently measured using artificial benchmarking tools. This reminds me of one of my old columns on this very issue as applied to enterprise middleware, entitled “The Performance Presumption” (PDF); the short version is that people often measure performance simply because performance is relatively easy to measure. The lesson is that you shouldn’t rely on generic benchmarks, but rather you should take the time to create specific benchmarks that mimic the app you want to develop, and base your decisions on the results of that exercise.

On top of all that, I don’t really understand the desire to keep writing new Erlang web frameworks for performance reasons. As I stated earlier, if a framework uses Erlang’s built-in packet decoding and socket handling, it won’t perform a great deal better than any other Erlang web framework. OTOH, if someone writes a new framework with the hope of providing a really nice new programming model — webmachine is a fantastic example of this — then they shouldn’t be “proving” how good the programming model is by trying to show how fast it is. Ever seen webmachine being advertised via performance benchmarks? Neither have I.

Let’s face it, the Erlang web development community isn’t large enough to support numerous web servers and frameworks. I’m sure some will disagree, but publishing artificial benchmarks designed to “prove” which is best IMO results mostly in just fragmenting the community. If you really have an itch to write a fast Erlang web server, you’d help the community much more by contributing to an existing one, including the Erlang inets web server included in Erlang/OTP and now powering the Erlang website. For Yaws, Klacke and I often take patches and suggestions from our users, and we gladly welcome solid contributions intended to improve Yaws performance. If you’re just dying to show off your chops, note that improving performance in a long-lived and highly stable codebase like Yaws without breaking anyone’s code is far more challenging than writing another new server that basically doesn’t differ much from what already exists.

Or perhaps better yet, contribute to the Erlang core. IMO the next major performance improvements in Erlang web servers will come not from minor tweaks in handling binaries or such things, but rather via radical improvements in the Erlang TCP driver or even from developing a whole new HTTP-specific driver. Unlike a war of artificial benchmarks among Erlang web servers, these approaches have a great chance to improve the lot of all Erlang web systems.

erlang.org RSS Feed: ACM SIGPLAN Erlang Workshop in Tokyo on September 23, 2011

Fri, 06 May 2011 23:08:27 +0000

ACM SIGPLAN Erlang Workshop

The Tenth ACM SIGPLAN Erlang Workshop will take place in Tokyo, Japan, on September 23, 2011. Please see the call for papers here .

Erlang Factory News: Book your place at the Erlang Factory London Now and save £100!

Wed, 04 May 2011 10:31:39 +0000

Book now to secure this fantastic rate!

Anders Conbere's Journal: Learning to Program is Easy

Wed, 04 May 2011 01:42:12 +0000

Years ago I spent a summer teaching science classes to kids aged 6 to 12 at the Oregon Museum of Science and Industry. This program wasn't particularly well managed, mostly a disaster, but quite a bit of fun. The most fun I had was helping teach a class on game programming to 5th and 6th graders. The class used a pretty nice implementation of Logo called MicroWorlds which benefited from having a simple GUI and IDE for the kids to use. Maybe today we'd use something like Scratch or EToys. Either way the class began with a simple introduction to making things happen in this virtual world, and some of the basics of the command structure. And within minutes there was a volley of hands that shot into the air each with it's very own bug.

That might not sounds like much, but by the time you've engaged someone to the point that they've typed in some commands and arrived at an unexpected occurrence... you've basically got them hook line and sinker. So that day and some of the next was often spent detailing the use of variables, the intricacies of syntax, and what rules exist and why they make their life easier. By the end of the week we would have flies narrowly dodging venus fly traps, and robots that had to defeat monsters.

I'm not saying every kid produced a work of genius, and some had more difficulties than others. But the nature of programming does not have to be a complex and difficult venture. Simply mapping input to a behavior gets you pretty far. This is a big part of the idea of Scheme. How do we build a language that is simple enough that our english majors can accomplish something in it, but powerful enough that our computer science students can develop complex ideas with it. The point is that most people can be taught to create at least rudimentary programs in just a few hours of tutorial.

And this really brings me to the reason I'm writing this post. And that is, that while programming might be easy, making applications, architecting large projects, putting all the various pieces together in a way that gives you the flexibility to accomplish what you might need to, but the structure to finishing what you have to, is amazingly difficult. And two different skills. You can teach someone to use python like a calculator, and to write functions that help them manage their bills. But the ability to take that knowledge and expand upon it to write a personal finance applications is much more difficult to teach.

Some of this is abstraction. Abstraction is difficult for humans, and I say this knowing that of all the creatures we've ever encountered we're likely the most capable abstractors in existence. But while that might be true, or brains are wired to work inside a rather constrained and physical world. There's a test that's often mentioned in literature where they ask the participants to accomplish a mathematical task using variable names, and then ask it again relating it to cards. In the first case most participants answer incorrectly, while in the next they do it with ease. The great difficult of mathematics as well as programming arrises from the depths of abstraction.

Just like Mathematics might be the deep study one abstract layer after another, complex application design mirrors this. While in mathematics at the bottom might be something like Category Theory from which derives Algebra from which derive the infinity of number systems from which derive Real and Complex numbers from which derive Real Analysis and then the Calculuses and on down the chain. A Complex application might have a User Interface which abstracts a Programable Interface which calls down into a Data Layer that then talks to a Persistence Layer that talks to the Operating System on down to the bare metal. Most of the extremely competent application architects I know might be considered a software version of a "tall, thin man". Someone who is familiar with all the ramifications of his design from the top layers to the cost at the bottom layers.

What I'm trying to say is that Programming might be easy. But the understanding of both the ramifications of the current environment (what can be done with the current language and tools) and the cost of the various layers below it, is hard. And that teaching THAT is much more difficult task than teaching someone how to use a fancy calculator. Books like SICP attempt to accomplish this task by dealing with abstraction in it's own right, by merging programing and mathematics and enabling a kind of cross pollination there of. But even that can leave a student with little understanding of how to put all the pieces together. I write this because my focus since college has been into the art of programming. And I'm beginning to understand where the limitations of my studies have been, and where I'm headed. And my new goal of trying to find more people to look at my code and find new ways of expressing those problems.

Erlang Factory News: Only 5 Places left at the Erlang Factory London!

Tue, 03 May 2011 15:02:00 +0000

There are only 5 chances left to book your place at the Erlang Factory London 2011 at the Very Early Bird Rate of £395 for the conference and £1195 for the University and conference. This is a saving of £200 off the standard price of the conference!When they are gone, they are gone!

Book now to secure this fantastic rate!

erlang.org RSS Feed: The Erlang Factory London is back!

Tue, 03 May 2011 09:10:31 +0000

The Erlang Factory London is back! The dates you need for your diary are 6th, 7th and 8th June for the Erlang University courses and 9th and 10th June for the Erlang Factory Conference.

There are 10 places left at the very Early bird rate of £395 which is a saving of £200! Book now to get your place!

Programming in the 21st Century: Impressed by Slow Code

Sat, 30 Apr 2011 06:00:00 +0000

At one time I was interested in--even enthralled by--low-level optimization.

Beautiful and clever tricks abound. Got a function call followed by a return statement? Replace the pair with a single jump instruction. Once you've realized that "load effective address" operations are actually doing math, then they can subsume short sequences of adds and shifts. On processors with fast "count leading zero bits" instructions, entire loops can be replaced with a couple of lines of linear code.

I spent a long time doing that before I realized it was a mechanical process.

I don't necessarily mean mechanical in the "a good compiler can do the same thing" sense, but that it's a raw engineering problem to take a function and make it faster. Take a simple routine that potentially loops through a lot of data, like a case insensitive string comparison. The first step is to get as many instructions out of the loop as possible. See if what remains can be rephrased using fewer or more efficient instructions. Can any of the calculations be replaced with a small table? Is there a way to process multiple elements at the same time using vector instructions?

The truth is that there's no magic in taking a well-understood, working function, analyzing it, and rewriting it in a way that involves doing slightly or even dramatically less work at run-time. If I ended up with a routine that was a bottleneck, I know I could take the time to make it faster. Or someone else could. Or if it was small enough I could post it to an assembly language programming forum and come back in a couple of days when the dust settled.

What's much more interesting is speeding up something complex, a program where all the time isn't going into a couple of obvious hotspots.

All of a sudden, that view through the low-level magnifying glass is misleading. Yes, that's clearly an N-squared algorithm right there, but it may not matter at all. (It might only get called with with low values of N, for example.) This loop here contains many extraneous instructions, but that's hardly a big picture view. None of this helps with understanding the overall data flow, how much computation is really being done, and where the potential for simplification lies.

Working at that level, it makes sense to use a language that keeps you from thinking about exactly how your code maps to the underlying hardware. It can take a bit of faith to set aside deeply ingrained instincts about performance and concerns with low-level benchmarks, but I've seen Python programs that ended up faster than C. I've seen complex programs running under the Erlang virtual machine that are done executing before my finger is off the return key.

And that's what's impressive: code that is so easy to label as slow upon first glance, code containing functions that can--in isolation--be definitively proven to be dozens or hundreds of times slower than what's possible on a given CPU, and yet the overall program is decidedly one of high performance.

(If you liked this, you might enjoy Timidity Does Not Convince.)

Learn You Some Erlang: Building an Application With OTP

Fri, 29 Apr 2011 12:00:00 +0000

This chapter lets us make practical use of the OTP behaviours seen so far. We do this by writing a process pool application that will let us handle resources and tasks. We explore ideas behind process trees, onion layer theories for processes and general views of how OTP can be used to write software.

Erlang Training and Consulting
- News: 20 April 2011: Registration is now open for the Erlang Factory London - Early Bird Rate Available

Wed, 20 Apr 2011 11:18:12 +0000

The registration for the Erlang Factory London 2011 is now open! Register now and you will save £100 by taking advantage of the Early-Bird offer. So far each Erlang Factory has been met with great interest so don't waste time. Register before it sells out!

The Erlang Factory London is the best place to exchange knowledge, experience and the vision for the future of Erlang. We have great speakers who will share all that with YOU. This conference has become one of the best Erlang networking events in Europe.

Among our first confirmed speakers for the Erlang Factory London are: Kostis Sagonas, the leader of the HIPE Team at Uppsala University, Marcus Kern, CTO at MIG, Steve Vinoski, distributed systems expert and Scott Lystig Fritchie, senior software engineer at Basho Technologies with many more still to be announced.

The list of speakers is available here, with many more to be announced.

In addition to the conference, there will be a three day Erlang University, allowing you to learn Erlang from the experts. You can choose from the following courses: Erlang Express, OTP Express, Erlang and Test Driven Development or Quick Check for Erlang Developers.

The Erlang University training courses will take place on the 6th, 7th and 8th June and the Erlang Factory conference on 9th and 10th June. Don't miss out - register now.

We will also be offering certification at Foundation and Intermediate level. You can register for this when you book your place at the Factory. Pricing for each exam is £195, or you can sit both for £345. For more information on Certification please visit the website.

Erlang Factory News: Erlang Factory London Open for Registration - Save £200

Wed, 20 Apr 2011 09:16:52 +0000

Be one of the first 50 people to book your place at the Erlang Factory London 2011 at the Very Early Bird Rate of £395 for the conference and £1195 for the University and conference. This is a saving of £200 off the standard price of the conference!

Book now to secure this fantastic rate!

Programming in the 21st Century: Follow the Vibrancy

Wed, 20 Apr 2011 06:00:00 +0000

Back in 1999 or 2000, I started reading a now-defunct Linux game news site. I thought the combination of enthusiastic people wanting to write video games and the excitement surrounding both Linux and open source would result in a vibrant, creative community.

Instead there were endless emulators and uninspired rewrites of stale old games.

I could theorize about why there was such a lack of spark, a lack of motivation to create anything distinctive and exciting. Perhaps most of the projects were intended to fulfill coding itches, not personal visions. I don't know. But I lost interest, and I stopped following that site

When I wanted to modernize my programming skills, I took a long look at Lisp. It's a beautiful and powerful language, but I was put off by the community. It was a justifiably smug community, yes, but it was an empty smugness. Where were the people using this amazing technology to build impressive applications? Why was everyone so touchy and defensive? That doesn't directly point at the language being flawed--not by any means--but it seemed an indication that something wasn't right, that maybe there was a reason that people driven to push boundaries and create new experiences weren't drawn to the tremendous purported advantages of Lisp. So I moved on.

Vibrancy is an indicator of worthwhile technology. If people are excited, if there's a community of developers more concerned with building things than advocating or justifying, then that's a good place to be. "Worthwhile" may not mean the best or fastest, but I'll take enthusiasm and creativity over either of those.

(If you liked this, you might enjoy The Pure Tech Side is the Dark Side.)

Scattered Thoughts: telehash: router

Tue, 19 Apr 2011 18:24:35 +0000

Now that we have all the necessary datastructures we can build the router itself. Most of the routing table logic is handled by the bit_tree and bucket modules. The router just ties these together and handles I/O.

Before actually running the routing table the router has to find out its own address, as it is seen from the outside world. It does this by sending +end signals to a list of known telehash nodes (eg telehash.org:42424).

record(bootstrap, { % the state of the router when bootstrapping
          timeout, % give up if no address received before this time
          addresses % list of addresses contacted to find out our address
         }).

bootstrap(Addresses, Timeout) ->
    ?INFO([bootstrapping]),
    State = #bootstrap{timeout=Timeout, addresses=Addresses},
    {ok, _Pid} = gen_server:start_link(?MODULE, State, []).
                          
init(State) ->
    switch:listen(),
    case State of
        #bootstrap{timeout=Timeout, addresses=Addresses} ->
            Telex = telex:end_signal(util:random_end()),
            lists:foreach(fun (Address) -> switch:send(Address, Telex) end, Addresses),
            erlang:send_after(Timeout, self(), giveup);
        #state{} ->
            ok
    end,
    {ok, State}.

Then we listen until we either get a reply with a _to field or run out of time.

handle_info({switch, {recv, From, Telex}}, #bootstrap{addresses=Addresses}=Bootstrap) ->
    % bootstrapping, waiting to receive a message telling us our own address
    case {lists:member(From, Addresses), telex:get(Telex, '_to')} of
        {true, {ok, Binary}} ->
            try util:to_end(util:binary_to_address(Binary)) of
                End ->
                    Self = util:to_bits(End),
                    Table = touched(From, Self, empty_table(Self)),
                    dialer:dial(End, [From], ?ROUTER_DIAL_TIMEOUT),
                    refresh(Self, Table),
                    ?INFO([bootstrap, finished, {self, Binary}, {from, From}]),
                    {noreply, #state{self=Self, pinged=sets:new(), table=Table}}
            catch
                _ ->
                    ?WARN([bootstrap, bad_self, {self, Binary}, {from, From}]),
                    {noreply, Bootstrap}
            end;
        _ ->
            {noreply, Bootstrap}
    end;

handle_info(giveup, #bootstrap{}=Bootstrap) ->
    % failed to bootstrap, die
    ?INFO([giveup, {state, Bootstrap}]),
    {stop, {shutdown, gaveup}, Bootstrap};

Once we know our own address we can fill in the state record and start managing the routing table.

-record(state, { % the state of the router in normal operation
          self, % the bits of the routers own end
          pinged, % set of addresses which have been pinged and not yet replied/timedout
          table % the routing table, a bit_tree containing buckets of nodes
         }).

One of the jobs of the router is to remove unresponsive nodes from the routing table. To check if a node is responsive we just a random +end signal and wait for a reply. If the node is unresponsive it gets marked as stale and we try to find a suitable replacement. The node won't actually be dropped from the table until a replacement is found - this prevents the table from getting flushed if our network connection goes down.

ping(To) ->
    Telex = telex:end_signal(util:random_end()),
    % do this in a message to self to avoid some awkward control flow
    self() ! {pinging, To},
    switch:send(To, Telex),
    erlang:send_after(?ROUTER_PING_TIMEOUT, self(), {timeout, Address}).

timedout(Address, Self, Table) ->
    bit_tree:update(
      fun (_Suffix, _Depth, _Gap, Bucket) ->
              case bucket:timedout(Address, Bucket) of
                  {node, Node, Update} ->
                      % try to touch this node, might be suitable replacement
                      ping(Node),
                      Update;
                  Update ->
                      Update
              end
      end,
      util:to_bits(Address),
      Self,
      Table
     ).

handle_info({pinging, Address}, #state{pinged=Pinged}=State) ->
    % do this in a message to self to avoid some awkward control flow
    ?INFO([recording_ping, {address, Address}]),
    Pinged2 = sets:add_element(Address, Pinged),
    {noreply, State#state{pinged=Pinged2}};
handle_info({timeout, Address}, #state{self=Self, pinged=Pinged, table=Table}=State) ->
    case lists:member(Address, Pinged) of
        true ->
            % ping timedout
            ?INFO([timeout, {address, Address}]),
            Table2 = timedout(Address, Self, Table),
            {ok, State#state{table=Table2}};
        false ->
            % address already replied
            {ok, State}
    end;

One of the rules of the router is that it should never pass on information about a node that it hasn't personally confirmed to exist. Once we receive a message from a node we know that it exists (later we will implement _ring/_line to protect against address spoofing):

touched(Address, Self, Table) ->
    bit_tree:update(
      fun (Suffix, _Depth, Gap, Bucket) ->
              May_split = (Gap < ?K), % !!! or (Depth < ?ROUTER_TABLE_EXPANSION)
              bucket:touched(Address, Suffix, now(), Bucket, May_split)
      end,
      util:to_bits(Address),
      Self,
      Table
     ).

On receiving a .see command we record all the contained addresses as potential nodes and ping them to try to confirm their existence.

seen(Address, Self, Table) ->
    bit_tree:update(
      fun (Suffix, _Depth, _Gap, Bucket) ->
              case bucket:seen(Address, Suffix, now(), Bucket) of
                  {node, Node, Update} ->
                      % check if this node is stale
                      ping(Node),
                      Update;
                  Update ->
                      Update
              end
      end,
      util:to_bits(Address),
      Self,
      Table
     ).

On receiving a +end signal we reply with a .see command containing the nearest K addresses which we have confirmed to exist.

see(To, End, Table) ->
    Telex = telex:see_command(nearest(?K, End, Table)),
    switch:send(To, Telex).

nearest(N, End, Table) when N>=0 ->
    Bits = util:to_bits(End),
    iter:take(
      N,
      iter:flatten(
        iter:map(
          fun ({_Prefix, Bucket}) -> bucket:by_dist(End, Bucket) end,
          bit_tree:iter(Bits, Table)))).

On receiving a message we have handle the above three cases, which gets a little ugly.

handle_info({switch, {recv, From, Telex}}, #state{self=Self, pinged=Pinged, table=Table}=State) ->
    % this counts as a reply
    Pinged2 = sets:del_element(From, Pinged),
    % touched the sender
    % !!! eventually will check _line here
    ?INFO([touched, {node, From}]),
    Table2 = touched(From, Self, Table),
    % maybe seen some nodes
    Table3 =
        case telex:get(Telex, '.see') of
            {ok, Binaries} ->
                try [util:binary_to_address(Bin) || Bin <- Binaries] of
                    Addresses ->
                        ?INFO([seen, {nodes, Addresses}, {from, From}]),
                        lists:foldl(fun (Address, Table_acc) -> seen(Address, Self, Table_acc) end, Table2, Addresses)
                catch
                    _ ->
                        ?INFO([bad_seen, {nodes, Binaries}, {from, From}]),
                        Table2
                end;
            _ ->
                Table2
        end,
    % maybe send some nodes back
    case telex:get(Telex, '+end') of
        {ok, Hex} ->
            try util:hex_to_end(Hex) of
                End ->
                    ?INFO([see, {'end', End}, {from, From}]),
                    see(From, End, Table3)
            catch
                _ ->
                    ?WARN([bad_see, {'end', Hex}, {from, From}])
            end;
        _ ->
            ok
    end,
    {noreply, State#state{pinged=Pinged2, table=Table2}};

The last responsibility of the router is to periodically refresh buckets which haven't recently seen any activity.

handle_info(refresh, #state{self=Self, table=Table}=State) ->
    ?INFO([refreshing_table]),
    refresh(Self, Table),
    {noreply, State};
handle_info({dialed, _, _}, State) ->
    % response from a bucket refresh, we don't care
    {noreply, State};

dialed(Address, Self, Table) ->
    bit_tree:update(
      fun (_Suffix, _Depth, _Gap, Bucket) ->
              bucket:dialed(now(), Bucket)
      end,
      util:to_bits(Address),
      Self,
      Table
     ).

needs_refresh(Bucket, Now) ->
    case bucket:last_dialed(Bucket) of
        never ->
            true;
        Last ->
            (timer:now_diff(Now, Last) div 1000) < ?ROUTER_REFRESH_TIME
    end.

refresh(Self, Table) ->
    Now = now(),
    iter:foreach(
      fun ({Prefix, Bucket}) ->
              case needs_refresh(Bucket, Now) of
                  true ->
                      ?INFO([refreshing_bucket, {prefix, Prefix}, {bucket, Bucket}]),
                      To = util:random_end(Prefix),
                      From = nearest(?K, To, Table),
                      dialer:dial(To, From, ?ROUTER_DIAL_TIMEOUT);
                  false ->
                      ok
              end
      end,
      bit_tree:iter(Self, Table)
     ),
    erlang:send_after(?ROUTER_REFRESH_TIME, self(), refresh),
    ok.

That's it. As usual the (untested) code is in the repo. The next post will probably deal with taps.

Scattered Thoughts: telehash: gen_event woes

Tue, 19 Apr 2011 17:33:57 +0000

I ran into some tricky bugs caused by a misconception I had about gen_event. Since this is not explicitly stated in the gen_event documentation I will say it here: gen_event does NOT spawn individual processes for each handler. Each handler is run sequentially in the event manager process.

Now obviously the documentation is not at fault here. I assumed that each handler got its own process solely because the callbacks resembled gen_server. However, a little googling reveals that several other people made the same mistake so I thought it was worth mentioning.

Here is how I found this out. I was working on the router implementation for telehash. When I tested the bootstrapping algorithm everything looked fine until the first dial, after which nothing else happened. Straight away I suspected a bug in the dialer, but repeating the exact same call in the console worked fine. After a few deadends I opened pman to look for anything suspicious but couldn't find the dialer process (because it doesn't exist, it's an event handler). I assumed that it was somehow crashing silently and wasted an hour or so reading and rereading the code and stepping through various calls in the debugger. No matter what I tried the dialer worked absolutely perfectly unless it was called by the router.

Eventually I noticed that the switch_event process was blocking inside a receive and the whole thing unravelled. The dialer is an event handler so when started it calls:

gen_event:add_handler(switch_event, dialer, State)

which is a synchronous call to the switch_event process. The router event handler is running inside the switch_event process so when the router tries to dial it deadlocks.

The moral of this story is RTFM.

This is easily fixed by changing

dialer:dial(End, [Address])

spawn(fun () -> dialer:dial(End, [Address]) end)

but there were more problems. Most of the event handlers used erlang:send_after to handle timeouts but since they all run in the same process they all receive each others timeouts. Also, every event handler is run sequentially so the switch_event process becomes a huge bottleneck.

The solution I settled on was to change each event handler into a gen_server and write a simple event handler that just forwards events to its owner. By using gen_event:add_sup_handler and listening for event handler exits we can keep the two in sync.

Erlang Factory News: Erlang Factory SF Bay Area 2011 - A Great Success

Tue, 19 Apr 2011 14:01:55 +0000

On the 24th - 25th March 2011, over 170 people gathered in the SF Bay Area for the 3rd Erlang Factory. With two Keynotes this year delivered by, Kostis Sagonas and Dan Ingalls, 10 extra talks compared to last year and a record-breaking number of delegates, this year's conference was nothing short of a tremendous success.

We would like to thank everyone that attended and keep your eyes out for the videos that will be released soon!

Erlang Factory News: Only 5 Places left at the Erlang Factory SF Bay Area!

Tue, 19 Apr 2011 14:01:55 +0000

We only have 5 places left at the Erlang Factory SF Bay Area so if you haven't booked yet, don't miss out and book now! The OTP Express course has now sold out but there are a few places left on the other courses at the University.

Erlang Factory News: Erlang Factory London 2011 - Talk Submission Open!

Tue, 19 Apr 2011 14:01:55 +0000

Do you have something you want to share with the programming world? A great project, something new or innovative then we want to hear from you. If you want to give a presentation at the Erlang Factory, you can find more information how to do that, here.

The submission deadline is the 11th April 2011.

Erlang Factory News: Early Bird Rate - 1 Week left to save over $200!

Tue, 19 Apr 2011 14:01:55 +0000

There is only 1 week left to book your place at the Early Bird Rate of $690 so don't miss out!

You will get the chance to, see some fantastic names from the Erlang world including, Joe Armstrong, Kostis Sagonas, Damien Katz, and Dan Ingalls, network with your fellow programmers and find out what’s new not only in the world of Erlang but in the world of programming.

Erlang Factory News: Erlang Factory SF Bay Area - Save over $200

Tue, 19 Apr 2011 14:01:55 +0000

We are offering a fantastic saving of over $200 when you register for the conference before the 1st of March 2011.

Programming in the 21st Century: Revisiting "Tricky When You Least Expect It"

Fri, 15 Apr 2011 06:00:00 +0000

Since writing Tricky When You Least Expect It in June 2010, I've gotten a number of responses offering better solutions to the angle_diff problem. The final version I presented in the original article was this:

angle_diff(Begin, End) ->
   D = End - Begin,
   DA = abs(D),
   case {DA > 180, D > 0} of
      {true, true} -> DA - 360;
      {true, _}    -> 360 - DA;
      _ -> D
   end.

But, maybe surprisingly, this function can be written in two lines:

angle_diff(Begin, End) ->
   (End - Begin + 540) rem 360 - 180.

The key is to shift the difference into the range -180 to 180 before the modulo operation. The "- 180" at the end adjusts it back. One quirk of Erlang is that the modulo operator (rem) gives a negative result if the first value is negative. That's easily fixed by adding 360 to the difference (180 + 360 = 540) to ensure that it's always positive. (Remember that adding 360 to an angle gives the same angle.)

So how did I miss this simpler solution? I got off track by by thinking I needed an absolute value, and things went downhill from there. I'd like to think if I could rewind and re-attempt the problem from scratch, then I'd see the error of my ways, but I suspect I'd miss it the second time, too. And that's what I was getting at when I wrote "Tricky When You Least Expect It": that you never know when it will take some real thought to solve a seemingly simple problem.

(Thanks to Samuel Tardieu, Benjamin Newman, and Greg Rosenblatt, who all sent almost identical solutions.)

Erlang Training and Consulting
- News: 13 April 2011: Ulf Wiger and Robert Virding to be on the judging panel at Spawnfest - 9 - 10 July 2011

Wed, 13 Apr 2011 15:02:44 +0000

We are happy to announce that our CTO, Ulf Wiger and Robert Virding will be on the panel of judges at this years Spawnfest. Spawnfest is an annual 48 hour development competition in which teams of skilled application developers get exactly one weekend to develop the best Erlang applications that they can. First contest is scheduled for July 9th and 10th. Please follow them on twitter to stay updated.

The judging panel also includes Bob Ippolito, Co-Founder/CTO of Mochi Media, Inc.

For more information on the event please see our events page.

Erlang Inside: Memory Models in Erlang vs Java

Wed, 13 Apr 2011 11:58:36 +0000

A view of Erlang (focused on the memory model) from a “Java Code Geek” – http://www.javacodegeeks.com/2011/04/erlang-vs-java-memory-architecture.html

Programming in the 21st Century: Caught-Up with 20 Years of UI Criticism

Mon, 11 Apr 2011 06:00:00 +0000

Interaction designers have leveled some harsh criticisms at the GUI status-quo over the last 20+ years. The mouse is an inefficient input device. The desktop metaphor is awkward and misguided. Users shouldn't be exposed to low-level details like the raw file-system and having to save their work.

And they were right.

But instead of better human/computer interaction, we got faster processors and hotter processors and multiple processors and entire processors devoted to 3D graphics. None of which are bad, mind you, but it was always odd to see such tremendous advances in hardware while the researchers promoting more pleasant user experiences wrote books that were eagerly read by people who enjoyed smirking at the wrong-headedness of an entire industry--yet who weren't motivated enough to do anything about it. Or so it seemed.

It's miraculous that in 2011, the biggest selling computers are mouse-free, run programs that take over the entire screen without the noise of a faux-desktop, and the entire concept of "saving" has been rendered obsolete.

Clearly, someone listened.

(If you liked this, you might enjoy Free Your Technical Aesthetic from the 1970s.)

Functional Jobs: Programmer - Lisp and other languages at Streamtech bv (Full-time)

Sun, 10 Apr 2011 20:42:51 +0000

Streamtech is a small Dutch company specialized in the development of high quality custom (web) applications. We are looking for programmers who like to write pragmatic but elegant code.

who we're looking for

Elegant code makes you happy.

To you, programming is not just a job but a hobby. You've been programming for fun for years.

You're not blind to practical demands, but you relish doing things the right way. When something seems to work but you're not sure how or why, you bang your head against it until you understand.

You like the net.

You love exploring interesting new languages, concepts and approaches.

Having played with Lisp, Haskell, Erlang, Prolog, Smalltalk, or other non-mainstream languages gets you bonus points. The same goes for language/compiler implementation, kernel hacking, cryptography, etc. Impress us with your need to learn cool stuff.

Because of the unfortunately strict immigration laws, already having permission to work in the Netherlands is also a plus.

the job

You'll be working on the development of our diverse web based projects – in streaming video, lawful interception, narrowcasting, and many other topics – together with the rest of the programming team, from our office in The Hague.

Together with the rest of the team, you'll make applications that are technically elegant and pleasant to work with, using whatever language and technologies suit the project best.

You will be involved in projects from A to Z – including finding out what exactly the customer needs, choosing the best approach for the job, and then making it so.

what we use

In a perfect world, we'd be Lisping all day long. In the real world, the needs of our clients don't always allow this, so there's also quite a lot of Python, and even some C and PHP.

We don't expect you to have experience with all the languages we use, but we do expect you to be able to learn new things and enjoy doing it.

what we offer

A challenging, fun job in a small but professional team that does not think Java, MVC and SOAP are the answer to every question.

REALITY.SYS: Adlige Posse

Sun, 10 Apr 2011 17:03:14 +0000

Eigentlich ist Guttenberg ist Sachen Öffentlichkeit sehr konsequent: immer wenn es darum ging, sich in Pose zu setzen, waren die Kameras willkommen (selbst eine Talk-Show aus Afghanistan durfte sich das gemeine Volk antun). Später aber, als es darum ging, für eigene Fehler gerade zu stehen, wurde zensiert was das Zeug hält. Zuerst versuchte er, vor “ausgewählten” Presse-Vertretern seine erste Stellungnahme zu den Plagiatsvorwürfen zu geben, während zur gleichen Zeit eine Bundespresse-Konferenz abgehalten wurde. Danach verbot er die Live-Übertragung seines Rücktritts aus dem Bundesverteidigungsministerium, so dass das gemeine Volk nur durch das zwischengeschaltete Handy einer NTV-Reporterin die “live Stimme” des ehemaligen Dr./Ministers hören konnte. Und jetzt erleben wir einen weiteren Zensurversuch. Typisch adliges aber sicherlich nicht “edles” Verhalten: immer schöne Posen halten, aber kaum Leistung erbringen. Dass solche “erlauchten” Kreise wenig von Fleißarbeit, Mühsal und Ehrlichkeit halten, dasselbe aber ihren Wählern einbläuen, muss hier nicht ausführlich erörtert werden. ABER: so hat es vielleicht in den Jahrhunderten zuvor funktioniert. In dieser Zeit, und das hat mittlerweile auch ein Herr KTG begriffen, reicht eine bloße Pose nicht mehr aus, um die Massen zu begeistern. Nicht einmal die Royal-versessenen Briten erlauben ihren blaublütigen Posern mehr zu sein, als Yellow-Press-Volontäre! Herr KTG ist nur ein typisches Beispiel für die Oberflächlichkeit der ehemaligen Herrschenden dieses Landes, die wohl immer noch davon träumen, durch bloßes Posing und “gute” Manieren das Volk zu steuern. Diese Zeiten sind für den größten Teil Deutschlands aber schon längst vorbei. Die feudale Matrix funktioniert nicht mehr. KTG, du bist raus!

Process-one Blogs: Sea Beyond 2011 Talk 7: Jukka Alakontiola on Nokia Push Notifications

Thu, 07 Apr 2011 12:53:26 +0000

Nokia Push Notification system has been developed by Nokia to send notifications on mobile directly.

Jukka Alakontiola presents the architecture of Nokia push and speaks about mobile related XMPP optimisations.

Do not miss this video, it features lots of great insights on mobile realtime services.

Here is the video of his presentation:

You can see the slides here:

Do not miss the event summary and the other videos from Sea Beyond event.

Erlang Training and Consulting
- Jobs: Erlang Developer, Mountain View, CA, USA

Wed, 06 Apr 2011 14:00:56 +0000

Erlang Developer (Mountain View, CA)
As a software engineer with experience in Erlang and call processing, you would be responsible for evolving the key call processing and event distribution systems at the heart of the platform. Your responsibilities will include designing and developing new features and capabilities in the Erlang based components.

Key Responsibilities
   •   Design and develop key features and capabilities in the Erlang based systems
   •   Maintain and evolve the call processing and event distribution components

Requirements
   •   Bachelors Degree or 5+ years of comparable experience in architecture and development using Erlang and OTP
   •   Experience in eunit and quickcheck
   •   Experience in developing call processing software, telephony systems
   •   Experience working in an agile environment (Scrum, Kanban or both)

Desired Skills
   •   Experience in Python, Perl, Java, Javascript
   •   Knowledge of telephony protocols (SIP, RTP, RTMP, etc.)
   •   Experience with Jira, Perforce
   •   Experience with AGILE development
   •   Experience with Test Driven Development
   •   Must be team-oriented, possess a positive attitude and work well with others
   •   Must be flexible and able to work accurately in a fast-paced environment
   •   Ability to work independently and deliver on schedule with little supervision
   •   Ability to quickly understand and articulate interactions in a complex technical environment
   •   Able to plan and execute own tasks in timely manner
   •   Passionate about software development, willing to learn new technology, self-motivated with high technical competency

Please send your CV and a cover letter when applying.

Scattered Thoughts: telehash: buckets

Wed, 30 Mar 2011 19:44:03 +0000

The other half of the routing table is the buckets which store node addresses.

Usual disclaimer: none of this is properly tested yet.

The Kademlia paper has much to say on the issue of routing, most of it contradictory. My takeaway from many readings and from browsing the source code of various different implementations is that the following points are the most important:

each bucket should contain at most K nodes
we should only ever report node addresses which we have personally confirmed exist
responsive nodes should never be removed from buckets
nodes should never be removed from buckets unless a suitable replacement exists

The first three points make the routing table very resistant to flooding and spoofing. In particular, they prevent a common attack for p2p networks where some bad guy floods the routing tables of all the other nodes so that all traffic is routed through nodes controlled by the bad guy. The last point prevents nodes from flushing their routing tables if their own network connection goes down.

I think the implementation I have come up with is fairly clean, if a little lengthy. Like the bit_tree I want the bucket to be completely pure. All side effects will be handled by the router itself. The main data structures are explained pretty well by the comments:

-define(K, ?DIAL_DEPTH).

-record(node, {
          address, % node #address{} record
          'end', % node end
          suffix, % the remaining bits of the nodes end left over from the bit_tree
          status, % one of [live, stale, cache]
          last_seen % for live/stale nodes, the time of the last received message. for cache nodes the time of the last .see reference to the node
         }).

-record(bucket, {
          nodes, % gb_tree mapping addresses to {Status, Last_seen}
          % remaining fields are pq's of nodes sorted by their last_seen field
          live, % nodes currently expected to be alive
          stale, % nodes which have not replied recently
          cache % potential nodes which we have not yet verified
         }). % invariant: pq_maps:size(live) + pq_maps:size(stale) <= ?K

The bucket is a two-stage data structure. This allows us the keep nodes of different statuses sorted by the last_seen time but still be able to get/delete nodes just knowing the address. The get_node function should make it clear how this works:

get_node(Address,
         #bucket{nodes=Nodes, live=Live, stale=Stale, cache=Cache}) ->
    case gb_trees:lookup(Address, Nodes) of
        {value, {Status, Last_seen}} ->
            case Status of
                live ->
                    {ok, pq_maps:get({Last_seen, Address}, Live)};
                stale ->
                    {ok, pq_maps:get({Last_seen, Address}, Stale)};
                cache ->
                    {ok, pq_maps:get({Last_seen, Address}, Cache)}
            end;
        none ->
            none
    end.

This is only long because records are purely a compile time structure ie we can't write Bucket#bucket.Status so we have to pattern match on Status instead. We also define add_node/2, del_node/2 and update_node/2, which look pretty similar, as well as to_list/1, from_list/1 and sizes/1.

The router is going to react to various events by calling the appropriate bucket functions and possibly sending out messages based on the result. The first event it has to handle is a node becoming unresponsive. The bucket will mark this node as stale and return a cache node which the router can attempt to verify.

% this address failed to reply in a timely manner
timedout(Address, Bucket) ->
    log:info([?MODULE, timing_out, Address, Bucket]),
    case get_node(Address, Bucket) of
        {ok, Node} ->
            case Node#node.status of
                live ->
                    % mark as stale, return a cache node that might be a suitable replacement
                    Bucket2 = update_node(Node#node{status=stale}, Bucket),
                    pop_cache_hi(Bucket2);
                _ ->
                    % if cache or stale already we don't care
                    ok(Bucket)
            end;
        none ->
            % wtf? we don't even know this node?
            % one way this could happen:
            % send N1, sendN1, timedout N1, add N2 (pushing N1 out of stale), timedout N1
            log:warning([?MODULE, unknown_node_timedout, Address, Bucket]),
            ok(Bucket)
    end.

% return most recently seen cache node, if any exist
pop_cache_hi(#bucket{cache=Cache}=Bucket) ->
    case pq_maps:pop_hi(Cache) of
        {_Key, Node, Cache2} ->
            {node, Node, ok(Bucket#bucket{cache=Cache2})};
        false ->
            ok(Bucket)
    end.

The next event is receiving a .see command. This may be as a result of a +end sent by the router but is more likely to be part of a dialing process happening elsewhere. The beauty of Kademlia is that the router can populate the routing table just by listening in on dialing attempts.

For each node listed in the .see command the router will call seen. This adds the node to the cache and returns the least recently seen live node so the router can check that it is still responsive.

% this address has been reported to exist by another node
seen(Address, Time, Suffix, Bucket) ->
    log:info([?MODULE, seeing, Address, Bucket]),
    case get_node(Address, Bucket) of
        {ok, Node} ->
            case Node#node.status of
                cache ->
                    % for cache nodes being in a .see is good enough
                    ok(update_node(Node#node{last_seen=Time}, Bucket));
                _ ->
                    % for live/stale nodes we require direct contact so ignore this
                    ok(Bucket)
            end;
        none ->
            % put node in cache, return a live node to ping
            Node = #node{
              address = Address,
              'end' = util:to_end(Address),
              suffix = Suffix,
              status = cache,
              last_seen = Time
             },
            Bucket2 = add_node(Node, Bucket),
            case peek_live_lo(Bucket) of
                none -> ok(Bucket2);
                {ok, Live_node} -> {node, Live_node, ok(Bucket2)}
            end
    end.

% return the oldest live node
peek_live_lo(#bucket{live=Live}) ->
    case pq_maps:peek_lo(Live) of
        none -> none;
        {_, Node} -> {ok, Node}
    end.

Any time we receive a message we learn that the node sending it exists (or not - we'll deal with address spoofing in a later post) so we can potentially mark it as a live node. The touched function checks if the node is already in the bucket or if it needs to be added.

% this address has been verified as actually existing
touched(Address, Suffix, Time, Bucket, May_split) ->
    log:info([?MODULE, touching, Address, Bucket]),
    case get_node(Address, Bucket) of
        {ok, Node} ->
            case Node#node.status of
                live ->
                    % update last_seen time
                    ok(update_node(Node#node{last_seen=Time}, Bucket));
                stale ->
                    % update last_seen time and promote to live
                    ok(update_node(Node#node{last_seen=Time, status=live}, Bucket));
                cache ->
                    % potentially promote the node to live
                    Bucket2 = del_node(Node, Bucket),
                    new_node(Address, Suffix, Time, Bucket2, May_split)
            end;
        none ->
            % potentially add the node to live
            new_node(Address, Suffix, Time, Bucket, May_split)
    end.

If the node needs to be added then touched calls new_node which decides if there is space in the bucket and, if so, adds the new node. If the bucket is full and May_split is true then new_node will split the bucket before adding the new node. Deciding whether or not splitting is allowed is the routers job.

% assumes Address is not already in Bucket, otherwise crashes
new_node(Address, Suffix, Time, Bucket, May_split) ->
    Node = #node{
      address = Address,
      'end' = util:to_end(Address),
      suffix = Suffix,
      status = undefined,
      last_seen = Time
     },
    {Lives, Stales, _} = sizes(Bucket),
    if
        Lives + Stales < ?K ->
            % space left in live
            log:info([?MODULE, adding, Node, Bucket]),
            ok(add_node(Node#node{status=live}, Bucket));
        (Lives < ?K) and (Stales > 0) ->
            % space left in live if we push something out of stale
            log:info([?MODULE, adding, Node, Bucket]),
            Bucket2 = drop_stale(Bucket),
            ok(add_node(Node#node{status=live}, Bucket2));
        May_split and (Suffix /= []) ->
            % allowed to split the bucket to make space
            log:info([?MODULE, splitting, Node, Bucket]),
            {split, BucketF, BucketT} = split(Bucket),
            [Bit | Suffix2] = Suffix,
            case Bit of
                false ->
                    BucketF2 = new_node(Address, Suffix2, Time, BucketF, May_split),
                    {split, BucketF2, BucketT};
                true ->
                    BucketT2 = new_node(Address, Suffix2, Time, BucketT, May_split),
                    {split, BucketF, BucketT2}
            end;
        true ->
            % not allowed to split, will have to go in the cache
            log:info([?MODULE, caching, Node, Bucket]),
            ok(add_node(Node#node{status=cache}, bucket))
    end.

% drop the oldest stale node, crashes if none exist
drop_stale(#bucket{stale=Stale}=Bucket) ->
    {_Key, _Node, Stale2} = pq_maps:pop_one_hi(Stale),
    Bucket#bucket{stale=Stale2}.

split(Bucket) ->
    Nodes = to_list(Bucket),
    NodesF = [Node#node{suffix=Suffix2} || #node{suffix=[false|Suffix2]}=Node <- Nodes],
    NodesT = [Node#node{suffix=Suffix2} || #node{suffix=[true|Suffix2]}=Node <- Nodes],
    {split, from_list(NodesF), from_list(NodesT)}.

Finally, upon receiving a +end signal the router needs to reply with a .see command listing the K nearest nodes to the specified end. This will be done using a combination of bit_tree:iter and bucket:nearest.

nearest(N, End, #bucket{live=Live, stale=Stale}) ->
    Nodes = pq_maps:to_list(Live) ++ pq_maps:to_list(Stale),
    Num_nodes = pq_maps:size(Live) + pq_maps:size(Stale),
    if
        Num_nodes =< N ->
            [Node#node.address || {_Key, Node} <- Nodes];
        true ->
            % !!! maybe should prefer to return live nodes even if further away
            Nodes_by_dist = [{util:distance(End, Node#node.'end'), Node} || {_Key, Node} <- pq_maps:to_list(Live)],
            {Closest, _} = lists:split(N, lists:sort(Nodes_by_dist)),
            [Node#node.address || {_Dist, Node} <- Closest]
    end.

As usual all the code is sitting in the repo.

Process-one Blogs: Sea Beyond 2011 Talk 6: Diana Cheng on OneSocialWeb

Tue, 29 Mar 2011 11:45:05 +0000

Diana Cheng, from Vodafone, introduces OneSocialWeb project.

Diana Cheng presents OneSocialWeb distributed social web initiative, primarily based on XMPP and Activitystreams standards.

Here is the video of her presentation:

You can see the slides here:

Do not miss the event summary and the other videos from Sea Beyond event.

Erlang Factory News: Erlang Factory SF Bay Area 2011 - A Great Success

Tue, 29 Mar 2011 11:34:01 +0000

Trapexit's Erlang Blog Filter: London Erlang User Group Slides

Mon, 28 Mar 2011 18:00:22 +0000

While attending QCon recently I also spoke at the London Erlang User Group meeting. I gave a quick talk about Erlang Native Implemented Functions (NIFs) as used in my little erlsha2 implementation. Slides are here (pdf).

Programming in the 21st Century: If You're Not Gonna Use It, Why Are You Building It?

Sat, 26 Mar 2011 06:00:00 +0000

Just about every image editing or photo editing program I've tried has a big collection of visual filters. There's one to make an image look like a mosaic, one to make it look like watercolors, and so on. Except for few of the most fundamental image adjustments, like saturation and sharpness, I never use any of them.

I have this suspicion that the programmers of these tools got hold of some image processing textbooks and implemented everything in them. If an algorithm had any tweakable parameters, then those were exposed to the user as sliders.

Honestly, that sounds like something I might have done in the past. The process of implementing those filters is purely technical--almost mechanical--yet it makes the feature list longer and more impressive. And they could be fun to code up. But no consideration is given to if those filters have any practical value.

Contrast this with apps like Instagram and Hipstamatic. Those programs use your phone's camera to grab images, then apply built-in filters to them. They're fully automatic; you can't make any manual adjustments. And yet unlike all of those filter-laden photo editors I've used in the past, I'm completely hooked on Hipstamatic. It rekindled my interest in photography, and I can't thank the authors enough.

What's the difference between those apps and old-fashioned photo editors?

The Hipstamatic and Instagram filters were designed with clear goals in mind: to emulate certain retro-camera aesthetics, to serve as starting points and inspirations for photographs. Or more succinctly: they were built to be used.

If you find yourself creating something, and you don't understand how it will be used, and you don't plan on using it yourself, then it's time to take a few steps back and reevaluate what you're doing.

(If you liked this, you might like Advice to Aimless, Excited Programmers.)

RJ's Blog: Erlang rebar tutorial: generating releases and upgrades

rj@metabrew.com (Richard Jones) — Sat, 26 Mar 2011 00:00:00 +0000

During my experiments with rebar, I made a simple example app for testing upgrades and releases. This article will walk you through using rebar to create an application, lay it out properly, package and deploy it, and create and install new versions without downtime.

The code accompanying this article is in various branches of github.com/RJ/erlang_rebar_example_project.

N.B. The OTP Design Principles docs are a good place to start if you want an overview of the OTP approach to Erlang apps and releases. However, rebar isn’t (yet) part of OTP, so consider that background reading. Rebar makes things much easier.

Creating the project

Build rebar:

$ cd ~/src
$ git clone https://github.com/basho/rebar.git
Initialized empty Git repository in /tmp/rebar/.git/
remote: Counting objects: 2651, done.
remote: Compressing objects: 100% (1344/1344), done.
remote: Total 2651 (delta 1540), reused 2227 (delta 1174)
Receiving objects: 100% (2651/2651), 622.99 KiB | 495 KiB/s, done.
Resolving deltas: 100% (1540/1540), done.
$ cd rebar && make
...snip....
==> rebar (compile)
Congratulations! You now have a self-contained script called "rebar" in
your current working directory. Place this script anywhere in your path
and you can use rebar to build OTP-compliant apps.

Now we’ll make a project directory called “dummy_proj”, copy rebar into it, and use rebar to generate a skeleton application:

$ mkdir -p ~/src/dummy_proj/apps
$ cd ~/src/dummy_proj/
$ cp ../rebar/rebar .
$ cd apps
$ ../rebar create-app appid=dummy_proj
==> dummy_proj (create-app)
Writing src/dummy_proj.app.src
Writing src/dummy_proj_app.erl
Writing src/dummy_proj_sup.erl

To the skeleton, I added a basic gen_server called dummy_proj_server, which just keeps track of the number of times it was poked, i.e. it holds some state, for demonstration purposes.

I also renamed dummy_proj_app.erl to just dummy_proj.erl, and added a start/0 function, which is useful when starting the application during developement, when not running from a generated release.

Compiling with rebar

You need a rebar.conf, place this in the top-level project directory:

{sub_dirs, [
            "apps/dummy_proj",
            "rel"
           ]}.
{erl_opts, [debug_info, fail_on_warning]}.

{require_otp_vsn, "R14"}.

And now to compile, you do:

$ ./rebar compile
==> dummy_proj (compile)
Compiled src/dummy_proj_sup.erl
Compiled src/dummy_proj.erl
Compiled src/dummy_proj_server.erl
==> rel (compile)
==> dummy_proj (compile)

Note that you now have .beam files in apps/dummy_proj/ebin/, and the .app.src generated apps/dummy_proj/ebin/dummy_proj.app for you, with a complete modules list.

N.B. I made a simple Makefile that calls ‘rebar compile’, because I’m too used to typing make. Find it in the git repo.

Running your app (development)

Here’s how you can start the application (and sasl, for nice error reporting):

$ erl -pa apps/*/ebin -boot start_sasl -s dummy_proj
...snip...
=INFO REPORT==== 16-Mar-2011::14:17:04 ===
Starting dummy_proj application...

=PROGRESS REPORT==== 16-Mar-2011::14:17:04 ===
          supervisor: {local,dummy_proj_sup}
             started: [{pid,<0.45.0>},
                       {name,dummy_proj_server},
                       {mfargs,{dummy_proj_server,start_link,[]} },
                       {restart_type,permanent},
                       {shutdown,5000},
                       {child_type,worker}]

=PROGRESS REPORT==== 16-Mar-2011::14:17:04 ===
         application: dummy_proj
          started_at: nonode@nohost
Eshell V5.8.1  (abort with ^G)
1> dummy_proj_server:num_pokes().
0
2> dummy_proj_server:poke().     
{ok,1}
3> dummy_proj_server:poke().
{ok,2}
4> dummy_proj_server:num_pokes().
2
5>

Now you have a nice sensibly structured Erlang project that you can compile with rebar. Exit the VM with q(). and let’s use rebar to package it up, so you can deploy it on a production box.

Generating your first release

When you generate a release with rebar, and indeed if you use the erlang tools manually (not recommended, just use rebar), you end up with the whole Erlang VM and required libraries packaged up under one directory.

This means you have a self-contained environment containing Erlang, the OTP libraries you need, and all your application code and dependencies. You can just tar it up, ship it over to another machine (of the same architecture, eg GNU/Linux 64-bit), and run it there.

Creating a node config

Use rebar to create a default node configuration in a rel subdirectory:

$ mkdir rel
$ cd rel/
$ ../rebar create-node nodeid=dummynode
==> rel (create-node)
Writing reltool.config
Writing files/erl
Writing files/nodetool
Writing files/dummynode
Writing files/app.config
Writing files/vm.args

You need to edit reltool.config a little; point to to your apps directory, and make sure the version number matches your .app.src file. You should also add dummy_app to the list of applications that are started as part of the release. Here’s reltool.conf from my v1 tag

Generating the release

Back in the top level directory, just run:

$ ./rebar generate
==> rel (generate)

Now have a look in rel/dummynode. This is the release directory containing everything you need to run your application.

We are going to be creating more releases later, so rename rel/dummynode to rel/dummynode_first, and then launch it using the handy script that rebar created for us:

$ cd rel/dummynode_first
$ ./bin/dummynode console
...snip...
Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:5] [hipe] [kernel-poll:true
=INFO REPORT==== 16-Mar-2011::13:29:59 ===
Starting dummy_proj application...
Eshell V5.8.1  (abort with ^G)
(dummynode@127.0.0.1)1> 
(dummynode@127.0.0.1)1> dummy_proj_server:num_pokes().
0
(dummynode@127.0.0.1)2> dummy_proj_server:poke().     
{ok,1}
(dummynode@127.0.0.1)3> dummy_proj_server:poke().
{ok,2}
(dummynode@127.0.0.1)4> dummy_proj_server:num_pokes().
2
(dummynode@127.0.0.1)5>

Now the release is running, we never want to have to restart it ever again, so open up another console because we want to leave that running whilst we work on version 2.

N.B. In a production environment, you would start with “./bin/dummynode start” so it runs in the background, and use “dummynode attach” to get a console.

Check the 'v1 branch' on github for code up to this point.

Upgrading to Version 2

Add the poke_twice() function to dummy_proj_server.

Change the version from “1” to “2”, in both apps/dummy_proj.app.src and rel/reltool.conf.

Here’s the github diff between v1...v2

Erlang application version numbers can be any string - I tend to use a date format with letter: “20110316a”, but you can use any scheme you want. I tag releases in git with the same version as the erlang application. We’ll just use “1”, “2”, “3” here for simplicity.

N.B. If you use {vsn, git} as the version in your .app.src, rebar will get the version string from the closest git tag.

Now build the new version:

$ ./rebar compile
$ ./rebar generate

So now you have rel/dummy_proj, containing a full release (VM included) of version 2. If you don’t care about online-upgrades, you could just kill your version 1 VM, and start version 2 from this new release directory.

Writing the .appup upgrade instructions

In order to make an upgrade, you must have a valid .appup file. This tells the erlang release_handler how to upgrade and downgrade between specific versions of your application.

Rebar has a (relatively new) command called ‘generate-appups’. I’ll show how it works, but ultimately we’ll write our .appup manually, and keep it in our project directory (in git).

$ ./rebar generate-appups previous_release=dummynode_first
==> rel (generate-appups)
Generated appup for dummy_proj
Appup generation complete
$ cat ./rel/dummynode/lib/dummy_proj-2/ebin/dummy_proj.appup
%% appup generated for dummy_proj by rebar ("2011/03/16 13:37:43")¬                                   
{"2", [{"1", [{update,dummy_proj_server,{advanced,[]}}]}], [{"1", []}]}.¬

Get rid of the autogenerated one, and create the appup file manually, in apps/dummy_proj/ebin/dummy_proj.appup:

{"2", 
    %% Upgrade instructions from 1 to 2
    [{"1", [
        {load_module, dummy_proj_server}    
    ]}], 
    %% Downgrade instructions from 2 to 1
    [{"1",[
        {load_module, dummy_proj_server}    
    ]}]
}.

This .appup contains instructions for upgrading and downgrading between versions “2” and “1”. Typically the downgrade instructions are the reverse of the upgrade instructions. Since we just added a function to our server process, without changing any internal state, we can just use load_module instructions. The Appup Cookbook explains the various upgrade instructions in depth.

Now generate again, overwriting the previous version 2. This will just make sure the .appup is part of the release directory:

$ ./rebar generate -f

And now, create the upgrade package:

$ ./rebar generate-upgrade previous_release=dummynode_first
==> rel (generate-upgrade)
dummynode_2 upgrade package created

The generate-upgrade command will look for rel/dummynode as the current version, and rel/dummynode_first as the previous version. It should have created the upgrade .tar.gz in rel:

$ ls -lh rel/
total 15M
drwxr-xr-x 8 rj rj 4.0K 2011-03-16 13:42 dummynode
drwxr-xr-x 8 rj rj 4.0K 2011-03-16 13:29 dummynode_first
-rw-r--r-- 1 rj rj  14M 2011-03-16 13:45 dummynode_2.tar.gz
drwxr-xr-x 2 rj rj 4.0K 2011-03-16 13:11 files
-rw-r--r-- 1 rj rj  922 2011-03-16 13:36 reltool.config

Installing the upgrade package

You should still have the VM running from dummynode_first. Make sure you called poke(), so the internal state is something other than the default. This will help illustrate that the upgrade worked seamlessly.

Copy the upgrade package to the releases directory of the running release:

$ cp rel/dummynode_2.tar.gz rel/dummynode_first/releases

Now, at the Erlang console where version 1 is running, we use release_handler to check which releases are currently available, and install our new one:

(dummynode@127.0.0.1)5> release_handler:which_releases().
[{"dummynode","1",[],permanent}]
(dummynode@127.0.0.1)6> release_handler:unpack_release("dummynode_2").
{ok,"2"}
(dummynode@127.0.0.1)7> release_handler:install_release("2").
{ok,"1",[]}   
(dummynode@127.0.0.1)8> dummy_proj_server:num_pokes().
2
(dummynode@127.0.0.1)9> dummy_proj_server:poke_twice().
{ok,4}
(dummynode@127.0.0.1)10> dummy_proj_server:num_pokes(). 
4
(dummynode@127.0.0.1)11> release_handler:which_releases().
[{"dummynode","2",
  ["kernel-2.14.1","stdlib-1.17.1","dummy_proj-2",
   "sasl-2.1.9.2","compiler-4.7.1","crypto-2.0.1",
   "syntax_tools-1.6.6","edoc-0.7.6.7","et-1.4.1","gs-1.5.13",
   "hipe-3.7.7","inets-5.5","mnesia-4.4.15","observer-0.9.8.3",
   "public_key-0.8","runtime_tools-1.8.4.1","ssl-4.0.1",
   "tools-2.6.6.1","webtool-0.8.7","wx-0.98.7","xmerl-1.2.6"],
  current},
 {"dummynode","1",[],permanent}]

The upgrade worked; you can see that the num_pokes() was preserved, and that the new poke_twice() function is available.

release_handler shows our version 2 as “current”, and the original version 1 as “permanent”. This means that although version 2 is running right now, if you restart the VM, version “1” will be booted up.

If you are happy with the upgrade, make it permanent, meaning it will boot instead of version 1 if you restart the VM:

(dummynode@127.0.0.1)12> release_handler:make_permanent("2").

Check the 'v2 branch' on github for code up to this point.

Version 3 and beyond

The upgrade from v1 to v2 was simple: we just added a fun without changing the internal #state{} record.

Erlang .appup files can do all sorts of clever stuff, allowing you to rewire your running applications during the upgrade process.

The Appup Cookbook details the various commands you can put in your .appup.

Let’s do an upgrade with a more complex appup - we’ll change the #state record in the dummy_proj_server process.

For version 3, we’ll track prods as well as pokes, which will require another field in the state record.

Here’s the github diff between v2...v3.

Check out the addition to .appup for this release:

{"3", 
    %% Upgrade instructions
    [{"2", [
        {update,dummy_app_server,{advanced,[from2to3]}}
    ]}], 
    %% Downgrade instructions
    [{"2",[
        {update,dummy_app_server,{advanced,[from3to2]}}
    ]}]
}.

This {update..} directive will result in the code_change function being called on the dummy_app_server. The purpose of code_change is to change the State from the old (v2) format, to the new (v3) format.

Although it’s not strictly necessary, I pass ‘from2to3’ as the ‘Extra’ field in the code_change call. This can be pattern matched on, and makes it clear in your code_change code exactly what version upgrade is expected.

Packaging and upgrading to v3

Move the generated release dir for v2:

$ mv rel/dummynode rel/dummynode_2

Compile and generate for v3, then create the upgrade package:

$ ./rebar compile
$ ./rebar generate
$ ./rebar generate-upgrade previous_release=dummynode_2

N.B. You need to provide the full, standalone generated release dir as the previous_release, you can’t use dummynode_first, even though that contains version 2 of your release.

As before, copy the upgrade package to the releases directory of the running release:

$ cp rel/dummynode_3.tar.gz rel/dummynode_first/releases

Now, at the Erlang console where you upgraded v1 to v2:

(dummynode@127.0.0.1)12> release_handler:unpack_release("dummynode_3").
{ok,"3"}
(dummynode@127.0.0.1)13> release_handler:install_release("3").
{ok,"2",[]}

Congratulations

Now you can deploy hot-code-upgrades the proper OTP way. Ideal for complex or large upgrades that change internal state or do require special upgrade hooks. Read the Appup Cookbook a few times, and test your upgrade packages in a staging environment before deploying. You can just tar up and copy the live environment to your staging box, to get an exact clone of the production system to test upgrades against.

Warning: a current issue with downgrades

To downgrade, you just install a previous release. However, there is currently a bug where release_handler chokes during downgrades to the first version, because of a discrepancy in the naming of .boot files in the release. release_handler has start.boot hardcoded, but rebar will generate appname.boot, with start.boot as a symlink. If you need to do downgrades, test this carefully before deploying; you may need to manually rename the boot file.

Cowboying out quick fixes

Appup files and generating releases is rather heavyweight. Here’s an overview of the process I’m using on IRCCloud at the moment

Complex upgrades are proper releases, with .appup

No way around it; bit of a pain to create and test, but glorious when you pull off a complex upgrade with zero downtime. Releases are tagged in git with the datetime and letter version, eg: v20110324a. If I do a second release that day, v20110324b.

‘Hotfixes’ preserve my sanity

If I make a quick fix that simply requires reloading a module with no risk of instability, I do the following:

Reset code to currently deployed tag, egv 20110324a
Write the fix
Commit as v200110324a-hotfix1
Build this version of the specific module that’s changed, and copy it into the production environment, eg to: /somewhere/dummyapp/lib/dummyapp-20110324a/ebin/
Reload the module, eg by using l(module_name). at the shell.

It’s of vital importance to have a repeatable process, so you know exactly which version of code (ie, the git tag) is currently running in production. If you can’t be sure, then it’s much harder to write successful upgrade code later on.

This process gives a reasonable balance between periodic ‘proper’ releases, during which any complex changes are made that rewire internal state, and quick fixes that just require a module reload.

Pitfalls to avoid

In the IRCCloud app, there’s one place that I don’t use a supervisor, but wish I had; the user process acts as a sort of supervisor for connection processes, because I needed exponential backoff / more control over restarting crashed children.

Don’t do that. release_handler isn’t aware of the child processes I spawn myself, so I can’t use the normal appup process to call code_change.

erlang.org RSS Feed: Mailing Lists Operating

Fri, 25 Mar 2011 18:52:58 +0000

The mailing lists at erlang.org are now back online after fixing a subtle Python "gotcha" configuration error.

Please report any posts that you feel slipped into the void, or double posts or whatnot!

Scattered Thoughts: telehash: bit_trees revisited

Thu, 24 Mar 2011 23:08:44 +0000

It has been suggested that the bit_trees presented in the last post are overly complicated. Indeed, in the cold light of the morning there is absolutely no need for that zipper. Without further ado, here is the much simpler version.

% implements the tree part of kademlias k-buckets
% a bit_tree maps ends (lists of bits) to buckets
% as far as the bit_tree is concerned the buckets are completely opaque
% the bit_tree also calculates various numbers needed for splitting decisions

-module(bit_tree).

-include("conf.hrl").

-export([empty/2, update/4, iter/2]).

% a bit_tree is either a leaf or a branch
-record(leaf, {
          size, % size of bucket
          bucket % some opaque bucket of stuff
         }).
-record(branch, {
          size, % size(childF) + size(childT)
          childF, % tree containing nodes whose next bit is false
          childT % tree containing nodes whose next bit is true
         }).

% --- api ---

empty(Size, Bucket) ->
    #leaf{size=Size, bucket=Bucket}.
                
update(Fun, Bits, Self, Tree) when is_function(Fun), is_list(Bits), is_list(Self) ->
    update(Fun, Bits, {self, Self}, 0, Tree).

update(Fun, Bits, Gap, Depth, #leaf{bucket=Bucket}) ->
    Gap_size =
        case Gap of
            {gap, G} -> G;
            {self, _} -> 0
        end,
    bucket_update_to_tree(Fun(Bits, Depth, Gap_size, Bucket));
update(Fun, Bits, Self, Depth, #branch{childF=ChildF, childT=ChildT}) ->
    [Next|Bits2] = Bits,
    Self2 =
        case Self of
            {gap, _} -> Self;
            {self, [Next|Rest]} -> {self, Rest};
            {self, [false|_]} -> {gap, tree_size(ChildF)};
            {self, [true|_]} -> {gap, tree_size(ChildT)}
        end,
    Depth2 = Depth+1,
    case Next of
        true ->
            ChildT2 = update(Fun, Bits2, Self2, Depth2, ChildT),
            Size = tree_size(ChildF) + tree_size(ChildT2),
            #branch{size=Size, childF=ChildF, childT=ChildT2};
        false ->
            ChildF2 = update(Fun, Bits2, Self2, Depth2, ChildF),
            Size = tree_size(ChildF2) + tree_size(ChildT),
            #branch{size=Size, childF=ChildF2, childT=ChildT}
    end.

% iterate through buckets in ascending order of xor distance to Bits
iter(Bits, Tree) ->
    iter(Bits, Tree, fun() -> done end).
                             
iter(_Bits, #leaf{bucket=Bucket}, Iter) ->
    fun () ->
            {Bucket, Iter}
    end;
iter([Bit|Bits], #branch{childF=ChildF, childT=ChildT}, Iter) ->
    case Bit of
        true ->
            iter(Bits, ChildT, iter(Bits, ChildF, Iter));
        false ->
            iter(Bits, ChildF, iter(Bits, ChildT, Iter))
    end.

% --- internal functions ---

tree_size(#leaf{size=Size}) ->
    Size;
tree_size(#branch{size=Size}) ->
    Size.

bucket_update_to_tree({ok, Size, Bucket}) ->
    #leaf{size=Size, bucket=Bucket};
bucket_update_to_tree({split, SplitF, SplitT}) ->
    ChildF = bucket_update_to_tree(SplitF),
    ChildT = bucket_update_to_tree(SplitT),
    #branch{size=tree_size(ChildF)+tree_size(ChildT), childF=ChildF, childT=ChildT}.

% --- end ---

And the corresponding test code.

% simple buckets used for testing bit_tree

-module(test_bucket).

-include("conf.hrl").

-export([bits/1, add/3, split/1, add_to_tree/2, make_tree/1, distance/2, list_from/2]).

-define(MAX_SIZE, 3).
-define(BITS, ?END_BITS).

bits(Int) ->
    util:to_bits(<<Int:?BITS>>).

add(Suffix, Int, Bucket) ->
    split([{Suffix, Int} | Bucket]).

split(Bucket) ->
    if
        length(Bucket) > ?MAX_SIZE ->
            BucketF = [{Suffix2, Int2} || {[false | Suffix2], Int2} <- Bucket],
            BucketT = [{Suffix2, Int2} || {[true | Suffix2], Int2} <- Bucket],
            {split, split(BucketF), split(BucketT)};
        true ->
            {ok, length(Bucket), Bucket}
    end.

add_to_tree(Int, Tree) ->
    bit_tree:update(
      fun (Suffix, _Depth, _Gap_size, Bucket) ->
              add(Suffix, Int, Bucket)
      end,
      bits(Int),
      bits(Int), % dont care about gap for now
      Tree).

make_tree(Ints) ->
    Tree = bit_tree:empty(0, []),
    lists:foldl(fun add_to_tree/2, Tree, Ints).

distance(IntA, IntB) ->
    util:distance({'end', <<IntA:?BITS>>}, {'end', <<IntB:?BITS>>}).

% output *should* be in ascending order
list_from(Int, Tree) ->
    List = util:iter_to_list(bit_tree:iter(bits(Int), Tree)),
    lists:map(
      fun (Bucket) ->
              lists:sort([{distance(Int, Elem), Elem} || {_,Elem} <- Bucket])
      end,
      List).

Scattered Thoughts: telehash: bit_trees

Thu, 24 Mar 2011 19:34:45 +0000

The next step in building a switch is managing a routing table. Actually, the next step is handling sessions via _ring/_line but I'm still mulling over the protocol so we'll skip to the routing table.

I'll add the usual 'I don't understand Kademlia and I don't test my code' disclaimer in here.

Routing in the Kademlia paper is described using what can best be called the 'mash everything together and be vague about the details' pattern. I want my switch to be a bit cleaner than that so I've split it into three modules. The first of these is the bit_tree.

The bit_tree is a suffix tree which maps ends (lists of bits) to buckets. The bit_tree neither knows nor cares what a bucket is and for now you don't either. The utility of this tree comes down to one important property: the floor of the log (base 2) of the XOR distance between two ends is the height of the smallest sub-tree which contains both of them. Got that? For example, if log(distance(EndA,EndB)) == 7.234... then the height of the smallest sub-tree containing both EndA and EndB is 7 nodes. This makes it easy to locate the nearest known nodes to a specified end, something we are supposed to do in response to a .see command.

So here is a bog-standard binary suffix tree:

% a bit_tree is either a leaf or a branch
-record(leaf, {
          size, % size of bucket
          bucket % some opaque bucket of stuff
         }).
-record(branch, {
          size, % size(childF) + size(childT)
          childF, % tree containing nodes whose next bit is false
          childT % % tree containing nodes whose next bit is true
         }).

When adding nodes to a bucket we need to keep track of certain numbers which will be used by the router to decide when to split buckets. Some of these are quite complicated so to make this easier we will work with a zipper-like structure instead of using leaf and branch directly. If you know what a zipper is the code in this post will make sense. If you don't know what a zipper is, go find out. When you come back the code in this post will make sense.

% zipper-esque structure marking a position in a bit_tree
-record(finger, {
          sizer, % a size function for buckets
          tree, % current sub-tree
          self, % the path *to* self (the nodes own end). either {down, Down_bits} or {up, Up_bits, Down_bits, Gap}
                % where Gap is the size of the largest tree containing self but not touching this finger
          depth, % the number of bits away from the root tree
          zipper % a list of {Bit, Tree} pairs marking branches NOT taken
         }).

The finger keeps track of where the nodes own end is located in the tree in order to calculate something I have termed the gap - the size of the largest sub-tree containing the nodes own end but not touching the finger.

The empty bit_tree is easy to define:

empty(Self, Bucket, Sizer) ->
    #finger{
       sizer = Sizer,
       tree = #leaf{size=Sizer(Bucket), bucket=Bucket},
       self = {down, Self},
       depth = 0,
       zipper = []
      }.

Moving around within the tree is a little more complicated but if you already went away and read about zippers it should feel familiar. Most of the work is in keeping track of the gap.

extend(Bits, #finger{tree=#leaf{}}=Finger) -> % must always end on a leaf
    {Bits, Finger};
extend([Next | Bits],
       #finger{
         tree = #branch{childF=ChildF, childT=ChildT},
         self = Self,
         depth = Depth,
         zipper = Zipper
        }=Finger) ->
    {Branch_taken, Branch_missed} =
        case Next of
            false -> {ChildF, ChildT};
            true -> {ChildT, ChildF}
        end,
    Self2 =
        case Self of
            {up, Up, Down, Gap} ->
                % already stepped out of gap
                {up, [not(Next)|Up], Down, Gap};
            {down, [Bit|Down]} when Bit == Next ->
                % still in the gap
                {down, Down};
            {down, [Bit|Down]} when Bit /= Next ->
                % leaving gap, check its size
                {up, [not(Next)], [Bit|Down], tree_size(Branch_missed)}
        end,
    Depth2 = Depth+1,
    Zipper2 = [{not(Next), Branch_missed} | Zipper],
    Finger2 = Finger#finger{
      tree = Branch_taken,
      self = Self2,
      depth = Depth2,
      zipper = Zipper2
     },
    extend(Bits, Finger2).

retract(0, Finger) ->
    Finger;
retract(N,
        #finger{
          tree = Tree,
          self = Self,
          depth = Depth,
          zipper = [{Last,Branch}|Zipper]
         }=Finger) when N>0 ->
    Size = tree_size(Tree) + tree_size(Branch),
    Tree2 =
        case Last of
            false -> #branch{size=Size, childF=Branch, childT=Tree};
            true -> #branch{size=Size, childF=Tree, childT=Branch}
        end,
    Self2 =
        case Self of
            {down, Down} ->
                % already in gap
                {down, [Last|Down]};
            {up, [], Down, _Gap} ->
                % just entered gap
                {down, [Last|Down]};
            {up, [Bit|Up], Down, Gap} ->
                % still outside gap
                true = (Bit==Last), % assert
                {up, Up, Down, Gap}
        end,
    Depth2 = Depth-1,
    Finger2 =
        Finger#finger{
          tree=Tree2,
          self=Self2,
          depth=Depth2,
          zipper=Zipper
         },
    retract(N-1, Finger2).

The extend and retract functions are only used internally. We export a much simpler function, move_to, which moves the finger to point at the bucket corresponding to the specified end.

move_to(Bits, #finger{depth=Depth}=Finger) when length(Bits) == ?END_BITS ->
    % !!! naive version
    extend(Bits, retract(Depth, Finger)).

We could make this more efficient by only retracting until the finger meets Bits partway up. For now I don't expect performance of the bit_tree to be an issue.

Now that we can find buckets we can modify them. Deciding when to split buckets is not the concern of the bit_tree so we delegate it to the caller.

update(Fun,
       #finger{
         sizer=Sizer,
         tree=#leaf{bucket=Bucket}
        }=Finger) ->
    Tree = bucket_update_to_tree(Sizer, Fun(Bucket)),
    Finger#finger{tree=Tree}.

bucket_update_to_tree(Sizer, {ok, Bucket}) ->
    #leaf{size=Sizer(Bucket), bucket=Bucket};
bucket_update_to_tree(Sizer, {split, SplitF, SplitT}) ->
    ChildF = bucket_update_to_tree(Sizer, SplitF),
    ChildT = bucket_update_to_tree(Sizer, SplitT),
    #branch{size=tree_size(ChildF)+tree_size(ChildT), childF=ChildF, childT=ChildT}.

In order to handle .see commands the iter function is used to return buckets in order of distance from the specified end. Here we are making use of the aforementioned nice properties of the bit_tree in order to efficiently return the buckets in order.

% iterate through buckets in ascending order of xor distance to (current position ++ Suffix)
iter(Suffix, #finger{tree=Tree, zipper=Zipper}) ->
    iter_buckets(Tree, Suffix, iter_zipper(Zipper, Suffix)).

% iterate through buckets in ascending order of xor distance to (current position ++ Suffix)
iter_zipper([], _Suffix) ->
    fun () ->
            done
    end;
iter_zipper([{Bit, Tree} | Zipper], Suffix) ->
    iter_buckets(Tree, Suffix, iter_zipper(Zipper, [not(Bit)|Suffix])).

% iterate through buckets in ascending order of xor distance to Bits, then hand over to Iter
iter_buckets(#leaf{bucket=Bucket}, _Bits, Iter) ->
    fun () ->
            {Bucket, Iter}
    end;
iter_buckets(#branch{childF=ChildF, childT=ChildT}, [Bit|Bits], Iter) ->
    case Bit of
        true ->
            iter_buckets(ChildT, Bits, iter_buckets(ChildF, Bits, Iter));
        false ->
            iter_buckets(ChildF, Bits, iter_buckets(ChildT, Bits, Iter))
    end.

It will typically be called like this:

{Suffix, Tree2} = bit_tree:move_to(util:to_bits(End), Tree),
bit_tree:iter(Suffix, Tree2)

Splitting the routing table into separate structures like this makes for easier testing. The bit_tree can be tested independently using really simple buckets where the elements are just integers and the buckets split when they reach more than three elements.

% simple buckets used for testing bit_tree

-module(test_bucket).

-include("conf.hrl").

-export([bits/1, add/3, split/1, move_to/2, add_to_tree/2, make_tree/2, distance/2, move_list_from/2, list_from/3]).

-define(MAX_SIZE, 3).
-define(BITS, ?END_BITS).

bits(Int) ->
    util:to_bits(<<Int:?BITS>>).

add(Suffix, Int, Bucket) ->
    split([{Suffix, Int} | Bucket]).

split(Bucket) ->
    if
        length(Bucket) > ?MAX_SIZE ->
            BucketF = [{Suffix2, Int2} || {[false | Suffix2], Int2} <- Bucket],
            BucketT = [{Suffix2, Int2} || {[true | Suffix2], Int2} <- Bucket],
            {split, split(BucketF), split(BucketT)};
        true ->
            {ok, Bucket}
    end.

move_to(Int, Tree) ->
    bit_tree:move_to(bits(Int), Tree).

add_to_tree(Int, Tree) ->
    {Suffix, Tree2} = move_to(Int, Tree),
    bit_tree:update(fun (Bucket) -> add(Suffix, Int, Bucket) end, Tree2).

make_tree(Int, Ints) ->
    Tree = bit_tree:empty(bits(Int), [], fun (Bucket) -> length(Bucket) end),
    lists:foldl(fun add_to_tree/2, Tree, Ints).

distance(IntA, IntB) ->
    util:distance({'end', <<IntA:?BITS>>}, {'end', <<IntB:?BITS>>}).

% output *should* be in ascending order
move_list_from(Int, Tree) ->
    {Suffix, Tree2} = bit_tree:move_to(bits(Int), Tree),
    list_from(Int, Suffix, Tree2).

list_from(Int, Suffix, Tree) ->
    List = util:iter_to_list(bit_tree:iter(Suffix, Tree)),
    lists:map(
      fun (Bucket) ->
              lists:sort([{distance(Int, Elem), Elem} || {_,Elem} <- Bucket])
      end,
      List).

We can play around with the test buckets a bit:

25> Tree = test_bucket:make_tree(47, lists:seq(1,1000)).
{finger,#Fun<test_bucket.1.121651971>,
        {leaf,1,[{[false,false,false],1000}]},
        {up,[false,true,false,false,false,false,false,true,true,
             true,true,true,true,true,true,true,true,true,true,true,true,
             true,true|...],
            [true,true,true,true,true,true,true,true,true,true,true,
             true,true,true,true,true,true,true,true,true,true,true|...],
            0},
        157,
        [{false,{branch,8,
                        {branch,4,
                                {leaf,2,[{[true],993},{[false],992}]},
                                {leaf,2,[{[true],995},{[false],994}]}},
                        {branch,4,
                                {leaf,2,[{[true],997},{[false],996}]},
                                {leaf,2,[{[true],999},{[false],998}]}}}},
         {true,{leaf,0,[]}},
         {false,{branch,32,
                        {branch,16,
                                {branch,8,
                                        {branch,4,
                                                {leaf,2,[{[true],961},{[...],...}]},
                                                {leaf,2,[{[...],...},{...}]}},
                                        {branch,4,
                                                {leaf,2,[{[...],...},{...}]},
                                                {leaf,2,[{...}|...]}}},
                                {branch,8,
                                        {branch,4,{leaf,2,[{[...],...},{...}]},{leaf,2,[{...}|...]}},
                                        {branch,4,{leaf,2,[{...}|...]},{leaf,2,[...]}}}},
                        {branch,16,
                                {branch,8,
                                        {branch,4,{leaf,2,[{[...],...},{...}]},{leaf,2,[{...}|...]}},
                                        {branch,4,{leaf,2,[{...}|...]},{leaf,2,[...]}}},
                                {branch,8,
                                        {branch,4,{leaf,2,[{...}|...]},{leaf,2,[...]}},
                                        {branch,4,{leaf,2,[...]},{leaf,2,...}}}}}},
         {false,{branch,64,
                        {branch,32,
                                {branch,16,
                                        {branch,8,
                                                {branch,4,{leaf,2,...},{leaf,...}},
                                                {branch,4,{leaf,...},{...}}},
                                        {branch,8,{branch,4,{leaf,...},{...}},{branch,4,{...},...}}},
                                {branch,16,
                                        {branch,8,{branch,4,{leaf,...},{...}},{branch,4,{...},...}},
                                        {branch,8,{branch,4,{...},...},{branch,4,...}}}},
                        {branch,32,
                                {branch,16,
                                        {branch,8,{branch,4,{leaf,...},{...}},{branch,4,{...},...}},
                                        {branch,8,{branch,4,{...},...},{branch,4,...}}},
                                {branch,16,
                                        {branch,8,{branch,4,{...},...},{branch,4,...}},
                                        {branch,8,{branch,4,...},{branch,...}}}}}},
         {false,{branch,128,
                        {branch,64,
                                {branch,32,
                                        {branch,16,
                                                {branch,8,{branch,...},{...}},
                                                {branch,8,{...},...}},
                                        {branch,16,{branch,8,{...},...},{branch,8,...}}},
                                {branch,32,
                                        {branch,16,{branch,8,{...},...},{branch,8,...}},
                                        {branch,16,{branch,8,...},{branch,...}}}},
                        {branch,64,
                                {branch,32,
                                        {branch,16,{branch,8,{...},...},{branch,8,...}},
                                        {branch,16,{branch,8,...},{branch,...}}},
                                {branch,32,
                                        {branch,16,{branch,8,...},{branch,...}},
                                        {branch,16,{branch,...},{...}}}}}},
         {false,{branch,256,
                        {branch,128,
                                {branch,64,
                                        {branch,32,{branch,16,{...},...},{branch,16,...}},
                                        {branch,32,{branch,16,...},{branch,...}}},
                                {branch,64,
                                        {branch,32,{branch,16,...},{branch,...}},
                                        {branch,32,{branch,...},{...}}}},
                        {branch,128,
                                {branch,64,
                                        {branch,32,{branch,16,...},{branch,...}},
                                        {branch,32,{branch,...},{...}}},
                                {branch,64,
                                        {branch,32,{branch,...},{...}},
                                        {branch,32,{...},...}}}}},
         {false,{branch,511,
                        {branch,255,
                                {branch,127,
                                        {branch,63,{branch,31,...},{branch,...}},
                                        {branch,64,{branch,...},{...}}},
                                {branch,128,
                                        {branch,64,{branch,...},{...}},
                                        {branch,64,{...},...}}},
                        {branch,256,
                                {branch,128,
                                        {branch,64,{branch,...},{...}},
                                        {branch,64,{...},...}},
                                {branch,128,{branch,64,{...},...},{branch,64,...}}}}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,[]}},
         {true,{leaf,0,...}},
         {true,{leaf,...}},
         {true,{...}},
         {true,...},
         {...}|...]}
26> List = test_bucket:move_list_from(657, Tree).
[[{0,657},{1,656}],
 [{2,659},{3,658}],
 [{4,661},{5,660}],
 [{6,663},{7,662}],
 [{8,665},{9,664}],
 [{10,667},{11,666}],
 [{12,669},{13,668}],
 [{14,671},{15,670}],
 [{16,641},{17,640}],
 [{18,643},{19,642}],
 [{20,645},{21,644}],
 [{22,647},{23,646}],
 [{24,649},{25,648}],
 [{26,651},{27,650}],
 [{28,653},{29,652}],
 [{30,655},{31,654}],
 [{32,689},{33,688}],
 [{34,691},{35,690}],
 [{36,693},{37,692}],
 [{38,695},{39,694}],
 [{40,697},{41,696}],
 [{42,699},{43,698}],
 [{44,701},{45,700}],
 [{46,703},{47,702}],
 [{48,673},{49,672}],
 [{50,675},{51,...}],
 [{52,...},{...}],
 [{...}|...],
 [...]|...]
27> lists:flatten(List) == lists:sort(lists:flatten(List)).
true

As usual the full code is in the repo.

erlang.org RSS Feed: New Site Look

Thu, 24 Mar 2011 18:11:55 +0000

Our demo site has now been launched as the regular erlang.org. We hope you like it. Report any problems.

Erlang Training and Consulting
- News: 24 March 2011: Erlang Solutions partners with Travelping to provide next generation AAA Solutions to telecom operators

Thu, 24 Mar 2011 17:17:22 +0000

Erlang Solutions (ESL), the driver behind the worldwide uptake of the Erlang development environment, today announced its partnership with Travelping, a Germany based software vendor offering solutions for network operators to build, manage and charge their broadband services securely and efficiently. Together, the two companies will offer operators highly scalable, robust and distributed, control plane solutions to build AAA and session control systems for their next generation networks (NGN).

For more information please see the press release.

Trapexit's Erlang Blog Filter: Don’t Lose Your ets Tables

Wed, 23 Mar 2011 15:01:51 +0000

In my recent QCon talk I talked about accidentally crashing an Erlang process on a customer’s subscription streaming video website running live in production. The code involved had not been used in production before, and the customer had decided somewhat unexpectedly to turn on a new feature that required it. The developer who wrote it had not tested it and had long since left the company.

The purpose of the code was to monitor bandwidth and session usage for each video subscriber to make sure they weren’t streaming more than they’d paid for. Concerned about the viability of the code, a colleague and I logged into the customer site (with their permission, of course), chose a subscriber at random, and, in an Erlang shell, I interactively invoked a function in the code in question to check that subscriber’s current bandwidth and session count. After a second check, we saw the numbers dropping, potentially indicating the subscriber was logging out, and we wanted to make sure all went well when the subscriber completely stopped streaming. After waiting a bit, I interactively called the function again, and — BAM! — the process holding session state for all paying customers crashed.

The original developer had used an Erlang ets table, an in-memory data store, to hold the subscriber data, and wrote something like this for lookups:

[SubscriberData] = ets:lookup(Table, Subscriber),

My interactive call from the shell looked up a nonexistent subscriber, so the result was the empty list [] rather than [SubscriberData], which caused a pattern mismatch and a badmatch exception. Uncaught, the exception crashed the process. Since the process owned the ets table, when it went down it took the ets table and all subscriber session data with it. It wasn’t so bad, since all it meant was that for a few hours a few subscribers potentially got a bit more video than they’d paid for, but still, it’s not at all the kind of design Erlang’s “Let It Crash” philosophy actually encourages. Crashing a process when something unexpected occurs is perfectly fine, since coding defensively introduces problems of its own, but you can still avoid losing your ets tables like this relatively easily.

Name an Heir

When you create an ets table you can also name a process to inherit the table should the creating process die:

TableId = ets:new(my_table, [{heir, SomeOtherProcess, HeirData}]),

If the creating process dies, the process SomeOtherProcess will receive a message of the form

{'ETS-TRANSFER', TableId, OldOwner, HeirData}

where TableId is the table identifier returned from ets:new, OldOwner is the pid of the process that owned the table, and HeirData is the data provided with the heir option passed to ets:new. Once it receives this message, SomeOtherProcess owns the table.

Give It Away

Alternatively, you can create an ets table and then give it to some other process to keep it:

TableId = ets:new(my_table, []),
ets:give_away(TableId, SomeOtherProcess, GiftData),

If the creating process dies, the process SomeOtherProcess will receive a message of the form

{'ETS-TRANSFER', TableId, OldOwner, GiftData}

where TableId is the table identifier returned from ets:new, OldOwner is the pid of the process that owned the table, and GiftData is the data provided in the ets:give_away call. Once it receives this message, SomeOtherProcess owns the table.

Table Manager

Instead of naming an heir or giving a table away, you can just have your Erlang supervisor process create a child process whose sole task is to own the table. This process creates the table as a named public table, thus allowing other processes to know its name and read/write it directly, with ets built-in concurrency protection dealing with any concurrency issues. Since the owner process does nothing more than create the table and then wait to be told to shut down, the likelihood of it crashing and taking the table with it is practically nil. The drawback here, though, is that the process actually using the table may have to coordinate with the owner process to ensure the table is available, and worse, it ends up using what is essentially a global variable — the table name — which can make code harder to read and maintain.

A Combination Approach

A nice way of managing ets tables, though, is to use a combination of the three previous techniques:

The Erlang supervisor creates a table manager process. Since all this process does is manage the table, the likelihood of it crashing is very low.
The table manager links itself to the table user process and traps exits, allowing it to receive an EXIT message if the table user process dies unexpectedly.
The table manager creates a table, names itself (self()) as the heir, and then gives it away to the table user process.
If the table user process dies, the table manager is informed of the process death and also inherits the table back.

Once it inherits the table, the table manager can then for example wait until the supervisor recreates the table user process, and then repeat the steps above to give the table to the new table user process. Other variations on this approach, like maybe a small pool of child process clones that cooperate to transfer the table between them in case of error, are of course also possible. Even though there are still process coordination issues here (but nothing difficult), I like this approach because it avoids global named tables and takes advantage of Erlang's supervision hierarchy.

The title of my QCon talk was "Let It Crash...Except When You Shouldn't." This scenario is an example of "when you shouldn't" — losing ets data due to a process crash is easily avoided.

Damien Katz: Couchbase SF Training Was Awesome

Mon, 21 Mar 2011 22:49:00 +0000

I had a blast teaching the first Couchbase CouchDB Training with training pro Alan McKean last week. 2 intensive days of hands on teaching and talking about Apache CouchDB to enthusiastic and excited people. It was actually a learning experience for me too, there's a lot in CouchDB I haven't had a chance to use yet :)

It's not too late to sign up for the remaining 3 cities on the Couchbase Training World Tour: Austin, London and Berlin.

Scattered Thoughts: telehash: dialing

Mon, 21 Mar 2011 15:14:39 +0000

The next step in building a telehash switch is being able to dial.

First a disclaimer: this post reflects my current understanding of TeleHash and Kademlia and is highly likely to be wrong. This code has only received minimal testing. Properly testing a p2p network is not something I'm entirely sure how to do yet. Expect to see more on that in later posts.

Each TeleHash node and each key in the DHT is identified by a 160 bit sha1 hash (aka end). In the original Kademlia paper the node ids are selected at random but in TeleHash they are the hashed address (IP:port) of the node. This means that malicious nodes don't get to choose where they are inserted in the DHT.

Kademlia routing is based on the XOR distance between ends. This forms a metric space over the set of ends.

distance(A, B) ->
    {'end', EndA} = to_end(A),
    {'end', EndB} = to_end(B),
    Bytes = lists:zip(binary_to_list(EndA), binary_to_list(EndB)),
    Xor = list_to_binary([ByteA bxor ByteB || {ByteA, ByteB} <- Bytes]),
    <<Dist:?END_BITS>> = Xor,
    Dist.

The Kademlia paper defines two constants, K and A. K controls the amount of redundant storage in the DHT and A controls the number of parallel requests issued by each node. To insert a key into the DHT a node must be able to locate the K nodes whose IDs are closest to the key. This process is called dialing.

Dialing works roughly as follows. Each node keeps track of all the other nodes it has seen. Upon receiving a +end signal a node will reply with a .see command containing the K nodes it is aware of which are closest to the specified end. To dial an end we send a +end signal to each of the K closest nodes we are aware of. Then to each node contained in the .see replies we send +end signals, and so on until we run out of nodes to contact.

Now this is nice and simple and will work but it generates a huge amount of load on the network. To reduce this Kademlia introduces two additional rules. First, we only send up to A signals at a time and don't send any new signals until previous signals have either generated a reply or timed out. Second, we finish early if at any point we have received replies from K nodes which are closer to the end than all the nodes we are waiting to contact. The Kademlia paper proves that under reasonable assumptions about the knowledge of each node this still has a very high chance to return the correct results.

The dialer process is an event handler which has two important data structures. The first stores the dialer configuration:

-record(conf, {
          target, % the end to dial
          timeout, % the timeout for the entire dialing process
          ref, caller % reply details
         }).

The second record stores the state of the dialing process. The principle around which the dialer is designed is that the state record is a reflection of the outside world and the sole job of the dialer is to keep this record up to date while maintaining the invariants in the comments. This is often the way that I write code and I feel that it needs it's own post once I can articulate it properly. It's certainly heavily informed both by the designs in Okasaki's Purely Functional Data Structures and by Conal Elliott's ideas about denotational semantics and type class morphisms.

-record(state, {
          fresh, % nodes which have not yet been contacted
          pinged, % nodes which have been contacted and have not replied
          waiting, % nodes in pinged which were contacted less than ?DIAL_TIMEOUT ago
          ponged, % nodes which have been contacted and have replied
          seen % all nodes which have been seen
         }). % invariant: pq:length(waiting) = ?A or pq:empty(fresh)

The dialer module exports the dial function which creates the records and starts the event handler.

dial(To, From, Timeout) ->
    log:info([?MODULE, dialing, To, From, Timeout]),
    Ref = erlang:make_ref(),
    Target = util:to_end(To),
    Conf = #conf{
      target = Target,
      timeout = Timeout,
      ref = Ref,
      caller = self()
     },
    Nodes = [{util:distance(Address, Target), Address}
             || Address <- From],
    State = #state{
      fresh=pq:from_list(Nodes),
      pinged=sets:new(),
      waiting=pq:empty(),
      ponged=pq:empty(),
      seen=sets:new()
     },
    ok = switch:add_handler(?MODULE, {Conf, State}),
    Ref.

The aim is to handle events and maintain the state invariants until we are finished. How do we define finished?

% is the dialing finished yet?
finished(#state{fresh=Fresh, waiting=Waiting, ponged=Ponged}) ->
    (pq:is_empty(Fresh) and pq:is_empty(Waiting)) % no way to continue
    or
    (case pq:length(Ponged) >= ?K of
         false ->
             false; % dont yet have K nodes
         true ->
             % finish if the K closest nodes we know are closer than all the nodes we haven't checked yet
             {Dist_fresh, _} = pq:peek(Fresh),
             {Dist_waiting, _} = pq:peek(Waiting),
             {Nodes, _} = pq:pop(Ponged, ?K),
             {Dist_ponged, _} = lists:last(Nodes),
             (Dist_ponged < Dist_fresh) and (Dist_ponged < Dist_waiting)
     end).

One of the invariants we aim to maintain is that either the fresh queue is empty or the length of the waiting queue is A. This ensures that we send out +end signals whenever possible. This invariant is maintained by calling the ping_nodes function after every event.

% contact nodes from fresh until the waiting list is full
ping_nodes(#conf{target=Target}, #state{fresh=Fresh, waiting=Waiting, pinged=Pinged}=State) ->
    Num = ?A - pq:length(Waiting),
    {Nodes, Fresh2} = pq:pop(Fresh, Num),
    Telex = {struct, [{'+end', util:end_to_hex(Target)}]},
    lists:foreach(
      fun ({Dist, Address}=Node) ->
              log:info([?MODULE, ping, Dist, Address]),
              switch:send(Address, Telex),
              erlang:send_after(?DIAL_TIMEOUT, self(), {timeout, Node})
      end,
      Nodes),
    Waiting2 = pq:push(Nodes, Waiting),
    Pinged2 = sets:union(Pinged, sets:from_list(Nodes)),
    State#state{fresh=Fresh2, waiting=Waiting2, pinged=Pinged2}.

We handle replies by moving the replying node from the waiting queue to the ponged queue and inserting the .see nodes into the fresh list. We cannot allow duplicate nodes so the seen set is kept up to date. The pinged set will be used later to ensure that we only accept replies from nodes we have already contacted and only accept one reply per node.

% handle a reply from a node
ponged(Node, See, #state{fresh=Fresh, waiting=Waiting, pinged=Pinged, ponged=Ponged, seen=Seen}=State) ->
    Waiting2 = pq:delete(Node, Waiting),
    Pinged2 = sets:del_element(Node, Pinged),
    Ponged2 = pq:push_one(Node, Ponged),
    New_nodes = lists:filter(fun (See_node) -> not(sets:is_element(See_node, Seen)) end, See),
    Fresh2 = pq:push(New_nodes, Fresh),
    Seen2 = sets:union(Seen, sets:from_list(See)),
    State#state{fresh=Fresh2, waiting=Waiting2, pinged=Pinged2, ponged=Ponged2, seen=Seen2}.

Once we are finished we need to return the results to the caller.

% return results to the caller
return(#conf{ref=Ref, caller=Caller}, #state{ponged=Ponged}) ->
    {Nodes, _} = pq:pop(Ponged, ?K),
    log:info([?MODULE, returning, Nodes]),
    Result = [Address || {_Dist, Address} <- Nodes],
    Caller ! {dialed, Ref, Result}.

Finally, after each event we call continue to decide whether to finish and return results or to carry on sending signals.

% either continue to dial or return results
% meant for use at the end of a gen_event callback
continue(Conf, State) ->
    case finished(State) of
        true ->
            return(Conf, State),
            remove_handler;
        false ->
            State2 = ping_nodes(Conf, State),
            {ok, {Conf, State2}}
    end.

The functions above are glued together by a gen_event handler. The handler is attached to the switch gen_event manager and receives an event for each telex arriving at the switch.

-behaviour(gen_event).
-export([init/1, handle_event/2, handle_call/2, handle_info/2, terminate/2, code_change/3]).

The init function is called when the handler is started. It sends out the first +end signals and sets a timer that tells the handler when to give up dialling.

init({#conf{timeout=Timeout}=Conf, State}) ->
    erlang:send_after(Timeout, self(), giveup),
    State2 = ping_nodes(Conf, State),
    {ok, {Conf, State2}}.

The giveup timeout is simple to deal with.

handle_info(giveup, {Conf, State}) ->
    log:info([?MODULE, giveup, Conf, State]),
    remove_handler;

As are the timeouts from individual signals.

handle_info({timeout, Node}, {Conf, #state{waiting=Waiting}=State}) ->
    log:info([?MODULE, timeout, Node]),
    State2 = State#state{waiting=pq:delete(Node, Waiting)},
    continue(Conf, State2);

The last callback is the messiest. This essentially just calls ponged and continue, but first has to sanity check the incoming message.

handle_event({recv, Address, Telex}, {#conf{target=Target}=Conf, #state{pinged=Pinged}=State}) ->
    case telex:get(Telex, '.see') of
        {error, not_found} ->
            {ok, {Conf, State}};
        {ok, Address_binaries} ->
            Dist = util:distance(Address, Target),
            Node = {Dist, Address},
            case sets:is_element(Node, Pinged) of % !!! command ids would make a better check
                false ->
                    {ok, {Conf, State}};
                true ->
                    try [{util:distance(Target, Bin), util:binary_to_address(Bin)} || Bin <- Address_binaries] of
                        Nodes ->
                            log:info([?MODULE, pong, Node, Nodes]),
                            State2 = ponged(Node, Nodes, State),
                            continue(Conf, State2)
                    catch
                        _:Error ->
                            log:info([?MODULE, bad_see, Address, Telex, Error, erlang:get_stacktrace()]),
                            {ok, {Conf, State}}
                    end
            end
    end;

That's pretty much it - we now (probably) have a working dialer. I spent a fair few hours teasing this apart but hopefully the end result is fairly simple to understand. The full code is in the repo as always.

4> switch:start_link().
{ok,<0.79.0>,<0.80.0>}
5> Root = {address, "208.68.163.247", 42424}.
{address,"208.68.163.247",42424}
6> dialer:dial_sync(Root, [Root], 10000).

=INFO REPORT==== 21-Mar-2011::14:48:06 ===
    pid: <0.35.0>
    dialer
    dialing
    {address,"208.68.163.247",42424}
    [{address,"208.68.163.247",42424}]
    10000

=INFO REPORT==== 21-Mar-2011::14:48:06 ===
    pid: <0.79.0>
    dialer
    ping
    0
    {address,"208.68.163.247",42424}

=INFO REPORT==== 21-Mar-2011::14:48:06 ===
    pid: <0.80.0>
    switch_event
    send
    {address,"208.68.163.247",42424}
    struct: [{<<"_to">>,<<"208.68.163.247:42424">>},
             {'+end',<<"38666817e1b38470644e004b9356c1622368fa57">>}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.80.0>
    switch_event
    recv
    {address,"208.68.163.247",42424}
    struct: [{<<"_ring">>,18115},
             {<<".see">>,
              [<<"204.232.205.180:42424">>,<<"208.68.163.247:42424">>]},
             {<<"_br">>,240},
             {<<"_to">>,<<"203.218.138.245:42424">>}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.79.0>
    dialer
    pong
    0: {address,"208.68.163.247",42424}
    [{535375931004298447338698443374311161987273280591,
      {address,"204.232.205.180",42424}},
     {0,{address,"208.68.163.247",42424}}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.79.0>
    dialer
    ping
    0
    {address,"208.68.163.247",42424}

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.79.0>
    dialer
    ping
    535375931004298447338698443374311161987273280591
    {address,"204.232.205.180",42424}

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.80.0>
    switch_event
    send
    {address,"208.68.163.247",42424}
    struct: [{<<"_to">>,<<"208.68.163.247:42424">>},
             {'+end',<<"38666817e1b38470644e004b9356c1622368fa57">>}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.80.0>
    switch_event
    send
    {address,"204.232.205.180",42424}
    struct: [{<<"_to">>,<<"204.232.205.180:42424">>},
             {'+end',<<"38666817e1b38470644e004b9356c1622368fa57">>}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.80.0>
    switch_event
    recv
    {address,"204.232.205.180",42424}
    struct: [{<<"_ring">>,16506},
             {<<".see">>,
              [<<"204.232.205.180:42424">>,<<"208.68.163.247:42424">>]},
             {<<"_br">>,162},
             {<<"_to">>,<<"203.218.138.245:42424">>}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.79.0>
    dialer
    pong
    535375931004298447338698443374311161987273280591: {address,
                                                       "204.232.205.180",
                                                       42424}
    [{535375931004298447338698443374311161987273280591,
      {address,"204.232.205.180",42424}},
     {0,{address,"208.68.163.247",42424}}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.80.0>
    switch_event
    recv
    {address,"208.68.163.247",42424}
    struct: [{<<"_ring">>,18115},
             {<<".see">>,
              [<<"204.232.205.180:42424">>,<<"208.68.163.247:42424">>]},
             {<<"_br">>,320},
             {<<"_to">>,<<"203.218.138.245:42424">>}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.79.0>
    dialer
    pong
    0: {address,"208.68.163.247",42424}
    [{535375931004298447338698443374311161987273280591,
      {address,"204.232.205.180",42424}},
     {0,{address,"208.68.163.247",42424}}]

=INFO REPORT==== 21-Mar-2011::14:48:07 ===
    pid: <0.79.0>
    dialer
    returning
    [{0,{address,"208.68.163.247",42424}},
     {535375931004298447338698443374311161987273280591,
      {address,"204.232.205.180",42424}}]
{ok,[{address,"208.68.163.247",42424},
     {address,"204.232.205.180",42424}]}

One last note: after I finished writing this I started thinking about what would happen if I run more than one dialer in parallel. Unlike Kademlia, TeleHash does not currently use command IDs so the dialer cannot tell if the response came in reply to its own command or in reply to the command of another dialer on the same node. It's the kind of bug that would be very rare in actual use but might be carefully exploited by a malicious node. Finding these kinds of bugs is going to be really hard.

Learn You Some Erlang: Who Supervises The Supervisors?

Fri, 18 Mar 2011 12:30:00 +0000

Right in time before the Bay Area Erlang Conference, the supervisors make their place in Learn You Some Erlang. We see how to set up an OTP supervisor, the restart strategies available, how to write children specifications and have a little demonstration where a band manager takes pleasure at firing band members.

Process-one Blogs: Sea Beyond 2011 Talk 5: Marek Foss on designing mobile collaboration software

Thu, 17 Mar 2011 15:46:51 +0000

Sea Beyond event was this year heavily focused on mobile real time applications.

Marek Foss, Chief Web Officer at ProcessOne, gave a talk at Sea Beyond 2011 event, sharing some useful design oriented patterns for collaborative real time mobile applications.

Here is the video of his presentation:

You can see the slides here:

Do not miss the event summary and the other videos from Sea Beyond event.

Scattered Thoughts: telehash: basics

Thu, 17 Mar 2011 12:47:18 +0000

TeleHash is a p2p network based on the Kademlia DHT that provides addressing and NAT traversal. These are problems that every p2p app has to deal with, including my poppi. Unfortunately there is no erlang implementation yet so I have to roll my own. The code so far lives here In this first post I'll just cover the absolute basics - sending, receiving, encoding and decoding messages.

TeleHash messages (telexes) are utf8-encoded json packets sent over udp. Luckily, mochijson2 uses utf8 by default so encoding/decoding is trivial.

encode(Telex) ->
    mochijson2:encode(Telex).

decode(Json) ->
    mochijson2:decode(Json).

The telex module also defines some convenience methods for working with json - get/2, set/3, update/4 - which are used like this:

2> T = telex:decode("{\"foo\":[\"bar\", {\"baz\":0}]}").
{struct,[{<<"foo">>,[<<"bar">>,{struct,[{<<"baz">>,0}]}]}]}
3> telex:get(T, foo).
[<<"bar">>,{struct,[{<<"baz">>,0}]}]
4> telex:get(T, {foo,2}).
{struct,[{<<"baz">>,0}]}
5> telex:get(T, {foo,2,baz}).
0
6> telex:set(T, {foo,2,baz}, 1).
{struct,[{<<"foo">>,[<<"bar">>,{struct,[{<<"baz">>,1}]}]}]}
7> telex:set(T, bigger, true).
{struct,[{<<"bigger">>,true},
         {<<"foo">>,[<<"bar">>,{struct,[{<<"baz">>,0}]}]}]}
8> telex:update(T, {foo,2,baz}, fun (X) -> X + 10 end, -1).
{struct,[{<<"foo">>,[<<"bar">>,{struct,[{<<"baz">>,10}]}]}]}

The next step is to be able to send and receive messages. The switch module runs a gen_server which manages the udp socket and a gen_event which allows other processes to subscribe to incoming messages.

-module(switch).

-include("conf.hrl").

-export([start_link/0, add_handler/2, add_sup_handler/2, send/2]).

-behaviour(gen_server).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).

-record(state, {socket}).
-define(EVENT, switch_event).
-define(SERVER, switch_server).

% --- api ---

start_link() ->
    {ok, Gen_event} = gen_event:start_link({local, ?EVENT}),
    {ok, Gen_server} = gen_server:start_link({local, ?SERVER}, ?MODULE, [], []),
    {ok, Gen_event, Gen_server}.

add_handler(Module, Args) ->
    gen_event:add_handler(?EVENT, Module, Args).

add_sup_handler(Module, Args) ->
    gen_event:add_sup_handler(?EVENT, Module, Args).

send({address, _Host, _Port}=Address, Telex) ->
    gen_server:cast(?SERVER, {telex, Address, Telex}).

% --- gen_server callbacks ---

init([]) ->
    {ok, Socket} = gen_udp:open(?PORT),
    {ok, #state{socket=Socket}}.

handle_call(_Request, _From, State) ->
    {reply, ok, State}.

handle_cast({telex, {address, Host, Port}, Telex}, #state{socket=Socket}=State) ->
    gen_udp:send(Socket, Host, Port, telex:encode(Telex)),
    {noreply, State};
handle_cast(_Msg, State) ->
    {noreply, State}.

handle_info({udp, Socket, Host, Port, Msg}, #state{socket=Socket}=State) ->
    Event = {telex, {address, Host, Port}, telex:decode(Msg)},
    gen_event:notify(?EVENT, Event),
    {noreply, State};
handle_info(_Info, State) ->
    {noreply, State}.

terminate(_Reason, #state{socket=Socket}) ->
    gen_udp:close(Socket),
    ok.

code_change(_OldVsn, State, _Extra) ->
  {ok, State}.

% --- end ---

To demonstrate this, let's write the simplest possible event handler:

-module(log).

-export([start/0]).
-export([info/1, warn/1, error/1]).

-behaviour(gen_event).
-export([init/1, handle_event/2, handle_call/2, handle_info/2, terminate/2, code_change/3]).

% --- api ---

start() ->
    switch:add_sup_handler(?MODULE, none).

info(Info) ->
    error_logger:info_report([{pid, self()} | Info]).

warn(Warn) ->
    error_logger:warning_report([{pid, self()} | Warn]).

error(Error) ->
    error_logger:error_report([{pid, self()} | Error]).

% --- gen_event callbacks ---

init(none) ->
    {ok, none}.

handle_event(Event, State) ->
    log:info([Event]),
    {ok, State}.

handle_call(_Request, State) ->
    {ok, ok, State}.

handle_info(_Info, State) ->
    {ok, State}.

terminate(_Reason, _State) ->
    ok.

code_change(_OldVsn, State, _Extra) ->
    {ok, State}.

% --- end ---

Here we have some wrappers around the standard error logger and an event handler which (after masses of gen_event boilerplate) simply logs every event.

This is enough functionality now to start talking to a TeleHash node:

1> c(util), c(telex), c(switch), c(log).
{ok,log}
2> switch:start_link().
{ok,<0.55.0>,<0.56.0>}
3> log:start().
ok
4> T = {struct, [{'+end', 'a9993e364706816aba3e25717850c26c9cd0d89d'}]}.
{struct,[{'+end',a9993e364706816aba3e25717850c26c9cd0d89d}]}
5> switch:send({address, "127.0.0.1", 55555}, T).
ok
6>
=INFO REPORT==== 17-Mar-2011::12:21:13 ===
    pid: <0.55.0>
    {telex,{address,{127,0,0,1},55555},
           {struct,[{<<"_ring">>,5932},
                    {<<".see">>,[]},
                    {<<"_br">>,51},
                    {<<"_to">>,<<"127.0.0.1:42424">>}]}}

Here we ask localhost:55555 for the nearest nodes it knows to the end 'a99...89d'. The reply is contained in the .see field (which is empty because localhost:55555 hasn't seeded itself yet and so doesn't know any nodes at all).

The next post will deal with dialing, at which point we will have a working announcer.

erlang.org RSS Feed: R14B02 released

Thu, 17 Mar 2011 09:47:43 +0000

Erlang/OTP R14B02 has been released as planned on March 16:th 2011. It is the second R14 service release.

See the release notes in the readme file

Download the new release from the download page.

Highlights:

The "halfword" emulator is now official. A 64-bit emulator that uses less memory than the full 64-bit emulator.
EDoc handles Erlang specifications and types.
All test suites now run with CommonTest

Functional Jobs: Junior Erlang Programmer at Klarna (Full-time)

Thu, 17 Mar 2011 09:06:49 +0000

The year was 2005 and e-commerce had long been precarious. With an idea to offer secure and simple payment solutions, Klarna was founded that same year. Six years have passed and we have grown to an ambitious and creative company of 500 employees. We have been awarded Company of the Year in -07 (bragging), increased our turnover by 13 570 % (statistics) and managed to become the Nordic market leader (facts). Klarna is partly owned by Investment AB Öresund and Sequoia Capital - a company that saw the same potential in Google, YouTube and Apple Computer, as they do in us. We think that is a good start.

Want to join us on a journey beyond the ordinary?

Klarna is looking for Passionate Junior Programmers to the office in Stockholm.

What do you want?

You do not just wish to learn the techniques and tools used by Facebook, Google, and Amazon to build high-performance, reliable network services, but you also want join in and further develop them. You want to learn Erlang, the language that makes it easy to program in multi-core and distributed environments. You want to be part of an exceptional development team, working with some of the most famous names in the Erlang world.

What do we want?

At Klarna we put development at the center and our development team currently consists of over 40 reputable developers and technicians. We have a great passion and we strive to be in the technological forefront of everything we do: reliable servers, user-friendly Web interface and financial services. Klarna is expanding further and we need more developers who can take our proprietary systems to new heights - a system that is the essence of a financial business that serves millions of consumers around the Nordic and north European region.

Who are you?

We are looking for someone who:

is a very good programmer and who loves to write qualitative code
wants to work with a functional programming language at a young and forward business
has a strong individual drive to get things done
is creative and dares to think outside the box
have strong analytical skills and can work independently

Since we have many different nationalities, and therefore several non-Swedish speaking co-workers, at Klarna, it's important that you have very good knowledge in English.

Note that we are hiring several people for this position! The selection process is ongoing, so apply immediately.

You want...

…to work at one of the fastest growing companies in Sweden where you set the limits.
…to be given the possibility to influence and make a difference within a rapidly growing organization on an international market.
…to have a workplace filled with job satisfaction, a drive to succeed and an entrepreneurial spirit.

We want...

… you to be as dedicated to us as we’ll be to you. We don’t want you to work for Klarna, we want you to work with Klarna.

Are you up for the challenge?

Send us your CV, a personal letter and a copy of your grade transcripts, all merged into a single pdf. Selection and interviews will be conducted continuously.

Application deadline: May 2nd 2011

Start date: As soon as possible

Contact: Lisa Stål, lisa [dot] stal [at] klarna [dot] com

Scattered Thoughts: transactional mealy machines

Wed, 16 Mar 2011 16:15:21 +0000

This is a hugely overdue post about an interesting system I worked on almost a year ago whilst at Smarkets and never got around to writing about. Unfortunately I don't have the code in front of me but the overall idea is simple enough to explain without examples.

Smarkets is a betting exchange (effectively a small stock exchange for buying and selling bets). The exchange system which handles all the money and manages the markets has quite stringent requirements. We want events to be serializable (because ordering is very important in a fast moving market), low latency and ideally distributed across more than one machine. However the exchange also has to handle a large number of bursty updates focused on a small number of records (popular markets, power users). I'm told that the early prototypes using postgres simply couldn't handle the high contention so a move to a more loosely coupled system was necessary.

The architecture in place when I arrived at Smarkets was based on this paper which I highly recommend reading. The main idea is that serializability across machines is difficult verging on impossible and that systems which try to paper over this (eg fully ACID distributed transactions) tend to be fragile at scale. The proposed solution is to identify specific sets of actions which must be serializable and handle each set with a single actor on a single machine. These actors then communicate with each other via asynchronous messages. In Smarkets' case the actors are individual markets, users, accounts and orders. These can be modeled nicely as mealy machines where the output value is a list of messages, hence the title.

This idea was very effective but the implementation at Smarkets was some of the scariest code in the repository (thanks mostly to being the oldest code). Each actor was implemented as a single erlang process which archived messages (using couchdb) after reading them. There was a lot of repetitive boilerplate code, it was hard to test (because the actors message each other directly) and worst of all there were ways to lose messages before they were archived (eg process inbox is lost if the process dies, messages between machines can be dropped silently).

I wrote a new system to handle the actor implementation whilst keeping the domain-specific logic of each actor mostly unchanged. Each actor is defined by a pair of callback functions (a behaviour, in erlang-speak). The init function sets the initial state of the actor. The transition function takes the current state and an incoming message and returns the new state and possibly some outgoing messages. Everything else is handled by a generic module which takes this behaviour and turns it into a running actor. Each actor consists of an inbox, outbox and a current state, all of which are persisted using mnesia. Each actor also has a unique id used for addressing messages. The transition process - pop a message off the inbox queue, run the transition function, store the new state, push outgoing messages to the outbox - is implemented as a single ACID transaction using mnesia. For actors on the same machine messages are moved directly from one actor’s outbox to another's inbox directly using mnesia transactions. For actors on different machines the outbox using erlang messages and sends repeatedly (with exponential backoff) until the receiver confirms receipt. The outbox attaches auto-incrementing message ids to each message which, together with the actor id of the sender, allows the receiver to ignore duplicate messages.

In this way the domain-specific logic is separated from message handling and storage. This led to much less repetition and a more maintainable system. It also made it easy to setup tests or replay past events without recreating the whole system. Last, but certainly not least, it can only lose messages if the database or disk fails and even then is easier to restore from backup than the previous system.

Note that this explanation is somewhat simplified. I have glossed over some fiddly implementation details like error handling (if an actor fails to handle a message the sender needs to be notified in many cases) and also left out extra features like subscribing to state changes (eg notify me when this order is filled). There is also a knack to designing actors which must cooperate without common knowledge). Hopefully the Smarkets team will find some time to open-source the actual code.

Erlang Training and Consulting
- News: 16 March 2011: Erlang/OTP R14B02 has been released

Wed, 16 Mar 2011 15:17:09 +0000

Erlang/OTP R14B02 has been released as previously announced.

See the release notes in the readme file

Download the new release from the download page.

Highlights:

EDoc handles Erlang specifications and types
All tests run with Common Test
halfword emulator is offcial. Usage --enable-halfword-emulator during configure. This is Faster and uses less memory than the standard 64-bit emulator.
Dependency generation for Makefiles has been added to the compiler and erlc.

Erlang Inside: Interview with Tino Breddin, Embedded Erlang developer, on the social science of tech communities.

Wed, 16 Mar 2011 14:18:17 +0000

Why are communities such as Erlang’s and Ruby’s so different? What makes them approchable or not? Is America the problem? Should we forcibly relocate all Ruby developers to Sweden? Tino talks about that question and others in this fascinating interview. Follow Tino on twitter at @tolbrino and follow me at @chaddepue. Get an alert when [...]

ejabberd@jabber.ru: ejabberd 3.0.0-alpha-2 - more alpha testing

Wed, 16 Mar 2011 12:12:17 +0000

This second alpha includes many bugfixes over alpha-1.

The complete list of related tickets can be found on:
http://redir.process-one.net/ejabberd-3.0.0-alpha2

Please note that the database schema used in this preliminary release is not yet definitive, and it will probably change in the next alpha and beta releases.

When compiling the source code, it is necessary to install exmpp

Programming in the 21st Century: Don't Distract New Programmers with OOP

Wed, 16 Mar 2011 06:00:00 +0000

When I get asked "What's a good first programming language to teach my [son / daughter / other-person-with-no-programming-experience]?" my answer has been the same for the last 5+ years: Python.

That may be unexpected, coming from someone who often talks about non-mainstream languages, but I stand by it.

Python is good for a wide range of simple and interesting problems that would be too much effort in C. (Seriously, a basic spellchecker can be implemented in a few lines of Python.) There are surprisingly few sticking points where the solution is easy to see, but there's a tricky mismatch between it and the core language features. Erlang has a couple of biggies. Try implementing any algorithm that's most naturally phrased in terms of in-place array updates, for example. In Python the sailing tends to be smooth. Arrays and dictionaries and sets cover a lot of ground.

There's one caveat to using Python as an introductory programming language: avoid the object-oriented features. You can't dodge them completely, as fundamental data types have useful methods associated with them, and that's okay. Just make use of what's already provided and resist talking about how to create classes, and especially avoid talking about any notions of object-oriented design where every little bit of data has to be wrapped up in a class.

The shift from procedural to OO brings with it a shift from thinking about problems and solutions to thinking about architecture. That's easy to see just by comparing a procedural Python program with an object-oriented one. The latter is almost always longer, full of extra interface and indentation and annotations. The temptation is to start moving trivial bits of code into classes and adding all these little methods and anticipating methods that aren't needed yet but might be someday.

When you're trying to help someone learn how to go from a problem statement to working code, the last thing you want is to get them sidetracked by faux-engineering busywork. Some people are going to run with those scraps of OO knowledge and build crazy class hierarchies and end up not as focused on on what they should be learning. Other people are going to lose interest because there's a layer of extra nonsense that makes programming even more cumbersome.

At some point, yes, you'll need to discuss how to create objects in Python, but resist for as long as you can.

(If you liked this, you might like How I Learned to Stop Worrying and Love Erlang's Process Dictionary.)

Erlang Factory News: Only 5 Places left at the Erlang Factory SF Bay Area!

Mon, 14 Mar 2011 16:01:17 +0000

Process-one Blogs: Sea Beyond 2011 Talk 4: Eric Cestari on XMPP over Websocket and GitLive web push

Mon, 14 Mar 2011 11:19:06 +0000

Eric Cestari, From ProcessOne, explains his work on Websocket integration in ejabberd and introduce GitLive as a demonstration of our web push platform.

This is an interesting view of the present and future of XMPP in the browser. The talk also shows that you can already today build web push service that already takes advantage of websockets for web platforms that supports it.

All details are in Eric Cestari's talk at Sea Beyond:

You can see the slides here:

Do not miss the event summary and the other videos from Sea Beyond event.

Programming in the 21st Century: Exploring Audio Files with Erlang

Sat, 12 Mar 2011 06:00:00 +0000

It takes surprisingly little Erlang code to dig into the contents of an uncompressed audio file. And it turns out that three of the most common uncompressed audio file formats--WAV, AIFF, and Apple's CAF--all follow the same general structure. Once you understand the basics of one, it's easy to deal with the others. AIFF is the trickiest of the three, so that's the one I'll use as an example.

First, load the entire file into a binary:

load(Filename) ->
   {ok, B} = file:read_file(Filename),
   B.

There's a small header: four characters spelling out "FORM", a length which doesn't matter, then four more characters spelling out "AIFF". The interesting part is the rest of the file, so let's just validate the header and put the rest of the file into a binary called B:

<<"FORM", _:32, "AIFF", B/binary>> = load(Filename).

The "rest of file" binary is broken into chunks that follow a simple format: a four character chunk name, the length of the data in the chunk (which doesn't include the header), and then the data itself. Here's a little function that breaks a binary into a list of {Chunk_Name, Contents} pairs:

chunkify(Binary) -> chunkify(Binary, []).
chunkify(<<N1,N2,N3,N4, Len:32,
   Data:Len/binary, Rest/binary>>, Chunks) ->
   Name = list_to_atom([N1,N2,N3,N4]),
   chunkify(adjust(Len, Rest), [{Name, Data}|Chunks]);
chunkify(<<>>, Chunks) ->
   Chunks.

Ignore the adjust function for now; I'll get back to that.

Given the results of chunkify, it's easy to find a specific chunk using lists:keyfind/3. Really, though, other than to test the chunkification code, there's rarely a reason to iterate through all the chunks in a file. It's nicer to return a function that makes lookups easy. Replace the last line of chunkify with this:

fun(Name) -> element(2, lists:keyfind(Name, 1, Chunks)) end.

The key info about sample rates and number of channels and all that is in a chunk called COMM and now we've got an easy way to get at and decode that chunk:

Chunks = chunkify(B).
<<Channels:16, Frames:32, Sample_Size:16, Rate:10/binary>> =
   Chunks('COMM').

The sound samples themselves are in a chunk called SSND. The first eight bytes of that chunk don't matter, so to decode that chunk it's just:

<<_:8/binary, Samples/binary>> = Chunks('SSND').

Okay, now the few weird bits of the AIFF format. First, if the size of a chunk is odd, then there's one extra pad byte following it. That's what the adjust function is for. It checks if a pad byte exists and removes it before decoding the rest of the binary. The second quirk is that the sample rate is encoded as a ten-byte extended floating point value, and most languages don't have support for them--including Erlang. There's an algorithm in the AIFF spec for encoding and decoding extended floats, and I translated it into Erlang.

Here's the complete code for the AIFF decoder:

load_aiff(Filename) ->
   <<"FORM", _:32, "AIFF", B/binary>> = load(Filename),
   Chunks = chunkify(B),
   <<Channels:16, Frames:32, Sample_Size:16, Rate:10/binary>> =
      Chunks('COMM'),
   <<_:8/binary, Samples/binary>> = Chunks('SSND'),
   {Channels, Frames, Sample_Size, extended_to_int(Rate), Samples}.

chunkify(Binary) -> chunkify(Binary, []).
chunkify(<<N1,N2,N3,N4, Length:32,
   Data:Length/binary, Rest/binary>>, Chunks) ->
   Name = list_to_atom([N1,N2,N3,N4]),
   chunkify(adjust(Length, Rest), [{Name, Data}|Chunks]);
chunkify(<<>>, Chunks) ->
   fun(Name) -> element(2, lists:keyfind(Name, 1, Chunks)) end.

adjust(Length, B) ->
   case Length band 1 of
      1 -> <<_:8, Rest/binary>> = B, Rest;
      _ -> B
   end.

extended_to_int(<<_, Exp, Mantissa:32, _:4/binary>>) ->
   extended_to_int(30 - Exp, Mantissa, 0).
extended_to_int(0, Mantissa, Last) ->
   Mantissa + (Last band 1);
extended_to_int(Exp, Mantissa, _Last) ->
   extended_to_int(Exp - 1, Mantissa bsr 1, Mantissa).

load(Filename) ->
   {ok, B} = file:read_file(Filename),
   B.

WAV and CAF both follow the same general structure of a header followed by chunks. WAV uses little-endian values, while the other two are big-endian. CAF doesn't have chunk alignment requirements, so that removes the need for adjust. And fortunately it's only AIFF that requires that ugly conversion from extended floating point in order to get the sample rate.

Process-one Blogs: Sea Beyond 2011 Talk 3: David Banes on Cleartext

Thu, 10 Mar 2011 13:48:46 +0000

David Banes introduces Cleartext new opensource product: an Enterprise Messaging 2.0, based on ejabberd for the server part and developed in Adobe Air for the client part.

The platform features of course EIM, Group Chat & Microblogging, has security and compliance designed in, and notably has Integrated Content Filtering, Malware Protection, URL & Image Filtering and Archiving with eDiscovery.

Enjoy David Bane's presentation:

You can see the slides here:

/p>

Do not miss the event summary and the other videos from Sea Beyond event.

Programming in the 21st Century: Accidental Innovation, Part 3

Wed, 09 Mar 2011 06:00:00 +0000

I didn't write the previous two installments so I could build up my ego. I wanted to give concrete examples of innovation and the circumstances surrounding it, to show that it's not magic or glamorous, to show that innovation is more than sitting down and saying "Okay, time to innovate!"

It's curious how often the "innovative" stamp is applied to things that don't fit any kind of reasonable definition of the word. How many times have you seen text like this on random company X's "About" page:

We develop innovative solutions which enable enterprise-class cloud computing infrastructure that...something something synergy...something something "outside the box."

I've seen that enough that I've formulated a simple rule: If you have to say that you're innovating, then you're not. Or in a less snarky way: Innovation in itself is an empty goal, so if you're using it in the mission statement for the work you're doing, then odds are the rest of the mission statement is equally vacant.

Really, the only way to innovate is to do so accidentally.

In both the examples I gave, I wasn't thinking about how to do things differently. I was thinking about how to solve a problem and only that problem. The results ended up being interesting because I didn't spend all my time fixated on what other people had done, to the point where that's all I could see. If I started designing a puzzle game in 2011, I'd know all about Tetris and all the knock-offs of Tetris and all the incremental changes and improvements that stemmed from Tetris. It would be difficult work within the restrictions of the label "puzzle game" and come up with something that transcends the boundaries of those restrictions.

Suppose it's the late 1990s, and your goal is to design a next generation graphical interface for desktop PCs--something better than Windows. Already you're sunk, because you're looking at Windows, you're thinking about Windows, and all of your decisions will be colored by a long exposure to Windows-like interfaces. There are icons, a desktop, resizable windows, some kind of task bar, etc. What you end up with will almost certainly not be completely identical to Windows, but it won't be innovative.

Now there are some interesting problems behind that vague goal of building a next generation GUI. The core question is how to let the user run multiple applications at the same time and switch between them. And that question has some interesting and wide-ranging answers. You can see the results of some of those lines of thinking in current systems, such as doing away with the "app in a movable window" idea and having each application take over the entire screen. Then the question becomes a different one: How to switch between multiple full-screen apps? This is all very different than starting with the desktop metaphor and trying to morph it into something innovative.

(If you liked this, you might like How to Think Like a Pioneer.)

Erlang Training and Consulting
- News: 15 February 2011: Ulf Wiger and Michal Ptaszek will be presenting at the ACCU 2011

Mon, 07 Mar 2011 18:02:36 +0000

Two of Erlang Solutions finest programmers will be talking at this years ACCU on April 13-16, 2011 at the Barcelo Oxford Hotel, Oxford, UK.

Ulf Wiger will give a 90 minute talk titled "How hard can it be? How never to learn from failure."

Abstract: Do we ever learn from failure? Already in the Mythical Man-month, Fred Brooks talked about how programmers constantly underestimate difficulty, and, if they didn't, they wouldn't even dare to get started writing code. Programmers get to grips with hard problems by trying to solve them. Boldly exclaiming "how hard can it be?", we dive in, and often find out exactly how hard it can be. We then try to teach others, but find that they are dead-set on learning the hard way too. In this talk, we will explore with practical examples how this dynamic leads to repeated disasters when scaled to large projects.

Michal Ptaszek will give a 90 minute talk on "ComputErl - Erlang-based Framework for Many Task Computing"

Abstract: This talk shows how the Erlang programming language can be used for creating a framework for distributing and coordinating the execution of many task computing problems. The goals of the proposed solution are (1) to disperse the computation into many tasks, (2) to support multiple well-known computation models (such as master-worker, map-reduce, and pipeline), (3) to exploit the advantages of Erlang for developing an efficient and scalable framework and (4) to build a system that can scale from small to large number of tasks with minimum effort. We present the ComputErl framework, and the promising results in preliminary experiments with benchmarks as well as real scientific applications deployed over hundreds of nodes.

Other speakers include John Hughes, Tom Preston-Werner and Michael Feathers. you can find more information here.

Process-one Blogs: Sea Beyond 2011 Talk 2: Christophe Romain, Karim Gemayel on Pubsub and distributed social networks

Mon, 07 Mar 2011 14:24:55 +0000

Christophe Romain and Karim Gemayel presents XMPP pubsub in the context of distributed social networking.

Christophe Romain and Karim Gemayel have been working on XMPP pubsub for several years now. They have build ejabberd implementation as well as custom improvements in various customers context, for features and scalability.

In this presentation, they explains XMPP pubsub and how it can be used to build distributed social networks.

Do not miss the event summary and the other videos from Sea Beyond event.

Damien Katz: Is Node.js an application server?

Mon, 07 Mar 2011 03:00:04 +0000

An insightful comment on Reddit about Node.js:

And the idea that you fully understand your own code is a bit suspect, too. My code's all nice and fast until somebody passes me in a POST request with a million keys, or decides to upload a 10GB file where I was expecting a 5KB file and I run a hash algorithm over it, or I accidentally use way more memory than I expected and push the system into swap, or any number of other things like that. My life would be a lot easier if my code never did anything I didn't expect.

I keep wondering where Node fits in a production environment and who writes the code that powers it.

Erlang, Testing and TDD: Erlang TDD hand on project – WorkerNet part 8

Sat, 05 Mar 2011 15:46:09 +0000

Last story to play on the workernet job layer,

“I want to be able to delete a job once it’s done, through any node.”

To design this – the test is written first (as usual)

done_job_deleted() ->
    wn_resource_layer:register(#wn_resource{name = "Laptop",
                                     type = [{perl,1}],
                                     resides = node()
                                    }),
    Path = create_file_at(?NODE_ROOT),
    File1 = #wn_file{id = "File1",file = Path,resides = node()},
    Job1 = #wn_job{id = "JobId",
                   files = [File1],
                   resources = [perl],
                   commands = ["perl -e 'print(\"HelloWorld\n\")'"],
                   timeout = 1000
                  },
    ok = wn_job_layer:register(Job1),
    ok = wn_job_layer:stream(user,"JobId"),
    timer:sleep(500),
    ok = wn_job_layer:delete("JobId"),
    ?assertMatch([#wn_file{id="File1"}],wn_file_layer:list_files()),
    ?assertEqual([],wn_job_layer:list_all_jobs()),
    ok.

Es usual, this will not even compile first as stuff is missing and dialyzer should get a headache about it. Now, everything is there except the wn_job_layer:delete/1 function.

Therefore, the first implementation goes into wn_job_layer.erl

-spec(delete(string()) -> ok | {error,term()}).
delete(Id) ->
    gen_server:call(?MODULE,{delete,Id}).

8!--- End Of Snippet -->

next the internal handle_call/3 clause for handling the delete request

handle_call({delete,Id},_,State) ->
    Result = try_delete(Id,State),
    {reply,Result,State};

this raises the need to implement the internal try_delete/2 function

try_delete(Id,State) ->
    case ets:lookup(State#state.jobs,Id) of
        [{Id,JobKeeperPid,_}] ->
            case wn_job_keeper:get_stored_result(JobKeeperPid) of
                {ok,Result} ->
                    ok = wn_job_keeper:delete(JobKeeperPid),
                    ets:delete(State#state.jobs,Id),
                    wn_file_layer:delete_file(node(),Result);
                X -> X
            end;
        [] ->
            {error,no_such_job}
    end.

That was all the code needed inside wn_job_layer.erl . However, the last internal function brought up the need for a new function; wn_job_keeper:delete/1 so the function is to be implemented next!

In wn_job_keeper.erl

-spec(delete(pid()) -> ok | {error,term()}).
delete(Pid) ->
    gen_fsm:sync_send_all_state_event(Pid,delete).

try_delete(Id,State) ->
    case ets:lookup(State#state.jobs,Id) of
        [{Id,JobKeeperPid,_}] ->
            case wn_job_keeper:get_stored_result(JobKeeperPid) of
                {ok,Result} ->
                    ok = wn_job_keeper:delete(JobKeeperPid),
                    ets:delete(State#state.jobs,Id),
                    wn_file_layer:delete_file(node(),Result);
                X -> X
            end;
        [] ->
            {error,no_such_job}
    end.

With a handle_call clause inside it for the delete request

handle_sync_event(delete,_,done,State) ->
    {stop,normal,ok,State};
handle_sync_event(delete,_,X,State) ->
    {reply,{error,not_done},X,State};

That would be all! Since so much had already been written in prior tests, this one turned out to be quite a breeze. Of course this passes both compilation, dialyzation and testing. The judge ‘make full’ turns it’s mighty eye on this

zen:worker_net-0.1 zenon$ make full
erlc -pa . -o ebin/  src/*.erl test/*.erl
erl -pa ebin/ -eval 'eunit:test(wn_resource_layer,[verbose]), init:stop().'
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> ======================== EUnit ========================
module 'wn_resource_layer'
  module 'wn_resource_layer_tests'
    wn_resource_layer_tests: local_resource_test_ (Can register resources locally)...[0.001 s] ok
    wn_resource_layer_tests: register_distributed (Can Register Distributed)...[0.006 s] ok
    wn_resource_layer_tests: register_restart_register (Can Register, Restart and Register)...[0.015 s] ok
    wn_resource_layer_tests: register_deregister (Can Register, Deregister and Register)...[0.013 s] ok
    [done in 6.324 s]
  [done in 6.325 s]
=======================================================
  All 4 tests passed.
erl -pa ebin/ -eval 'eunit:test(wn_file_layer,[verbose]), init:stop().'
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> ======================== EUnit ========================
module 'wn_file_layer'
  module 'wn_file_layer_tests'
    wn_file_layer_tests: file_layer_local_test_ (Can store file locally)...[0.326 s] ok
    wn_file_layer_tests: file_layer_local_test_ (Can retrieve files locally)...[0.003 s] ok
    wn_file_layer_tests: file_layer_local_test_ (Can delete files locally)...[0.002 s] ok
    wn_file_layer_tests: can_store_distributed (Can store file distributed)...[0.024 s] ok
    wn_file_layer_tests: can_retrieve_distributed (Can retrieve file distributed)...[0.018 s] ok
    wn_file_layer_tests: can_delete_distributed (Can delete file distributed)...[0.020 s] ok
    wn_file_layer_tests: must_retain (Must retain information between node kill and node restart)...[0.391 s] ok
    [done in 2.348 s]
  [done in 2.348 s]
=======================================================
  All 7 tests passed.
erl -pa ebin/ -eval 'eunit:test(wn_job_layer,[verbose]), init:stop().'
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> ======================== EUnit ========================
module 'wn_job_layer'
  module 'wn_job_layer_tests'
    wn_job_layer_tests: local_test_ (Can register locally)...[0.005 s] ok
    wn_job_layer_tests: local_test_ (Executed locally)...{1299,338969,204767} : file_fetching_done
{1299,338969,204771} : executing_commands
{1299,338969,218209} : {executing,"more EUnitFile"}
{1299,338969,225651} : "1,2,3"
{1299,338969,226358} : no_more_commands
{1299,338969,226362} : building_result_tgz
{1299,338969,226462} : done
[1.018 s] ok
    wn_job_layer_tests: local_test_ (Executed queue)...{1299,338970,229138} : file_fetching_done
{1299,338970,229141} : executing_commands
{1299,338970,229539} : {executing,"file EunitFile"}
{1299,338970,239052} : "EunitFile: ASCII text"
{1299,338970,239301} : no_more_commands
{1299,338970,239304} : building_result_tgz
{1299,338970,239476} : done
{1299,338970,243238} : file_fetching_done
{1299,338970,243250} : executing_commands
{1299,338970,243360} : {executing,"cat EUnitFile"}
{1299,338970,250730} : "1,2,3"
{1299,338970,250782} : no_more_commands
{1299,338970,250785} : building_result_tgz
{1299,338970,250856} : done
[1.105 s] ok
    wn_job_layer_tests: local_test_ (Queueus on resource type amount)...[0.001 s] ok
    wn_job_layer_tests: local_test_ (Canceled in queue)...[0.001 s] ok
    wn_job_layer_tests: local_test_ (Done Job Stored in file layer)...[0.606 s] ok
    wn_job_layer_tests: local_test_ (Done Job canceled)...{1299,338971,966369} : file_fetching_done
{1299,338971,966377} : executing_commands
{1299,338971,966651} : {executing,"perl -e 'print(\"HelloWorld\n\")'"}
{1299,338972,3168} : "HelloWorld"
{1299,338972,3405} : no_more_commands
{1299,338972,3408} : building_result_tgz
{1299,338972,3508} : done
[0.504 s] ok
    [done in 3.305 s]
  [done in 3.305 s]
=======================================================
  All 7 tests passed.
dialyzer src/*.erl test/*.erl
  Checking whether the PLT /Users/zenon/.dialyzer_plt is up-to-date... yes
  Proceeding with analysis...
Unknown functions:
  eunit:test/1
 done in 0m6.92s
done (passed successfully)
zen:worker_net-0.1 zenon$

The end

This concludes the TDD hands on project for the WorkerNet – the source can now be found through my github repository for this.

git://github.com/Gianfrancoalongi/WorkerNet.git

RedHotErlang: Using SimpleBridge

Sat, 05 Mar 2011 08:38:23 +0000

I like [SimpleBridge][1]; written by [Rusty][2] and part of [Nitrogen][3]. I often use it as a separate component, i.e without using Nitrogen. SimpleBridge provides an abstraction above the underlying HTTP engine. Apart from being a nice programmi...

Erlang Inside: Interview with Kostis Sagonas – On Erlang tools, type systems, and how HiPE compares to JIT

Fri, 04 Mar 2011 15:22:45 +0000

Chad DePue long form interview with Kostis Sagonas, leader of the HiPE and Dialyzer teams and a speaker at the upcoming Erlang Factory SF Bay.

Process-one Blogs: Sea Beyond 2011 Talk 1: Mickaël Rémond on ProcessOne and TextOne protocol

Thu, 03 Mar 2011 20:14:01 +0000

Mickaël Rémond speaks about a few of ProcessOne focus for 2011 and presents TextOne Protocol.

Mickaël Rémond, ProcessOne's CEO, explains at Sea Beyond what you can expect soon from ProcessOne on a few XMPP related topics.

He then introduces TextOne Protocol, XMPP related optimisation for mobile usage.

Enjoy the presentation here:

You can see the slides here:

You can watch Sea Beyond video summary to get an overview of the full event.

Damien Katz: So You Wanna Learn About CouchDB?

Thu, 03 Mar 2011 01:43:28 +0000

CouchDB World Tour Coming! Along with Alan McKean, I and other Couchbase staff will be doing 5 training sessions in 5 different cities starting in March. I'll be teaching the San Francisco one :) We've developed some incredible material that I'm really excited to present. So go ahead and sign up, and bring a friend!

Claire was so inspired she wrote a song about it:

http://vimeo.com/20499717

Erlang, Testing and TDD: gianfrancoalongi

Tue, 01 Mar 2011 21:07:42 +0000

This evenings entertainment is

“Once a job is done, the result should be stored in the file layer, together with the logs.“

A new test needs to be devised to prove the result is stored in the file layer once a job is done. Of course the test is put in wn_job_layer_tests.erl . The test can be seen below – and as always – the test is written first.

stored_in_file_layer() ->
    %% (1)
    wn_resource_layer:register(#wn_resource{name = "Laptop",
                                     type = [{laptop,1}],
                                     resides = node()
                                    }),
    %% (2)
    Path = create_file_at(?NODE_ROOT),
    %% (3)
    File1 = #wn_file{id = "File1",file = Path,resides = node()},
    Job1 = #wn_job{id = "JobId",
                   files = [File1],
                   resources = [laptop],
                   commands = ["cat EunitFile","touch MadeFile"],
                   timeout = 100
                  },
    %% (4)
    ok = wn_job_layer:register(Job1),
    %% (5)
    timer:sleep(500),

    %% (6)
    {ok,T} = wn_job_layer:finished_at("JobId"),

    %% (7)
    TimeSuffix = wn_util:time_marker_to_string(T),
    ExpectedFileName = "JobId_result_"++TimeSuffix++".tgz",
    ExpectedFileId = "JobId_result",
    ExpectedFiles = [F1,F2,F3] = ["EUnitFile","MadeFile","Log.txt"],

    %% (8)
    {ok,ResultId} = wn_job_layer:stored_result("JobId"),
    ?assertEqual(ExpectedFileId,ResultId),

    %% (9)
    [_,Res] = wn_file_layer:list_files(),
    ?assertEqual(ExpectedFileName,filename:basename(Res#wn_file.file)),
    ?assertEqual(ExpectedFileId,Res#wn_file.id),

    %% (10)
    {ok,Tar} = wn_file_layer:retrieve_file(node(),ResultId),
    ?assertMatch({ok,ExpectedFiles},erl_tar:table(Tar,[compressed])),

    %% (11)
    erl_tar:extract(Tar,[{files,[F3]},compressed]),
    {ok,IoDev} = file:open("local_stream.txt",[write]),
    ok  = wn_job_layer:stream(IoDev,"JobId"),
    timer:sleep(100),
    file:close(IoDev),
    {ok,LocalStreamBin} = file:read_file("local_stream.txt"),
    {ok,JobLogBin} = file:read_file(F3),
    ?assertEqual(LocalStreamBin,JobLogBin),    

    %% (12)
    erl_tar:extract(Tar,[{files,[F1]},compressed]),
    {ok,LocalEunitBin} = file:read_file(Path),
    {ok,JobEunitBin} = file:read_file(F1),
    ?assertEqual(LocalEunitBin,JobEunitBin),

    %% (13)
    erl_tar:extract(Tar,[{files,[F2]},compressed]),
    ?assertEqual({ok,<<>>},file:read_file(F2)),

    %% (14)
    ok = file:delete("local_stream.txt"),
    ok = file:delete(F1),
    ok = file:delete(F2),
    ok = file:delete(F3),
    ok = file:delete(Tar).

This test is one of the bigger ones, and deserves some guidance. From the top to bottom: (1) resource registration, (2) creation of the file I wish to send as part of the job, (3) file and job declaration, (4) job registration, (5) waiting for the job to finish (one of the structural weaknesses of this test) and (6) job finish checking by getting the timestamp at the moment of finish. (7) Setting up the expected result values for comparison, (8) getting the result-id and comparing it with the expected result. (9) Listing all files in the file layer and comparing with expected result (note that there SHOULD be two files in the layer – [1] First EunitFile, [2] the result package). (10) Retrieving the expected tgz file and checking that it should contain only two specific files (the log and the sent file). (11) Extraction of the payload logfile and checking with a streamed version that it’s the same. (12) Extraction of the job file to see that it’s there, and comparison to see it’s the same content. (13) Testing that the created file is the one which we sent back, (14) Cleanup of the extracted files and the tgz result.

So, of to write the code – first the wn_util:time_marker_to_string/1 function. One important aspect is that this utility function should not be allowed to exists for the sole purpose of the testing – however as we shall see, this function does have other applications and thus survives.

In wn_util.erl, (skipping to show the export, that part should be bread and butter by now)

-spec(time_marker_to_string(time_marker()) -> string()).
time_marker_to_string({Date,Time,Now}) ->
    F = fun(X) -> string:join([integer_to_list(Y)||Y<-tuple_to_list(X)],"_")
        end,
    F(Date)++"_"++F(Time)++"_"++F(Now).

following the test-code, the expected filename is already given in (7), and it’s time to write up the wn_job_layer:stored_result/1 function, taking the JobId as parameter and giving the file-name.

Next the missing part in (8), in wn_job_layer.erl , the API function

-spec(stored_result(string()) -> {ok,string()} | {error,term()}).
stored_result(Id) ->
    gen_server:call(?MODULE,{stored_result,Id}).

and internal logic for handling it

handle_call({stored_result,Id},_,State) ->
    Result = try_get_stored(Id,State),
    {reply,Result,State};

and the seen helper function for stored result retrieval.

try_get_stored(Id,State) ->
    case ets:lookup(State#state.jobs,Id) of
        [{Id,JobKeeperPid,_}] ->
            wn_job_keeper:get_stored_result(JobKeeperPid);
        [] -> {error,no_such_job}
    end.

This directly leads to the need in fleshing out the wn_job_keeper code. Thus, time to write the wn_job_keeper:get_stored_result/1 function. However, as you are not inside my head (thanks for that!) I might need to give a quick explanation. The idea is that the job keeper holds the name of the stored result, not the result itself. So it’s not the actual tgz we get here. Remember that all files should be handled through the wn_file_layer.

In wn_job_keeper.erl: In order to maintain the stored result in the state, a new field has to be added (and while doing so I took the liberty of making the state type-declarations nicer)

-record(state, {job :: #wn_job{},
             info :: [{time(),term()}],
             result :: [{time(),term()}],
             stored_result :: string(),
             stream :: [pid()],
             done_at :: undefined | time_marker()
               }).

onward and downwards (in the source file), the API function

-spec(get_stored_result(pid()) -> {ok,string()} | {error,term()}).
get_stored_result(Pid) ->
    gen_fsm:sync_send_all_state_event(Pid,get_stored_result).

alongside the internal function for handling the call

handle_sync_event(get_stored_result,_,X,State) ->
    {reply,
     case State#state.stored_result of
         undefined -> {error,no_result};
         R -> {ok,R}
     end,X,State};

while working my way through this I noticed a bug in the get_done clause of handle_sync_event with done as StateName, and fixed it (should now be what is seen below)

handle_sync_event(get_done,_From,done,State) ->
    {reply,{ok,State#state.done_at},done,State};

The natural question is now – who called some API function to put the value we are retrieving? The wn_job_worker would seem to be a good place to do this from since the job_worker knows when it’s finished and hence can generate the resulting file, put it into the file layer and signal the wn_job_keeper.

So, going to wn_job_worker.erl, to the part when all commands have been executed, (the handle_info clause of the gen_server) – this is what I put there (not working – needs to be implemented)

handle_info({Port,eof},#state{port = Port} = State) ->
    case State#state.commands of
        [] ->
            wn_job_keeper:info(State#state.pid,no_more_commands),
            wn_job_keeper:info(State#state.pid,building_result_tgz),
            ok = file:set_cwd(State#state.olddir),
            Keeper = State#state.pid,
            TimeMarker = wn_util:time_marker(),
            wn_job_keeper:done(Keeper,TimeMarker),
            Job = wn_job_keeper:get_job(Keeper),
            Logs = wn_job_keeper:logs(Keeper),
            {Id,Name} = make_tgz_result(TimeMarker,
                                        State#state.workdir,
                                        Job,Logs),
            ok = wn_file_layer:add_file(#wn_file{id = Id,
                                                 file = Name,
                                                 resides = node()}),
            file:delete(Name),
            wn_job_keeper:set_stored_result(Keeper,Id),
            {stop,normal,State};
        [C|Commands] ->
            wn_job_keeper:info(State#state.pid,{executing,C}),
            NewPort = erlang:open_port({spawn,C},[eof,{line,2048}]),
            {noreply,State#state{commands = Commands,port = NewPort}}
    end;

This block of code jumps out of the current worker-dir, creates a resulting tgz, adds all files in the worker-dir to it, signals done to the wn_job_keeper and puts the tgz file into the file layer. Of course this is designing as we go and hence there are some functions missing. Finishing off the stuff here first, the local make_tgz_result/4 function

make_tgz_result(TimeMarker,Dir,WnJob,Logs) ->
    Id = WnJob#wn_job.id++"_result",
    Name = Id++"_"++wn_util:time_marker_to_string(TimeMarker)++".tgz",
    {ok,TarFile} = erl_tar:open(Name,[write,compressed]),
    Files = filelib:wildcard(Dir++"*"),
    lists:foreach(
      fun(File) ->
              ok = erl_tar:add(TarFile,File,filename:basename(File),[])
      end,Files),
    LogStr = lists:foldl(
               fun({TimeMark,Entry},Str) ->
                       Str++io_lib:format("~p : ~p~n",[TimeMark,Entry])
               end,"",lists:sort(lists:append([Lines || {_,Lines} <- Logs]))),
    ok = erl_tar:add(TarFile,erlang:list_to_binary(LogStr),"Log.txt",[]),
    erl_tar:close(TarFile),
    {Id,Name}.

luckily this one does not rely on any more internal / external functions. So that closes this branch of the dev. Backtracking to the handle_info clause, there are some new external functions which has to be developed, one of them is the wn_util:time_marker/0
function.

In wn_util.erl i add the following

-spec(time_marker() -> time_marker()).
time_marker() ->
    {date(),time(),now()}.

the way of signalling that a job is done has now changed, and the second argument is the time_marker of when this happened – this as the wn_job_worker must create the resulting tgz file with the correct time-stamp in the filename.

So, to change the done-signaling, the new lines in wn_job_keeper.erl are

-spec(done(pid(),time_marker()) -> ok).
done(Pid,TimeMarker) ->
    gen_fsm:send_all_state_event(Pid,{done,TimeMarker}).

the internal clause for the handle_event to process this change is

handle_event({done,TimeMarker}, working, #state{info = Info} = State) ->
    Now = now(),
    stream_msg(State#state.stream,{Now,done}),
    {next_state, done, State#state{info = [{Now,done}|Info],
                                   done_at = TimeMarker
                                  }};

Next – still handle_info of wn_job_worker.erl there is a new function to be implemented, wn_job_keeper:logs/1 which will retrieve the stdout result of the processing to be put into the Log.txt file that was seen in make_tgz_result/4.

Going into wn_job_keeper.erl , i add the following type-spec and function

-spec(logs(pid()) -> [{info|result,[{time(),term()}]}]).
logs(Pid) ->
    gen_fsm:sync_send_all_state_event(Pid,get_logs).

and the function clause to handle the log retrieval request

handle_sync_event(get_logs,_From,X,State) ->
    {reply,[{info,State#state.info},
            {result,State#state.result}],X,State};

Going back to the handle_info clause in wn_job_worker.erl, the next new thing to implement is the function for setting the result inside the wn_job_keeper. The implementation is thus naturally put in wn_job_keeper.erl.

-spec(set_stored_result(pid(),string()) -> ok).
set_stored_result(Pid,Id) ->
    gen_fsm:sync_send_all_state_event(Pid,{stored_result,Id}).

Of course we also need the clause to handle this synchronous request

handle_sync_event({stored_result,Id},_,X,State) ->
    {reply,ok,X,State#state{stored_result = Id}};

very straight forward and nothing fishy on that sandwich. Ending the wn_job_worker’s handle_info clause, some changes where made in the wn_resource_process.erl. The wn_resource_process should now no longer signal done to the wn_job_keeper as the wn_job_worker does it herself.

As the right of done-signalling has been revoked from the wn_resource_process, some code modification had to be done to the handle_info clause in wn_resource_process.erl

handle_info({'EXIT',WorkerPid,_}, State) ->
    {noreply,
     begin
      [{WorkerPid,_,QueueType}] = ets:lookup(State#state.working,WorkerPid),
      true = ets:delete(State#state.working,WorkerPid),

      case lists:keytake(QueueType,1,State#state.queues) of
        {value,{_,[]},_} ->
          case ets:lookup(State#state.slots,QueueType) of
             [{QueueType,infinity}] -> ignore;
             [{QueueType,X}] -> ets:insert(State#state.slots,{QueueType,X+1})
          end,
          State;
        {value,{Type,[QueuedPid|R]},Queues} ->
          case try_dispatch_job(QueuedPid,State,QueueType) of
             ok ->
               State#state{queues = [{Type,R}|Queues]};
            {error,taken} ->
               State
          end
         end
     end}.

As the wn_job_worker:make_tgz_result/4 dictated, the input in the Log.txt file changed what the streamer should get – in unison with (11) in the test. Thus this format change has to be accommodated in the wn_job_layer:stream/2 function

-spec(stream(term(),string()) -> ok | {error,term()}).
stream(IoDev,Id) ->
    Stream = fun(F) -> receive {T,E} -> io:format(IoDev,"~p : ~p~n",[T,E]) end, F(F) end,
    StreamPid = spawn_link(fun() -> Stream(Stream) end),
    case gen_server:call(?MODULE,{stream,StreamPid,Id}) of
        ok -> ok;
        Err ->
            exit(StreamPid,Err),
            Err
    end.

Also, the wn_job_layer:replay/2 function had to be changed

replay(Pid,#state{info = Info,result=Result}) ->
    lists:foreach(fun(Entry) -> Pid ! Entry end,
                  lists:sort(Info++Result)).

Finally! This satisfies the test! However, there is one point of concern. One of the previous tests can cause this one to fail! The culprit is

 {"Queueus on resource type amount", fun queues_on_resource_types_amount/0},

That culprit-test queued several jobs but never removed them properly – causing some interesting effects on the current working directory of the processes. The effect of having this intermediate test interfering is a good example of bad test behaviour. In the best case – we would be expecting each test to be set up in it’s own clean universe, and to die in that same universe as well, without overspilling into other parallel realities!

As can be seen – the make test shows the failure

1> ======================== EUnit ========================
module 'wn_job_layer'
  module 'wn_job_layer_tests'
    wn_job_layer_tests: local_test_ (Can register locally)...[0.004 s] ok
    wn_job_layer_tests: local_test_ (Executed locally)...{1298,929708,203063} : file_fetching_done
{1298,929708,203067} : executing_commands
{1298,929708,203217} : {executing,"more EUnitFile"}
{1298,929708,210033} : "1,2,3"
{1298,929708,210134} : no_more_commands
{1298,929708,210137} : building_result_tgz
{1298,929708,210187} : done
[1.004 s] ok
    wn_job_layer_tests: local_test_ (Executed queue)...{1298,929709,215132} : file_fetching_done
{1298,929709,215134} : executing_commands
{1298,929709,215221} : {executing,"file EunitFile"}
{1298,929709,222661} : "EunitFile: ASCII text"
{1298,929709,222721} : no_more_commands
{1298,929709,222725} : building_result_tgz
{1298,929709,222801} : done
{1298,929709,225999} : file_fetching_done
{1298,929709,226001} : executing_commands
{1298,929709,226189} : {executing,"cat EUnitFile"}
{1298,929709,231033} : "1,2,3"
{1298,929709,231087} : no_more_commands
{1298,929709,231090} : building_result_tgz
{1298,929709,231149} : done
[1.104 s] ok
    wn_job_layer_tests: local_test_ (Queueus on resource type amount)...[0.001 s] ok
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
                                                                                                                       wn_job_layer_tests: local_test_ (Canceled in queue)...ok
    wn_job_layer_tests: local_test_ (Done Job Stored in file layer)...
=ERROR REPORT==== 28-Feb-2011::21:48:30 ===
** Generic server <0.174.0> terminating
** Last message in was {'$gen_cast',{signal,<0.178.0>,a}}
** When Server state == {state,"/Users/zenon/ErlangBlog/worker_net-0.1/node_root/",
                               [{a,[]}],
                               127001,122904}
** Reason for termination ==
** {{badmatch,
        {error,
            {function_clause,
                [{wn_job_worker,'-init/1-fun-0-',[{error,enoent}]},
                 {wn_job_worker,init,1},
                 {gen_server,init_it,6},
                 {proc_lib,init_p_do_apply,3}]}}},
    [{wn_resource_process,try_dispatch_job,3},
     {wn_resource_process,handle_cast,2},
     {gen_server,handle_msg,5},
     {proc_lib,init_p_do_apply,3}]}
*failed*
::error:{badmatch,{error,not_done}}
  in function wn_job_layer_tests:stored_in_file_layer/0

    [done in 2.687 s]
  [done in 2.687 s]
=======================================================
  Failed: 1.  Skipped: 0.  Passed: 5.
zen:worker_net-0.1 zenon$

While the version in where we comment out the test in question succeeds quite nice

1> ======================== EUnit ========================
module 'wn_job_layer'
  module 'wn_job_layer_tests'
    wn_job_layer_tests: local_test_ (Can register locally)...[0.005 s] ok
    wn_job_layer_tests: local_test_ (Executed locally)...{1298,929858,474814} : file_fetching_done
{1298,929858,474817} : executing_commands
{1298,929858,474949} : {executing,"more EUnitFile"}
{1298,929858,481366} : "1,2,3"
{1298,929858,481451} : no_more_commands
{1298,929858,481454} : building_result_tgz
{1298,929858,481489} : done
[1.005 s] ok
    wn_job_layer_tests: local_test_ (Executed queue)...{1298,929859,486690} : file_fetching_done
{1298,929859,486693} : executing_commands
{1298,929859,486733} : {executing,"file EunitFile"}
{1298,929859,493820} : "EunitFile: ASCII text"
{1298,929859,493937} : no_more_commands
{1298,929859,493939} : building_result_tgz
{1298,929859,494065} : done
{1298,929859,497623} : file_fetching_done
{1298,929859,497627} : executing_commands
{1298,929859,497716} : {executing,"cat EUnitFile"}
{1298,929859,503066} : "1,2,3"
{1298,929859,503118} : no_more_commands
{1298,929859,503120} : building_result_tgz
{1298,929859,503188} : done
[1.105 s] ok
    wn_job_layer_tests: local_test_ (Canceled in queue)...ok
    wn_job_layer_tests: local_test_ (Done Job Stored in file layer)...[0.607 s] ok
    [done in 2.766 s]
  [done in 2.766 s]
=======================================================
  All 5 tests passed.

This was fixed by constraining the previous test (and making it more actual to what it should test)

queues_on_resource_types_amount() ->
    wn_resource_layer:register(#wn_resource{name = "Laptop",
                                            type = [{a,0},{b,0}],
                                            resides = node()
                                           }),
    Queued = fun() -> wn_resource_layer:queued(node(),"Laptop") end,
    Job1 = #wn_job{id = "JobId",files = [],resources = [a]},
    Job2 = Job1#wn_job{id = "JobId2", resources = [a]},
    Job3 = Job1#wn_job{id = "JobId3", resources = [a,b]},

    wn_job_layer:register(Job1),
    ?assertMatch({ok,[{a,[Job1]},{b,[]}]},Queued()),

    wn_job_layer:register(Job2),
    ?assertMatch({ok,[{a,[Job1, Job2]},{b,[]}]},Queued()),

    wn_job_layer:register(Job3),
    ?assertMatch({ok,[{b,[Job3]},
                      {a,[Job1,Job2,Job3]}
                     ]},Queued()).

This story has now ended – with the following output of ‘make full’

zen:worker_net-0.1 zenon$ make full
erlc -pa . -o ebin/  src/*.erl test/*.erl
erl -pa ebin/ -eval 'eunit:test(wn_resource_layer,[verbose]), init:stop().'
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> ======================== EUnit ========================
module 'wn_resource_layer'
  module 'wn_resource_layer_tests'
    wn_resource_layer_tests: local_resource_test_ (Can register resources locally)...[0.001 s] ok
    wn_resource_layer_tests: register_distributed (Can Register Distributed)...[0.004 s] ok
    wn_resource_layer_tests: register_restart_register (Can Register, Restart and Register)...[0.010 s] ok
    wn_resource_layer_tests: register_deregister (Can Register, Deregister and Register)...[0.008 s] ok
    [done in 0.894 s]
  [done in 0.894 s]
=======================================================
  All 4 tests passed.
erl -pa ebin/ -eval 'eunit:test(wn_file_layer,[verbose]), init:stop().'
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> ======================== EUnit ========================
module 'wn_file_layer'
  module 'wn_file_layer_tests'
   $wn_file_layer_tests: file_layer_local_test_ (Can store file locally)...[0.001 s] ok
    wn_file_layer_tests: file_layer_local_test_ (Can retrieve files locally)...[0.002 s] ok
    wn_file_layer_tests: file_layer_local_test_ (Can delete files locally)...[0.001 s] ok
    wn_file_layer_tests: can_store_distributed (Can store file distributed)...[0.016 s] ok
    wn_file_layer_tests: can_retrieve_distributed (Can retrieve file distributed)...[0.017 s] ok
    wn_file_layer_tests: can_delete_distributed (Can delete file distributed)...[0.018 s] ok
    wn_file_layer_tests: must_retain (Must retain information between node kill and node restart)...[0.321 s] ok
    [done in 1.667 s]
  [done in 1.667 s]
=======================================================
  All 7 tests passed.
erl -pa ebin/ -eval 'eunit:test(wn_job_layer,[verbose]), init:stop().'
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> ======================== EUnit ========================
module 'wn_job_layer'
  module 'wn_job_layer_tests'
    wn_job_layer_tests: local_test_ (Can register locally)...[0.004 s] ok
    wn_job_layer_tests: local_test_ (Executed locally)...{1299,10791,575180} : file_fetching_done
{1299,10791,575182} : executing_commands
{1299,10791,575370} : {executing,"more EUnitFile"}
{1299,10791,582067} : "1,2,3"
{1299,10791,582181} : no_more_commands
{1299,10791,582183} : building_result_tgz
{1299,10791,582278} : done
[1.005 s] ok
    wn_job_layer_tests: local_test_ (Executed queue)...{1299,10792,587205} : file_fetching_done
{1299,10792,587208} : executing_commands
{1299,10792,587299} : {executing,"file EunitFile"}
{1299,10792,594990} : "EunitFile: ASCII text"
{1299,10792,595034} : no_more_commands
{1299,10792,595037} : building_result_tgz
{1299,10792,595117} : done
{1299,10792,598436} : file_fetching_done
{1299,10792,598446} : executing_commands
{1299,10792,598525} : {executing,"cat EUnitFile"}
{1299,10792,603488} : "1,2,3"
{1299,10792,603533} : no_more_commands
{1299,10792,603536} : building_result_tgz
{1299,10792,603593} : done
[1.104 s] ok
    wn_job_layer_tests: local_test_ (Queueus on resource type amount)...ok
    wn_job_layer_tests: local_test_ (Canceled in queue)...ok
    wn_job_layer_tests: local_test_ (Done Job Stored in file layer)...[0.609 s] ok
    [done in 2.773 s]
  [done in 2.773 s]
=======================================================
  All 6 tests passed.
dialyzer src/*.erl test/*.erl
  Checking whether the PLT /Users/zenon/.dialyzer_plt is up-to-date... yes
  Proceeding with analysis...
Unknown functions:
  eunit:test/1
 done in 0m5.21s
done (passed successfully)

The status of the stories is now

Done stories

“I want to be able to describe a job as

A set of Files [#wn_file{}]
The possible resource types needed (disjunction)
A list of commands for running the job
A timeout the job is given while running (in seconds)
A job id – primary key for the job“

“I want to be able to register a job in the job layer, through any node.“

“Several jobs should be queued on the same resource type(s), being processed one and one in order as they where queued“

“I want to be able to list all jobs in the system, through any node.”

“The output generated by a job should be possible to see in realtime, through any node and stored to logfiles.”

“A job must be possible to cancel before it starts running, through any node.“

“Once a job is done, the result should be stored in the file layer, together with the logs.“

Not done stories

“I want to be able to delete a job once it’s done, through any node.”

In the next post, I shall write the test and implementation of the current final story on the job layer – to delete a job from any node – once it’s done.

Cheers

Programming in the 21st Century: Accidental Innovation, Part 2

Tue, 01 Mar 2011 06:00:00 +0000

In 1995 I was writing a book, a collection of interviews with people who wrote video and computer games in the 1980s. I had the inside track on the whereabouts of many of those game designers--this was before they were easy to find via Google--and decided to make use of that knowledge. But at the time technology books that weren't "how to" were a tough sell, so after a number of rejections from publishers I set the project aside.

A year later, my wife and I were running a small game development company, and those interviews resurfaced as a potential product. We were already set-up to handle orders and mail games to customers, so the book could be entirely digital. But what format to use? PDF readers were clunky and slow. Not everyone had Microsoft Word. Then it hit me: What about using HTML as a portable document format? I know, I know, it's designed to be a portable document format, but I was thinking of it as an off-line format, not just for viewing websites. I hadn't seen anyone do this yet. It was before HTML became a common format for documentation and help files.

And so for the next couple of years people paid $20 for Halcyon Days: Interviews with Classic Computer and Video Game Programmers. Shipped via U.S. mail. On a 3 1/2 inch floppy disc. Five years later I put it on the web for free.

Even though I made the leap of using HTML for off-line e-books, and web browsers as the readers, I still didn't realize how ubiquitous HTML and browsers would become. I don't remember the details of how it happened, but I asked John Romero to write the introduction, which he enthusiastically did. I mentioned that I was looking for a distribution format for those people who didn't use browsers, and his comment was (paraphrased): "Are you crazy! Don't look backward! This is the future!"

Obviously, he was right.

Process-one Blogs: Sea Beyond 2011: Video summary

Mon, 28 Feb 2011 16:30:44 +0000

Sea Beyond is an event on Realtime communication and collaboration organized by ProcessOne. Second edition took place in Paris on February 3rd, 2011. Focus this year was on mobile realtime collaboration and federated social networks.

Here is the summary video of the event:

We will release video of each demo and presentation in the "Developer Sandbox" part of the event and all the talks from the "lighthouse" conference part.

Here is the schedule for the release:

March 3rd: Mickaël Rémond, ProcessOne: TextOne protocol presentation: Improving XMPP for mobile messaging.
March 7th: Christophe Romain and Karim Gemayel, ProcessOne: Pubsub introduction and use case in the context of federated social networks.
March 10th: David Banes, Cleartext: Cleartext microblogging extension for the XMPP protocol. He will demo and Cleartext IM and Microblogging Desktop App as well as server component. Don't miss David's surprise announcement ;)
March 14th: Alexandre Eisenchteter and Thierry Bomandouki, AF83: UC Engine (Unified Collaboration Engine) and tools to help developers build real-time applications or integrate real-time features to existing applications.
March 17th: Eric Cestari, ProcessOne: XMPP based web socket implementation and emphasizes the benefit compared to previous techniques like BOSH socket. Don't miss the launch of the showcase real time pubsub push over websocket, Gitlive.
March 21st: Marek Foss, ProcessOne: Different approaches for designing mobile collaboration software.
March 24th: Diana Cheng, Vodafone: OneSocialWeb: XMPP-based distributed social network for web and mobile usage.
March 28th: Jukka Alakontiola, Nokia: Nokia Push notification platform and API.
March 31st: Julien Genestoux, Superfeedr: XMPP and pubsubhubbub for real-time large scale events and notification distribution.

Stay tuned for this extraordinary event videos and prepare yourself to attend to the next edition, hopefully in the United States in mid 2011.

RedHotErlang: External access to Intranet Polish

Thu, 24 Feb 2011 19:25:02 +0000

We are making heavy use of [Polish][1] for all our translation efforts. So far we have been using our in-house personnel to do the actual translation work. However, increasing demands for both quality and quantity made us look for external translator...

Erlang Training and Consulting
- News: 24 February 2011: Erlang Factory London 2011 - Talk Submission Open

Thu, 24 Feb 2011 10:17:03 +0000

The Erlang Factory London 2011 is returning once again and will be held on 6th – 10th June 2011 and we have opened the call for talk submissions.

Do you have what it takes to lead a conference session or tutorial? Have you made an interesting innovation, open source application or product with Erlang/OTP? Have you used Erlang in a real-world project and want to present a case study? Or have you developed a cool tool and want to give a hands-on tutorial? Tell us about it.

We welcome submissions from developers, system designers, testers, community leaders, inventors, CTOs, evangelists, students, researchers and entrepreneurs alike. Participants at the Erlang Factory want to hear about how Erlang/OTP is used real-world scenarios, talks about tricks, tools and applications which increase productivity or enable developers to write better code.

The deadline for talk submissions is the 13th April 2011 and you can submit yours here.

You can follow @erlangfactory on twitter for the latest news.

ProTest news: Erlang Factory London 2011 - Talk Submission Open

Thu, 24 Feb 2011 00:00:00 +0000

The Erlang Factory London 2011 is returning once again and will be held on 6th – 10th June 2011 and we have opened the call for talk submissions.

You can follow @erlangfactory on twitter for the latest news.

Erlang Factory News: Erlang Factory London 2011 - Talk Submission Open!

Wed, 23 Feb 2011 11:31:13 +0000

The submission deadline is the 11th April 2011.

Erlang Inside: Erlang Factory San Francisco Bay last day for early bird special of $200 off is FEB-28

Mon, 21 Feb 2011 22:34:12 +0000

Erlang Factory San Francisco Bay last day for early bird special of $200 off is FEB-28

Erlang Factory News: Early Bird Rate - 1 Week left to save over $200!

Mon, 21 Feb 2011 18:01:33 +0000

There is only 1 week left to book your place at the Early Bird Rate of $690 so don't miss out!

Programming in the 21st Century: Accidental Innovation, Part 1

Sat, 19 Feb 2011 06:00:00 +0000

In the mid-1980s I was writing 8-bit computer games. Looking back, it was the epitome of indie. I came up with the idea, worked out the design, drew the art, wrote the code, made the sound effects, all without any kind of collaboration or outside opinions. Then I'd send the finished game off to a magazine like ANALOG Computing or Antic, get a check for a few hundred dollars, and show up in print six months or a year later.

Where I got the ideas for those games is a good question. I was clearly influenced by frequent visits to arcades, but I also didn't just want to rip-off designs and write my own versions of the games I played there.

I remember seeing screenshots of some public domain games that appeared to have puzzle elements, though I never actually played them so I didn't know for sure. The puzzley aspects may have been entirely in my head. I had a flash of an idea about arranging objects so that no two of the same type could touch. Initially those objects were oranges, lemons, and limes, which made no sense, and there was still no gameplay mechanic. Somewhere in there I hit upon the idea of dropping the fruits in a pile and switched the theme to be about cans of radioactive waste that would hit critical mass if the same types were near each other for more than a moment.

Here's what I ended up with, designed and implemented in five days, start to finish: Uncle Henry's Nuclear Waste Dump.

At first glance, this may look like any of the dozens of post-Tetris puzzle games. There's a pit that you drop stuff into, and you move back and forth over the top if it. Stuff piles up at the bottom, and the primary gameplay is in deciding where to drop the next randomly-determined item. It's all very familiar, and except that the goal is reversed--to fill the pit instead preventing it from filling--there's nothing remarkable here.

Except that Tetris wasn't released in the United States until 1987 and Uncle Henry's Nuclear Waste Dump was published in 1986. I didn't even know what Tetris was until a few years later.

Was Uncle Henry's as good as Tetris? No. Not a chance. I missed the obvious idea of guiding each piece as it fell and instead used the heavy-handed mechanic of releasing the piece before a timer expired (that's the number in the upper right corner). And keeping three waste types separated wasn't nearly as addictive as fitting irregular shapes together. Overall the game wasn't what it could have been, and yet it's interesting that it's instantly recognizable as having elements of a genre that didn't exist at the time.

So close.

(While looking for the above screenshot, I found that someone wrote a clone in 2006. The part about the game ending when the pile gets too high sounds like the gameplay isn't exactly the same, but the countdown timer for dropping a can is still there.)

BeerRiot Blog: Baseball + Riak Map/Reduce: the Movie

Fri, 18 Feb 2011 13:09:19 +0000

If you have been following my posts about using Riak’s map/reduce system to compute baseball statistics via the luwak_mr library, or if they’re still sitting in your ‘read later’ pile, you may be interested in a presentation I gave yesterday on the same topic.

Video of that presentation is available on Vimeo. It covers most of the content in the blog posts, while also providing a little extra background about why luwak_mr was necessary for the work.

Erlang Training and Consulting
- News: 18 February 2011: Programme Announced for the Erlang Factory SF Bay Area 2011

Fri, 18 Feb 2011 12:17:21 +0000

The programme has been announced for this year's Erlang Factory SF Bay Area. The 3rd Erlang Factory SF Bay Area will take place at the Hilton Hotel San Francisco Airport on the 21st -25th March.

Keynote talks this year will come from Kostis Sagonas who will discuss Half a Dozen Tools for Modern Erlang Program Development and Dan Ingalls who will explore From Metacircular IDE to End User Programming, It's just not that hard.

With other speakers including Damien Katz, Creator of CouchDB and CouchOne Founder, Robert Virding, inventor of Erlang and Bluetail co-founder, Cliff Moon, Dynomite creator and committer and Noah Gift, co-author of "Python For Unix and Linux System Administration" this years Factory boasts the best programme we have ever put together.

The Early Bird Rate of $690 for the just the conference or $1490 for the University course with free entry to the conference is available until the 28th February. This is a saving of over $200 so don’t miss out and book now.

Erlang Training and Consulting
- News: 11 February 2011: Erlang Solutions are proudly sponsoring QCon London 2011

Fri, 18 Feb 2011 12:17:21 +0000

This year's Qcon London will be on the 9th - 11th March 2011 at the The Queen Elizabeth II Conference Centre London and has a great line up of names from the programming world.

As well as sponsoring the event, we will also have the Erlang Trio, Robert Virding, Ulf Wiger and Francesco Cesarini speaking at the conference. We will have a stand there, so please drop by and say hi and feel free to talk to us about your Erlang needs.

We look forward to seeing you there.

Erlang Training and Consulting
- News: 02 February 2011: Erlang Solutions Appoints new MD of Nordic Countries

Fri, 18 Feb 2011 12:17:21 +0000

Anders Waldner joins the Erlang Solutions expert team as its new Managing Director of Nordic countries, with immediate effect. He will be based in our Stockholm office. Anders responsibilities will include overseeing the operational aspects of the business and managing the workforce to ensure Erlang Solutions’ continued and rapid expansion in the Nordic market.

We are delighted by the 23 years experience in the technology industry that Anders will bring to Erlang Solutions. Before joining Erlang Solutions he worked with Logica, a business and technology service company. Within Logica he held a number of roles including acting CEO, Director of Operations, Senior Business Developer and Group Head of e-Commerce.

All of us here at Erlang Solutions look forward to working with Anders.

For more information please see the press release.

Erlang Training and Consulting
- News: 31 January 2011: Call for Summer Interns 2011

Fri, 18 Feb 2011 12:17:21 +0000

Erlang Solutions regularly provides summer internships for IT undergraduate students with a basic understanding of Erlang or other functional programming languages. Many former interns have said that this was the best learning experience they have ever had. We will be interviewing for this year's summer internships over the next couple of months.

With offices in 3 exciting European locations, Stockholm, London and Krakow, the internships offer you the chance to work with some of the best Erlang programmers around.

The Internships can be tailored around the University holidays or be taken as Erasmus Scholarships.

For more information and to apply, please visit our Jobs section.

Erlang Training and Consulting
- Jobs: Student Software Developer (abroad) , Abroad

Fri, 18 Feb 2011 12:16:02 +0000

For students of Computer Sciences and Information Technology

Your Chance:

„...Opening the door to start your career after graduation.“

We offer you the chance to deploy your expert knowledges in practice and prepare your entry into professional life as a computer scientist. As an intern at Travelping you are involved in current projects and take on responsible tasks in software development, network and system design and application engineering. You get the opportunity to enlarge your methodical and social skills in both the team and independently .We also provide support and technical environment for writing your thesis.. We are happy to assist you in choosing a suitable subject and guide you from planning to completion of the work.

Your advantages:

Deepening and expansion of IT knowledge in the professional world
Internship compensation
Potential employment prospects in Travelping
Support and technical know-how
Provision of modern working materials

Our requirements:

You should have completed the undergraduate level and a basic knowledge of telecommunication networks. Required is the safe use of linux operating systems and some experience developing web applications. Both experience in PHPMySQL or PostgreSQL based websites and in the technologies JavaScript, jQuery and Ajax is also required. Familiarity and/or experience with the webtechnologies HTML5, CouchDB, node.js, Django and Erlang/OTP is highly desired, but not required. Finally, you should be characterized by general criteria such as a high willingness to learn and stringency, by openness and a solution-oriented way of thinking.

“... get the chance to learn about your potential employer.“

Please send your application like the following

Motivation letter
CV (with photo)
Certifications & Reviews

Learn You Some Erlang: Event Handlers

Tue, 15 Feb 2011 12:30:00 +0000

Event handlers are one of the most underappreciated behaviours available in OTP (generally only used for alarms and loggers). In this chapter, we see them from a broader angle. We'll be seeing the principles behind event handlers, their generic OTP form with gen_event, and then how event managers and handlers can be used in the context of an Olympic curling game where we need to react to a bunch of events happening as we go.

Erlang Inside: Andrew Thompson on optimizing egitd (Erlang’s git daemon, written by github)

Sun, 13 Feb 2011 00:13:34 +0000

Here's a gripping series on optimizing egitd by using OTP best practices.

Damien Katz: CouchOne + Membase = Couchbase

Wed, 09 Feb 2011 18:09:22 +0000

I've got some news I'm extremely excited to finally announce: a merger between CouchOne and Membase!

A little background, I met James Phillips, the co-founder of Membase, for the first time in December. I'd heard a little about Membase up to that point, but I was most impressed with some of their high profile users. For example, Membase is a key part of Zynga, where giving millions of users a fast, low latency experience is critical.

Membase has been targeting large scale mission critical apps, being able to scale out quickly and support millions of users, and getting impressive traction. They'd been going after a very specific pain point, a completely different part of the market than what we were targeting. They've focused on performance and scalability and exploiting all the power and memory available on modern servers. Simple, Fast, Elastic.

At CouchOne we've been focusing on very different problems: mobile, sync and offline use cases. We make it easy to build applications that travel with you, allowing you access to your important data no matter the network conditions. Slow and unreliable connectivity means many businesses can't rely on the cloud for mission critical apps, all their data is gone when their network is down. But with Couch powered apps on your phone, tablet, putting data directly on the machines at the edge of the network, you have your apps and data with you at all times and safely backed up to the cloud.

Couchbase!

What James had is the vision to see the great fit between the two companies. While independently we were both doing very well, we both have a lot of growing to do yet. And amazingly, the direction Membase needed to grow, we were already doing very well. And in the direction we needed to grow, Membase was already doing very well. Not only were the part of the stack we were focusing different and complementary, but the way we built out our teams was different and complementary. I'm not sure we could have planned it any better, and we didn't plan it at all!

And so I'm thrilled to announce Couchbase, a merging of both our companies and our technology!

Technologically, we'll be joining the products together to create a high volume, low latency, elastic clustered Couchbase server system. A Couch that's Simple, Fast, Elastic with all the reliability and power of CouchDB. We'll also continue to support the Membase API, for both backwards compatibility and it's performance advantages over HTTP. We will be the only solution out there that can scale to Zynga sized workloads and down to phones and tablets and everything in between, supporting millions of users and keeping everything in sync.

For existing CouchDB users, we will fully support CouchDB's HTTP API with all its associated benefits: seamless integration with other HTTP based infrastructure, a universally supported, human-readable protocol and direct-browser access just to name a few.

Together as Couchbase, we'll have the fastest, most scalable (both scale up and scale down) NoSQL solution. We will become the standard storage for mobile devices, and the standard server technology for syncing them all together. Our unified solution will dramatically simplify your technology stack and maintenance for building fast responsive apps that scale to millions of users, and also scaling down to phones so people can work and play even when not connected to the network.

My role at Couchbase will be CTO, overseeing the technical direction of the company. Dustin Sallings will be the chief architect. Bob Weiderhold will be CEO and co-founder James Phillips will continue to be product-oriented maniac :) CouchOne co-founders Chris Anderson and Jan Lehnardt will take roles to lead our mobile efforts and to work with our developers and community.

What's in it for you?

It's all upside! In the short-term we'll be able to provide a much better developer and support experience for both for CouchOne and Membase technologies, and move the development speed ahead much faster. The long term benefits are that CouchDB users will acquire the high performance, high scale easy-fast-elastic capabilities of Membase, while Membase users will acquire CouchDB's indexing features (map/reduce views, lucene, R-Tree GeoCouch), replication, reliability, and an easy path to mobile.

This is hot stuff! 2011 is the year of Couchbase!

1011 Ltd Blog: I'm building a WebGL modelling tool for 3D printing

Wed, 09 Feb 2011 15:10:00 +0000

2 weeks ago Father Christmas delivered a MakerBot Thing-O-Matic 3D Printer to my office. I think I understand how it must have felt to own an Altair 8800 back in the day when the world was on the brink of the consumer computing revolution. Consumer 3D printing could be another such revolution. But to print something in 3D you have to create a 3D model, and my survey of the software tools available for 3D modelling has been very disappointing. They broadly fall into the following categories:

Erlang Inside: Erlang Factory SF Bay – March 21st-25th

Wed, 09 Feb 2011 12:09:50 +0000

Erlang Factory is back in the US http://www.erlang-factory.com/conference/SFBay2011 – new speakers and a 3 day course from Learn You Some Erlang’s Fred Trottier-Hébert. UPDATE: More speakers than last year, with speakers from from Basho, AT&T, CouchBase, and Ericsson (among others). Early bird rate is over Feb 28th. That is also the deadline to book the [...]

Erlang Factory News: Erlang Factory SF Bay Area - Save over $200

Wed, 09 Feb 2011 10:46:21 +0000

We are offering a fantastic saving of over $200 when you register for the conference before the 1st of March 2011.

Scattered Thoughts: tunneling in china

Mon, 07 Feb 2011 09:25:23 +0000

In Shanghai I found that ssh was blocked at the protocol level so even running sshd on port 80 doesn't work. I don't whether this is widespread or whether it was our hotel in particular that was blocking it. Regardless, I found a workaround using httptunnel.

On a server outside China:

sudo apt-get install httptunnel
sudo hts -F localhost:22 80

On your client machine:

sudo apt-get install httptunnel
sudo htc -F 22 my.server.com:80

Now you can use ssh to your hearts content:

ssh user@localhost
scp /some/file user@localhost:/some/file
darcs push user@localhost:/some/repo

Process-one Blogs: gitlive: display GitHub pushes in realtime on your website

Thu, 03 Feb 2011 14:26:01 +0000

gitlive is a new service leveraging some of the new (and some of the proven) technologies in ejabberd.

It's a widget displaying commits from GitHub ... live!

Usage is very simple, just head over to the service for detailed instructions and sample usage.

It is also deployed on the Tsung and ejabberd homepages (see the “Commits” section on the bottom right).

There are two main components:

ejabberd with custom modules
client code in javascript

The client component will attempt to connect using the most efficient protocol to ejabberd but will fallback on BOSH is necessary. It will then connect anonymously and subscribe to the PubSub projects nodes.

To be continued after the jump!

Transport protocol

Most efficient protocol?

The client will attempt to connect using the WebSocket transport to ejabberd. WebSockets are a bidirectional protocol that solve one of the problems HTTP can't solve — by design. Many crutches have been built on HTTP to solve this problem. In the XMPP world it's BOSH. But it's not efficient on the server and on the client (think mobile applications).

WebSockets will solve all this. "Will solve" because even though we have implementations in the latest versions of Safari, Chrome, Firefox and Opera, it is still an IETF draft. Little progress has been done on the subject in the last year, and the latest iterations of the protocol are not implemented. For a good overview of the issues, Dave Cridland has you covered.

But still there is an installed base we can build on. So there, we have developped a native websocket server implementation for ejabberd. And for browsers not supporting websockets, we have Adobe Flash™.

Flash?

Yes, the irony isn't lost on us either. A nice project available on GitHub, web-socket-js, implements the WebSocket API on the client and the protocol towards the server. So if you have Flash and no native WebSocket support, the gitlive widget will load the Flash file. It's only 18Kb.

Back to BOSH

If all the previous techniques fail, the widget will attempt a BOSH connection. It's many times slower than websockets and consumes more resources on client and server. But it's very failsafe as it works everywhere vanilla HTTP works.

This combination of transport mechanisms make sure that gitlive notifications can be viewed with any browser, any time.

Front-end

The rest of the code is pretty much straightforward. The client code uses a combination of StropheJS and jQuery, and the commits are templated with mustache.js. Clients connect using anonymous authentication and subscribe to the project nodes.

It also uses the p1:rebind feature, a mechanism enabling fast rebinds to a previous connection.

You can actually customize the commits by overriding the CSS and mustache templates:

var GithubPush = {nodes:["cstar/erlang-oauth"],

location:"#commits",

css: "http://web.site/your.css",

template: "inline mustache template"};

This is default mustache template:

"<li class='gh_commit' id='{{id}}'>"

+ "<a class='gh_author' href='https://github.com/{{username}}'>"

+ "<img src=' http://www.gravatar.com/avatar/{{avatarhash}}?s=70'/>{{name}}</a>"


  + "<div class='push'><h5><a href='{{repository_link}}'>{{repository_name}}</a></h5>"
  + "<h4><a href='{{link}}'>{{title}}</a></h4>"
  + "<abbr class='timeago' title='{{time}}'>{{time}}</abbr></div>"
  + "<pre>{{message}}</pre><a class='watermark' href='http://gitlive.com'>GitLive</a></li>"
"<li class='gh_commit' id='{{id}}'>"
+ "<a class='gh_author' href='https://github.com/{{username}}'>"
+ "<img src=' http://www.gravatar.com/avatar/{{avatarhash}}?s=70'/>{{name}}</a>"
+ "<div class='push'><h5><a href='{{repository_link}}'>{{repository_name}}</a></h5>"
+ "<h4><a href='{{link}}'>{{title}}</a></h4>"
+ "<abbr class='timeago' title='{{time}}'>{{time}}</abbr></div>"
+ "<pre>{{message}}</pre><a class='watermark' href='http://gitlive.com'>GitLive</a></li>"

Note: that would be nice if you'd leave a link back to http://gitlive.com/ with your custom templates.

The default CSS is available here.

Back-end

It is a very streamlined ejabberd 2.2. The only module activated is mod_pubsub. On the HTTP side, we have our custom websocket module, the http-bind module and the GitHub webhook.

After each push, GitHub will POST on the webhook a JSON payload with all the meta-information about the the commits just pushed. The GitHub webhook will create the pubsub node if necessary. (That's why you need to Test Hook on the GitHub admin inteface). The JSON payload is converted to an Atom PubSub payload, and published on the node. ejabberd takes care of the rest, distributing the payloads to subscribers.

Bugs

There are still a couple of bugs that I hid under the carpet. But this time, and this time only, I am going to remove the carpet:

BOSH connections are not reattached. The last item is not sent in that case.
WebSocket connections need to resubscribe to the channels every time even after rebind, putting an unnecessary load on the storage backend.

Both bugs will be corrected soon, and workarounds are deployed to silence these bugs in the meantime.

Future plans

Though we plan to keep this product simple, but we will federate the gitlive server with the XMPP community. You will be able to get GitHub push notifications in real time in your favorite XMPP client. Provided it can parse Atom payloads. A little birdie told me that OneTeam will in a not-so-distant future.

This product is also the first of a series enabling websites to add realtime notifications, events, or ... Javascript.

Process-one Blogs: af83 presents UCengine

Thu, 03 Feb 2011 13:57:05 +0000

Alexandre Eisenchteter and Thierry Bomandouki, from af83, have presented UCengine at SeaBeyond.

UCengine, which stands for Unified Collaboration Engine, is an erlang server for generic publish and subscribe, with persistence. It is written for customized real time collaboration applications, under the AGPL license. It exposes a ReSTful HTTP API with JSON as its primary exchange format and can store the broadcasted events in a database.

Here is short example of an event in a JSON format:

{ "type"    : "chat.message.new",
  "datetime": 1292593766682,
  "id"      : "44099145822544409583064798111089",
  "meeting" : "introduction",
  "from"    : "ucengine@example.com",
  "metadata": {
    "text": "Hi there ! I'm an event !"
  }
}

An UX framework is also available, shipped with Javascript and Ruby on Rails libraries to access the API, under MIT or GPL licenses. Additionnally, several interactive jQuery widgets are available for UI building. It lets developpers build complete custom applications in a flexible fashion.

Check out UCengine website, and also check out the code!

Process-one Blogs: Cleartext Enterprise Messaging 2.0

Thu, 03 Feb 2011 12:00:21 +0000

David Banes, from Cleartext, has released a new opensource product: an Enterprise Messaging 2.0, based on ejabberd.

At the SeaBeyond event to day in Paris, David Banes, CEO of Cleartext, has introduced a new way of doing microblogging. It is a client-compatible, standards-based protocol extension, based on message stanzas. It is not relying on PubSub nor XEP-0277: Microblogging over XMPP, that may not be ready yet, or not completely address all the requirements.

It features of course EIM, Group Chat & Microblogging, has security and compliance designed in, and notably has Integrated Content Filtering, Malware Protection, URL & Image Filtering and Archiving with eDiscovery.

Inspired by "XMPP:The Definitive Guide", it is modelled on ProcessOne Twitter Gateway, and it is a Python XMPP component (XEP-0114). It is NOT federated (yet)!

The client has text-based commands:

"me" - shows who you are, your username and jid (Jabber ID)
"ers" - shows your followers
"ing" - shows who you follow 
"f username" - follow this user
"u username" - unfollow this user "
"d username message text" - send direct message to the user 
"@username message text" - mention a user, a public message to a user 
"s" - show saved searches 
"s word" - save live search term 
"us word" - delete live search term 
"help" - show this help

Custom Stanzas

To Tweet:

<message id="156"
         from="dbanes@cleartext.com/9643007101295859746231576"
         to="cleartext.cleartext.com"
         type="chat">
  <body>it's time for end of month expenses please</body>
  <active xmlns="http://jabber.org/protocol/chatstates"/>
  <x from="dbanes@cleartext.com/9643007101295859746231576"
     stamp="2011-01-24T10:49:05Z" 
     xmlns="urn:xmpp:delay"/>
  <x xmlns="http://cleartext.net/mblog">
    <buddy type="sender"> 
    <displayName>dbanes@cleartext.com</displayName> 
    <userName>dbanes</userName> 
    <jid>dbanes@cleartext.com</jid> 
    <avatar type="hash">56a03afdd9ab286d43c8f1715fac013d4c53f15a</avatar> 
    <serviceJid>cleartext.cleartext.com</serviceJid>
</buddy> </x>
</message>

Incoming Tweet:

<message from='cleartext.cleartext.com'
         to='dbanes@cleartext.com'
         type='chat'>
    <body>@admin: my expenses will be late</body>
    <x xmlns='http://cleartext.net/mblog'>
      <buddy type='sender'>
        <displayName>Ozzie</displayName>
        <userName>admin</userName>
        <jid>admin@cleartext.com</jid>
        <avatar type='hash'>ed90623da0e6963516abce705dd24bf0e84a03bd</avatar>
        <serviceJid>cleartext.cleartext.com</serviceJid>
        <text xmlns='http://cleartext.net/mblog'>my expenses will be late</text>
      </buddy>
    </x>
</message>

Follow command:

<message id="41"
         from="dbanes@cleartext.com/4732502611296587389793170"
         to="twitter.cleartext.com"
         type="chat">
  <body>f noornet</body>
  <active xmlns="http://jabber.org/protocol/chatstates"/>
  <x from="dbanes@cleartext.com/4732502611296587389793170"
     stamp="2011-02-01T19:23:38Z"
     xmlns="urn:xmpp:delay"/>
</message>

Server response:

<message from='twitter.cleartext.com'
         to='dbanes@cleartext.com/4732502611296587389793170'
         type='chat'>
  <body>User noornet has been added to your following list</body>
</message>

If you need more information, got to the Cleartext website, and follow David Banes's GitHub account.

Process-one Blogs: OneSocialWeb for ejabberd

Thu, 03 Feb 2011 10:30:42 +0000

OneSocialWeb, the decentralized social network, now runs on ejabberd.

At SeaBeyond event, Mickaël Rémond has demoed a functionnal service based on OneSocialWeb and ejabberd. It works with the default web interface, but it' not full featured yet.

Diana Cheng from Vodafone will present and demo the full OneSocialWeb stack, the XMPP-based decentralized social network. Check in the afternoon for the full report.

Process-one Blogs: TextOne by example (part 3): Facebook friend search and federation

Tue, 01 Feb 2011 13:59:40 +0000

TextOne integrates well with Facebook, in a non-intrusive way.

We know that many of our users have already a good deal of friends in different places. We use the address book on the phone to simplify chat and discovery of contact using email (see TextOne by example Part 2: email fedearation). In two words, TextOne federates with email, thus you can chat with email users.

However, many of you also have a large network of friends on Facebook. We deal with that situation in two ways: Facebook application and Facebook messaging federation (email).

Facebook application

If you log into Facebook and point your browser to http://apps.facebook.com/textone/ you can add it to your applications. This is a Facebook application that allows you to link your TextOne ID (username) to your Facebook account.

On the following screen you can put your TextOne credentials:

If they are correct, your TextOne ID is broadcast on your wall to your friends (or every one if your prefer to select that option):

Once this is done, TextOne Facebook application presents you with your Facebook contacts that have already done the same process:

With the Facebook app it is easier to find your Facebook friends that are also using TextOne.

Facebook Messages federation

With the upcoming release of Facebook Messages (email + chat + internal Facebook messages, all in one), you get new opportunity to communicate. Throught the strong support of email in TextOne, you can have live chat conversations with Facebook contacts, without needing a Facebook account.

We had the chance to play with Facebook Messages beta and here is a preview of what is coming for everyone.

In this case, my contact initiated a conversation from Facebook Messages to my TextOne email address:

Here is what the conversation looks like from TextOne:

So, it means you can transparently discuss with Facebook contacts using Facebook Messages from TextOne. It even supports photos sending and receiving.

Do not forget however that you are not forced to use Facebook email addresses to communicate with your contact. They probably have a standard email address, supplied by more conventional providers.

Our philosophy? We respect our users

We want to give you the opportunity to link TextOne to your Facebook account and share your ID with friends in a controlled and non-intrusive way. We do not spam you with a message on your mobile each time a user is joining: you will receive a message only if your friends asks it specifically.

We want also to separate the Facebook application from the iPhone and Android application to keep the application lightweight and not cluttered the experience with any Facebook settings inside the mobile app. If you are not a Facebook user yet, there is no reason to bother you with Facebook linking in our application. Moreover, this web approach works with all mobile clients, existing and future.

This is the same reason why we will never bother you to beg for an iTunes rating your our application. We appreciate if you give us a good rating on iTunes, but will not clutter our application experience until you end up doing so.

Finally, we provide you with a tool that is privacy friendly and work in a clean and simple way. We hope you will enjoy and share the way we respect our users.

We know that with this radical stance, our application usage will be slower to take off, but we are convinced that doing the "right thing" for our users will pay off on the long term.

Read the whole "TextOne by example" series:

Functional Jobs: Software Developer (Functional Programming) at Jane Street (Full-time)

Thu, 27 Jan 2011 22:17:37 +0000

Jane Street is looking to hire great software developers with an interest in functional programming. OCaml, a statically typed functional programming with similarities to Haskell, Scheme, Erlang, F# and SML, is our language of choice. We've got the largest team of OCaml developers in any industrial setting, and probably the world's largest OCaml codebase. We use OCaml for running our entire business, supporting everything from research to systems administration to trading systems. If you're interested in seeing how functional programming plays out in the real world, there's no better place.

The atmosphere is informal and intellectual. There is a focus on education, and people learn about software and trading, both through formal classes and on the job. The work is challenging, and you get to see the practical impact of your efforts in quick and dramatic terms. Jane Street is also small enough that people have the freedom to get involved in many different areas of the business. Compensation is highly competitive, and there's a lot of room for growth.

You can learn more about Jane Street and our technology from our main site, janestreet.com. You can also look at a a talk given at CMU about why Jane Street uses functional programming (ocaml.janestreet.com/?q=node/61) our programming blog (ocaml.janestreet.com), and some papers we've written about our experience using functional programming in the real world (janestreet.com/technology/articles.php)

We also have extensive benefits, including:

90% book reimbursement for work-related books
90% tuition reimbursement for continuing education
Excellent, zero-premium medical and dental insurance
Free lunch delivered daily from a selection of restaurants
Catered breakfasts and fresh brewed Peet's coffee
An on-site, private gym in New York with towel service
Kitchens fully stocked with a variety of snack choices
Full company 401(k) match up to 6% of salary, vests immediately
Three weeks of paid vacation for new hires in the US
16 weeks fully paid maternity/paternity leave for primary caregivers, plus additional unpaid leave

More information at http://janestreet.com/workplace/benefits.php

Process-one Blogs: XWave: A tribute to Google Wave team

Thu, 27 Jan 2011 17:11:15 +0000

Wave is not dead and it lives at ProcessOne.

Google Wave is a strange beast. Demo was acclaimed in may 2009 at Google I/O, with people very excited by the annoucement of the Wave project. At this time, it was clear for anyone with project management that the ambition of the project was a five year target.

The launch phase wanted to immediately target the end-user, while the platform should have been a platform for developers, to build an ecosystem. That's what the Wave team had in mind and on that level the Wave platform was quite a success. ProcessOne was in the Wave area sandbox at Google I/O 2010, and several companies were presenting interesting products. The ecosystem was being built. The developers were enthousiastic and the potential was there. One or two more years should have been sufficient to bring that to a true platform and ecosystem.

This technical video of Wave in the enterprise is convincing for developers:

When you have to work with many customers and partners with different bug trackers, I guess you can see the potential of a trully federated bug tracking system for example.

A fiasco?

Launched in fanfare, Google was however disappointed by the end result after just one year. Given the discrepency between the scope of the ambition and the expectation put by Google, this is where the fiasco lies. Google should have either not launched Wave at all or not kill it after just one year.

The fiasco lies as well in the way Google handled the end of Google Wave. Having brought partners in that development, they just announcement in the middle of the summer that the initiative was stopped. They made no contact with partners nor gave them any opportunity at this time to gather to organise the future of Wave.

Wave did a few things wrong. One of them is the fact that the initiative did not from the begining include federation with mail. This is something possible to design however. From the end-user point of view it has been a fatal flaw, giving one more inbox to check, when it could have been the killer feature.

The Google Team did many things right however. The choice of the XMPP protocol for federation was just their best idea. XMPP is now the universal protocol for federation. The vibrant XMPP community had this part right and Instant Messaging on top of XMPP is a success showing that tens of different implementations running on thousands of domains can work together.

What is the situation with Wave now?

The Wave in a Box initiative has moved to Apache incubator and to be frank, things are getting wrong. What is plain wrong? The team now behind the protocol, I am not sure who they are, is killing the best vision of the protocol to build another HTTP based hack, like PubSubHubbub. PubSubHubbub is ugly, for example a notification does not include nor cannot include the payload, because the source cannot be trusted: the server receiving the notification has to connect back to the originating server to make sure he really sent the notification.

Federation is the best and killer feature of XMPP. There is not a single protocol that can compete with this one to build federated distributed system.

You have XMPP pubsub. The protocol has its own flaw (we are working on improvements) but it is no comparaison to Pubsubhubbub hack. HTTP is good for a class of protocol, but when you need sessions, when you need user ID, when you need internet-wide addressability, and when you need federation, it is simply not designed for this task (I plan to explain soon why HTTP misuse is responsible for today's online privacy concerns).

Many projects fail because they do not take the existing solution on federation into account. Diaspora have build a nice web interface but ignored XMPP federation and even protocol design. Diaspora-X / Diaspora-X2 is doing better at it, because they plugged it to XMPP. That's the reason our social initiative is focusing on OneSocialWeb, because they have the federation right.

So, now the Wave protocol / Wave in a Box team is heading the way or trying to simulate session / connection and federation on top of HTTP. They are simply trying to fix the wrong part of Wave. Sure, they will be something quick because that's the force of HTTP, but it will be deeply broken, and will need to be patched and add dirty hacks over time.

XWave = XMPP + Wave

We at ProcessOne are still commited to Wave, but Wave as originally designed by the Google Wave team. As we do not agree to were the Wave protocol is going now, we will stick to the original Wave ambition. We will call it XWave protocol as in XMPP Wave and as a tribute to Google Wave team vision. WaveOne server by ProcessOne, presented at Google I/O 2010, will thus be the reference implementation. As we are free from relaxing Wave compliance given the recent turn they are taking, we will keep on our work in building a dual headed protocol for Wave store using both original Wave protocol and XMPP pubsub.

You can expect new products and services based on it to be release this year.

Wave is not dead and its vision will live in XWave: Wave + XMPP vision. There is time when I know we must choose long term goal over rapid short term benefit.

Meet us at Sea Beyond event on the future of collaboration in Paris on the 3rd of february 2011.

Erlang Training and Consulting
- Jobs: Undergraduate Summer Interns, Krakow, London, Stockholm

Mon, 24 Jan 2011 13:45:54 +0000

Erlang Solutions has fifteen summer internship positions in its three locations in London, Stockholm and Krakow. As in previous years, we are looking for bright, dedicated, motivated, curious and self-driven individuals in their second to fourth year of studies in the field of Computer Science. Exposure to Erlang and Unix derivatives (preferably Linux/Mac OSX) are a must. The right candidates will be working with exciting new projects (SMS, Telecom, E-Commerce, Instant Messaging, Banking) with some of the best Erlang developers around. And as a bonus, you will be working in our new offices in these exciting European locations.

To apply please submit your CV with a covering letter, letting us know about yourself, your hobby projects, and why you want to work with us at Erlang Solutions. We need your applications by the 31st March.

Erlang Training and Consulting
- News: 21 January 2011: New Offices for the Erlang Solutions Team in Krakow, Poland

Fri, 21 Jan 2011 14:32:18 +0000

The Erlang Solutions team in Krakow has once again outgrown their space. We have moved to a new central location in the old town, only a 5 minute walk from the main train station and the central square. The new offices have their own training facilities capable of hosting Erlang User Group meetings, the Evening School of Erlang and Hackathons, providing a hub for the growing local Erlang community. It will also allow the existing team to comfortably triple in size in the upcoming year.

You can see the views from the other offices on the Erlang Solutions Facebook group.

Erlang Training and Consulting
- News: 19 January 2011: Erlang Solutions appoints new CEO

Wed, 19 Jan 2011 10:17:15 +0000

Stuart Whitfield joins the Erlang Solutions expert team as its new Chief Executive Officer, with immediate effect. He will be based primarily in our London office but will oversee all of the Erlang Solutions offices worldwide.

Stuart’s role will be varied with his key responsibility being the day-to-day management of the company, including operations, regional offices and finances, ensuring Erlang Solutions meets its objectives and continues its rapid expansion. Stuart has previously been involved with Erlang Solutions as a part-time, non-executive director, during which time the company doubled its turnover, whilst maintaining its profit margins.

Stuart has been impressed with the range of talent within the company and looks forward to working with everyone to develop the company.

All of us here at Erlang Solutions look forward to working with Stuart.

For more information please see the press release.

Erlang Training and Consulting
- News: 18 January 2011: Erlang Solutions launches a course in Test Driven Development with Erlang

Tue, 18 Jan 2011 17:03:02 +0000

The course is aimed at Support and Test Engineers, and Software Developers who have knowledge of basic Erlang. Ultimately the course will allow you to write more reliable and maintainable software thus saving time and money.

The first trip out for this course will be at the Erlang Factory SF Bay Area 2011 and it will also be delivered by Fred Hebert. Registration is currently open and a Very Early Bird Discount is being offered which will save you $400 off the cost of the course and give you entry to the 2-day conference. This is a fantastic discount and one that is only valid for the first 50 people to register.

The course will also be running in London in June. More information can be found on our website.

ProTest news: Erlang Solutions launches a course in Test Driven Development with Erlang

Tue, 18 Jan 2011 00:00:00 +0000

The course will also be running in London in June. More information can be found on our website.

Process-one Blogs: Pre-SeaBeyond training: “ejabberd and Erlang: understanding the big picture”

Fri, 14 Jan 2011 16:00:00 +0000

Before the SeaBeyond event, ProcessOne is organising a training around ejabberd and Erlang.

On the 2nd of February (just the day before SeaBeyond), ProcessOne is inviting developers for a training around our core technologies: “ejabberd and Erlang: understanding the big picture”.

Training: “ejabberd and Erlang: understanding the big picture”

This training covers the following subjects:

    ejabberd, a scalable and reliable messaging platfrom
    Erlang/OTP, the technology behind ejabberd
    Basic understanding of ejabberd
        Config files option
        Basic system administration task (start/stop/simple diagnostic)
        Setting up ejabberd clustering
    Advanced ejabberd administration tasks
        Components (Internal and external components)
        Handling Web clients
    ejabberd databases
        Working with relational databases
        Understanding Mnesia role in ejabberd
        Backup and restore
    Practical mentoring session

Training will be done by core ejabberd developer and long time erlang hackers.

The participation cost is fixed to €500 per person, please contact us to book the training. We will be delighted to welcome you.

Meet our CEO

For Project Managers, our CEO is available for meetings around your projects. Book an appointment by contacting us.

Erlang Training and Consulting
- News: 13 January 2011: Registration is now open for the Erlang Factory SF Bay Area 2011 - Save $400!

Thu, 13 Jan 2011 14:17:32 +0000

Registration for the third Erlang Factory Bay Area Conference and the Erlang University courses is now open. Register now and the first 50 people will receive a Very Early Bird Discount of $400!

The Erlang Factory Conference will take place at the Hilton San Francisco Airport Hotel on 24th - 25th March 2011. As in previous years, the conference is aimed at Erlang enthusiasts from architects to newbies.

You will get the chance to, see some fantastic names from the Erlang world including, Joe Armstrong and Robert Virding– the inventors of Erlang, Damien Katz – the creator of CouchDB, Kostis Sagonas - Leader of the HiPE team at Uppsala University and inventor of Dialyzer

Stay tuned and follow the @erlangfactory on twitter for further announcements or check the website where you can also register to receive the Very Early Bird Discount

Erlang Factory News: Erlang Factory SF Bay Area - Registration Open - Save $400!!

Thu, 13 Jan 2011 14:01:34 +0000

We have opened registration for the Erlang Factory SF Bay Area and are offering the first 50 people to register a discount of $400 off the cost of admission.

You will get the chance to, see some fantastic names from the Erlang world including, Joe Armstrong, Kostis Sagonas, Damien Katz, Justin Sheehy, network with your fellow programmers and find out what’s new not only in the world of Erlang but in the world of programming.

ProTest news: Programme Announced for the Erlang Factory SF Bay Area 2011

Thu, 13 Jan 2011 00:00:00 +0000

The programme has been announced for this year's Erlang Factory SF Bay Area. The 3rd Erlang Factory SF Bay Area will take place at the Hilton Hotel San Francisco Airport on the 21st -25th March.

Process-one Blogs: Major hosted.IM update

Wed, 12 Jan 2011 17:32:39 +0000

Our XMPP services hosted.IM has seen a major update.

We are proud to announce the latest update on hosted.IM including the following new features:

Complete redesign of the site, enhancing user experience and usability.
Multi User Chatroom control (MUC) panel to manage your chatrooms.
CORS support (Cross-Origin Resource Sharing) to allow accessing our HTTP Connection Manager for BOSH from other web chat clients.

This release also includes other fixes and improvements suggested by our users.

We continue improving our service and welcome your feedback. Much more to come soon. Thank you!

Links:

Erlang Factory News: More speakers have been announced for the Erlang Factory SF Bay Area

Wed, 12 Jan 2011 12:01:23 +0000

With the latest round of names announced, the Erlang Factory looks set to be full of talent, fantastic speakers and inspiring projects.

We are pleased to announce that speakers include, Geoff Cant, game messaging expert, Cliff Moon, Dynomite author, Noah Gift of AT&T Interactive and merle creator, Joe Williams.

Registration will be opening soon. You can keep up to date with all the latest announcements by checking the website or following us on twitter @erlangfactory.

Functional Jobs: Python and Erlang Engineers at Smarkets (Full-time)

Tue, 11 Jan 2011 17:36:31 +0000

A bit about Smarkets

Smarkets is a real-time online trading platform with an Erlang backend and Python frontend. Smarkets is one of the Wall Street Journal’s Top 10 Tech Startups to watch in Europe, was shortlisted for the Startups Awards, and is heavily driven by user-focused design and a focus on technology and engineering as a first class discipline. We use agile product development techniques and release changes frequently.

We are also heavily invested in world-class technical talent from such pedigree as Wolverine Trading, UBS, Last.fm, and Yahoo!; and are looking to foster that reputation within the community and investors.

A bit about the job and working environment

The part of our stack which runs behind the website is one of the few websites that uses Erlang as a primary language, and is supported by other well-known Erlang projects such as RabbitMQ and CouchDB.

The non-Erlang pieces of our stack are entirely written in Python, and relies heavily on asynchronous programming techniques and REST.

Key Responsibilities

Build new products and services that augment our exchange offering;
Improve existing market exchange infrastructure and services;
Program primarily using Python and Erlang; and
Work in a start up environment where getting things done is crucial.

Requirements

Computer Science (or related technical field) degree or equivalent professional experience;
The ability to learn quickly with enthusiasm;
The ability to work independently and lead design/architecture of new features;
Knowledge of transaction processing and relational database design; and
Prior experience delivering solid web-related products in a GNU/Linux environment.

Bonus points

Functional programming experience;
Experience with web technologies and methodologies like web servers, REST, and protocol buffers;
Using Git in a multi-developer environment.

Other information about the job

This is a full time, on site job;
Pay is £30k – £60k, depending on experience plus options;
The job is based in Clerkenwell, London, United Kingdom;
We’re open to helping you relocate to London for the right candidate.

Links

If you’re looking for a flexible, agile team to work with; want to work somewhere free of big corporate politics; would like a cool project to work on; and fit the above criteria we’d love to hear from you! Please provide a CV and some examples of your recent or relevant work.

Functional Jobs: Junior Erlang Programmer at Klarna (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

The year was 2005 and e-commerce had long been precarious. With an idea to offer secure and simple payment solutions, Klarna was founded that same year. Six years have passed and we have grown to an ambitious and creative company of over 400 employees. We have been awarded Company of the Year in -07 (bragging), increased our turnover by 13 570 % (statistics) and managed to become the Nordic market leader (facts). Klarna is partly owned by Investment AB Öresund and Sequoia Capital - a company that saw the same potential in Google, YouTube and Apple Computer, as they do in us. We think that is a good start.

Take the opportunity to develop in the new wave of web interface!

What do you want?

What do we want?

At Klarna we put development at the center and our development team currently consists of over 40 reputable developers and technicians whom together have over 100 years experience in Erlang and over 300 years of programming experience. We have a great passion and we strive to be in the technological forefront of everything we do: reliable servers, user-friendly Web interface and financial services.

Klarna is expanding and we need more developers who can take our proprietary systems to new heights - a system that is the essence of a financial business that serves millions of consumers around the Nordic region.

We are looking for someone who:

is a very good programmer and who loves to write qualitative code
wants to work with a functional programming language
has a strong individual drive to get things done is creative and dares to think outside the box
have strong analytical skills
can work independently
wants to work at a young and forward business

Note that we are hiring several people for this position! The selection process is ongoing, so apply immediately. The process also includes various tests.

Location: Stockholm

Start: As soon as possible

Are you up for the challenge?

Functional Jobs: Senior Programmers at Klarna (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

The year was 2005 and e-commerce had long been precarious. With an idea to offer secure and simple payment solutions, Klarna was founded that same year. Five years have passed and we have grown to an ambitious and creative company of 400 employees. We have been awarded Company of the Year in -07 (bragging), increased our turnover by 13 570 % (statistics) and managed to become the Nordic market leader (facts). Klarna is partly owned by Investment AB Öresund and Sequoia Capital - a company that saw the same potential in Google, YouTube and Apple Computer, as they do in us. We think that is a good start.

Our company

Klarna has since it's formation in 2005 focused on building a world class development department. Six years later we are well on the way to achieving this goal. With almost 45 reputable developers with over 100 years of combined experience in Erlang and over 300 years of programming experience. We are passionate about what we are achieving and working hard to keep at the forefront of delivering high quality financial services with extremely high levels of reliability and usability.

Facebook, Google and Amazon have with great success been using new technologies and tools to build high performance and reliable services on the Internet. Now Klarna is looking for senior programmers who want to take these technologies even further. These positions are especially interesting to those of you wishing to join an exceptional development team with some of the worlds most renown Erlang programmers.

The position

Due to our continuous expansion we are looking for new colleagues who can help us develop the Klarna system. A system which is the core of our financial services that serve millions of consumers around Scandinavia. With us you will be working with Erlang - all to make it easy to program in mulitcore and distributed environments.

As a developer you work will help Klarna reach 1,5 million consumers, directly affect your 400 fellow Klarna employees and support the 6000 e-stores that daily work in the Klarna system. Your code will analyse and process transactions for several billion kronor per year in a complex and distributed environment.

In your work you will cooperate closely with the rest of the development department in an atmosphere of performance and entrepreneurship. We have chosen to create a development department that stimulates every developer and gives them great personal freedom. We develop using the Kanban agile methodology and each team has great freedom to organise it's own work.

We are looking for those of you who:

are very talented programmers, at least 8 years of programming or 4 years of Erlang programming
have a strong desire to get things done
want to work with the latest technologies within web (e.g. Erlang and Ajax)
are creative and innovative
are analytical
can work independently
love to program in functional languages (e.g. Erlang, Haskell, Scheme) or modern high level languages such as Scala, Ruby and Python

We look positively on candidates with experience of:

GUI-development, internet banks, distributed databases, Scrum, electronic payment methods, statistical analysis, Erlang, XML, HTML, Yaws, OTP, Mnesia.

Note that we are hiring several people for this position! The selection process is ongoing, so apply immediately. The process also includes various tests.

Location: Stockholm

Start: As soon as possible

Functional Jobs: Technical Web Developer at Streamtech (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

We are on the lookout for programmers who like to write pragmatic but elegant code.

who we're looking for

Elegant code makes you happy.

To you, programming is not just a job but a hobby. You've been programming for fun for years.

You're not blind to practical demands, but you relish doing things the right way. When something seems to work but you're not sure how or why, you bang your head against it until you understand.

You like the net.

You love exploring interesting new languages, concepts and approaches.

Having played with Lisp, Haskell, Erlang, Prolog, Smalltalk, or other "alternative" languages gets you bonus points. The same goes for language/compiler implementation, kernel hacking and cryptography.

Because of the unfortunately strict immigration laws, already having permission to work in the Netherlands is also a plus.

the job

You'll be working on the development of our diverse web based projects – in streaming video, lawful interception and online ad management, to name a few topics – together with the rest of the programming team, from our office in The Hague.

Together with the rest of the team, you'll make applications that are technically elegant and pleasant to work with, using whatever language and technologies suit the project best.

You will be involved in projects from A to Z – including finding out what exactly the customer needs, choosing the best approach for the job, and then developing it.

what we use

We work with a variety of languages (Python, Lisp, PHP, Java and C, among others): whatever we think is best suited for the job. We don't expect you to have experience with all the languages we use, but we do expect you to be able to learn new things and enjoy doing it.

what we offer

A challenging, fun job in a small but professional team that does not think Java, OOP and XML are the answer to every question.

Functional Jobs: Senior Erlang Engineer at ExactTarget (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

ExactTarget is seeking a Senior Erlang Engineer to play a pivotal role in the development of new features for our industry leading one-to-one marketing platform. The successful candidate is a versatile developer with excellent proficiency in using Erlang in production environments and keen attention to detail that permits them to own the final quality assurance of their solutions. The successful candidate must be highly adaptable, capable of learning new technologies and APIs and demonstrate passion and enthusiasm for designing and implementing products.

Primary Responsibilities

Design/implement software solutions based upon needs and requirements.
Research issues identified by QA and develop succinct solutions.

Required Skills/Experience

Bachelor’s degree or equivalent experience in software engineering, computer science, or computer & information systems.
Excellent knowledge of Erlang and memcached.
Proven experience with building and scaling high-traffic web services.
Excellent written and verbal communication skills.
Proven experience developing solutions to complex and unusual technology challenges.

Preferred Skills/Experience

Intermediate/expert-level proficiency with Erlang, Amazon Web Services, message queuing architectures or developing SaaS API’s.
Intermediate/expert-level proficiency with agile methods, git, or continuous integration.

Functional Jobs: Software Engineer at Quora (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

Quora is a continually improving collection of questions and answers. We believe it has the potential to become one of the largest sources of high-quality information available.

We're looking for engineers and product designers to help. Some of the challenges are highly algorithmic, such as coming up with ways to organize and categorize the information effectively so that users can efficiently find what they need; others are very technical, such as working to make a web application that is complex while still being very fast; and many of the challenges are in product design, such as figuring out a way to set up and grow a healthy community and constructing intuitive interfaces for users. Quora is based in downtown Palo Alto, CA and is funded by Benchmark Capital.

We are committed to building a cutting-edge technology company that develops software the right way and is a place where engineers love to work. Some of the tools we are using include Python, Pylons, nginx, memcached, Thrift, and git. We're using continuous deployment and EC2, so all code you write will go live to production within minutes no matter what time it is. Both founders are developers. We want to build a fantastically strong engineering team and the first engineers that join us should set the tone for that.

Requirements:

B.S., M.S., or Ph.D. in Computer Science or equivalent
Extraordinary software engineering talent
Substantial experience developing web applications
Knowledge of TCP/IP and network programming
Experience with compilers, distributed systems, and machine learning a plus
Proficiency programming in Python a plus
Familiarity with functional or declarative programming languages (OCaml, Erlang) a plus
The number of years of professional experience you have isn't important to us
You should be ready to make this startup the primary focus of your life

In the short run, the most important thing is to scale and improve our first product, and so it is critical that you can immediately make big contributions to our complex web-based application. In addition to working with us in writing the software, the ideal candidate should be proficient outside the technical realm--making good product decisions, designing user interfaces that are both pretty and functional, being an effective marketer and recruiter, and being capable of handling anything else that comes up. You should also be fearless and broad inside the technical realm, happy to tackle CSS and JavaScript as readily as you are to take on optimizing the performance of a backend service in C or Java. You should be ready to jump down into C and write an interpreter for a simple language if necessary, or hook up a cluster of machines to run a distributed machine learning job you wrote. In general, you should be very good at getting things done outside your comfort zone.

You will be required to make good decisions about important things without much oversight. You should be comfortable with autonomy and ownership of large areas of product and infrastructure. It's a plus if you have worked on projects in your spare time, such as programming games in middle school or high school. An ideal candidate will have the potential to grow into a strong leadership role over time--in either a technical or management capacity (i.e. eventually the equivalent of a Google Fellow or a V.P.)

Functional Jobs: Erlang/OTP Developer at Cloudant (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

At Cloudant we're building a hosted data platform based on the Apache CouchDB project. We're serious about data management, analytics, search and distributed systems. Our founding team consists of 3 physics PhDs from MIT and our company includes experts in database tech, devops, analytics, UI/UX, and search.

We're looking for talented and passionate software engineers to join us at our Boston and Seattle offices. You should be a technical generalist ready and eager to jump into any problem and able to learn the skills necessary to do so. You will be working closely with our CTO on the core parts of our data platform and have the opportunity to influence product development at the highest levels. Our only requirement is that you have experience writing Erlang/OTP applications. While not necessary, experience with the other tools we use for our platform (C/C++, CouchDB) is a huge plus. We're more impressed by talent, passion, and ability to execute well in a small team (since we are a small team). Here is a list of requirements and other things that can't hurt your chances:

REQUIREMENTS

Mastery of Erlang/OTP Applications for distributed systems

LIST OF THINGS THAT CAN'T HURT

Experience developing in C/C++
Familiarity with CouchDB and other NoSQL databases
Notable experience working on open-source projects
An advanced degree in Computer Science or equivalent technical field

Cloudant is a VC-funded software startup with offices in Boston and Seattle. Compensation includes full time salary, generous equity compensation plan, and benefits. More info at: http://cloudant.com

Functional Jobs: Clojure Developer at Runa (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

The language: The majority of our codebase is in Clojure. We're looking for accomplished Clojure programmers - be it in a professional setting, or open-source projects. If you're just getting started with Clojure, that is OK too, as long as you also have experience with other languages such as Haskell, Erlang, an ML, or a Scheme/Lisp, or Prolog. Significant Ruby or Python experience (the languages, not just Rails or equivalent) might be OK also.

The bottom line is that we're looking for smart engineers who have taken the trouble to learn/use these other languages because they're unsatisfied with the popular ones.

Bottom-up: You should be able to think in a meta manner. You should understand what it means to create a system from the bottom-up... Runa is driven by several domain-specific languages - all the way from analytics through to what an end consumer sees on a merchant website. You need to be able to think in layers and abstractions, and understand that building up complexity by composing simpler pieces is the natural way to grow software.

Distributed systems: Runa is a complex system - with a lot of moving parts. Such a distributed system comes with globs of incidental complexity. We need to minimize this complexity, but also deal with it effectively. You should have experience with such systems, and have battle-scars that will help avoid mistakes and improve the ability to debug the system, increase testability, maintainability, and so on. You should be familiar with architecture patterns of distributed systems.

Big data: We collect a lot of data. A lot. And we need to quickly analyze all that data to extract insights and to create reports on the behavior of visitors vis a vis dozens of parameters. If you have experience wrangling large volumes of data, we want you!

Performance: As you can imagine, the Runa SaaS is extremely time-sensitive, since it needs to respond to end-users in real-time, as they browse merchant sites. You should know how to profile and optimize large systems. Experience with various tools is a plus.

Scaling: Again, Runa is a SaaS that services online merchants. Each time we add a client, we have to handle all their traffic. Each page on a merchant website may call our servers multiple times. We need to be able to scale the system up, and provide predictable and reliable throughput under loaded conditions. We should be able to determine how much capacity is needed at various points in the system. We need to be fault-tolerant and reliable (and never negatively impact a client website). Experience with scaling large systems is a plus, as is experience with various tools.

General engineering: We're a small engineering team and we want to keep it as small as we can. Everyone needs to play multiple roles. You should be ready to roll up your sleeves and jump in to handle whatever comes up. It isn't all about Clojure - it's about getting things done.

Agile/Lean: We use Mingle to drive our development process. You should be familiar with various agile/lean methodologies. We are fanatics about automated testing so you should have extensive experience with various testing tools/frameworks/philosophies and understand designing for testability.

Experience/education: You should have at least 6 years of software development experience, most of them spent dealing with issues in the fields above. A degree in electrical engineering or computer science (or equivalent) is required.

FPS: Experience with Quake, Counter Strike, or any such game is a real bonus.

Want to prove that traditional models of software development are lacking? That more can be done with less? Join the elite engineering team at Runa!

Functional Jobs: Sr. Software Engineer – Skype Connect at Skype (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

The Wider Brief

As a member of the Skype for Business Team, you will help build a new suite of services serving millions of business users. Our growing San Francisco-area team will leverage open standards to deliver communication services--including voice, video, chat and presence--to the web and enable radically new Skype applications and transform the way businesses collaborate.

The Business Specific brief

Skype for Business builds on the community of Skype users to deliver collaboration experiences targeted at small teams in businesses of all sizes. By focusing on the way real teams work together we are making collaboration services more relevant to more people. The Skype community and the availability of the Skype experience (PCs, MACs, phones, TVs and on the web) allows us to bring the Skype collaboration experience to users on whatever device they use. This coupled with the quality of the experience and the excitement and familiarity users already have for Skype experiences provides us with a unique opportunity to deliver wonderful, ubiquitous and genuinely useful collaboration experience that make Skype for Business users truly awesome

Skype is seeking a motivated Software Engineer with a passion for delivering quality software solutions. You will be an integral part of a cross-functional team including Product Managers, Testers/Quality Engineers and Architects. This is not a manual testing job. You will be designing and developing a suite of web based collaboration and communication services for millions of business users.

Key Responsibilities

World-class Software Solutions

Work within an agile development environment with the Product Managers on a daily basis to scope, develop and deliver world-class software solutions to support business requirements
Play an active role in the design, development and unit testing of production software
Collaborate with and assist the Product Managers to develop and execute release planning

Development Process

Write code!
Plan and estimate work effort and help to continuously improve the agile development process.

Service Building

Build efficient and reusable services that can scale to support large numbers of users building on top of scalable infrastructure
Participate and contribute to architecture meetings for defining how backend systems will work for massively deployed web applications
Coordinate with internal development teams in multiple geographies to deliver high quality integrated solutions on time and on budget

Leadership

The candidate should have demonstrated skills and experience in working within / contributing to high performing teams, while mentoring junior team members.
Lead one or more teams of developers delivering collaboration SaaS

Expertise/Experience

Proven track record of building and deploying large-scale distributed systems
Extensive programming experience in an Object Oriented programming language. e.g. C++, Java, etc.
Moderate experience with multiple scripting languages such as Python, Perl, Bash.
Working knowledge of Erlang or some modern functional language desirable.
Good working and applied knowledge of networking protocols including TCP and UDP
Good working and applied knowledge of communication and web protocols including HTTP, SIP/SIMPLE and XMPP
Extensive experience programming multi-threaded systems.
Experience working with Cloud storage technologies such as Cassandra, Riak, Voldemort is desired.
Applied working knowledge of Service-oriented architecture (SOA).
Worked with messaging technologies such as Protocol Buffers, Thrift and/or Message Queues.
Experience developing software in LAMP environments.
Extensive experience using and/or designing RESTful APIs
A working knowledge of Agile Development or an enthusiasm and passion to learn.
Experience developing complex network and telecommunications or Web applications services such as Azure, S4 or another large-scale internet service are desirable.
A minimum of 4 - 6 years experience leading software development team in a world class development environment.

Problem Solving Skills/Analytics

Strong quantitative and qualitative analytical skills, with extensive technical troubleshooting and the ability to use all aspects of testing, including functional, regression, load and system testing.

Interpersonal Skills

Confident and articulate in expressing their ideas to a mixed audience with varying views and challenges
Cultural sensitivity and a proven ability to work with teams and partners across multiple geographies
A sense of personal ownership is required, together with the ability to deliver to tight deadlines while managing internal stakeholder expectations

Motivation and Passion

The candidate should be creative and passionate about generating new ideas with a drive to explore areas where others haven't gone.
Have the stamina and tenacity to deliver great software solutions, with a passion for working with other developers of all personality types

Qualifications

BS or MS degree in Computer Science or Engineering OR equivalent years of work experience.
Min of 4 - 6 years experience of applicable work experience

Functional Jobs: Member of Research Staff at Nokia (Full-time)

Mon, 10 Jan 2011 05:00:00 +0000

Nokia produces in a year more than 500 million mobile devices with rich communication, media and sensing capabilities. In addition to the device manufacturing being significant logistics operations producing terabytes of data, Nokia's emerging services, software and content platforms result in various streams of diverse data.

Data Insight team in Nokia Research Center Palo Alto foacuses on the challenges and opportunities of such data. The team aims at performing world-class research on processes and techniques to derive insights to data. The team's approach combines the theoretical foundations of data analysis and computer science to hands-on skills on rapid development of scalable systems with practical relevance. The work includes gathering, analyzing, developing methods and building statistical and computational models from real-world datasets. The research topics include scalable mining of services and manufacturing, adaptivity in mobile terminals, and massively distributed information processing and collective mining in the planetary-scale networks of people and devices.

This position gives you an opportunity to contribute to one of the most exciting open-source Map/Reduce implementations in the world, Disco (http://discoproject.org), using Erlang, Python and Javascript. You will engineer scalable infrastructure for Big Data from the ground up, together with specialists in large-scale computing, visualization, data mining and machine learning.

Qualifications

We are looking for candidates with strong experience using Erlang, Python, and Javascript and who are familiar with problems of scalability and distributed computing. Experience with other languages, such as C, Haskell or Standard ML, is a plus. Candidates should be comfortable thinking in the Map/Reduce paradigm and anxious to develop platforms for Big Data, as well as tools for data analysis and machine learning that go well beyond the state-of-the-art.

Process-one Blogs: TextOne by example (part 2): email federation

Fri, 07 Jan 2011 16:32:38 +0000

You might like it or not but the largest messaging community in the world is email. It is almost instant. Everyone has an email address. If you look at your address book, you will see lots of email addresses already.

Email has drawbacks but email is both a widespread communication endpoint (everyone has an email address on the internet) and a widespread identity.

TextOne make use of both.

TextOne, XMPP and email federation

First, TextOne federates with email as a default. It takes your address book as defaut contact list so that you do not start TextOne with no one to talk to.

That's a really good start.

Create a conversation, by selecting contacts, whether they are email addresses or TextOne id:

Now, you can use all instant messaging features of TextOne over email:

Send email: Select a contact from address book and send a message to him. The contact will receive a short message in his mailbox.
Receive email: Your contact can reply to the email he receives from your TextOne. He can even initiate a conversation. Every TextOne account has a matching email address on @text-one.com domain.
Chat-like experience by cleaning up the email messages: The text received by email is cleaned up by the TextOne server. For example, it removes signatures, subject if redundant, previous email quotes remaining in case of reply...
Support all TextOne features with email contact
- Groupchat: Yes, you can do groupchat with several people using their email. You can even create groupchats and mix contacts with TextOne users and email addresses in the same conversation. It will be transparent and every one can participate, keeping the same initial list of participants.
- Image transfer: You can send photos to email contact and receive photos (attachment) using your TextOne email address. This is again transparent and we take care of optimising the incoming images for mobile size.

As a summary, TextOne offers both a JID (XMPP address) and an email address, in the form of username@text-one.com. TextOne does not do like Gmail/Gtalk, it goes way beyond, by fusioning both communication forms, in one simple, beautiful and efficient smartphone application.

Email address as migration path

The email is also used in TextOne to provide a migration path for your contacts to TextOne messaging.

The first thing to check is to make sure that you have filed your email address in the setting screen. Once, you have filed it, you will receive an email for verification. When validating your email becomes trusted:

It means you can use it for retrieving lost password (TextOne key), which is not possible if you did not fill it.
It means it can be used by the server for TextOne message rerouting. If one of your contacts has TextOne and has declared his email, he will receive a TextOne message even if you only know his email. The client will even learn that TextOne id and use it directly in the future.

Example screenshots:

Settings screen:
Input email address:

What does it imply from your user perspective?

As your contacts slowly adopt TextOne, you will transparently switch to chatting using their TextOne id instead of their email address. It will be more real time use push, etc. This is transparent and you have nothing special to do about it. That's the beauty of it. The community is already large and will grow seamlessly.

Future

What needs to be done:

We will still further improve our incoming email clean up filter. If you find cases that could be optimised, please send then to us and we will implement corresponding filters.
Send us you suggestions. We are listening!

Conclusion

TextOne is not only federated with XMPP world, it is also federated with email. It means that using TextOne you have access to the world largest messaging community ever.

Isn't it cool ?

Read the whole "TextOne by example" series:

RedHotErlang: Rewriting history in Subversion with the help of Erlang

Mon, 03 Jan 2011 21:31:25 +0000

Our development department at [Klarna](http://www.klarna.com) has grown quite a lot the last year, and because we are trying to be as agile as we can, using scrum and kanban, this has meant more teams, more branches, and more coordination and merging...

REALITY.SYS: The Social Network Soundtrack

Wed, 29 Dec 2010 19:40:43 +0000

Für alle die nicht nur das geniale Album von Trent Reznor & Atticus Ross zu The Social Network mögen, sondern auch die ganzen Musikstücke aus den Filmszenen hören möchten.

Opening scene at the bar with his girlfriend - Ball And Biscuit – White Stripes
Blogging on Live Journal after breakup - Man Fi Cool – Roots Manuva
(unknown scene) - Dreadlock Holiday – Eric Stewart and Graham Gouldman (performed by 10cc)
Caribbean party – So I Wait – Three Hour Tour
A capella singing group / thefacebook launches – I Swear – UCLA’s Bruin Harmony [ link to original by All-4-One]
(unknown scene) - Like A Bad Girl Should – The Cramps [link to version by the Torpedo Monkeys]
Talk about moving to Palo Alto (?)- Ital Visions – Roots Manuva
Intern coding & drinking contest - California Uber Alles – Dead Kennedys
Swimming pool zipline - West Coast Poplock – Ronnie Hudson
Club with Sean Parker - The Sound Of Violence – Dennis De Laat
Rowing competition in England - In the Hall of the Mountain King Op. 23 – Trent Reznor and Atticus Ross [link to version by San Francisco Symphony Orchestra]
After the rowing competition - Arrival Of The Queen Of Sheba – Lumiere String Quartet
Girls playing videogames - Crazy Baldhead – Bob Marley & The Wailers
(unknown scene) – Hesitation (Instrumental Version) – Exray’s
Zuckerberg in the car with Sean Parker on the way to play trick on Case investments - Underground Rapstar (Post-Mortem Remix) – Cage
Eduardo contract (?) - Bleed Forever – Super Furry Animals
House party - Black Book Lodge – Gluecifer
Tries to add Erica as a friend / end credits - Baby, You’re A Rich Man – The Beatles [cover version of the song]
Original music for The Social Network – Trent Reznor and Atticus Ross

Diese Liste ist auch hier zu finden (inkl. einiger Musikvideos).

Programming in the 21st Century: A Three-Year Retrospective

Wed, 29 Dec 2010 06:00:00 +0000

Programming in the 21st Century: Write Code Like You Just Learned How to Program

Fri, 24 Dec 2010 06:00:00 +0000

I'm reading Do More Faster, which is more than a bit of an advertisement for the TechStars start-up incubator, but it's a good read nonetheless. What struck me is that several of the people who went through the program, successfully enough to at least get initial funding, didn't know how to program. They learned it so they could implement their start-up ideas.

Think about that. It's like having a song idea and learning to play an instrument so you can make it real. I suspect that the learning process in this case would horrify most professional musicians, but that horror doesn't necessarily mean that it's a bad idea, or that the end result won't be successful. After all, look at how many bands find success without the benefit of a degree in music theory.

I already knew how to program when I took an "Intro to BASIC" class in high school. One project was to make a visual demo using the sixteen-color, low-res mode of the Apple II. I quickly put together something algorithmic, looping across the screen coordinates and drawing lines and changing colors. It took me about half an hour to write and tweak, and I was done.

I seriously underestimated what people would create.

One guy presented this amazing demo full of animation and shaded images. I'm talking crazy stuff, like a skull that dripped blood from its eye into a rising pool at the bottom of the screen. And that was just one segment of his project. I was stunned. Clearly I wasn't the hotshot programmer I thought was.

I eventually saw the BASIC listing for his program. It was hundreds and hundreds of lines of statements to change colors and draw points and lines. There were no loops or variables. To animate the blood he drew a red pixel, waited, then drew another red pixel below it. All the coordinates were hard-coded. How did he keep track of where to draw stuff? He had a piece of graph paper that he updated as he went.

My prior experience hurt me in this case. I was thinking about the program, and how I could write something that was concise and clean. The guy who wrote the skull demo wasn't worried about any of that. He didn't care about what the program looked like or how maintainable it was. He just wanted a way to present his vision.

There's a lesson there that's easy to forget--or ignore. It's extremely difficult to be simultaneously concerned with the end-user experience of whatever it is that you're building and the architecture of the program that delivers that experience. Maybe impossible. I think the only way to pull it off is to simply not care about the latter. Write comically straightforward code, as if you just learned to program, and go out of your way avoid wearing any kind of software engineering hat--unless what you really want to be is a software engineer, and not the designer of an experience.

(If you liked this, you might like Coding as Performance.)

Process-one Blogs: TextOne by example (part 1): XMPP federation

Thu, 23 Dec 2010 11:00:37 +0000

TextOne application is intended to be a simplified superset of the standard XMPP protocol, specially designed for mobile usage.

TextOne application is available on iPhone and Android.

Being based on a standard protocol means that you can tap into the power federation. What's this strange thing? It means that you can send messages to users and receive replies even if they are using a different XMPP messaging service than the one you are using.

Quick demonstration with TextOne iPhone app in version 2.0:

Log in to TextOne with your account
Think about a friend using Gtalk chat service
Send him a message to its address user@gmail.com
Wait that he accepts you as a contact, since Gmail requires that to receive messages: you will be notified when that happens
Start chatting together

See TextOne in action:

New message:
Gtalk address entered:
First message sent:
Your contact has accepted you:
Go on messaging:

That's it!

Simple, isn't it ?

What does it means for TextOne? It means that this beautiful texting application is not closed like all of its counterparts. It is open to the world. In the long run, it will make a huge difference.

Read the whole "TextOne by example" series:

1011 Ltd Blog: Evolving aircraft control values using a Genetic Algorithm

Wed, 22 Dec 2010 17:16:00 +0000

If you've ever had to tune PID control values, you might realise what a pain they can be. I have started developing an Erlang UAV autopilot, with the principle of leveraging in-silico simulation as much as possible. You can see some videos of those experiments here (especially the one labelled "First bumpy, but successful landing in X-plane with Erlang autopilot").

Erlang Training and Consulting
- News: 22 December 2010: Merry Christmas and a Happy New Year from Erlang Solutions

Wed, 22 Dec 2010 15:32:16 +0000

Erlang Solutions would like to wish everyone a Merry Christmas and a Happy New Year. This year we have decided that instead of sending Christmas cards, we would donate the money to a worthwhile cause. After much discussion we agreed that this year’s worthwhile cause would be Kiva.

Kiva's mission is to connect people, through lending, for the sake of alleviating poverty. Kiva empowers individuals to lend to an entrepreneur across the globe. By combining microfinance with the internet, Kiva is creating a global community of people connected through lending. Lenders can choose which project they would like to donate to and can then track the progress of this project.

Over the next year we will track our projects and keep you all up to date with how they are going. You can find out more about Kiva and how you to can become a lender on their website.

In the mean time we wish you happy holidays, a great rest, and we sincerely hope your software doesn't crash on you while you're gone (Well, if it is Erlang, it shouldn't)!! May snowflakes fall upon you like thousands of tiny concurrent processes.

Learn You Some Erlang: Rage Against The Finite-State Machines

Tue, 21 Dec 2010 12:30:00 +0000

Rage Against The Finite-State Machines introduces the reader to the concept of finite-state machines and how they're generalised in the OTP framework. We get to write an asynchronous concurrent protocol for a trading system for items between two players for a fictive game. We then implement the protocol by using the gen_fsm behaviour in the OTP way.

Erlang Factory News: New Speaker Added for the Erlang Factory SF Bay Area - Dave Smith

Mon, 20 Dec 2010 18:01:22 +0000

We are pleased to announce yet another top name from the world of Erlang will be speaking at next years conference.

Dave Smith (Dizzy) who is the author of rebar and bitcask wil be discussing and exploring how to use Rebar in a typical development environment for everything from simple Erlang code to port drivers and embedded Erlang servers. Dizzy is a speaker not to miss!

Registration will be opening soon. You can keep up to date with all the latest announcements by checking the website or following us on twitter @erlangfactory.

Process-one Blogs: Tweet.IM update: retweets, hashtags and timeline refresh

Mon, 20 Dec 2010 14:49:02 +0000

New interesting features have been deployed on Tweet.IM.

Our Twitter gateway, and thus its installation on Tweet.IM, has received new features, enabling further integration of IM and microblogging.

These will be available to clients that display formatted text, well known as XHTML-IM by the specialists. For this, you will need to have XHTML enabled on Tweet.IM by sending xhtml on. Reminder: type help to have a list of all available commands

Here are these features:

Support for the display of retweet (official retweet format)
Allow to force timeline fetch manually, by typing update
Render hash tag as a search link, which is then clickable

Here is a mouse hover of a hash tag: clicking on this one will open a new browser tab to the Twitter hashtag

With these new features, we provide more fine-grained enhancements, enabling seamless integration between real-time push technologies based on XMPP and microblogging based on the web.

Planet Trapexit - Erlang/OTP News

REALITY.SYS: Stierkampf mal anders.

REALITY.SYS: Immer dieses Händewaschen…

ejabberd@jabber.ru: ejabberd 2.1.8 - Fix PubSub

Process-one Blogs: “Google Social”: A paradox of its own

ejabberd@jabber.ru: ejabberd 2.1.7, 3.0.0-alpha-3 and exmpp 0.9.7 -- security release

ejabberd 2.1.7

erlang.org RSS Feed: R14B03 released

Erlang Factory News: Countdown to the Erlang Factory London!

Programming in the 21st Century: Constantly Create

Damien Katz: I'm in Boston next week for a Couchbase Meetup

Erlang Training and Consulting - News: 18 May 2011: Radio-Electronics publishes an article on Open Telecommunications Platform, OTP for Open Communications by Francesco Cesarini

Erlang Factory News: Early Bird Rate - Book before 22 May to save £100

Trapexit's Erlang Blog Filter: More Erlang Web Server Benchmarking

Erlang Inside: Motivated Reasoning and Erlang vs Python vs Node

Erlang Inside: Erlang in Haskell

Erlang Training and Consulting - News: 11 May 2011: Juan Puig has been invited to talk at The University of Coruña's Computer Science faculties 25th Anniversary celebrations!

Scattered Thoughts: mist

Trapexit's Erlang Blog Filter: Erlang Web Server Benchmarking

erlang.org RSS Feed: ACM SIGPLAN Erlang Workshop in Tokyo on September 23, 2011

Erlang Factory News: Book your place at the Erlang Factory London Now and save £100!

Anders Conbere's Journal: Learning to Program is Easy

Erlang Factory News: Only 5 Places left at the Erlang Factory London!

erlang.org RSS Feed: The Erlang Factory London is back!

Programming in the 21st Century: Impressed by Slow Code

Learn You Some Erlang: Building an Application With OTP

Erlang Training and Consulting - News: 20 April 2011: Registration is now open for the Erlang Factory London - Early Bird Rate Available

Erlang Factory News: Erlang Factory London Open for Registration - Save £200

Programming in the 21st Century: Follow the Vibrancy

Scattered Thoughts: telehash: router

Scattered Thoughts: telehash: gen_event woes

Erlang Factory News: Erlang Factory SF Bay Area 2011 - A Great Success

Erlang Factory News: Only 5 Places left at the Erlang Factory SF Bay Area!

Erlang Factory News: Erlang Factory London 2011 - Talk Submission Open!

Erlang Factory News: Early Bird Rate - 1 Week left to save over $200!

Erlang Factory News: Erlang Factory SF Bay Area - Save over $200

Programming in the 21st Century: Revisiting "Tricky When You Least Expect It"

Erlang Training and Consulting - News: 13 April 2011: Ulf Wiger and Robert Virding to be on the judging panel at Spawnfest - 9 - 10 July 2011

Erlang Inside: Memory Models in Erlang vs Java

Programming in the 21st Century: Caught-Up with 20 Years of UI Criticism

Functional Jobs: Programmer - Lisp and other languages at Streamtech bv (Full-time)

who we're looking for

the job

what we use

what we offer

REALITY.SYS: Adlige Posse

Process-one Blogs: Sea Beyond 2011 Talk 7: Jukka Alakontiola on Nokia Push Notifications

Erlang Training and Consulting - Jobs: Erlang Developer, Mountain View, CA, USA

Scattered Thoughts: telehash: buckets

Process-one Blogs: Sea Beyond 2011 Talk 6: Diana Cheng on OneSocialWeb

Erlang Factory News: Erlang Factory SF Bay Area 2011 - A Great Success

Trapexit's Erlang Blog Filter: London Erlang User Group Slides

Programming in the 21st Century: If You're Not Gonna Use It, Why Are You Building It?

RJ's Blog: Erlang rebar tutorial: generating releases and upgrades

Creating the project

Compiling with rebar

Running your app (development)

Generating your first release

Creating a node config

Generating the release

Upgrading to Version 2

Writing the .appup upgrade instructions

Installing the upgrade package

Version 3 and beyond

Packaging and upgrading to v3

Congratulations

Cowboying out quick fixes

Complex upgrades are proper releases, with .appup

‘Hotfixes’ preserve my sanity

Pitfalls to avoid

erlang.org RSS Feed: Mailing Lists Operating

Scattered Thoughts: telehash: bit_trees revisited

Scattered Thoughts: telehash: bit_trees

erlang.org RSS Feed: New Site Look

Erlang Training and Consulting - News: 24 March 2011: Erlang Solutions partners with Travelping to provide next generation AAA Solutions to telecom operators

Trapexit's Erlang Blog Filter: Don’t Lose Your ets Tables

Name an Heir

Give It Away

Table Manager

A Combination Approach

Erlang Training and Consulting
- News: 18 May 2011: Radio-Electronics publishes an article on Open Telecommunications Platform, OTP for Open Communications by Francesco Cesarini

Erlang Training and Consulting
- News: 11 May 2011: Juan Puig has been invited to talk at The University of Coruña's Computer Science faculties 25th Anniversary celebrations!

Erlang Training and Consulting
- News: 20 April 2011: Registration is now open for the Erlang Factory London - Early Bird Rate Available

Erlang Training and Consulting
- News: 13 April 2011: Ulf Wiger and Robert Virding to be on the judging panel at Spawnfest - 9 - 10 July 2011

Erlang Training and Consulting
- Jobs: Erlang Developer, Mountain View, CA, USA

Erlang Training and Consulting
- News: 24 March 2011: Erlang Solutions partners with Travelping to provide next generation AAA Solutions to telecom operators

Erlang Training and Consulting
- News: 16 March 2011: Erlang/OTP R14B02 has been released

Erlang Training and Consulting
- News: 15 February 2011: Ulf Wiger and Michal Ptaszek will be presenting at the ACCU 2011

Erlang Training and Consulting
- News: 24 February 2011: Erlang Factory London 2011 - Talk Submission Open

Erlang Training and Consulting
- News: 18 February 2011: Programme Announced for the Erlang Factory SF Bay Area 2011

Erlang Training and Consulting
- News: 11 February 2011: Erlang Solutions are proudly sponsoring QCon London 2011

Erlang Training and Consulting
- News: 02 February 2011: Erlang Solutions Appoints new MD of Nordic Countries

Erlang Training and Consulting
- News: 31 January 2011: Call for Summer Interns 2011

Erlang Training and Consulting
- Jobs: Student Software Developer (abroad) , Abroad