Planet Trapexit - Erlang/OTP News

Erlang Software Framework: EIF on AWS and an AMI for Erlang

2008-07-02T19:20:55+00:00

I got an email this morning that made me realize how long it had been since I had updated this site. I've been a little bit distracted since I got my book deal. Originally I submitted a proposal to write a book on Erlang, but through a series of events ...

Trapexit News Feed: Trapexit News :: New Cookbook Entry on Tracing!

2008-07-02T11:51:04+00:00

Author: francesco
Subject: New Cookbook Entry on Tracing!
Posted: Wed Jul 02, 2008 11:40 am (GMT 0)
Topic Replies: 0

Marstein has kindly contrinuted a new page to the Erlang Cookbook on tracing processes. You can view his contribution here... Or you can add your own recepie in trapexit's wiki based Erlang Cookbook.
_________________
--
http://www.erlang-consulting.com

21st Century Code Works: Notes from Erlang Exchange

2008-07-02T04:09:00+00:00

I was at the Erlang Exchange in London last week, which I really enjoyed. It was really great to see Joe Armstrong, and to see the faces behind the projects, e.g. Claes Wikström (Yaws, Bluetail, Kreditor, Tail-f), Steve Vinoski, Eric Merrit and Martin Logan (Erlware), Matthias Radestock (LShift & RabbitMQ), Mickaël Rémond (ejabberd) etc. etc.

Something that Joe said reminded me of a comment that Keith made about 6 months ago. I think this is something that gets lost in the hype around Erlang and multicore computing:

Erlang was designed to program fault tolerant systems.

It was not designed to program multicore computers. It was designed 15-20 years ago when multicore-everywhere was not even on the horizon. As a side-effect, it is a good language for multicore systems, since the only way to actually achieve fault tolerance is to have independant processes (read here for some more on this).

I think it's important for us as to remember to advocate this, and instead of marketing Erlang/OTP as "the application system for multicore", we should be marketing it as "the application system for fault tolerance". Oh, and by the way "this is one of the few systems that will actually use those cores that are coming your way".

The second point that I have been pondering, was brought on my Gordon Guthrie's talk about Erlang/OTP vs Google Apps as an application engine. At this moment, there aren't many application systems to choose from, Erlang/OTP, Google Apps and Amazon EC2 come to mind. In the context of the talk, an Application System (A/S) would be something that takes care of a lot of the non-functional requirements of you system, e.g. reliability, scalability (+distribution) etc.

For some businesses, using Google Apps or EC2 would be a good fit, but then you're tied into the value chain of that business, and in some instances you just cannot (for legal and other reasons) put your data on servers you don't own. If you go the OTP route you have full control over your application(s), but there's a lot of extra infrastructure that you will have to supply yourself.

If you start thinking of Erlang/OTP in this way, you realise that the terms should be reversed and it should actually be OTP/Erlang. The platform is actually the important thing. The platform is the thing that gives you the reliability, scalability, distribution and hot code swapping etc. that you require. Erlang becomes this awesome language that you use to build things on the platform. Looking at things from a different perspective can lead to some interesting insights.

If I'm standing at this vantage point, heated discussions like the recent meme-storm in the blogosphere of "Erlang vs Scala" become much less relevant. Is there a platform that gives you reliability, scalability, distribution, hot code replacement that uses Scala as a language? No? Oh. What a pity...

Trapexit's Erlang Blog Filter: Convenience Over Correctness

2008-07-02T02:42:58+00:00

My latest Internet Computing column, “Convenience Over Correctness,” (PDF) is now available. It continues the exploration of problems with RPC-oriented distributed programming approaches that I’ve been writing about in each of my three prior columns this year, as well as in columns from years gone by and in the erlang-questions mailing list.

For years we’ve known RPC and its descendants to be fundamentally flawed, yet many still willingly use the approach. Why? I believe the reason is simply convenience. Regardless of RPC’s well-understood problems, many developers continue to go down the RPC-oriented path because it conveniently fits the abstractions of the popular general-purpose programming languages they limit themselves to using. Making a function or method call to a remote or distributed function, object, or service appear just like any other function or method call allows such developers to stay within the comfortable confines of their language. Those who choose this approach essentially decide that developer convenience and comfort is more important than dealing with hard distribution issues like latency, concurrency, reliability, scalability, and partial failure.

Is this convenience for the developer the right thing to focus on? I really, really don’t think it is. There are ways of developing robust distributed applications that don’t require code-generation toolkits, piles of special code annotations, or brittle enterprisey frameworks. Perhaps the wonderful programming language renaissance we’re currently experiencing will help us to finally see the light and put tired old broken abstractions like RPC permanently out to pasture.

LShift on Erlang: RabbitMQ XMPP gateway released

2008-07-01T11:31:37+00:00

I’m pleased to announce that our XMPP gateway for exposing a RabbitMQ instance to the global XMPP network has been released (documentation, browse or check out code, download snapshot).

Update: Because it depends on a newer release of RabbitMQ than 1.3.0, you will also need to check out the server and codegen code from our public mercurial repositories, or download them as snapshots: server, codegen.

The mod_rabbitmq module implements an ejabberd extension module which gateways AMQP (as implemented by RabbitMQ) to XMPP.

By bridging between the two systems, we benefit from:

XMPP’s internet-scale addressing and federation
XMPP’s presence model
AMQP’s store-and-forward capability
AMQP’s routing and filtering (using exchanges and bindings)

The current implementation is a very simple mapping between the two systems. Its simplicity keeps the code short, but only exposes a subset of AMQP features to the XMPP network, and vice versa.

Yariv's Blog: Numenta, AI, and the Web

2008-06-30T04:26:22+00:00

Last week, I attended the Numenta workshop. I didn’t know much about Numenta before I went. My friend’s excitement about the technology Numenta is building piqued my curiosity, so I decided to check it out. It seemed that almost everyone else in the conference had read Jeff Hawkins’s On Intelligence and at least experimented with Numeta’s tools, so I felt like a real n00b. I’m happy I went, though, because I learned about some interesting ideas and technologies.

Jeff Hawkins, Numenta’s founder, has been fascinated with the workings of the brain throughout his career, but only two decades into it, after he founded Palm and Handspring, was he able to devote his efforts to artificial intelligence. In On Intelligence, Hawkins discusses his theories on the brain’s functions in detail. Numenta, a company he founded with Dileep George and Donna Dubinsky, aims to put these ideas to work in commercial and research applications.

Numenta is a platform company. The platform they develop is NuPIC (Numenta Platform for Intelligent Computing), a software toolkit essentially for building pattern classifiers. The fundamental concept behind NuPIC is called HTM (Hierarchical Temporal Memory). It postulates that the cortex learns to recognize patterns using a combination of two basic algorithms: hierarchical belief propagation, and the detection of invariants in a sequence of transformations in time. I won’t get into what this all means because there’s plenty of documentation on the Numenta website. I recommend browsing it if you find this interesting.

HTM is not just theory. Although NuPIC is in a very early stage, companies are applying NuPIC to a wide range of problems, including vision, voice recognition, finance, motion recognition (recognizing motion capture data to detect if a person is walking, running, sitting, etc) and games. This is just a small subset of its potential uses.

Using Nupic in its current state isn’t trivial. It provides the building blocks for HTM pattern classifiers, but application developers still have to do a good deal of work to tune the parameters of their HTM (How many nodes? How many levels in the hierarchy? How much training data to use? How many categories? What transformations to apply to the input over time to train the system?) to their problem domain. Also, some important features haven’t been implemented yet. For example, although NuPIC can be pretty effective at classifying images that contain a single object against a plain background (with enough training), it isn’t designed to recognize objects in images with noisy backgrounds or with multiple objects. (The problem of how to identify interesting objects in a scene is called the “attention” problem. To solve it you need to have a mechanism by which the top nodes could send feedback down to the bottom nodes. Hawkins said Numenta will tackle it in a future release.)

One reason I find Numenta so interesting is that I believe that NuPIC, or something like it, will play a role in the evolution of the Web. The current generation of web applications (Web 2.0 apps, if you will) is effective at aggregating massive amounts of data in different verticals (pictures, videos, bookmarks, status messages, paintings), slicing and dicing it in different ways, searching it, and displaying it in an organized fashion. Mashups provide additional context for the data gathered in the different silos of the web (Kosmix is a good example), but they don’t add any real “intelligence” to the mix, i.e. they don’t extract new knowledge from the data they aggregate. Numenta’s technology could be used to implement a new layer of intelligence on top of existing services by training it to recognize spacial and temporal pattens in the data they’ve collected. For example, imagine a Flickr API that let you submit an image and Flickr would tell you what the objects in the image are and where the picture was taken. Or a Facebook API for identifying the people in a picture. Or a Skype API for recognizing the speaker from a voice sample (creepy, I know). Or a HotOrNot API for automatically classifying the hotness of a person (ok, bad example :) ). Or a YouTube API for identifying the objects and events in a video clip. Or a icanhascheezburger API for automatically classifying the LOLness of a cat (well… maybe not :) ).

If this happens, maybe some day a mashup of these web services will be used to build something that resembles real AI. If (when?) someone manages to build a real-life WALL*E (great movie!), I think there’s a good chance its HTMs will be trained on the vast amounts of data gathered on the web.

Interesting times ahead. Stay tuned.

Web 2.0 and beyond: Erlang Exchange 2008

Roberto Saccon — 2008-06-30T02:53:58+00:00

The last two days I have spent at the Erlang Exchange Conference. It was great to met other Erlang hackers in person I just knew from their blog, emails or mailing list. Below the slides from my talk:

Programming in the 21st Century: Want to Write a Compiler? Just Read These Two Papers.

James Hague — 2008-06-29T06:00:00+00:00

Imagine you don't know anything about programming, and you want learn how to do it. You take a look at Amazon.com, and there's a highly recommended set of books by Knute or something with a promising title, The Art of Computer Programming, so you buy them. Now imagine that it's more than just a poor choice, but that all the books on programming are at written at that level.

That's the situation with books about writing compilers.

It's not that they're bad books, they're just too broadly scoped, and the authors present so much information that it's hard to know where to begin. Some books are better than others, but there are still the thick chapters about converting regular expressions into executable state machines and different types of grammars and so on. After slogging through it all you will have undoubtedly expanded your knowledge, but you're no closer to actually writing a working compiler.

Not surprisingly, the opaqueness of these books has led to the myth that compilers are hard to write.

The best source for breaking this myth is Jack Crenshaw's series, Let's Write a Compiler!, which started in 1988. This is one of those gems of technical writing where what's assumed to be a complex topic ends up being suitable for a first year programming class. He focuses on compilers of the Turbo Pascal class: single pass, parsing and code generation are intermingled, and only the most basic of optimizations are applied to the resulting code. The original tutorials used Pascal as the implementation language, but there's a C version out there, too. If you're truly adventurous, Marcel Hendrix has done a Forth translation (and as Forth is an interactive language, it's easier to experiment with and understand than the C or Pascal sources).

As good as it is, Crenshaw's series has two major omissions. The first is that there's no internal representation of the program at all. That is, no abstract syntax tree. It is indeed possible to bypass this step if you're willing to give up flexibility, but the main reason it's not in the tutorials is because manipulating trees in Pascal is out of sync with the simplicity of the rest of the code he presents. If you're working in a higher level language--Python, Ruby, Erlang, Haskell, Lisp--then this worry goes away. It's trivially easy to create and manipulate tree-like representations of data. Indeed, this is what Lisp, Erlang, and Haskell were designed for.

That brings me to A Nanopass Framework for Compiler Education [PDF] by Sarkar, Waddell, and Dybvig. The details of this paper aren't quite as important as the general concept: a compiler is nothing more than a series of transformations of the internal representation of a program. The authors promote using dozens or hundreds of compiler passes, each being as simple as possible. Don't combine transformations; keep them separate. The framework mentioned in the title is a way of specifying the inputs and outputs for each pass. The code is in Scheme, which is dynamically typed, so data is validated at runtime.

After writing a compiler or two, then go ahead and plunk down the cash for the infamous Dragon Book or one of the alternatives. Maybe. Or you might not need them at all.

Yariv's Blog: Twoorl Goes Multilingual

2008-06-29T03:24:47+00:00

Since its launch, Twoorl users have helped translate it to Spanish, German, French, Korean, Polish, Portuguese (Brazilian) and Russian. This is an awesome contribution from the Twoorl community. Big thanks to everyone who contributed a translation!

If you’re fluent in a language that Twoorl hasn’t been translated into and you’d like to contribute a translation for it, obtain the file twoorl_eng.erl, translate the english strings, and email me the modified file. (Please make sure the file is encoded in UTF-8.) If you’re familiar with Git, you can also clone the repository, make the changes, and send me a message through GitHub to pull your updates. Thanks in advance!

Namsisi: The Erlang Exchange

2008-06-26T14:44:23+00:00

I’m currently attending the Erlang Exchange conference in London.

So far I’ve attended two presentations in the main track before lunch and two presentations in the Tools & Gadgets track. I’m going to give a brief summary below. All the talks are being streamed on Yaika.

Main Track

Joe Armstrong on Software: Erlang & SMP

This is the talk of Joe Armstrong which kind of aligns with his book. It is quite basic and more aimed at Erlang beginners that experts. It contains a few interesting bits however, mostly the stuff about Symmetric Multi-Processing (SMP) and multi core processors. Not specifically Erlang related but at least message passing oriented, or as Joe calls it, Concurrency Oriented Programming (COP). All in all, “old stuff” (in a good sense) with some really interesting insights.

Introducing Erlang to Motorola: The Journey to Success

This talk was given by Nicholas Gunder and Torben Hoffman (which I enjoyed the OTP course together with) from Motorola (which interesting enough is now using Erlang, the language developed by their main competitor Ericsson). It was more of a manager talk than a technical talk, although they sometimes go all the way down to the hardware level, for example when talking about the Gumstix computers they are using.

This was a really nice talk, Nicholas and Torben are really funny guys. It gave a good insight into how they pushed Erlang from being a basically non-existant technology inside Motorola to a technology about which the managers ask “is there’s any other projects we could apply this to?” Their main conclusion was, I think, “have something finished to show.” That means, whenever pushing Erlang to management, show them something working. This is among other things a terrific illustrator on how productive Erlang is, which also compared to C is one of the selling points.

Tools & Gadgets Track

Erlang D-Trace

Given by Garry Bulmer, the talk was about intergrating Erlang with the OS X/Unix tracing facility DTrace. DTrace basically lets you trace on “anything” in the operating system and user space programs. You can trace on functions, accumulate data etc using your own DTrace scripts. Erlang already has an extremely powerful tracing and debugging facility. This integration aims to give a bigger picture on how your Erlang application and the rest of the OS and other applications interact and perform. Unfortunately due to GPL issues, DTrace is not available on Linux, only on OS X and other Unix variants.

What Bulmer and his team did, was to extend the Erlang VM with DTrace hooks so that tracing can be done on process creating and other Erlang primitives. The goal is to become a good complement to the internal Erlang tracing and debuggnig facilities.

Building Web Applications in Erlang

A talk about Erlang and web development given by Michal Slask and Xingdong Bian from Erlang Training & Consulting (disclaimer: I work for ETC). It was about the new web development platform that ETC is releasing. It takes its template philosophy from JSP, so that with tags such as list and choose you can call backend Erlang modules expanding to dynamic HTML code. All in a Model View Controller (MVC) fashion.

More about this soon so keep an eye on the Erlang Training & Consulting website and the Erlang mailing list!

Erlang & Ajax Web

Roberto Saccon gived a talk on integrating Ajax technologies with Erlang. The talk focused heavily on Coment and Roberts implementation in Erlang, ErlyComet. The talk is focusing on the implemantion and the results, both mistakes and successes. It is like a overview of both Ajax (for the Erlang community) and a collection of do’s and don’ts when using Ajax/Comet with Erlang. All in all, a web 2.0 presentation with an Erlang flavour.

The Rest of the Conference

This is it so far, maybe more posts will come later. Today it is just the Faxien & Sinan presentation I’m planning to attend.

EazyErl!: Parsing Binaries with erlang, lamers inside

rolphin — 2008-06-26T13:25:39+00:00

It's seems that there's a really high expectation on "parsing binaries" with erlang and reach others languages performance. For me this is a complete nonsense.
What's the meaning of "parsing a binary", a binary is not a text, it's a binary, a sequence of bytes...
A sequence must be defined by its length. You must know before reading anything, the size that'll be needed store what's coming.
Every crap software you can find has always prefered to use strcpy instead of memcpy. Whatever the language you use, you MUST know the size of what you're working with, this is not an advice this is mandatory.

From the post above, you can find that the only delimiter seems to be "\r\n". So if someone sends you 4Gb of data not ending with "\r\n" you'll keep reading it... (and of course blow your memory because this was not supposed to be)

While working at low level with C and flex scanners, I've always ask me this question: "What's the max size of the element I can accept ?". This simple question helps me build software that don't break with a simple 'perl print Ax60000' trick...

So is HTTP badly designed, because delimiters are "\r\n" and headers can spread on multiple lines ? The answer is absolutely YES.
Was'it difficult to build something more secure, using prefixed elements with their size ? The answer is absolutely NO ! (take ajp13 for example...)

Now that erlang is becoming more and more popular, lamers are lurking in the erlang direction. This is life, but will the erlang mailling list suffer from this ? The answer is yes :/

Someone with knowledge must not try to resolve someone's else problem, he must help him by asking the good question. (do you know the size a priori ?)

Why parsing binaries in java is faster than erlang ? Who cares, since parsing binaries is of course stupid !
Parsing real world protocol with erlang is lightning fast, both for writing and for executing. So teach lamers how to build real protocols and don't try help them with some trickery.

That's my rant for today :)

Process-one Blogs: Talk at Erlang Exchange Conference, London, UK on June 27th

2008-06-25T13:44:00+00:00

I will be talking in a London conference about ejabberd XMPP application server and Tsung benchmarking (HTTP and XMPP).

I will be at Erlang Exchange next friday (27th June, 2008), to talk about:

ejabberd for web 2.0 development: We have designed ejabberd as an XMPP application server that happens to be ideal to build web 2.0 applications and social networks. The talk will rely on two majors use cases: building a large scale community chat system (Major League of Baseball) and building a Twitter-like microblogging platform on top of ejabberd.
Tsung tutorial: This Tsung tutorial will present the fundamental concepts underlying Tsung architecture and will provide the required knowledge to write relevant benchmark scenarios. You will then learn how to create a scenario to benchmark an example web application and how to analyse the results of the benchmark.

I would be very glad to meet existing or future ejabberd and Tsung users next friday

More details: My talks and the conference programme.

Joe Armstrong: Invasion of Privacy

Joe Armstrong — 2008-06-24T13:53:06+00:00

On 18 June the Swedish Parliament passed a law giving sweeping new powers to the FRA (Swedish Defense Radio Establishment) allowing them to wiretap people in Sweden through phone conversations, email, text messages and more. All people in Sweden using electronic communication can have their communication monitored despite the fact that they are not suspected of committing any crime. In my view

Hypothetical Labs: Oh Frabjous Day!

kevin — 2008-06-24T10:16:09+00:00

(With humble apologies to Lewis Carroll, my favorite purveyor of kid-safe absurdity)

I’ve finally wrestled frabjous into a form approximating acceptable quality — you can check it out on GitHub here.

So, what is frabjous? It’s a wrapper around Erlang’s parse transforms with some of the sharp edges filed off. Frabjous insulates developers from the tedium of parse transforms in two ways.

First, it interfaces directly with the compiler and handles all of the low-level details around iterating over the AST and other parse transform muck. Frabjous knows all about multi-pass transforms and provides a persistent state mechanism for parse transformers so they can remember where they’ve been and what they’re doing.

Second, frabjous defines a new behavior, ast_transformer, to use when you want to write a frabjous-based parse transformer. Why is this a good thing? Well, ast_transformer defines a number of helpful functions like start/1 and is_complete/1 which model allow a transformer to track its state and implement multiple passes. ast_transformer also defines a standardized calling convention for transforming AST nodes using the function transform/3. No more inventing your own calling convention each time you write a parse transformer.

Frabjous integrates into the Erlang compiler via a header file named “frabjous.hrl“. Just use include_lib to include the header and your code is now frabjous-enabled. Wiring up AST transformers is done using the frabjous attribute.

A picture is worth a thousand words so let’s see what a source file which uses frabjous and its AST transformers looks like. This same is taken from frabjous/examples/src/pt_test.erl.

-module(pt_test).

-define(SERVER, ?MODULE).

-include_lib("frabjous/include/frabjous.hrl").

-frabjous([{remote_call, [{debug, true}]}]).

-behaviour(gen_server).

The example includes frabjous.hrl which “turns on” frabjous. It then goes on to configure a single frabjous AST transformer named remote_call and runs it in debug mode. When frabjous AST transformers are run in debug mode they emit before and after snapshots of the AST. This is indispensable when you’re debugging a transformer, either on its own or when it’s part of a chain of transformers. Yep, frabjous understands how to chain a series of transformers together.

The remote_call transformer takes attribute entries like this:

-call_gen_server([{server, ?SERVER}, {name, add}, {vars, [x,y]}]).

and turns them into functions like this, complete with matching export statements:

add(X, Y) -> gen_server:call(?SERVER, {add, X, Y}).

Why? Because I got really, really, really tired of writing the same message sending wrapper function over and over each time I write a gen_server.

These are still early days for frabjous so the code is a bit rough. There are several things I want to improve soon including:

Unit tests (!!!)
Teaching frabjous how to handle more than the first level of the AST nodes
Adding more functions to frabjous_helper to make AST node generation easier

If you’re interested in frabjous, please check it out and let me know what you think!

Dukes of Erl: Tokyocabinet and Mnesia

Paul Mineiro — 2008-06-23T22:25:59+00:00

As Daisy has already indicated, it is now possible to plug arbitrary storage strategies into Mnesia. For those who are familiar with mnesia_access, this is different; mnesia_access only covers reads and writes, not schema manipulations, and has other deficiencies that it render it useless for adding a new storage type in practice (what mnesia_access is great for is changing the semantics of mnesia operations, e.g., mnesia_frag). This project lets you make tables that are essentially indistinguishable from the built-in mnesia table types (ram_copies, disc_copies, disc_only_copies).

Anyway our goal was to get a good on-disk ordered_set table type, since we've found ordered_sets very useful for the kinds of problems we're solving but we're tired of being limited by memory. After looking around for a while Tokyocabinet emerged as our favorite for underlying implementation. We considered BDB and libmysql, but Tokyocabinet seemed simple and faster, and had a more accommodating license. So we ported Tokyocabinet to Erlang and then used the above storage API to connect to Mnesia.

As a side benefit Tokyocabinet might also be preferred to dets even for set-type applications because of the lack of file size limit and high performance. Tokyocabinet actually has a set-type storage strategy that we'd like to define an Erlang Term Store for, but as of this post, the set-type store doesn't support cursor positioning based upon a key, which makes the implementation of next tedious (although not impossible). So I'm waiting on the author to add that call; if that happens, we could have a nicer on-disk set-type table as well.

Everything is available on google code: mnesiaex (storage API) and tcerl (erlang port of Tokyocabinet).

Also Daisy (aka Joel Reymont) is really nice to work with.

Hypothetical Labs: I Wrote What?

kevin — 2008-06-22T12:59:47+00:00

Late last evening I was finally able to carve out some time to work on frabjous. Frabjous, for those who missed out on my only other post about it, is a parse transform framework for Erlang aimed at making writing parse transforms easier.

But that’s not important right now.

The important part is I was ready to release it last night and decided against it. It’s got unit tests, examples, and nice descriptive comments in all the public functions (and some of the private ones, too). But the code is, well, utter crap.

I wrote the original core of frabjous the day I flew out to Engine Yard’s offices. I figured with a day of air travel and my laptop I could get a lot of hacking done. I did. A lot of hacking but not much designing. The up side is I understand parse transforms really well. The down side is the code is way less organized and efficient than it could be.

Plain and simple, the code isn’t pretty. I have a simple benchmark for code prettiness which works equally well for OOP and functional languages. If I have to stare at the code for more than a minute or two to understand a given function, method, or class, then the code isn’t pretty. That’s it. Think of it as inscrutability quotient. The cleaner the code, aka pretty, the less time I have to spend understanding it. The longer I have to stare before I understand it the less pretty the code becomes. I wrote frabjous and I had several multi-minute “WTF?!” moments.

So, frabjous may or may not be released today. But when it is released it’ll be pretty even if i have to go medieval on it.

EazyErl!: Quick Tip, list join

rolphin — 2008-06-20T10:20:00+00:00

Another "lists:join" :


Join = fun([X|Rest], D) -> 
  [ X | [ [D,E] || E <- Rest ] ] 
  end.

Usage:


io:format("~s~n", [Join(["a", "b", "cde", "h", "k", "lm"], $,)]).
a,b,cde,h,k,lm
ok

Hypothetical Labs: Episode 4 Errata

kevin — 2008-06-19T17:27:41+00:00

See here for the gory details.

*sigh*

A hearty thanks to Jon Gretar for finding it. I’d much rather know about a bug and fix it than let it languish in public.

Also, if you haven’t told me what you’d like to see in the next episodes, please drop a comment on here. Right now I’m leaning towards doing a RESTian API via Yaws or Mochiweb (haven’t decided which), followed by unit testing, and then OTP. But there’s still time to make your case! Comment today!

Hypothetical Labs: Erlang By Example: Episode 4

kevin — 2008-06-18T18:48:49+00:00

Mike Clark and his merry band of busy hamsters have published episode 4 in my screencast series Erlang By Example. The episode is the longest so far, clocking in at an epic 39 minutes. I chose mnesia as the episode’s focus and used it to implement store and forwarding for messages in the chat system. If you’re interested in using mnesia, wondering how to write qlc queries, or just curious what a non-trivial Erlang application looks like, I recommend checking this episode out.

I’m also soliciting comments on what topic I should cover next in the series. The choices and their expected durations are, in no particular order:

Using OTP (2-3 episodes)
Adding a REST endpoint using either Mochiweb or Yaws (2 episodes)
Writing chat clients in another language like Python or Ruby (2-3 episodes)
Clarification: This is meant to illustrate how to interface other languages to Erlang via Erlang’s ports or other libraries such as erlectricity
Unit testing and dialyzer (1 episode)
Insert the burning topic I missed here

Please comment on this post and let me know what you’d like to see next!

EazyErl!: Ubuntu and Ghostscript

rolphin — 2008-06-18T14:38:51+00:00

On my development box, I've recently upgraded my ubuntu. The ghostscript package was also upgraded, but my erlang webservice wasn't able anymore to draw any mobile tag...

I've found that the new ghostscript binary 'gs' has new command line parameters that are incompatible with their previous version...

I used to initialize gs like this:


 Cmd = "gs -sDEVICE=pngalpha -q -dNOPLATFONTS -dNOPAUSE -dGraphicsAlphaBits=2 -sOutputFile=- -",

But that no longer works since this flushing is done only when the process quits...

The correct command line is then:


 Cmd = "gs -sDEVICE=pngalpha -q -dGraphicsAlphaBits=2 -sOutputFile=%stdout -dNOPROMPT",

Everything works fine now, but I've re-read the very long Ghostscript documentation, and the solution comes from this page.

21st Century Code Works: Migrating a native Erlang interface to RESTful Mochiweb (with a bit of TDD)

2008-06-17T06:07:00+00:00

The title is a bit of a mouthful, but it does contain in essence what I will show you:

1. I will convert an existing native erlang CRUD interface (a simple client-server) to use a HTTP layer.
2. The HTTP calls will be RESTful.
3. I will be using Mochiweb.
4. I will show you how to do a bit of Test-Driven-Development (TDD) for this exercise using EUnit.

As an introduction, I suggest you read Steve Vinoski's "RESTful Services with Erlang and Yaws", and "A RESTful web service demo in yaws" by Nick Gerakines. They are both excellent posts, and Nick's post has a lot of interesting details, including some OTP goodness. Also check out Daryn's posts for a Rails twist on this topic.

I will NOT be showing much error handling, simply returning a 501 response if the REST Url has
some mistake in it.

This is the plan:
1. We will start with a working set of tests for a native CRUD interface. I will not show you the actual implementation, but it is simple enough to do yourself using a process dictionary.
2. I will show a very simple REST interface with Mochiweb, that just returns the request method type as a plaintext string. This will be tested using Inets, the native Erlang OTP internet module.
3. We will write a translators to and from plaintext and native erlang terms (again, unit tested).
4. We will combine all these into the REST interface, tested using Inets again.

Right, here is the unit tests for the CRUD interface:

crud_test_() -> 
 {setup,
 fun() -> start() end,
 fun(_) -> stop() end,
 fun(_) ->
 [
 ?_assert(ok == create(#person{id = 1})),
 ?_assert(already_exists == create(#person{id=1})),
 ?_assert(#person{id=1} == retrieve(1)),
 ?_assert(ok == update(#person{id = 1, name="Ben"})),
 ?_assert(#person{id=1, name="Ben"} == retrieve(1)),
 ?_assert(ok == delete(1)),
 ?_assert(undefined == delete(2)),
 ?_assert(undefined == update(#person{id = 2}))
 ]
 end}.

The CRUD interface is container in a module "crud", and start() and stop() are methods to start and stop the CRUD server. For EUnit, you can define a test with a setup, teardown and tests, and this is the form that I've used here. There are many forms of tests for EUnit, and I suggest you consult the EUnit documentation (contained in the EUnit distribution).

EUnit automatically creates a test() function on the module when you include "eunit.hrl", so let's run the crud tests:


75> crud:test().
All 8 tests successful.
ok

We know that are CRUD interface is working OK.

Onto Step 2! If you're not familiar with Mochiweb, it's a lightweight web server developed by the guys at Mochi Media. It's very easy to integrate into you application, and has very little configuration.

Here are the tests for the simple Mochiweb demo:


-define(URL, "http://127.0.0.1:8888").

rest_server_test_() ->
 {setup,
 fun() -> inets:start(), start_simple() end,
 fun(_) -> inets:stop(), stop_simple() end,
 fun(_) ->
 [
 ?_assert(http_result('GET') =:= "GET"),
 ?_assert(http_result('PUT') =:= "PUT"),
 ?_assert(http_result('POST') =:= "POST"),
 ?_assert(http_result('DELETE') =:= "DELETE")
 ]
 end}.

parse_result(Result) ->
 {ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
 ResultBody.

rest_server_test_() ->
 {setup,
 fun() -> inets:start(), start_simple() end,
 fun(_) -> inets:stop(), stop_simple() end,
 fun(_) ->
 [
 ?_assert(http_result('GET') =:= "GET"),
 ?_assert(http_result('PUT') =:= "PUT"),
 ?_assert(http_result('POST') =:= "POST"),
 ?_assert(http_result('DELETE') =:= "DELETE")
 ]
 end}.

start_simple() starts the simple version of the server and stop_simple() stops it. We're just using the base URL, and the server just returns a plain-text string of the request method. The http:request() functions are part of Inets (documentation here). You will notice that the put and post functions have extra parameters (which we will use later for the request body).

Here's the solution:


start_simple() ->
 mochiweb_http:start(
 [{ip, "127.0.0.1"},
 {loop, {?MODULE, simple_response}}]).
stop_simple() ->
 mochiweb_http:stop().
 

simple_response(Req, Method) ->
 Req:ok({"text/plain", atom_to_list(Method)}).
simple_response(Req) ->
 simple_response(Req, Req:get(method)).

We've started Mochiweb, and told it that the "simple_response" function in the current module should be used to handle requests. It takes one parameter, which contains the request data. The data can be queried with different methods to extract data from it, e.g. Req:get(method), gives you the method used.

The Req:ok() function is used to respond to a request, and we simply return plain text (with the text/plain MIME type).

And let's run the tests:


77> rest_server:test().
...
All 4 tests successful.
ok

Nice. The next bit of code we need is to translate Erlang terms to and from plain text. We'll use Base64 encoding for this. Erlang also has very hand term to binary and binary to term functions that we can use to achieve the translation.

The tests for the translation:


term_to_plaintext_test_() ->
 A = anatom,
 B = {a, 23, "abc"},
 C = {props, [{a,3},{b,4}]},
 [
 ?_assert(A == ptt(ttp(A))),
 ?_assert(B == ptt(ttp(B))),
 ?_assert(C == ptt(ttp(C)))
 ].

Where "ttp" would be "term_to_plaintext" and "ptt" is
"plaintext_to_term". Here's the implementation:


ttp(Term) ->
 base64:encode_to_string(term_to_binary(Term)).

ptt(PlainText) ->
 binary_to_term(base64:decode(PlainText)).

And the test result:


77> rest_server:test().
...
All 7 tests successful.
ok

(7 tests, since we have the first 4 and now the extra 3). Now things are getting a bit more complicated. We now have to ensure that the response to the request is converted from the native Erlang terms to plaintext, this result is then returned, and we also need a function to convert the request result into the native form.

Here we go:


result_to_terms(Result) ->
 {ok, {{_Version, 200, _ReasonPhrase}, _Headers, ResultBody}} = Result,
 ptt(ResultBody).

rest_api('GET', Path) ->
 Result = http:request(get, {?URL ++ Path, []}, [], []),
 result_to_terms(Result);
rest_api('DELETE', Path) ->
 Result = http:request(delete, {?URL ++ Path, []}, [], []),
 result_to_terms(Result).

rest_api('POST', Path, Data) ->
 Result = http:request(post, {?URL ++ Path, [], [], ttp(Data)}, [], []),
 result_to_terms(Result);
rest_api('PUT', Path, Data) ->
 Result = http:request(put, {?URL ++ Path, [], [], ttp(Data)}, [], []),
 result_to_terms(Result).

crud_server_test_() ->
 {setup,
 fun() -> crud:start(), inets:start(), start() end,
 fun(_) -> stop(), inets:stop(), crud:stop() end,
 fun(_) ->
 [
 ?_assert(ok == rest_api('POST', "/person", #person{id = 1})),
 ?_assert(already_exists == rest_api('POST', "/person", #person{id = 1})),
 ?_assert(#person{id=1} == rest_api('GET', "/person/1")),
 ?_assert(ok == rest_api('PUT', "/person/1", #person{name="Ben"})),
 ?_assert(#person{id=1, name="Ben"} == rest_api('GET', "/person/1")),
 ?_assert(ok == rest_api('DELETE', "/person/1")),
 ?_assert(undefined == rest_api('DELETE', "/person/1")),
 ?_assert(undefined == rest_api('PUT', "/person/1", #person{id = 2}))
 ]
 end}.

There is a bit of duplication here, but for illustration I've kept the functions seperate. You will notice that the data is converted to plaintext for the PUT and POST requests. result_to_terms() converts the HTTP result from the plain text to the native Erlang terms.

You can scroll back to the original CRUD tests, and notice that the tests do exactly the same thing as on the CRUD interface, but we would have to do a "GET" + "/person/1" to get person 1, instead of doing a crud:retrieve(1).

The mapping between the CRUD and REST method are as follows:
GET <=> RETRIEVE
POST <=> CREATE
PUT <=> UPDATE
DELETE <=>DELETE

Here's the final product:


start() ->
 mochiweb_http:start(
 [{ip, "127.0.0.1"},
 {loop, {?MODULE, crud_response}}]).
stop() ->
 mochiweb_http:stop().


crud_response(Req, 'GET', "/person/" ++ IdString) ->
 Id = list_to_integer(IdString),
 Response = crud:retrieve(Id),
 Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'DELETE', "/person/" ++ IdString) ->
 Id = list_to_integer(IdString),
 Response = crud:delete(Id),
 Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'POST', "/person") ->
 Body = Req:recv_body(),
 Person = ptt(Body),
 Response = crud:create(Person),
 Req:ok({"text/plain", ttp(Response)});

crud_response(Req, 'PUT', "/person/" ++ IdString) ->
 Id = list_to_integer(IdString),
 Body = Req:recv_body(),
 PersonWithNewValues = ptt(Body),
 UpdatedPerson = #person{id = Id, 
 name = PersonWithNewValues#person.name, 
 email_address = PersonWithNewValues#person.email_address},
 Response = crud:update(UpdatedPerson),
 Req:ok({"text/plain", ttp(Response)});
 
crud_response(Req, _Method, Path) ->
 Req:respond({501, [], Path}).

crud_response(Req) ->
 crud_response(Req, Req:get(method), Req:get(path)).

Once again, there is some duplication, but it's easier to grasp when we split the functions. The GET and DELETE requests are simple to handle, since we just convert the Id string to an integer (which could throw an exception, you would have to handle that in some way), and call the CRUD interface.

POST is not much more complicated, we just get the body of the request using Req:recv_body().

The PUT function has a problem. There is duplication of the Id of the person, in both the URL "/person/1", and the actual term, #person{id=1,...}. I'm not sure how to handle this, comments are welcome. Perhaps you can generate an error response if the Ids don't match. Or you can make sure the posted record does NOT contain an Id field, and return an error if it does.

The solution as I've given it, uses the fields in the record, and ignores the Id of the record, using the Id in the URL instead.

The last function generates an error if the request could not be matched.

And the moment of truth:

81> rest_server:test().
...
All 15 tests successful.
ok

Looks so simple now, but I can asure you that it took some effort to get all the tests to pass!

Well I hope I've shown you something that you didn't know before, or even encouraged you to learn some Erlang.

Comments and criticisms are most welcome :)

Dukes of Erl: generating app files

Paul Mineiro — 2008-06-15T13:46:46+00:00

We automatically generate our .app files around here from analysis of the source code. However this trick is buried inside fw-template-erlang which most (all?) people outside our group do not use. I thought I'd highlight how it works.

A typical OTP application specification file looks like:


{ application, walkenfs,
[
  { description, "Distributed filesystem." },
  { vsn, "0.1.7" },
  { modules, [ walkenfssup, walkenfsfragmentsrv, walkenfs, walkenfssrv ] },
  { registered, [ walkenfsfragmentsrv ] },
  { applications, [ kernel, stdlib   ] },
  { mod, { walkenfs, [] } },
  { env, [ { linked_in, false }, { mount_opts, "" }, { mount_point, "/walken" }, { prefix, walken }, { read_data_context, async_dirty }, { write_data_context, sync_dirty }, { read_meta_context, transaction }, { write_meta_context, sync_transaction }, { attr_timeout_ms, 0 }, { entry_timeout_ms, 0 }, { reuse_node, true }, { start_timeout, infinity }, { stop_timeout, 1800000 }, { init_fragments, 7 }, { init_copies, 3 }, { copy_type, n_disc_only_copies }, { buggy_ets, auto_detect }, { port_exit_stop, true } ] }

]
}.

There are alot of bits here that can filled in automatically. It helps to think in terms of the template:


{ application, @FW_PACKAGE_NAME@,
[
  { description, "@FW_PACKAGE_SHORT_DESCRIPTION@" },
  { vsn, "@FW_PACKAGE_VERSION@" },
  { modules, [ @FW_ERL_APP_MODULES@ ] },
  { registered, [ @FW_ERL_APP_REGISTERED@ ] },
  { applications, [ kernel, stdlib @FW_ERL_PREREQ_APPLICATIONS@ ] },
  @FW_ERL_APP_MOD_LINE@
  { env, @FW_ERL_APP_ENVIRONMENT@ }
  @FW_ERL_APP_EXTRA@
]
}.

This is an "automake" style template where @VARIABLE@ indicates something that will be substituted.

Now we can say where each of these templates values comes from when using fw-template-erlang:

FW_PACKAGE_NAME: specified by the developer. if you're using automake you've had to specify this already via AC_INIT, so it's good to reuse that rather than require duplicate specification.
FW_PACKAGE_SHORT_DESCRIPTION: specified by the developer.
FW_PACKAGE_VERSION: specified by the developer; like the package name, if you're using automake this information is already contained in the arguments to AC_INIT so reuse that.
FW_ERL_APP_MODULES: generated by analysis of the source code directory.
FW_ERL_APP_REGISTERED: generated by analysis of the source code directory.
FW_ERL_PREREQ_APPLICATIONS: specified by the developer.
FW_ERL_MOD_LINE: partially generated by analysis of the source code directory, partially specified by the developer (see below).
FW_ERL_APP_ENVIRONMENT: specified by the developer.
FW_ERL_APP_EXTRA: specified by the developer. this is for unusual extra directives like included applications.

So the interesting bits come down to FW_ERL_APP_MODULES, FW_ERL_APP_REGISTERED, and FW_ERL_MOD_LINE being generated automatically via inspection of the source code.

First, a note on intervention. Anytime you are attempting to automate a task that was previous done by humans, it's helps to allow a human to override any of the automatic settings that you generate by default. That way, you can be correct only 95% of the time and still be a huge timesaver. In practice we've found that although it easy to come up with code examples that flummox the automatic strategies, such code is never actually written by people in the normal course of their work. In fact, we've found that application files we get from the universe are often missing registered process names that our automatic strategy finds. So empirically we're running at 100% correct so far, but fw-template-erlang still contains the capability to override any automatically generated value.

Now the actual trick to computing all the stuff is that Erlang contains an Erlang compiler as a library. Here's an example of leveraging this to generate all the module names from a set of files; any file that contains a

-fwskip()

directive will be ignored.


-module (find_modules).
-export ([ main/1 ]).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

print_module (Dir, F) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, Mod, { Mod, _, Forms } } ->
case is_skipped (Forms) of
  true ->
    ok;
  false ->
    port_command (get (three), io_lib:format ("~p~n", [ Mod ]))
end;
_ ->
ok
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
  Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
  put (three, Three),
  lists:foreach (fun (F) -> print_module (Dir, F) end, Rest)
after
  { ok, FileNames } = file:list_dir (Dir),
  lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
  file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

The output is on file descriptor 3 in order to isolate the desired output from various messages being output by the Erlang code. This was originally an escript but to be compatible with older versions of Erlang we changed it to run as follows:


#! /bin/sh

# NB: this script is called find-modules.sh

# unfortunately, escript is a recent innovation ...

ERL_CRASH_DUMP=/dev/null
export ERL_CRASH_DUMP

erl -pa "${FW_ROOT}/share/fw/template/erlang/" -pa "${FW_ROOT}/share/fw.local/template/erlang" -pa "${top_srcdir}/fw/template/erlang/" -pa "${top_srcdir}/fw.local/template/erlang" -noshell -noinput -eval '
find_modules:main (init:get_plain_arguments ()),
halt (0).
' -extra "$@" 3>&1 >/dev/null

In practice this gets executed like


find src -name '*.erl' -print | xargs find-modules.sh "$tmpdir" |
perl -ne 'chomp;
     next if $seen{$_}++;
     print ", " if $f;
     print $_;
     $f = 1;'

Ok, that was the easy one! In fact, the problem of finding all the module names seems so easy that one could be tempted to solve it with grep (hmmm: what about preprocesser directives ?). However the point is that nothing parses Erlang as well as Erlang, so why reinvent the wheel.

Computing the mod line in the application specification is just slightly harder. Basically we scan source code for a module which has a

-behaviour(application)

attribute and assume that is the start module. The Erlang compiler allows behaviour to be (mis)spelt American-style so we allow that as well.


-module (find_start_module).
-export ([ main/1 ]).

is_application ([]) -> false;
is_application ([ { attribute, _, behaviour, [ application ] } | _ ]) -> true;
is_application ([ { attribute, _, behavior, [ application ] } | _ ]) -> true;
is_application ([ _ | Rest ]) -> is_application (Rest).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

find_start_module (_, []) ->
ok;
find_start_module (Dir, [ F | Rest ]) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, Mod, { Mod, _, Forms } } ->
  case is_application (Forms) and not is_skipped (Forms) of
    true ->
      port_command (get (three), io_lib:format ("~p~n", [ Mod ]));
    false ->
      find_start_module (Dir, Rest)
  end;
_ ->
  find_start_module (Dir, Rest)
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
put (three, Three),
find_start_module (Dir, Rest)
after
{ ok, FileNames } = file:list_dir (Dir),
lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

If there are no applications in the set of files being analyzed, we output nothing for the mod line in the application file, otherwise we output

{ mod, { @FW_ERL_APP_START_MODULE@, @FW_ERL_APP_START_ARGS@ } },

where FW_ERL_APP_START_MODULE is the output of the above command and FW_ERL_APP_START_ARGS is indicated by the programmer (we use [] by default for FW_ERL_APP_START_ARGS and in practice never use the field). Obviously, if multiple modules implementing the application behaviour are found then this will not work. Our style has been to put each application we make in a separate directory to facilitate automatic analysis.

Anyway now we can tackle the more challenging problem of finding registered processes. There are many library calls that end up registering a process name; the ones we recognize are:

supervisor child specs that contain calls of the form

{ _, start, [ { local, xxx }, ... ] } -> xxx being registered
{ _, start_link, [ { local, xxx }, ... ] } -> xxx being registered

function calls of the form

Module:start ({ local, xxx }, ...) -> xxx being registered
Module:start_link ({ local, xxx }, ...) -> xxx being registered

calls to erlang:register (xxx, ...) -> xxx being registered

There's alot of code here but the general idea is the same: walk the forms generated by compile and pattern match on one of the above cases, outputting discovered registered process names to file descriptor 3.


-module (find_registered).
-export ([ main/1 ]).

%% looks like fun_clauses are the same as clauses (?)
print_registered_fun_clauses (Clauses) ->
print_registered_clauses (Clauses).

%% looks like icr_clauses are the same as clauses (?)
print_registered_icr_clauses (Clauses) ->
print_registered_clauses (Clauses).

print_registered_inits (Inits) ->
lists:foreach (fun ({ record_field, _, _, E }) ->
               print_registered_expr (E);
                 (_) ->
               ok
             end,
             Inits).

print_registered_upds (Upds) ->
lists:foreach (fun ({ record_field, _, _, E }) ->
               print_registered_expr (E);
                 (_) ->
               ok
             end,
             Upds).

% hmmm ... pretty sure patterns are not supposed to have side-effects ...
print_registered_pattern (_) -> ok.
print_registered_pattern_group (_) -> ok.

print_registered_quals (Qs) ->
lists:foreach (fun ({ generate, _, P, E }) ->
               print_registered_pattern (P),
               print_registered_expr (E);
                 ({ b_generate, _, P, E }) ->
               print_registered_pattern (P),
               print_registered_expr (E);
                 (E) ->
               print_registered_expr (E)
             end,
             Qs).

print_registered_expr ({ cons, _, H, T }) ->
print_registered_expr (H),
print_registered_expr (T);
print_registered_expr ({ lc, _, E, Qs }) ->
print_registered_expr (E),
print_registered_quals (Qs);
print_registered_expr ({ bc, _, E, Qs }) ->
print_registered_expr (E),
print_registered_quals (Qs);

%% Ok, here's some meat:
%% the "supervisor child spec" rule
%% { _, start, [ { local, xxx }, ... ] } -> xxx being registered
%% { _, start_link, [ { local, xxx }, ... ] } -> xxx being registered
print_registered_expr ({ tuple, _,
                     Exprs=[ _,
                       { atom, _, Func },
                       { cons,
                         _,
                         { tuple, _, [ { atom, _, local },
                                       { atom, _, Name } ] },
                         _ } ] }) when (Func =:= start) or
                                       (Func =:= start_link) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_exprs (Exprs);

print_registered_expr ({ tuple, _, Exprs }) ->
print_registered_exprs (Exprs);
print_registered_expr ({ record_index, _, _, E }) ->
print_registered_expr (E);
print_registered_expr ({ record, _, _, Inits }) ->
print_registered_inits (Inits);
print_registered_expr ({ record_field, _, E0, _, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ record, _, E, _, Upds }) ->
print_registered_expr (E),
print_registered_upds (Upds);
print_registered_expr ({ record_field, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ block, _, Exprs }) ->
print_registered_exprs (Exprs);
print_registered_expr ({ 'if', _, IcrClauses }) ->
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'case', _, E, IcrClauses }) ->
print_registered_expr (E),
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'receive', _, IcrClauses }) ->
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'receive', _, IcrClauses, E, Exprs }) ->
print_registered_icr_clauses (IcrClauses),
print_registered_expr (E),
print_registered_exprs (Exprs);
print_registered_expr ({ 'try', _, Exprs0, IcrClauses0, IcrClauses1, Exprs1 }) ->
print_registered_exprs (Exprs0),
print_registered_icr_clauses (IcrClauses0),
print_registered_icr_clauses (IcrClauses1),
print_registered_exprs (Exprs1);
print_registered_expr ({ 'fun', _, Body }) ->
case Body of
  { clauses, Cs } ->
    print_registered_fun_clauses (Cs);
  _ ->
    ok
end;

%% Ok, here's some meat:
%% Module:start ({ local, xxx }, ...) -> xxx being registered
%% Module:start_link ({ local, xxx }, ...) -> xxx being registered

print_registered_expr ({ call,
                     _,
                     E={ remote, _, _, { atom, _, Func } },
                     Exprs=[ { tuple, _, [ { atom, _, local },
                                           { atom, _, Name } ] } | _ ] })
                        when (Func =:= start) or
                             (Func =:= start_link) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_expr (E),
print_registered_exprs (Exprs);

%% Ok, here's some meat:
%% erlang:register (xxx, ...) -> xxx being registered

print_registered_expr ({ call,
                     _,
                     { remote,
                       _,
                       { atom, _, erlang },
                       { atom, _, register } },
                     Exprs=[ { atom, _, Name } | _ ] }) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_exprs (Exprs);

print_registered_expr ({ call, _, E, Exprs }) ->
print_registered_expr (E),
print_registered_exprs (Exprs);
print_registered_expr ({ 'catch', _, E }) ->
print_registered_expr (E);
print_registered_expr ({ 'query', _, E }) ->
print_registered_expr (E);
print_registered_expr ({ match, _, P, E }) ->
print_registered_pattern (P),
print_registered_expr (E);
print_registered_expr ({ bin, _, PatternGrp }) ->
print_registered_pattern_group (PatternGrp);
print_registered_expr ({ op, _, _, E }) ->
print_registered_expr (E);
print_registered_expr ({ op, _, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ remote, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr (_) ->
ok.

print_registered_exprs (Exprs) ->
lists:foreach (fun (E) -> print_registered_expr (E) end, Exprs).

print_registered_clauses (Clauses) ->
lists:foreach (fun ({ clause, _, _, _, Exprs }) ->
                 print_registered_exprs (Exprs);
                   (_) ->
                 ok
               end,
               Clauses).

print_registered_forms (Forms) ->
lists:foreach (fun ({ function, _, _, _, Clauses }) ->
                 print_registered_clauses (Clauses);
                   (_) ->
                 ok
               end,
               Forms).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

print_registered (Dir, F) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, _, { _, _, Forms } } ->
  case is_skipped (Forms) of
    true ->
      ok;
    false ->
      print_registered_forms (Forms)
  end;
_ ->
  ok
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
put (three, Three),
lists:foreach (fun (F) -> print_registered (Dir, F) end, Rest)
after
{ ok, FileNames } = file:list_dir (Dir),
lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

It's definitely possible to fool the above code. For instance:


Hidden = therealname,
erlang:register (Hidden, SomePid).

However in practice developers tend to use either explicitly name their registered process or at worst obscure them with a macro (which, by going through the Erlang compiler, we get expanded for free), so this automatic strategy has been working very well.

You can download fw-template-erlang from google code and get the complete source code from above plus some intuition about how it is called. One thing that's been on our TODO list is to combine all these analysis into one pass on the source code; currently we pass through the source code three times, which is clearly suboptimal, although the process is speedy enough that it is only noticeable when analyzing larger projects (like, when we converted yaws and mnesia to fw-template-erlang for our internal use).

Web 2.0 and beyond: Javascript syntax highligting for Firebug

Roberto Saccon — 2008-06-14T14:41:35+00:00

The syntax highligting capabilities of CodeMirror, my favorite Javascript based code editor (for Web IDE-type applications) found their way into Firebug, currently available as additional extension.

Dukes of Erl: Erlang is so clear.

RoscoeOTPColtrain — 2008-06-13T11:50:50+00:00

Ah the clearness of erlang.

Someone on the yaws list gets errors and asks:

Can any one please explain me what this error is about and how to
overcome this one?

and gets the reply

Well - it's fairly clear. It says:

{undef,[{compile,file,
["/home/phanikar/.yaws/yaws/default/m1.erl",
[binary,return_errors, ...

That means that when .yaws page has been transformed into an
.erl page, the yaws server cannot compile the generated .erl file
You need to have the erlang compiler in the load path

I'm going to assume the author was being sarcastic about that being clear, I mean I get that it couldn't compile something.. but.. no hint of what reasons it couldn't compile.

maybe to quote the awesome tude of of the dynamic window manager guys:

This keeps its userbase small and elitist. No novices asking stupid questions.

Well despite some of these problems erlang is super awesome. It just needs a little more new blood to inject new attitudes of usability [ or some other language will evbentually just steal it's good ideas, which would be a shame.. we have them now. ] Also new attitudes around easier sharing of libraries. [ the erlware guys get that. ].

I'm pretty proud to see Uncle Jesse and the gang here trying to contribute improvements all the time. Better mnesia on disk tables in the works, yaws and erlang patches, erlrc, automatic mnesia fragmentation mangment and loads more. I mean we're building a big and real distributed mnesia backed system so we're learning a lot, and we're certainly not the only ones contributing lately.

So, as a novice who asks stupid questions, I'm telling the rest of you out there not to give up if you run into a few weird error messages or attitudes from language scholars who have been so entrenched they don't see what seems weird at first. Despite those few small speed bumps, there is some really powerful awesome stuff going on here. So, don't let em be elitist barge in there and get some erlang for yourself.

P.S. Pattern matching rules!!!

EazyErl!: Quick Bash Script for Checking HTTP Headers

rolphin — 2008-06-13T11:09:07+00:00

Whenever it comes to efficiently configure Web Servers and setting headers like ETAg, Cache-Control, or Expires and you have more than one server, you need to check them all since any user may hit different servers.

This script is meant to call wget on every IP returned by dig and display http response headers.
Usage is simple:


./script.sh http://www.example.com/ressource/with-long-cache-and-no-etag/big-image.jpg

Here's the code:


#!/bin/bash

BIN=${0##*/}
URL=${1?usage: $BIN url}
HOST=${URL#http://*}
REQ=${HOST#*/}
HOST=${HOST%%/*}

function check
{
	wget -S --header="Host: $HOST" "http://$line/$REQ" -O/dev/null
}

dig +short $HOST | \
while read line
do
	case $line in	
		[a-z]*)
		;;
		[0-9]*.[0-9]*.[0-9]*)
			echo === Testing $HOST with IP $line
			check $line
		;;
		*)
		;;
	esac
done

Trapexit's Erlang Blog Filter: Back from JAOO Au

2008-06-12T05:16:55+00:00

I recently returned from giving two talks at each of the Brisbane and Sydney JAOO conferences. JAOO has long been my favorite conference, and even though this was the first time JAOO was held in Australia, my understanding is that overall it was quite a success. There were a couple hundred attendees in each city, and I heard a number of attendees echo something along the lines of “we really need developer conferences like this that aren’t marketing and aren’t academic.” That’s precisely why I like JAOO so much — almost all of the JAOO speakers are practitioners themselves, so as a developer you get a lot of well-grounded useful info rather than product pitches or unproven research.

My talks were “Building RESTful Services with Erlang and Yaws” and “Multilanguage Programming.” For first-time talks both went OK, but they could have been better. The biggest problem with both was that I already had too many slides for each one for an hour but when I got there I learned that a scheduling mix-up had reduced the presentation time slots to only 45 minutes, which then meant I had way too many slides for each talk. The first talk tries to cover a lot of ground, for example, and it’s hard to remove any of it and still have it make sense. Oh well. I warned the attendees up front about the problem, and since I got reasonable reviews for both talks I hope that means I left the attendees with some useful info.

There’s talk in the blogosphere of a strong revival in interest in programming languages, and I’d have to say there’s definitely something to that notion because I didn’t expect so many would be interested in the multilanguage programming talk. I guess I expected to encounter some defensive dyed-in-the-wool “X” programmers, where “X” often seems to be Java, who would just argue that multilanguage programming is a bad idea, but there were none. Instead I was encouraged by the number of developers who came to the talk and seemed truly interested in the idea of learning and using multiple languages in order to do their jobs in the best possible way.

Anyway, I strongly recommend that you go to JAOO Aarhus if you can.

Next up: Erlang eXchange in London, which I also strongly recommend. Overall it looks like a very strong program, and I’m really looking forward not only to giving my “Enterprise Integration” keynote there, but better yet getting to meet more folks in the Erlang community in person. I hope I’ll see you there.

Ulf Wiger: Indentation-sensitive Erlang 3

2008-06-11T14:13:12+00:00

So, maybe I’m thick enough not to realise in advance when I’m barking up the wrong tree, or perhaps I just like to follow things through to the bitter end, just to know what exactly didn’t work…?

I’ve gone one more round with my indentation-sensitive Erlang scanner/parser. For a while, it looked like I was winning, but eventually, I had to admit defeat in the face of funs and record constructors.

Again, the approach I thought I’d take was to add indentation tokens to the normal scanner, and then add rule clauses to the normal grammar to make it understand both indentation-sensitive code, and all the code you’re accustomed to writing. The normal parser is an LR(1) grammar, which means that the additions have to be quite symmetrical in order to work.

This seemed to work in just about all the places that mattered, but I was stumped by funs and record constructors. The main with funs was that you could no longer write them in the conventional way (failing one of my preconditions), and record constructors usually cannot be written without including a few commans in a way that was …uhm, less than obvious.

I include the test module, which pretty much illustrates what works, and what doesn’t. At this point, I doubt that I can achieve better than a 90% solution, which is probably just good enough to eventually drive people crazy.

It’s been fun, though. I’m have no particular craving for indentation-sensitive syntax myself, so I thought I’d tackle this as a learning experience. Not for the first time, I feel like concluding that retrofitting concepts onto existing programming languages is usually very difficult to do well.

The code is at http://svn.ulf.wiger.net/indent/branches/0.3

-module(test).

-record(r,{a,b}).

-compile(export_all).
-scan(indentation).

f(X) ->
    X+2

g(X) ->
    X+4
.          % dot can either be 'outdented' or
           % terminating last line

g1(X) ->
    X+4.

h(X) ->
    Y = case X of
          a ->
            {a}
          b ->
            {b}
    end          % end is optional
    Y

h1(X) ->
    Y = case X of
          a ->
            {a}
          b ->
            {b}
    Y

h2(X) ->
    case X of
        a ->
            1
        b ->
            2

h3(X) ->
    case X of
        a -> 1;  % must use semicolon here 
        b -> 2

i(a) -> a
i(b) -> b

j(A,B) ->
    {A
    B}

j1(A,B,C) ->
    {A
     B     % indents must be greater than 1;
     C}    % else they count as aligned

k() ->
    S = "a string "
        "spanning multiple "
        "lines"
    {S}

l0() ->
    fun(1) -> 1; (2) -> 2 end.

l1() ->
    fun            % must break line here and indent.  
      (1) ->
            1
      (2) ->
            2

%%% This, alas, doesn't work 
%%%
%%% l2() ->
%%%     fun(1) ->
%%%             1;
%%%        (2) ->
%%%             2
%%%     end

m() ->
    #r{a = 1, b = 2}.

m1() ->
    R = #r{a = 1}
    R#r.a

%%% indentation syntax works poorly for record assignment.
%%% Both commas are needed 
m2() ->
    R = #r{a = 1,
           b = 2},
    R

test() ->
    2 = f(0),
    4 = f(2),
    4 = g(0),
    4 = g1(0),
    8 = g(4),
    {a} = h(a),
    {b} = h(b),
    {a} = h1(a),
    {b} = h1(b),
    1 = h2(a),
    2 = h2(b),
    1 = h3(a),
    2 = h3(b),
    a = i(a),
    b = i(b),
    {a,b} = j(a,b),
    {a,b,c} = j1(a,b,c),
    {"a string spanning multiple lines"} = k(),
    F0 = l0(), 1 = F0(1), 2 = F0(2),
    F1 = l1(), 1 = F1(1), 2 = F1(2),
    {r,1,2} = m(),
    1 = m1(),
    ok.

Dukes of Erl: I'm in ur Erlangz, upgrading ur applicationz

Paul Mineiro — 2008-06-11T12:37:08+00:00

Hot code loading is one of the sexiest features of Erlang; after all, who wants downtime? If you're running mnesia, it's also very convenient, because as we've learned mnesia does not like to be shut down (more precisely, it makes no attempt to incrementally replicate changes across brief outages, instead doing full table (re)copying; something probably nobody cares about because they never turn it off).

Erlang's hot code loading support consists of low level language and runtime features, and a high level strategy embodied by release handler. The latter is a very Java-esque strategy of bundling up the complete new state of the system and upgrading everything. Without being normative let's just say that we enjoy using our OS package manager to manage the software on our boxes, and so we were interested in integrating Erlang hot code loading with the OS package manager.

The result is erlrc, which when operating looks something like this:

pmineiro@ub32srvvmw-199% sudo apt-get install egerl
Reading package lists... Done
Building dependency tree    
Reading state information... Done
The following packages will be upgraded:
egerl
1 upgraded, 0 newly installed, 0 to remove and 34 not upgraded.
Need to get 0B/113kB of archives.
After unpacking 0B of additional disk space will be used.
WARNING: The following packages cannot be authenticated!  egerl
Install these packages without verification [y/N]? y
(Reading database ... 48221 files and directories currently installed.)
Preparing to replace egerl 4.0.1 (using .../archives/egerl_4.1.0_all.deb) ...
Unpacking replacement egerl ...
erlrc-upgrade: Upgrading 'egerl': (cb8eec1a1b85ec017517d3e51c5aee7b) upgraded
Setting up egerl (4.1.0) ...
erlrc-start: Starting 'egerl': (cb8eec1a1b85ec017517d3e51c5aee7b) already_running

Basically, I issued an apt command, the result was to hot upgrade the corresponding Erlang application on the running Erlang node on the machine.

In addition to integration with the OS package manager, erlrc also features an extensible boot procedure (basically, using a boot directory instead of a boot file to determine what to start, which makes packaging easier); and in addition erlrc will automatically generate appup files which correspond to most of the common cases outlined in the appup cookbook. Of course, Erlang packages made with fw-template-erlang are automatically instrumented for erlrc (but the instrumentation is inert if erlrc is not installed).

If you're interested, there's more extensive documentation on google code.

Caoyuan's Blog: Parsing Performance of Scala for NetBeans

2008-06-11T03:30:33+00:00

I'm re-considering the indexing mechanism of Scala for NetBeans. The indexing is used to store meta-info of parsed templates/methods etc, for auto-completion and document/source offsets searching. Currently, the parsing phases include lexer, syntax and semantic analysis, not include type inference and type check.

With a basic performance testing on all Scala standard library and liftweb's library, the maxima parsing time seems less than 1s, the average parsing time is around 0.2s, not bad.

So, this may let the indexing feature a lot simple, I can store classes/objects/traits' meta-info only, instead of including their type parameters and their members (fields/methods, scoped importing etc), these additional information can be got via re-parsing the source file or querying the class file.

Philip Robinson: "Dynamic" module generation with compile-time macros

Philip Robinson — 2008-06-11T00:26:06+00:00

Over on the Erlang Questions mailing list, Jacob Perkins asked:

"I have a few files that are basically lists of words. What I want to do is generate a module whose functions will return each list of words. So if I have two files named "adjectives" and "nouns", then the generated module (let's call it 'grammar') should have two functions, adjectives() and nouns(), that return their respective list of words.

How can I do this?"

If I had some method of running Erlang code at compile-time - and I do - then I could write a program to slurp in those adjective and noun files and generate the appropriate functions.

adjectives.txt:


this
that
these

nouns.txt:


apple
banana
carrot

grammar.erl:


-module(grammar).
-export([nouns/0, adjectives/0]).
-compile({parse_transform, emp2}).

-macro({grammar_macro, generate, [nouns]}).
-macro({grammar_macro, generate, [adjectives]}).

grammar_macro.erl:


-module(grammar_macro).
-export([generate/1]).

generate(NameFun) ->
    {ok,Fd} = file:open(atom_to_list(NameFun) ++ ".txt", [read]),
    generate(Fd, NameFun, []).

generate(Fd, NameFun, Words) ->
    case io:get_line(Fd, "") of
        eof ->
            ok=file:close(Fd),
            io_lib:format("~w() -> ~w.~n", [NameFun, lists:reverse(Words)]);
        Word -> generate(Fd, NameFun, [Word | Words])
        end.

And in the Erlang shell:


1> c(grammar_macro), c(grammar).
{ok,grammar}
2> grammar:nouns().
["apple\n","banana\n","carrot\n"]
3> grammar:adjectives().
["this\n","that\n","these\n"]

Programming in the 21st Century: A Spellchecker Used to Be a Major Feat of Software Engineering

James Hague — 2008-06-08T06:00:00+00:00

Here's the situation: it's 1984, and you're assigned to write the spellchecker for a new MS-DOS word processor. Some users, but not many, will have 640K of memory in their PCs. You need to support systems with as little as 256K. That a quarter megabyte to contain the word processor, the document being edited, and the memory needed by the operating system. Oh, and the spellchecker.

For reference, on my MacBook, the standard dictionary in /usr/share/dict/words is 2,486,813 bytes and contains 234,936 words.

An enticing first option is a data format that's more compressed than raw text. The UNIX dictionary contains stop and stopped and stopping, so there's a lot of repetition. A clever trie implementation might do the trick...but we'll need a big decrease to go from 2+ megabytes to a hundred K or so.

In fact, even if we could represent each word in the spellchecker dictionary as a single byte, we need almost all the full 256K just for that, and of course the single byte representation isn't going to work. So not only does keeping the whole dictionary in RAM look hopeless, but so does keeping the actual dictionary on disk with only an index in RAM.

Now it gets messy. We could try taking a subset of the dictionary, one containing the most common words, and heavily compressing that so it fits in memory. Then we come up with a slower, disk-based mechanism for looking up the rest of the words. Or maybe we jump directly to a completely disk-based solution using a custom database of sorts (remembering, too, that we can't assume the user has a hard disk, so the dictionary still needs to be crunched onto a 360K floppy disk).

On top of this, we need to handle some other features, such as the user adding new words to the dictionary.

Writing a spellchecker in the mid-1980s was a hard problem. Programmers came up with some impressive data compression methods in response to the spellchecker challenge. Likewise there were some very clever data structures for quickly finding words in a compressed dictionary. This was a problem that could take months of focused effort to work out a solution to. (And, for the record, reducing the size of the dictionary from 200,000+ to 50,000 or even 20,000 words was a reasonable option, but even that doesn't leave the door open for a naive approach.)

Fast forward to today. A program to load /usr/share/dict/words into a hash table is 3-5 lines of Perl or Python, depending on how terse you mind being. Looking up a word in this hash table dictionary is a trivial expression, one built into the language. And that's it. Sure, you could come up with some ways to decrease the load time or reduce the memory footprint, but that's icing and likely won't be needed. The basic implementation is so mindlessly trivial that it could be an exercise for the reader in an early chapter of any Python tutorial.

That's progress.

Web 2.0 and beyond: Google App Engine limitations and workarounds

Roberto Saccon — 2008-06-06T13:11:28+00:00

One of the current limitations of the Google App Engine is the lack of scheduled background processes. But early adapters are creative to circumnavigate this and other limitations. The most simple solution to this problem has William Vambenepe: he uses Google Reader to subscribe to a dummy RSS feed on his App Engine hosted test app, that results in some kind of regular polling with an interval

Web 2.0 and beyond: The missing feature of the Web: a DNS REST API

Roberto Saccon — 2008-06-05T21:04:41+00:00

I have been waiting for years, that a quality DNS provider will a offer a REST API for setting DNS Records, but there is still no such a thing on the market. There is a SOAP based offering from Nettica (unfortunately I have an SOAP allergy). And there are the REST APIs some service providers are exposing for updating dynamic IPs. DNS Made Easy, one of my preferred DNS providers, told me about a