Planet Trapexit - Erlang/OTP News

Erlang Factory News: New Speaker for the Erlang Factory SF Bay Area 2010

Fri, 12 Mar 2010 12:45:55 +0000

We would like to announce Geoff Cant as a Speaker for the Erlang Factory SF Bay Area. Geoff will give a talk on "Enet: TCP/IP in Pure(ish) Erlang".

To see all Speakers please click here. The programme of the conference is available here.

Reminder: The Erlang Factory in SF Bay Area is just over a week away! Do not miss out on what will be the biggest gathering of Erlang expertise in the USA, register today!

Process-one Blogs: OneTeam XMPP client entering private alpha

Thu, 11 Mar 2010 14:00:21 +0000

Our OneTeam XMPP client for Linux, Mac OS X, and Windows is entering a private alpha period.

OneTeam is a multiplatform XMPP client for Linux, Mac OS X, and Windows. We have slowly worked on it during long months (and even years, it is being developed since 2006) but never had enough time and resources to release it properly.

The time to launch OneTeam in the wild and interoperable internet is soon arriving. We believe we have achieved our goal to deliver some real enterprise-class user experience. As a proof, we are using it inhouse on a daily basis as a productivity tool, without any major glitch.

For the record, OneTeam works as a Firefox extension, as well as a standalone application, on the three major platforms, thanks to the XUL technology. OneTeam has a nice dashboard-based interface, and handy user interactions improvements everywhere. This XMPP client supports a wide range of features, including chat and presence of course, but also Multi-User Chat (groupchat), message threading, history browsing, etc. It also features Jingle voice calls, as well as the exclusive Jingle Nodes feature demoed at FOSDEM, that works much like Skype. On the more technical side, there is the XML console, as well as a very interesting remote debugging console.

So, we are now releasing OneTeam, but in alpha quality for now, to the curious ones, and those willing to help and test. Please feel free to contact us, if you want to join the alpha testing, and report issues. Please e-mail nverite at process-one.net for a OneTeam ride.

Programming in the 21st Century: Eleven Years of Erlang

Wed, 10 Mar 2010 06:00:00 +0000

I've written about how I started using Erlang. A good question is why, after eleven years, am I still using it?

For the record, I do use other languages. I enjoy writing Python code, and I've taught other people how to use Python. This website is statically generated by a Perl program that I had fun writing. And I dabble in various languages of the month which have cropped up. (Another website I used to maintain was generated by a script that I kept reimplementing. It started out written in Perl, but transitioned through at least REBOL, J, and Erlang before I was through.)

One of the two big reasons I've stuck with Erlang is because of its simplicity. The functional core of Erlang can and has been described in a couple of short chapters. Knowledge of four data types--numbers, atoms, lists, tuples--is enough for most programming problems. Binaries and funs can be tackled later. This simplicity is good, because the difficult part of Erlang and any mostly-functional language is in learning to write code without destructive updates. The language itself shouldn't pour complexity on top of that.

There are many possibilities for extending Erlang with new data types, with an alternative to records being high on the list. Should strings be split off from lists into a distinct entity? What about arrays of floats, so there's no need to box each value? How about a "machine integer" type that's represented without tagging and that doesn't get automatically promoted to an arbitrarily sized "big number" when needed?

All of those additional types are optimizations. Lists work just fine as strings, but even the most naive implementation of strings as unicode arrays would take half the memory of the equivalent lists, and that's powerful enticement. When Knuth warned of premature optimization, I like to think he wasn't talking so much about obfuscating code in the process of micro-optimizing for speed, but he was pointing out that code is made faster by specializing it. The process of specialization reduces your options, and you end up with a solution that's more focused and at the same time more brittle. You don't want to do that until you really need to.

It may be an overreaction to my years of optimization-focused programming, but I like the philosophy of making the Erlang system fast without just caving in and providing C-style abilities. I know how to write low-level C. And now I know how to write good high-level functional code. If I had been presented with a menu of optimization-oriented data types in Erlang, that might never have happened. I'd be writing C in the guise of Erlang.

The second reason I'm still using Erlang is because I understand it. I don't mean I know how to code in it, I mean I get it all the way down. I know more or less what transformations are applied by the compiler and the BEAM loader. I know how the BEAM virtual machine works. And unlike most languages, Erlang holds together as a full system. You could decide to ditch all existing C compilers and CPUs and start over completely, and Erlang could serve as a foundation for this new world of computing. The ECOMP project (warning: PowerPoint) proved that an FPGA running the Erlang VM directly gives impressive results.

Let me zoom in on one specific detail of the Erlang runtime. If you take an arbitrary piece of data in a language of the Lua or Python family, at the lowest-level it ends up wrapped inside a C struct. There's a type field, maybe a reference count, and because it's a heap allocated block of memory there's other hidden overhead that comes along with any dynamic allocation (such as the size of the block). Lua is unabashedly reliant on malloc-like heap management for just about everything.

Erlang memory handling is much more basic. There's a block of memory per process, and it grows from bottom to top until full. Most data objects aren't wrapped in structs. A tuple, for example, is one cell of data for the length followed by the number of cells in the tuple. The system identifies it as a tuple by tagging the pointer to the tuple. You know the memory used for a tuple is always 1 + N, period. Were I trying to optimize data representation by hand, with the caveat that type info needs to be included, it would be tough to do significantly better.

I'm sure some people are correctly pointing out that this is how most Lisp and Scheme systems have worked since those languages were developed. There's nothing preventing an imperative language from using the same methods (and indeed this is sometimes the case).

Erlang takes this further by having a separate block of memory for each process, so when the block gets full only that particular block needs to be garbage collected. If it's a 64K block, it's takes microseconds to collect, as compared to potentially traversing a heap containing the hundreds of megabytes of data in the full running system. Disallowing destructive updates allows some nice optimizations in the garbage collector, because pointers are guaranteed to reference older objects (this is sometimes called a "unidirectional heap"). Together these are much simpler than building a real-time garbage collector that can survive under the pressure of giant heaps.

Would I use Erlang for everything? Of course not. Erlang is clearly a bad match for some types of programming. It would be silly to force-fit Erlang into the iPhone, for example, with Apple promoting Objective C as the one true way. But it's the best mix of power and simplicity that I've come across.

Erlang Inside: Zotonic destroys WordPress and rethinks the CMS with Erlang

Tue, 09 Mar 2010 18:50:24 +0000

A chat with Marc Worrell, Lead Architect of Zotonic - a new Content Management System written entirely in Erlang.

Programming in the 21st Century: It Made Sense in 1978

Thu, 04 Mar 2010 06:00:00 +0000

Whenever I see this list of memory cell sizes, it strikes me as antiquated:

BYTE = 8 bits
WORD = 16 bits
LONG = 32 bits

Those names were standard for both the Intel x86 and Motorola 68000 families of processors, and it's easy to see where they came from. "Word" isn't synonymous with a 16-bit value; it refers to the fundamental data size that a computer architecture is built to operate upon. On a 16-bit CPU like the 8086, a word is naturally 16-bits.

Now it's 2010, and it's silly to think of a 16-bit value as a basic enough unit of data to get to the designation "word." "Long" is similarly out of place, as 32-bit microprocessors have been around for over 25 years, and yet the standard memory cell size is still labeled in a way that makes it sound abnormally large.

The PowerPC folks got this right back in the early 1990s with this nomenclature:

BYTE = 8 bits
HALFWORD = 16 bits
WORD = 32 bits

That made sense in 1991, and it's still rational today. (64-bit is now common, but the jump isn't nearly as critical as it was the last time memory cell size doubled. The PowerPC name for "64-bits" is "doubleword.")

Occasionally you need to reevaluate your assumptions and not just cling to something because it's always been that way.

Process-one Blogs: Talkr.IM XMPP/Jabber server gets Apple Push notifications

Wed, 03 Mar 2010 21:00:16 +0000

The free public XMPP/Jabber server Talkr.IM gets Apple Push notifications for iPhone.

The ProcessOne's free, open, public XMPP/Jabber server Talkr.IM is gaining a new feature: Apple Push notifications. This well enable users to get notified of offline messages when they arrive, as well as simulate a continued XMPP session on iPhone devices that still do not implement multitasking.

For the latter feature, since the iPhone still does not accept to run applications in the background, if you want to use another application, you will have to shutdown your XMPP client and thus disconnect from Talkr.IM. With the Push feature enabled on your XMPP client (if this one supports Apple Push notifications), you can tell your contacts you are still online when you are using other applications. You will then automatically receive notifications of new messages. And when you re-open your XMPP client, you will receive these messages.

OneTeam for iPhone has the Apple Push feature enabled: you will be able to use it on you Talkr.IM account. Just use you Jabber ID, like username@talkr.im, and for a simple configuration, go to the "Settings" tab, then "Push settings".

The Talkr.IM XMPP server will stop at 10:00am CET. This means:

01:00am Los Angeles
04:00am New York
12:00 Moscow
18:00 Tokyo

UPDATE: the Apple Push module is installed on Talkr.IM, and working fine. You can use it on OneTeam for iPhone. You can run a APNS module on your own ejabberd server with IMpush.

Process-one Blogs: ejabberd tip: simple health check

Wed, 03 Mar 2010 16:30:28 +0000

An ejabberd health check mechanism might be useful in your deployments, but using the HTTP file server might be overkill...

Some people want to do regular health checks on ejabberd, which is always a good idea. Doing a health check over XMPP might not be the solution, here is at least two reasons:

with a pure ad-hoc solution like a cron script, you might need an XMPP library you might not want to install on your servers
monitoring systems might not talk XMPP at all...

So people first think of the internal ejabberd HTTP file server, which then they have to configure through a specific listener and the right module... possibly with logs, they would have to manage (rotate, parse, clean, etc.).

Sometimes people just don't think there is already an HTTP listener on their internal BOSH Connection Manager. You would argue that BOSH uses the HTTP POST method. But the ejabberd BOSH Connection Manager replies to simple GET:

wget http://myserver.net/http-bind

And ejabberd will answer:

ejabberd mod_http_bind v1.2

An implementation of XMPP over BOSH (XEP-0206)

This web page is only informative. To use HTTP-Bind you need a Jabber/XMPP client that supports it.

Erlang Factory News: Countdown to the Erlang Factory SF Bay Area

Wed, 03 Mar 2010 15:46:07 +0000

There are only 3 weeks left to the biggest Erlang gathering in the USA. We have 43 speakers giving 35 talks in 6 tracks! The programme of the conference can be viewed here. If you haven't registered yet, you can still do this by clicking here.

'I'm attending', 'I'm speaking' and 'See you there' banners are now available here. You can add them to your blog, Twitter page or any website. You are also welcome to follow @erlangfactory on Twitter.

Tragically L33T: The Lizard Brain, the Dip, and other Godinisms

Wed, 03 Mar 2010 08:00:00 +0000

Seth Godin is a really smart guy. More importantly, he takes his ideas and does something with them.

I came across Seth in an interview he did with Merlin Mann. He talked a lot about fear and the lizard brain: the part of our brain that is only hungry, scared, selfish and horny. Its the part of our brains that ruled in high school.

It will also sabotage us if we let it.

Whenever we get close to completing something big or interesting or important to us, the lizard brain senses a threat. “What if we fail?” “Ridicule is painful!” “Better to go along in life quietly, then do something that might get us noticed.”

The lizard brain tells us to quit when success is near. Seth mentioned in his interview that he listens to his lizard brain… and then does the opposite. If his lizard brain is quiet, he probably isn’t doing anything important. If his lizard brain is screaming bloody murder, he knows he is on the right track.

I picked up a couple of Seth’s books, including Tribes, The Dip, and his new book, Linchpin. As I come across more interesting tidbits I will share them.

Programming in the 21st Century: Dehumidifiers, Gravy, and Coding

Wed, 03 Mar 2010 06:00:00 +0000

For a few months I did freelance humor writing. Greeting cards, cartoon captions, that sort of thing. My sole income was from the following slogan, which ended up on a button:

Once I've gathered enough information for the almighty Zontaar, I'm outta here!

Sitting down and cranking out dozens of funny lines was hard. Harder than I expected. I gave it up because it was too draining (and because I wasn't making any money, but I digress).

Periodically I decide I want to boost my creativity. I carry around a notebook and write down conversations, lists, brainstormed ideas, randomness. I recently found one of these notebooks, so I can give some actual samples of its contents. Below half a page of "Luxury Housing Developments in Central Illinois Farmland" (e.g., Arctic Highlands), there's a long list titled "Ridiculous Things." Here are a few:

salads
spackle
key fobs
wine tastings
mulch
hair scrunchies
asphalt
Fry Daddy^TM
cinder blocks
relish
Frito Pie
aeration shoes

Okay, okay, I'll stop. But you get the idea.

As with the humor writing, I remember this taking lots of effort, and it took real focus to keep going. Did this improve my creativity? I'd like to think so. It certainly got me thinking in new directions and about different topics. It also made me realize something fundamental: technical creativity, such as optimizing code or thinking up clever engineering solutions, is completely different from the "normal" creativity that goes into writing stories or taking photos.

Years ago, I followed the development of an indie game. This was back when writing 3D games for non-accelerated VGA cards was cutting edge. The author was astounding in his coding brilliance. He kept pulling out trick after trick, and he wasn't shy about posting key routines for others to use. Eventually the game was released...and promptly forgotten. It may have been a technical masterpiece, but it was terrible as game, completely unplayable.

I still like a good solution to a programming problem. I still like figuring out how to rewrite a function with half the code. But technical creativity is only one form of creativity.

RedHotErlang: Returning a file with Nitrogen

Tue, 02 Mar 2010 19:41:01 +0000

Let's say I want to create a file download link in my Nitrogen application. First I create a src/pages/web_file.erl file, and fill it with some code: <pre><code>-module(web_file). -export([main/0, title/0, body/0, event/1]). -include...

LShift on Erlang: Memory matters - even in Erlang

Sun, 28 Feb 2010 21:42:43 +0000

Some time ago we got an interesting bug report for RabbitMQ. Surprisingly, unlike other complex bugs, this one is easy to describe:

At some point basic.get suddenly starts being very slow - about 9 times slower!

Basic.get doesn’t do anything complex - it just pops a message from a queue. This behaviour was quite unexpected. Our initial tests confirmed that we have a problem when a queue contains thousands of elements:

queue_length: 90001  basic_get 3333 times took: 1421.250ms
queue_length: 83335  basic_get 3333 times took: 1576.664ms
queue_length: 60004  basic_get 3333 times took: 1403.086ms
queue_length: 53338  basic_get 3333 times took: 9659.434ms [ look at that! ]
queue_length: 50005  basic_get 3333 times took: 9885.598ms
queue_length: 46672  basic_get 3333 times took: 8562.136ms

Let me repeat that. Usually popping a message from a queue takes Xms. At some point, it slows down to 9*Xms.

It turned out that the problem is with the queue:len() function, which is executed during the basic.get. Actually, queue:len() calls only erlang:length() builtin. At some point it switches to the “slow” mode.

Erlang:length() is a builtin that iterates through a linked list and counts it’s length. It’s complexity is O(N), where N is the length of the list. This function is implemented in the VM so it’s expected to be very, very fast.

The problem is not with erlang:length() being slow. It’s about being unpredictably slow. Let’s take a look at Erlang interpreter source code (erl_bif_guard.c:erts_gc_length_1). Here’s the main loop for erlang:length():

i=0
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}

It does nothing unusual - it just iterates through list elements. However, recompiling Erlang with some debugging information confirms that the problem is indeed here:

clock_gettime(CLOCK_REALTIME, &t0);
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}
clock_gettime(CLOCK_REALTIME, &t1);
td_ms = TIMESPEC_NSEC_SUBTRACT(t1, t0) / 1000000.0;
if (i > 200000 || td_ms > 2.0) {
    fprintf(stderr, "gc_length_1(%p)=%i %.3fms\n\r", reg[live], i, td_ms);
}

gc_length_1(0x7f4dbfa7fc19)=499999 2.221ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.197ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.208ms
(hibernation)
gc_length_1(0x7f4db0572049)=499999 13.793ms
gc_length_1(0x7f4db0572049)=499999 12.806ms
gc_length_1(0x7f4db0572049)=499999 12.531ms

This confirms Matthias’ initial guess - the slowdown starts after Erlang process hibernation.

For those who aren’t Erlang experts: Hibernation is an operation that compacts an Erlang process. It does aggressive garbage collection and reduces the memory footprint of a process to absolute minimum.

The intended result of hibernation is recovering free memory from the process. However its side effect is a new memory layout of objects allocated on the heap.

Ah, how could I have forgotten! The memory is nowadays slow! What happens, is that before hibernation list elements are aligned differently, more dense. Whereas after hibernation they are sparse. It’s easy to test it - let’s count the average distance between pointers to list elements:

gc_length_1(0x7f5c626fbc19)=499999 2.229ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.349ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.345ms avg=16.000 dev=0.023
(hibernation)
gc_length_1(0x7f5c61f7d049)=499999 13.800ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.726ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.367ms avg=136.000 dev=0.266

Confirmed! Standard deviation is surprisingly small, so we can read the numbers as:

Before hibernation list elements are aligned exactly one after another, values are somewhere else.
After hibernation list elements are interleaved with values.

This behavior does make sense. In most cases when you traverse the list, you actually do something with the values. After hibernation, when you access list item, the value will be already loaded to the CPU cache.

Knowing the mechanism, it’s easy to write a test case that reproduces the problem.

The average distance between pointers in my case is constant - the standard deviation is negligible. This information has a practical implication - we can “predict” where the next pointer will be. Let’s use that information to “fix” the Erlang VM by prefetching memory!

while (is_list(list)) {
    i++;
    list2 = CDR(list_val(list));
    __builtin_prefetch((char*)list2 + 128*((long)list2-(long)list));
    list = list2;
}

Test script running on original Erlang VM:

length: 300001  avg:0.888792ms dev:0.061587ms
length: 300001  avg:0.881030ms dev:0.040961ms
length: 300001  avg:0.875158ms dev:0.019436ms
hibernate
length: 300001  avg:14.861762ms dev:0.150635ms
length: 300001  avg:14.833733ms dev:0.017405ms
length: 300001  avg:14.884861ms dev:0.220119ms

Patched Erlang VM:

length: 300001  avg:0.742822ms dev:0.029322ms
length: 300001  avg:0.739149ms dev:0.012897ms
length: 300001  avg:0.739465ms dev:0.014417ms
hibernate
length: 300001  avg:7.543693ms dev:0.284355ms
length: 300001  avg:7.342802ms dev:0.330158ms
length: 300001  avg:7.265960ms dev:0.053176ms

The test runs only a tiny bit faster for the “fast” case (dense conses) and twice as fast for the “slow” case (sparse conses).

Should this patch be merged into mainline Erlang? Not really. I have set the prefetch multiplier value to 128 and I don’t even know if it’s optimal. This was only an experiment. But it was fun to see how low-level system architecture can affect high-level applications.

Tragically L33T: Moving my Blog to Jekyll

Sun, 28 Feb 2010 08:00:00 +0000

In the past I have not blogged very often. In fact I seem to blog less often than Wordpress releases a security patch. This was making me nervous and, combined with the issues of writing posts offline at events like NFJS, I decided a change was in order.

Enter Jekyll, the static page blog generator behind Github Pages. So far the workflow of managing text files in a Git repository is working well for me. Not being able to leave well enough alone I created a Rakefile to manage certain tasks like create a tagcloud for the sidebar, creating tag specific pages listing posts, and creating a draft post.

Creating a draft post was pretty straightforward:

desc 'create a new post in draft mode'
task :new => [:require_input] do
  title = ask("Title: ")
  filename = title.downcase.gsub(/[^a-z0-9]/,"-")
  template=File.read "lib/post_template.markdown"
  File.open("_drafts/#{filename}.markdown", 'w+') do |f| 
    f << template.gsub(/POST_TITLE/, title)
  end
  sh "git add _drafts/#{filename}.markdown"
end

Publishing a draft to the blog will consist of a git mv of the draft file to the _posts directory with the data appended to the filename.

As for comments I have switched over to Disqus, which allowed me to import my Wordpress comments and link to them on my Jekyll blog.

RedHotErlang: Nurturing creativity and the passionate programmer

Sat, 27 Feb 2010 09:11:02 +0000

I happened to read an article in the Harvard Business Review the other day. The article was about how Pixar was fostering creativity among their employees. I realized that I've been nurturing my creativity for years by spending a <br...

Chicago Erlang User Group: Erlang Development Cycles Presentations

noreply@blogger.com (Barry Nicholson) — Wed, 24 Feb 2010 13:54:00 +0000

Here's the video from the Development Cycles Talk given on February 17, 2010. Lots of great ideas in here!Development Cycles Part 1 tools used can be found at http://code.google.com/p/faxienthe github erlware accountand sinan can be found with faxien by running "faxien install-release sinan"Also, check out http://ecb.sourceforge.net/ for the emacs IDE mode and http://cedet.sourceforge.net/ and

Kevin Scaldeferri's Weblog: Alpha Male Programmers Are Driving Women Out

Wed, 24 Feb 2010 10:57:04 +0000

Yesterday, DHH made an argument that alpha male programmers aren’t keeping women out of the tech field. I’m of the opinion that he’s wrong and that his argument is flawed, and in a moment I’ll explain why, but let me get a few things out of the way so they don’t distract from the rest of my argument.

First, I don’t think that “alpha males” or the “rockstar” mentality are the only causes of under-representation of women in technology. As far as I can tell, the causes are many, varied, generally difficult to deal with, and in one or two cases may even be valid reasons why we might ultimately accept some degree of imbalance in the field. Second, this is not strictly a gender issue. Some men are also driven away by this behavior, although I think women are more likely to be; and also some “alpha males” happen to be female. Third, this is not a personal attack on DHH or any other individual, although some people might read parts of it in that way. But my goal here is that the range of individuals who find themselves uncomfortably reflected in what I say, but don’t simply reject it all out of hand, might view that discomfort as an opportunity for personal growth. Finally, I am certainly not claiming that I am perfect in this regard. I’ve made mistakes in the past; I will make them again. I simply hope that my friends will be kind enough to point out my mistakes to me, so that I can try to do better.

Okay, now that that’s all out of the way...

I first claim that DHH is wrong. My proof is empirical and anecdotal, but fortunately for me, I’m on the side of this debate that gets to use those techniques. I.e., I’m asserting existence of a phenomenon, rather than non-existence. I know numerous women who have been driven away by alpha male behavior. In some cases, they simply moved to a different team, or a different company. In other cases, they switched to non-programmer roles. And some left the industry entirely. I know this because they told me. They described specific individuals and specific behaviors which drove them away.

With some frequency in these debates, male programmers will claim that they don’t know any women who have left the field for this reason (or who have experienced sexism in the field, or who were offended by the example under discussion, or even just the milder claim that DHH makes, that no one has any idea what to do). I can explain this in only one of two ways: either they don’t know any women in the field to begin with or don’t talk with them beyond the minimal professional requirements, or women are not telling them because they are part of the problem. Perhaps it would be more effective if these women directly confronted the people causing the problem, but the fact of the matter is that most people, men and women, dislike conflict. We’re much more comfortable griping to a sympathetic friend than to the cause of our unhappiness. So consider me something akin to an anonymizing proxy. Without revealing names or too many of the specifics, please trust me when I say that almost every woman in the field experiences and complains about this.

Now I also said that DHH’s argument is flawed, and I will spend the rest of this post pointing out the various flaws I see.

DHH claims alpha males cannot be a problem in programming because the average male programmer is “meek, tame, and introverted” compared to other fields. First off, “alpha males” are by definition not average; they are the most dominant individuals of a group. And, it may even be possible that the general meekness or introversion of programmers makes it easier for a small set of individuals to dominate the interaction, rather than reaching a condition of detente between a group of uniformly assertive individuals. Second, presumably DHH does not interact with these people from other fields in a professional context. A point repeatedly stressed in this recent “pr0n star” controversy is that it’s not an issue of people being anti-sex or anti-porn; it’s about what’s appropriate in the workplace, or in a professional context. Standards for behavior in a social context are different. Third, he speaks in terms of whether these other men are more or less “R-rated”. This is not the point. Women are just as “R-rated” as men. They curse. They talk about sex (often more explicitly than men). The issue is not about whether we say or do adult things, it’s about whether we respect each other as human beings and whether we understand the societal norms of what is and is not appropriate in particular contexts. In fact, in this regard, I’ll defend DHH. Saying “fuck” (in the exclamatory, non-sexual usage) in the course of a technical presentation is not problematic in this day and age within the technology community. I think most of us swear freely in the course of struggling with a nasty bug or production problem. This is a normative expression of frustration within our community, and it does not oppress or disrespect other members of the community. (As far as I know. It’s possible that people just aren’t telling me that it upsets them when I curse at my monitor. If that’s the case, I hope someone will tell me.) Finally, DHH observes that these other fields have a more even mix of men and women. What he misses is that when the distribution is relatively equal it is generally easier and more comfortable for men to be men and women to be women. It is perhaps counterintuitive, but environments which are heavily skewed call for greater sensitivity to gender or other cultural differences simply because it is so easy to unintentionally create an oppressive or exclusionary atmosphere.

In the final paragraphs of his post, DHH suggests that somehow by respecting women we are squashing some other sort of “edge” and diversity in the community. I’m a little puzzled by what he means by this, and I’m sort of afraid that he thinks that being a heterosexual male who likes to look at scantily-clad women (or who openly admits as much) is somehow “edgy”. It’s not. By definition, hetero males like women; and it’s well established that men tend to be visually oriented. Pointing out that you fall in this category does not make you “diverse”, it makes you a completely typical representative of your gender and orientation. No one needs to be reminded of it.

Moreover, it might be true that maximal gains are had by pushing the edges (although I don’t think that one should naively assume that analogy from physics or economics applies to social endeavors), but for this to be work there has to be negative feedback when boundaries are crossed. If the edge-walkers want society to accept their behavior, they must be prepared to apologize, to make reparations, and to correct their course when they go over the line. This is the difference between a trend-setter and a sociopath.

There’s quite a bit more that I could say on this issue, but I fear this may be becoming too long already, and I think it’s probably best to focus only on the arguments presented in this particular post at the moment. To summarize things in a couple of sentences, the phenomenon of women being discouraged by alpha male behavior is real. You merely need to talk, and listen, to women in the field to verify this. (But you might have to earn some trust first.) Comparisons with men in other fields in non-professional settings do not have much relevance to the matter at hand. Claims that respecting the feelings and experiences of a minority group is damaging to the community overall are extraordinary and require extraordinary support. Being a thought leader and being offensive are two very different things.

It’s really quite discouraging that so much of this discussion still seems mired in the question of whether a problem even exists, or whether it is desirable and possible to address the problem. This lack of acceptance leads both to the explicit refusals to acknowledge the validity of the complaints of the offended, as well as the phenomena of false apologies and insincere claims that “I would help if only I knew how (and if it doesn’t require any actual change of behavior on my part)”. Male programmers need to pull their heads out of the sand. The evidence, both hard statistical data and anecdotal, is overwhelming. It also is not hard to find advice about what can be done differently. The hard part is moving from a vague desire for diversity and balance to serious, meaningful, sometimes painful self-examination and commitment to change and improvement. It’s not easy to admit flaws in yourself, to acknowledge when you’ve hurt another person, or to make a true apology. Change doesn’t happen overnight or simply because you say that you want it to. It takes work, but it’s an important part of being a human being and being a member of a community.

Comments

Kevin Scaldeferri's Weblog: Speeding up a Rails request by 150ms by changing 1 line

Wed, 24 Feb 2010 10:57:04 +0000

We’re pretty obsessed with performance at Gilt Groupe. You can get a taste for what we’re dealing with, and how we’re dealing with it, from our recent presentation at RailsConf.

One of the techniques we’re using is to precompute what certain high-volume pages will look like at a given time in the future, and store the result as static HTML that we serve to the actual users at that time. For ease of initial development, and because there’s still a fair bit of business logic involved in determining which version of a particular page to serve, this was done inside our normal controller actions which look for a static file to serve, before falling back to generating it dynamically.

We’re now running on Rails 2.3 and, of course, Rails Metal is the new hotness in 2.3. I spent the last couple days looking into how much improvement in static file serving we would see by moving it into the Metal layer. Based on most of what I’ve read, I expected we might shave off a couple milliseconds. This expectation turned out to be dramatically wrong.

Metal components operate outside the realm of the usual Rails timing and logging components, so you don’t get any internal measurements of page performance. Instead, I fired up ab to measure the serving times externally. What I found for the page I was benchmarking was that the Metal implementation took about 5ms. The old controller action took 170ms. But, wait... the Rails logs were only reporting 8ms for that action. Something was fishy.

I started inserting timers at various places in the Rails stack, trying to figure out where the other 160ms was going. A little bit was routing logic and other miscellaneous overhead, but even setting a timer around the very entry points into the Rails request serving path, I was only seeing 15ms being spent. This was getting really puzzling, because at this point where a Rack response is returned to the web server, I expected things to look identical between Metal and ActionController. However, looking more closely at the response objects I discovered the critical difference. The Metal response returns an [String], while the controller returned an ActionController::Response.

I went into the Rails source and found the each method for ActionController::Response. Here it is:

   def each(&callback)
     if @body.respond_to?(:call)
       @writer = lambda { |x| callback.call(x) }
       @body.call(self, self)
     elsif @body.is_a?(String)
       @body.each_line(&callback)
     else
       @body.each(&callback)
     end

     @writer = callback
     @block.call(self) if @block
   end

The critical line is the case where the body is a String. The code iterates over each line in the response. Each line is written individually to the network socket. In the case of the particular page I was looking at, that was 1300 writes. Ouch.

To confirm this was the problem, I changed that line to

      yield @body

With the whole body being sent in a single write, ab reported 15ms per request, right in line with what I measured inside Rails.

1 line changed. 150ms gained. Not too bad.

This sort of performance pessimization we uncovered is particularly insidious because it’s completely invisible to all the usual Rails monitoring tools. It doesn’t show up in your logged response time; you won’t see it in NewRelic or TuneUp. The only way you’re going to find out about it is by running an external benchmarking tool. Of course, this is always a good idea, but it’s easy to forget to do it, because the tools that work inside the Rails ecosystem are so nice. But the lesson here is, if you’re working on performance optimizations, make sure to always get a second opinion.

Comments

Trapexit News Feed: Trapexit News :: Erlang Open Source crawler!

Wed, 24 Feb 2010 10:56:53 +0000

Author: admin
Subject: Erlang Open Source crawler!
Posted: Thu Nov 19, 2009 4:22 pm (GMT 0)
Topic Replies: 0

Trapexit is proud to announce its own Erlang open source crawler. We crawl the web for you, gathering searchable information on all open source Erlang projects from all the major repositories. You can discuss these projects in the forum, add documentation in the trapexit wiki or rate the various contributions. To view our new project index repository, visit http://projects.trapexit.org.

Trapexit News Feed: Trapexit News :: Navigating The Erlang Source!

Wed, 24 Feb 2010 10:56:53 +0000

Author: francesco
Subject: Navigating The Erlang Source!
Posted: Thu Aug 20, 2009 2:24 pm (GMT 0)
Topic Replies: 0

While trapexit user zghst was looking at the erlang source, there was quite a bit of information about it he felt was missing or hard to find. As a result, he started a wiki page to help new people like him find their way around it: A Guide To The Erlang Source. Why not give him a hand in improving it?

Erlang Factory News: Early Bird Saver - only two weeks left!

Wed, 24 Feb 2010 10:56:23 +0000

You only have two weeks left to take advantage of our Early Bird special offers. Sign-up for the Erlang Factory before 28th February 2010 and SAVE $200 off the standard price!

When registering, you can also sign up for the Erlang University courses and choose from: Erlang Express, Erlang OTP Express, QuickCheck for Erlang developers and Web Programming with Erlang. The discount applies to the courses as well.

Don't miss out - book your place today!

Erlang Factory News: Book your hotel room for the Erlang Factory!

Wed, 24 Feb 2010 10:56:23 +0000

The 2010 Erlang Factory will be held at the Hilton San Francisco Airport. You can now book your hotel room there for the GREAT CONFERENCE RATE! For rooms with a Kingsize bed, delegates will pay only $109.00 per night – room only, with FREE PARKING and FREE WI-FI in the rooms for delegates.

To book for the special Erlang Factory rate click HERE.

RiboComments/Erlang: Generating prime numbers with Erlang and Java

Wed, 24 Feb 2010 10:56:23 +0000

During my work with the course-ware of a new Erlang course, I experimented with one of the programming assignments to compare the threading performance of Erlang versus Java. The assignment is one of the classical programs from teaching concurrent programming: How to generate prime numbers using a pipeline of sieve ...

RiboComments/Erlang: The critical section problem in Erlang

Wed, 24 Feb 2010 10:56:23 +0000

The programming language Erlang is based on micro-threads and asynchronous message passing. There is a (naive) belief that critical section problems cannot arise in languages based solely on message passing. The justification for this stand-point is the absence of mutex synchronization primitives, which is absolutely essential in shared-data based concurrent ...

EazyErl!: Filtering lines efficiently

noreply@blogger.com (rolphin) — Wed, 24 Feb 2010 09:00:02 +0000

Whenever you are dealing with log lines or that you're program is filtering data you always have to handle 'escaping' efficiently.

While developing a log module using a gen_event, I needed to escape simple quotes.
Sometime thoses quotes were already escaped...

I've found this regexp to handle gracefully the case:

re:replace( Bin, "(?<!\\\\)'", "\\\\'", [ global ] ).

Not so easy to read, and because of the various backslashes, this regexp needed some tests
before being fully usable.
Basically this regexp only filter simple quote when they're not already escaped...

Now that I've my filter for quotes and I need a filter for newline characters.
Everytime you find some newlines in your log files, you can be sure that many tools already in your network will not treat them efficiently, worse this may break everything after ...

So I came across this regexp:

re:replace( Bin, "[\\n\\r]+", " ", [ global ] ).

Finally having two filter functions for every line, I wanted to be able to add or remove easily any function.
You can reference functions with this notation:

fun filterquotes/1

Or a list of functions:

[ fun filterquotes/1, fun filternewlines/1 ]

With this notation and the famous 'lists:foldl', I can filter a line with many functions quite nicely:

% Data is the accumulator, but we don't change it :)
        lists:foldl( fun( Fun, Data ) ->
                         Fun(Data) 
        end, Bin, [ fun filterquotes/1, fun filternewline/1 ]).

Here's the code:

filter( Bin ) when is_atom(Bin) ->
        Bin;
filter( Bin ) when is_integer(Bin) ->
        Bin;
filter( Bin ) ->
        lists:foldl( fun( Fun, Data ) ->
                         Fun(Data) 
        end, Bin, [ fun filterquotes/1, fun filternewline/1 ]).

filterquotes(Bin) ->
        re:replace( Bin, "(?<!\\\\)'", "\\\\'", [ global ] ).

filternewline( Bin ) ->
        re:replace( Bin, "[\\n\\r]+", " ", [ global ] ).

EazyErl!: Reading an openssl .priv.key file and extracting the key

noreply@blogger.com (rolphin) — Tue, 23 Feb 2010 22:23:26 +0000

Extracting the private key from a .priv.key file is simple.
The private key is encrypted using a AES-128 with your passphrase.

The initial vector is also stored in the file, you can extract it directly from the first line:

get_salt( "Salted__", Salt:8/binary, Rest/binary>> ) ->
        {Salt, Rest}.

The last part is handled by some md5() of your passphrase and the initial vector:

Key = crypto:md5([ Password, Salt ]),
        IV = crypto:md5([ Key, Password, Salt ]),
        crypto:aes_cbc_128_decrypt( Key, IV, Rest).

Now the full module:

-module(priv_key).

-compile(export_all).

priv_key_file( File, Password ) ->
        {ok, Bin} = file:read_file(File),
        {Salt, Rest} = get_salt(Bin),
        Key = crypto:md5([ Password, Salt ]),
        IV = crypto:md5([ Key, Password, Salt ]),
        crypto:aes_cbc_128_decrypt( Key, IV, Rest).
        
get_salt( "Salted__", Salt:8/binary, Rest/binary>> ) ->
        {Salt, Rest}.

Nicolas Charpentier's blog: Code availability on git-hub

Sun, 21 Feb 2010 08:54:32 +0000

Bored to switch code between my private subversion repository and git-hub for my open-source projects, I decided to use exclusively git-hub for them. My trac wiki pages are also moved to git-hub for improve the documentation of those projects. Selenium-RC Erlang binding Rake tasks to build erlang code Small mock library for erlang (Please don’t use mock to test [...]

Process-one Blogs: Jingle Nodes on Talkr.IM

Thu, 18 Feb 2010 14:00:00 +0000

A Jingle Nodes relay has been installed on the Talkr.IM XMPP service.

The public XMPP server Talkr.IM has received its own Jingle Nodes relay on: xmpp:jn.talkr.im. You can browse the Talkr.IM services via your Service Discovery interface in your XMPP client.

This new exclusive feature on Talkr.IM XMPP server enables Jingle calls by providing a public service for media relaying. Basically, this means that XMPP clients which can handle Jingle Nodes, will be able to join other Jingle Nodes enabled clients to establish voice calls even when both are hidden behind network barriers (like NAT, for the techies).

One many entities of the XMPP federated network (clients, servers) will have Jingle Nodes enabled, this will be possible to establish calls, in a totally P2P manner, much like Skype, but with much more features like the control of who can use your relay. Read the Jingle Nodes website and the Jingle Nodes XEP to know more.

This Talkr.IM Jingle Nodes relay is one of the very first step to a simpler Jingle user experience worldwide.

Visit the Talkr.IM website, and create your own Talkr.IM XMPP account freely: https://www.talkr.im/signup.

Process-one Blogs: OneTeam for iPhone in version 3.2.5, with Facebook chat

Wed, 17 Feb 2010 16:08:11 +0000

OneTeam for iPhone has been published on the AppStore in version 3.2.5, bringing bugfixes, and Facebook chat based on XMPP.

The version 3.2.5 of our XMPP client OneTeam for iPhone has hit the shelves of the Apple's application store. It is recommended that you upgrade it.

OneTeam 3.2.4 for iPhone had already fixed bug related to Multi-User Chats (also known as groupchats): in some cases, OneTeam used to crash on joining password-protected and members-only MUC.

OneTeam 3.2.5 for iPhone now allows OneTeam users to connect to the XMPP interface to Facebook chat. You can now connect with Jabber-ID set your "your_username@chat.facebook.com", and password set to your Facebook password.

Be careful, Facebook's XMPP chat interface does not offer MUC services, nor is federated, which means you still cannot chat with your friends on Talkr.IM or Google's Talk/GMail, but only with Facebook friends.

Check out OneTeam for iPhone on our website, as well as Apple's AppStore.

Chicago Erlang User Group: OTP Packaging Video 12/16/2009

noreply@blogger.com (Barry Nicholson) — Tue, 16 Feb 2010 15:36:00 +0000

Here's the video from the December 16, 2009 Erlang/OTP Packaging talk given to the Chicago Erlang Users Group by Martin Logan.Part 1 Part 2 Part 3

Chicago Erlang User Group: Chicago Erlang User Group Meeting Februrary 17th

noreply@blogger.com (Jordan Wilberding) — Thu, 11 Feb 2010 12:59:00 +0000

The time and location:We are confirmed for Wednesday February 17th at 6:00PM. Orbitz has agreed again to let us use their office space for the meetup. We need all the names of the people that will attend so we can place them on the security list. Please email to RSVP at martinjlogan -at- gmail.The address for the talk is 500 W. Madison St, Chicago IL (the Olgilvie transportation center). Come up

Programming in the 21st Century: Optimizing for Fan Noise

Wed, 10 Feb 2010 06:00:00 +0000

The first money I ever earned, outside of getting an allowance, was writing assembly language games for an 8-bit home computer magazine called ANALOG Computing. Those games ended up as pages of printed listings of lines like this:

1050 DATA 4CBC08A6A4BC7D09A20986B7B980
0995E895D4B99E099DC91C9DB51CA90095C0C8
CA10E8A20086A88E7D1D8E7E,608

A typical game could be 75 to 125+ of those lines (and those "three" lines above count as one; it's word-wrapped for a 40-column display). On the printed page they were a wall of hex digits. And people typed them in by hand--I typed them in by hand--in what can only be described as a painstaking process. Just try reading that data to yourself and typing it into a text editor. Go ahead: 4C-BC-08-A6...

Typos were easy to make. That's the purpose of the "608" at the end of the line. It's a checksum verified by a separate "correctness checker" utility.

There was a strong incentive for the authors of these games to optimize their code. Not for speed, but to minimize the number of characters that people who bought the magazine had to type. Warning, 6502 code ahead! This:

   LDA #0
   TAY

was two fewer printed digits than this:

   LDA #0
   LDY #0

Across a 4K or 6K game, those savings mattered. Two characters here, four characters there, maybe the total line count could be reduced by four lines, six lines, ten lines. This had nothing to do with actual code performance. Even on a sub-2MHz processor those scattered few cycles were noise. But finding your place in the current line, saying "A6," then typing "A" and "6" took time. Measurable time. Time that was worth optimizing.

Most of the discussions I see about optimization are less concrete. It's always "speed" and "memory," but in the way someone with a big house and a good job says "I need more money." Optimization only matters if you're optimizing something where you can feel the difference, and you can't feel even thousands of bytes or nanoseconds. Optimizing for program understandability...I'll buy that, but it's more of an internal thing. There's one concern that really does matter these days, and it's not abstract in the least: power consumption.

It's more than just battery life. If a running program means I get an hour less work done before looking for a place to plug in, that's not horrible. The experience is the same, just shorter. But power consumption equals heat and that's what really matters to me: if the CPU load in my MacBook cranks up then it gets hot, and that causes the fan to spin up like a jet on the runway, which defeats the purpose of having a nice little notebook that I can bring places. I can't edit music tracks with a roaring fan like that, and it's not something I'd want next to me on the plane or one table over at the coffee shop. Of course it doesn't loudly whine like that most of the time, only when doing something that pushes the system hard.

What matters in 2010 is optimizing for fan noise.

If you're not buying this, take a look at Apple's stats about power consumption and thermal output of iMacs (which, remember, are systems where the CPU and fan are right there on your desk in the same enclosure as the monitor). There's a big difference in power consumption, and corresponding heat generated, between a CPU idling and at max load. That means it's the programs you are running which are directly responsible for both length of battery charge and how loudly the fan spins.

Obvious? Perhaps, but this is something that didn't occur with most popular 8-bit and 16-bit processors, because those chips never idled. They always ran flat-out all the time, even if just in a busy loop waiting for interrupts to hit. With the iMacs, there's a trend toward the difference between idle and max load increasing as the clock speed of the processor increases. The worst case is the early 2009 24-inch iMac: 387.3 BTU/h at idle, 710.3 BTU/h at max load, for a difference of 323 BTU/h. (For comparison, that difference is larger than the entire maximum thermal output of the 20-inch iMac CPU: 298.5 BTU/h.)

The utmost in processing speed, which once was the goal, now has a price associated with it. At the same time that manufacturers cite impressive benchmark numbers, there's also the implicit assumption that you don't really want to hit those numbers in the everyday use of a mobile computer. Get all those cores going all the time, including the vector floating point units, and you get rewarded with forty minutes of use on a full battery charge with the fan whooshing the whole time. And if you optimize your code purely for speed, you're getting what you asked for.

Realistically, is there anything you can do? Yes, but it means you have to break free from the mindset that all of a computer's power is there for the taking. Doubling the speed of a program by moving from one to four cores is a win if you're looking at the raw benchmark numbers, but an overall loss in terms of computation per watt. Ideas that sounded good in the days of CPU cycles being a free resource, such as anticipating a time-consuming task that the user might request and starting it in the background, are now questionable features. Ditto for persistent unnecessary animations.

Nanoseconds are abstract. The sound waves generated by poorly designed applications are not.

RedHotErlang: inotify for erlang

Sat, 06 Feb 2010 16:44:06 +0000

inotify is a cool linux kernel service, that lets you subscribe to file system events. For example, if you are interested in reacting to changes in the file "~/.emacs", and since you are not a lamer you abhor polling, you would like to be able to ...

RedHotErlang: Dialyzer

Fri, 05 Feb 2010 21:59:19 +0000

At times you sit there at the typical Erlang conferences and hear Kostis Sagonis speak of dialyzer, typer and all the funky tools. It sounds just so great. The downside is of course that all the tools require an effort on your part. You need to re...

Process-one Blogs: ProcessOne at FOSDEM and XMPP Summit

Thu, 04 Feb 2010 17:19:01 +0000

ProcessOne will be present at FOSDEM and XMPP Summit.

This week-end, we will be joining the XMPP community at Brussels, for the FOSDEM, as well as the XMPP-specific events: XMPP hackfest and XMPP Summit.

The FOSDEM, or Free and open source software developers' european meeting, will be held on Saturday, as well as Sunday, at ULB Campus Solbosch, with myriads of FOSS people for different countries and profiles.
Before and after this event, the XMPP community will gather on Friday for the XMPP hackfest, and Monday for the XMPP Summit.

The XMPP hackfest is the place where coders can code and test interop, while the XMPP Summit is more oriented towards specifications improving.

ProcessOne will be there and demo three of our software products:

OneTeam, our Firefox extension, providing interesting features
OneTeam for iPhone, enabling the always-on XMPP experience to a pocket near you
IMGateways, our transports, especially our Twitter gateway, as well as our Google Wave gateway

Feel free to join, as we would like to show you our software, and we would like to have your opinion on these.

Learn You Some Erlang: Errors and Exceptions

Tue, 02 Feb 2010 15:30:00 +0000

Erlang being a language built for high reliability, it is natural for it to have exceptions. This chapter focuses on how to handle and raise exceptions in the functional subset of Erlang. I go over compile-time errors and warning, run-time errors, the difference between errors, exits and throws, the different constructs to handle these and then finally, an example of how to use throws for non-local returns.

LShift on Erlang: RabbitMQ-shovel: Message Relocation Equipment

Mon, 01 Feb 2010 12:13:52 +0000

In several applications, it’s very useful to be able to take messages out of one RabbitMQ broker, and insert them into another. Many people on our mailing list have being asking for such a shovel, and we’ve recently been able to devote some time to writing one. This takes the form of a plugin for Rabbit, and whilst it hasn’t been through QA just yet, we’re announcing it so people who would like to play and even suggest further features for inclusion can do so sooner rather than later.

The shovel is written on top of the Erlang client. It supports both direct and network connections to nodes, SSL support, the ability to declare resources on nodes it connects to, basic round-~~robin~~rabbit balancing of both source and destination nodes, and allows you to configure many parameters controlling how messages are consumed from the source, and how they’re published to the destination. Multiple shovels can be specified, their statuses queried, and shovels can repeatedly reconnect to nodes in the event of failure.

The plugin is available from http://hg.rabbitmq.com/rabbitmq-shovel/, and is released under the MPL v1.1. There is a README included which contains full documentation. This is replicated below.

RabbitMQ-shovel

Introduction

This is a plug-in for RabbitMQ that shovels messages from a queue on one broker to an exchange on another broker. The two brokers may be the same. The plug-in allows several shovels to be specified at the same time. Each shovel may have a number of source and destination brokers specified, and one of each is chosen whenever the shovel attempts to make a connection: this permits simple round-rabbit load balancing.

Resources can be declared upon connection to both the source and destination brokers, and parameters can be specified for both the reception and publishing of messages.

Requirements

Currently, you must build the server from source, under branch bug16653. You must also have checked out the rabbitmq-public-umbrella hg repository, and have the rabbitmq-erlang-client built. From scratch, the following commands should build RabbitMQ with the shovel plug-in:

hg clone http://hg.rabbitmq.com/rabbitmq-public-umbrella
cd rabbitmq-public-umbrella
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-erlang-client
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-shovel
cd rabbitmq-server
hg up -C bug16653
make -j
mkdir -p plugins
cd plugins
ln -s ../../rabbitmq-erlang-client
ln -s ../../rabbitmq-shovel
cd ../../rabbitmq-erlang-client
make
cd ../rabbitmq-shovel
make
cd ../rabbitmq-server
./scripts/rabbitmq-activate-plugins
make cleandb run

Configuration

The RabbitMQ configuration file specifies the shovel configurations. This exists by default, in /etc/rabbitmq/rabbitmq.config under Linux systems, %RABBITMQ_BASE%\rabbitmq.config under Windows or somewhere else under OS X. This file configures both RabbitMQ-server and all the plugins installed in it. It is an Erlang-syntax file of the form:

[{section1, [section1-config]},
 {section2, [section2-config]},
 …
 {sectionN, [sectionN-config]}
].

thus a list of tuples, where the left element of each tuple names the applications being configured. Don’t forget the last element of the list doesn’t have a trailing comma, and don’t forget the full-stop is needed after closing the list. Hence if you configure RabbitMQ-server and the RabbitMQ-shovel, then the configuration file may have a structure like this:

[{rabbit,        [configuration-for-RabbitMQ-server]},
 {rabbit-shovel, [configuration-for-RabbitMQ-shovel]}
].

A full example of the shovel configuration is:

{rabbit_shovel,
  [{shovels,
    [{my_first_shovel,
      [{sources,      [{brokers,
                          ["amqp://fred:secret@host1.domain/my_vhost",
                           "amqp://john:secret@host2.domain/my_vhost"
                          ]},
                       {declarations,
                          ['queue.declare',
                           {'queue.bind',
                                  [{exchange, "my_exchange">>},
                                   {queue,    >}]}
                          ]}]},
       {destinations, [{broker, "amqp://"},
                       {declarations,
                          [{'exchange.declare',
                                  [{exchange, "my_exchange">>},
                                   {type, "direct">>},
                                   durable]}
                          ]}]},
       {queue, >},
       {qos, 10},
       {auto_ack, false},
       {tx_size, 0},
       {delivery_mode, keep},
       {publish_fields, [{exchange, "my_exchange">>},
                         {routing_key, "from_shovel">>}]},
       {reconnect, 5}
      ]}
     ]
   }]
}


Firstly, all shovels are named. Here we have one shovel, called
‘my_first_shovel’. We can have multiple shovels if you wish. Every
shovel must have all sub-fields specified: sources, destinations, qos,
auto_ack, delivery_mode, publish_fields, reconnect.

Sources and Destinations

Sources and destinations specify respectively where messages are
fetched from and delivered too. One of ‘broker’ and ‘brokers’ must be
specified, and ‘broker’ is simply shorthand for when only one broker
needs specifying. Using ‘brokers’ allows a list of brokers to be
specified: whenever the connection to a broker is lost, another one is
chosen at random from the list and a connection attempt is made to
that. The syntax for broker URIs is:

amqp://username:password@host:port/vhost

If username or password are omitted, the default values of guest and
guest are used. If the vhost is omitted, the default value of / is
used. If the host is omitted, then the plugin uses the “direct”
connection internally rather than a network connection: this means it
connects to the RabbitMQ-server node on which it is running without
going through the network stack. This is much more efficient. If port
is omitted then the default value is used (5672 or 5671 if SSL is
used).

SSL is implemented, for which additional parameters are needed:

amqps://username:password@host:port/vhost?cacertfile=/path/to/cacert.pem&certfile=/path/to/certfile.pem&keyfile=/path/to/keyfile.pem&verify=verifyOption&fail_if_no_peer_cert=failOption

(note, this is a single line)

All five parameters (3 paths: cacertfile, certfile and keyfile; 2
options: verify, fail_if_no_peer_cert) must be specified. See the SSL
guide at http://www.rabbitmq.com/ssl.html#configure-erlang for details
of SSL in RabbitMQ in general and specifically for the Erlang client
(on which the shovel is built).

Note that SSL cannot be used with the direct connection (i.e. a host
must be specified when using SSL), and that it is preferable to use
the non-SSL direct connection when connecting to the same node that’s
running the shovel.

Resource Declarations

Both sources and destinations can have an optional ‘declarations’
clause. The value of this is a list, consisting of AMQP Methods. If
default values are sufficient, then the method name alone can be
specified - e.g. ‘queue.declare’. If parameters need to be set then
the method should be given as a tuple, with the right hand side a
proplist specifying which fields need altering from their default
values. E.g:

    {'exchange.declare',[{exchange, "my_exchange">>},
                         {type, "direct">>},
                         durable]},


One very useful feature here is the Most-Recently-Declared-Queue
feature, in which RabbitMQ remembers the name of the most recently
declared queue. This means that you can declare a private queue, and
then bind it to exchanges without ever needing to know its name.

queue :: binary

This parameter specifies the name of the queue on the source brokers
to consume from. This queue must exist. Use the resource declarations
to create the queue (or ensure it exists) first. Note again that the
Most-Recently-Declared-Queue feature can be used here, thus an
anonymous queue can be used: use <<>> to indicate the
Most-Recently-Declared-Queue.

qos :: non-negative-integer

The shovel consumes from a queue. The QoS controls how many messages
are sent to the shovel in advance of the message the shovel is
currently processing.

auto_ack :: boolean

Setting this to ‘true’ turns on the no_ack flag when subscribing to
the source queue.

tx_size :: non-negative-integer

When set to 0, transactions are not used. Other values make publishes
transactional, with a commit every N messages. In lieu of the auto-ack
option, when transactions are not used, messages are acknowledged to
the source immediately after every publish. When transactions are
used, acks are only issued to the source on receipt of the commit-ok
message from the destination. This can thus be used to guarantee that
messages are only acknowledged (and thus forgotten about by the source
broker) when they are guaranteed to have been received by the
destination broker.

delivery_mode :: ‘keep’ | 0 | 2

This affects the delivery_mode field when publishing to the
destination. A value of ‘keep’ means that the same delivery_mode
should be used as when the message was originally published to the
source broker. 0 and 2 override the original setting.

publish_fields

This is a list of tuples which override fields in the publish method
when publishing to the destination. This can be used to direct
messages to a particular exchange on the destination, for example, or
change the routing key. By default, the routing key of the message as
it is received by the shovel is passed through, but this can be
overridden as necessary.

reconnect :: non-negative-integer

When an error occurs, the shovel will disconnect from both the source
and destination broker immediately. This will force uncommitted
transactions at the destination to be rolled back, and delivered but
unacknowledged messages from the source to be requeued. The shovel
will then try connecting again. If this is unsuccessful, then it’s not
a good idea for the shovel to very quickly and repeatedly try to
reconnect. The value specified here is the number of seconds to wait
between each connection attempt.

Note that if set to 0, the shovel will never try to reconnect: it’ll
stop after the first error.

Obtaining shovel statuses

From the broker Erlang prompt, call
rabbit_shovel_status:status(). This will return a list, with one row
for each configured shovel. Each row has three fields: the shovel
name, the shovel status, and the timestamp (a local calendar time of
{{YYYY,MM,DD},{HH,MM,SS}}). There are 3 possible statuses:


’starting’: The shovel is starting up, connecting and creating
resources.
‘running’: The shovel is up and running, shovelling messages.
{’terminated’, Reason}: Something’s gone wrong. The Reason should give
a further indication of where the fault lies.

Simon Willison's Weblog: Hot Code Loading in Node.js

Sun, 31 Jan 2010 13:57:53 +0000

Hot Code Loading in Node.js. Blaine Cook’s patch for Node.js that enables Erlang-style hot code loading, so you can switch out your application logic without restarting the server or affecting existing requests. This could make deploying new versions of Node applications trivial. I’d love to see a Node hosting service that allows you to simply upload a script file and have it execute on the Web.

Trapexit's Erlang Blog Filter: Erlang Factory SF Bay Area 2010

Sat, 30 Jan 2010 21:29:40 +0000

Interested in Erlang? You might consider attending Erlang Factory SF Bay Area 2010. Below is a message that Francesco Cesarini, conference organizer and co-author of the most excellent book Erlang Programming, sent to the erlang-questions list yesterday providing more information about the conference, especially pointing out that the Very Early Bird registration deadline is tomorrow (Sunday January 31). Hope to see you there!

Hi All,

a note to say that we are almost done with the programme for the 2010 SF Bay Area Erlang Factory. This year, we are lucky to have keynote speakers such as Joe Armstrong, Bjarne Dacker, Kenneth Lundin and Steve Vinoski. They will be giving four of the 35 scheduled talks on the 25th and 26th of March in the San Francisco Bay Area. The almost complete programme is available here:

http://erlang-factory.com/conference/SFBay2010/programme

The conference will be preceded by three days of University courses taught by experts such as Simon Thompson, John Hughes, Thomas Arts, Henry Nystrom and Kevin Smith. Come and learn Erlang, OTP, QuickCheck or Web Development with Erlang. More information on the courses are here:

http://erlang-factory.com/conference/SFBay2010/university

This *Sunday* (January 31st) is the deadline for the very early bird deadline. Register by Sunday night and save $400 on the on-site registration price.

The conference hotel and venue is the SF Airport Hilton, a short BART / Caltrain ride from SF and the Valley. We have secured a very competitive price of US$109 per room and night at the conference hotel, this being one of the reasons for us choosing it. The other is the lower price of the venue, allowing us to pass on the savings to the delegates through a higher very early bird discount. We are planning an ErlLounge open to everyone who can’t make the two days on the 25th, and hope we will be able to surpass last year’s success. If you have thoughts or questions, you are welcome to drop me a line.

Hope to see you all there!

Francesco
–
http://www.erlang-solutions.com

Erlang Inside: Erlang Factory SF Bay 2010 – Francesco Cesarini on the Conference and Old-School vs New-School Erlangers

Sat, 30 Jan 2010 18:19:20 +0000

The Erlang Factory’s 2010 conference is March 25th and 26th in San Francisco, with the university three days before, starting the 22nd. The conference is at the Hilton San Francisco Airport, with three tracks, each on a different theme. The number of tracks gives attendees an unusually broad set of choices for talks. When attending [...]

Programming in the 21st Century: What to do About Erlang's Records?

Sat, 30 Jan 2010 06:00:00 +0000

The second most common complaint about Erlang, right after confusion about commas and semicolons as separators, is about records. Gotta give those complainers some credit, because they've got taste. Statically defined records are out of place in a highly dynamic language. There have been various proposals over the years, including Richard O'Keefe's abstract syntax patterns and Joe Armstrong's structs. Getting one of those implemented needs the solid support of the Erlang system maintainers, and it's understandably difficult to commit to such a sweeping change to the language. So what are the alternatives to records that can be used right now?

To clarify, I'm really talking about smallish, purely functional dictionaries. For large amounts of data there's already the gb_trees module, plus several others with similar purposes.

In Python, a technique I often use is to return a small dictionary with a couple of named values in it. I could use a tuple, but a dictionary removes the need to worry about order. This is straightforward in Erlang, too:

fun(length) -> 46;
   (width)  -> 17;
   (color)  -> sea_green
end.

Getting the value corresponding to a key is easy enough:

Result(color)

This is handy, but only in certain situations. One shortcoming is that there's no way to iterate through the keys. Well, there's this idea:

fun(keys)   -> [length, width, color];
   (length) -> 46;
   (width)  -> 17;
   (color)  -> sea_green
end.

Now there's a way to get a list of keys, but there's room for error: each key appears twice in the code. The second issue is there's no simple way to take one dictionary and create a new one with a value added or removed. This road is becoming messy to go down, so here's more data-driven representation:

[{length, 46}, {width, 17}, {color, sea_green}]

That's just a list of key/value pairs, which is searchable via the fast, written-in-C function lists:keyfind. New values can be appended to the head of the list, and there are other functions in the lists module for deleting and replacing values. Iteration is also easy: it's just a list.

We still haven't bettered records in all ways. A big win for records, and this is something few purely functional data structures handle well, is the ability to create a new version where multiple keys get different values. For example, start with the above list and create this:

[{length, 200}, {width, 1400}, {color, sea_green}]

If we knew that only those three keys were allowed, fine, but that's cheating. The whole point of dictionaries is that we can put all sorts of stuff in there, and it doesn't change how the dictionary is manipulated. The general solution is to delete all the keys that should have new values, then insert the new key/value pairs at the head of the list. Or step through the list and see if the current key is one that has a new value and replace it. These are not linear algorithms, unfortunately. And you've got the same problem if you want to change multiple values in a gb_tree at the same time.

What I've been using, and I admit that this isn't perfect, is the key/value list approach, but forcing the lists to be sorted.This allows the original list and a list of changes to be merged together in linear time. The downside is that I have to remember to keep a literal list in sorted order (or write a parse transform to do this for me).

There's still one more feature of records that can't be emulated: extracting / comparing values using Erlang's standard pattern matching capabilities. It's not a terrible omission, but there's no way to dodge this one: it needs compiler and runtime system support.

Programming in the 21st Century: Nothing Like a Little Bit of Magic

Fri, 29 Jan 2010 06:00:00 +0000

Like so many other people, I was enthralled by the iPad introduction. I haven't held or even seen an iPad in person yet, but that video hit me on a number of levels. It's a combination of brand new hardware--almost dramatically so--and uses for it that are coming from a completely different line of thinking. I realized it's been a long time since I felt that way about the introduction of a new computer.

I remember the first time I tried a black and white 128K Mac in a retail store. A mouse! Really tiny pixels! Pull-down menus! Graphics and text mixed together! And the only demo program was what made the whole experience click: MacPaint.

I remember when the Atari 520ST was announced. Half a megabyte of memory! Staggering amounts of power for less than $1000! A Mac-like interface but in full color! Some of the demos were simple slideshows of 16-color 320x200 images, done with a program called NeoChrome, but I had never seen anything like them before.

I remember when the Amiga debuted that same year. Real multitasking! Digitized sound! Stereo! Hardware for moving around big bitmaps instead of just tiny sprites! Images showing thousands of colors at once! Just the bouncing ball demo was outside what I expected to ever see on a computer. And there was a flight-sim with filled polygon graphics. Behind the scenes it was the fancy hardware enabling it all, but it was the optimism and feeling of new possibilities that fueled the excitement.

I remember when the Macintosh II came out, with 24-bit color and impossibly high display resolutions for the time. It seemed like a supercomputer on a desk, the kind of thing that only high-end graphics researchers would have previously had access to.

PCs never hit me so unexpectedly and all at once, but there were a few years when 3D hardware started appearing where it felt like the old rules had been thrown out and imagining the future was more important than looking back on the same set of ideas.

Am I going to buy an iPad? I don't know yet. I never bought most of the systems listed above. But I am glad I've been experiencing that old optimism caused by a mix of hardware and software that suddenly invalidates many of the old, comfortable rules and opens up territory that hasn't been endlessly trod upon.

Process-one Blogs: Google Wave XMPP notification gateway

Thu, 28 Jan 2010 10:00:35 +0000

At ProcessOne, we have developed an experimental notification Google Wave XMPP gateway.

I wrote it because, in the current state of Wave usage, my contacts always had to write me an XMPP message or send me an email to explain that I have content available.

I felt this was for me a large deterrent for using it and quite paradoxical in a real-time web world.

To solve this problem and benefit from the largely used XMPP protocol for notification, I wrote a Google Wave gateway. For now, it is quite raw because it relies on GWT client protocol and lacks any mechanism from Google to do delegated authentication. However, we are going to implement a more standard client protocol and delegated authentication as Google makes those features available.

Anyway, here is a small demo of how to use it to give you an idea of the user experience.

It is deployed on talkr.IM server, so you can give it a try.

Enjoy!

Process-one Blogs: Talkr.IM service update: Google Wave gateway

Wed, 27 Jan 2010 10:30:37 +0000

The Talkr.IM XMPP/Jabber service will undergo today a service update, adding a gateway to Google Wave.

ProcessOne's public XMPP service Talkr.IM will receive a new feature for all users: a Google Wave gateway. This gateways will enable Talkr.IM XMPP users to be notified of modified Waves.

Our Google Wave gateway will notify the subscribed users of all the changes in a Wave, by providing an direct link to a maximized Wave window. The other windows (Navigation, Contacts, Search) will be minimized.

We are planning a service interruption at:

10:00 UTC
11:00 Paris
13:00 Moscow
02:00 Los Angeles
05:00 New York
19:00 Tokyo

We will remind our online users a few minutes before.

Update: The GWave transport is online, feel free to test and play with it! Warning, this is an alpha software, you my meet bugs.

Caoyuan's Blog: Progress of Migrating AIOTrade to Scala #2

Mon, 25 Jan 2010 10:17:34 +0000

My next step is to push AIOTrade to another level of UE, with hotkey for symbols, indicators etc, with better UI.

So far, my Scala adventure went almost smoothly, I'm thinking about the data exchange between client and data source, maybe the actors from Scala, LiftWeb or AKKA will bring some fresh atmosphere.

LShift on Erlang: Plugin exchange types for RabbitMQ

Fri, 22 Jan 2010 14:38:54 +0000

An obvious extension point for an AMQP broker is the addition of new types of exchange. An exchange type essentially represents an algorithm for dispatching messages to queues, usually based on the message’s routing key, given how the queues are bound to the exchange — it’s a message routing algorithm.

At a minimum, supporting new exchange types requires only some scaffolding to plug in to (an exchange type registry) and a hook for routing messages. However, this wouldn’t support some more interesting use cases, and in particular it didn’t support our motivating use case. Exchange types that want to keep their own state need to be initialised, and be notified about other lifecycle events.

The ~~branch bug22169~~ default branch of RabbitMQ supports plugin exchange types, by providing a behaviour for exchange type modules to implement, and an exchange type registry to map a module to a type (i.e., what the client supplies in the type field of exchange.declare).

The behaviour requires exported hooks for validating exchange declarations, creating exchanges, recovering durable exchanges, publishing to an exchange (this is where the routing comes in), maintaining bindings, and deleting an exchange. RabbitMQ continues to maintain the database of exchanges and bindings, and calls the hooks after it’s done its own bookkeeping.

For simplicity, the hooks are not called atomically with the bookkeeping; so, it is possible for instance to publish to a new exchange for which the hook has not completed. However, provided there are no asynchronous operations in the hook implementation, the hook will have completed by the time the OK message is sent to the client. This is no more racey than AMQP itself, with consistency at the channel level where operations follow a single thread of control.

Here’s an example that simply io:formats things as they happen:

-module(rabbit_exchange_type_debug).
-include("rabbit.hrl").

-behaviour(rabbit_exchange_type).

-export([description/0, publish/2]).
-export([validate/1, create/1, recover/2, delete/2, add_binding/2, remove_bindings/2]).
-export([register_debug_types/0]).
-include(”rabbit_exchange_type_spec.hrl”).

-rabbit_boot_step({debug_exchange_types, 
                   [{description, "debugging exchange types"},
                    {mfa, {?MODULE, register_debug_types, []}},
                    {requires, rabbit_exchange_type_registry},
                    {enables, exchange_recovery}]}).

description() ->
    [{name, <<"debug">>},
     {description, <<"Debugging exchange">>}].

backing_module(#exchange{ type = Type }) ->
    %% Presume that Type is EITHER one of the standard types –
    %% i.e,. that we have been registered this module as direct,
    %% topic, fanout or match — or, for testing purposes, it’s
    %% registered (as in the boot steps above) as debug_direct, etc.
    Type1 = case atom_to_list(Type) of
                “x-debug-” ++ T -> T; 
                              T -> T
            end,
    list_to_existing_atom(”rabbit_exchange_type_” ++ Type1).

publish(Exchange, Delivery) ->
    io:format(”Publish ~p to ~p~n”, [Delivery, Exchange]),
    Module = backing_module(Exchange),
    Module:publish(Exchange, Delivery).

validate(X) ->
    io:format(”Validate ~p~n”, [X]),
    (backing_module(X)):validate(X).

create(X) ->
    io:format(”Create ~p~n”, [X]),
    (backing_module(X)):create(X).

recover(X, Bs) ->
    io:format(”Recover ~p with bindings ~p~n”, [X, Bs]),
    (backing_module(X)):recover(X, Bs).

delete(X, Bs) ->
    io:format(”Delete ~p with bindings ~p~n”, [X, Bs]),
    (backing_module(X)):delete(X, Bs).

add_binding(X, B) ->
    io:format(”Add binding ~p to ~p~n”, [B, X]),
    (backing_module(X)):add_binding(X, B).

remove_bindings(X, Bs) ->
    io:format(”Delete bindings ~p from ~p~n”, [Bs, X]),
    (backing_module(X)):remove_bindings(X, B).

register_debug_types() ->
    lists:foreach(
      fun (T) ->
              rabbit_exchange_type_registry:register(T, ?MODULE)
      end,
      [<<"x-debug-direct">>,
       <<"x-debug-topic">>,
       <<"x-debug-fanout">>,
       <<"x-debug-headers">>]).

Bit by important bit:

-behaviour(rabbit_exchange_type).
-export([description/0, publish/2]).
-export([validate/1, create/1, recover/2, delete/2, add_binding/2, remove_bindings/2]).

rabbit_exchange_type specifies these exported callbacks.

-include("rabbit_exchange_type_spec.hrl").

This include has the specs for each of the exported functions, if you’re using specs.

-rabbit_boot_step({debug_exchange_types, 
                   [{description, "debugging exchange types"},
                    {mfa, {?MODULE, register_debug_types, []}},
                    {requires, rabbit_exchange_type_registry},
                    {enables, exchange_recovery}]}).

This uses the new boot sequence mechanism to register the exchange type during boot. The “enables” and “requires” say that the function given as mfa above must be run after the exchange type registry is available, but before any exchanges are recovered.

publish(Exchange, Delivery) ->
    io:format("Publish ~p to ~p~n", [Delivery, Exchange]),
    Module = backing_module(Exchange),
    Module:publish(Exchange, Delivery).

This exchange type simply delegates to a “backing” exchange type.

register_debug_types() ->
    lists:foreach(
      fun (T) ->
              rabbit_exchange_type:register(T, ?MODULE)
      end,
      [<<"x-debug-direct">>,
       <<"x-debug-topic">>,
       <<"x-debug-fanout">>,
       <<"x-debug-headers">>]).

rabbit_exchange_type_registry maintains a registry of type to module; because of this indirection, we can register this module as many different types, then check the declared type of the exchange in our hook to see which type we’re expected to be. Note that the AMQP specification requires the “x-” prefix for non-standard exchange types.

~~This should reach default branch soon after RabbitMQ 1.7.1 is released.~~ This is in RabbitMQ’s default branch, and should be in the next release (1.7.2).

You can drop modules straight into src, but they are better packaged as plugins — follow the drill at the plugin development guide (and note that your plugin may only need to be a library application).

[EDITS: Updated sample code to keep up with name changes, and correct use of list_to_atom to list_to_existing_atom] [Further EDITS: slight API change, now in default branch]

Process-one Blogs: Talkr.IM service update: gateways

Fri, 22 Jan 2010 10:10:42 +0000

The Talkr.IM XMPP/Jabber service will undergo today a service update, adding gateways to legacy IM systems.

ProcessOne's XMPP service Talkr.IM will receive new features for all users: IM gateways. These gateways will enable XMPP users to join legacy IM systems with their existing accounts. This will seamlessly merge the legacy IM contact lists to the main XMPP contact list ("roster"). Of course the main features are presence and chat, but these gateways also enable the status exchange, as well as the avatar, and much more.

We had already deployed an IRC gateway, the new gateways (or "transport" in XMPP terms) are the following:

AIM
ICQ
MSN/WLM
Yahoo!

This adds IM contact points to your addressbook, and enable the inclusion of millions of legacy IM users to XMPP users.

We are planning a service interruption at:

13:00 UTC
14:00 Paris
16:00 Moscow
05:00 Los Angeles
08:00 New York
22:00 Tokyo

We will remind our online users a few minutes before.

Update: our service update is done, you can now register your legacy IM accounts and play with them.

Process-one Blogs: [ANN] ejabberd 2.1.2 bugfix release

Mon, 18 Jan 2010 16:03:16 +0000

We are pleased to announce ejabberd 2.1.2, which contains several bugfixes over last month's maintenance release.

Brief summary of changes:

Fix SASL PLAIN authentication message for RFC4616 compliance
Fix support for old Erlang/OTP R10 and R11
If server start fails due to config file problem, display some lines and stop node
PubSub and PEP: several improvements and bugfixes
WebAdmin: fix offline message displaying
When server stops with new stop_kindly command: inform users, clients, MUC

Check the Release Notes for a more complete list of changes:
http://www.process-one.net/en/ejabberd/release_notes/release_note_ejabberd_2.1.2

If you upgrade from ejabberd 2.0.5 or older, read carefully the release notes of ejabberd 2.1.0 too, because there were several changes in the installation path and the configuration options.

The list of solved tickets since previous version is available on ProcessOne bug tracker:
http://redir.process-one.net/ejabberd-2.1.2

ejabberd 2.1.2 is available as source code package and binary installers for Linux 32 bits, 64 bits, Mac OS X Intel, and Windows:
http://www.process-one.net/en/ejabberd/downloads

Caoyuan's Blog: Progress of Migrating AIOTrade to Scala

Sun, 17 Jan 2010 22:24:41 +0000

Well, I've done most parts of migrating AIOTrade to Scala, not all features return yet. I gain lots of experiences of inter-op between Scala and Java, since AIOTrade has to be integrated into an existed Java framework NetBeans Platform. And also, whole project is now managed by Maven instead of Ant, which reduces lots of pain of dependencies upon crossing sub-projects.

This project is now hosted on ~~kenai.com~~ http://sf.net/projects/humaitrader, you can check out the code to get an overview of how to integrated Maven + Scala + NetBeans Modules. Of course, all were done with NetBeans Scala plugin.

LOC of this project so far:

$ ./cloc.pl --read-lang-def=lang_defs.txt ~/myprjs/aiotrade.kn/opensource/
     677 text files.
     617 unique files.
     154 files ignored.
http://cloc.sourceforge.net v 1.08  T=3.0 s (167.7 files/s, 21373.7 lines/s)
-------------------------------------------------------------------------------
Language          files     blank   comment      code    scale   3rd gen. equiv
-------------------------------------------------------------------------------
Scala               353      7981     16301     27180 x   1.36 =       36964.80
Java                 43      1148       833      6946 x   1.36 =        9446.56
XML                 104       231       389      2414 x   1.90 =        4586.60
Bourne Shell          2        81        81       488 x   3.81 =        1859.28
HTML                  1         7        15        26 x   1.90 =          49.40
-------------------------------------------------------------------------------
SUM:                503      9448     17619     37054 x   1.43 =       52906.64
-------------------------------------------------------------------------------

A screen snapshot:

Dukes of Erl: Minor Erlang Interface Tricks

noreply@blogger.com (Paul Mineiro) — Thu, 14 Jan 2010 12:41:28 +0000

erlrc requires that you drop a file whose name is your node name and whose contents is the node cookie into a particular directory so that the packaging system can find running Erlang VMs and ask them to do hot-code upgrades. It's not that hard, but I figured I would put some shell scripts into the erlrc google code project demonstrating how to do this and other minor tricks that make it a little nicer to talk to an Erlang VM from the (non-Erlang) command line.
All of these scripts are driven by a POSIX shell syntax configuration file describing an Erlang VM, which can be pretty short. Here's one I'm using right now for my personal stuff:

% cat /usr/share/myerlnode/myerlnode.rc
node_name_file='/etc/erlrc.d/nodes/erlang'

run_extra_args='+A 64 -noshell -noinput -s crypto -s mnesia -eval "case erlrc_boot:boot () of ok -> ok; _ -> init:stop () end" >erlang.out 2>erlang.err &'

node_name_file (the only required config setting) defines what the node name will be (via the basename). This config file drives four scripts:

erlstart-run-erlang: starts an Erlang VM. doesn't do that much, really: creates the node file for erlrc, and sets the heart command in case you want to use heart.
erlstart-remsh: starts a remote shell on an Erlang VM.
erlstart-eval: takes the argument, evals it on an Erlang VM, and prints the result to standard out. very useful shell script glue for maintenance scripts.
erlstart-etop: runs etop on an Erlang VM.

If you put your config file in the default location (/etc/erlstart.rc) or export an environment variable indicating the location (ERLSTART_CONFIG_FILE) then things are pretty zero-configuration. Doing this kind of thing at the (non-Erlang) shell gets very addictive:

% erlstart-eval 'application:which_applications()'
[{anwhereos,"Rest API for time series persistence and retrieval.","0.1.0"},
{drurlyjsclientsrv,"Serve the drurly jsclient from the drurly server.",
                "0.0.1"},
{drurly,"Social sharing server.","2.2.0"},
{mcedemo,"TinyMCE + Nitrogen demo.","1.3.0"},
{inets,"INETS  CXC 138 49","5.0.12"},
{mochiweb,"MochiWeb is an Erlang library for building lightweight HTTP servers.",
       "0.2009.05.26"},
{nitrogen,"Nitrogen web framework for Erlang.","0.2009.05.12.3"},
{nitromce,"A Nitrogen element which corresponds to a TinyMCE editor instance.",
       "4.0.1"},
{sgte,"String template language for Erlang.","0.7.1"},
{signzor,"Erlang library to generate signed printable encodings.","0.0.1"},
{tcerl,"Erlang driver for tokyocabinet.","1.3.1h"},
{webmachine,"An Erlang REST framework.","0.2009.09.24b"},
{erlrc,"Extensible application management.","0.2.3"},
{mnesia,"Mnesia storage API extensions.","4.4.7.6.1"},
{crypto,"CRYPTO version 1","1.5.3"},
{sasl,"SASL  CXC 138 11","2.1.5.4"},
{stdlib,"ERTS  CXC 138 10","1.15.5"},
{kernel,"ERTS  CXC 138 10","2.12.5"}]

erlstart-run-erlang is fairly low level, so for an init script I would do something like the following: first, a little stub installed into /etc/rc.d

#! /bin/sh

# chkconfig: 2345 20 80
# description: Control my personal Erlang node.

exec myerlnodectl "$@"

and then the actual guts installed as myerlnodectl

#! /bin/sh

eval_with_main_node () \
{
  erl -name myerlnodetmp$$ \
      -hidden \
      -setcookie "$cookie" \
      -noshell -noinput \
      -eval "MainNode = list_to_atom (\"$1\"), $2" \
      -s erlang halt
}

get_hostname () \
{
  erl -name myerlnodetmp$$ -setcookie $$ -noshell -noinput -eval '
    [ Host ] = tl (string:tokens (atom_to_list (node ()), "@")),
    io:format ("~s~n", [ Host ])
  ' -s init stop
}

id=`basename "$0"`

if test -d /root
then
  HOME=${HOME-/root}
else if test -d /var/root
  then
    HOME=${HOME-/var/root}
  fi
fi
export HOME

ERLSTART_CONFIG_FILE=${ERLSTART_CONFIG_FILE-/usr/share/myerlnode/myerlnode.rc}
export ERLSTART_CONFIG_FILE

. "$ERLSTART_CONFIG_FILE"

cookie=${cookie-turg}
user=${user-erlang}
hostname=${hostname-`get_hostname`}
node_name=`basename "$node_name_file"`
full_name="$node_name@$hostname"
shutdown_file=${shutdown_file-/var/run/myerlnode.shutting_down}

ERL_CRASH_DUMP=${ERL_CRASH_DUMP-/dev/null}
export ERL_CRASH_DUMP

case ${1-"status"} in
start)
  test "`id -u`" -eq 0 || exec sudo $0 "$@"

  printf "Starting Erlang... "

  if test -f "$node_name_file" &&                                     \
     test true = "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "already started."
      exit 0
    fi

  pid=`eval_with_main_node "$full_name" '
         io:format ("~p", [
           case rpc:call (MainNode, os, getpid, []) of
             { badrpc, _ } -> undefined;
             Pid -> list_to_integer (Pid)
           end ])'`

  test "$pid" -gt 0 2>/dev/null && {
    test -f "$shutdown_file" && {
      oldpid=`cat "$shutdown_file"`
      test "$pid" -eq "$oldpid" && {
        echo "shutdown in progress (pid = '$pid')." 1>&2
        exit 1
      }
    }
  }

  rm -f "$shutdown_file"

  # check for -s shell support
  su -l -s /bin/sh $user -c true >/dev/null 2>/dev/null

  if test $? = 0
    then
      dashs="-s /bin/sh"
    else
      dashs=""
    fi

  ${niceness+ nice -n $niceness}                                      \
  su -l $dashs "$user" -c                                             \
    "env cookie=\"$cookie\" erlstart-run-erlang \"$ERLSTART_CONFIG_FILE\""

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case net_adm:ping (MainNode) of
                      pong ->
                        ok;
                      pang ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1)
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: could not connect to node after 10 seconds" 1>&2
    exit 1
  }

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case rpc:call (MainNode, init, get_status, []) of
                      { started, _ } ->
                        ok;
                      { starting, _ } ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1);
                      { Status, _ } ->
                        { failed, Status }
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: node did not boot after 10 seconds" 1>&2
    exit 1
  }

echo "done."
;;

stop)
  test "`id -u`" -eq 0 || exec sudo $0 "$@"

  printf "Stopping Erlang... "

  if test ! -f "$node_name_file" ||                                   \
     test true != "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "not running."
      exit 0
    fi

  pid=`erlstart-eval 'os:getpid ()' 2>/dev/null`

  test "$pid" -gt 0 2>/dev/null && {
    printf '%s' $pid > "$shutdown_file"
    chown $user:$user "$shutdown_file"
  }

  erlstart-eval 'init:stop ()' >/dev/null 2>/dev/null

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case net_adm:ping (MainNode) of
                      pong ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1);
                      pang ->
                        ok
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: node still responsive after 100 seconds" 1>&2
    exit 1
  }

  rm -f "$node_name_file"

  echo "done."
  ;;

status)
  if test -f "$node_name_file" &&                                     \
     test true = "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "Erlang is running"
    else
      echo "Erlang is not running"
    fi

  ;;

*)
  echo "$id: unknown command $1" 1>&2
  exit 1
esac

exit 0

Hopefully you found that inspirational.

Ulf Wiger: QCon 2010 Concurrency Track

Wed, 06 Jan 2010 16:01:40 +0000

I will be hosting the “Concurrency Challenge” track at QCon 2010. The speakers list is now ready with all titles and abstracts in place.

I am quite pleased with the mix and the dignity of the speakers. One of the things we wanted to highlight is that there is no one-size-fits-all concurrency model, and it’s time for us to start being a little bit more specific about what kind of concurrency we have in mind.

Each setting brings its own set of challenges, and in this track, you can expect to get a good orientation from some brilliant people, who know how to tell a story and are not afraid of offering their opinion.

Process-one Blogs: ProcessOne, a year in review

Mon, 04 Jan 2010 13:00:12 +0000

At ProcessOne, we wanted to wish you all a very happy 2010 year. To give you a good sense of all what has been accomplished, we have written this small review of the 2009 year at ProcessOne. We hope you will enjoy the reading.

Software

First, we have released ejabberd 2.0.3 to 2.0.5, and we finally made it with ejabberd 2.1 (with its bugfix 2.1.1). It has received much scalability improvements, as well as a big PubSub boost. Even more changes have made it an even more rock-solid and featureful real-time server. A STUN server has been added in ejabberd, for multimedia purposes. The version 3.0 is in the works, based on the opensource exmpp library, freshly released this year also.

On the desktop, OneChannel, the PubSub client working on AIR, has been released, optionally using the xmpp-sandbox.org preconfigured real-time feeds.

Our gateway packaging is stable, scalable and works pretty well, covering Microsoft Live Messenger, Yahoo! IM, AIM and ICQ, XMPP, and also non-IM systems, like Twitter and Wave gateway.

On the mobile front, OneTeam for iPhone, the mobile XMPP client, has been released in versions 2.1, 3.0, 3.1, and 3.2, and we believe it is the best XMPP client out there, with room for improvements and innovations in the coming year.

OneWeb, the extension for Firefox, has been released as an alpha product, with a Fennec port (Firefox mobile). It basically adds data sync and remote control between browsers, demoing the power of XMPP in the browser.

Our supervision console TeamLeader has had many improvements in 2009 and is now clean and efficient, being deployed on large customer projets.

Our XMPP desktop client OneTeam for Firefox has become a strong client and should become a major component in our offering in 2010.

We also are offering a module for the APNS, or Apple Push Notifications Service, for the iPhones.

Announced

We have announced and demoed the OneTeam Media Server, our Flash server implementation, acting as a media relay and much more, for multimedia applications.

The ProcessOne Wave Server is in the work, gaining features and improving fast. A demo on Tkabber has been done, showing very basic client/server interactions, like create a wavelet, join and edit it, the basics of collaborative real-time editing of a text document.

Our OCS Bridge enables a real federation of ejabberd servers with the Microsoft's IM+VoIP corporate servers, while this one only painfully offers gateways to XMPP since this year.

Services

We have given birth to a number of services:

We have opened the ProcessOne Labs, showing all our non-production ready, but promising products.
Hosted.IM offers for XMPP hosting services for small businesses.
IMstore enables customers to buy products and services, the fast and easy way, on the web via a simple form.
The APNS, or Apple Push Notifications Services, runs on our infrastructure for OneTeam for iPhone clients using our OneTeam.IM and Talkr.IM services.
Tweet.IM is a XMPP to Twitter gateway, offering nice features like delivery modes.
xmpp-sandbox.org is the XMPP service for developers.
Talkr.IM is our XMPP service for the masses.

Events, Talks & Articles

Here are the places where we have been, and what has been said about us:

In December, we have organized the Sea Beyond event, for real-time communications.
The concept of "Web Augmented Reality" came to birth.
A Strategic Guide to Instant Messaging and Security has been published.
Facebook has been caught working on XMPP with ejabberd.
id Software's QuakeLive online game is using ejabberd for its ingame chat infrastructure.
Mickaël Rémond went to Mobile World Congress, Erlang Factory in San Jose and London, XMPP Summit in Brussel and Palo Alto, Erlang User Conference in Stockholm, Le Web in Paris.
We have been covered by Datamonitor and Giga OM.
The series "Scalable XMPP bots" has been published, promoting the use of exmpp (I, II, III).

Roadmap 2010

In 2010, our focus will be mainly on the following points:

Hosted.IM will be offering new simple and extendable services.
OneTeam for Firefox will be released, with WAR (Web Augmented Reality), and much more innovative features.
ejabberd will become more industrial, with 3.0 getting to production ready state, better TeamLeader support, good test suite coverage and more modules on the IMstore.
Talkr.IM will offer the seamless migration from/to another XMPP service.

Conclusion

It has been a busy year, with many achievements, much of which we cannot talk about, or we have just forgotten. But the coming year 2010 will be even more exciting.

Me Dev, You Jane: Erlang Quick Tip: The user_default module

Sun, 03 Jan 2010 15:09:38 +0000

If you are anything like me you spend a lot of your time in the erlang shell debuging and testing your code. And some commands are used more than others but they may be a bit on the long side and you may make regular errors while typing it. One of these wonderful functions is make:all([load]). that will compile using your Emakefile and then magically reload the newly compiled modules into the running shell. What I reglarly do is to type load instead of [load].

Enter the user_default module. Any function defined in the user defined user_default module or in the Erlang defined shell_default module will be allowed to run without typing in the module name. One of these functions that you may use often is the c(module\_name). function. First start by creating a directory for your utility modules. I like having it in the ~/.ebin/ diretory. Now add that direcory to the code path by opening or creating the file ~/.erlang and inserting this line:

code:add_pathz("/Users/username/.ebin").

Create your user_default.erl file and but in the following:

-module (user_default).
-export ([sync/0]).

%% Compiles all files in Emakefile and load into current shell.
sync() ->
  make:all([load]).

Then compile it with erlc user_default.erl and start an Erlang shell. Now inside a project you can run sync(). and it will recompile and load you project.

For those interested I will put up my ~/.ebin code up at github.com/JonGretar/erlang_user_utilities. Please fork and make your own changes and share with the world.

Joe's Blog: Fun with the CouchDB _changes feed and RabbitMQ.

Fri, 01 Jan 2010 23:14:01 +0000

I was recently introduced to yajl-ruby, ruby bindings to the C based yajl json parsing/encoding libraries. After discovering that it can parse HTTP streams it seemed like it would be a perfect fit for use with CouchDB. A while back I wrote some code to push update notifications to RabbitMQ and a commenter mentioned using the _changes feed instead. Combining the _changes feed and yajl-ruby’s HttpStream seemed like a good way to do it.

The _changes feed is a running list of all the documents that have changed in a database listed in order by sequence number. This is similar to update notifications but gives more information such as the document IDs and is HTTP based (with multiple feed styles) rather than stdout. Additionally you can create design document filters which can be specified as a query parameter to give you only the parts of the feed you want. All in all _changes is a pretty powerful feature.

Now for the fun stuff, the code. There are a few dependencies I used to do this, specifically focused on making it fast. As such I used EventMachine based libraries for AMQP and HTTP requests. The first bit of code takes the _changes feed for the “test” database, parses the feed, uses the document ID to request that document and publish it to the queue. One key item to note is that this code requires the latest yajl-ruby from github to run properly. Additionally, this works nicely with feed=continuous so it grabs the documents as they are changed without a need for polling.

Note that there is a variable for since, this allows you to start from a specific sequence number so you can skip over old changes.

The next bit of code works from the other side of the queue. It subscribes to the queue, parses the JSON, performs some operations on it and puts the results back into another CouchDB database called “results”.

What could it be used for? My first thought is some sort of parallel computation, boot up a few dozen EC2 nodes and start dumping data into CouchDB. Have all those nodes pop messages off the queue, process them and dump the results back into Couch. Legitimately one could chain these together to process the results again. The queue ends up being a simple job management system with the EC2 nodes popping new messages as they finish processing them. With a little bit of work, features and the right use case I think could be a pretty powerful system.

Check out the code, my other projects and follow me on twitter @williamsjoe.

[edit: made a slight improvement to changes_sub.rb on 20100107]

Anders Conbere's Journal: Atom feeds for igor... difficult

Thu, 31 Dec 2009 21:17:20 +0000

The first problem was actually reading the atom specs which I had neglected to do. Most of my previous work with atom had been in either creating elements used in various locations (Google data is all passed as Atom elements, as well as moments of injecting atom elements into xmpp streams). So I cobbled together what I thought was a decent first attempt at a Jinja template for building the atom file, published it and redirected my browser there... only to see nothing.

The XML was being generated, but not parsed.

The problems left were:

writing out dates in RFC:3339 format
building correct id's for elements

for RFC 3339 I found a nice python library in the public domain for producing the correct format from a datetime object. To you Henry Precheur I am quite grateful.

Generating correct id's was a little more tricky. First I tried generating uuid's but upon reading Mark Pilgrims post on atom element id's I ended up writing a small function to generate tag: urns based on the time when a post was published (I'm not sure this is a /good/ solution given that I'm working with version control systems and you might legitimately have many articles published at the same time).

In the end I'm fairly content with the result, it required a morning's effort, more time than I had wanted to devote to atom publishing, but still successful.

Erlang Inside: Mustache – GitHub’s framework agnostic templating ported to Erlang

Wed, 30 Dec 2009 14:00:29 +0000

Mustache is an Erlang port of the Ruby framework of the same name, originally written by Chris Wanstrath, one of the founders of GitHub. A powerful feature of this type of framework is the ability to write a ‘template’ that you could use in a Rails application and then reuse in an Erlang application with [...]

Process-one Blogs: Merry Christmas and Happy New Year !

Fri, 25 Dec 2009 09:30:01 +0000

Year 2009 has been for us full of opportunities and allowed us to launch new promising software, like OneChannel the first XMPP publish and subscribe client, and to demonstrate our ability to innovate.

We still have lots of new features, software and improvements in the work: Oneteam Media Server, ProcessOne Wave server, new XMPP client, new online service portfolio, and many other surprises.

Year 2010 is going to be a great year for ProcessOne.

Thank you all for your ongoing support. Let's build the realtime web together !

Dukes of Erl: erlrc and rpm

noreply@blogger.com (Paul Mineiro) — Wed, 23 Dec 2009 15:12:51 +0000

Two of the Dukes now work at OpenX, which is starting to dip its toe into the Erlang waters. They use CentOS so they agreed to fund porting framewerk and erlrc to rpm; previously I'd only used them with deb.

A bit of background: erlrc is a set of Erlang modules and shell scripts that are designed to be integrated into packaging hooks so that installation, upgrade, or removal of a package causes corresponding hot-code activity to happen inside registered Erlang VMs on the box. Since erlrc was designed to easily integrate with multiple package managers, getting it to work with rpm was mostly about me understanding rpm's package hooks model. The result is an experience like this,


% sudo yum -q -y install lyet
erlrc-start: Starting 'lyet': (erlang) started
% sudo yum -q -y remove lyet
erlrc-stop: Stopping 'lyet': (erlang) unloaded

i.e., installing an rpm causes a running Erlang VM on the box to hot-code load the new modules and start the associated application, and removing the rpm causes the associated application to be stopped and the corresponding modules to be hot-code unloaded.

If you use fw-template-erlang than the appropriate packaging hooks are added for you automatically, both for deb and now rpm. However even manual creation of rpm spec files is pretty easy:

erlrc-stop is called in %preun if the installation count indicates removal
erlrc-upgrade is called in %preun if the installation count indicates upgrade
erlrc-start is called in %posttrans

Also, the erlrc shell scripts want to know the previously installed version, so I call rpm -q in a %pretrans hook and save the result. Longer term, erlrc should probably ask the Erlang VM it is talking to what version is running to eliminate the need for this argument (I was a bit surprised that rpm doesn't provide this argument to the package hook like debian does; it seems very useful for creating version-specific upgrade fixes).

Process-one Blogs: [ANN] exmpp 0.9.2 new release

Wed, 23 Dec 2009 14:56:24 +0000

We are pleased to announce a new release of exmpp.

exmpp is a XMPP library written in Erlang and released under the Erlang Public License. It helps the development of XMPP entities by providing functions to build and interpret XML stanzas. Once compiled, check the generated HTML documentation.

List of main changes since the previous release six months ago:

Adding support for stream error stanzas in exmpp_session when logged in
Autodetect compilation parameters for Mac OS X Snow Leopard
Component support for exmpp, and some fixes
Fix segmentation fault with exmpp and expat
New functions exmpp_client_pubsub:discover_nodes/1 and 2
Stringprep Bidi checking doesn't reject strings with RandALCat and LCat characters
queryns was not defined for IQ packets in received_packet record

The API of exmpp 0.9.2 is considered to be stable, so there aren't any
major changes expected in the API before reaching 1.0.0.

exmpp home page:
http://exmpp.org/

Download exmpp 0.9.2 source code package from:
http://download.process-one.net/exmpp/

You can also check the ProcessOne Labs page:
http://www.process-one.net/en/labs/

LShift on Erlang: Merry Christmas: Toke — Tokyo Cabinet driver for Erlang

Mon, 21 Dec 2009 17:01:01 +0000

Tokyo Cabinet is a rather excellent key-value store, with the ability to write to disk in a sane way (i.e. not just repeatedly dumping the same data over and over again), operate in bounded memory, and go really fast. I like it a lot, and there’s a likelihood that there’ll be a RabbitMQ plugin fairly soon that’ll use Tokyo Cabinet to improve the new persister yet further. Toke is an Erlang linked-in driver that allows you to use Tokyo Cabinet from Erlang.

There is already a Tokyo Cabinet driver for Erlang, tcerl, however, I couldn’t make it work: even after fixing the C so that it compiles (I hit this bug), I still couldn’t make it work. Inspecting the code, I get the feeling it’s bit-rotted - the Tokyo Cabinet API has moved on, and tcerl hasn’t kept up.

The other issue with tcerl is that it’s not a linked-in driver. Erlang allows two different types of drivers: the first are external C programs — these have a main() and run in their own process. Communication is done by stdin/stdout. These are a bit safer because if they crash they don’t take out the Erlang VM, but they’re never going to be blazingly fast. Toke, on the other hand, is a fully linked-in driver. It dynamically links with the Erlang VM, exists in the same address space and goes as fast as it possibly can (using the Erlang driver callbacks which avoid all copying of data passed from the Erlang). My tests show it’s about three times slower driving Tokyo Cabinet from Erlang via Toke, than driving it natively through C (which is quite good: some googling suggests both the Ruby and Python bindings to Tokyo Cabinet are rather slower). Toke is also about twice as slow as the Erlang ets module, which is in-memory only.

Toke only implements the Tokyo Cabinet hash table (tchdb*) functions, and doesn’t even support all of those: I only wrapped exactly what I needed. You’ll want to read the documentation for Tokyo Cabinet for these. The functions implemented are as follows (refer to the Tokyo Cabinet documentation to explain these further):

toke_drv:new/1 — Set up the driver with a new TCHDB object.
toke_drv:delete/1 — Destroy the driver’s TCHDB object.
toke_drv:tune/5 — Tune the driver’s TCHDB Object.
toke_drv:set_cache/2 — Set the number of records to cache.
toke_drv:set_xm_size/2 — Set the extra amount of memory mapped in.
toke_drv:set_df_unit/2 — Set the steps between auto defrag.
toke_drv:open/3 — Open a db.
toke_drv:close/1 — Close an open db.
toke_drv:insert/3 — Insert. If the key already exists, value is updated.
toke_drv:insert_new/3 — Insert new. If the key already exists, the old value is silently kept.
toke_drv:insert_concat/3 — Concatenate the supplied value with an existing value for this key.
toke_drv:insert_async/3 — Asynchronously insert. If the key already exists, value is updated.
toke_drv:delete/2 — Delete a key from the db.
toke_drv:get/2 — Fetch a key from the db. Returns ‘not_found’ on occasion.
toke_drv:fold/3 — Fold over every value in the db. This internally uses the iteration functions. It’s just wrapped up as a fold to make it appear more functional.
toke_drv:stop/1 — Stop the driver and close the port.

You should be able to use Mercurial to clone it:

# hg clone http://hg.opensource.lshift.net/toke/

Make sure you have Tokyo Cabinet installed (ideally from source. If you’re using a package, make sure you have the development headers available. If you do compile from source, make sure you ldconfig to make your system pick up the new library once it’s installed.). Then it should just be a case of make. There’s also a make run target that starts up an Erlang shell with the paths set up correctly for testing Toke:

toke# make run
erl -pa ebin +K true +A30
Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:30] [hipe] [kernel-poll:true]

Eshell V5.7.4  (abort with ^G)
1> toke_test:test().
passed
2>

Toke is licensed under the MPL. As ever, feedback is very welcome, as are patches!

RedHotErlang: Websockets and Yaws

Fri, 18 Dec 2009 22:17:31 +0000

For years I've wanted to start tech blogging, however not having my own RSS/blogging software always put me off. After all, what lame webserver hacker am I, blogging on someone elses RSS software. Now, finally my friend Tobbe Tornquist did ...