Erlang/OTP News feeds

Welcome to Planet Trapexit. On this page, you will find the latest entries from all the news feeds currently in our database. If you know feeds that would be suitable here (that is, that relate to Erlang/OTP in any way, please add them here.

July 02, 2009

Caoyuan's Blog

Scala Plugin Version 1 for NetBeans 6.7 Released

>>> Updated on July 1, 2009

There are couple of reports on "Could not connect to compilation daemon.", mostly under Windows OS. It's a known issue, to resolve it, please check the following:

  1. SCALA_HOME is set to a fresh installed Scala runtime
  2. PATH includes $SCALA_HOME/bin
  3. If the build task still complain, try to run "fsc" or "scala" in a command window first

===

I'm pleased to announce the availability of Scala plugin version 1 for NetBeans 6.7

What's new:

  • Use fsc instead of scalac as the project building compiler (If you've set SCALA_HOME, make sure $SCALA_HOME/bin is included in your PATH environment).
  • Fixed setting breakpoint in closure statement.
  • A basic import-fixer (Right click on source, then choose "Fix Imports").
  • Code assistant for local vars and functions.
  • Run/Debug single file.

To download, please go to: https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=256544&release_id=686747

For more information, please see http://wiki.netbeans.org/Scala

Bug reports are welcome.

It works on NetBeans 6.7 RC1 or above.

by dcaoyuan at July 02, 2009 12:15 AM

July 01, 2009

Caoyuan's Blog

Rats! Plugin for NetBeans#1: Syntax Highlighting

I've used Rats! parser generator heavily on Scala/Erlang plugin for NetBeans, but not write a plugin for Rats! itself.

So I spent my weekend days on a simple Rats! editor module, which implemented syntax highlighting. It's built on Scala.

Here's the snapshot:

nn

It will be available in couple of days.

by dcaoyuan at July 01, 2009 09:42 PM

Joe's Blog

Introducing haproxy_join, and how to use it with Chef.

Inspired by Holger Just’s haproxy configuration tool I decided to write one my own that worked better for my setup and haproxy_join was born. It’s a simple Ruby script that allows you to break up a monolithic haproxy configuration file in to pieces.

haproxy_join expects files and directories to be in the following scheme:

HAPROXY_PATH/conf/global.cfg (file)
HAPROXY_PATH/conf/defaults.cfg (file)
HAPROXY_PATH/conf/frontend.cfg (file)
HAPROXY_PATH/conf/frontend.d (dir of frontend configs)
HAPROXY_PATH/conf/backend.d (dir backend configs)

The HAPROXY_PATH and resulting configuration file are specified when running the haproxy_join command. On most systems it would look like the following, all you need to do is break up your current config and put it in the above structure.

haproxy_join haproxy.cfg /etc/haproxy/

haproxy_join will also attempt to backup your configuration file before writing a new one.

This works great in tandem with a tool like Chef, allowing you to have Chef manage each small configuration file with a template and haproxy_join to concatenate them together each time they are changed. You can achieve this by using a Chef recipe based on the default opscode haproxy recipe and a slightly modified haproxy init script based on Holger Just’s haproxy init script. The recipe will notify haproxy Chef to restart haproxy if a configuration has changed and the init script will run haproxy_join before it restarts haproxy. I have posted an example of the cookbook recipe and the init script. Neither of these have been heavily used/tested so try them out before you put them into production.

Hope this helps anyone with large haproxy configurations. Let me know if you have any questions in the comments.

by joe at July 01, 2009 09:24 PM

Simon Willison's Weblog

PubSub-over-Webhooks with RabbitHub

PubSub-over-Webhooks with RabbitHub. RabbitMQ, the Erlang-powered AMQP message queue, is growing an HTTP interface based on webhooks and PubSubHubBub.

July 01, 2009 08:22 PM

Dukes of Erl

osmos

I just released the first version of osmos, a pure Erlang library that provides on-disk ordered set tables which allow thousands of updates per second with ACID properties.

It achieves that update rate using a rather different structure from a traditional B-tree based table (like the ones used by most RDBMSs or provided by DBM libraries like BDB or tokyocabinet): an incremental merge sort with user-defined merge semantics.

Motivation

Ordinarily, the rate of updates to an on-disk table is limited by the need to seek around and update an index in place. With a typical seek time on the order of 10 ms, this makes it challenging to scale past about 100 updates/s. Most strategies for going beyond that involve some kind of partitioning, either over multiple disks, multiple machines, or both.

However, a few key observations[1] point to a different strategy:
  1. The reason for updating an index on every write is the expectation that reads are much more frequent than writes, so that read efficiency is the dominating factor. But if writes are far more frequent than reads, you can use some kind of lazy updating to delay the work until absolutely necessary, and combine the work from multiple updates.
  2. An extreme example of a write-dominated, lazily updated database is a full-text inverted index for a search engine. To build one, you might typically read in billions of term-document pairs, sort them by term using some form of external merge sort, and then create a highly optimized index before ever handling a single query.
  3. A merge sort can operate continuously, by injecting new records in sorted batches, and then merging the batches as necessary to maintain a set of sorted files with exponentially increasing sizes. And crucially, this kind of incremental merge sort process can allow relatively efficient searches of the data while it's operating, by binary-searching each sorted file. (An example of this is the incremental indexing provided by Lucene.)
This gives you an ordered set table with a slight increase in the cost of a search (with N records, maybe an extra factor of log N). But the cost of a write is tiny, and mostly delayed: about log N future comparisons during merging, and log N future disk writes, but since all the disk writes are sequential, they will be buffered, and writing to the table requires no explicit seeking at all.[2]

User-defined merging

Things get even more interesting when you let the user control how records are merged. In the osmos model, there is at most one record for any given key; if two records with the same key are encountered during the merge sort, the user's merge function is called to merge the two records into a single record.

The merge function can be any function

  Merge(Key, EarlierValue, LaterValue)  -> MergedValue
that is associative, i.e.,

  Merge(K, Merge(K, V1, V2), V3) =:=
  Merge(K, V1, Merge(K, V2, V3))
for any key K and any consecutive sequence of values V1, V2, V3.

This allows a wide variety of semantics for writing to the table. For example:
  • If the merge function always returns the later value, then a write replaces any previous value, like an ordinary key-value store.
  • If the values are integers, and the merge function returns the sum of the two values, then writing to the table acts like transactionally incrementing a counter.
Similarly, you could use any associative function of two numbers; you could apply such a function to each element of a vector of numbers; or you could apply a different function to each element. For example, to keep a minimum, maximum, and average over some set of keys, you could use something like:

  merge(_K, N1, N2)
      when is_number(N1), is_number(N2) ->
    {min(N1, N2), max(N1, N2), N1 + N2, 2};
  merge(_K, N1, {Min2, Max2, Sum2, Count2})
      when is_number(N1) ->
    {min(N1, Min2), max(N1, Max2),
     N1 + Sum2, 1 + Count2};
  merge(_K, {Min1, Max1, Sum1, Count1}, N2)
      when is_number(N2) ->
    {min(Min1, N2), max(Max1, N2),
     Sum1 + N2, Count1 + 1};
  merge(_K, {Min1, Max1, Sum1, Count1},
            {Min2, Max2, Sum2, Count2}) ->
    {min(Min1, Min2), max(Max1, Max2),
     Sum1 + Sum2, Count1 + Count2}.
This lets you write single numbers as values, but read back either {Min, Max, Sum, Count} tuples (if more than one number has been written for a key) or single numbers (if that was the only value written). To do this with an ordinary key-value table and multiple writers would require expensive transactions, but with osmos, operations like this are no more expensive than replacement, but still ACID.

As you can see, keeping statistics for reporting (when the reports are queried infrequently relative to data collection) is one of the killer applications for a merge sort table.

Among the possibilities for even wackier merge operations are:
  • Always return the earlier value. (What was the first value that occurred for this key?)
  • Take the union of two sets. (What are all the values that have occurred for this key?)
  • Take the intersection of two sets. (What values have always occurred for this key?)
  • Multiply two NxN matrices, e.g., to keep track of a series of rotations applied to a vector in RN.
  • Compose a series of arbitrary operations applied to a space of replaceable objects, e.g.:

      merge(_K, _, V) when ?is_value(V) ->
        V;
      merge(_K, V, Op) when ?is_value(V),
                            ?is_operation(Op) ->
        do_operation(Op, V);
      merge(_K, V, Ops) when ?is_value(V),
                             is_list(Ops) ->
        lists:foldl (fun (Op, V) ->
                       do_operation(Op, V)
                     end,
                     V,
                     Ops);
      merge(_K, Op1, Op2) when ?is_operation(Op1),
                               ?is_operation(Op2) ->
        [Op1, Op2];
      merge(_K, Op, Ops) when ?is_operation(Op),
                              is_list(Ops) ->
        [Op | Ops];
      merge(_K, Ops, Op) when is_list(Ops),
                              ?is_operation(Op) ->
        Ops ++ [Op];
      merge(_K, Ops1, Ops2) when is_list(Ops1),
                                 is_list(Ops2) ->
        Ops1 ++ Ops2.
    The values could be employee records, and the operations could be things like, “change street address to X,” “increase salary by Y%.” Using this pattern, you can get extremely cheap transactional safety for any single-key operation, as long as your merge function implements it.

API

The basic API is quite simple:

  {ok, Table} = osmos:open(Table, [{directory, D}, {format, F}])
to open a table named Table with the format F in the directory D;

  ok = osmos:write(Table, Key, Value)
to write a record to the table;

  case osmos:read(Table, Key) of
    {ok, Value} -> ...;
    not_found   -> ...
  end
to read the record for a key; and

  ok = osmos:close(Table)
to close the table.

You can also iterate over a range of keys in chunks using osmos:select_range/5 and osmos:select_continue/3. The results from a select provide a consistent snapshot of the table, meaning that the results always reflect the contents of the table at the time of the original call to select_range. (In other words, any writes that happen between subsequent calls to select_continue won't affect the results.)

A table format is a record with the following fields:
  • block_size::integer(): block size of the table in bytes, controlling the size of disk reads (which are always whole blocks), and the fanout of the on-disk search trees.
  • key_format::#osmos_format{}: on-disk format for keys. (A pair of functions to convert some set of terms to binaries and back.)
  • key_less::(KeyA, KeyB) -> bool(): comparison function defining the order of keys in the table. Takes two native-format keys, and returns true if the first argument is less than the second argument.
  • value_format::#osmos_format{}: on-disk format for values.
  • merge::(Key, EarlierValue, LaterValue) -> MergedValue: the merge function described above.
  • short_circuit::(Key, Value) -> bool(): function which allows searches of the table to be terminated early (short-circuited) if it can be determined from a record that any earlier records with the same key are irrelevant.
  • delete::(Key, Value) -> bool(): function controlling when records are deleted from the table.
There are several pre-canned formats available from the function osmos_table_format:new/3, or you can build your own #osmos_table_format{} record as needed.

Performance

The file tests/osmos_benchmark_1.erl in the source distribution contains a benchmark that uses variable-length binaries as keys (some more frequent than others, with an average length of 15 bytes), and nonnegative integers as values, encoded in 64 bits, where merging takes the sum of the two values. One process writes random keys and values as fast as it can, while another process reads random keys with a 10 ms sleep between reads.

I ran the benchmark for 15 minutes on a 2.2 GHz dual-core MacBook, and got the following numbers:
  • 5028735 total records written, for an average of 5587 writes/second (including the time to compute random keys, etc.)
  • an average of 109.8 microseconds per write call, which would mean a theoretical maximum write rate of 9111 writes/second (for this table format, machine, etc.)
  • the median time per write call was 30 microseconds, and the 90th percentile was 54 microseconds, indicating that the vast majority of writes are extremely quick
  • an average of 1900 microseconds (1.9 ms) per read call
  • the median time per read call was 979 microseconds, and the 90th percentile was 2567 microseconds

I reran the benchmark for 2 hours on the same machine, and got the following numbers:
  • 17247676 total records written, for an average of 2396 writes/second
  • an average of 329.7 microseconds per write call, for a theoretical maximum write rate of 3033 writes/second
  • the median time per write call was 39 microseconds, and the 90th percentile was 88 microseconds
  • an average of 13076 microseconds (13 ms) per read call
  • the median time per read call was 6081 microseconds, and the 90th percentile was 34094 microseconds
The table had about 400 MB of data on the disk (in 7 files) at the end of the 2-hour run. This shows that read performance does start to suffer a bit as the amount of data on the disk grows, but writes remain very fast. (In fact, if there were no reads competing with writes, I wouldn't expect the write times to degrade even that much, since all that's happening synchronously during a write call is a buffered write to a journal file, and an insert into an in-memory tree.)
[1] I have to thank my good friend Dave Benson for introducing me to these ideas, and the generalized merge sort table. His excellent library GSK provides a very similar table API (GskTable) for people who want to write single-threaded servers in C. (I ripped off several of his design ideas, including prefix compression and the variable-length integer encoding.)
[2] Of course there may be implicit seeking due to the need to access multiple files at the same time. But for sequential access, the OS and disk hardware can mitigate this to a large degree as long as the number of files isn't too large.

by Michael Radford (noreply@blogger.com) at July 01, 2009 03:45 PM

Process-one Blogs

OneTeam 3.0 for iPhone

The new version 3.0 of OneTeam for iPhone has been released, with the push notification feature from Apple Push Notification Service (APNS).

When moving with your iPhone, as a pedestrian, bike rider, car driver or in public transportation, you might have to shutdown the OneTeam for iPhone application while staying connected to your XMPP server. It is convenient for battery saving purposes, or because you want to use another application (iPhone OS 3.0 still does not authorize background applications).

When an event occurs on your XMPP server, it is sent to the APNS (Apple Push Notification Service), which instantly relays it to your iPhone. A push notification is then poped up on your screen, with a sound played. The OneTeam badge (icon) is then changed, showing a new event has arrived, or the number of events waiting for you to read.

image

You simply have to tap the OneTeam icon to read your new event. The reconnection will be fast and seamless.

This feature is very handy, since you can use your iPhone the way you want (browse the web, use another application) while still being instantly notified as if you were using OneTeam on the background.

This push notification feature is also deployed on OneTeam.im XMPP server, and ejabberd has a component, IMpush, that can send push notifications to the APNS. This makes ejabberd and OneTeam the first XMPP client and server solution that supports the Apple push system.

OneTeam for iPhone has also received fixes and improvements: groupchat, file transfer, image sending... Here are a few screenshots:

An example of Groupchat:

image

See the Push configuration screen:

image

And the general OneTeam confguration:

image

A few comments on the experience after having used OneTeam for iPhone with push and groupchat:

  • "Battery life is not impacted by being always connected to my XMPP server on mobile. I have used all mobile XMPP client and this is the first time I can truely stay connected all the time. I feel this is really really the premise of the mobile revolution for mobile instant messaging and mobile XMPP."
  • "OneTeam for iPhone client has the type of feature you would expect from a good desktop XMPP client and not a mobile one. You can use groupchat, file transfer. This is really what I need professionally."

OneTeam for iPhone will be available on the Apple's App Store in a few days, after the usual moderation period at Apple.

IMpush is sold as soon as today on the IMstore to prepare ejabberd servers ahead of OneTeam 3.0 with push.

by Nicolas Vérité at July 01, 2009 02:18 PM

Erlang Inside

erlang:lists/1

Welcome to the start of the erlang:lists series where we list some interesting happenings in the world of Erlang.

erlang:lists(CouchDBNaked) - Harish Mallipeddi, a performance engineer at Yahoo, has put together a great post on the internals of CouchDB, and how to even use some of the couch source to build your own B-Tree based mini-application.

erlang:lists(ErlangFactory) - Slides and videos are up from the 2009 London Erlang Factory conference. About half have videos and most have slides available. Not sure yet if they will all have videos or if there are reasons for not showing the films for some speakers.

Each week I’ll mention a few must-see articles, blog entries or videos. If you have something to bring to my attention - contact me at chad at inakanetworks com.

by Chad DePue at July 01, 2009 02:15 AM

June 30, 2009

Erlang Training and Consulting
- News

30 June 2009: The Erlang Factory London is still the largest Erlang event!

The London Erlang Factory retained its place as the biggest gathering of Erlang talent in 2009, surpassing even the Palo Alto Factory! With more than 40 speakers and a series of 10-minute talks at the Erlounge, the London Factory brought together Erlangers from five continents. The Erlang Universities also proved popular and allowed delegates to combine their training with the conference for a great value-adding experience.

The presentation slides and videos of the talks can be found on the Erlang Factory website . You can still follow @erlangfactory on Twitter and join us on Facebook where we have put some photos from the event.

June 30, 2009 01:53 PM

Process-one Blogs

ProcessOne presentations at Erlang Factory

There have been two presentations from ProcessOne at Erlang Factory. Here are the slides of the presentations.

The first presentation by me (Mickaël Rémond) was titled "OneTeam Media Server: Adding Video to Instant Messaging with Erlang".

OneTeam Media Server
View more documents from Mickaël Rémond.

The second presentation by Geoff Cant was title "Whitelabel Erlang" and give feedback on how Erlang can be used to write scalable hosted application for business.

Erlang White Label
View more presentations from Mickaël Rémond.

by Mickaël Rémond at June 30, 2009 01:34 PM

June 29, 2009

Joe Armstrong

Content Editable

I've been playing with the HTML contentEditable mode in Firefox.One word awesome.I quickly managed to put together the basis of a seamless editor. This is described in a seven part article.Part1 - toggling content editablePart2 - Adding some stylePart3 - Adding editing buttonsPart4 - A seamless structure editorPart5 - All the buttonsPart6 - Storing the results foreverPart7 - Integration with

by Joe Armstrong (noreply@blogger.com) at June 29, 2009 12:26 PM

June 24, 2009

Damien Katz

StackOverflow Podcast

Yesterday I did a StackOverflow podcast with Joel Spolsky and Jeff Atwood. We talked about CouchDB and Erlang, among other things: StackOverflow Episode 59

by Damien Katz at June 24, 2009 07:27 PM

Erlang Training and Consulting
- News

11 June 2009: Get a 35% discount on the latest Erlang book

Francesco Cesarini and Simon Thompson will be signing their new book Erlang Programming at the Erlang Factory. Buy it on a day and get 35% discount off the cover price.

June 24, 2009 04:52 PM

08 June 2009: Open Source Light Weight HTTP 1.1 Client

After recent experiences and discussions with the Erlang community, ETC have decided to develop and release an HTTP/1.1 lightweight client. You can find more information in the Erlang Open Source section of our website or in the contributions section on Trapexit.

June 24, 2009 04:52 PM

13 May 2009: Erlang Training and Consulting at OSCON 2009!

Erlang Training and Consulting has been selected to represent Erlang at O’Reilly’s Open Source Convention in San Jose, California July 20 - 24, 2009. We will jumpstart the week with a tutorial on Practical Erlang Programming and follow up with Erlang for Five Nines, a non technical introduction to Erlang. If you are planning on attending, use discount code OS09PGM to receive a 15% discount. See you there!

June 24, 2009 04:52 PM

29 March 2009: Sponsoring the 2009 Powered By Erlang Eurobot Team!

Erlang Training & Consulting is co-sponsoring the development of a “Powered by Erlang” IANO Robot. This Intelligent Autonomous Nasty Object will compete against other international solutions in the in the 2009 Eurobot competition building (and destroying) a temple! It is developed by the Computer and Telecommunication Engineering department of the University of Catania (Italy). For more information click here...

June 24, 2009 04:52 PM

03 March 2009: Meet us at QCON London 2009 !

Erlang Training and Consulting’s Francesco Cesarini and Ulf Wiger will be giving a presentation on Erlang and Multicore and one on Concurrent Erlang Architectures at QCON in London next week. We will also be track hosts for the Functional and Concurrent Programming Languages Applied. See you there!

June 24, 2009 04:52 PM

16 Febuary 2009: Welcoming Ulf Wiger as Erlang Training and Consulting's New CTO

Ulf has used Erlang since 1992, and made a name for himself as the Chief Designer of Ericsson's AXD 301 - for many years the main showcase of Erlang's strengths. With his 20-year track record of building high-availability systems and strong ties to the Erlang community, Ulf will be a powerful addition to our management team.

June 24, 2009 04:52 PM

04 Febuary 2009: Announcing the 2009 SF Bay Area Erlang Factory!

The Erlang Factory comes to Palo Alto in the San Francisco Bay Area April 27th - May 1st! This promises to be the largest gathering of Erlang expertise since last year's eXchange in London. Confirmed speakers include Robert Virding, Ulf Wiger, Richard Carlsson, Kevin Smith, John Hughes, Ezra Zygmuntowicz, Mickael Remond and many many more. Tentative tracks include Tools and Gadgets, Erlang and TDD, Erlang and IM, Relax with CouchDB and Cool Applications. Running together with the Factory is the Erlang University - 3-day courses on Erlang, OTP Design Patterns, Quick Check and CouchDB, allowing you to combine a training course, talks and tutorials in the same week. Come and meet Erlang inventors and experts who have been using the language long before it was released as open source, network with committers of the open source applications or debate and discuss the latest features and libraries. For more information on the Erlang Factory, visit our Erlang Factory site.

June 24, 2009 04:52 PM

01 Febuary 2009: Erlang in South America!

Erlang Training & Consulting gets its first contract in South America. This prestigious consulting job brings to six the number of continents on which it operates. Previous continents include Africa, Australia, Asia, North America and Europe. Is there anyone in Antarctica we can help?

June 24, 2009 04:52 PM

11 Novmber 2008: Announcing our Training Schedule for 2009!

As a very busy and successful year with lots of scheduled courses draws to a close, we can now confirm our Training Schedule for 2009.

Next year we will be offering courses in Dubai and Singapore in addition to our usual ones in UK, Sweden, Poland, USA and South Africa. For details, please see our Training Schedule.

June 24, 2009 04:52 PM

03 Novmber 2008: Releasing the Erlang Web 1.1 Platform as Open Source!

The Erlang Web is an open source framework for the rapid deployment of web based interfaces. By separating the HTML generation, glue and logic while retaining it in the same memory space, we provide a framework which gives the developer better control of content management where reusability is the key. Erlang Training and Consulting has been using the Erlang Web in commercial applications for three years, and in order to increase the user base and available generic components, has decided to release it as open source. For more information and to download the latest version of the code, read documentation and view examples, presentaitons and tutorials, visit the Erlang Web site. You can also meet us at the Erlang User Conference, where we will be presenting the Erlang Web.

June 24, 2009 04:52 PM

Erlang Training and Consulting
- Jobs

Senior Erlang Developer, London, UK

Gambit Research is a West London software company developing market information and trading software for the betting industry.
We are looking to hire an experienced Erlang developer to work on extending and improving an existing trading system. This role will involve working as part of a small team of Erlang developers and working closely with Python developers and the trading team.
You should have experience of developing reliable, fault tolerant applications in Erlang. Experience with UNIX, networking, PostgreSQL and Python would also be highly desirable.

  • Starting salary £30,000 plus, depending on experience.
  • 25 days holiday per year.
  • Discretionary bonus scheme.
  • Flexible working hours.
  • Relaxed and informal working environment.

June 24, 2009 04:50 PM

Seattle Area Erlang Contractors, Seattle, USA

We are looking for 2-3 Erlang developers with 3-5 years experience with Erlang development for a short term engagement (4-6 months).

Required Skills: Erlang/OTP, Mnesia,Yaws, MochiWeb, EUnit. Test-Driven Development experience (Unit/Integration/System. Strong verbal and written communication skills.

Additional Desired Skills: Network Programming, Enterprise Application Development, Real-Time Concurrent Programming,Performance Analysis and Tuning, Stress-Testing.

Compensation is based upon experience. No relocation/travel, prefer local candidates.

June 24, 2009 04:50 PM

Undergraduate Summer Interns, London, UK

Erlang Training and Consulting is currently interviewing for its summer internships. As in previous years, we are looking for bright, dedicated, motivated, curious and self-driven individuals in their second to fourth year of studies the field of Computer Science. Exposure to Erlang and Unix derivatives (preferably Linux) are a must. The right candidates will be working with exciting new projects (SMS, Telecom, E-Commerce, Instant Messaging, Banking) with some of the best Erlang developers around. And as a bonus, you will be in the heart of London. Applicants from all over are encouraged to seek the positions (We have 9 nationalities represented in our offices), but to apply, you have to be eligible to work in the EU. Let us have your CVs and a covering letter, letting us know about yourselves, your hobby projects, and why you want to work with us at Erlang Training and Consulting. We need your application at the latest on the 30th of April.

June 24, 2009 04:50 PM

Systems Administrator, London, UK

We are seeking a competent Systems Administrator to develop our business and development environment.

1. The requirement

The ideal applicant will need to have experience of setting up an office infrastructure for a software development company where there are hosted components and office based components. We also have offices in Krakow and Uppsala that should have access to the infrastructure. Our goal is to phase out Microsoft and move completely over to Open Office and Open Source offerings so somebody who understands and believes in this approach is hugely desirable.

The following are a broad list of the technologies and application we are either using, or intend to use and which the incumbent would be expected, in the longer term, to manage and improve.

OS: Windows XP and Vista, All flavours of Linux, UNIX.
DB's: mySQL, Postgresql, Berkley DB
Collaborative environments: MS Exchange and Zimbra
Office suites: MS Office and Open Office,
Set up of VPNs
Set up and maintenance of Firewalls
Wireless Networks
PBX's: Nortel BCM50
Wiki's
Single Sign on
File share and Backups
Process for signing up new users and removing leavers
Policies for access privileges
Security and encryption
Experience of CRM systems like SugarCRM or Salesforce and Trouble Ticketing systems like Bugzilla or Trac would be desirable
Knowledge of email and IM protocols are useful
Experience of dealing with service providers and selecting best ADSL, SDSL services is also a plus as well as SLA's

Selection and recommendations for hardware purchases Ability to work alone and under pressure

A hands on and "I can do it" attitude

Confident person

A creative thinker



2. The company offers:
A supportive and friendly working atmosphere together with the opportunity to progress.
Salary: 22,000GBP - 28,000GBP + Discretionary Bonus

Benefits: Private Health Insurance, Life Protection, 20 days holiday, Bike Scheme, Stakeholder Pension Scheme.

Training and advancement is on offer once the successful candidate has proven themselves within the company as a dedicated member of the team.

Only applicants who have a legal right to work within the UK will be considered.

June 24, 2009 04:50 PM

Marketing and Communications Assistant, London, UK

We are seeking a highly motivated and ambitious marketing assistant to support the Operations Manager with the development and execution of marketing plans

1. The Role:

  • To understand the marketing communications requirements of the business
  • To work closely in cooperation with the organisation for the creation and implementation of each project
  • Undertake a focused telemarketing role to help in the delivery of qualified leads
  • Maintain the prospect and customer database and report leads via the CRM system
  • Work closely with the business to develop a communications strategy
  • To assist in co-ordination of conferences and training events worldwide
  • To support the business to deliver customer programmes
  • Support the business to ensure plans, measurement and the delivery are implemented
  • Attend weekly status and planning meetings

2. The Person:
  • A good all round marketing mix background within B2B and/or IT/telecoms focused environment
  • Knowledge of Salesforce.com an advantage
  • Understanding and experience of marketing communications delivery
  • Excellent communications skills with peers and more senior level
  • Ability to manage numerous projects at any one time and prioritise workloads
  • Demonstrable experience of assisting with managing successful marketing communications campaigns
  • Ability to work alone and under pressure
  • A hands on and `I can do it` attitude
  • Confident person
  • A creative thinker

3. The company offers:
  • A supportive and friendly working atmosphere together with the opportunity to progress.
  • Salary: 20,000GBP - 25,000GBP + Discretionary Bonus

Benefits: Private Health Insurance, Life Protection, 20 days holiday, Bike Scheme, Stakeholder Pension Scheme.

Training and advancement is on offer once the successful candidate has proven themselves within the company as a dedicated member of the team.

Only applicants who have a legal right to work within the UK will be considered.

June 24, 2009 04:50 PM

Book keeper/Administrator, London, UK

Duties will include, but will not be limited to:

1. The Role

  • Accounting duties
  • Purchase and sales ledger
  • Bank reconciliations
  • Production of VAT and Year End Returns
  • Monthly Payroll including PAYE, NIC and all relevant Year End Returns
  • Preparation of profit and loss accounts
  • Preparation of management accounts
  • Monthly reporting
  • Preparation of year end files for Auditors
  • Administration duties
  • Diary management
  • Administration
  • Monthly reporting
  • Ad hoc duties as required

2. The Candidate:
  • Accounts experience as above is essential.
  • A good working knowledge of Sage Line 50 would be a distinct advantage.
  • Excellent organisational skills, self motivated, adaptable and excellent communication and IT skills.
  • Desire to work within a small team.

3. The company offers:
  • A supportive and friendly working atmosphere together with the opportunity to progress.
  • Salary: 20,000GBP - 25,000GBP + Discretionary Bonus

Benefits: Private Health Insurance, Life Protection, 20 days holiday, Bike Scheme, Stakeholder Pension Scheme.

Training and advancement is on offer once the successful candidate has proven themselves within the company as a dedicated member of the team.

Only applicants with a suitable amount of UK based finance experience and who have a legal right to work within the UK will be considered.

June 24, 2009 04:50 PM

Experienced Server Engineer, Los Angeles, California

Do you know the difference between static and dynamic typing? Are you obsessed with ducks? Do you want to jump ship from large corporations!? If so, then we're looking for you.

Our high energy startup is committed to revolutionizing the way people think about their mobile phones. It has the potential to capture hundreds of thousand of users in the uncharted space of mobile social networking. Our service promotes, assists & inspires face-to-face connections between friends and family through the use of digital devices while maintaining an uncompromising commitment to privacy & security.

To make this happen we focus on working hard, fostering a creative job setting, working harder, indulging in our successes, and then working even harder still you could say our ambition teeters on the insane, the good kind though.

We are seeking an experienced server engineer to join our small but rapidly growing team. This individual will thrive in a startup environment and is passionate about creating server software that scales to hundreds of thousands of concurrent users and beyond.

You will be involved in all phases of the software lifecycle, from specification to design, coding, and deployment. We will enjoy finding someone who takes pride in the quality of his/her work.

Requirements:

Expert level server-side coding experience (Python, Erlang,Jabber/XMPP, C/C++)
Object oriented programming background
Experienced with functional programming
Knowledge of distributed programming
Strong knowledge of Postgres or MySQL and development of database-backed applications.
Extensive experience with working with data intensive applications including data manipulation, structures and integration.
Scalability, security and performance are at the front of your mind.

For this reason, you should have exposure to high-volume, high-availability apps and have a knack in building distributed systems.

To apply for this job, you need to be eligible to work in the US. This employer will not be sponsoring candidates.

If interested, send us your CV.

June 24, 2009 04:50 PM

June 23, 2009

Process-one Blogs

Datamonitor coverage

Mickaël Rémond, CEO of ProcessOne, was covered by a Datamonitor study: "CEO Viewpoint: Social Networks Mobility = Market Opportunity".

In this Analyst Opinion series by Daniel Okubo, it is observed that social networks and mobile network operators are currently deploying IM and presence features based on XMPP.

In a competitive and recessive context, the leaders and newcomers look to differentiate in order to increase their market shares. Real time communication based on open standard like XMPP has proven a good way to value existing users base.

This document is available at Datamonitor ResearchStore, at the price of $150, or $375 for a multi-user license. ProcessOne is not affiliated with DataMonitor and does not earn any commission on sales of this document, but we thought it could be usefull to know for customers interested in ProcessOne.

by Mickaël Rémond at June 23, 2009 03:44 PM

June 22, 2009

Joe's Blog

The Specials.

A Message To You Rudy

Click here to view the embedded video.

Ghost Town

Click here to view the embedded video.

Yup. The Specials.

by joe at June 22, 2009 04:03 AM

June 21, 2009

Caoyuan's Blog

Erlang Plugin Version 1 for NetBeans 6.7 Released

I'm pleased to announce Erlang plugin (ErlyBird) version 1 for NetBeans 6.7 is released.

NetBeans 6.7 RC3 or above is a requirement.

What's new:

  • It's rewritten in Scala instead of Java
  • More reliable instant rename
  • Display extracted document information from source comment when doing auto-completion.

To download, please go to: https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=226387

To install:

  • Open NetBeans, go to "Tools" -> "Plugins", click on "Downloaded" tab title, click on "Add Plugins..." button, choose the directory where the Erlang plugin are unzipped, select all listed *.nbm files, following the instructions.
  • Make sure your Erlang bin path is under OS environment PATH, you can also check/set your OTP path: From [Tools]->[Erlang Platform], fill in the full path of your 'erl.exe' or 'erl' file in "Interpreter", for instance: "C:/erl/bin/erl.exe". Or open the "Brows" dialog to locate the erlang installation.
  • When you open/create an Erlang project first time, the OTP libs will be indexed. Take a coffee and wait, the indexing time varies from 10 to 30 minutes depending on your computer.
Feedback and bug reports are welcome.

by dcaoyuan at June 21, 2009 02:04 AM

June 20, 2009

plok

EU Summer Tour

After the US Spring Tour in April this year, I’m about to embark on the EU Summer Tour.

I’ll be visiting London, Amsterdam, Zurich and the Gran Canaria. Here’s when, how and why:

June 22nd–26th: CouchDB University & Factory, London

The CouchDB University is a a three day training course where J Chris and I teach a select group of students everything about CouchDB. With little prior knowledge, we’ll leave you with being able to build amazing CouchDB applications at small and large scale as well as extend CouchDB itself.

The CouchDB Factory is a track at the Erlang Factory running all day Friday.

June 29th–30th: Kings of Code, Amsterdam

Kings of Code looks like it is going to be a kick-ass web developer conference featuring some of my favourite web people: Geoffrey Grosenbach Joe Stump & Francisco Tolmasky. I’m fairly confident that the other speakers will be among my favourites after Kings of Code :)

J Chris will be talking about CouchDB.

It’s still in discussion, but I might talk about CouchDB and Erlang for web developers on one of the side events.

July 1st–2nd: ICOODB, Zurich

I’ll be taking the night train from Amsterdam zu Zurich to give a three hour tutorial as well as a 60 minute presentation on CouchDB at the International Conference on Object Databases. CouchDB is strictly not an object oriented database, but it stores objects and is of interest to the research community that meets in Zurich.

Prof. Stefan Edlich invited me to speak at ICOODB and I’m very happy I can make it.

July 1st–7th: GUADEC, Gran Canaria

Canonical, the kind folks behind the Ubuntu Linux distribution are pushing CouchDB to become a centerpiece of the Ubuntu desktop data synchronization infrastructure. Merrily sync your contacts, calendar data between your machines, an online backup service and share select data with your peers. And yeah UbuntuOne is also related :)

Canonical is flying me out to attend the Linux Desktop Summit to talk to desktop application developers and show them how cool CouchDB is and where it is useful for them.

Also, Gran Canria, I couldn’t say no. Thank you Canonical!

As much as I am excited about the travels and meeting all you out there, I’ll be missing three weeks in my favourite city, Berlin and it makes me a little sad.

by jan@apache.org (Jan) at June 20, 2009 12:09 PM

June 19, 2009

Hypothetical Labs

Win32 Linked-In Drivers and A Project Idea

I’ve been living in linked-in driver land for the past few weeks. I guess that’s what I get for publishing basic_erl_driver. It’s been interesting work so I certainly haven’t minded doing it.

Well, I didn’t mind it until I had to build one of my drivers on Windows. It’s been a long time since I’ve done much Win32 development but I used to know my way around Visual C++ pretty well. “I used to do this shit for a living. How hard could this be?”, I thought to myself.

Three words: famous last words.

Three more words: C++, linker, XML.

Two final words: sheer frustration.

After wrestling with Visual Studio Express C++ Edition 9.0 (or whatever crazy name MS uses) for a day and a half I finally have a working driver. Here’s the nuggets o’ knowledge I learned along the way. These notes assume you’re porting a working driver from OS X/Linux/BSD to Windows although most of them are applicable if you’re starting from scratch on Windows, too. They also assume you’re building code from inside of Visual Studio because, frankly, I don’t care enough about Windows to figure out nmake, too.

  • Start with a blank Win32 DLL project. Don’t let Visual Studio create stdafx and friends. Down that path lies madness. You’re much better off to create a blank project and import your code.
  • Add the erts-<version_number>/include directory to your project’s include path. You’ll need this so the compiler can resolve erl_driver.h.
  • Don’t forget to include string.h before erl_driver.h. erl_driver.h uses memcpy() without including string.h. Including string.h first will fix a compiler warning and a possible runtime crash.
  • Very Important: Right-click your project and navigate to the Linker -> Manifest File option. Enter this magical text into the Additional Manifest Dependencies field: 
    type='win32' name='Microsoft.VC90.CRT' version='9.0.21022.8' processArchitecture='x86' publicKeyToken='1fc8b3b9a1e183b'

    I believe this tells the compiler to link the DLL with the Visual C++ 9.0 runtime. This is the equivalent of the C stdlib on other platforms, I think. Erlang will be unable to load your driver if you skip this step.

    • Observation: Who in their right fucking mind thought it was a good idea to add XML descriptors to the link step of building a C library?! I take back every bad thing I’ve ever said about Java’s use of XML. SOAP, with all of its angle brackety warts, almost looks sane in comparison.
  • With the previous step accomplished you’ll need to distribute the VC++ runtime with your driver. Luckily, MS provides the code in a redistributable form inside the Microsoft Visual Studio 9.0\VC\redist\x86 directory structure. Grab all of the files in there and plop them into the same directory as your driver’s DLL file.

    • There are probably better ways to accomplish the same goal given my dated platform knowledge. If you know of a better way to do this, please drop a comment and set me straight!

    One of the drivers I’ve been working on is to allow SpiderMonkey to interface with Erlang. I’ve asked and am waiting for permission from my client to open source the code since I think it’d be generally useful. If that doesn’t pan out I’m strongly considering doing another driver — open source from the beginning — to interface with V8 or possibly WebKit’s engine. I’ve got some ideas on how JSON could smooth the interop between the two languages. Anyone up for either working on or sponsoring work on this?

    by kevin at June 19, 2009 02:29 AM

    June 17, 2009

    Undefined And Most Likely Disastrous

    Webmachine Tutorial: File Uploads

    I've vented about the client side of HTTP file uploads in my post below, but once you've got the client side down, you still have to deal with a messy multipart/form-data request body on the server. Fortunately, version 1.3 of Basho's homegrown and production-proven HTTP toolkit, Webmachine, has just been released, with support for parsing multipart bodies.

    I've checked in a fully-functional file-upload demo to BitBucket here, and I'll go through the interesting bits of using webmachine_multipart in this post.

    Before parsing a multipart body, you need to parse out the browser-generated part separator, or "boundary", from the Content-Type header.


     ContentType = wrq:get_req_header("content-type", ReqData),
     Boundary = string:substr(ContentType, string:str(ContentType, "boundary=")
                              + length("boundary=")),

    Once you have the boundary string, Webmachine provides two options for parsing the actual request body. The first, webmachine_multipart:get_all_parts/2 parses an entire body and returns a structure with the body metadata and content. The second option, which I used here, uses the new-in-1.3 streaming body API to read data in chunks off the wire. This API returns a function that allows you to read successive elements of the multipart body lazily. Here's the relevant usage of this API in the file upload example (called from process_post/2):

    %% (from process_post/2...)
    {FileName, FileSize, Content} = 
        get_streamed_body(webmachine_multipart:stream_parts(
                          wrq:stream_req_body(ReqData, 1024), 
                          Boundary), [],[]),
        ....

    get_streamed_body(done_parts, FileName, Acc) ->
       Bin = iolist_to_binary(lists:reverse(Acc)),
       {FileName, size(Bin)/1024.0, Bin};
    get_streamed_body({{"filedata", {Params, _Hdrs}, Content}, Next}, Props, Acc) ->
       FileName = binary_to_list(proplists:get_value("filename">>, Params)),
       get_streamed_body(Next(),[FileName|Props],[Content|Acc]).
    The function takes the result of wrq:stream_req_body.2 and returns a 3-tuple of {FileName::string(), FileSize::float(), Bin::binary()} describing the uploaded file and it's contents.

    With that data, once can write the uploaded file to a disk and return a new page indicating success, along with the filename and file size, which is what the rest of process_post in my example does.

    For completeness, here's the entire process_post/2 function:

    process_post(ReqData, C) ->
        ContentType = wrq:get_req_header("content-type", ReqData),
        Boundary = string:substr(ContentType, string:str(ContentType, "boundary=") 
                                 + length("boundary=")),
        {FileName, FileSize, Content} = get_streamed_body(
                                          webmachine_multipart:stream_parts(
                                            wrq:stream_req_body(ReqData, 1024), 
                                            Boundary), [],[]),
        StorePath = filename:join([?STORE_PATH,"/", FileName]),
        filelib:ensure_dir(StorePath),
        file:write_file(StorePath, Content),
        NewRD = wrq:append_to_response_body(
                  io_lib:format(
                    "<html><head><title>Upload Complete</title></head>"
                    "<body><h1>Upload Complete</h1>"
                    "<p>Received file ~s (~.2fK)</p></body></html>",
                    [filename:basename(StorePath), FileSize]), ReqData),
        {true, NewRD, C}.
    The entire source for the file upload resource can be found here. It's 42 lines total, including boilerplate and supporting HTML.

    Happy Uploading!

    by noreply@blogger.com (andy) at June 17, 2009 05:28 PM

    Dukes of Erl

    Keeping it simple with flatula

    Paul has blogged about overcoming mnesia performance issues in the past, but I don't think we've talked much about the ultimate strategy -- keeping data out of mnesia altogether.

    When we first started serving ads, we stored information about every single ad impression in a huge mnesia database, for retrieval on click, and for building behavioral profiles. Almost needless to say, this didn't scale very far. We spent many a day last summer delving into mnesia internals, fixing corrupted table fragments after node crashes, bemoaning how long it took new nodes to join the schema under heavy load, and so on.

    One of the simplest and most effective changes that got us out of this mess was not to store any per-impression data in mnesia at all -- instead, we started logging the data to flat files on disk, and storing a small pointer to the data in a cookie so we could read it back the next time we saw the user. Hardly a revolutionary solution . . . it's well-known that disk seeking is the enemy of performance. The hardest part was coming to realizations like, "Hmm, I guess we don't really care if a node goes down and we lose part of that data!"

    We've open-sourced one of the main components that enabled this strategy: flatula, an Erlang application that manages write-once "tables" that are really just collections of flat files. It looks a bit like dets, except that it doesn't support deletions, updates, or iteration, and you can't make up the keys. But when you don't need those things, it's hard to imagine a more efficient way to store data.

    If you'd like to learn more, there's a brief tutorial on the Google Code site.

    by Michael Radford (noreply@blogger.com) at June 17, 2009 04:58 PM

    BeerRiot Blog

    Webmachine Bloggy Goodness

    Webmachine (and Erlang use in general) is picking up. If you're looking for a few more people to follow in relation to webmachine, I have some suggestions.

    by Bryan at June 17, 2009 02:18 PM

    ErlyBird

    Erlang Plugin Version 1 for NetBeans 6.7 Released

    <p>
    I'm pleased to announce Erlang plugin (ErlyBird) version 1 for NetBeans 6.7 is released.
    <p>
    NetBeans 6.7 RC3 or above is a requirement.
    <p>
    What's new:
    <ul>
    <li>It's rewritten in Scala instead of Java</li>
    <li>More reliable instant rename</li>
    <li>Display extracted document information from source comment when doing auto-completion.</li>
    </ul>
    <p>
    To download, please go to: https://sourceforge.net/project/showfiles.php?group_id=192439&package_id=226387
    <p>
    To install:
    <ul>
    <li>Open NetBeans, go to "Tools" -> "Plugins", click on "Downloaded" tab title, click on "Add Plugins..." button, choose the directory where the Erlang plugin are unzipped, select all listed *.nbm files, following the instructions.</li>
    <li>Make sure your Erlang bin path is under OS environment PATH, you can also check/set your OTP path: From [Tools]->[Options], click on 'Miscellanous', then expand 'Erlang Installation', fill in the full path of your 'erl.exe' or 'erl' file. For instance: "C:/erl/bin/erl.exe"
    <li>When you open/create an Erlang project first time, the OTP libs will be indexed. Take a coffee and wait, the indexing time varies from 10 to 30 minutes depending on your computer.</li>
    </ul>
    Feedback and bug reports are welcome.
    <p> (0 comments)

    by dcaoyuan@users.sourceforge.net (Caoyuan Deng) at June 17, 2009 07:45 AM

    RabbitMQ Announcements

    [rabbitmq-announce] RabbitMQ 1.6.0 release

    by Matthias Radestock at June 17, 2009 12:00 AM

    June 16, 2009

    plok

    Caveats of Evaluating Databases

    This is part two in a small series about measuring software performance. There’s a lot of common sense covered, but I feel it necessary to shed some light.

    If you haven’t, check out part one.

    Say you want to find out what’s behind the buzz of all these new #nosql databases. There’s a large number to choose from today. All options come in varying degrees of maturity and characteristics so it’d be nice to know what solves your problem best. A non-exhaustive list of these databases or storage systems include Memcache[DB], Tokyo Cabinet / Tyrant, Project Voldemort, Scalaris, Dynamite, Redis, Persevere, MongoDB, Solr or my favourite CouchDB. And these are just some of the open source ones.

    This article is not a comprehensive comparison of any of the mentioned systems. Instead it tries to give you an idea about what to look for when evaluating a storage system or how to take into perspective evaluations and benchmarks others have done.

    We’ll look at some of the technical aspects of data storage systems: Applying common sense when reading benchmarks; b-trees and hashing; speed vs. concurrency; networked systems and their problems; low level data storage (disks’n stuff); and data reliability on single-nodes and multi-node systems.

    There are a lot of other reasons to decide for or against a project based on a lot of non-technical criteria, but things like commercial support or a healthy open source community are not part of this article.

    Astounding Numbers

    From time to time you see some crazy numbers posted to the reddits of the internets that claim fantastic performance.

    The (imaginary) SuperfastDB can store 450,000 items per second!.

    Wow.

    No word on where the items are stored (in memory? on a harddrive? Spindles? Solid State?), what an item is exactly and how big it is, the rest of the hardware this was run on and how to reproduce it.

    But boy, 450,000 a second!

    My shoes can do 650,000 a second, but you’ve got to figure out what.

    Context is as important as reproducibility. The last article here established that finding out that my system and your system come up with different numbers is not much of a help. Any sort of serious test must come with a set of scripts or programs and comprehensive instructions on how the tests were run.

    Everything “cool” in computer science has been around for 25+ years. Actual innovation is rare. Advancements in hardware and new combinations of existing solutions make for new stuff coming out each day (that’s a good thing), but the fundamental rules are the same for all. We’re all running von Neumann machines, quicksort is still pretty quick and hashes and b-trees rule the storage world.

    Let’s recap.

    Hashes & Trees

    Hashing revolves around the idea of O(1) lookups. Allocate a number of buckets, create a function that gives you a number of a bucket for any data item you might want to store, make sure no two data items hit the same bucket (or work around that). Runtime characteristics include that you only need to ask your function where to look for or store your data and the allocation of your set of buckets: If you need to store more items than you have buckets, some more work is required which gives you O(N) operations that you can’t ignore in practice.

    D5563B63-7B48-4280-A31F-EDB37DB78416.jpg

    The other elephant in the room are b-trees. The fundamental idea here is to get to your data in a minimal number of steps traversing a tree because making a step is expensive, but reading your data is very fast comparatively. Steps are expensive because they translate to a head seek (that is the time your spinning hard drive needs to position the reading arm to find the spot to read your data from), but reading from a harddrive once the reading head is in place is fast.

    6720EE64-4DFC-4298-B3BA-0145746C6523.jpg

    There are a bunch of more interesting lookup structure like R-Trees for spacial queries, but they are mostly used for secondary indexes on top a regular data set that lives in a hash or b-tree.

    Concurrency vs. Speed

    Concurrency is hard. The devil lies in the details and when briefly looking at things, the details are often overlooked. Suites the devil.

    Creating storage systems that assume only one access occurs at a time is relatively easy. If resources are shared concurrently, things become tricky. The two larger schools of thought (and practice) are locking and no-locking (heh).

    Locking means that the database has to maintain information for everybody who wants to write to a part of the database, and what part it is.

    No locking, or optimistic locking or MVCC moves that burden to the person who is trying to write to the database. She must prove that she won’t be overwriting any existing data.

    The trade-offs here are a leaner request handing on the server that works well with remote & concurrent clients at the expense of more complexity on the client (the person who wants to store something in our database).

    Hybrid approaches are possible too: While MVCC is used internally, the database’s clients can rely on database-side locking (e.g. PostgreSQL or InnoDB).

    Networks

    Just a quick note: We already talk about client and server here. There is a strong case for embedded databases like SQLite that don’t expose a concurrent user model to the outside. The program that needs an embedded database just includes it.

    Another approach to using databases is having a dedicated computer running a database system and sharing it over the network with any number of clients using this database server. They can often be “a bunch of servers” or a cluster. More on that later.

    A separate database server (networked or not) will need to spend some time to deal with connections, network failures, unspecified client behaviour and so on. The upside is a piece of infrastructure that can be maintained separately. An embedded database will thus be faster but probably won’t solve all of your problems and it will always be tied to your application.

    fsync(): Reliability vs. Speed

    When people tell me “SuperfastDB does 450,000 a second!” I ask “How many fsync()s is that?”. Let me explain:

    A database system uses operating system services to use any hardware. The operating systems exposes a harddrive through a filesystem. The database systems talks to the filesystem and asks it to store or retrieve data in its behalf. The filesystem then goes ahead and tries to satisfy the database’s requests.

    (I’ll not talk about databases that can use raw block devices to store data. They exist but they are not as common as those who use the filsystem.)

    The filesystem also tries to be clever – for good reasons. When the database requests a piece of data, the filesystem will not only find that piece and return it, it will also store it in a cache to avoid having to actually talk to the harddrive the next time this piece of data gets requested. When the data changes, the filesystem either removes it from the cache or updates it with the harddrive. It might even go further and only store the new data that comes in with a write request into the cache and rely on a periodic task to write all of the cache back to the drive. Writing a bunch of of pieces at once is more efficient than storing each one on its own.

    More efficient equals to faster and faster is good, right? Well, it depends: If all goes well, this approach is a nice one. But you know computers, things will not go well 100% of the time. The failure scenarios are endless, but they boil down to the question: “What happens when your machine dies and you have data that has only been written to memory?” — The answer isn’t too hard: That data is lost. If there is a delay between a write request finishing and data being written (or “flushed”) to disk any data that has been “written” during the delay period is subject to lost.

    There are cases where this is not a problem; in other cases it is. A developer should have the chance to decide. (Note that even your hardware could be lying to you about having stored data, but I’ll punt on this one, get proper hardware).

    So, flushing to disk needs to happen before you can rest assured your data has been stored. Your operating system has an API call that forces the filesystem to write its cache to disk. It is called fsync() (on UNIX systems) and it is an expensive operation. You can only do so many fsync()s in a second and it is not a great many.

    The 450,000 items were most likely just written to memory and not to disk.

    Space & In-Place

    When writing files to disk (at the end of the day, your data ends up in one file or another on the filesystem) that represents what lives in a database, there are multiple options to handle updates.

    An update is a change to your data item, for example, a new phone number. The intuitive way to handle this is to go and find the old phone number in the file, and overwrite it with the new number. Easy.

    There are several problems with this approach: What to do if the new phone number is longer than the old one (say you added an international calling prefix)? The new number needs to be written to a different place and the change in location must be recorded. Not too big of an issue.

    Back to failure scenarios: Again, the reasons can be manifold, but what happens when we’ve (over-)written the first 4 digits of the old with the new number and then the server dies, power goes away or the database server crashes? The next time you want to read the phone number you get a mix of the old and the new one (if you are lucky) and you don’t exactly know that this is the case and which parts are missing. Your database file is inconsistent and you need to run a integrity check to find missing bits and correct half-written bytes. In the worst case that means scanning your entire database file a few times before you resolved all inconstancies. If you have a lot of data, that can take days.

    To solve this, you always write the new phone number to a new place in the database file and only when it has been fsync()ed to disk, you update the location of the phone number (and then flush that update to disk as well). You will never end up in a scenario where your database file can end up an inconsistent state and after a crash you are back online without an integrity check.

    The trade-off for consistency is write-speed (remember fsync()s are expensive) for consistency-check-speed after a failure.

    A nice bonus is that if the “new place in the database” is the end of the file, you keep your disk-drive head busy with writing data to disk instead of seeking all over the place (remember: seeks are expensive).

    Distribution, Sharding & Resharding

    So far, we’ve been looking at scenarios that involve a single database. We learned a great deal (I hope), but in reality we often deal with more than one database. The simplest reason to have two databases is for redundancy. Failures can bring down your database temporarily or even permanently. If it is a temporary issue, waiting a bit (or a bit longer) to get up and running again might be an option, but often, an application or service should be available at all times. A fatal failure where a database server is lost beyond repair, your data is gone if you haven’t stored it in a second place.

    “I’ll just make two copies, easy!”. Yup easy, until you look at the details (that damn devil again!).

    It’s all about failures again. Consider a single read request. A client connects to a server and asks for a data item. The server looks it up and returns the data to the client. All is well. At any point things can go wrong. The network connection can drop (or slow down so much that client or server assume it dropped), the client can disappear (because of a network failure or crash) as can the server. Clients, servers and the protocols they speak need to be built around the assumption that any of these things (and many more) can go wrong. If any parts is not designed to handle error cases, your system will do funny things, but it won’t reliably store and manage your data.

    Add complexity: With each write target (store in two places) the possibility of error and the need for proper error handling grows exponentially. When evaluating a distributed storage system, looking at how errors are handled is vital.

    Another reason to distribute data among multiple servers is capacity. The three metrics of interest here are read requests, write requests and data. If you have more requests or data than a single machine can handle, you need to move to multiple machines. Each metric calls for different strategies, but they often go along with each other. The need for fault tolerance that I discussed above needs to be considered alongside.

    Growing read capacity is relatively easy once you covered the base case where the source for reading data might not be the same as the the target for writing data and that there can be a mismatch (cf. eventual consistency).

    Distributing writes and data works by designating two machines with 50% of the operations. A clever intermediate, a proxy server for example, decides which request goes where and all is well, we can store twice as much and we can store at twice the speed. When we need to grow bigger yet, we add another server and tell the proxy server to distribute the load equally among them. Adding a proxy for distribution introduces a single point of failure and you don’t want these; there’s added complexity with this approach.

    resharding.png

    The diagram shows that there is another step needed that wasn’t included in the above description. The new “node” needs to have a copy of all data items that are assigned to him and are currently living on the two existing nodes. The process of moving data items to new nodes is called resharding and needs to happen every time a new node is added.

    Resharding can be an expensive operation if you have a lot of data. Techniques like consistent hashing help with minimising the amount of items that need to move. If you are looking at a sharding database, you want to understand how the sharding is performed and if you like the trade-offs.

    CAP Theorem

    The CAP Theorem states that out of consistency, availability and partition tolerance, a system can choose to support two at any given moment, but never three.

    cap.png

    Consistency guarantees that all clients that talk to cluster of nodes will always get to read the same data. Write operations are atomic on all nodes.

    Availability guarantees that in any (reasonable) failure scenario, clients are still able to access their data.

    Partition tolerance guarantees that when nodes in the cluster lose their network connection and two or more completely separated sub-clusters emerge, the system will still be able to store and retrieve data.

    Please Talk! (To Developers)

    If you are aiming for a comparative benchmark of two or more systems, you should run your procedure by they authors. I found developers are happy to help out with benchmarks by clearing up misconceptions or sharing tricks to speed things up (which you can choose to ignore, if you are looking for out-of-the box comparison, but this is rarely useful).

    by jan@apache.org (Jan) at June 16, 2009 09:57 PM

    Undefined And Most Likely Disastrous

    HTTP File Upload Woes

    I recently implemented support for email attachments in our sales prospecting application, Basho Open, and learned how painful file uploads on the Web can be. I've been on the server side since 2005, so this was my first step back into the HTML/CSS/Javascript waters in a long time.

    Using only HTML and HTTP (i.e. no Flash), you're required to use a form and a file input element. This means native controls, styling nightmares, no progress reporting, and iframe hacks if you want to support uploads without page reloads. Since our user base supports Flash, I broadened my search for solutions. (For the gory details the client side issues of file uploads, see here).

    We use jQuery, and a quick search for "jQuery file upload plugin" yielded Uploadify, which had a slick demo and seemed like it would meet my needs. Slightly annoying was the fact that Uploadify assumes you're using PHP on the server side, with the client-side JS code expecting certain responses that I had to reverse-engineer to make it work with Webmachine. No big deal, until I tested it on our staging cluster. Turns out Flash doesn't like wildcard SSL certs, which we use. Showstopper.

    With about a day left to complete the feature, I went back to a pure-HTML+HTTP solution. I searched around a bit, and found a couple of valuable resources:


    • Jack Born's Multiple File Upload Magic With Unobtrusive Javascript was very informative, but the solution he outlines requires a page reload to actually perform the upload, which due to the nature of our app wasn't acceptable.

    • This page, which outlines the invisible-iframe technique I ultimately ended up adapting and using.

    Even with the deadline looming, I managed to pull off HTML/HTTP-only solution and gussy it up enough to not be completely embarrassing. After some cleanup, I hope to release my final solution as a jQuery plugin. Some browser inconsistencies did pop up, namely:

    • Firefox populates the value of the <input> element with the basename of the file, while IE provides the entire client-side path to the file, complicating display issues.

    • Firefox also provides the size of the input file on the client side, allowing the developer to perform max-size checks on the client before a user tries to upload a larger-than-allowed file.


    And that's just dealing with the client side. Next post: Easy (server-side) file upload processing with Webmachine.

    by noreply@blogger.com (andy) at June 16, 2009 08:14 PM

    June 15, 2009

    Erlang Training and Consulting
    - Jobs

    Senior Erlang Developer, London, UK, permanent position

    Gambit Research is a West London software company developing market
    information and trading software for the betting industry.

    We are looking to hire an experienced Erlang developer to work on extending and
    improving an existing trading system. This role will involve working as part
    of a small team of Erlang developers and working closely with Python developers
    and the trading team.

    You should have experience of developing reliable, fault tolerant
    applications in
    Erlang. Experience with UNIX, networking, PostgreSQL and Python would also be
    highly desirable.

    * Starting salary £30,000 plus, depending on experience.
    * 25 days holiday per year.
    * Discretionary bonus scheme.
    * Flexible working hours.
    * Relaxed and informal working environment.

    by Erlang Training and Consulting at June 15, 2009 11:59 PM

    Programming in the 21st Century

    How to Crash Erlang

    Now that's a loaded title, and I know some people will immediately see it as a personal slam on Erlang or ammunition for berating the language in various forums. I mean neither of these. Crashing a particular language, even so-called safe interpreted implementations, is not particularly challenging. Running out of memory or stack space are two easy options that work for most languages. There are pathological cases for regular expressions that may not truly crash, but result in such an extended period of unresponsiveness on large data sets that the difference is moot. In any language that allows directly linking to arbitrary operating system functions...well, that's just too easy.

    Erlang, offering more complex features than many languages, has some particularly interesting edge cases.

    Run out of atoms. Atoms in Erlang are analogous to symbols in Lisp--that is, symbolic, non-string identifiers that make code more readable, like green or unknown_value--with one exception. Atoms in Erlang are not garbage collected. Once an atom has been created, it lives as long as the Erlang node is running. An easy way to crash the Erlang virtual machine is to loop from 1 to some large number, calling integer_to_list and then list_to_atom on the current loop index. The atom table will fill up with unused entries, eventually bringing the runtime system to halt.

    Why is this is allowed? Because garbage collecting atoms would involve a pass over all data in all processes, something the garbage collector was specifically designed to avoid. And in practice, running out of atoms will only happen if you write code that's generating new atoms on the fly.

    Run out of processes. Or similarly, "run out of memory because you've spawned so many processes." While the sequential core of Erlang leans toward being purely functional, the concurrent side is decidedly imperative. If you spawn a non-terminating, unlinked process, and manage to lose the process id for it, then it will just sit there, waiting forever. You've got a process leak.

    Flood the mailbox for a process. This is something that most new Erlang programmers do sooner or later. One process sends messages to another process without waiting for a reply, and a missing or incorrect pattern in the receive statement causes the receiver to ignore all messages...so they keep piling up until the mailbox fills all available memory, and that's that. Another reminder that concurrency in Erlang is imperative.

    Create too many large binaries in a single process. Large--greater than 64 byte--binaries are allocated outside of the per-process heap and are reference counted. The catch is that the reference count indicates how many processes have access to the binary, not how many different pointers there are to it within a process. That makes the runtime system simpler, but it's not bulletproof. When garbage collection occurs for a process, unreferenced binaries are deleted, but that's only when garbage collection occurs. It's possible to create a large process with a slowly growing heap, and create so much binary garbage that the system runs out of memory before garbage collection occurs. Unlikely, yes, but possible.

    by James Hague at June 15, 2009 06:00 AM

    June 14, 2009

    Programming in the 21st Century

    Digging Deeper into Sufficiently Smartness

    (If you haven't read On Being Sufficiently Smart, go ahead and do so, otherwise this short note won't have any context.)

    I frequently write Erlang code that builds a list which ends up backward, so I call lists:reverse at the very end to flip it around. This is a common idiom in functional languages.

    lists:reverse is a built-in function in Erlang, meaning it's implemented in C, but for the sake of argument let's say that it's written in Erlang instead. This is super easy, so why not? 

    reverse(L) -> reverse(L, []).
reverse([H|T], Acc) ->
   reverse(T, [H|Acc]);
reverse([], Acc) ->
   Acc.
    Now suppose there's another function that uses reverse at the very end, just before returning: 
    collect_digits(L) -> collect_digits(L, []).
collect_digits([H|T], Acc) when H >= $0, H =< $9 ->
   collect_digits(T, [H|Acc]);
collect_digits(_, Acc) ->
   reverse(Acc).
    This function returns a list of ASCII digits that prefix a list, so collect_digits("1234.0") returns "1234". And now one more "suppose": suppose that one time we decide that we really need to process the result of collect_digits backward, so we do this: 
    reverse(collect_digits(List))
    The question is, can the compiler detect that there's a double reverse? In theory, the last reverse could be dropped from collect_digits in the generated code, and each call to collect_digits could be automatically wrapped in a call to reverse. If there ends up being two calls to reverse, then get rid of both of them, because it's just wasted effort to double-reverse a list.

    With lists:reverse as a built-in, this is easy enough. But can it be deduced simply from the raw source code that reverse(reverse(List)) can be replaced with List? Is that effort easier than simply special-casing the list reversal function?

    by James Hague at June 14, 2009 06:00 AM

    June 12, 2009

    Hypothetical Labs

    Hands-On Erlang Cancelled

    Due to a host of sour economic factors I’ve decided to cancel the Hands-On Erlang training class scheduled for mid-July in Chicago. All registered attendees will be receiving a complete refund by the end of today.

    I’d like to thank the attendees who did enroll and everyone else who took the time to share their thoughts about training with me. While this isn’t the outcome I wanted the entire process has been a learning experience.

    I am working on some alternative training and code review ideas which I hope to announce in the next few weeks.

    by kevin at June 12, 2009 06:36 PM

    June 11, 2009

    Erlang Training and Consulting
    - News

    Get a 35% discount on the latest Erlang book

    Francesco Cesarini and Simon Thompson will be signing their new book Erlang Programming at the Erlang Factory. Buy it on a day and get 35% discount off the cover price.

    by Erlang Training and Consulting at June 11, 2009 11:59 PM

    Best of Erlang

    Erlang Progamming available online

    The "Erlang Programming" book by Francesco Cesarini and Simon Thompson is now available online, via O'Reilly's Safari service.

    by Dein Bär (noreply@blogger.com) at June 11, 2009 04:26 PM

    Joe Armstrong Talk: Functions + Messages + Concurrency = Erlang

    A very good talk (video and slides) by Joe Armstrong from Qcon hosted at InfoQ:

    Functions + Messages + Concurrency = Erlang
    I was glad to hear several of Joe's talks and enjoyed this one as well

    by Dein Bär (noreply@blogger.com) at June 11, 2009 04:22 PM

    Dukes of Erl

    Let parse transform

    So the problem of intermediate variable naming came up again on erlang questions.

    Subject:Versioned variable names
    From: Attila Rajmund Nohl
    Date: Tue, 9 Jun 2009 17:12:34 +0200
    Hello!

    I think there wasn't any grumbling this month about the
    immutable local variables in Erlang, so here's real world
    code I've found just today:

    % Take away underscore and replace with hyphen
    MO1 = re:replace(MO, "_", "-", [{return, list}, global]),
    MO2 = toupper(MO1),
    % Replace zeros
    MO3 = re:replace(MO2,
                  "RX0",
                  "RXO",
                  [{return, list}, global]),
    % Insert hyphen if missing
    MO5 = case re:run(MO3, "-", [{capture, none}]) of
      nomatch ->
    insert_hyphen(MO3);
      match ->
    MO3
    end,
    ...

    Mikael Pettersson pointed out that this really has less to do with immutable local variables and more to do with the lack of a let expression. That was insightful, and since a let expression can be considered syntactic sugar for a lambda expression, I realized that a parse transform could provide let like functionality. Let is a reserved keyword in Erlang so I used lyet instead.

    Essentially the parse transform rewrites
    lyet:lyet (A = B, C)
    as
    (fun (A) -> C end) (B)
    so the above code could be rewritten as
    Result = lyet:lyet (
    % Take away underscore and replace with hyphen
    MO = re:replace(MO, "_", "-", [{return, list}, global]),
    MO = toupper(MO),
    % Replace zeros
    MO = re:replace(MO,
                 "RX0",
                 "RXO",
                 [{return, list}, global]),
    % Insert hyphen if missing
    case re:run(MO, "-", [{capture, none}]) of
    nomatch ->
      insert_hyphen(MO);
    match ->
      MO
    end),
    You must provide at least one argument to lyet:lyet. All but the last argument to lyet:lyet must be an assignment, and the last argument has to be a single expression (but you can use begin and end for a block of expressions inside the lyet). As you can see above, you can reuse a variable name across the assignment arguments to lyet:lyet. You can even use lyet:lyet on the right hand side of the assignments, or as part of the expression argument. Some examples of usage are present in the unit test.

    Update: per Ulf's suggestion, the parse transform also recognizes the local call let_ in addition to the remote call lyet:lyet. It definitely looks nicer with let_.

    The software is available on Google code.

    by Paul Mineiro (noreply@blogger.com) at June 11, 2009 10:27 AM

    Erlang Eclipse IDE

    erlIDE 0.6.3 supports R13B01

    We discovered that R13B01 created some incompatibilities and this release fixed them. There is also an assortment of small bug fixes (but not many, this time).

    Get it from http://erlide.org/update (0 comments)

    by vladdu@users.sourceforge.net (Vlad Dumitrescu) at June 11, 2009 07:37 AM

    June 10, 2009

    Erlang Factory News

    Erlounge open to non-conference delegates

    The Erlounge at the Erlang Factory  will be open, not only to delegates from the Erlang Factory, but also to others who are interested in Erlang. Members of the London Erlang-User group are especially welcome. Attending is free, but to allow us to plan, non-conference attendees must register here.

    From 18:00-19:30 on Thursday 25th, in an environment of beer, cool drinks and snacks, there will be a series of 10-minute talks as the first part of the Erlounge. User-groups and other Erlang interested parties will be given the opportunity to introduce themselves , their interests, their projects etc. If you wish to give a 10-minute talk, please submit your talk here.

    Details of the location of the Erlounge you can find here.

    June 10, 2009 10:06 AM

    June 09, 2009

    Trapexit News Feed

    Trapexit News :: HTTP/1.1 client

    Author: oscarh
    Subject: HTTP/1.1 client
    Posted: Tue Jun 09, 2009 3:33 pm (GMT 0)
    Topic Replies: 0

    After recent experiences and discussions with the Erlang community, ETC have decided to develop and release an HTTP/1.1 client. You can find more information under the open source section of our website and on bitbucket. You can also view this and other user contributions in the User Contribution Section.

    June 09, 2009 03:36 PM

    June 08, 2009

    Erlang Training and Consulting
    - News

    Open Source Light Weight HTTP 1.1 Client

    After recent experiences and discussions with the Erlang community, ETC have decided to develop and release an HTTP/1.1 lightweight client. You can find more information in the Erlang Open Source section of our website or in the contributions section on Trapexit.

    by Erlang Training and Consulting at June 08, 2009 11:59 PM

    Hypothetical Labs

    It’s erlounge RDU time again

    Wow. Another month has gone by and it’s time for another erlounge. Things have gotten a bit stale of late and I’ve no-one to blame but myself. It’s time I fixed that.

    So.

    This month, and subsequent months, we’re going to start splitting the meetings into two parts. The first, lasting 30-45 minutes will be a directed learning session focusing on some aspect of Erlang. This month we’ll focus on the basics of Erlang, getting it installed and getting acquainted with the syntax. In subsequent months we’ll move onto writing concurrent programs and finally exploring the power of distributed Erlang.

    The second half of the meeting, lasting for at least 30 minutes, will be an informal session where people can continue working on programming exercises, socialize, or give a 5-10 minute lightning talk about something cool they’re doing with Erlang. We’ve got a small but growing Erlang community who’s doing some pretty cool stuff like Mark Imbriaco who put Erlang to work at 37 Signals and Sean Cribbs who’s written a Twitter bot in Erlang.

    Click here to get more info about the meetup and RSVP.

    by kevin at June 08, 2009 06:59 PM

    Erlang Inside

    Sneak Preview of O’Reilly’s new book: Erlang Programming, by Francesco Cesarini and Simon Thompson

    For the past two years in the Erlang community, when someone says “I used The Book to learn Erlang,” it was an unequivocal reference to Joe Armstrong’s book by The Pragmatic Programmers. It has been the guide to learning Erlang, responsible for inducting a whole new generation of developers to the language.

    Yet, now that Erlang Programming, by Francesco Cesarini and Simon Thompson, and published by O’Reilly, is out, the use of the phrase “The Book” will be followed by the question “Which One?” Erlang Programming is a worthy contender for the best Erlang book, and evidence of the growth and maturity of the community.

    I received a pre-release copy of the book, which is scheduled to be published in a few weeks, and I spent the weekend exploring it.

    First, the inevitable comparison between the two books. Joe Armstrong is probably best described as a prophet of the Erlang world. His book is a polemic for parallel programming in general and Erlang specifically. Why shared nothing is everything, why message passing and the actor model are important, and why ‘5 nines’ doesn’t make Erlang break a sweat. Joe set out to tell the world why any system sufficiently advanced is an attempt at rebuilding Erlang. Joe succeeded wildly - he produced a book that gently preaches and occasionally dazzles us with one line IP protocol parsing. Speaking for myself, it was the major influence that drew me into the Erlang world.

    Francesco and Simon’s work is altogether different    . It’s still for the beginner, but it has the luxury of living in a world where many readers already know why they want to learn Erlang. In fact, it’s partially due to the success of Joe’s book that this one can be so good. So they don’t have to proselytize. They can teach and tell stories. We’re ready to listen and we know why we’re here.

    They produced a great book. They could have titled it “Practical Erlang Programming” - its the best way to describe the book. In just a few days of reading i’ve learned practical tips on the best way to structure processes and pass requests through the system to the reason why defensive programming designed to handle every possible input without failing can make your programs much more diffcult to debug.

    The book starts with a great summary of the happenings in the Erlang world in the past few years, useful to show to your boss if he’s still not sure the language has legs. Next, they dive into a summary of “Basic Erlang” - a great primer on data types, atoms, guards, lists, comparing lists and tuples, and the shell. It’s the best introduction to Erlang syntax i’ve read yet.

    It moves on into a description of sequential programming, how to use case statements, guards, the “BIFs” (built in functions), and much more. It even tells us things they’ve left out because they promote bad programming practices, such as the process dictionary, which is an easy way to subvert “shared nothing”.

    They provide a good intro to the Erlang debugger, and then dive into concurrent programming. The overview of message passing assumes you know why you would want to pass messages, and spends more time talking about the best way to architect a multi-process system than the specifics of low-level message passing.

    A section on Design Patterns gives a great overview of the patterns used in Erlang development, including client/server, finite state machine, and event handler patterns. In general, you can tell that these guys have built a lot of real-time systems with Erlang. Tips for patterns to avoid and real-world reasons for why using a particular pattern is wise or foolish pepper the work with depth that only comes from experience.

    Error Handling, dynamic functions, and OTP are all given detailed treatment in their own chapters.

    Unit testing and Test-Driven development are covered in a separate chapter, with specifics on EUnit. The culture of TDD, which recently has become so popular in the Rails community, is clearly not as prevalent in the Erlang world, so it’s important that this is included.

    How to interact with Ruby, Java, and C via Erlang ports is taught, along with a brief mention of how to accomplish interactions between Perl, Python, Haskell and PHP.

    ETS and Dets tables are covered as well as Mnesia. The reasons to use and not to use Mnesia are explained thoroughly, with discussions on gotchas in bringing up Mnesia nodes with extremely large tables. Unfortunately CouchDB and MySQL are mentioned but using them is not discussed. This is a real world need - perhaps a whole book on data access with Erlang is in order?

    The book concludes in Chapter 20 with a discussion of coding style, best practices, and optimizing programs. I can say that my Erlang code will improve just from reading this section and this chapter alone is worth the price of the book.

    What did they leave out? For the beginner, one more chapter on web programming and relational database access would have been a bonus, but the book is thorough enough without it.

    So which book would I recommend? If you can afford both, they complement each other well. But, if you had to buy one, I’d recommend this one, mostly because of the practicality and depth. It builds from a great introduction to the Erlang language to in-depth discussions of the most powerful parts of the platform, with a solid focus on best practices throughout. Joe’s book has made the converts, now Francesco and Simon’s equips the Erlang Faithful with the tools to conquer the world…

    The Safari version of the O’Reilly work will be available around June 15th, and the print version should be available the week of June 22nd.

    by Chad DePue at June 08, 2009 09:37 AM

    June 06, 2009

    Joe's Blog

    Sending CouchDB Update Notifications to RabbitMQ.

    Working at Cloudant I use CouchDB on a daily basis. This evening for fun I decided to write some Ruby to take update notifications and push them into RabbitMQ. There are other examples of using the update notifications and Ruby in Couch such as the view updater out on the Couch wiki. It turned out super simple. There are a few AMQP libraries for Ruby, in this example I am going to use carrot.  It’s based on the  amqp library without all the eventmachine stuff. So here it goes:

    couch_amqp.rb :

    #!/usr/bin/ruby

    require ‘rubygems’
    require ‘carrot’

    def main
    queue = “couchdb”
    run = true
    couchq = Carrot.queue(:queue => queue)

    while run do

    notifications = gets

    if notifications == nil
    run = false
    else
    couchq.publish(notifications)
    end

    end
    end

    main

    As you can tell we connect to a queue called “couchdb” on by default this is on localhost. Next we have a loop that continually runs and grabs updates from stdin. I then publish each notification to the queue and that’s that. To get the messages out of the queue I used irb and carrot.

    [user@host ~]$ irb
    irb(main):001:0> require ‘rubygems’
    => true
    irb(main):002:0> require ‘carrot’
    => true
    irb(main):003:0> couchq = Carrot.queue(:queue => “couchdb”)
    => #<Carrot::AMQP::Queue:0×7f8d2284b640 <snip>
    irb(main):004:0> couchq.pop
    => “{\”type\”:\”updated\”,\”db\”:\”test1\”}\n”

    So yeah, pretty simple stuff. Go ahead relax! :)

    [EDIT 06/05/2009 2326 PST : Don't forget to add the entry to your local.ini]

    [update_notification]

    couch_amqp=/PATH/TO/couch_amqp.rb

    by joe at June 06, 2009 06:09 AM

    Damien Katz

    Why CouchDB?

    Why CouchDB?, the first chapter from upcoming CouchDB O'Reilly Book.

    couchdb_book_cover.png

    by Damien Katz at June 06, 2009 01:32 AM

    June 05, 2009

    Tony Arcieri

    Dear Viacom: You're Doing It Wrong

    I've been very excited about the upcoming release of The Beatles: Rock Band after hearing about it earlier this week. It's the first PS3 game sold on disc I'm going to snap up since, well, Rock Band 2. Beyond the simple fact that it's Rock Band loaded with Beatles music (and I love the Beatles), I've been rather impressed by the visual style, particularly this intro video directed by Pete Candeland of Gorillaz music video fame:


    Hopefully this video hasn't been flagged for a copyright violation by the time you read this post. It rules

    However, I was rather surprised to see that a particular copy of the intro video was flagge