September 02, 2010

Process-one Blogs

TextOne for iPhone, with photo sharing!

Our iPhone version of TextOne, the simple and fast messenger for smartphones, has received a nice photo sharing feature.

TextOne, the simple, light and fast smartphone messenger, is now offering a new feature to share photos with your friends.

Easy, a new icon appears at the top right of the conversation screen, you just need to tap it.

There, you can choose to take a picture or select one from your albums... and add the text you wish.

This will appear in the conversation as a taped instant photo, like any other messages.

You can also browse the photos exchanged with your contacts.

Of course, this also works with TextOne Pink!

Photo sharing in TextOne will let you share good moments with the people to whom you are close. Or alternatively play ;-)

And finally, you can play with it

by Nicolas Vérité at September 02, 2010 08:43 AM

August 31, 2010

Programming in the 21st Century

Common Sense, Part 1

There's a photo of mine in the September 2010 issue of Popular Photography. I'm excited about it; my photo credits are few and far between, and it brings back the feelings I had when I wrote for magazines long ago. Completely ignoring the subject of the image, there are couple of surprising facts about it.

The first is that it was a taken on a circa-2004 Canon PowerShot G5, a camera with a maximum resolution of five megapixels.

The second is that it's a doubly-compressed JPEG. The original photo was a JPEG, then I adjusted the colors and contrast a bit, and saved it out as a new JPEG. Each save lost some of the image quality. I was perfectly willing to submit the adjusted photo as a giant TIFF to avoid that second compression step, but was told not to worry about it; the JPEG would be fine.

Yet there it is: the five megapixel, doubly-compressed photo, printed across almost two pages of the magazine. And those two technical facts are irrelevant. I can't tell the difference; it looks great in print.

Now it is an impressionistic shot, so it could just be that the technical flaws aren't noticeable in this case. Fortunately, I have another anecdote to back it up.

Last year I was in New Mexico and took a lot of photos. After I got back home, I decided to get a few photo books printed. The source images were all twelve megapixel JPEGs, but the book layout software recommended a six megapixel limit. I cut the resolution in half, again twice-compressing them. When I got the finished books back, the full-page photos were sharp and beautiful.

The standard, pedantic advice about printing photos is that resolution is everything. Shoot as high as possible. Better yet, save everything as RAW files, so there's no lossy compression. Any JPEG compression below maximum is unacceptable. Double-compression is an error of the highest order, one only made by rank amateurs. And so it goes. But I know from personal experience that while it sounds authoritative, and while it's most likely given in a well-meaning manner, it's advice that's endlessly repeated in a loose, "how could it possibly be wrong?" sort of way and never actually tested.

by James Hague at August 31, 2010 06:00 AM

August 30, 2010

Erlang Inside

Russia’s numerous contributions to the Erlang community – Dmitrii Dimandt, creator of erlanger.ru

Erlang Inside is sponsored by Inaka Networks, Erlang consultants combining experience with Telecom to build high performance web applications. This week, Erlang Inside has a guest post from Dmitrii Dimandt, a Russian developer and creator of the site erlanger.ru. I’d like to talk about the Russian Erlang community and what it’s doing with Erlang. In [...]

by Chad DePue at August 30, 2010 04:58 PM

August 29, 2010

Chicago Erlang User Group

6/2/2010 OTP Plus

Here's the video from the group meeting on 6/2/2010 on "How to use OTP". OTP makes coding with Erlang just so much easier. OTP just abstracts so much coding away. Jon Patton discussed using gen_servers and Tristan Sloughter discussed using gen_supervisors. Great talks guys!!A couple of other impromptu talks were also given. Martin Logan talked about terminology and how we can discuss and

by noreply@blogger.com (Barry Nicholson) at August 29, 2010 09:15 PM

August 28, 2010

Nicolas Charpentier's blog

Next language to learn: Haskell

Remembering  ”The pragmatic programmer: Fro, Journeyman to Master”, each developer should learn a new language each year. This year, I will give its chance to Haskell. Why Haskell ? Scala was a candidate for this year but Haskell has a strong reputation in the functional world. I came to functional with Erlang because of its concurrent and distribution [...]

by charpi at August 28, 2010 07:36 PM

Programming in the 21st Century

Personal Programming

I've mentioned before that this site is generated by a small Perl script. How small? Exactly 6838 bytes, which includes comments and an HTML template. Mentioning Perl may horrify you if you came here to read about Erlang, but it's a good match for the problem. Those 6838 bytes have been so pleasant to work on that I wanted to talk about them a bit.

I've used well-known blogging applications, and each time I've come away with the same bad taste, one that's caused by a combination of quirky rich formatting and having to edit text in a small window inside of a browser. I don't want to worry about presentation details: choosing fonts, line spacing, etc. It's surprising how often there are subtle mismatches between the formatting shown in a WYSIWYG editing window and what the final result looks like. Where did that extra blank line come from? Why do some paragraphs have padding below them but others don't?

I decided to see if I could bypass all of that and have a folder of entries marked-up with basic annotations, then have a way to convert that entire folder into a real site. And that's pretty much what I ended up with. The sidebar and "Previously" list and dated permalink are all automatically generated. Ditto for the atom feed and archives page. The command-line formatter lets me rebuild any page, defaulting to the newest entry. If I want to change the overall layout of the site, I can regenerate all of the pages in a second or so.

There are still legitimate questions about the path I chose. "Why not grab an open source program and modify it to fit your needs?" "You do realize that you decided to write an entirely new system from scratch, just because you didn't like a few things in existing programs; surely that's a serious net loss?"

My response is simple: I did it because it was easy. If I get annoyed with some feature of Microsoft Word, I'm not going to think even for a second about writing my own alternative. But the site generation program just took a bit of tinkering here and there over the course of a weekend. It never felt daunting. It didn't require anything I'd label as "engineering." I spent more time getting the site design and style sheet right, something I would have done even if I used other software.

Since then, I've made small adjustments and additions to the original script. I added the archive page nine months later. Earlier this year I added a fix for some smaller feed aggregation sites that don't properly handle relative links. Just today I added mark-up support for block quotes. That last one took ten minutes and four lines of code.

If this suddenly got complicated, if I needed to support reader comments and ten different feed formats and who knows what else, I'd give it up. I have no interest in turning these 6838 bytes into something that requires a grand architecture to keep from collapsing. But there's some magic in a solution that's direct, reliable, easy to understand, and one that fits my personal vision of how it should work.

(If you liked this, you might like Micro-Build Systems and the Death of a Prominent DSL.)

by James Hague at August 28, 2010 06:00 AM

August 25, 2010

Erlang Factory News

Erlang User Conference Registration Open with a Very-Early Bird Rate!

Registration for the Sixteenth International Erlang User Conference and Erlang University courses are now open. The first 100 people to register will receive a Very-Early Bird Discount and pay only SEK 750 including VAT! DON’T MISS OUT as last year's conference sold out.

August 25, 2010 01:46 PM

August 19, 2010

Process-one Blogs

Tsung 1.3.3 has been released

Tsung 1.3.3 fixes severals bugs. It fixes support for SSL with erlang R14A.

Tsung 1.3.3 brings some minor bugfixes.

• fixed parent proxy not working in 1.3.x (tested with 1.3.2 and 1.3.0).
• fixed url substitution is broken in some cases
• fixed Tsung not using sessions with low probabilities
• fixed SSL not working with erlang R14A
• fixed failure when a proxy is used and an URL substitution is set
• fixed HTTP cookies support when a proxy is used
• fixed hanging at the beginning using distributed setup
• fixed "if" statement, which is not allowed in a transaction

by Marek Foss at August 19, 2010 04:18 PM

August 09, 2010

Chicago Erlang User Group

ErlangCamp Chicago 2010 Registration Open

We are pleased to announce ErlangCamp Chicago 2010 (http://erlangcamp.com/) has gone live and is ready to take registrations. If you want to learn how to build production ready Erlang/OTP systems, this workshop is for you.

by noreply@blogger.com (Jordan Wilberding) at August 09, 2010 01:31 AM

August 07, 2010

EazyErl!

erlang: Simple Debug macro

When testing things I need some quick way to debug "a la" printf() style. Old habits :)

Here's is the simple debug macro I use:

-ifdef(debug).
-define(DEBUG(Format, Args), io:format("~s.~w: DEBUG: " ++ Format, [ ?MODULE, ?LINE | Args])).
-else.
-define(DEBUG(Format, Args), true).
-endif.

-include("debug.hrl").

-module(utils_compile).

-export([c/1, c/2, d/1, d/2]).

c(FilePattern) ->
 c(FilePattern, "../ebin").

c(FilePattern, OutDir) ->
 filelib:fold_files(".", FilePattern ++ ".erl$", false, 
 fun(X, _Acc) -> 
  io:format("Compiling ~p~n", [ X ]), 
  compile:file(X, [{outdir, OutDir}, report]) end, []).

d(FilePattern) ->
 d(FilePattern, "../ebin").

d(FilePattern, OutDir) ->
 filelib:fold_files(".", FilePattern ++ ".erl$", false, 
 fun(X, _Acc) -> 
  io:format("Compiling (debug) ~p~n", [ X ]), 
  compile:file(X, [{outdir, OutDir}, {d, debug}, report]) end, []).

by rolphin (noreply@blogger.com) at August 07, 2010 02:46 PM

August 06, 2010

Erlang Inside

ErlangCamp Chicago 2010 – Oct 23rd/24th

Incredibly cheap at $100.00 USD - Two days of Erlang/OTP sessions in Chicago - check it out. http://erlangcamp.com/

by Chad DePue at August 06, 2010 05:35 PM

1011 Ltd Blog

Why Google Wave "failed"

Google Announced this week that they "don't plan to continue Wave as a standalone product", which was a surprise. The twitterverse and technology press went crazy with the news. Even the Financial Times reported on the demise of Wave. I've been involved on the fringes of Wave, building clients, gadgets, prototyping servers, using the web client extensively, but since Google I/O 2010 I don't think I've touched it. Why did it "fail", and how can other projects learn from the mistakes?

August 06, 2010 09:19 AM

August 03, 2010

Process-one Blogs

ejabberd 2.1.5 and exmpp 0.9.5 bugfix releases

We are pleased to announce the bugfix releases ejabberd 2.1.5 and exmpp 0.9.5.

Regarding ejabberd 2.1.5:

The main changes are:

• Erlang/OTP R12 support fixed
• Erlang/OTP R14A support added
• OpenSSL 0.9.8 or higher is required
• BOSH: New optional connection attribute process-delay
• C2S: Don't ask for client certificate when using TLS
• C2S: Inform client that SSL session caching is disabled
• CTL: Fix problem when FIREWALL_WINDOW options for erl kernel were used
• CTL: Some systems delete the lock dir; in such case don't use Flock at all
• Caps: Support all the hash functions required by XEP-0115
• Config: Fix typo in --enable-transient_supervisors
• Config: New configure option: --enable-nif
• Extauth: Support parallel script running
• MUC: Allow admins to see private rooms in disco
• ODBC: Correct handling of SQL boolean types
• ODBC: Discard too old queued requests (the caller has already got a timeout)
• ODBC: Fixes wrong SQL escaping when --enable-full-xml is set
• ODBC: Use ets instead of asking supervisor in ejabberd_odbc_sup:get_pids/1
• Pubsub: Enforce disco features results
• S2S: When logging s2s out connection attempt or success, log if TLS is used
• Shared Rosters: When account is deleted, delete also member of stored rosters

Release Notes

Check the Release Notes for a more complete list of changes:
http://www.process-one.net/en/ejabberd/release_notes/release_note_ejabberd_2.1.5

If you upgrade from ejabberd 2.0.5 or older, read carefully the release notes of ejabberd 2.1.0 too, because there were several changes in the  installation path and the configuration options.

Links

The list of solved tickets since previous version is available on ProcessOne bug tracker:
http://redir.process-one.net/ejabberd-2.1.5

ejabberd 2.1.5 is available as source code package and binary installers for Linux 32 bits, 64 bits, Mac OS X Intel, and Windows:
http://www.process-one.net/en/ejabberd/downloads

Regarding exmpp 0.9.5:

Brief summary of changes:

• Add method to retrieve underling connection properties
• Configurable Zlib support
• Fix BOSH that didn't work
• Modify exmpp_component to use exmpp_socket instead of exmpp_tcp
• Negotiate zlib compression before OR after SASL
• Raise requirement to Erlang/OTP R12B-5
• Replace calls to OTP module ssl_pkix with public_key
• STARTTLS support
• Session enhancements
• Stream compression support for session

Links

exmpp home page:

http://support.process-one.net/doc/display/EXMPP/ or easier to remember: http://exmpp.org/.

Download exmpp 0.9.5 source code package from:

http://download.process-one.net/exmpp/

You can also check the ProcessOne Labs page:

http://www.process-one.net/en/labs/

by Jérôme Sautret at August 03, 2010 03:35 PM

July 30, 2010

Hypothetical Labs

Part 1: Introducing riak_core

• What is riak_core?

  riak_core is a single OTP application which provides all the services necessary to write a modern, well-behaved distributed application. riak_core began as part of Riak. Since the code was generally useful in building all kinds of distributed applications we decided to refactor and separate the core bits into their own codebase to make it easier to use.

  Distributed systems are complex and some of that complexity shows in the amount of features available in riak_core. Rather than dive deeply into code, I’m going to separate the features into broad categories and give an overview of each.

  Note: If you’re the impatient type and want to skip ahead and start reading code, you can check out the source to riak_core via hg or git.

  Node Liveness & Membership

  riak_core_node_watcher is the process responsible for tracking the status of nodes within a riak_core cluster. It uses net_kernel to efficiently monitor many nodes. riak_core_node_watcher also has the capability to take a node out of the cluster programmatically. This is useful in situations where a brief node outage is necessary but you don’t want to stop the server software completely.

  riak_core_node_watcher also provides an API for advertising and locating services around the cluster. This is useful in clusters where nodes provide a specialized service, like a CUDA compute node, which is used by other nodes in the cluster.

  riak_core_node_watch_events cooperates with riak_core_node_watcher to generate events based on node activity, i.e. joining or leaving the cluster, etc. Interested parties can register callback functions which will be called as events occur.

  Partitioning & Distributing Work

  riak_core uses a master/worker configuration on each node to manage the execution of work units. Consistent hashing is used to determine which target node(s) to send the request and the master process on each node farms out the request to the actual workers. riak_core calls worker processes vnodes. The coordinating process is the vnode_master.

  The partitioning and distribution logic inside riak_core also handles hinted handoff when required. Hinted handoff occurs as a result of a node failure or outage. In order to assure availability, most clustered systems will use operational nodes in place of down nodes. When the down node comes back the cluster needs to migrate the data from its temporary home on the substitute nodes to the data’s permanent home on the restored node. This process is called hinted handoff and is managed by components inside riak_core. riak_core also handles migrating partitions to new nodes when they join the cluster such that all work continues to be evenly partitioned to all cluster members.

  riak_core_vnode_master starts all the worker vnodes on a given node and routes requests to
  the vnodes as the cluster runs.

  riak_core_vnode is an OTP behavior wrapping all the boilerplate logic required to implement a vnode. Application-specific vnodes need to implement a handful of callback functions in order to participate in handoff sessions and receive work units from the master.

  Cluster State

  A riak_core cluster stores global state in a ring structure. The state information is transferred between nodes in the cluster in a controlled manner to keep all cluster members in sync. This process is referred to as “gossiping”.

  riak_core_ring is the module used to create and manipulate the ring state data shared by all nodes in the cluster. Ring state data includes items like partition ownership and cluster-specific ring metadata. Riak KV stores bucket metadata in the ring metadata, for example.

  riak_core_ring_manager manages the cluster ring for a node. It is the main entry point for application code accessing the ring, via riak_core_ring_manager:get_my_ring/1, and also keeps a persistent snapshot of the ring in sync with the current ring state.

  riak_core_gossip manages the ring gossip process and insures the ring is generally consistent across the cluster.

• What’s the plan?

  Over the next several months I’m going to cover the process of building a real application in a series of posts to this blog where each post covers some aspect of system building with riak_core. All of the source to the application will be published under the Apache2 licensed and shared via a public repo on github.

  And what type of application will we build? Since the goal of this series is to illustrate how to build distributed systems using riak_core and also satisfy my own technical curiosity I’ve decided to build a distributed graph database. A graph database should provide enough use cases to really exercise riak_core while at the same time not obscuring the core learning experience in tons of complexity.

  Thanks to Sean Cribbs and Andy Gross for providing helpful review and feedback.

by kevin at July 30, 2010 05:30 PM

July 29, 2010

Learn You Some Erlang

Errors and Processes

Errors and Processes might be the last chapter released before Learn You Some Erlang hits its one year mark (see: http://ferd.ca/learn-you-some-erlang-a-year-in.html). Thanks to all dedicated readers and reviewers. This chapter covers the building blocks of error management with concurrent Erlang: linking processes together, handling different signals sent between processes when they crash, monitoring actors and naming them. This should be the last material required before we build a real concurrent application in Erlang.

July 29, 2010 12:30 PM

Programming in the 21st Century

Stop the Vertical Tab Madness

In One Small Step Toward Reducing Programming Language Complexity I added "Who even knows what "\v" (vertical tab) does?" as an off the cuff comment. Re-reading that made me realize something that's blatantly obvious in retrospect, so obvious that I've gone all this time without noticing it:

No one is actually using the vertical tab escape sequence.

And I truly mean "no one." If I could stealthily patch the compiler for any language supporting the "\v" escape so I'd receive mail whenever it occurred in source code, then I could trace actual uses of it. I'm willing to bet that all the mail would come from beginners trying to figure out what the heck "\v" actually does, and then giving up when they realize it doesn't do anything. That's because it doesn't do anything, except with some particular printers and terminal emulators, and in those cases you're better off not relying on it anyway.

And yet this crazy old feature, one that no one understands or uses, one that doesn't even do anything in most cases, not only gets special syntax in modern programming languages, it's consistently given space in the documentation, even in tutorials. It's in the official Lua docs. It's in MIT web course notes about printf. It's in the books Programming in Python 3 and Python Essential Reference. It's in an introduction to Python strings. It's in the standard Erlang documentation, too.

[Insert conspiracy theory involving the Illuminati here.]

Here's my simple plea: stop it. Stop mentioning vertical tabs in tutorials and language references. Drop the "\v" sequence in all future programming languages. Retroactively remove it from recent languages, like Python 3.0. Yes, ASCII character number 11 isn't going away, but there's no reason to draw attention to a relic of computing past.

Surprisingly, the "\v" sequence was removed from one language during the last decade: Perl. And even more surprisngly, there's a 2010 proposal to add escaped character sequences, including vertical tab, to the famously minimal language, Forth.

(If you liked this, you might like Kilobyte Constants, a Simple and Beautiful Idea that Hasn't Caught On.)

by James Hague at July 29, 2010 06:00 AM

July 28, 2010

Joe's Blog

SurgeCon 2010

If you haven’t heard about Surge, it’s a new web operations conference presented by the smart folks at OmniTI. They have amassed a good list of speakers including guys like John Allspaw and Theo Schlossnagle. I also happen to have been invited to talk about the cloud, Cloudant and all sorts of good stuff.

by joe at July 28, 2010 03:00 AM

July 24, 2010

Programming in the 21st Century

One Small Step Toward Reducing Programming Language Complexity

I've taught Python a couple of times. Something that experience made clear to me is just how many concepts and features there are, even in a language designed to be simple. I kept finding myself saying "Oh, and there's one more thing..."

Take something that you'd run into early on, like displaying what's in a dictionary:

for key, value in dictionary.iteritems():
   print key, value

"Content-type: text/plain" + NL + NL

"Content-type: text/plain" ++ NL ++ NL

r"c:\my_projects\input.txt"

by James Hague at July 24, 2010 06:00 AM

July 22, 2010

Erlang Inside

Erlang and REST – an interview with Steve Vinoski

Interview recorded at Erlang Factory, where Steve Vinoski discusses the key concepts of RPC frameworks such as CORBA, and how that compares to REST, and how Erlang fits into both worlds

by Chad DePue at July 22, 2010 12:23 PM

July 19, 2010

Joe's Blog

Adding Health Checks to Deckard from Chef.

Recently, we (at Cloudant) open sourced Deckard, a HTTP content check monitoring system based on CouchDB. One of the best bits about using Couch is that it gives you a ReST API and with Deckard it can be used to add new health checks. Doing a simple PUT adds new URLs to monitor. At Cloudant we love Chef and use it for everything. Chef has things called resources and providers. Resources are abstractions that describe the state you want a machine to be in. Providers perform the actions described by a resource. A good example is using the package resource on Centos uses yum while on Ubuntu it uses apt-get. The resource abstracts that away, letting the provider (and node) deal with the specifics on how to install the package. This makes your recipes nice and DRY, use the same code to install packages on all sorts of platforms. There are resources and providers for anything from installing packages to even one I wrote for executing Erlang code via erl_call. One resource that works well with Deckard is the HTTP request resource, using it makes it very easy to add health checks from your cookbooks. We use something like the following code to add checks to new nodes at Cloudant:

This code will add the document describing the check to the monitor_content_check database and then create a file so we can use “not_if” and Chef won’t attempt to add the check twice. Pretty cool stuff and even more reason that everything should have an API. Even cooler than this example would be to use Chef Search to do the same thing but I’ll save that for another blog post.

by joe at July 19, 2010 08:52 PM

July 17, 2010

Trapexit's Erlang Blog Filter

New Column, New Interview

A couple newsworthy items:

• The July/August 2010 issue (PDF) of The Functional Web column is available, this time with guest columnist Aaron Bedra writing about using Clojure and the Compojure web framework on the Google App Engine platform. Pretty interesting read, and plenty of code examples.

• Sadek Drobi interviewed me about Erlang and REST at Erlang Factory back in March, and the interview is now up at InfoQ.

by steve at July 17, 2010 06:01 PM

Programming in the 21st Century

Free Your Technical Aesthetic from the 1970s

In the early 1990s, I used Unix professionally for a few years. It wasn't the official Unix, nor was it Linux, but Sun's variant called SunOS. By "used" I mean I wrote commercial, embedded software entirely a Unix environment. I edited 10,000+ line files in vi. Not vim. The original "one file loaded at a time" vi.

At the time, Unix felt clunky and old. I spent a lot of time in a library room down the hall, going through the shelves of manuals. It took me a long time to discover the umask command for changing the default file permissions and to understand the difference between .bashrc and .bash_profile and how to use tar.

By way of comparison, on my home PC I used a third-party command shell called 4DOS (later 4NT, and it's still available for Windows 7 as TCC LE). It had a wonderful command line history mechanism: type part of a command, then press up-arrow. The bash bang-notation felt like some weird mainframe relic. 4DOS had a built-in, full-screen text file viewer. The Unix equivalent was the minimalist less command. 4DOS help was colorful and pretty and hyperlinked. Documentation paged through as man pages was several steps backward.

The Unix system nailed the core tech that consumer-level computers were way behind on: stability and responsiveness in a networked, multitasking environment. It was ugly, but reliable.

In 2006, I got back into using Unix again (aside from some day-job stuff with Linux ten years ago) in the guise of OS X on a MacBook. The umask command is still there. Ditto for .bashrc and .bash_profile and all the odd command line switches for tar and the clunky bang-notation for history. I'm torn between wonderment that all those same quirks and design choices still live on...and shocked incredulity that all those same quirks and design choices live on.

Enough time has passed since the silly days of crazed Linux advocacy that I'm comfortable pointing out the three reasons Unix makes sense:

1. It works.
2. It's reliable.
3. It stays constant.

But don't--do not--ever, make the mistake of those benefits being a reason to use Unix as a basis for your technical or design aesthetic. Yes, there are some textbook cases where pipelining commands together is impressive, but that's a minor point. Yes, having a small tool for a specific job sometimes works, but it just as often doesn't. ("Those days are dead and gone and the eulogy was delivered by Perl," Rob Pike, 2004.) Use Unix-like systems because of the three benefits above, and simultaneously realize that it's a crusty old system from a bygone era. If you put it up on a pedestal as a thing of beauty, you lose all hope of breaking away from a sadly outdated programmer aesthetic.

(If you liked this, you might like My Road to Erlang.)

by James Hague at July 17, 2010 06:00 AM

July 14, 2010

Erlang Factory News

Erlang User Conference 2010 Talk Submission Open!

The 2010 Erlang User Conference in Stockholm will be held on Tuesday 16th November 2010. The venue will again be the Old Astoria Cinema in Nybrogatan.

On Monday November 15th there will be a day of tutorials. This will allow delegates to again optimise their visit to Stockholm to take in both these events.

We have opened the submissions page. To submit a proposal for a talk or tutorial, visit the Talk Submission page. 

More details will be forthcoming in the near future. Keep a look out or join the Erlang Factory Mailing list!

July 14, 2010 07:46 PM

July 09, 2010

Programming in the 21st Century

Explaining Functional Programming to Eight-Year-Olds

"Map" and "fold" are two fundamentals of functional programming. One of them is trivially easy to understand and use. The other is not, but that has more to do with trying to fit it into a particular view of functional programming than with it actually being tricky.

There's not much to say about map. Given a list and a function, you create a new list by applying the same function to each element. There's even special syntax for this in some languages which removes any confusion about whether the calling sequence is map(Function, List) or map(List, Function). Here's Erlang code to increment the values in List:

[X + 1 || X <- List]

lists:foldl(fun(X, Sum) -> X + Sum end, 0, List)

1 2 3 4 5 6

1 + 2 + 3 + 4 + 5 + 6

by James Hague at July 09, 2010 06:00 AM

July 08, 2010

Erlang Inside

PostgreSQL Erlang client library epgsql supports asynchronous messages from LISTEN/NOTIFY events

If you use PostgreSQL with Erlang, you're probably already familiar with the epgsql client library. Something that slipped my attention was the addition of...

by Chad DePue at July 08, 2010 07:27 PM

July 05, 2010

Process-one Blogs

ProcessOne releases OneCached, a Memcached in erlang

ProcessOne has just released OneCached, a Memcached server and client implementation written in Erlang.

OneCached is a new Memcached server and client implementation written from scratch in Erlang by ProcessOne.

From the Memcached website:

What is Memcached?

Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

Memcached is simple yet powerful. Its simple design promotes quick deployment, ease of development, and solves many problems facing large data caches. Its API is available for most popular languages.

OneCached supports the set, add, replace, get, incr, decr, delete, flush_all and quit commands. It doesn't handle expiration time.

You call pull the source code from the public repository at: https://git.process-one.net/onecached

git clone git://git.process-one.net/onecached/mainline.git

To compile, just run make, and to start, just type:

bin/onecachedctl start

OneCached is released under the Erlang Public License (EPL), version 1.1. It is available from ProcessOne Labs.

by Nicolas Vérité at July 05, 2010 04:29 PM

Erlang Inside

New Erlang job board – totally-erlang.com

The Zotonic guys put together a new Erlang job board at http://totally-erlang.com/

by Chad DePue at July 05, 2010 11:27 AM

July 03, 2010

RedHotErlang

Playing with LFE macros

Summertime, it's warm, it's sunny, I'm on vaccation, it's [LFE][1]-time. [Lisp macros][2] are cool. With a Lisp macro you do program transformations where you have the full language available to express these transformations. This means that you *ea...

by tobbe at July 03, 2010 06:48 AM

July 01, 2010

Programming in the 21st Century

What Do People Like?

I wrote Flickr as a Business Simulator in earnest, but I think it was interpreted more as a theoretical piece. When you build something with the eventual goal of releasing it to the world, the key question is "Will people like this?" And, really, you just won't know until you do it. There's nothing like the actions of tens of thousands of independently acting individuals who have no regard for your watertight theories. What Flickr provides is a way to make lots of quick little "product" releases, and see if your expectations line up with reality. Is this my primary use of Flickr? No! But the educational opportunity is there, regardless. Click on a photo to go to the Flickr page.

The rest of this entry is an annotated list of some photos I've posted to Flickr, with both my conjectures of how they'd be received and what actually happened. In each case the photo comes first with the commentary following.



I sat on this photo for a while after shooting it. I thought it was cliche, something everyone had already seen many times. A real photographer wouldn't bother to recreate such an image. Then I posted it...and it got an immediate string of comments and favorites. It's still one my top ten overall Flickr photos according to the stats system. My invented emphasis on originality didn't matter.



I thought these skid marks in front of a local liquor superstore were photoworthy, but the result didn't grab me. Like the sunset wheat photo, it took on a life of its own on Flickr. Was the hook in the impossibility of those tire tracks? That they look like a signature? Why was I unable to see the appeal before uploading it? It even ended up--with permission--in Scott Berkun's The Myths of Innovation (O'Reilly, 2007).



This one I liked immediately. The red arrow. The odd framing. The blown-out white background that makes the rust pop. The Flickr reaction...well there wasn't one. Is the industrial decay photo niche saturated? Would it have been a hit if I worked at getting hundreds of dedicated followers first? Or maybe I like it because it's better than other photos I've taken recently, but not all that great in absolute terms?



Oh so cleverly titled "I'm Lovin' IT!" I knew this was a novelty. It pulled in some novelty linkage as a ha-ha photo of the day sort of thing. It didn't get anywhere near the exposure of that Yahoo ad next to the 404 distance on a home run fence. The traffic from "I'm Lovin' IT!" was transient, adding points to the view counter, but as they weren't Flickr users they didn't add comments or favorites. In the end it was an empty success.

by James Hague at July 01, 2010 06:00 AM

June 30, 2010

Process-one Blogs

Hosted.IM gets gateways to legacy IM systems

The XMPP service for your domains has just received a much demanded feature: gateways to legacy IM systems.

Hosted.IM is the cloud service for everything IM, by ProcessOne. This service is free for up to 5 users teams.

We have added gateways to main legacy IM systems in the world:

• AOL: aol.hosted.im
• ICQ: icq.hosted.im
• MSN: msn.hosted.im
• Twitter: twitter.hosted.im
• Yahoo: yahoo.hosted.im

These gateways are also free for up to 5 users teams.

by Nicolas Vérité at June 30, 2010 03:08 PM

June 29, 2010

Erlang Inside

erldocs.com updated with R14A support, mochiweb, and available for your own projects

Dale Harvey just recently updated erldocs.com with R14A support. If you are doing any erlang development, or even looking to just learn a bit about erlang, I highly recommend using erldocs over the official documentation.

by Chad DePue at June 29, 2010 12:53 PM

Programming in the 21st Century

Tricky When You Least Expect It

Here's a problem: You've got a satellite dish that can be rotated to any absolute angle from 0 to 360 degrees. If you think of the dish as being attached to a pole sticking out of the ground, that's what the dish rotates around. Given a starting angle and a desired angle, how many degrees do you rotate the dish by?

An example should clarify this. If the initial angle is 0 degrees, and the goal is to be at 10 degrees, that's easy. You rotate by 10 degrees. If you're at 10 degrees and you want to end up at 8 degrees, then rotate -2 degrees. It looks at lot like all you have to do is subtract the starting angle from the ending angle, and that's that.

But if the starting angle is 10 and the ending angle is 350...hmmm. 350 - 10 = 340, but that's the long way around. No one would do that. It makes more sense to rotate by -20 degrees. With this in mind and some experimenting, here's a reasonable looking solution (in Erlang, but it could easily be any language):

angle_diff(Begin, End) ->
   D = End - Begin,
   DA = abs(D),
   case DA > 180 of
      true -> -(360 - DA);
      _ -> D
   end.

angle_diff(Begin, End) ->
   D = End - Begin,
   DA = abs(D),
   case {DA > 180, D > 0} of
      {true, true} -> DA - 360;
      {true, _}    -> 360 - DA;
      _ -> D
   end.

by James Hague at June 29, 2010 06:00 AM

June 28, 2010

BGH's Blog

Erlang Programming Exercise: 5-2

Exercise 5-2 involves changing the frequency server from earlier in the chapter. Here is the task list:

• Restrict the deallocation of a frequency so that only the client that allocated it can deallocate it.
• Fix a bug: Deallocating a frequency that isn't allocated crashes the server
• Only allow shutdown of the frequency server if there are no allocated frequencies
• Limit the number of frequencies a client can allocate to 3

I worked through the tasks and then had a bit of fun with the server. I changed the shutdown sequence so that deallocations work while shutting down but allocations return an error. Once all of the outstanding frequencies have been returned it will complete the shutdown.

As you can see in the code below, I was playing around with typer and dialyzer. Please take a look and let me know if you have any suggestions.

Thanks!
Halzy

[erlang]
-module(frequency).
-export([start/0, stop/0, allocate/0, deallocate/1, deallocate_all/0]).

-export([init/0]).

-define(SERVER, ?MODULE).

-type frequency() :: pos_integer().
-type used_frequency() :: { frequency(), pid() }.

-record(state, {
free = [] :: [frequency()],
allocated = [] :: [used_frequency()],
shutting_down = false :: boolean(),
max_allocations = 3 :: pos_integer()
}).

%% These are the start functions used to create and
%% initialize the server.

start() ->
register(?SERVER, spawn(?MODULE, init, [])).

-spec init() -> ok.
init() ->
State = #state{
free=get_frequencies()
},
loop(State).

% Hard Coded
-spec get_frequencies() -> [frequency()].
get_frequencies() ->
[10, 11, 12, 13, 14, 15].

%% The client Functions

-spec stop() -> ok.
stop() ->
call(stop).

-spec allocate() -> {error, atom()} | pos_integer().
allocate() ->
call(allocate).

-spec deallocate(frequency()) -> ok.
deallocate(Freq) ->
call({deallocate, Freq}).

-spec deallocate_all() -> ok.
deallocate_all() ->
call(deallocate_all).

%% We hide all message passing and the message
%% protocol in a functional interface

call(Message) ->
?SERVER ! {request, self(), Message},
receive
{reply, Reply} ->
Reply
end.

%% The Main Loop

-spec loop(#state{}) -> ok.
%% [bgh] if we are shutting down, and have no outstanding frequencies, end
loop(#state{shutting_down=true, allocated=[]}) ->
ok;
loop(#state{shutting_down=ShuttingDown}=State) ->
receive
{request, Pid, allocate} when false==ShuttingDown ->
{UpdatedState, Reply} = allocate(State, Pid),
reply(Pid, Reply),
loop(UpdatedState);
{request, Pid, stop} when false==ShuttingDown ->
reply(Pid, ok),
loop(State#state{shutting_down=true});
{request, Pid, {deallocate, Freq}} ->
UpdatedState = deallocate(State, Pid, Freq),
reply(Pid, ok),
loop(UpdatedState);
{request, Pid, deallocate_all} ->
UpdatedState = deallocate_all(State, Pid),
reply(Pid, ok),
loop(UpdatedState);
{request, Pid, _} when true == ShuttingDown ->
reply(Pid, {error, shutting_down}),
loop(State)
end.

reply(Pid, Reply) ->
Pid ! {reply, Reply}.

%% The Internal Help Functions used to allocate and
%% deallocate frequencies.

-spec allocate(#state{free::[]}, pid()) ->
{#state{}, {error, no_frequency}};
(#state{free::[frequency(),...]}, pid()) ->
{#state{}, {error, max_allocations}} |
{#state{}, {ok, frequency()}}.
allocate(#state{free=[]}=State, _Pid) ->
{State, {error, no_frequency}};
allocate(#state{free=[Freq|Free], allocated=Allocated}=State, Pid) ->
case can_allocate(Allocated, Pid, State#state.max_allocations) of
false ->
{State, {error, max_allocations}};
true ->
{State#state{free=Free, allocated=[{Freq, Pid}|Allocated]}, {ok, Freq}}
end.

-spec deallocate(#state{}, pid(), frequency()) -> #state{}.
deallocate(#state{free=Free, allocated=Allocated}=State, Pid, Freq) ->
case lists:keytake(Freq, 1, Allocated) of
false ->
State;
{value, {Freq, Pid}, NewAllocated} ->
State#state{free=[Freq|Free], allocated=NewAllocated};
{value, {Freq, _WrongPid}, _NewAllocated} ->
State
end.

-spec pid_match(used_frequency(), pid()) -> boolean().
pid_match({_Freq, Pid}, Pid) -> true;
pid_match({_Freq, _OtherPid}, _Pid) -> false.

-spec deallocate_all(#state{}, pid()) -> #state{}.
deallocate_all(#state{free=Free, allocated=Allocated}=State, Pid) ->
{ToDeallocate, UpdatedAllocated} = lists:partition(fun(Elem) -> pid_match(Elem, Pid) end, Allocated),
{ReleasedFreqs, _Pids} = lists:unzip(ToDeallocate),
UpdatedFree = ReleasedFreqs ++ Free,
State#state{free=UpdatedFree, allocated=UpdatedAllocated}.

-spec can_allocate([used_frequency()], pid(), pos_integer()) -> boolean().
can_allocate(Allocated, Pid, MaxFrequencies) ->
{PidFrequencies, _Other} = lists:partition(fun(Elem) -> pid_match(Elem, Pid) end, Allocated),
length(PidFrequencies) MaxFrequencies.
[/erlang]

by halzy at June 28, 2010 03:19 PM

June 25, 2010

Erlang Inside

Spanish Language Erlang List

If you’re looking for Spanish language group to discuss Erlang, Mariano Guerra recently started “ErlAr“, a mailing list for all things Erlang, but in Castellano. Mariano is the author of Efene, a Javascript-syntax language that compiles to Erlang bytecode. I’m planning an interview with Mariano on Efene soon. There is a small but growing group [...]

by Chad DePue at June 25, 2010 01:52 AM

June 24, 2010

1011 Ltd Blog

Elegant XML SAX parsing using Erlang's pattern matching

Yesterday I wrote some code for parsing an XML file with Erlang's built-in SAX parser. It turned out that the events generated by the SAX parser, combined with the pattern matching in Erlang yielded a really elegant solution.

June 24, 2010 05:00 PM

Learn You Some Erlang

More On Multiprocessing

This chapter's a bit of an extension on the practical aspects of the Hitchhiker's Guide to Concurrency. New multiprocessing and concurrent aspects of Erlang are seen, including how to hold state in processes, useful abstractions when passing messages, how to handle timeouts and then selective receives for prioritizing messages. I'm publishing this chapter in English on the night of our French-Canadian national holiday, so enjoy!

June 24, 2010 12:00 AM

June 23, 2010

Process-one Blogs

Case study: Arc International

Arc International has migrated away from a proprietary solution to the open source, and open standards ejabberd and Psi, realising cost-savings and productivity increase.

Arc International is well known for distributing famous tableware brands such as Luminarc®, Arcoroc, Chef&Sommelier, Pyrex and Cristal d’Arques Paris. With 12,000 employees around the world, from Europe to Asia, and the Middle East to North America, they needed to lower the costs of their previous solutions.

They have chosen ejabberd, as an open-source, scalable XMPP platform. In conjunction, with the Psi cllent Arc International has achive their goals of cost-savings and high-quality service.

The cost-savings have benefitted from a lower TCP on IT, as well as reduced telephone bills, while providing a same feature-set to the employees worldwide.

Read our case study on the website, or as a PDF (582 KB).

by Nicolas Vérité at June 23, 2010 10:22 AM

June 22, 2010

RedHotErlang

Dizzy thoughts

On the bus home today I was reading the first chapter of [Programming Clojure][1]. The text, which is excellent, discuss the motives behind Clojure. The bus however, was stinking hot and I started to doze off. I was in a strange state of being half a...

by tobbe at June 22, 2010 08:52 PM

June 20, 2010

RedHotErlang

Permutations

Permutations ========== Permuting things is fun. Rearranging all the furniture's in a room is one kind of permutation. Generating all combinations of the string "ABC" will give you five more strings. "ACB" "BAC" "BCA" "CAB" "CBA" In general you w...

by tony at June 20, 2010 03:29 PM

June 19, 2010

RedHotErlang

Erlang Factory, London 2010

I went to London recently to attend the Erlang Factory conference. We were a bunch of people coming from [Klarna][1], as you can see below: [1]: http://www.klarna.com/ <img width="50%" src="/blob/ab62b2625d4ec6b6f16b631508b87c48"/> And if you do s...

by tobbe at June 19, 2010 05:22 PM

June 18, 2010

1011 Ltd Blog

HOWTO Choose a NoSQL Database

The past year has seen an EXPLOSION in NoSQL databases. It feels like every month there a new contender to consider. The choice is fantastic, but can also be problematic. If there are so many options, how do you choose which one fits your problem the best? The most information I've seen to date focusses on the features and performance of each. Those are important, but...

June 18, 2010 01:30 PM

Erlang Factory News

Erlang User Conference Date Announced

The 2010 Erlang User Conference in Stockholm will be held on Tuesday 16th November 2010. The venue will again be the Old Astoria Cinema in Nybrogatan.

On Monday 15th November there will be a ProTest (Property-based Testing) seminar series held at the Ericsson conference faciilities. This will allow delegates to again optimise their visit to Stockholm to take in both these events.

More details will be forthcoming in the near future. Keep a look out!

June 18, 2010 10:45 AM

June 16, 2010

Trapexit News Feed

Trapexit News :: tryerlang.org is Live!

Author: admin
Subject: tryerlang.org is Live!
Posted: Wed Jun 16, 2010 1:29 pm (GMT 0)
Topic Replies: 0

tryerlang.org is an interactive hands-on tutorial that allows you to try the power of Erlang directly in the browser. Try the power of the Erlang shell without installing anything to your machine! Visit: http://www.tryerlang.org

June 16, 2010 01:31 PM

June 14, 2010

1011 Ltd Blog

CV Bling

I love mainting my CV. As much as I love doing the dishes. This time around though, I'm hoping to make things a little bit more exciting and a little bit less painful for the future.

June 14, 2010 09:55 PM

June 13, 2010

Programming in the 21st Century

A Ramble Through Erlang IO Lists

The IO List is a handy data type in Erlang, but not one that's not often discussed in tutorials. It's any binary. Or any list containing integers between 0 and 255. Or any arbitrarily nested list containing either of those two things. Like this:

[10, 20, "hello", <<"hello",65>>, [<<1,2,3>>, 0, 255]]

quote(String) -> $" ++ String ++ $".

quote(String) -> [$", String, $"].

[Directory, $/, Filename, ".png"]

by James Hague at June 13, 2010 06:00 AM

June 07, 2010

Erlang at Work

epgsql Moved to GitHub

epgsql development has picked up steam in the past few days with changes to support some of the upcoming features of PostgreSQL 9.0, bug fixes, and support for delivering asynchronous notices and notifications from the LISTEN/NOTIFY mechanism which will be much improved in PostgreSQL 9.0.

I've also moved development to GitHub since git has become my primary version control system. You'll find the new master repository here:

http://github.com/wg/epgsql

The old mercurial repositories will no longer be updated, but I'll leave them up for the time being. If you're a mercurial user please accept my apologies. I'm a big fan of hg but maintaining two repositories would be too much work, and git has a few killer features that convinced me to switch.

by Will Glozer (noreply@blogger.com) at June 07, 2010 10:43 PM

June 04, 2010

Joe's Blog

Just Opensourced: Gaff and Deckard

This post was stolen from my original post on the Cloudant blog.

Today we released two open source projects that have been in use internally at Cloudant for some time now, Gaff and Deckard.

All of our infrastructure is in the cloud and as such we need a way for disperate systems to all request resources, this is where Gaff comes in. Gaff is a pubsub daemon for asynchronously talking to cloud APIs using AMQP. Currently it supports a subset of the Dynect (DNS), Slicehost and EC2 APIs and uses geemus‘ awesome fog Ruby library. The basic workflow for Gaff is to send JSON-RPC formated messages to an AMQP exchange with a routing key corresponding to the API you are talking to, you could be sending these messages from a web application or another service.  Each message gets routed to an API specific queue and is picked up by Gaff and turned into the appropriate API call, starting, stopping, modifying your servers on EC2 or elsewhere.

We have a lot of CouchDB instances to keep tabs on to do this we wrote Deckard. Deckard is a HTTP check monitoring system based on CouchDB. Yo dawg! What better than to monitor CouchDB with CouchDB (and some Ruby)? Deckard supports basic HTTP content checks, email alerts, SMS alerts (via email) for on-call rotations, basic maintenance scheduling, replication latency alerts (between two Couches) and even has EC2 Elastic IP support for failover between two EC2 instances. Best of all since it’s based on Couch you get an API for free, just PUT a doc in the HTTP checks database and you get a new HTTP check the next time Deckard runs.

Checkout these and my other projects on GitHub and follow Cloudant and myself on Twitter.

by joe at June 04, 2010 09:18 PM

Process-one Blogs

ejabberd 2.1.4 and exmpp 0.9.4 bugfix releases

We are pleased to announce the bugfix releases ejabberd 2.1.4 and exmpp 0.9.4.

Regarding ejabberd 2.1.4:

Brief summary of changes:

• Bounce messages when closing c2s session
• Bugfixes when handling Service Discovery to contacts
• Don't send error stanza as reply to error stanza
• Don't store blocked messages in offline queue
• Extauth: Optionally cache extauth users in mnesia
• Full support for XEP-0115 Entity Capabilities v1.5
• HTTP-Bind: Remove 100ms delay, export functions for prebinding methods
• LDAP: Inband password change, Extensible match support, and ldap_tls_verify
• Localization: Updated most translations
• MUC: Refactor code to reduce calls to get_affiliation and get_role
• ODBC: Add created_at column also to PostgreSQL schema
• PAM: New option pam_userinfotype to provide username or JID
• Pubsub: Send node notification. Enforce presence_based_delivery, show_values
• Vcard: Automatic vcard avatar addition in presence
• WebAdmin: New Access rule webadmin_view for read-only

Check the Release Notes for a more complete list of changes:

http://www.process-one.net/en/ejabberd/release_notes/release_note_ejabberd_2.1.4

If you upgrade from ejabberd 2.0.5 or older, read carefully the release notes of ejabberd 2.1.0 too, because there were several changes in the installation path and the configuration options.

The list of solved tickets since previous version is available on ProcessOne bug tracker:
http://redir.process-one.net/ejabberd-2.1.4

Download

ejabberd 2.1.4 is available as source code package and binary installers for Linux 32 bits, 64 bits, Mac OS X Intel, and Windows:
http://www.process-one.net/en/ejabberd/downloads

Regarding exmpp 0.9.4:

Brief summary of changes:

• Add unexpected-request with cancel needed by pubsub
• Caps: Add exmpp_caps to the compilation scripts
• Caps: Move Caps erlang records to a header file
• Handle the case that the payload is incorrectly defined in error stanzas
• Preliminary documentation of functions from ejabberd 2 available in exmpp
• Remove unexpected-request with cancel as this is pubsub specific
• Support for SRV DNS query

exmpp home page:

http://support.process-one.net/doc/display/EXMPP/
or easier to remember: http://exmpp.org/

Download

Download exmpp 0.9.4 source code package from:

http://download.process-one.net/exmpp/

You can also check the ProcessOne Labs page:
http://www.process-one.net/en/labs/

by Jérôme Sautret at June 04, 2010 11:16 AM

June 01, 2010

Process-one Blogs

ProcessOne at Google I/O: slides and photos

ProcessOne has joined the developer Sandbox in Google I/O 2010, in the wave area.

by Nicolas Vérité at June 01, 2010 05:38 PM

Erlang Factory News

Only 1 week left!

There is only 1 week left to the Erlang Factory London. The Erlang University courses will start on 7th June and will be followed by the two-day Conference which starts on Thursday 10th.

There are still a few remaining places left on the Erlang University training courses: Erlang Express, Erlang OTP Express and Web Programming with Erlang.

If you haven't registered yet, you can still get a last minute seat - Book Now!

June 01, 2010 09:46 AM

May 31, 2010

Joe's Blog

Beyond BigData.

BigData is a big deal. It’s changing how we look at data and analytics, but it isn’t the end. What are the enablers of BigData? First and foremost, cheap computing resources (CPU, disks, memory, bandwidth, etc) all thanks to Moore’s Law. Today even startups have the ability to afford huge amounts of computing power, the likes previously only the big boys could afford. Additionally, this has given rise to commodity hardware and cloud computing, which only furthers the proliferation of large amounts cheap, quickly-provisioned, computing resources. Second, to apply all that power, we have open source data processing systems based on years of distributed systems research, like Hadoop, and many incarnations of NoSQL. The development of open source data processing sytems has allowed proliferation of systems that scale, which only the highly capitalized could afford, until recently. These two things alone have allowed for the democratization of BigData. A guy in a garage can process terabytes of data with little more than a credit card and elbow grease.

With all these tools and recently acquired computing power, where are we going? Of course we can expect datasets to continue to grow, and the computational complexity of our data processing to increase, as well as compute power to continue to rise (GPGPUs, multicore and so on). In addition, I anticipate the emergence of something I’m calling NewData. NewData will build on what we have currently with the BigData, but will include some trends just beginning to take off. First, the development of ubiquitous public APIs (Meatcloud Manifesto). Public APIs have yet to proliferate to all online systems. As a consequence, there is still a lot of screen scraping going on. By having easily query-able and parse-able datasets available through ubiquitous APIs, consuming the internet with machines is easier making the application of BigData more powerful. Netflix is a good example of this. Second and similarly enabling will be the development of standardized public datasets. Current datasets are generally hard to find and use, standardized dataset formats will enable BigData analysis to be more productive and not waste time munging. Data.gov is a start. These two developments are yet to be fully realized in current systems but will allow for the rise of NewData. As these developments begin to roll out we will begin to see changes to how our BigData systems look. NewData systems will be less concerned with how big the data is and what it looks like, but will emphasize derivation of more information from the data. Bradford Cross gets this, and as a result FlightCaster is an early example of what I mean by NewData.

The scale of data and computations is an important issue, but the data age is less about the raw size of your data, and more about the cool stuff you can do with it.

Asking the right questions of the data is important, especially if you’re trying to do cool stuff. The Freakonomics guys proved this a few times over. NewData will be about creating value from data, and asking the right questions is worth as much as the answers. The key enablers of this will be using new found APIs and datasets to combine data from disperate sources in ways that BigData couldn’t. Asking questions that we wouldn’t have thought to ask of BigData. Where BigData was about a handful of datasets at most, NewData will be about dozens of datasets. The mashup is the cornerstone of NewData.

That being said, we will need new systems to process this data and enable us to ask these questions. NewData analysis will need inter-process communication and collaboration. Currently, systems like Hadoop process data by splitting the data up and processing chunks in parallel on hundreds to thousands of machines. Processes are isolated from the other processes. This will continue, but NewData will require more from these systems to ask deeper questions. Complex inter-process communication will be needed to ask these questions. Think of the simplicity of writing Map/Reduce jobs, the robustness of Hadoop, the workflow and dataflow of Cascading and DryadLINQ, respectively, and the power of a message passing system like MPI. These jobs will likely include large in-memory collaborative computations across thousands of machines. Where data locality was key in BigData, both data and memory-locality (NUMA/ccNUMA) will be important in NewData.

It is clear that BigData still has some runway before NewData takes over. However, if the trends in the democratization of compute and processing continue (beyond Hadoop and EC2), and the opening of APIs and datasets proliferate online and off, NewData and it’s new questions, mashups, and systems are inevitable. Where having readily available compute resources and the software to use it defined BigData, NewData will be defined solely by asking the right questions, the algorithms to derive answers, and the systems used to produce them.

Thanks to Mike Miller, Bradford Stephens and my awesome wife Erin for the help on this article.

Follow me on twitter.

by joe at May 31, 2010 04:54 PM

May 29, 2010

RedHotErlang

Rebar template support for gettext

I have added [gettext][1] support to my [rebar templates][2]. This makes it very easy to internationalize a Nitrogen webapp. To show how this works I'll create a simple example. First, we create the project using rebar: [1]: http://github.com/etnt/g...

by tobbe at May 29, 2010 02:11 PM

May 24, 2010

Erlang Factory News

Early-Bird Registration Ends in Just ONE WEEK!

Early-Bird Registration for the Erlang Factory London 2010 ends on 31st May, so you have only one week to save up to £200 (incl. VAT)!

There are only 2 weeks left to the biggest Erlang gathering in London. List of the speakers, the tracks and talks as well as the programme of the conference are now available on www.erlang-factory.com/conference/London2010


Don't miss out and register today.

May 24, 2010 11:30 AM

May 21, 2010

Process-one Blogs

WaveOne, a wave server and client by ProcessOne

WaveOne, the new wave server developped by ProcessOne, has been demoed at Google I/O.

ProcessOne has been invited to join the Wave area in Google Developer Sandbox at Google I/O 2010 in San Francisco.

Mickaël Rémond, the founder of ProcessOne, has been invited by Google in the context of the Google I/O event, to write a blog post to introduce WaveOne, our own wave server implementation.

WaveOne is basically an ejabberd module, so it is written in Erlang. It features an Operationnal Transform engine, as well as a Wave Store. It means that with the help of a client, you can - on your own WaveOne server installation - create, edit, store and share wavelets.

The WaveOne extension to our XMPP client OneTeam enables you to manage your wavelets (and blips), like create, edit, as well as add participants to wavelets, still on your own WaveOne server installation.

With the WaveOne client and server, you can collaborate in real-time internally, as well as externally, thanks to the federation capability. WaveOne supports federation with Google Wave in both directions. You can host the wavelets on your local server and have remote participants or you can do the reverse and participate to wavelets hosted on the Google server. This enables you to add participants, or be added as participant, to wavelets hosted on any wave server. Everything works in real time with character by character support.

Here is a screenshot that shows a wavelet (containing two blips) edited both on Google Wave and ProcessOne's WaveOne:

The result is much like having your own e-mail or XMPP client and server under your own control. This is built on the power of XMPP-based federation.

A handy feature of our WaveOne implementation is the capability to share a textarea of a webpage, for collaborative real-time text editing. This has been demoed at Google I/O, and has received much interest. Here are some sample screenshots.

The user right-clicks in a textarea, and shares it:

He adds participants to the wavelet:

All participants can collaboratively edit the text in real-time:

by Nicolas Vérité at May 21, 2010 02:00 PM

Learn You Some Erlang

The Hitchhiker's Guide to Concurrency

I was a bit slow to produce this chapter, but I'll blame that on my laptop dying on me and playoffs Hockey (watching a game as I'm writing this!) Anyway, this is the first chapter to really talk about Erlang's concurrency. Because of this, I'm taking a theoretical approach and visiting the language's history a little bit, explaining the rationale and doing a quick overview of how processes work in Erlang. I also take a look at the concepts of concurrency and parallelism to make sure everyone will be on the same level no matter what their background is. I also show the basic building blocks of concurrent Erlang: spawning processes, sending messages and then receiving them.

May 21, 2010 01:30 PM

May 19, 2010

Process-one Blogs

TextOne for Android, simple texting for even more smartphones

TextOne is now available for Android devices, enabling smartphone lovers to text friends and contacts the smart and easy way.

ProcessOne has released an early version of TextOne, the smart text messaging application for smartphones, for the Android platform. As on the iPhone, it features one-to-one chats, and simple groupchats. TextOne also provides reliability mechanisms like visual delivery notification. Of course TextOne for Android and TextOne for iPhone are fully interoperable.

TextOne is better than:

• SMS:
  • because you can exchange messages freely at no cost
  • because you can groupchat
• IM/chat:
  • because you have reliability mechanisms
  • because you don't need a useless presence icon, just send ans receive simply

As an early version, you may experience some bugs and weirdnesses. You can report them on the dedicated TextOne forum.

TextOne for Android works starting from Android 1.5.

Here are some screenshots, so you can have an idea on its look and feel:

The conversation list:

And the conversation screen:

Please feel free to download and play with TextOne for Android in version 0.8.4. It is currently available on the Android Market.

TextOne Android is free during the beta period. A TextOne extended version for Android 2.0 to use the native content management feature is scheduled.

by Nicolas Vérité at May 19, 2010 03:17 PM

May 18, 2010

RedHotErlang

New Layout

RedHotErlang has got a new layout. The old layout had been around for some years now and I wanted to move the code base to use Nitrogen-2.0. So, gone is the desktop style structure with no visible page loads. Now the site is using a more normal page ...

by tobbe at May 18, 2010 06:59 PM

May 13, 2010

BGH's Blog

Silence

The blog has been a bit quiet. I've been rewriting some of our server software at work in Erlang and it's taken up most of my free time. I'm about done with it, so we should see the posts pick up again soon.

Cheers,
Halzy

by halzy at May 13, 2010 02:49 PM

Programming in the 21st Century

How to Think Like a Pioneer

Here's an experiment to try at home: do a Google image search for "integrated development environment." Take some time to go through the first several pages of pictures.

Even if you had no idea what an IDE was, the patterns are obvious. There's a big area with text in it, and on the left side of the screen is a pane that looks a lot like Windows Explorer: a collection of folders and files in a tree view, each with icons. Some folders are closed and have little plus signs next to them. Others are expanded and you can see the files contained within them. To be perfectly fair, this project area is sometimes on the right side of the window instead of the left. Variety is the spice of life and all that.

Why have IDEs settled into this pattern of having a project view take up twenty percent or more of the entire left or right side of the window?

The answer is shallower than you may expect. Someone who decides to create an IDE uses the existing programs he or she is familiar with as models. In other words, "because that's how it's supposed to be."

There's no solid reason the project view pane has to be the way it usually is. In fact, there are some good arguments against doing it that way. First, you are typically either looking at the project view or you're editing a file, not doing both at the same time. Yet the project view is always there, taking up screen real estate, sometimes a lot of screen real estate if highly nested folders are expanded. That view is made even wider by having icons to the left of each filename, even though most projects consist of one or two file types and they could be differentiated with color or an outline instead of a contrived icon attempting to evoke "file of Java code."

Second, a long, thin pane containing files and folders doesn't give a particularly deep or even interesting view of a large project. Open a folder and it might fill the whole pane, and all the other folders are now offscreen.

Are there better options? Sure! The first problem, of losing screen space to a persistent view of the project hierarchy, can be alleviated by a hot key that brings up a full-screen overlay. When you want to see the project as a whole, hit a key. Press escape to make it go away (or double-click a file to edit it).

The data presented in this overlay doesn't need to be a tree view. A simple option is to group related files into colored boxes, much like folders, except you can see the contents the whole time. With the narrow, vertical format out of the way, there can be multiple columns of boxes filling the space. Now you can see 100+ files at a time instead of a dozen.

It might make more sense to display modules as shapes, each connected by lines to the modules which are imported. Or provide multiple visualizations of the project, each for a different purpose.

Someone, sometime, and probably not all that long ago, came up with the canonical "project view on the left side of the window" design for IDEs. And you may wonder how that person arrived at that solution. After all, there were no prior IDEs to use for guidance. I think the answer is a basic one: because it's a solution that worked and was better than what came before. No magic. No in-depth comparison of a dozen possibilities. Clearly a way to see all the files in a project is better than a raw window in a text editor where there's no concept of "project" whatsoever. That solution wasn't the best solution that could ever exist in the entire future history of IDEs, but it sure fooled people into thinking it was.

If you want to think like a pioneer, focus on the problem you're trying to solve. The actual problem. Don't jump directly to what everyone else is doing and then rephrase your problem in terms of that solution. In the IDE case, the problem is "How can I present a visual overview of a project," not "How can I write a tree viewer like in all other other IDEs I've ever seen?"

by James Hague at May 13, 2010 06:00 AM

May 10, 2010

1011 Ltd Blog

Google Wave @ BCS SPA - slides

Here are the slides from the presentation I gave at BCS SPA on the 5th of May. They're really not that useful without actually hearing the talk, because I don't believe in creating slides that people just read. But you might find them useful if you were there.

May 10, 2010 06:01 PM