Notes taken from the 2009 Commercial Users of Functional Programming, Edinburgh Scotland.
Growth figures for languages
A call to arms to the functional programming community.
Bryan talks about how Real World Haskell (the book) came to be, and the response that it has received since publication. He will also discuss the opportunities presented, and the challenges faced, by functional languages in open source and in industry.
Questions:
Implementing a Domain-Specific Language for Derivative Pricing with Scala
In this talk I shall explain how Scala has been used at EDF Trading. We have used Scala to implement Diesel: a domain-specific language used within our in-house system for pricing/risk-managing/hedging commodity derivatives. Diesel can represent commodity derivatives naturally and can then be executed to perform a Monte Carlo simulation. Compared to writing derivative-specific Monte Carlo simulations in a general-purpose computer language, the per-derivative development time has been reduced from weeks to hours.
We have been using Scala within a large code base of 300k lines of Java on a on a business-critical, production system, where performance requirements are key. Based on this experience, the second part of the talk will discuss the suitability of Scala for use within this context, comparing the risks of using a niche language against its productivity benefits.
DIESEL: EDFT's domain-specific language for pricing custom derivatives Very concise notation that implements textual descriptions of how to do the pricing Quite concise description of the AST. clear language design Elegant semantics
Illustrates interesting use of mixed OO/Functional style, supports macros,
Question:
I'm Co-Founder & CTO at 'hover.in', the in-text content & ad delivery platform that let's blogs and website publishers to push client-side events to the cloud. The start-up predominantly runs off the LYME stack ( Linux / Yaws / Mnesia / Erlang ) .
From our experiences at 'hover.in' I'd like to discuss why we chose using Erlang and got about using it as our bridge across our multi-node cluster. In particular the architectural decisions that went into making our distributed python crawler back-end running off Mnesia with its sharing & fragmentation strategies for tables that span several millions of rows, load-balancing to our 3-node Yaws web servers, tweaks to solve file descriptor & efficiency bottlenecks, experiments in DHT's, our cache worker implementations, our messaging queues, cron's & trade offs in dispatching jobs also throw light on design choices that can fit in distributed and heterogeneous environments. We have also recently built our own in-memory cache workers, persistent stats & logging system, and in the process of now building our own A/B testing framework, that we'd love to talk about.
After an initial quite 1 1/2 years of Erlang in production, we've just launched our developer blog which might give a closer insight of our work:
a more product-oriented interview
business: ad for what happens when you hover over a word on your blog "in-text broadcasting"
Questions:
Teleconferencing over High-res Maps with Haskell
A public emergency is the ultimate example of multitasking and collaboration over a wide area. Crowd control, first aid, fire, police, and search and rescue teams all have to provide a timely and coordinated response to save lives and property. This requires situational awareness, meaning that emergency managers know what and where their resources are on the ground, and what the ground's features are. Managers who direct emergency operations centers have few technological tools to help them with these complicated, critical tasks. They are, in fact, still largely paper based.
The Big Board aids in filling this technological gap by providing aerial photography, facility for dynamic, geo-referenced content creation, and a mechanism for simulcasting the content to everyone participating in the management of an emergency. The goal is that it should give emergency managers a reliable and flexible way to coordinate on their most natural medium (maps), be as easy to use as pen and paper, quick to train a new user on, and able to run in many environments, some quite technologically out of date.
In The Big Board, users open a map, join a virtual conference room, and then can collaboratively mark up the map with paths, polygons, and placemarks and add content behind these using a wiki that is generated as part of every conference room. As they mark up the map, their annotations are simulcasted to everyone else in the same conference room. Additionally, web-services serving geo-referenced content can be integrated onto the map to overlay things such as real-time vehicle location and sensor data.
This application has been written from the ground up in Haskell, with a PostGIS backend. Haskell was chosen for its laziness, reliability of applications written in it, rapid development, multicore scalability, and native compilation. In this presentation I will describe:
I will also give a demonstration of the application running, showing how it can be used to coordinate a disaster response.
Questions:
The First Substantial Line of Business Application in F#
We have developed MarketingPlatform(tm) a marketing automation solution delivered as Software as a Service with F# as the primary language. MarketingPlatform(tm) is a solution for marketers in direct marketing and in channel marketing who would like to gain a timely and deep understanding of what is working and what is not working in their marketing campaigns. Marketers are than facilitated in the execution and delivery of campaigns, using this insight to create relevant communications to each individual. It is divided into four tightly integrated campaign management steps of Measure, Analyze, Design and Execute.
Measure: How well are my campaigns working? Are they meeting the goals?
Analyze: Why are the campaigns exceeding goals, or falling short of goals? What sub-segments are worthy of further communications with?
Design: Create communications that are relevant to each individual, driven by the data and insight gained from measurement and analysis.
Execute: Deliver the communications through the most appropriate channels, including email, print and mail, texting and purls.
F# is a functional language with first class functions and composability, a pattern matching language and strongly typed. At the same time it as a .NET CLR language, fully interoperable with the vast libraries, and fully capable of OOP. So we believed that we would get the best of both worlds — a modern productive language and a full set of libraries to build on. The product is in the market and selling well, so we were rewarded by our decisions. Where was F# applied within the application, and where not?
Most of the business logic was implemented in F#, as was most of the data handling. C# was retained for building the ASP.NET user interface, just because the F# tool support is not there yet.
The application was developed in less time with significantly less code — probably a quarter. The code is more readable and easier to adapt as new market requirements come in.
We were able to get to market much sooner with more feature, and the overall cost of the development was significantly lower.
Conclusion: F# is an excellent language for line of business applications. It somewhat shortens the development cycle and greatly shortens the enhancement cycle.
Use laziness for SQL access.
Questions:
Functional Programming at Facebook
We use Erlang, ML, and Haskell for a few small projects but (as I'm sure you guessed) we'd like to focus on the lessons we've learned using Erlang for building Facebook Chat. The channel servers are at the heart of our Chat infrastructure; they receive messages from our web servers and queue them for HTTP delivery. Presence information, the knowledge of who is currently using Chat, is aggregated and shipped to a separate service. The service has been operational since our launch about a year ago and we continue to develop and improve it frequently. We're also developing an XMPP interface to Facebook Chat based on the ejabberd project. A lot of functionality has been added to interact with the many moving parts of our infrastructure.
Erlang has been the right tool for the job. It's well-known that Erlang is a good choice for communications applications and real-time chat in particular, and we'd like to mention that Erlang's strong support for concurrency, distribution, hot code loading, and online debugging have been indispensable, and that it's those same weaknesses that lead us away from C++ when first designing the service. However, we'd like to focus on the factors inside and outside Facebook that enabled us to choose Erlang in the first place. Most importantly, we needed good support for language-independent RPC. Fortunately we rely heavily on Thrift for all our services. Both the channel and Jabber servers need to speak to programs written in PHP and C++, and having infrastructure and engineers comfortable debugging it already in place when we began was invaluable. Our service management tools as well are language-independent; all our services regardless of language are controlled using the same interface. Facebook and the Thrift project are also fortunate to have an active community. The original Thrift binding for Erlang was implemented in-house but substantial portions were reimplemented by outside contributors. As well we've leveraged existing open source products including ejabberd and Bob Ippolito's Mochiweb framework.
The other half of our story consists of the barriers presented and risks taken in choosing Erlang. I mentioned above that much of Facebook's infrastructure is language-agnostic, but our build and deploy systems unfortunately are not. Many of our internal tools weren't written with Erlang or (in particular) hot code reloading in mind, so we needed to write many one-offs to support all the runtime goodness that Erlang affords us. Facebook runs many services, most of which are in C++, Python, or Java, and only two in Erlang, so in some respects we've needed to work harder to integrate. We at Facebook also tend to work in large codebases and share the responsibility of bug fixing. Unfortunately, our Erlang projects live outside of our main repository, and the extra learning necessary to develop or just fix bugs in Erlang tends to keep us isolated from the pool of talent in our department. Unsurprisingly, the job of fixing Chat bugs falls to one of a very small group, whereas bugs in our PHP code have a small army of PHP generalists waiting to squash them. However Erlang has such a steep learning curve only because most undergraduate programs don't stress functional programming, and Facebook in particular doesn't expose most engineers to functional programming, or even information about the strengths and weaknesses of FP versus PHP, C++, and Python. In particular we'd like to touch on the prevalence of object-oriented programming in education and in practice, and the tendency of engineers (even engineers with FP experience) to guide their design by OOP principles even when they don't fit. The recurring theme is "the right tool for the job" — actor-based, functional, imperative, and object-oriented programming are all valuable tools for modeling particular problems, but the most important lessons in our experience are how to lower the barriers to choosing the right tool, and how to arm engineers with the knowledge they need to make an appropriate choice.
Questions
FMD - Functional development in Excel
Barclays Capital, like many other investment banks, uses Microsoft Excel as a rapid application development environment for pricing models and other tools. While this may sound bizarre, Excel's flexibility and convenience renders it an immensely useful tool for the Front Office, and our Traders are extremely Excel literate.
Excel combines two programming models
The functional model allows very rapid development and great transparency, but the limitations of Excel's built-in functions drives developers to use VBA. Soon the spreadsheet dependency graph is lost and the spreadsheet layer is relegated to a GUI on top of tens/hundreds of thousands of lines of VBA. The logic is tightly tied to Excel, and a server-side implementation involves a complete rewrite in another language, which carries both operational risk and developmental cost.
FMD (‘Functional Model Deployment’) prevents these problems by embedding a functional language cleanly into Excel, effectively extending Excel to be a higher order functional environment. Now complex models can be developed without leaving the pure spreadsheet domain:
Before
After
The business have fully supported this approach, and are enthusiastic about using FMD - as Simon Peyton Jones identified elsewhere, “Excel is the world's most popular functional language”. From their point of view, functional programming in Excel is just an extension of what they've been doing for years!
Clear & Simple: Composing a Marketplace
Glyde is a web-based marketplace, with the goal of making buying and selling as easy as possible. We also strive to provide a responsive, error-free experience. These goals represent challenges which functional programming helps mitigate.
First, easy-to-use often means complex-to-build, but FP's high-level abstractions and referential transparency help us to manage this complexity. Second, as we handle people's money, mistakes could be damaging, but the expressiveness of FP give us confidence in our code, and a lack of side effects facilitates testing. Third, a responsive site is a pleasure to use, so we write parallelizable logic, which FP's avoidance of side effects makes feasible.
This talk will describe Glyde's use of JoCaml and Scala and discuss our experiences with these tools.
Birth of the Industrial Haskell Group
It has long been thought that commercial users of Haskell could benefit from an organisation to support their needs, and that as a side-effect the wider Haskell community could benefit from the actions of such an organisation. The stronger community would in turn benefit the commercial users, in a positive feedback cycle.
At last year's CUFP, users of several FP languages raised the issue that there was no organisation that they could pay to do the important but boring work of maintaining and improving common infrastructure. Shortly after CUFP, in partnership with major commercial users of Haskell such as Galois and Amgen, we started to set wheels in motion, and in March 2009 we announced the birth of the Industrial Haskell Group (IHG).
The IHG is starting off with a limited set of activities, but already it is having an impact on the state of the Haskell development platform. We expect that as it expands, it will become a significant force driving Haskell forwards.
In this presentation we will talk about the motivation leading to the formation of the IHG, how it has worked thus far and what lessons we can learn that might benefit other FP communities. We will also look at how we can encourage the positive feedback cycle between commercial users and the wider community.