Aeron – Proof of the benefits of open development
- Open Source Software
By Martin Thompson, High-Performance & Distributed Systems Specialist.
A precept of open-source work is that “given enough eyeballs, all bugs are shallow,” first said by Eric S. Raymond in The Cathedral and the Bazaar, and dubbed “Linus’ Law” in 1999.
Based on my experience of the benefits of developing Aeron – the high throughput, low latency messaging system – in the open with multiple collaborators, and across a variety of organisations, Linus’ Law is one I agree with, but I’m proposing an open source corollary to it:
Given sufficiently diverse collaboration, superior solutions can eventually emerge
Open source projects allow collaborators to work across the walled-gardens of their organisations and welcome creative solutions.
Open source means someone banging away on the software as a hobby can work with someone labouring inside a large corporation, even with a developer who is only able to devote a little bit at a time – it makes idea-sharing and creative teamwork both possible and productive.
The development of Aeron is representative of that approach; it happened in the open and under a liberal open-source software (OSS) licence (Apache 2.0).
It took time to develop and get to a point of operational stability where financial institutions would be able to trust it, let alone eventually come to favour it. For Aeron, the emphasis is on eventually.
Todd Montgomery and I collaborated and enjoyed sponsorship to develop Aeron while carrying on other interests over a number of years.
In time, Aeron took on a life of its own and allowed us, and later others, to contribute to the open-source project. This shared effort resulted in the product you see today – used to some extent in nearly all financial organisations globally where the performance of communications and messaging is a key requirement.
Another key word in the formulation of our corollary is “can”, as in “superior solutions can eventually emerge”, but it’s not a given.
The sponsorship of large institutions and or corporations can provide a stable hull that’s necessary for weathering the inevitable storms associated with all software development projects. Having someone there to backstop your efforts and ensure that your developers receive some kind of compensation as they code can really help to keep a project progressing.
In this brief description of the Aeron experience and some of the benefits of development in the open, I’ll discuss:
- the influence on quality
- collaboration – do’s and don’ts
- sponsored development
- learning opportunities
- corporate stewardship
I argue that the development of Aeron, and its high quality, is inextricably linked to its status as an open-source project.
Context and background
To get the best experts on a topic to collaborate with any significant effort usually requires the formation of a company and those experts to become employees. This is a hurdle to innovation as this would require the experts, who are often in different companies, to quit and then join the new company, as their existing employers typically have IP rights over whatever they produce.
Open source can be a way to sidestep this hurdle as experts can collaborate in the open for common interest. This is in fact how Aeron began.
Influence on quality
The first really interesting benefit of working on OSS is that it is often treated more like a craft than the more common utilitarian approach to commercial software.
Open source software development is often driven by passion and desire toward something not just functional but also pleasing, like well-made art. This public nature instils pride and the desire to leave a legacy.
On top of the quality benefits derived from the pride of the creators, there are benefits from peer and public review of the code. Though for some developers showing their code can be intimidating, for others working collaboratively makes them stretch and strive to be better.
While with proprietary software, customers can provide feedback on its usage, they usually do not have access to the source code. In contrast, having easy access to open-source code results in significantly more review.
With the use of open-source tools like GitHub and Pull Requests, quality communication improves. Users can provide a failing test to illustrate a bug, which can then be worked on collaboratively to achieve a rapid fix. This means bug reporting is much more effective than any narrative description of bugs, or the events that trigger an error.
Converting a closed-source project to open-source is one strategy for getting the benefits of collaboration, but it is even better to develop in the open from the start.
As code for features is developed, a public audience can provide early feedback to help steer the solution. This community feedback can reduce the cost or even avoid the need to change released code, given the breaking implications of later changes.
To be fair, though developing in the open has huge benefits, it is important to realise this does not come without consequences. Not all feedback is useful and sometimes the noise levels can drown out the signal. Managing the community feedback requires experience and a careful touch.
Collaboration – do’s and don’ts
One negative strategy that occurs is the situation in which companies make use of open-source software and find it would be better for them if the OSS had feature X.
They then complain within the community that the software product does not have feature X and try to guilt, or in effect bully, the core contributors into adding this feature. This is not only a terrible way to collaborate, it is a very unpleasant way to behave as an individual or company.
Better ways to collaborate on the OSS effort are to either sponsor the core contributors to add feature X, or work with them to add specific features via contributed efforts and/or sponsorship.
Aeron has had its fair share of demanding customers who did not contribute in any way, yet they benefited from the product. Likewise, Aeron has also had a number of good customers who sponsored development and/or contributed effort.
⇒ Learn more about the G-Research Open-Source Software team
Example of preferred collaboration
One such example of collaboration occurred after the Spectre and Meltdown mitigations got applied to operating systems. These mitigations resulted in a significant cost increase for system calls and soft page faults.
In detail: when Aeron establishes connection for a stream between producers and consumers, it needs to allocate memory for buffering that stream and provide a history to recover any potential loss.
This memory is contained in a memory-mapped file which by default is sparse and faulted-in on demand. The faulting-in on demand increased latency due to the soft page fault cost increasing after the mitigations were applied.
Aeron has the option to pre-touch the memory on connection establishment to avoid later latency pauses. The pre-touching of memory results in many soft page faults using the cross-platform algorithm Aeron employs.
There are OS-specific system calls which can be used to initialise the memory in a user process, thus avoiding the later soft page faults. Each operating system takes a different approach which must be considered.
At the time, a group at G-Research contacted the Aeron development team and said that it would like to help with extending the C-based Aeron media driver to allow this OS-specific memory initialisation support on Linux.
The Aeron team then collaborated with G-Research to provide this on Linux and Windows so Aeron could benefit from faster and more efficient connection establishment. The end result was that the wider software community now benefits from this useful feature on multiple operating systems, and it happened transparently, for everyone.
Assessing contributions
Open-source software users contributing their time or sponsoring development is one of the best expressions of value for a new feature. This is a much stronger indicator than a request for a new feature that is not backed by “putting skin in the game”.
It is equally important for the core team to understand any contributions which are made so that they can be supported longer-term, as there is no guarantee the contributor will be available in the future. Thus accepting contributions is a tricky balance, but when well-managed is positive for an OSS project.
In my experience, I have rejected some contributions due to being of obvious poor quality; some I accepted, though later regretted, due to the effort required to improve the quality. However, in the round I believe the majority of contributions are positive.
Additionally, end users will often ask how they should contribute. This does not have to be with lots of code. It can be small – but really important contributions such as tests, documentation or just answering community questions – all are very welcome.
Sponsored development
What does “sponsored development” mean in the case of open-source software development?
Companies that use OSS often require a feature, or set of features, to match their requirements. Often these features can be common across the community, and one or more companies can offer to pay for the work effort by core contributors to develop the feature. This benefits the company in that they get their required feature, plus it gets supported in the project longer-term.
Wise companies realise the cost of maintaining software which has been developed internally can be significantly greater than the initial development effort. Thus sponsoring OSS can not only be good for the community, it is often the most cost-effective way to achieve quality software.
Correspondingly, contributors who work on an OSS project often stay with it even if they change employers, especially if their employers encourage open-source software contributions. This results in the people maintaining the software being the ones who know it best.
Notably, this is often very different from most internally-developed software, which may not be well understood as individuals move on externally to a different employer or just to a different division within the company.
Learning opportunities
The peer review that comes with development in the open is really special. Not only can recognised experts in the field contribute feedback with less friction, the less-experienced can ask questions, often revealing great insight and spawning new avenues of thinking. None of us can aspire to produce great software if we have not seen what some of the best software looks like.
In fact, the nature of closed-source is one of the greatest hurdles to our profession’s advancement. How do we learn when we have limited examples to learn from? By developing openly and transparently other developers can read code and learn from it.
Often, they ask questions that help the original authors to consider how they can do things better. To quote the author Stephen King – when asked what makes a great writer, he responded with, “to be a great writer you must first learn to be a great reader.”
All developers should spend more time reading software developed by others. Sometimes the “others” include your previous self, and the longer you stay with a project the more likely it is that you will have to maintain your own code as you revisit it.
Stewardship
At Real Logic, and now Adaptive, I take our stewardship of OSS very seriously, and want to avoid creating “abandonware”. I appreciate the benefits outlined above and make heavy use of Aeron open-source software myself.
As its caretaker I see my responsibility to maintain the quality and active development of Aeron. I also appreciate that some companies need more than community support to meet service levels, and for that we offer additional commercial support.
As a steward, I would encourage users who benefit from open-source software to contribute back in any way they can. This can range from simply publicly supporting the OSS and declaring their usage, to commercially supporting the contributors with sponsorship, or even by purchasing support offerings.
In conclusion, building open-source software projects collaboratively and in the open from the start gives significant advantages and helps produce a higher-quality, more resilient, and superior product.