Why picking a license isn’t the first step in going open source

I was at a conference in December last year where I was giving a talk on ARI in Asterisk 12. After my talk, one of the attendees came up, and asked if he could talk to me about running an open source project. I don’t consider myself an expert on the subject by any stretch of the imagination – yes, I’m the project lead for Asterisk, but I learn something new about open source every day! But, being daring, I said yes. His question was approximately thus:

“Once upon a time, I wrote a tool that I sold for a small amount of money to a large number of companies, some of whom are rather large. Over the past many years, I’ve re-written said tool such that it’s a full on project, and I’ve realized that to get it done and available, there are a number of things I could use help with. I’d like to take it open source so that I can finally get this software out there. What open source license should I pick?”

I have to admit that I had a quick, off-the-cuff reply: “don’t pick the GPL”. This certainly surprised him – Asterisk, after all, is dual licensed under the GPLv2, such that Digium can choose to commercially license Asterisk. Our business model certainly benefits from this licensing model. He was, in fact, seriously considering GPLv2 for his project as well – specifically because he wanted to dual license his project. It was, I admitted immediately to him, a glib answer, but one that stemmed from my experience over the past couple of years dealing with the GPL. While a popular and powerful license, the dual licensing aspect is tricky: you have to have a good understanding of what can and cannot be used with a GPL licensed application. This can get particularly sticky when you consider all of the various software distribution models that are out there, some of which were not as prevalent when the GPL was written. While licensing your project under the GPL requires a good amount of knowledge, providing a commercial license compounds this; you now have to manage exceptions to the GPL, which has both public perception issues as well as real legal issues. None of this is fun (at least, I don’t find it fun), and it certainly would be a lot of work for him. But I realized I spoke too quickly, because there was a much more fundamental issue at play here: what, exactly did he want to do with his project?

In other words: what is his motivation in open sourcing his project?

I’m going to start with the assumption that this is not a moral question. If it is a moral question, then everything that follows here is moot. If you morally feel that all software should be free – as in libre – then there is no question, no dilemma. You open source the bloody thing: in fact, it should have been free when you started! But I assumed that this wasn’t his case – because if so, than the answer of which license you pick is more of a personal preference (although I’m sure some morally inclined people would disagree; it’s still easier than where I’m going with this next).

If his goal was to have people use his software, then open sourcing his project makes sense – and it’s fine to discuss which license to pick.

If his goal is to make money, then open sourcing his project at this time is crazy.

Why is that?

It is not easy to make money on open source software. Yes, lots of companies do it: Red Hat, Digium (yay!), Canonical, etc. However, if you look closely at companies that make money on open source software – producing it, not just using it – you’ll find that they rely on more than open source. They have some other mechanism to generate revenue:

  • Many companies have something that is not free – both in the libre and the beer sense. Think licensing costs, commercial add-on modules which are neither open or free, etc. If your product is software, you want to find some way to monetize that software. While most open source licenses don’t care if you charge for your software, the end result is that unless you restrict your software in some fashion or provide some added value outside of the open source software, you won’t make any money off your software. You have to hold something back or restrict the software in some way in order to generate revenue strictly from your software.
  • Paid support and/or training on the software. However, as a primary means for a business model, this is not a recipe for large-scale success. This is for two reasons: (1) it does not scale. You may make money yourself by providing support for your software, but as the popularity grows, you will lose ground. There’s only one of you, and the support you provide will eat into your actual development effort. This can, ironically, cause your project to lose popularity. If you find someone to help you, you can hire them; but, assuming they are equally as smart and talented and dedicated as you, you’ve only prolonged the problem. You can’t hire every customer to support your project. (2) Relying only on support implies your software is impenetrable, difficult, and buggy. As you fix that, people will no longer need to pay you (or pay you as much) for support. The more your software matures, the less necessary you are. That is a good thing, but it does undermine this business model.
  • Providing said software as a service, if the project allows for it. However, this may not be enough, as it is difficult to maintain a competitive edge unless the software is substantially complex. If you build a service on top of open source software, someone else will build one too. And another one. And the race to zero begins. Since all modifications have to be made and distributed freely (although SaaS has circumvented many of the older licenses, much to the consternation of the FSF), there is little competitive advantage from one service provider to another (unless, of course, you don’t have to distribute your changes: but that’s hardly open and libre). Note that some open source licenses are much more permissive than the FSF licenses in this regard however – but creating services from your software has a high cost associated with it as well (infrastructure isn’t free). It is, however, one of the more attractive business models: witness Rackspace, hosted VoIP providers, etc.
  • Sell hardware. However, hardware, in general, is not a growth market. This also has a prohibitively high infrastructure/startup cost, which makes it even less palatable.

There are pro’s and con’s to any mechanism used to make money off of open source software. Regardless, the long and short of it is: you need a business plan. In fact, if your primary product is open source software, you need an even more solid business plan than your average software company. And you definitely need a business plan before you choose how to license your software, if you want to make a living from your project.

Do you have a complex project with a targeted niche where your expertise will be required? GPL may be appropriate. Are you planning on using your project as part of a larger whole? A more permissive license may be fine. But all of these are subject to the much more important question of: how do you want to make your money. Because once you ring the bell on the license, you’re going to be stuck with that business plan, and it is very hard to change it.

All of this only matters if you want to make money from your project. That is not the only reason to write software. For developers, it often isn’t even the most important one – and if what you want is to see your work get used, then by all means – license however you feel best to you. Permissive licenses will get your software used the most and the fastest; but there’s something to be said for protecting the freedom of your software as well.

But always know your motivations before you release your software. Otherwise, you may end up with a lot of unnecessary regrets.

As an epilogue to this whole post: the individual wanted to make money. He was sure that since he made a little bit of money with the tool, that he’d make a lot with the project – and he viewed open sourcing it as a way of getting free help. There’s a whole host of problems with viewing open source development in that light as well – not the least of which is that you don’t control what your open source contributors work on. I’m not sure he appreciated the sentiment of needing a concrete business plan either; we in the programming/engineering field often don’t like to think through such things. Still, I wished him the best of luck – he has a tough road ahead.

Oh Discordia

I found out today that the man who hired me out of college and mentored me at my first job died in his sleep on New Years Eve. He was only 44.

They believe it was complications from diabetes. He was, however, relatively healthy overall. Finding out that he is dead is so completely unexpected that there aren’t really words to describe it. There aren’t words to describe the entire situation; it is horrible.

I owe a lot to this man. When I find myself in a difficult spot with co-workers, I often think, how would he defuse the situation? When I find myself dreading a phone call or dealing with a situation, I remember him sighing and picking up the phone time and time again to pull my bacon out of the fire with some irate customer. When I wonder what I should say to someone who is frustrated, I remember his advice to me while I was on the road and working late at night. He shaped my understanding of business, and what it means to have a career you can be proud of.

More than what he gave me, he was, quite frankly, just an awesome guy. A lover of dogs, a talented woodworking craftsman, a die-hard Philadelphia sports fan, a tenacious worker with a savvy understanding of technology, a man who could throw an amazing party, a loving husband, a great man and a great friend.

The world is darker with him gone.

Asterisk 12: Now Available (with PJSIP)

We released Asterisk 12 on Friday. So, that’s a thing.

There’s a lot to talk about in Asterisk 12 – as mentioned when we released the alpha, there’s a lot in Asterisk 12 that makes it unlike just about any prior release of Asterisk. A new SIP channel driver, all new bridging architecture (which is pretty much the whole “make a call” part of Asterisk), redone CDR engine (borne out of necessity, not choice), CEL refactoring, a new internal publish/subscribe message bus (Stasis), AMI v2.0.0 (build on Stasis), the Asterisk REST Interface (ARI, also built on Stasis), not to mention the death of chan_agent – replaced by AppAgentPool, completely refactored Parking, adhoc multi-party bridging, rebuilt features, the list goes on and on. Phew. It’s a lot of new code.

Lots of blog posts to write on all that content.

I’ve been thinking a lot about how we got here. The SIP channel driver is easiest: I’ll start there.

Ever since I was fortunate enough to find myself working on Asterisk – now two years, five months ago (where does the time go!) – it was clear chan_sip was a problem. Some people would probably have called it “the problem”. You can see why – output of sloccount below:

  • 1.4 – 15,596
  • 1.6.2 – 19,804
  • 1.8 – 23,730
  • 11 – 25,823
  • 12 – 25,674

Now, I’m not one for measuring much by SLOC. Say what you will about Bill Gates, but he got it right when he compared measuring software progress by SLOC to measuring aircraft building progress by weight. That aside, no matter who you are, I think you can make two statements from those numbers:

  1. The numbers go up – which means we just kept on piling more crap into chan_sip
  2. The SLOC is too damned high

sloc_is_too_highI don’t care who you are, 15000 lines of code in a single file is hard to wrap your head around. 25000 is just silly. To be honest, it’s laughable. (As an aside: when I first started working on Asterisk and saw chan_sip (and app_voicemail, but that’s another story), I thought – “oh hell, what did I get myself into”. And that wasn’t a good “oh hell”.)

It isn’t like people haven’t tried to kill it off. Numerous folks in the Asterisk community have taken a shot at it – sometimes starting from scratch, sometimes trying to refactor it. For various reasons, the efforts failed. That isn’t an insult to anyone who attempted it: writing a SIP channel driver from scratch or refactoring chan_sip is an enormous task. It’s flat out daunting. It isn’t just writing a SIP stack – it’s integrating it all into Asterisk. And, giving credit to chan_sip, there’s a lot of functionality bundled into that 15000/25000 lines of code. Forget calling – everything from a SIP registrar, SIP presence agent (kinda, anyway), events, T.38 fax state, a bajillion configuration parameters is implemented in there. The 10% rule applies especially to this situation – you need to implement 90% of the functionality in chan_sip to have a useful SIP stack, and the 10% that you don’t implement is going to be the 10% everyone cares about. And none of them will agree on what that 10% is.

As early as Asterisk 11, a number of folks that I worked with had starting thinking about what it would take to rewrite chan_sip. At the time, I was a sceptic. I’m a firm believer in not writing things from scratch: 90% of new software projects fail (thanks Richard Kulisz for the link to that – I knew the statistic from somewhere, but couldn’t find it.) (And there’s a lot of these 90/10 rules, aren’t there? I wonder if there’s a reason for that?). Since so many new software projects fail – and writing a new SIP channel driver would certainly be one – I figured refactoring the existing chan_sip would be a better course. But I was wrong.

Refactoring chan_sip would be the better course if it had more structure, but the truth is: it doesn’t. Messages come in and get roughly routed to SIP message type handlers; that’s about it. There’s code that gets re-used for different types of messages, and change their behaviour based on the type of message. But a lot of that is just bad implementation and bad organizational design; worse is the bad end user design. You can’t fix users.conf; you can’t fix peers/friends/users in sip.conf. (And no, the rules for peers versus friends isn’t consistent.) Those are things that even if you had a perfect implementation you’d still have to live with, and that’s not something I think any one wants to support forever.

But, I don’t believe that not liking something is justification for replacing it. After all, chan_sip makes a lot of phone calls. And that’s not nothing. So, why write a new SIP channel driver?

As someone who has to act as the Scrum master during sprint planning and our daily scrums, I know we spend a lot of time on chan_sip. Running the statistics, about 25% of the bugs in Asterisk are against chan_sip; anecdotally, half of the time we spend is on chan_sip. When we’re in full out bug fix mode – which is a lot of the time – that’s half of an eight man development team doing nothing but fixing bugs in one file. Given its structure, even with a lot of testing, we still find ourselves introducing bugs as we fix new ones. Each regression means we spent more than twice the cost on the original bug: the cost to fix it, the cost to deal with triaging and diagnosing the resulting regression report, the cost to actually fix the regression, the impact to whatever issue doesn’t get fixed because we’re now fixing the regression, the cost to write the test to ensure we don’t regress again, etc. What’s more, all of the time spent patching those bugs is time that isn’t spent on new features, new patches from the community, improving the Asterisk architecture, and generally moving the project forward.

Trying to maintain chan_sip is like running in place: you’re doing a lot, but you aren’t going anywhere.

A few weeks before AstriDevCon in 2012, we were convinced that we should do something. There wasn’t any one meeting, but over the months leading up to it, the thought of rewriting chan_sip crossed a number of people’s minds. There were a few factors converging that motivated us prior to the developer conference:

  1. In my mind, the knowledge that we were spending half of the team’s energy on merely maintaining a thing was motivation enough to do something about it.
  2. The shelving of Asterisk SCF. Regardless of the how and the why that occurred, the result was that we had learned a lot about how to write a SIP stack that was not chan_sip. Knowing that it could be done was a powerful motivator to do it in Asterisk.
  3. Asterisk 11. We had spent a lot of time and energy making that as good of an LTS as we thought we could: so if we were going to do something major in Asterisk, the time was definitely now.

As we went into AstriDevCon, the foremost question was, “would the developer community be behind it? Would they want to go along with us?”

As it turned out: yes. In fact, I wasn’t the first one to bring it up at AstriDevCon – Jared Smith (of BlueHost) was. And once the ball got rolling, there wasn’t any stopping it.

The rest, as they say, is history. And so, on Friday, the successor to chan_sip was released to the world.

There’s a long ways to go still. Say what you will about chan_sip (and you can say a lot), it is interoperable with a lot of devices, equipment, ITSPs, and all sorts of other random SIP stacks. It does “just” work. And chan_pjsip – despite all of our testing and the knowledge that it is built on a proven SIP stack, PJSIP – has not been deployed. It will take time to make it as interoperable as chan_sip is. But we’ll get there, and the first biggest step has been taken.

The best part: all of the above was but one project in Asterisk 12! Two equally large projects were undertaken at the same time – the core refactoring and Stasis/ARI – because if you’re going to re-architect your project, you might as well do as much as you can. But more on that next time.

Agile and Open Source

Googling any combination of the keywords Agile and Open Source yields dismal results. In general, what you’ll find are a plethora of Open Source Agile tools (most of which are mediocre at best – whatever happened to “Individuals and interactions over processes and tools”?) and almost no source on marrying Agile and Open Source software development. To Twitter to complain!

And thus enters the power of the internet. Not surprisingly, both Kevin and Russell commented back – and Russell linked to another blog post and a mailing list thread he was involved in that cast some cold water on marrying Agile – or at least Scrum – and FOSS.

Disclaimer: I’m a big fan of Agile development.

Background

I came to Agile development painfully. I’d say prior to doing Agile development – in some flavor – I did one of two forms of development:

  1. Heroic Development: Also called “no process”. This is where planning is minimal, requirements are barely known, and you only release software through the blood, sweat, and tears of you and your compatriots. The highs are high; the lows are abysmal. Projects typically succeed once (we released 1.0!) and then fail miserably as projects gain complexity. Things drift. I’d guess that many FOSS projects fall into the Heroic Development category, which may account for why so many fail (and many do fail – I don’t think 95% is far off).
  2. Mythical Man-Month Development: This goes by many names. Waterfall. Spiral. Iterative (which is often confused with being Agile). It’s an approach that marries well with 1970′s programming. It sometimes makes sense with mission critical embedded software development, where environments are well known, requirements are set in stone, and releases are few (if not just one). In general, however, it is a terrible, terrible approach to software development for any modern project.

Working in one of those two development models for years taught me a lot. I watched a lot of projects fail despite my best efforts; I watched a lot of good code get thrown away because the customer (who is that? Write more code!) never knew what was being written for them. It made me yearn for some way of doing my job that didn’t suck the life out of me. So I have a hard time thinking that Agile – which, while not perfect, is the closest I’ve seen to a sustainable, working model of software development – can’t mesh with what is essentially a way of organizing a project. Agile doesn’t say “close your source code!”. Nor does being Open Source imply chaos.

So… wherein lies the conflict?

What is Agile?

Agile is all the rage. There’s Agile Certification Courses all over the internet; I’m not sure what a certificate in being Agile is supposed to tell me, or anyone else. There’s Agile development tools everywhere you look (“eBook: Agile Portfolio Management Drives Innovation”. I just threw up in my mouth a little). While I appreciate the usefulness of good tools, so much of this is obviously snake oil that it is hard to tell what is genuinely good from what is crap. When in doubt, I find it’s always good to go back to the source.

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Does anything advertised in the pages I linked to emphasize the concepts on the left side of those sentences? Not really… and I have a suspicion that people view Agile development through the prism that has been foisted on them by said salesman. So since the internet isn’t much help on this one, how can we apply Agile methodologies to Open Source?

At first glance, nothing in the manifesto contradicts the principles underlying FOSS projects. In fact, every aspect of it feels directly in line with how FOSS projects are managed and run. FOSS thrives on individual interactions, although it tends to focus on IRC and mailing lists over personal interaction. FOSS is all about working software (and could usually use more documentation – after all, we still value the things on the right side of those sentences). FOSS is usually driven by customers: it is through deployments and customer needs that many developers on FOSS make their livelihood, and you’d be hard pressed to find any contracts in any official capacity in a FOSS project. FOSS projects are all about change: plans are usually loose and non-binding, if they exist at all.

So what’s the problem?

Implementation Concerns

As is so often the case, the devil is in the details. Implementations of Agile development tend to be hard to implement in open source projects. Extreme Programming emphasizes pair programming; that is obviously nearly impossible in a FOSS project. Demos are difficult – who do you demo to, and who is your product owner? Scrum has the daily standups, which are difficult to organize across a plethora of time zones (although that issue is not unique to FOSS: any company with distributed teams runs into that problem.) How do you organize a backlog? Or obtain consensus on said organization? How do you coordinate participation in user story development? The vary roles in said participation (scrum master/product owner/etc.) get fuzzy quickly.

Rather than get stuck in the muck on one particular Agile implementation, I think it’s a good idea to look at the principles behind the manifesto. These tend to feed more directly into these implementations, and can show the root of some of these conflicts:

  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

    No problems here. Write software!

  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

    Change is usually welcome in FOSS projects; I can’t think of an instance where this wasn’t the case. It’s usually hard to think of cases where requirements were defined in stone in the first place.

  3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

    This can be more difficult in FOSS. Everyone has a different time scale that they work on, and coordinating that across many organizations/individuals with different constraints is impossible. I think the important point here is to release as much as possible for the project, with a reasonably predictable schedule. We balance this in Asterisk – we have bug fixes every six months; new major versions every year. In general, this seems to be striking the right balance – but it’s also always a contentious discussion every time we bring up potential changes. I’m not sure what the answer is here, but I can say that delaying releases has always backfired. The longer the gap in releases, the more difficult the recovery period is as we iron out things missed during the test cycles. Regression releases are common in such scenarios.

  4. Business people and developers must work together daily throughout the project.

    No problems here – businesses often drive the features that go into FOSS and sponsor development.

  5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

    This is more of a dig against centralized command and control management than anything else. Companies that view two “Engineer 3″ developers as being equivalent and interchangeable don’t understand this concept.

  6. The most efficient and effective method of conveying information to and within a development
    team is face-to-face conversation.

    And… here we are. This is the one that sticks in the craw of FOSS. But even if face-to-face conversation isn’t always possible in a FOSS project, there are some points that I think that FOSS projects have to admit.

    • Face-to-face conversation is the most efficient and effective mode of conveying information. Nuance and subtleties don’t translate well in IRC or mailing lists. Witness the power of FOSS conferences and many open source contributors desire to meet up with fellow contributors.
    • Even when face-to-face conversation isn’t possible, open source projects do live and die by their communications. Negative communication, ill will, and malcontents are poison to the lifeblood of FOSS. Emphasizing positive, productive communication has to be a cornerstone of any FOSS project.
  7. Working software is the primary measure of progress.

    Yup!

  8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

    This is something FOSS could learn from. I think many newcomers to a FOSS project fall into the trap of taking on too much and getting involved in too many places. Participating at a constant, steady pace is often more valuable than someone who is involved for a short period of time and wanders off to another project when they burn themselves out.

  9. Continuous attention to technical excellence and good design enhances agility.

    Again, this is something that many FOSS projects have learned at their own peril. Early in many FOSS projects, the tendency is to accept any code that “works” – for some definition of work (it worked on my machine! My customers love it!) This can lead to compromised design, which eventually constrains the project and makes it difficult to adapt to changing requirements. The chaotic nature of such projects can make it difficult to address long standing design flaws that creep in over time; a lack of adherence to technical excellence and good design can quickly exacerbate this situation. Witness the effort of unscrewing the damage of masquerades in Asterisk.

  10. Simplicity–the art of maximizing the amount of work not done–is essential.

    This is rarely a problem in FOSS. In general, contributions are made because someone, somewhere, found them useful. Rarely is code contributed that has no practical purpose.

  11. The best architectures, requirements, and designs emerge from self-organizing teams.

    This marries well with FOSS: the people participating in the project want to be there. Respect in a project is earned, not bestowed.

  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

    Again, this can be difficult for FOSS projects to do, but it can be done. Periodic retrospectives are possible for any project, even if they don’t align with a “sprint”.

Conclusions?

After going through all of them, there aren’t many conflicts. The difficultly seems to lie directly with the implementations of Agile, and not with Open Source. What is needed is an Agile Development methodology directed at Open Source projects, but one that still meets the principles that guide Agile methodology.

This has been some rambling thoughts, none of which resolved the reason I started thinking about this; namely, how do I better coordinate Agile development with running an Open Source project. There are still real world concerns to wrestle with, some of which still need some more thought.

I do think Thierry Carrez nails the essential differences between Agile and Open Source:

The goals are also different. The main goal of Agile in my opinion is to maximize a development team productivity. Optimizing the team velocity so that the most can be achieved by a given-size team. The main goal of Open source project management is not to maximize productivity. It’s to maximize contributions. Produce the most, with the largest group of people possible.

He’s right; however, I have to quibble a bit. The tone makes it sound as if maximizing contributions will trump maximizing team productivity; while that can be true, it isn’t always. This is a bitter pill for that most advocates of Open Source projects don’t want to swallow. Not all contributions are created equal. A large number of trivial contributions are useful, but are not equivalent to a few good major, meaningful contributions. It’s perfectly fine to maximize contributions in an open source project, but not at the expense of the larger strategic goals for the project. Agile feels like the answer to this; the question is how to implement it effectively while still maintaining the large collaborate nature of an Open Source project.

Asterisk 12 on a Raspberry Pi

So, like a lot of people, as soon as I could I got my hands on a Raspberry Pi (I got mine from Adafruit and was very happy with the experience). If you don’t know what a Raspberry Pi is, it’s the greatest thing since sliced bread – a small single board computer that runs Linux for only thirty-five bucks. Along with a case, a good power supply, and a few other odds and ends, it’s still a steal, coming in somewhere under seventy-five bucks. They’re ridiculously flexible. They can be used for a whole host of microcontroller projects, but are also powerful enough to act as mini home “servers”. For awhile now, I’ve planned to set mine up as a home Asterisk system. In particular, my poor wife – who happens to work from home – has had to make do without much in the way of good business communications (there’s a story in there somewhere about a cobbler and his kids not having any shoes). Unfortunately, just about every waking moment for the past six months has been spent on getting Asterisk 12 developed and released, so my poor Pi has sat on a desk collecting dust.

Today, however, no more. I got the Pi out, plugged it in, connected it to my home network, and got tinkering. The goal: get Asterisk 12 running on a Raspberry Pi. By running, I mean just running – configuration is a bit beyond the scope of today (or this blog post) – but if I can get myself to the CLI prompt, that will do. As an aside, this is realistic: the Raspberry Pi does not compile quickly. I ended up running a lot of errands between steps, so this whole project was stretched out quite a bit over today.

It’s not often that a developer gets to take a step back and look at things from a user’s perspective, so this should be a lot of fun.

As another aside, I think like most folks using a Raspberry Pi for the first time, I found the experience to be quite a pleasure. While I would never claim that I’m a Linux guru (I did work in Microsoft shops for quite a long time; I still find myself moving back and forth between the two worlds), I can get around just fine in the major Linux distros and have no problems setting up a Linux system. Even still, I was still pleasantly surprised at the configuration tools that come with the Pi. They really nailed their target audience, and even for those of us who use Linux daily, they still made life nice and easy.

Moving on!

I spent the first bit just configuring the Pi with the tools. I set it up with a static IP address behind my home router, updated Wheezy, changed the default user’s password to something secure, and changed the hostname to ‘mjordan-pi’ (terribly original, I know. I’m not good at naming things – I leave that to Mr. Joshua Colp on our team)

At this point, I figured I should be good to go on the actual task of getting Asterisk downloaded, installed, and configured. While the folks at Asterisk for Raspberry Pi have done a spectacular job of documenting and making it easy to install Asterisk on a Pi, I’m going to depart from their guidelines here. While I love FreePBX, I’m going to eschew a GUI as well as MySQL. I really want a relatively stream-lined install of Asterisk 12, with .conf files for configuration, access through the CLI, and everything done by hand. When we get to configuration, you’ll note that I’m going to take a pretty slow and conservative approach to configuration – but that will let me look at each module and step and really digest what I’m doing. We’re going old school here. Should be fun!

Step 1: Get Asterisk

Since I’m working through ssh from my laptop, everything is going to be done through the shell. That means downloading Asterisk using wget from downloads.asterisk.org.

pi@mjordan-pi ~ $ wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-12-current.tar.gz
--2013-09-01 16:27:28-- http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-12-current.tar.gz
 Resolving downloads.asterisk.org (downloads.asterisk.org)... 76.164.171.238, 2001:470:e0d4::ee
 Connecting to downloads.asterisk.org (downloads.asterisk.org)|76.164.171.238|:80... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 56720157 (54M) [application/x-gzip]
 Saving to: `asterisk-12-current.tar.gz'
100%[====================================================================================================>] 56,720,157 982K/s in 61s
2013-09-01 16:28:29 (905 KB/s) - `asterisk-12-current.tar.gz' saved [56720157/56720157]

Hooray! We got it. Now to untar it:

pi@mjordan-pi ~ $ tar -zvxf asterisk-12-current.tar.gz

As you may notice, Asterisk 12 is a bit large (what can I say, we did a lot of work). Some of that heft comes from the exported documentation from the Asterisk wiki, in the form of the Asterisk Administrators Guide. Since I don’t really need that on my Pi, I’m going to go ahead and get rid of it, as well as the tarball now that I’ve extracted it.

pi@mjordan-pi ~ $ rm asterisk-12-current.tar.gz
pi@mjordan-pi ~ $ rm asterisk-12.0.0-alpha1/Asterisk-Admin-Guide-12.html.zip
pi@mjordan-pi ~ $ rm asterisk-12.0.0-alpha1/doc/Asterisk-Admin-Guide-12.pdf

Random note here. You might be wondering why the PDF is in the doc subfolder, while the zip of the HTML documentation is in the base directory. When we were making the Alpha tarball, we had a number of problems crop up in the scripts that make the release. There’s a lot of moving parts in that process, in particular with making the release summaries and pulling the documentation from Confluence. We had some connectivity issues going back from the release build virtual machine to the issue tracker, such that the connection would drop – or have some random fit at least – while it was trying to obtain information about Asterisk issues. After the fourth or fifth exception thrown by the script (various socket errors, HTTP timeouts, and other random badness), we had managed to pull all of the data but – for various reasons – not in the correct locations in the directory that was destined to become the tarball. So the location of the zip file is a goof on my part – I had to move it manually, and accidentally moved it into the wrong location. All of this went to show that it’s a well known fact that whatever can go wrong in your systems will go wrong just as you’re trying to push something live, particularly if you’re supposed to meet with your colleagues for a celebratory beer in five minutes. Every. Freaking. Time.

Anyway, let’s see how far we get running configure:

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ ./configure
configure: error: *** termcap support not found (on modern systems, this typically means the ncurses development package is missing)

That’s not terribly surprising. In fact, http://www.raspberry-asterisk.org/faq/ has a good list of dependencies you’ll need if your’e installing Asterisk from source. As it is, I don’t really want all of the libraries listed in the FAQ. For example, I know I won’t be integrating Asterisk with MySQL, as I am – for the time – eschewing any Realtime configuration. But most of the rest are needed, and, as the Raspberry Pi is slow, it’s good to think things through and get things right the first time.

Step 2: Install Asterisk Dependencies

Let’s get what we do know:

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ apt-get install build-essential libsqlite3-dev libxml2-dev libncurses5-dev libncursesw5-dev libiksemel-dev libssl-dev libeditline-dev libedit-dev curl libcurl4-gnutls-dev

Since this is Asterisk 12, however, I’ll want a few other things as well. The CHANGES and UPGRADE.txt file tell you the additional dependencies – let’s got those as well:

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ apt-get install libjansson4 libjansson-dev libuuid1 uuid-dev libxslt1-dev liburiparser-dev liburiparser1

We could, at this point, configure and build Asterisk – but there’s one more thing I want. Since this is Asterisk 12, we have the brand new SIP stack to try out, and I’m definitely going to be using it over the legacy chan_sip channel driver. But, to get it working, we’ll need PJSIP.

Step 3: Get PJSIP

As of today (and I’m hopeful this isn’t a permanent situation), Asterisk needs a particular flavor of pjproject (I’m going to use the terms pjproject/PJSIP interchangeably here. There is a difference, but let’s just pretend there isn’t for the purposes of this blog post). There’s instructions on the wiki for how to obtain, configure, and install pjproject for use with Asterisk:

https://wiki.asterisk.org/wiki/display/AST/Installing+pjproject

Following along with the instructions, we first need git. Let’s get that:

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ apt-get install git

And then we can clone pjproject from our repo on github:

pi@mjordan-pi $ git clone https://github.com/asterisk/pjproject pjproject
pi@mjordan-pi $ cd pjproject

Now here’s the tricky part. What parameters should we pass to the configure script?

At a minimum, from the wiki article we know we need to tell it to install in /usr, and to enable shared objects with –enable-shared. But pjproject embeds a lot of third party libraries, which will conflict if we want to use them in Asterisk (or at least, will generally not play nicely). It is very important to know what you have on your system what installing pjproject, and what you want to use in Asterisk.

In my case, I don’t have libsrtp installed and I’m not going to use SRTP in Asterisk, so I can ignore that. I also don’t have libspeex installed, and I don’t really care about using libspeex in Asterisk either. The same goes for the GSM library. So, in the end I ended up with the following:

pi@mjordan-pi ~/pjproject $ ./configure --prefix=/usr --enable-shared --disable-sound --disable-video --disable-resample

After a little bit, we get the configuration success message:

Configurations for current target have been written to 'build.mak', and 'os-auto.mak' in various build directories, and pjlib/include/pj/compat/os_auto.h.
Further customizations can be put in:
 - 'user.mak'
 - 'pjlib/include/pj/config_site.h'
The next step now is to run 'make dep' and 'make'.

I went ahead and skipped ‘make dep’, since we don’t really need to do that step. Thus… the fateful command was typed:

pi@mjordan-pi ~/pjproject $ make

This took a long time.

if test ! -d ../bin/samples/armv6l-unknown-linux-gnu; then mkdir -p ../bin/samples/armv6l-unknown-linux-gnu; fi
 gcc -o ../bin/samples/armv6l-unknown-linux-gnu/vid_streamutil \
 output/sample-armv6l-unknown-linux-gnu/vid_streamutil.o -L/home/pi/pjproject/pjlib/lib -L/home/pi/pjproject/pjlib-util/lib -L/home/pi/pjproject/pjnath/lib -L/home/pi/pjproject/pjmedia/lib -L/home/pi/pjproject/pjsip/lib -L/home/pi/pjproject/third_party/lib -lpjsua -lpjsip-ua -lpjsip-simple -lpjsip -lpjmedia-codec -lpjmedia -lpjmedia-videodev -lpjmedia-audiodev -lpjnath -lpjlib-util -lmilenage -lsrtp -lgsmcodec -lspeex -lilbccodec -lg7221codec -lpj -lm -luuid -lm -lnsl -lrt -lpthread -lcrypto -lssl
 make[2]: Leaving directory `/home/pi/pjproject/pjsip-apps/build'
 make[1]: Leaving directory `/home/pi/pjproject/pjsip-apps/build'

Huzzah! Let’s get this bad boy installed.

pi@mjordan-pi ~/pjproject $ sudo make install

And after a bit:

for d in pjlib pjlib-util pjnath pjmedia pjsip; do \
 cp -RLf $d/include/* /usr/include/; \
 done
 mkdir -p /usr/lib/pkgconfig
 sed -e "s!@PREFIX@!/usr!" libpjproject.pc.in | \
 sed -e "s!@INCLUDEDIR@!/usr/include!" | \
 sed -e "s!@LIBDIR@!/usr/lib!" | \
 sed -e "s/@PJ_VERSION@/2.1.0/" | \
 sed -e "s!@PJ_LDLIBS@!-lpjsua -lpjsip-ua -lpjsip-simple -lpjsip -lpjmedia-codec -lpjmedia -lpjmedia-videodev -lpjmedia-audiodev -lpjnath -lpjlib-util -lmilenage -lsrtp -lgsmcodec -lspeex -lilbccodec -lg7221codec -lpj -lm -luuid -lm -lnsl -lrt -lpthread -lcrypto -lssl!" | \
 sed -e "s!@PJ_INSTALL_CFLAGS@!-I/usr/include -DPJ_AUTOCONF=1 -O2 -DPJ_IS_BIG_ENDIAN=0 -DPJ_IS_LITTLE_ENDIAN=1 -fPIC!" > //usr/lib/pkgconfig/libpjproject.pc

Let’s verify we got it.

pi@mjordan-pi ~/pjproject $ pkg-config --list-all | grep pjproject
libpjproject libpjproject - Multimedia communication library
pi@mjordan-pi ~/pjproject $

Yay! As an aside, Asterisk uses pkg-config to locate libpjproject – so if this step fails, something has gone horribly wrong and Asterisk is not going to find libpjproject either. So this is always a useful step to perform, regardless of the platform you’re installing Asterisk on.

Step 4: Build and Install Asterisk

Back to Asterisk!

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ ./configure --with-pjproject

I specified –with-pjproject, because I really wanted to know if it failed. It takes a bit to compile, and this way the configure script will tell me. Otherwise, the first time I’ll find out whether or not I have the dependencies for the new SIP stack is when I fire up menuselect.

configure: Menuselect build configuration successfully completed
 $$$$$$$$$$$$$$$$.
configure: Package configured for:
 configure: OS type : linux-gnueabihf
 configure: Host CPU : armv6l
 configure: build-cpu:vendor:os: armv6l : unknown : linux-gnueabihf :
 configure: host-cpu:vendor:os: armv6l : unknown : linux-gnueabihf :
 pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $

And now for some configuration via menuselect.

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ make menuselect

I’m going to be a bit conservative here. There’s a lot of things I just won’t need for this Asterisk install. Rather than build everything and exclude them all via modules.conf, I’m going to disable them here. That way my overall installation will just be smaller (space is a premium on a pi), and I’ll be less likely to leave something lying around in Asterisk that doesn’t need to be there.

The following are what I disabled in menuselect:

Compiler flags:

  • DONT_OPTIMIZE – this may seem odd, but since I’m running an alpha of Asterisk 12, if I have a problem, I want to be able to fix it. This will make backtraces a bit more usable.

Applications:

  • app_agent_pool – I’m not going to be using Queues or Agents.
  • app_authenticate – I’m not going to need to Authenticate people myself.
  • app_gelgenuserevent – I’m not using CEL.
  • app_forkcdr – I hate this application, but that’s probably because I had to rewrite the CDR engine in Asterisk 12. While there’s one or two valid reasons to fork a CDR, most of what ForkCDR has traditionally done is silly.
  • app_macro – Use GoSubs everyone!
  • app_milliwatt – I’ll use something else to annoy people.
  • app_page – I don’t see myself setting up a home paging system right now. I’m only planning on connecting one or two SIP phones.
  • app_privacy – Nope!
  • app_queue – If I need a tech support queue for my house, I’ve got bigger problems.
  • app_sendtext – Nope.
  • app_speech_utils – Maybe someday, but probably not. I’d rather play around with calendaring or conferencing than speech manipulation of any kind.
  • app_system – Until I have a use for it, I view this as one big security vulnerability.
  • Everything in extended, except for app_zapateller. It may be fun to try and zap a telemarketer.

CDR:

Everything but cdr_custom. I’ll probably end up logging call records using this, since we don’t tend to have more than a handful of calls a day.

CEL:

Disable everything.

Channel Drivers:

  • chan_iax2 – I’m not going to set up an IAX2 trunk, and all my phones are SIP.
  • chan_multicast_rtp – Since I’m not doing any paging, I don’t need this channel driver.
  • chan_sip – I’m committing to chan_pjsip!
  • I left chan_motif to play around with some day. This may be useful for some soft phones or XMPP text message integration.
  • Everything in the extended/deprecated sections was disabled.

Codecs:

I left all of the codecs enabled.

Formats:

  • format_jpeg – I’m not going to do any fax at this point.
  • format_vox – I shouldn’t need this either.

Functions:

  • I decided to leave most functions. Most function modules are rather small and don’t take up much space, and I find that you never know when you’re going to need one.
  • func_frame_trace – This is only ever used for development (and isn’t used often then)
  • func_pitchshift – Funny, but not super useful.
  • func_sysinfo – Interesting, but I shouldn’t need it.
  • func_env – I shouldn’t need to muck around with environment variables from the dialplan.

PBX:

  • pbx_ael – Classic dialplan all they way for me.
  • pbx_dundi – I don’t think I’ll be needing anything as complex as DUNDi for my little Pi.
  • pbx_realtime – Definitely not. I’m not a fan of realtime dialplan, for a variety of reasons.

Resources:

  • I’m going to leave a number of resources that I may end up using someday, such as res_calendar. The ones I know for sure I won’t use I’ll disable.
  • res_adsi – Nope. I’m not sure you can even find ADSI devices still.
  • res_config_curl – Nope, not doing Realtime. I’ll come back and enable it if I decide to do Realtime some day.
  • res_config_sqlite3 – Nope, for the same reason as res_config_curl.
  • res_fax – Ew, fax. Not going to mess with fax at home.
  • res_pjsip_info_dtmf – I shouldn’t need DTMF over INFO.
  • res_pjsip_endpoint_identifier_anonymous – I don’t want anonymous access.
  • res_pjsip_one_touch_record_info – The SIP devices I’m using won’t need this.
  • res_pjsip_t38 – Fax: just say no.
  • res_rtp_multicast – Since I’m not doing any paging, I don’t need res_rtp_multicast.
  • res_smdi – Same reason as res_adsi.
  • res_speech – Nope, for the same reason as the speech utilities.
  • Everything in extended, except: res_http_websocket. That should be fun to play with at some point, and ARI needs it.

MoH/Sounds:

  • I enabled g.722 sounds/MoH. Wideband audio!

Phew. Time to save changes and finally compile.

menuselect changes saved!
make[1]: Leaving directory `/home/pi/asterisk-12.0.0-alpha1'
pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ make
...
Building Documentation For: channels pbx apps codecs formats cdr cel bridges funcs tests main res addons
+--------- Asterisk Build Complete ---------+
+ Asterisk has successfully been built, and +
+ can be installed by running: +
+ +
+ make install +
+-------------------------------------------+
pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $

Woot! We’re compiled. Time to install:

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ sudo make install

+---- Asterisk Installation Complete -------+
+ +
+ YOU MUST READ THE SECURITY DOCUMENT +
+ +
+ Asterisk has successfully been installed. +
+ If you would like to install the sample +
+ configuration files (overwriting any +
+ existing config files), run: +
+ +
+ make samples +
+ +
+----------------- or ---------------------+
+ +
+ You can go ahead and install the asterisk +
+ program documentation now or later run: +
+ +
+ make progdocs +
+ +
+ **Note** This requires that you have +
+ doxygen installed on your local system +
+-------------------------------------------+
pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $

And we’re installed!

I’m going to set up the bare minimum Asterisk configuration files to get Asterisk up and running. I’m not going to do anything more than get the CLI prompt up for now – I’ll save getting a phone configured and howler monkeys playing back for another time.

Step 5: Configure Asterisk

asterisk.conf

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ sudo nano /etc/asterisk/asterisk.conf

[options]
verbose = 5
debug = 0
systemname = mjordan-pi

As you can see, a pretty simple asterisk.conf. Not much else is needed for this, at least for now.

modules.conf

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ sudo nano /etc/asterisk/modules.conf

[modules]
autoload = no

; -- Bridges --

load => bridge_builtin_features.so
load => bridge_builtin_interval_features.so
load => bridge_holding.so
load => bridge_native_rtp.so
load => bridge_simple.so
load => bridge_softmix.so

; -- Formats --

load => format_g719.so
load => format_g723.so
load => format_g726.so
load => format_g729.so
load => format_gsm.so
load => format_h263.so
load => format_h264.so
load => format_ilbc.so
load => format_pcm.so
load => format_siren14.so
load => format_siren7.so
load => format_sln.so
load => format_wav_gsm.so
load => format_wav.so

; -- Codecs --

load => codec_adpcm.so
load => codec_alaw.so
load => codec_a_mu.so
load => codec_g722.so
load => codec_g726.so
load => codec_gsm.so
load => codec_ilbc.so
load => codec_lpc10.so
load => codec_resample.so
load => codec_ulaw.so

; -- PBX --

load => pbx_config.so
load => pbx_loopback.so
load => pbx_spool.so

Rather than go with autoloading, I’ve chosen to explicitly load modules. For now, I’ve only specified the “basics” – codecs and formats, the bridge modules (now used a lot in Asterisk 12), and the ancillary PBX modules. This will let me catch configuration errors easier – if you autoload, I find most people tend to ignore the warnings and errors.

extensions.conf

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ sudo nano /etc/asterisk/extensions.conf

[general]

[default]

Nothing needed in there right now!

Step 6: Profit

And now, for the moment of truth:

pi@mjordan-pi ~/asterisk-12.0.0-alpha1 $ sudo asterisk -cvvvvg

...

Loading codec_adpcm.so.
== Registered translator 'adpcmtolin' from format adpcm to slin, table cost, 900000, computational cost 1
== Registered translator 'lintoadpcm' from format slin to adpcm, table cost, 600000, computational cost 1
codec_adpcm.so => (Adaptive Differential PCM Coder/Decoder)
Loading codec_g726.so.
== Registered translator 'g726tolin' from format g726 to slin, table cost, 900000, computational cost 60000
== Registered translator 'lintog726' from format slin to g726, table cost, 600000, computational cost 120000
== Registered translator 'g726aal2tolin' from format g726aal2 to slin, table cost, 900000, computational cost 60000
== Registered translator 'lintog726aal2' from format slin to g726aal2, table cost, 600000, computational cost 120000
codec_g726.so => (ITU G.726-32kbps G726 Transcoder)
Asterisk Ready.
*CLI>

Woohoo!

Asterisk 12 Alpha

Last Friday (August 30th), we released Asterisk 12.0.0-alpha1.

This was an incredibly ambitious project. We started work on it directly after AstriDevCon last year (October 2012), so we’ve been hard at work on it for about ten months. Really, I would say that it hasn’t even really been that long – the project probably didn’t take its full shape until January of this year. So, in reality, over only the past eight months we’ve:

  • Completely re-architected bridging in Asterisk. Every phone call you make through Asterisk – where you actually talk to someone else – is probably going through the new Bridging API. The old bridging code was easily some of the most complex code in Asterisk as well – and while the new API is certainly very complex, it actually provides an abstraction of the bridging operations that the old code (which wasn’t really didn’t provide any abstraction, or an API) couldn’t ever do. Also, it has a completely different threading model that let us nearly completely annihilate masquerades.
  • Wrote a completely new SIP stack. The old SIP channel driver is 35000 lines of code. The new SIP stack and channel driver has nearly the same feature parity. That’s pretty crazy.
  • Wrote a new API – ARI (REST! JSON! WebSockets!) that exposes the communications objects in Asterisk to the outside world. You can, for really the first time, write your own communications applications in the language of your choosing, without having to mix and match AMI/AGI. You can also do things through ARI that you could never really do through AMI/AGI (asynchronous control of media, fine grained bridging operations, etc.)

And that doesn’t take into account the ancillary changes: re-writing all core bridging features, CDRs, CEL, every AMI event. It doesn’t take into account the infrastructure to make it all possible – the Stasis message bus, threadpools, and the Sorcery data access layer. And the list goes on.

How we got here wasn’t straight forward. At AstriDevCon last year, we took away only two objectives: write a new SIP channel driver, and stop exposing internal Asterisk implementation details through the APIs. We didn’t know we were going to have to gut Asterisk’s bridging code. We didn’t know we would end up having to re-write Parking, all Transfers (core and externally initiated), CDRs or CEL. We did not know we would need a more flexible publish/subscribe message bus in Asterisk, one that would end up obsoleting the existing event subsystem.

The moral of the story is: when you start a project, you have to have a goal. But the path to that goal is not a straight line. I’ll have to think for awhile how that line curved to where we find ourselves today.

I won’t say I’m not a little proud of this release. This was an incredibly ambitious project, and the team at Digium really rocked it out. We aren’t a very large team either – besides myself, there are only seven other developers. For us to get this done, with shipped code, is easily the greatest success of my professional career. That isn’t to say there isn’t a ton left to do: it is, after all, only in an alpha state. But I’m damned proud of what we’ve accomplished so far.

CHANGES

Asterisk, like many other projects, has a CHANGES file.

Ostensibly, the purpose of this file is to provide a user approaching a new major version with a summary of the major changes in the project (hence, “CHANGES”). For the user approaching the new version, it should give them insight into new features and capabilities, as well as caution them against breaking changes as they upgrade.

In reality, CHANGES files can read something like this (written from a slightly Asterisk mindset):

  • Added new channel variable FOOBAR
  • The table app_coolapp now has a new column, widget_power. This stores the power of the widget.

And so on.

Ugh.

Let’s break that down a bit. When does the channel variable FOOBAR get set? Why? What are the possible values? Why should I care? Can I set the variable in the dialplan? If I do, what does the program do?

The database column. That’s handy. What is the datatype? If it is a string type (which it probably is), how large should it be? Is the schema change optional?

And so on.

This is nothing against any project mind you – or any developer for that matter. The fact of the matter is, writing good entries in a CHANGES file is hard.

  1. You have to remember to actually do it. Often we’re so excited about the feature – or so relieved that it’s done (or a combination thereof) – and that it finally got put up for code review – that the CHANGES file is an after thought. It’s one of the most common findings on a code review for a new feature.
  2. You have to be able to intelligently summarize what is often a complex idea. This is a life skill that a lot of people don’t have.
  3. You have to think like a user, not a programmer. Again, a life skill that takes effort.

Over the past few major versions, we’ve tried awfully hard to get better about updating the CHANGES file. Comparing some of the entries from various versions:

1.6.0

  * Added the jittertargetextra configuration option.

1.6.1

* Added two new dialplan functions from libspeex for audio gain control and
denoise, AGC() and DENOISE(). Both functions can be applied to the tx and
rx directions of a channel from the dialplan.

1.6.2

* Added support for setting the domain in the URI for caller of an
outbound call by using the SIPFROMDOMAIN channel variable.

1.8

* Added ‘R’ option to app_queue. This option stops moh and indicates ringing
to the caller when an Agent’s phone is ringing. This can be used to indicate
to the caller that their call is about to be picked up, which is nice when
one has been on hold for an extened period of time.

10

* Addition of the JITTERBUFFER dialplan function. This function allows
for jitterbuffering to occur on the read side of a channel. By using
this function conference applications such as ConfBridge and MeetMe can
have the rx streams jitterbuffered before conference mixing occurs.

11

* Added settings recordonfeature and recordofffeature. When receiving an INFO
request with a “Record:” header, this will turn the requested feature on/off.
Allowed values are ‘automon’, ‘automixmon’, and blank to disable. Note that
dynamic features must be enabled and configured properly on the requesting
channel for this to function properly.

Admittedly, some of these may be a tad cherry picked, particularly for the older versions. Hey, at least we have one. And at least we keep trying to improve.

Now: back to updating the CHANGES file for Asterisk 12.

Follow

Get every new post delivered to your Inbox.