Room 106 Thursday Mar. 05 - SCaLE 23x

This opening panel at Astrocon 2026, featuring Chris May from Sangoma and other key figures, introduces the event and discusses its focus on various topics related to Asterisk and FreePBX. The speakers highlight exciting advancements in voice AI and performance improvements within the Asterisk platform, emphasizing its role as an application framework for voice communication. Notable updates include performance enhancements, the integration of AI for speech recognition, and developments in emergency calling systems. The panel aims to engage attendees and encourage discussions about leveraging Asterisk's features in innovative ways. Scheduled talks will dive deeper into specific projects and community contributions surrounding Asterisk and FreePBX.

All right. Mike. >> Yep. >> Seems to be working. >> Whoa. I think we're here. >> Oh, it was dead for a second. >> Yeah. >> Good morning. >> Howdy. Thank you all for coming to Astrocon 2026 as part of scale the Southern California Linux Expo 23. our opening panel, myself, Chris May, I'm the open source solutions advocate at Sangoma and Mike Berdine, engineer on the asterisk

project and Mike White, VP of open source at Sangoma. Talk a little bit about our day and the event, the next couple of days, what those are going to look like. Everybody can see the slides, okay, on the screen. Not too bright. All right, let's go. >> Great. >> Well, that's so then, um, Josh Culp, uh, is with us virtually. Uh, you can feel free to ask

a question if it's, uh, you specific one to Josh. We'll we're going to loop him in, uh, when we have some Q&A at the end of this opening panel, as well as some other presentations that we have later on. So, this is our first time here at Scale. So, thank you to all the new, uh, attendees to Astrocon. We've been in Florida for a few years now.

Uh before that we're Georgia, uh Las Vegas, Arizona, a lot of places. Um sounds like we may have been here about 20 years ago. Uh so welcome, uh back to those who that's last time they attended Astrocon. Uh this myself, this is my fifth Astrocon now. And I think maybe Mike might have a few more Astracons under his belt than me. >> Yeah, the same. >> Well,

we're at five. And and I know Mike White has a few uh extras. So >> yeah, I think I'm around 13 at this point. Um anyone else been to Astrocon before. Maybe a quick show of hands to see who's been here. Welcome. Thank you for coming back. >> Cool. >> Forward. >> Very cool. >> So and and thanks again all the speakers who did come and as

well. So thank you to um >> Thank you. Yeah. Thank you to everybody. >> So there's a a pretty nice lineup of talks. Uh we're we're definitely going to be talking a lot about AI uh voice AI in particular as it relates to asterisk and all the cool new things and projects that people have been building with with asterisk integrated at the core for a lot of

uh speech to text text to speech. So we have also a few free PBX talks. Free PBX of course is the most widely deployed open source guey for asterisk uh the telefan toolkit. We uh have had a number of recent performance I'd say leaps and asterisk over the past few years or past year in particular. >> We can talk about those as well. Josh is going to

be giving a talk on that >> and we have of course a number of great stories about people who have been using asterisk in a variety of ways and some other uh projects that maybe are more tangentially related to asterisk as part of another solution. Just a bit of uh housekeeping details. This is being recorded as well as streamed live on YouTube through the scale YouTube channel.

So if you do click over to the live stream, you should be able to see each of the rooms. In particular, uh our room at 9, I think we're the first ones to get started. So hopefully uh everyone on the internet can throw a few questions in as well. Uh I know that the asterisk IRC uh channel has uh some discussion uh happening in regards to our

our live stream feed. So we're going to try to field a few questions maybe from the internet at large if we can. This is uh important though that we are streaming it live. So if you are speaking asking a question very important that you speak into the microphone or else it won't get picked up on the recording. We do have some uh cordless handheld mics that we're

going to walk around with. We do have Q&A so we can come right to you and you can speak into that and then we'll try to repeat the question as well. Whoever is speaking as a reminder to our speakers just to summarize quickly the question as it was asked so that we can make sure that it gets into the final recorded product that is put out. So

again live stream on YouTube but then cut into individual sections later on when the the clips are isolated into individual portions. So, we do have breaks scheduled. Of course, there's uh a bunch of snacks in the back. So, feel free to take advantage of those 15-minute breaks. Don't have to be in your chair the entire time. We do have some tables up front for those of you

that do need to open your laptop and not place on your lap the the whole time. Feel free to use those. Power is in every row, so you should be able to plug in and and charge. The uh local Wi-Fi, you should have access and information on your instructions. We're in a bit of a of a dead zone when it comes to cell signals. So, uh, you

know, be alert of that. But the Wi-Fi is pretty solid. They dropped a lot of APs in the area. Uh, lunch, we have a lunch break for for an hour at noon. The, uh, restrooms are out if you make a left out the exit and then immediate left uh, just right down the hall. There's also, uh, water in the back and there's water right outside. Uh if

you if you did get a t-shirt, please uh if you signed up ahead of AstroCon, please grab your t-shirt from the booth out front and get your badge scanned. Those uh who did attend pay the extra attendance fee for Astrocon, you're definitely getting a shirt. Uh everyone else, please try to wait until our booth on Saturday. Uh we will we will have a Sangoma booth over in

the exhibit hall, so you can pick up your shirt there if you didn't pre-register for Astrocon. We're trying to reserve those to paid attendees, but we do have u a few other uh pieces of swag as well. If you're a speaker, please make sure you grab a mug uh and and talk to the our desk about Like I said, there'll be u u a few other um

points to wear your t-shirt tomorrow for the group picture uh at the end of the day. So, and we're we're going to try to wrap up right around 2:00, which is when the exhibit hall opens. So if you, you know, are heading back later tomorrow, you might have a few minutes to visit some of the other sponsor booths over in the exhibit hall in the other building.

So some housekeeping. >> All right. So uh we'll start by uh some announce uh announcements for the Asteros project itself. Um uh version 23 released on October 15th, which is a standard release. Um we also saw uh the end of asterisk 18 getting uh its its updates um which you know was a very good release very widely adopted uh was a good standard for a while um

we've made a lot of changes uh for ARRI which we'll talk about um I have a presentation on that there's a couple other presentations on that mostly dealing with uh websockets and some other changes um there's also been a big push as uh Chris alluded to this year about performance right We're kind of pushing asterisk as being more of an application platform. So that means ARI, that

means performance, that means, you know, being able to handle loads beyond just a typical PBX. Um, and of course more um and we'll have more uh information as we go through. Um, there'll be a full project update uh given by Josh at the end um that we hope you all hang around for. Ah, free PBX news. Currently, two supported versions. Uh, 15 went EOL in October. So

that means we're on version 16 and 17. Uh this is a big uh deal for us. We're we're supporting two OSS underneath SNG7 for our version 16 of Free PBX and version 17 moved on to Debian 12. Uh we do have a few different ISOs that are coming out. We have reproducible build steps to make your own ISO from some Debian input ISOs. So that's been a

focus. Uh you ask where an ISO is and the tools are in your hands to make your own. So that's exciting. Uh we we have a uh version 18 milestone in GitHub. You can see a few of the tasks and issues that have been assigned over there for our planning with Debian 13 which was released Debian 13 was released last summer and we're are starting the process

of moving on to Debian 13 with free PBX 18 and the planning for that later this month into the beginning of of next month. So that uh is going to be covered in our big talk uh tomorrow. Um as well as sort of like a summary of asterisk, there'll also be a summary of what's new in free PBX including a lot of uh security advisories including uh

four last night. >> So that brings us to uh how is doing? Um it's doing great. Uh we have uh active participation across the forums. Um plenty of contributions. We're getting lots of issues coming in all the time. Um, our forums have over uh 8,700 new posts and 63 uh 630 new participants over the past year. Probably even a little bit higher as of uh as of

now. Over 1.7 million downloads. Um 1,800 uh or more commits across all the branches at this point. You know, we try to keep most of them current, but there are a couple that are uh different between those. And we had uh 67 individual contributors last year. That includes, you know, members of the team internally within Singoma and also the community members. >> Free PBX, how's it doing?

Uh, Tango, our free PBX mascot, Tango the Frog, is terrific. We, uh, have been tracking some forum site traffic here. Our community.freepbx.org site. So, we're a discoursebased site where a lot of discussion happens about free PBX as well as in our GitHub issues. Of course, this is a map sort of of our traffic. Uh can almost see a spike from automated crawlers and whatnot over the past

year as we occasionally are feeding in LLM with 20 plus years of content uh from previous mailing lists and other forums aggregated in this modern uh research. So it's a really great tool for training LLM. Still a lot of great Q&A regular posts. I just thought this graph was interesting to show that uh we are we are still increasing traffic and it's uh it's a lot of

high quality stuff in uh in in regards to the forums. So hopefully you get a chance if you don't uh we're we're all of course on the forums. Asterisk also has community forums so you can uh look us up and chat with us there. Uh there's a lot of different um threads, topics and and things that are actively discussed and new Q&A as well. So bit more

of that in the talk tomorrow for free PBX but we are definitely seeing an increase in traffic. So >> cool. >> So what's new in asterisk? Uh performance uh performance improvements. Again we mentioned this a couple of times. Uh the big one was switching to uh task pool and stasis and pjip. So it's a new way of handling uh the different multi you the different uh cued

and threaded options. Um we've got ARRI outbound websockets and HTTP requests over websockets. So this means your entire ARRI applications point can be over websocket. You don't have to have multiple different uh protocols. Um that includes of course the chain websocket channel driver which is mentioned here is great for AI and actually uh came came about through some of the feedback and things we had over some

studies we had done the year prior. Um, and I'll be giving a presentation on this later on on how to integrate those three pieces together to build uh your voice application or almost sort of an external module for Asterisk at that point. Um, and then uh the the Josh will be giving his uh project update at the end of Astrocon. We'll really be getting into detail on

it. Um and you know we always encourage you when you're looking at asterisk you know some things that have been around for a while um that you haven't considered you know in new use cases can be very useful in the future you know the obvious example that comes uh to mind is the external media which is you know how we had done some examples uh for AI

based assistance there before um but you know using chance spy and external media you can build you know something listening in on your calls and responding to your voice things like that so definitely look tools we already have and on top of the new ones we're adding for you. >> So, um I mentioned, excuse me, we have a lot of uh great knowledge based content as well.

The Sangoma wiki, uh this is an update that we had a lot of different wikis that contain a lot of different information about free PBX and ways to plug in and these have been kind of coalesed into a rag and that is accessible through the helps. website. So you can it's really great for basic questions like how do I connect a zip trunk to free PBX and

it will walk through and collect some different documents to show you a few different ways summarize that into some easy steps. So that kind of search AI assisted has been something that we've taken advantage of as as a company. So there's a lot of other areas where you can uh learn more about uh other Sangoma products through that interface but we direct a lot of people there.

Now it's also placed open internal support tickets and that kind of thing with Sango. So that uh is in in combination with better search tools for our forums and whatnot. So the uh we also have a number of um uh AI text to speech uh integrations. So, we have drop downs integrated, for example, in the free PBX interface where you can choose a different backend LLM to

type in your text that you want for your prompts and retrieve back the audio that then you can integrate into your IVR without having to leave the free PVX web interface and you can choose between a number of different providers uh based on your tokens. We also have some internal Sangoma tools that are integrated with that. So of course those things you know unless you're running a

local model you know would cost tokens or whatnot but that is now a little bit more fluid experience to really helpful like to speed up your IVR uh creation. Yeah Mike you have a thought >> and absolutely this is uh found in system recordings in free PBX. So if you're using free PBX um like Chris said there's the ability now to to select your LLM type in

your IVR message add that to the system and then apply it to whichever IVR you choose. Um, you know, as Chris mentioned also, it's built into our Scribe product. So, if you are using that for some of the sentiment analysis summary or uh other pieces like tagging that we're using, you can also use that for sound files in the IVR as well. So, pretty neat piece of

uh new tech that we've added in there and uh lots of folks have been been happy to use it. >> Yeah, hopefully everyone appreciates the uh the sparkle stars behind us there for, you know, that's the key like AI ingredient to have the sparkles. So um you know there's also been a lot of forum updates. Uh we had a pretty big overhaul of that. Uh talk about

a lot more of these free PBX things in the talk tomorrow. There are a couple of new repos going to talk about uh in as far as moving some of our uh actions in GitHub into separate repos. We had some integration for tagging issues. We've kind of expanded that modeled a lot after the asterisk CI actions. uh and then uh that is spent particularly for our new

weblate project which I'll talk about uh tomorrow as well and uh plenty of new modules in in free PBX of course uh open source that we'll we'll talk about some of those and a lot of other you know commercial developments that people have had in various places and so there's uh still a lot happening in uh in free PBX we'll talk about more cool >> so let's

talk about some upcoming stuff for asterisk uh we're going to have even further uh performance enhancements uh hints in particular Josh is working out at probably right now as he's uh as he's watching us. Um some ARI early br uh bridging improvements uh create and dial combined together to uh reduce some weird off-nominal uh situations. You know there can be times where between those two states where

can run into some odd stuff a little bit better. All right. Uh an ARI multiple channel variable get and set. Uh so you can do those together in one request instead of having to have multiple requests to get and set the different variables. Um and then uh an ARRI attachable state which would just basically attaching an arbitrary set of data uh to channels and bridges that you

can retrieve or set with those events. you think further improvements um to building your you know sort of off asterisk state machine via ARI >> and and can I just add that that ARRI is the asterisk rest interface >> so it's sort of a acronym and in an acronym >> yes rest within within rest um yeah and you'll see there's you'll see a lot a lot more

talk and focus on that over over the next two days cool so we're going to switch over to questions so give us a We're gonna head out into the microphones. Switch microphones over and we'll we'll come around for some questions. >> Do I need to switch >> and we switch off? Anything in live. >> Check. Check. Josh, >> check. Check. Check. >> Can you hear me? We're

listening for him. >> Josh, can you hear us? Yes, I can hear you. >> Hi. >> All right, first question. >> Yeah, let me introduce myself. So, I am head of engineering at Parlins Corporation. Um, we're based in Wuburn, Massachusetts. What we do is we build healthcare voice AI assistant and um my first time in Astricon. I I wanted to get to Astrocon last year in Florida,

but I just wasn't able to. I got sick. So, glad to be here. Um so coming into this conference one of the biggest questions that I had in my mind is architecturally when you look at asteris right it's a single threaded application that is really serving all the calls or maybe I'm wrong but I think what I would really like to understand from you is that is

there any plans in the road map to really sort of take a into a multi-threaded system or come up with ways so that it's able to really deal with a lot more volume of calls I would love to get your thoughts on Josh, do you want to step in or >> Oh, I could talk about this for ages. Uh, and I will in my performance talk at

the end of the day. So, please uh please come and view that. Um, asterisk itself is multi-threaded. Um, it is actually more multi-threaded than it used to be in years past, particularly our SIP stack. um media handling has always been multi-threaded and the performance improvements that I've been doing over the past while um have been able to push the ceiling of how many calls and such you

can do in asterisk even further. So um it it's it will continue to be as was said in the um presentation here uh it's we're going to keep pushing it as far as we can. Um so uh what I will say is um come to my performance talk and you'll get some better insights into that >> there. Thank you. >> Great. Any other questions from the group?

>> Everyone's an expert already. That's good to know. Don't be shy. Here we go. All right. Hi, I'm Jeff Lorsier. I'm the CTO of Stratos Talk. We're a hosted PBX company. Uh, one thing that I've been playing a lot with is ARRI. And for the first time in over a decade, I've managed to crash asterisk multiple times in a day. And I just wonder uh is that

something that's going to stop happening? >> Uh, if you file an issue with a back trace, then there's a better chance of >> Yeah, that's definitely a priority for us. So, Those those will make it to the top of the queue. >> Thank you. My name is Ed Nicholson. I'm with 0x1B and I have not deployed Asterisk in quite some time. This time I'm looking to deploy

it to a Kubernetes cluster. Is that something I do you have hints for that? I I can offer we had a great talk about this last year at Astrocon and it is up on our YouTube channel. Uh it was an extensive deployment with gosh it may have been 100,000 endpoints uh for a larger telco. So uh yeah, this is definitely deployed at scale in Kubernetes in a

number of areas and there's some great resources that we have available at our our separate asterisk YouTube channel not the not part of the scale channel but so recorded talk from last year. >> Once again don't be shy if anyone has a a quick question or any way we can help you out. Well, that was an easy one for you today, Josh. >> I mean, I'm not

done. I still got mine >> perfect. >> This is only half today. It's only half >> This is only half from Josh here today. >> All right. Well, back to you, gentlemen. >> Great. Well, I think we're um set. Uh maybe we have one more slide or I I guess it's just a summary of you know who we are and thanks Josh and and what uh you

know we'll be around um I guess for a couple of days like I mentioned we'll have a Sangoma booth as well in the exhibit hall so if we don't get a chance to catch up today or tomorrow during Astrocon you can come and ask us more questions there on Saturday and Sunday. Uh thank you again. Uh and it's going to take a couple minute break uh here

before I fire up. We do have a bit of a change in schedule at 9:30. So I need to boot up and then we'll present on dynamic location routing in regards to emergency calling. So topic I'm particularly interested in. Uh we we've had um some uh difficulties with uh speaking arrangements. I think you'll hear about that at every other track here. Uh uh so we are doing

the best we can, but I will be uh talking on that topic in just a couple of minutes. So, please uh grab some coffee, snacks in the back, and we'll get started at about uh 9:30 to 9:35. Thank you. >> Excellent. >> Good work, Chris. Good work, Mike. check. Check. Okay. Good morning again. Everybody hear me? Okay, great. Great. Thanks again for coming uh to Astrocon 2026.

My name is Chris May. I'm the open-source solutions advocate at Sangoma. This is my talk about dynamic location routing and other emergency calling ideas for accurate and prompt response. First, just wanted to start with a quick show of hands. Everyone who uh has anyone who's ever dialed 911 like is that Okay, good qu Yeah. Yeah. Okay, that's just about everybody. Great. Okay. And then um if that

was the last time that you made a voice telephone call, please raise your hand. Okay, there's no hands up. Great. So, I I heard that there's a lot of anxiety around making voice phone calls, but this is still one of those places where when you need help, it's very good that you can call and get it there quickly. My apologies to the uh Europeans in the audience.

I believe it's 211 and 112. Thank you. So that shows shows how much you know. And it's something that we it it's so common. We we kind of take it for it almost like a a default like an advantage. You know that we but we have to be conscious of how we configure our phone systems so that we are in compliance with all the new rules and

regulations and make sure that the fire truck and the ambulance know exactly where to go. So this is a very important part of my talk. Hopefully this motivates you to at the end review your dial plan, be comfortable with deploying these kinds of systems. The goal here is that you can say to yourself, I um of there it is. Probably should have turned off the screen locking.

just to get started here, I will advance my slides manually. So, first I'll talk a little bit about myself and then do some legislative review. Of course, this is a very regulated environment, so it's good to know what's happened. There's been some changes in the past 5 to 10 years that you should be aware of if you're deploying systems like this. So, it's a little educational opportunity

here. Um, I come up with an acronym not uh it maybe packle would look better as clap, but it's for uh parse, alert, call back, and locate. uh and you'll see why it's in that order in particular in just a bit. But look at some practical examples of doing this in asterisk and free PBX and some of our new features that we're moving forward with a PID

flow which is not a dance move but it is a new uh format for transferring location information that has got more popularity and look at a couple of examples of how to do it in free PBX. And if you do have a question, um, you know, feel free. We can run around with the mic and and grab it and interrupt in the middle, but do please ask

those into the microphones. So, about me, uh, this is, uh, the picture inside from last year's Astracon. So, this is still a popular meme. Uh, that is me at my desk previously and then kind of, uh, looking at it. So, everyone's familiar with the exhibit meme. I heard you all like Astracon. I've been uh deploying these systems for about 20 years independently before I join joined Sangoma

about a year and a half ago mostly in small medium businesses. So I had to deal a lot with these questions about emergency calling and how to configure these systems and especially keep a breast of all the new developments and regulations. So that was important to me. So and uh I just put an Italian spin on the end of it because the uh word astri uh means

means star in in Italian. So, legislative I uh first off, I am not a lawyer and this is not legal advice. Uh I refrained from inappropriate acronyms and just went with uh it. So, that's I'll I'll I'll feature that on several slides here as a bit of a disclaimer. Again, not a lawyer. Uh you should ask lawyers though for help. I like personally when I deployed systems

to start at the front desk. Legal kind of would come second. Sometimes it came first, but often when I'd walk in the building, the first person to answer the phone is there at the front desk. And they are the ones that if you want a satisfied customer and client, start there. Ask them questions, figure out how they do things, and have them help you test and buy

into the system. Uh, also, it's great if you have uh friends, firefighters, police officers, uh, ambulance drivers, ask them. Uh they have an organization called Nenina, the National Emergency Numbering Association. They have meetings and things too that talk about all these 911 developments and improvements. Kind of comes as a shock sometimes, but you know, you just think about situations where, you know, a large building and someone

dials 911 in a corner with hundreds of people and the fire truck rolls up to the front door and no one has any idea where the 911 call came from. And and that could that's a problem. So uh also private security uh in particular very interested in these kinds of solutions is when you're thinking about like how to fund a project like this that you're interested in

uh that that's their entire business model is responding quickly to emergencies before the public safety answering point gets a hold of it right so that is a group to consider tele of course the telephone company's been solving this problem for 50 years or more now so that's a great resource and then MLTS or PBX operators like yourselves or your future selves. So, this is another great place.

Um, there's been a lot of laws. Actually, it was interesting. 911 itself started in 1968, but it wasn't a federal thing until and mandated until 1999. And even then, it was, you know, sort of a a larger um, you know, question that when it made it through Congress, it wasn't unanimous. We'll talk about that in another couple slides, but these are just some key dates to think

about. Uh, this came to VoIP in 2008. very difficult to geollocate IP addresses. So, how do you get the, you know, firetruck in the right place? And then a couple of new laws that came out, like I said, about 5 to 10 years ago that really affect how we program a lot of PBX MLTS systems. That's a uh suit of ready fox. I I think when 911

was being rolled out for uh children, this was something that was used to popularize how to work with it. So, u you know, shout out to the fries in the audience if there's any. I don't see any right now, but maybe later. Um, so the the the 1999 bill, this really looked at landlines and and also with wireless phones, but you know, the tech was kind of

early. It it it basically the phone book, right? Like, you know, some people have made the the joke that the uh the phone company is a massive billing operation that provides telephone connectivity as an ancillary service. So, but they know where they are. they know where to send the bill. So when someone calls 911, they know where to send the fire truck, right? Like this is part

of these. So there's some terms like this that have been floated around that you may be familiar with. You know, we're not going to talk about these in details, but it what we we don't really consider these as much now in the VO world when we talk about this stuff like we have, you know, different terminology, but these are kind of some of the basics where and

it's still a huge part of the system. And then in uh 2008 it sort of moved to cover the wild west of VoIP and this was an issue where GE geograph the geography of your IP device very difficult of course any system in knows this. So how do we know how do we map that to physical addresses? You may have seen um you know in your cell

phone provider when you switch to Wi-Fi calling that you have to enter your address information. That's kind of a solution to this problem. Sometimes uh clients like the Sangoma talk line for example will switch over to your cellular network to make the 911 call but still a lot of problems you know with GPS in large buildings. How do you pinpoint what floor you're on? And this particular

law got something called NG 911 moving along which we're kind of going to you know merge and pick up speed and and talk about you know that at the end but this is some of the ball and got rolling here in this bill. Um and Nenah was a big part of that. This uh car's law came about. Tragic accident. Um again, you know, if you're bored by

this, you know, feel free. I find it interesting, but feel free to grab a snack. Uh this this was something that a a person didn't know that they need to dial 9 before 911 from the hotel, which personally I think terrible way to configure a phone system. Never do that. Uh that is just not something that even should be in the mindset like and this law addressed

that. So, it took a few years to get it passed and now it's mandated. So, on new systems, you need to make sure that direct dialing of 911 is allowed. So, we'll look at these a little bit more detail, but it also required that you notify people. So, someone needs to know that you dialed 911. The the example I mentioned in the front of the building that

if they don't know when the fire truck shows up who dialed 911, that's another problem that this bill tried to address. And then providing a call back number. So if the call drops, you you need to think about what happens from the 911 operator. They try to call you back and they need to be able to get through. So these are parse, alert, and call back. Talk

about those a lot. That's the PAC. And the first part, we'll look at each of these in detail. The parsing, right? So you need to make sure that your phone system, whatever phone system it is, needs to be able to dial 911 exactly like that. That's it. So you you can have a nine as well, but it it can't be required. So you have options. You can

do 9911. You could do 1911, 91911, whatever you like. Uh maybe you map 112 over to it if you have a bunch of European users. I mean, it's entirely, you know, possible that you have a dial plan like that. But the key part is that the 911 needs to go out as that from any phone that looks like a phone. So the alert now what does the

alert look like? Well, it needs to go somewhere where somebody can do something about it ideally, right? So a lot of times we see this take the form of an email notification to someone who can act on it, which is one way to go about it. If you have a actively monitored security station, maybe it's a screen pop on their Windows PC. If you have a text

message option with your provider, you could send out a page to people. And you know, with we're dealing with phone systems, uh so why not a phone call, right? Like I I think uh you know we'll get you know thinking and looking at a few more ideas here but hopefully it starts some you know uh marbles turning around where you can consider like well what if you

know you can get someone else at your building or your school or your hotel involved early on in the call. So I mean you have a resource like a private security officer that knows or front desk operator who knows exactly that layout of the building. the the the fire truck, they don't know the ambulance. They don't they don't know necessarily what's going on inside that building, but

you have local resources and maybe they can get on the call and talk to everybody. Like we we have conference calls, right? Like, you know, why it boggles me that it's just a one-way trip between the person calling 911 and the PAP operator when there's all these other people around who could potentially lend assistance in in seconds instead of waiting minutes for the truck to arrive. So

just some ideas to think about. Again, I am not a lawyer and this is not legal advice. There's the IT disclaimer in the bottom right corner. So just some ideas. Callback number. That's the CPAC parse alert call This is important because you the main problem is that you don't want to broadcast your main company number as the callback number. If that main company number goes to an

IVR or an AI or whatnot, it needs to go to a human being. Whatever number that that 911 operator sees, needs to be able to connect back to someone who can do something about it. So, you don't want the 911 operator calling your main number and having to press, you know, ex enter your extension number one, two. I have no idea what extension just called. You know,

I just see this phone number showed up. They need more information. So, you know, just imagine the situation where someone's in an office cubicle in a corner, incapacitated. They choked on something, God forbid, and then they try to talk and they can't. And your main number showed up and now how do you reach them? How do you even know? So, it happens. Um, you know, people that

you want to think about like with it doesn't have to be one single person either. This could be a group of phones. You know, we have ring groups for example in free PBX. So, you can call back a bunch of different people and one of them can answer and say, "All right, I didn't dial 911, but I know that like this area is somewhere I got to

start looking around. I'm going to get up and check to see what's going on." Like, you know, who called in my local area of the room 911. So, that's an idea how to maybe position your call back numbers. Now, the locate part. This is a couple of different parts here, but the key part is that you need to have something called a dispatchable location relayed to the

911 operator. So, this can't be at a multi-building campus, you know, just the street address with there's thousands of rooms to choose from. It needs to be specific where the ambulance can go, what room, what floor. Maybe it's a corner of a building. Maybe it's a specific building number if it's small within this campus, but you need to be able to locate the caller. That's the key

part. So, what this looks like in short form, these uh code words, code words, you know, uh building BLG, this is a common way to, you know, send this data across. We're going to look at how those fit into your SIP invites in in just a couple of minutes with the PID flow. So parse alert call backlocate. Now I I want to show a bit why this

is such a huge I I talked to somebody yesterday who was just starting MSP saw free PBX sign and said I need a PBX and I said do you I just I just put up the rollup banner and and they asked do you have you heard of free PBX I said and they said no and but they know they need one and like this is this a

huge opportunity with free PBX and the tools we've been putting in place to to tackle these kinds of issues in particular I'm not going to name names but I just added this slide so we can go through it together of a uh certain hotel uh sticker in the front door. I always like to look at these to see what I should do in case of emergency. And

and a couple things caught my eye. I don't want to break it down too much, but so first, if there's smoke or fire inside your room, you're asked, now mind you, this is by the door. This is taped on the door and your and your phone in the hotel is over by the bed, right? And the cord isn't I know the cord is not long enough. And

if you dial 911 and state your location, your name and address, you you're on the phone by the bed trying to read off the name and address that's printed on this sign because who knows what the name and ad you know the name of the hotel, but do do you always know the address? So now you're ripping the phone out of the wall. And then and then

after you're done with the 911 call, you're also supposed to call the hotel operator at zero, which shout out to all the operators. You know, who thought that dialing zero for the operator was dead? Well, not at a hotel, right? Everybody just picks up dial zero. So, the thought is like, what if what if somehow these things could be combined so that maybe the front desk knew

when the 911 call happened, right? So, and then the the fourth step, alert others in the area. Well, okay, here's a thought. All of these phones are in this hotel. What if what if they rang at the same time and played back some sort of automated message that there's a fire like you know mind-b blown so before you get to the fire alarm right it it uh

this is an unmet need uh parse alert call back locate there's still a lot to do here like I said these are newer laws past 5 to 10 years existing systems largely weren't affected this really is focused on newer systems except probably for the you know the the parsing of the 911 part you really need to you know get that done. But and then I I found

interesting a little bit further down now if you are ordered to leave the room and it's hot, you you do dial 911 and state your location, name and address, but then it asks you to call the hotel operator and and state your location. So I I was, you know, like, okay, I know that the first part was just call the hotel operator. So So my point is

that it doesn't seem like there was this sign was not uh I I did not help make this sign. I just took the picture. So there's, you know, definitely some room for improvement here. So ideas to think about. Uh QR code there. We have some step by step in and in free PBX how to do it. Uh a lot of these things. Um of course there's a

lot of ways to skin this cat being asteris free PBX open source software. Uh but we have some great resources there on the Sangoma knowledgebased wiki. So if you uh you know take a look at that code and read up a lot of this is some of these things are coming actually direct from that page. But now that we move into some of these ideas, so couple

of questions. Um, if you are configuring a phone system like this, think about how you've been laying it out. Um, and what you have been doing with your dial plan. Uh, you know, make sure you're not in a spot where 911 got put into some other use or that you have some other like, you know, definitely do not start your extension numbers at 910. um you know

those kinds of ideas like I I personally like to start around the 700s. It gives me a few options in the front of the menu for things and then make sure that I don't interfere with 911 at the end if I'm just you know doing like a standard you know 100 kind of phone system small medium business. Um and then if you need uh if you still

have analog lines like that's a different story because those are really kind of fixed like that's more of that earlier like MSAG stuff like phone book right those are really tied in that system's pretty solid from a different perspective. We're going to be looking at more like SIP in particular. And uh then you're using hot desking. That's something else to consider. Hotesking is the idea that you

can log into a phone wherever you happen to pick up a cubicle and that's your extension for the day. And then you go home and you come back the next day, you go to a different cubicle, now that's your extension. So that that's some different considerations, but there's allowance for it in the in the guey in the wiki. Um what does parsing look like in your 911?

And this is looking at the free PBX configs in particular. So just some basic steps. You're going to go into your connectivity outbound routes, add an outbound route, set it to emergency, and then uh choose the appropriate trunk your your telco in your trunk sequence for how these calls are going to go out. This is just a little snippet of the free PBX guey for those who

haven't seen it. We'll have a couple of pictures like this. um those dial plans I talked about earlier, you know, we're going to want to make sure that we, you know, include fixes for those errors. So, just kind of reiterate that that parsing is happening with the 9911. Also, you need to make sure you know 911 goes out. And then a great idea for testing uh is

933. So, this is something that uh I our wholesale carrier services, Sangomao, number of providers offer this where you can dial 933 and have it read back to you a verification of your 911 location address information. Not every telode does this, but it's an automated system. It'll say, you know, 123 Main Street in a very robotic voice. So, a couple of thoughts on uh parsing that outbound

route. This is sort of what it looks like in the free PBX guey. You can see uh here 911 the first uh in underneath dial pattern or the match patterns. Don't worry so much about the prepending or the caller ID matching. Those aren't you know going to be something you want to consider here. Leave those blank. Um but this is where you remove the prefix. So you

know pure asteris style plan there's you know you can do this kind of stuff as well but you know the free prefix guy makes it a little bit easier to see. So this is going to allow for example 1 911 uh 9 911 and 91 911 and instead though it'll strip off those leading digits and send it out as 911. So what does it look like step

by step user dials one of these numbers it's going to look at their emergency caller ID setting for their extension. So this is all in free PBX how the processing kind of happens just a flowchart of this. If they do have an emergency caller ID, then we're going to set that outgoing caller ID to the extensions emergency caller ID and that's what's going to get routed to

the the telco. And if we they don't have an use the outbound route caller ID. So, this can get a little confusing if you haven't. But again, test test test. Dial 933, you know, see what goes out and what gets sent. This is how to do it in free PBX. Now, we have a question about there's a couple other things to consider if we're talking about forcing

a trunk caller ID or overwriting extensions. So, you still have a few other things that not as common settings, but the two main things to look at are whether or not you have that emergency caller ID and if you do, then that's what gets used and if you don't, then the outbound route caller ID subject to a couple of caveats. So some alert notifications. These are all

things uh configurable in in free PBX. uh shameless self plug for my GitHub which worked on uh some uh public domain always be conferencing some of those ideas about conferencing. It's uh in AAL the asterisk extension language. So it's not standard comp format. uh it it's a little different to look at but uh talks about some of these ways to to solve these problems using conference calls

and other ways to notify and of course free PBX offers a number of these tools now integrated as well um in particular the open- source paging module so that means that you can drop the uh the caller into and this we're going to focus on this one into an outbound group where you can broadcast the 911 call to a bunch of phones. So they can actually listen

in if this is duplex is set to no. Uh this is straight out of our Sang knowledgebased wiki here. So they can hear what's happening uh on the call which can be very useful uh in certain situations to immediately let people know not only you know uh like that a 911 call was dialed but now you're also like playing back the actual content of the 911 call

to others who can run over there and do something about it. Right? So this is one way that might be appropriate for your environment to configure it. Walks through a couple of steps there. So this is uh what that looks like. Setting up a paging group, calling a 911 notify, including some extensions in there. Um so going through some of the steps. Again, this all be, you

know, online later. Um under your additional settings in that outbound route, you'd select for notifications that that particular paging group. So that's uh covered on the Sangoma wiki as well. Uh looking at version 15, I mentioned that's EOL, but you know this I believe um was the it was either 14 or 15 was the first version to kind of allow some of this. I think it was

15, but we're the current version is 17. So all right, a little bit of planning now on the call back. This is that question of you know what happens if the call gets dropped? So the line's cut, person can't talk, what have you. Very clearly I I mentioned this before, don't send them to an IVR. make sure that they're going somewhere where a human being is going

to handle that call back. It might not be the case that everyone in your building has a phone number. Well, that's okay. Like, you can still have a ring group and and assign some DIDs just for this purpose to be used in this way. So, we're going to look at that one a little bit more. Um, a three- ring group example of what this looks like for

a callback. So kind of a ven diagram to consider. So let's say we have a couple of phones in the lobby. Uh front desk and security 70172 that's their callback number. Now let's say we have floor one. We have uh three more phones 3, four and five. Those are in the ring group as well as the front desk and security. And then we're looking at floor two

uh which has another four extensions and in their ring group they're also in the ring group for callbacks with 701 and 702. So you can see that front desk and security are in all three ring groups. So you advertise one of these three DIDs based on what extension number in the box is calling 911. And when the call drops that call back number gets called back all

of those phones will ring. So that's, you know, one way to keep a lot of people in the loop. So maybe it's more important that you don't call back everyone on floor one, floor two. You know, you just want to focus on those primary front desk and security and then these other floors, but that's that might not be granular enough, right? Like floor two is maybe a

really big space and and what happens, you know, these rooms are far apart. So how do we know, you know, exactly who called? like we we're still relying on this call back number to patch that call through from the pap. So here's some thoughts on you know the how we do this locate showing this previous diagram. So what if you did buy a phone number for every

extension for 91 and this is a perfectly reasonable strategy, right? Like you can buy uh single trunking service retail. Every single phone gets a phone number, gets a trunk and you just stuff this into the address line too and it says building in there and problem solved. you can go on like your phone bill is going to be a little higher if that makes sense because maybe

you've got a bunch of power users that you know you this this is a reasonable strategy for you perfect it'll work right but it can get expensive if you're talking a school limited funds you know you're looking at perhaps you know many many different buildings this is kind of some of the discussions I've had with uh a shout out to some of our carrier uh services team

in in Sangoma this is something that they have had to deal with and how do we solve this question. So um uh if you need to chat on that at that level another Mike not up here but uh today Mike Dazi is great on this um he has a lot of you know uh lot of my testing that we're going to be looking at was using our

own uh VOIPE innovations our wholesale carrier services stuff from Sangoma to test this and get this stuff working. What I'm going to show now with our PID flow and what we've been you know generally calling dynamic location routing. So, there's a few questions like, and this is why it's important to check with your SIP provider. We do have that option. Not all SIP providers are going to

allow PID flow. What you don't want to do is stuff your invite full of PID flow and have your router fragment it because it's too large and it does I mean, that's the worst case scenario, right? we're sending UDP and it gets split up on an emergency call and or if you know your your carrier looks at it and says that's an invalid I'm I'm scrubbing every

packet you know down to make sure that that doesn't contain stuff I don't know about uh because it could be something and I'm just going to delete it worst you know so so you got to check with that make sure your carrier does this and this is something why I mentioned specifically the the Sang wholesale carriers that this is I got a test account for free I

work at Sang right so um then you know you want to make sure that you're using a modern version of of asterisk and and in particular PJ SIP and free PBX you you don't need to um you know have the very latest but you know current is good and then uh this this particular example involves a little bit of asterisk config file so appropriate asterisk conversation uh

for Astrocon I did give a talk about it a little bit at a free PBX uh world summit last summer but uh this there is a module in the works But um you know a lot of different ideas on on how to do that. So this is also something where you know your extension map very important to keep track uh where are your extensions right and so

uh link layer discovery protocol has some extensions where you you can configure your switches to relay information to the phone so they can send it out in the invite but like that's not probably there in a lot of places just yet. So we're not going to talk about that today. Now I mentioned pitflow is not a a new uh type of electronic dance music. Uh it's the

presence of information data format location object. It's been in asterisk for several years. Uh you may hear about it geoloccation uh dynamic location routing. So this is just narrowing down our locations in an XML format in the invite and elsewhere. And you can use a few different ways to do this. Uh, for example, we're not going to talk about it much, but for wireless, you know, GPS,

although I I mean GPS is only so good, right? Because in in a large building, um, you know, if you think about it, like right now we're in a cellular dead zone, but we've got, you know, we do have Wi-Fi, but we have wired network here. I mean, a a phone in a physical building, it's 10 stories tall and underground or something like that. I mean, can

be programmed to know exactly where it is and relay that information more accurately like every time. So, like I I mean I I think picking up a quote unquote landline or physical SIP phone that's tied into location is a far better choice than trying to dial 911 and in a in a GPS sketchy environment like that. So, a lot of different um you know, we're going to

talk about the civic address, a lot of different lot of different RFC's with uh different bits of information. Uh I know George, one of the asterisk engineers, worked a lot on this stuff and there's definitely a lot of uh things that are fleshed out in some that aren't in others. This has been a big talk in uh in in Europe in particular. They're also moving forward and

actually at the carrier level mandating some of these things before like the actual uh you know the rest of the network is ready. So it's a process around the world as part of this NG 911 stuff and it's probably going to be rolling out globally. So this is a good idea for our worldwide audience of free PvX and Asterisk users, right? A lot of good docs and

the internal uh Asterisk documentation site too from that link at the bottom. We're going to look at the civic address in particular. This is what it looks like uh in the XML format. Um you know, in particular the location portion of it here. Uh so hopefully that makes sense and what each of those stand for. Uh and you you uh you want to see like in particular

this LOC, this location, this is this is this detailed uh like room number, suite number, particular corner where the ambulance is going to go. So, we're going to focus on that and with a couple of practical dynamic location routing examples. So, there's two different ways to look at this. Your room numbers match your extension numbers. Anybody show hands who sets that that up that way? Room numbers

match extension numbers. Oh, gosh. All right. Thanks. Thanks, Mike. Yeah. Okay. Yeah. Obligatory. All right. Well, great job. um if you do that uh if not then here's an idea for overloading your caller ID name with um some variable location details and so again this is uh you know focus on some free PBX and how to extend your asterisk configs to do this um a few common

steps to either scenario so you're going to need to log in to your system set up a couple of files uh for geoloccation over SSH that you're then going to go back into the edit with the free PBX guey this uh geoloccation comp file I'm extend this. Um, so in free PBX, whenever you need to do something special, it's a good idea to put it in a

file that a configuration file that ends with underscorecustom.com that gets picked up and loaded in um by free PBX. Then once you have one of these files, you can go into the configuration file and are now back into the free PBX guy and go down to that file and edit your extensions_custom.com. So do a lot of work in there. Now, this is where uh you know, we

get into the you'll see the red part here on your carrier. So, some of these names you're going to want to change uh to adapt to what the name of your your SIP trunk carrier is. But, a couple of uh options. This is our carrier office in Pittsburgh. So, I I configured that in. Now, you're not actually editing raw XML. This is asterisk. So, we're not gonna

ask you to edit raw XML. That's going to happen for you automatically. You know, the packets go out, right? So you're still eding you're editing a comp file um you know so that can be used by some classy Sangoma phones here and that uh you can see the details kind of just looks like a mailing address on the bottom a little different format um how to break

up that address and now this is uh the old uh Digim office which uh related uh if you're interested in free PBX learning more we have some training coming up uh in the spring uh at this locations where a lot of hardware and is is uh manufactured and and is so there's one of our our boxes with a 16port T1 card just came out uh a couple

years ago so not too long. So that's pretty wild. Um you get I have a breakout, you know, panel for it and then it would go in that box. You can kind of see it wants to make its way in. So that's what that address looks like. Uh and then you kind of glue these together with your location and these these profiles. And I just called it

CO for the carrier office and DO for the Digim office and a few other these are like the minimal minimal requirements you know for this config. And that wraps up the geoloccation custom. Now you're going to go back into another custom file which is already there. You didn't have to touch this one because this is just part of the free PBX. And this is a way to

extend your PJ SIP config. So asterisk uh configuration very flexible but you can uh just um sort of it's a templated structure right so you can add in a few things so I've already set up in my free PBX vip in y you know my unique numbers so your zip trunks I have a couple of zip trunks and that config was generated in free PBX and I'm

just extending it a little bit right by gluing in that particular geoloccation profile and and how I'm going to connect those to the phones. And I want to say prefer incoming means that the extension 701 I'm going to use what it's configured to do. So there's a couple different options there about how you can handle that. Um you could override what the phone is sending, right? You

can force it in a in a different way, but here we're we're gluing it in with the phone's endpoint. Nothing to do on the phone itself really. This is in the again in the asterisk configuration in the free PBX friendly version of asterisk configuration. And uh there's a key part here. If you are uh if you have any parts missing or you disable a trunk um then

you're going to have a bad time. So you know careful when you do this. This is the last couple of slides here about wrap up. Um this is again scenario one. This is room numbers, match extension numbers, which doesn't seem like anybody's doing. Uh, but this is what you you you look at the key line here. Uh, third from the bottom, we talked about that loc. That's

what's going to change that little bit of XML that goes out by gluing it into and and and adding the word room in there. And that is what then gets relayed on. And when you verify your address with a 933, you'll hear back uh 123 Main Street, room 103. Pittsburgh, PA. And second example, this is where we're kind of overloading the caller ID name. So what we've

done here is in the free PBX extensions in the caller ID name in the guey, you actually typed in the room information and that is then so it could be maybe it's sweet 567, but it's extension one two three. So in the caller ID name you have something else written and that is what gets sent then as part of this you know information. So here um you

know how you configure a caller ID name if it's not that important um or you can extend it. So I you know I'm thinking more of like a K12 like type situation school right like it doesn't really matter what the name so much is on the extension for a classroom. I just need to know that it's room. In that caller ID name, I might have a four-digit

extension, but the room number is is one, two, three, and that's the key part in the caller ID name that I want to that I want to send out, right? Is the room number because it's it's but I'll also include the extension number at the end there. So, I'm getting like both parts that information and that's what gets sent in the XML, you know, out your out

your trunk. So, that is the end of my presentation. Right on time. Uh if there's any questions, uh happy to uh run around and ask those. But I I hope that this gives you enough um insight and thoughts to make sure that you know you're again your your dial plan is is sending out 911. Exactly. No nine required. Uh and that you have a few other ideas

about how to go about this and and know that this is a huge area for uh you know innovation and expansion and uh help with uh for help for a lot of people. So thanks again um for coming to this talk about dynamic location routing. Throwing it at the last minute but uh happy uh to answer any questions. Okay seeing none we will um proceed according to

our schedule. I believe we have a 15minute break scheduled uh before is that correct? for Alex's talk, right? Okay. All right. So, are you ready to start in 15? Okay, great. All right. Uh, thank you so much. Uh, again, appreciate it. And we'll see you back here in about uh 10 10 15 minutes. Check, check, check, check, check, check. Does it work? >> Does it? >> You

do. I don't hear me. >> Oh, okay. Too Check. >> Is it? Um, is this too quiet? Well, this might be a matter of where I put the mic. Check. One, two. Check. One, two. Try it. Check. One, two. Good morning again everyone. Woo. Uh my name is Chris May, open source solutions advocate at Sangoma. Uh besides opening panel and a prompt impromptu presentation, I will be

mceing uh much of the event. Just trying to give a little lead in to folks who are here to present all these wonderful things that they're doing with asterisk and free PBX. Uh last talk about 911 and dynamic location routing. you know, looking at some history and legislative review and where we're at with that system. Uh, this is another great talk from Alex Bellish. He is uh

talking about some history of the PSTN, the public switch telefan network. So, if you've ever made a phone call, you've used the PSTN. Uh, I guess unless you've purely used Wi-Fi calling to call other users over Wi-Fi, then maybe you've avoided it. But, uh, the corner case there. So, uh, Alex is a Chameleio developer, uh, and and contributor, speaks at a number of open- source conferences for

a number of years, has been a valued contributor and member of Astrocon. We're happy that he is back to talk about the PSTN core. Alex, >> hello. Yes. Uh, thank you for that intro. So, as I mentioned, uh, I specialize in SIP service delivery platform architecture and call routing for VoIP service providers. So I'm actually on the carrier and PSTN facing side of all this which makes

me a little bit unusual because a lot of times I'm dealing with things that customers do you know higher up the application stack where all the value is created and I feel myself quite distant from all that. Um but um I've been doing this for almost 20 years and I actually got my start in the small ISP world uh back when that was a thing and I

got interested in voice and telefan at kind of an opportune time in the middle of the 2000s when asterisk was just breaking through in terms of mind share and open source voipe was having its uh first moment. Um, so you know, it's funny because I actually got into asterisk in 2005 or so and I thought all this SIP stuff was a little hokey and wouldn't really take

off apart from the handsets and I was very interested in TDM and analog line interface hardware for asterisk and I thought that you know TDM ISDN SS7 that was the future and that was the really serious stuff and went to work for Seellex and Vo companies a lot of legacy infrastructure for a while to kind of invest in that viewpoint. point and of course by 2008 or

2009 I had to sort of change my tune uh and reverse course as SIP you know the as its role expanded inside the carrier world uh first inside carrier networks and later between them. you know in this my purpose in saying all this is that it kind of brings us uh in a I guess characteristically way to uh the real point of this talk which is that

you know we think I think a lot of folks who do VOIPE they think of SIP providers as throwing calls over the fence to an obscure and kind of secretive and predominantly SS7based PSTN in one sense and you know that continues to be true but just about as soon SIP hit any kind of critical mass in the early mid two 2000s. I remember there were a lot

of triumphant proclamations of the IP peering revolution that the PSTN is dead. Long live the PSTN. You don't need it anymore. You could just send all your calls over IP now. And those of you who have been around long enough to remember, you know, how this started out, remember that >> Alex, sorry, quick check. Are you advancing slides right now? >> Uh, I am not. >> Very

well. Sorry. My my my apologies. >> but I could be. Um, so what I was just going to say there is that, you know, you remember Enum and E164.org and well-intentioned efforts like that, but it wasn't long before you had people with more kind of filthy Lucer in mind who were racing to proclaim that the PSTN was over. And of course, you know, naturally the new IP

peering fabric and highdeinition voice was happening at, you know, their so-called I, you know, peering clearing house or IP tandem or whatever. But it just wasn't taking. I actually, you know, made a lot of fun of them in those years. And, you know, year after year, this was going to be the year of SIP to sip peering in the way it was going to be the year

of the Linux desktop or whatnot. And I hey, I I did desktop Linux for 23 years. I ran six dros. I've been doing it since 97. So, I earned my right to make that um take that shot even at scale. Um but um you know the basic problem with all this IP pering stuff was um combination of regulatory and commercial and engineering encumbrances and then just the

the critical mass and network effect reality that you know all your all your Facebook friends aren't going to come with you to your new non-f Facebook uh social network even though Facebook is so in shitified but um you know, things do change and when they change they typically change in kind of discontinuous step functions and I think in the last 10 years that's really happened in the

PSTM and that's what I'm really here to talk about. So just want to take you back to the moment when the AT&T monopoly was broken up uh that you know before 1982 or so AT&T was a nationwide near monopoly and it served 81% of subscribers. Uh the remainder were served by what we uh affectionately call local yals who remained interconnected with uh AT&T as necessary. And the

Reagan administration broke up AT&T into seven regional bell operating companies or box often called baby bells. And the idea here was that the box would maintain a monopoly on what we call local loop services within their region but that the inter that there would be con competition introduced into the long distance portion which is the long-distance calls that flow between these operating regions. Uh so box could

not provide interlata services while interchange carriers could not provide intraata services. We're going to get to the concept of LATA here in a minute, but the concept of LATA, local access and transport area. Raise your hand if you've heard of it. Okay. Um, it emerged out of this breakup to delineate local loop versus longdistance services for the purpose of affectuating the divevestature. So, a lot of people

think I used to think that this was a concept AT&T invented, and that's partly true, but it it was previously vertically integrated to a degree that didn't require that. Instead, laddas were something that arose precisely when the government went to AT&T and said, "Well, please provide a plan to break up yourself." And so the essential effect of this this uh breakup of AT&T which most telecom people

refer to as either divevestature or the MFJ the modification of final judgment due to the changes to the plan that had to be made before the final ruling in 1984 was first and foremost the replacement of the national monopoly of AT&T with functionally southern regional monopolies. And um you remember what these are at least if you're of a certain age. uh Ameritech, Bell Atlantic, Bell South, Ninx,

Pacific, Tellesus, Southwest Bell, US West. Um the second effect that was more salient to the anti- monopoly mission of the time was to introduce competition into the long-distance market and to prevent elements of the old AT&T, that is the baby bells, from leveraging their vertical integration in the long-distance market. They were permitted to maintain their monopoly on local services within their region, but that required defining exactly

what local means. Um, just as a side note, you know, some of the reconsolidation that's happened since then is really confusing. So, like for example, I live in the southeast of the US and, you know, Bell South was bought by AT&T in 2006, which I think a lot of people believe was the original AT&T monopoly, reasserting itself and reconstituting itself, but that's not really true. Um, what

actually happened is that Southwestern Bell or SBC Communications, its underlying holding company, bought out the carcass of AT&T and the late 90s, renamed itself to AT&T since it's bought it, you know, it bought its brand and trademarks, of course, and then uh, you know, psychopathically wore its skin and presented itself out to the world as AT&T and that's what acquired Bell South. Um, so I'm sure some

of you know having been around telecom for a while, but the arc of re, you know, the rest of the mergers and reconsolidation since then has led to a blizzard of confusion. Uh, you know, Bell Atlantic was merged with 9X and GTE and became Verizon and US West was acquired by Quest. Quest was acquired by Centry Link, Centry Link rebranded as Lumen. Lumen later also bought Level

3. Um and of course you know SBC renamed itself to AT&T bought Bell South and lots of other things and so on and so forth. the local access and transport area that came out of divevestature um is a geographic or administrative area that defined what it meant to provide local wline services as opposed to long-distance services loosely put. And this is a these are some excerpts of

a US-wide LATA map. So for example, in my home state of Georgia, you find there's about six laddas. They do not cleanly observe state boundaries as you note uh as you might note. And um loosely they can be said to do so. But the idea here was that if you are a baby bell and you are a a call or increasingly data circuits and other services are

originating in the same lad as the one to which they are terminating then you are providing intra latta services and that is within your remit as an as a baby bell. But if those services are originating in one ladder and terminating in another, they become what are called interla services and you needed a special type of regulatory classification known as an inter exchange carrier orc to provide

this and you know that was I'm sure you remember your you know MCI and sprint commercials from the 90s and late 80s and stuff like that that you're you're thinking along the right lines if you're um thinking about exes that way. So the architecture of the bell system after divevestature looks kind of like this and I'm not sure how visible that diagram really is on that screen.

Um unfortunately I can't really don't have a reasonable way of making it any bigger but it was a hierarchical hub and spoke structure which is broadly true of telephone networks just about everywhere. So, there were two types of C central offices, which is what they're called in America, but they're usually called telephone exchanges everywhere else on the planet. Um, one was called an end office, which had

a subscriberfacing switch that served actual people with actual telephone lines at home. So, you know, your copper lines ran into that switch and either got punched down into a main distribution frame or, you know, were demultiplexed off of some sort of TDM aggregate. Not necessarily TDM aggregate. And you know when you called your grandma in the same town that you lived in that just got switched by

that switch into that switch out of that switch. But if the subscribers were calling not necessarily each other in that same exact rate center but a someone else who lived in the same ladder then that was going to be served by a different end office switch. And in order for those switches to communicate they didn't just all have lines to each other and some sort of full

mesh. they had a two two tiers of switching. They had kind of a end office switch then they had a switch of switches and that switch of switches was called a tandem. Uh so when one end office switch needed to talk to another end office switch they would just go through the tandem. However, if the traffic needed to leave the ladder it would jump on an inter

exchange carrier orc link. And one of the ideas of divevestature was to allow you know competition in this space. you could choose which hauled your traffic to another ladder. And this is a world where there are many different names for the same thing. If you've been around telefan long enough to remember when end offices were called central offices, um you know, subscriber switches, class 5 switches, class

5 offices, and so on. Tandemss were called toll offices, class 4 switches, class 4 offices, and many other things. Some of them savory. Um so I'm going to stick to end office and subscriberf facing switch here for the end office and tandem for intrata and exec switching. Um this tendency to have lots of extends to other parts of the system as well. Um at first the baby

bells were called box then they came to be known as arbox. I knew plenty of people in my managed modem days who called them reebox like the shoe. Um and this tendency is pervasive in this world. Um so this diagram here deliberately shows two tandemss in one ladam and one tandem in another lad which illustrates the variation in how tandemss were designed. Some lattas had one tandem.

The lad I live in 438 which covers the midsection of Georgia has three tandemss which of course have you know big fat interconnections to each other. It's also fairly common for buildings that have a class 4 and a class 5 switch or that is to say a tandem and an end office to be in the same building. Um that's seems almost universal in some ways around the

mid80s you know pointto-point T1s frame relay ATM links stuff like that came to be more significant as well and they followed a similar logic in terms of their design through the physical elements of this system but that would be a whole presentation in itself. that's how the system worked and nothing really very substantial changed until the telecommunications act of 96 which I'm sure most of us have

heard of that was uh undertaken in the Clinton years and it represented a very very large structural change to the telecommunications architecture of the US. Um it's there are absolute volumes and reams that one could say about all the different spheres that its tentacles touched. But for purposes of this discussion, the main thing that it did was it opened up intraata services to competition by requiring the

incumbents in that latter that is the bells that were already there to interconnect with a new category of entity called a competitive local exchange carrier or select. I'm imagining most have heard of the term select here. Yeah. Okay. Um, it mandated lots of other things that would go into introducing an element like that into Intel data services. Number portability, very very complicated reciprocal compensation structures wherein operators

pay each other to take their their calls onto their network. Um, various elements of the Bell system that had to be unbundled for resale. Um that is to say that they were regulatory obligated to allow competitors to potentially sell parts of their network both retail services and the physical componentry. Um lots of implications for the broadband world as well which is where I started So in essence

a celic was an entirely new kind of company. Obviously, the incumbent carrier, the Ilic, which for most people was the baby bell, had the phone lines and the switches already in the area. So, it was the idea wasn't really that a CEC would start building a parallel duplicate of this from day one running along the same rights of way or whatever. Instead, the idea was that Selex

could offer new kinds of services over diversified network architectures and kind of overlay this. And the main requirement as far as seellexs were concerned was that the incumbent interconnect with them. So in other words, if you were operating for example in the midsection of Georgia where I live in LA 438, you wanted to start a select and you went through the process, Bell South had to interconnect

with you. Now the exact terms and conditions were to a very large extent determined by them, but they had to interconnect with you. This was mandatory. So the way this would work in practice is that you would go to the eyeick and you would say I'm going to I'm going to simplify here a little bit and say I'm going to pay half the cost of this fiber

to go to this location that it in practice it was usually an end office known as a POI or point of interconnect and they have to pay the other half and as part of this kind of cost sharing arrangement you would physically plug into one of their end offices but your goal wasn't really to talk to an endoff switch your goal was to talk to a tandem

since the tandem is the the heart of the beast, the heart of this whole nexus. So you're you would physically have trunks that would or physically have a link that would go to a end office POI. But what you really were after logically was tandem connections. And so you would buy tandem trunks through the incumbent's network and they would drag your traffic over their network from your

POI and drag it back to their tandem for you. um you would have to pay them for this privilege and the anatomy of that cost structure is extremely complex and varied. Um so selex were also lex that's the le part of select so they were local exchange carriers so the scope of their operation was ostensibly latter by latter as well although the to a much lesser extent

than the baby bells which were explicitly prohibited from uh offering inter exchange services. So that's a lot. So let me offer an amended version of that diagram. Select interconnects in the I like tandem. So here I've zoomed in on one of the laddas in my previous diagram. Uh and you can kind of see that there are two new actors here represented by orange links and you can

see that they I know I'm probably spending a lot of time on the physical aspect of how they connected but really their goal is to go to the two death stars in the middle and those are the tandemss. And so you would buy tandem links to connect to the tandemss of the ILC in your latter. And if you wanted to operate in multiple laddas, you would do

it for every ladder. And this was expensive, difficult, and arduous. Um, there's a whole another aside here about that as a matter of historical interest about whether you actually have to connect to one tandem or all the tandemss that might be present in your ladder. And if you had to go around to if you could connect to one tandem and have the bell drag around the traffic

to its other tandemss, that was known as multi-tandom access and was allowed in some places and not others. But anyway, most of the time you needed to buy your own tandem trunks from the eye. That is to say, reserve a portion of their network capacity to haul your traffic to all their tandemss in that lad. So after TA96, you could argue that the role of the Bell

or Eye Tandem as a switch of switches became even more central in some ways because they weren't just switching the the ELIC switches, they're switching everybody's switches, burgeoning wireless operators, paging companies, long-distance companies, and now the local Selex, too. So the tandemss are where the EXC trunks landed, bringing in calls from other laddas. There were places where intratic companies outside the baby bell's footprint the local yokos

were interconnected with the baby bell as well and of course you know that's where the selex went as well. More importantly as selex traffic footprint got bigger and bigger there were also places where selex exchanged traffic with each other. So in all cases the upstream I like tandem was the default gateway to the world the default route the switch of switches the plexus the nexus the end

all beall and suddenly there was a lot more intercarrier traffic to move. So what did Selex actually do? They did a lot of different things. Um in the 90s you may remember they sold outsourced managed modem to uh internet providers. They did some broadband buildout in the sense that they built fiber rings and especially in metro areas got you know leased fiber or some cases trenched fiber

and got various office buildings connected to it and then could offer services to tenants on the 17th floor off their ring stuff like that. There was a brief opportunity to resell the own uh retail services i.e dial tone through an arrangement called UNIP that was required by TA96, but that didn't last very long. But I think for purposes of this discussion, the element of what Selex did

that we really need to focus on is that they became the back end of the VOIPE industry. That is to say, they started to provide wholesale PSTN origination and termination to the VOIPE Um, and I'm going to tell you why. Um, one thing to consider is that just to go back to the LATA idea for a second, one does not simply become a nationwide select. Um, this

wasn't LA by LATA process and it's very complex. Um, interconnection agreements are often hundreds of pages and laboriously spell out all kinds of intercarrier compensation mechanisms and exactly who pays for every inch of what in this equation at the latter level. Although from an administrative point of view, it's often simplest to opt into an existing interconnection agreement of someone that's someone else already has filed. That's done

all the time. And there's al there's what like around 200 laddas nationwide. So one does not simply become a nationwide select. So by volume most selects narrowly specialized to offer stuff in their local areas which was sometimes called you know wholesale origination or termination fairly early on. There were definitely folks who got a particularly cute ICA with their baby bell in their particular area and specialized in

offering wholesale termination or origination just there. And some of you may be familiar with some of the arbitrage and intercon and intercarrier compensation games that were played in this era. But some companies methodically obtained select status in more and more laddas and eventually in just about every latter to the point where they could offer inbound and outbound termination just about anywhere in the country. This was in

a way that's often understated and underappreciated, often driven by investments during the managed modem era. Uh when you had to offer dialup numbers just about everywhere and the you probably know that back in the days of net zero and various dialup providers in the 90s and this explosion of cheap dialup service, it was often level three managed modem that was the underlying uh facilities provider for that.

though the ISP would slap its label on it, but you were dialing into level 3's modems. And this is how they grew a lot of their network. So the appearance of VoIP in the early 2000s provided this type of managed modem oriented select with a new opportunity because if you think about it, dial-up aggregation hardware is just a bunch of DSPs in a box that would take

typically take a large TDM link and converting PCM audio from TDM frames to IP packets is also kind of a DSP workload. So there's a lot of fungibility here to the point where some of the same boxes were often used to do this. So not a lot of people know this, but the the reason level 3 had such a comprehensive nationwide origination footprint back in the day

to where if you were doing VOIPE in the 2000s and you were thinking about the DID anywhere company, it was probably level three or you know some reseller of level three like Bandwidth. Um it was because they leveraged their managed modem network into that. Um, it's not an accident that Bandwidth started out as a level three reseller before they got their own select first in the highest

volume, most lucrative metro markets and later in other kinds of markets. Um, so that that's not too different to the story of a lot of other nationwide facilities based wholesale providers that we know that are in our vocabulary So for most of the 2000s and 2010s, VoIP providers bought DIDs, PSTN origination and outbound services that is PSTN termination from underlying carriers which were large selexs. Some depending

on the scale and size of what was going on, it might be hyper local, but chances are it was probably regional or national or from a reseller who sold the services of underlying carriers. So this led to a lot of specialization where businessto business interfaces and processes were made to be consumable to this industry of ITSPs you know number porting uh you know CNAM all that kind

of stuff. Um I were not interested and are not interested in selling services to the VOIPE industry by and large and to some extent are regulatory hamburger from doing so as well. So this meant that translating VOIPE calls to and from the PSTN was an opportunity left entirely to Selex. So as I said the PRA in practice being a VOIPE business in this era meant at some

level buying the services of an underlying carrier or reselling the services of someone who was selling the services of an underlying carrier. You were a reseller of Seelic footprint unless you happen to be vertically integrated with a Celick. Um so underlying carrier meant one of an igopoly of large selexs who were set up to provide you know SIP to PSTN conversion if you want to think of

it that way in a way that is consumable to over-the-top VOIPE service providers and so you know throughout all this the tandem remained the big aggregation router where all the traffic met by and large not exclusively you can imagine that it's possible for a select that does SIP termination to get a private SIP interconnect to another major select that also does termination via direct circuit that doesn't

that bypasses the IC tandem and this sometimes happened especially in dense metro areas but often times it was just a problem if it wasn't economical at the end of the day it just doesn't matter you weren't passing enough traffic between you know level three and global crossing to where it made sense in most places to bother to do that. So they just ran it all through the

eye tandem more or less you know that's as mobile operators came to play a bigger and bigger role um they too connected to the tandemss and so you wanted to call a cell phone I tandem wanted to call landline tandem wanted to make a call from your VoIP handset to a call that would ultimately terminate in the VoIP handset of another VoIP provider that is also using

a different um underlying carrier for the inbound So at the end of the day, if you needed to complete a intrata call, you know, holla at the I like tandem. Um, so technically Selex needed an EXC license to move traffic into LATAD, but it wasn't terribly difficult for them to obtain this status either for the same entity or for different entities that they both owned, which in

some cases were connected to Elix. Still, you know, by and large, the tandem was the meeting point. The ELEC tandem switches were TDM and they spoke SS7. That meant that a large segment of the industry continued to exist as a necessary adaptation layer because you know VoIP TSPs lacked both the technical means and the desire to invest heavily in TDM gear for legacy interconnection and because they

didn't have the select status required to interconnect with the ILC in the first place and we'll get to where that changed but you kind of start to see where this picture is going right so any future evolution of how calls and are sent and received among different providers was really going to be a question of how the world, you know, comes to lesser degrees orbit around the

So things did quietly shift during the ensuing years. Uh I already mentioned that mobile became more central to the point of now, you know, I'm pretty sure that if there's an age cut off below which you think of calls as just going to and from cell phones and don't even pause to consider any other kind. Um, but as landlines declined, making and receiving calls from PSTN subscribers

increasingly just meant sending them to, you know, T-Mobile, Sprint, and AT&T or Singular, whatever they were called that at that time. Um, and Verizon and mobile operators, unlike Elix were more willing to do private SIP interconnects with major Selex. Uh, this became increasingly true over time for reasons I'll touch on briefly. Um, a lot of phone lines that people perceive to be landlines because they go to

traditional phone systems simply became VoIP quietly under the hood. There's the much wanted VOIPE uh, conversion, but then there's also the kind where you have a, you know, Lucent Partner Messaging 2 system from the early 90s that works just fine, but now you get your phone lines from your cable company instead, the same people that provide you internet. And you don't even really think about it. They're

just phone jacks you plug into on the back of some box. And apart from the fact that they're both down at the same time, there's really no reason to get get into it that deeply. So, a lot of business lines in North America today, such as they continue to exist, are broken out of the back of an ATA or an IAD type device and are provided by

a Seelic or a major cable operator. there is of course the whole IPZ thing. Anybody here heard of IPZ? IES. Um there's probably a good reason for that. So throughout the 2010s, the FCC did via a series of orders slowly erode this monopoly that selects have on owing or on uh sorry owning numbering resources and um and in general doing things that make them a first class

citizen in the Now, that doesn't mean that today you can start a VOIPE company, get an IPZ certificate for it, and participate in the PSTN on the same terms as Selex. We're simply not there. But things are further along in that direction than they were before. And finally, you know, IP tandem operations did become more sophisticated and gain more critical mass. And this was to a very

large extent a commercial accomplishment. So, obviously, the most well-known one is neutral tandem. you know then became Intelquent and was acquired by Cinch and I'm not sure what they're called this week but neutral tandem is increasingly or I'm going to call it that which really dates myself but neutral tandem Cinch is essentially a uh has become more or less the consolidated agreed upon one-stop shop for exchanging

traffic via IP um that's not entirely true, but it's substantially true. I think there was a watershed moment in the late mid late 2010s where really things did start to change because as late as 2015 I was on record saying ah you know the PS the SS7 TDM PSTN will always be here and to some extent it still is. uh there's just so many layers in this

thicket that are hard to untangle. I think it is fair to say that we live in a much more endto-end IP world in 2026 than we did in 2015 or even 2018. And if I were to draw you a picture, give you an architectural diagram of the PSTN both from a commercial and and an engineering perspective 10 years ago, I think it just would look very different

than it does now. And I think um one of the main reasons for that has just been the move of mobile operators to IMS Volty cores that are SIP internally where it's only a very natural leap for them to interconnect via SIP to other carriers. Um, at the same time, the emergence of a of a plausible and durable peering fabric in one place, you know, in the

form of Cinch and some other folks, uh, has made it far more reasonable to say we're going to bypass the ILC tandem. Now the idea that you're just not going to connect to the IC tandem at all anymore as a select is not plausible or on the horizon. But in terms of the utilization and the investments you have to make the the bar has become much lower.

One of the main reasons that it's become much lower is that you don't actually need an SS7 speaking TDM switch anymore to do that. There are very as the traffic volume pumped through the eye tandem decreases there's a bunch more stuff that you can do now to offload some of that cost. I would say there they broadly work in two directions. One is that you get your

own SIGRAN box that well there's a couple of different options there. You can get a piece of equipment that takes SIP in one end and spits out SS7 on the other and still get all the ANF links from your to the tandem. Or you can more or less outsource this problem entirely to someone who handles all the SS7 interfacing for you and you just send them SIP.

There's kind of an intermediate setting where you can speak SS7 directly but using a protocol called SIGRAN which is essentially SS7 over IP uh someone else converts it to real SS7 for you. So there's that has become a lot more pervasive. There's an FCC push to phase out certain aspects of the Telecommunications Act of 96 section 251c2 incumbent specific TDM interconnection obligations by 2028. And what that

really means is it used to be that you you as a CELC have definitive interconnection request rights. You must you can demand that the interconnect with you but to a very large extent the gets to set the terms and say well this has to be TDM. uh that's slowly being pushed out and you know the probably one of the most shifts that has made this more advantageous

for for Selex in terms of private IP interconnects is just that then industry consolidation has made it so that you send traffic to a smaller number of places it's you know probably 70% your mobile operators and a smattering of other large selexs that represent a very small igopoly that basically feeds the whole industry. Yes. And also cable majors and also you have to figure out how to

reach the the various nonarbachals and so on. But you can patch this together in a way that's much more feasible than it used to be in terms of network footprint. So um what that means is that if you are a select and you want to have a private SIP interconnect to T-Mobile, you can interconnect with them not on a laborious latter by latter basis but just at

a few IP pops around the country and you can pretty much just peer with everybody that way else through Cinch and anyone that you still can't reach. I guess you still have to fall back to old reliable I like tandem but there's become a lot more endto-end IP solutions to that problem and of course that has lowered the capital barriers to entry. So, we're undergoing this metamorphosis

right now in a way that if you had asked me 8 to 10 years ago, which to the mind of someone as old as me is not a particularly long time. Um, I would have given you a different answer then about where we're at about whether the eyel tandem is still where the buck stops. That is very rapidly not becoming the case. And um I there's obviously

a great deal that one can say on this topic and I don't want to you know bloat the context window with too much uh information and just TA96 in particular is a very complex topic and had manifold implications for many many different parts of telecom. Um, the one other thing I wanted to say about IPZ real quick is just that, um, you know, a relatively small number

of VoIP providers have IPZ status due to its limitations. Um, IPZ did gain considerably more traction in the 2010s, but there are parts of the PSTN where TDM is still mandatory, especially small rural carrier connections that still require TDM under existing AAS or even state rules. An IPZ only Vo provider isn't going to build nationwide TDM facilities. So in practice, they're still going to have to ride

a TDM interconnected select that already has trunks stitched into every ILC tandem everywhere for, you know, to to reinforce the parts of their network where they can't connect to someone privately. And so when you're doing that analysis, you're thinking, well, maybe I should just send them everything. It's cheap. So also number portability, intra interlateral routing, intercarrier compensation, new generations of 911 and of course as I mentioned

already the limitations on uh IPZ providers as information service providers ability to demand interconnection from Elix you know all limit the utility of IPZ. So I don't know that we today can say that becoming an as an ITSP that if you gain I your IPZ certificate then you can play on level footing with TDM interconnected selex but we're going in that direction. So with all that said

thank you very much. I be happy to take some questions if you have a couple. Go ahead. >> Oh sorry. >> Yeah, sure. Thanks. I'm not sure if it's in my perview to point at people or not. >> Hi, first off, thank you for the presentation. Um, I wonder if you can touch on stir shaken, traffic pumping. >> Selex have done obviously a lot of good for

competition, but there's a lot of bad actors out there and if we are moving away from the PSTN, why is stir shaken such a problem? Why is number spoofing such a problem? And and what do you see going forward? And is this a 10-year fix? Is it a never fix? Just like email, we have spam. What do you think? >> so just to restate the question, it's

about the stir shaken and >> traffic pumping and number spoofing. I can say relatively little about stir shaken implementation bumps and the main reason for that is just that it's such an complex situation by situation topic as to why it doesn't deliver what it promised. Certainly one reason is that there's pieces of call routing that require TDM in the loop and while there are various mechanisms proposed

to propagate stir shaken through that piece uh it's not there. Um the traffic pumping thing is interesting. So I don't know how many of you are familiar with this but without restating a very arduous history you remember the free conference calling scandals of the mid late 2000s that stem from something called access charge arbitrage. Basically, there were Selexs around the country. Well, there were around the country

that operated in small rural areas. And the amount of money that they were able to charge inbound uh IC's that were delivering interlata traffic to them was much higher per regulation than would normally be the case in metro areas as telecom defines them. And the way telecom defines metro areas that almost everything's a metro area. And the reason for that is it was effectively a kind of

cost subsidy for their for the fact that these places are hyper rural, have lower population densities, low incomes, it's relatively expensive per subscriber to provide service to them. And so the idea was, you know, if you're in hyperural Iowa or South Dakota and you're providing landlines to farm houses, this is very expensive. And one of the ways we're going to reward you is you can charge quite

a lot for the small number of longdist calls that you would ordinarily receive being such a small place. Well, as broadband and voipe, you know, propagated, it became possible to uh exploit this dynamic by through traffic pumping. You would put down a server in the operating footprint of some carrier that is based in that territory physically and get a tremendous amount of inbound traffic that under normal

plausible ordinary kind of organic conditions would never go there by setting up some sort of free conference calling, you know, Jesus line, whatever. And um then you know you always have two or three thousand calls lit up only you're getting a penny a minute and splitting it with the local carrier and this was there was all kinds of stuff like that that went on for most of

the last 20 years longer and then of course the reciprocal comp boom before that in the modem days. um to the to the point of traffic pumping it the number of areas in which you can do this that are still suspended from kind of ordinary metro NECA tariff pricing discipline has just gotten much smaller. So these opportunities haven't totally gotten away from some particularly enterprising actors but

it's just become much harder um and the places where you can still collect substantial access revenue have just shrunk to almost nothing. So I mean of the market will find a way to arbitrage away any inefficiencies like that but certainly they were so much more numerous even you know 101 15 years ago than they are now. Um and so you mentioned stir shaken traffic >> number spoofing.

>> Number spoofing. Well really relates to the stir shaken thing. Um, stir shaken has not delivered what it promised in my opinion and I'm sure most of you in the room who are familiar with it might agree. There's so much to say on that topic really that I don't know that I could really address it in the available time. I'm also not a specialist in stir shaken.

So that question is probably best taken to people who are um I can sort of theorize at a fairly general level but I think some of the implement you know having listened to some talks and panels on this my sense is that some of the reasons why it hasn't quite panned out are so very implementationally and situationally specific that if you really want to learn more about

that you have to actually get into the nitty-gritty of where and why. >> was a it was a great question. I I it reminds me uh I think Warren Buffett's right-hand man, Charlie Muncher, that said, uh, show me the incentives and I'll show you the behavior. >> So, that that's a shorter version. I'm sure others have said that before. Any more questions for Alex, our PSTN expert

historical analysis here? seeing none, uh, thank you, Alex. All right. You ready to go? >> Okay, that was a great talk. Uh, just a little bit more logistics reminder. Uh if you haven't got your t-shirt yet, uh we we have two more talks, short talks, uh between now and noon and then we will uh disperse for lunch for 1 hour on your own. Uh there's a number

of uh restaurants uh and things available on the other side on the north side of the the facility here. Um across the street, the restrooms are on the left. We're uh hopefully um if you did have a chance, grab your t-shirt at that time. If you did register for AstroCon, so we have plenty of those available for those who did sign up. Again, if you didn't, uh

please do come back to our booth on on Saturday or Friday afternoon to pick up leftover t-shirts. Uh if you can scan your badge, that would be great, too, just so we can double check who did when you especially when you get your shirt. I know there's a number of other events going on. So thanks again to those who are filtering in from other events. It's a

really great opportunity to share some of this uh knowledge with you from uh the telefan perspective for a number of years. Um we're going to get started I think with in at 11:30. So just a couple minutes to grab another snack if you'd like. We do have uh tables up front in case you can't see them in the back. So it's a place instead of putting your

laptop on your lap if you'd like to sit at a table uh and plenty of power to plug in as well. So, all right, I think we'll get started in a couple minutes with Noah's talk on free PBX security and and then a brief AI talk, our our first uh AI talk coming up here in 15 to 20 minutes of the day. So, excited for Robert's talk

here in a few minutes after that. So, test. Test. Yep. >> All right, everyone. I think we'll get uh started back up here just a second. So, thanks again, Astrocon 2026. It's part of Scale 23X. My name is Chris May. uh with Sangoma. If you don't get a chance to talk to me today or tomorrow, I'll be over at our booth on Saturday and Sunday to talk

more. The Sangoma booth over in the exhibit hall. We uh have the conference through about 2:00 tomorrow. So, you'll have time to get up and head over there. And then all day Saturday and Sunday, we're we're we'll be available with a whole lot more swag. And so if you grab some stickers we've got outside, we've got uh shirts for people who signed up. Like I mentioned, we

also have a few pens and notebooks. If we did just dim the lighting a bit, so if you are having trouble seeing it, you want to, you know, move up a row or two. I know that's a rather small screen to see, but we are streaming this live on YouTube. So if you want to watch from the very back on your phone, I guess you could stream

it. Uh Wi-Fi password is penguins. Um and then this uh particular talk I'm really excited about. Uh his talk on on free PBX security vulnerabilities. Uh made mention 15 we had last year uh pushed out four last night for publication. It is an ongoing issue uh meta issue. We we really are taking a a strong focus on improving security across all of our uh you know portions

of asterisk and and free PBX. In particular, we have uh some great researchers like Noah who have been responsibly disclosing issues uh using our GitHub issue tracker security repository. That is the proper place to report issues for both free PBX and the asterisk project. You can do so privately. We use the GitHub tools to so if you don't have a GitHub account I would encourage you before

you file post something publicly. Please do let us know and give us a heads up. As I mentioned in my talk earlier about 911 dynamic location routing, we this is a system that is used for uh you know some some some critical systems to provide emergency connectivity and and we want to make sure that it's as secure as possible and and that requires your help and and

letting us know about things that you find very important so that we can address them and ask you to come and and talk about it. So, uh Noah King, thanks so much. >> Awesome. Thank you. Yeah. Hey everybody, my name is Noah King. Uh I'm a senior offensive security engineer for Horizon 3 AI. So let's um let's dive into a couple of different things. I'll kind of

give you a story about what we do and how we work with different vendors. Uh the vulnerability timeline kind of starts off. Um there was an off bypass vulnerability not discovered by Horizon 3. We just saw it start trending in the news. And so a lot of times we'll see these articles. Uh hacker news is a big one. Free PBX servers were targeted by a zero day

flaw. Emergency patch now available. Uh when that one came out, there was no known exploit for it. And so what we do at Horizon 3 is we're a cyber security company. We have a lot of different aspects that we focus on. Um, but customers can drop us like on a Docker container in their network and we'll go out there and we'll pin test autonomously all their infrastructure

from a internal and an external standpoint. So, one of the areas that we focus on is external, what's on the internet, what could be exploitable. And we have a lot of different customers that pay for our services. So, we're actively monitoring and trying to find these vulnerabilities before malicious actors do. So when this uh vulnerability first came out, we started reverse engineering it. We pull down the

software, we stand it up in a lab and try to reproduce or we'll use customer targets if they allow us to to pro them. And so as we were looking at it, it took a couple of weeks and eventually the exploit did come out. We were really close to reverse engineering ourselves, but it was discovered in a honeypot that somebody put on the internet. It looks like

a real free PBX, but it's there to just capture exploits uh and use that for information for generating exploits. So once that third party exploit became public, we already have free PBX downloaded. We were looking for this authentication bypass ourselves and we actually found a handful of different vulnerabilities that were related and so we went ahead through the process of reporting those and um some of the

timeline October or August 26 the first CVE comes out August 28th we start our rapid response looking for that off bypass. So a couple days later, uh September 10th, there was finally an exploit found and so we add that to Node Zero offering. We also informed any of our customers that had it exposed. We also at the same time did some research and we found our own

zero days. And if you're not aware, a zero day is just a vulnerability that exists in software that currently doesn't have a patch and it's most likely not reported as well. Um so once we published those uh or disclosed in the free PBX in October 14th there was an advisory for two CVEEs. Uh one was SQL injections and the other one was a file upload and then

there was a third vulnerability in December 9th that had a patch for an authentication bypass. Uh finally on the same day once free PBX releases all of their information there's a safe patch for customers to use then we publish our information our research. This allows security teams and companies to find and have a safe way to test for these things instead of being exploited or wondering am

I exploitable they can they can go and run it in their infrastructure and we're production safe. So we don't focus on taking down their t uh their instances. So, continuing with the news articles, this is the one that came out by the hacker news after uh we reported our vulnerabilities and it kind of hit the the mainstream. So, vulnerabilities in general in enterprise software, they they make

headlines often. They attract the attention of threat actors. So, there's nation states, there's a groups, there all different types of malicious actors. Anybody that even goes on to the internet, knows how to find free PBX on the internet using like open source tools, uh, they can just go grab an exploit and start trying it themselves. Not that it's legal or anything. Um, but rapid response is one

of our key capabilities. And so, like I said earlier, we research and analyze zero days and these end days to find them in customer environments. So, this text might be a little hard to read, but uh I do have a link to all the research, all the blog and everything if you need to review that. But the first one was SQL injections that we found. Um, in

general, if you get a SQL injection, they're they're pretty nasty vulnerabilities. Um, in this case, you were able to pretty much do whatever you want. you could gain remote code execution on the free PBX server or in this case we're just doing a post to uh an admin config endpoint for a new custom extension and if you look down at the very bottom there's an ID equals

and that says insert into AMP users username and password and then we're sending hex encoded values and then over on the right where the frog is pointing you can see that that SQL injection added a user to the database they could then just inherently log in. Um, additionally, there's there's paths to code execution as well, but um, SQL injection in general is pretty bad. Uh, these cases,

they did require authentication. So, somebody already has to be logged in to execute these attacks. So, from an unauthenticated perspective, not inherently at risk, but it is vulnerabilities in the system that get patched. Now the next part is that makes it interesting is if we can find an authentication bypass then we can chain it to the SQL injection. So a lot of times attackers will chain together

So in this case this is more the complete um exploit chain that that does a lot of things. Uh there's three different parts of the payload that the frog is pointing at. And so this this vulnerability was an offype web server. So in free pedbacks you can change your authentication type. You can use web server, there's database and there's two others. Uh none and I forget what

the other one is. But um I went ahead and changed that to web server that is kind of relying on Apache to do some of that front-end authentication before it redirects you to free PBX. And so what um web server requires I noticed when you turned it on and I started trying to make requests to free PBX it's like you need an authorization header and I was

like okay well an authorization so I went and started just filling that out manually. Uh the O header was just it wanted basic O and so there's a base 64 encoded string and I think I just used like a random admin admin user looked at the regular format in a valid session and then sent it off and it it came back and it just worked fine. So

that was the first piece just passing that authorization header of basic with an encoded um encoded username and password. Then uh I wanted to chain this further. So I had an off bypass. I could chain it with the SQL injections or I also was just looking through a lot of the endpoint manager because that's where we saw the previous off bypass was from. So we were looking

in that area and I saw that you could upload your own custom firmware for uh an endpoint. I saw that that was a file upload. You could just kind of choose any firmware off your local machine. I inspected the request and noticed the uh if you look at the second frog um it was passing a path and so it was like TFTP boot and I think it's

like a firmware folder on free PBX. I was just able to manipulate the path by doing um uh like a file inclusion or a path traversal. So dot dot slash just to back out of those directories and then place the file in var www html which is where all the public html and UI for free PBX lives so anybody can publicly browse it. And then the third

piece is uploading uh an actual file. The file is webshell.php. There wasn't any restrictions really to what kind of files you could upload or firmware. So I just uploaded a web shell that says that executes some PHP code. If you send a command, then it's going to execute the command on the system. So, this is chaining together authentication bypass through an authorization header, uploading custom firmware with

a PHP web shell in the varwh directly. So, it's publicly accessible. And so, what that looks like is I have free PBX on the left and then on the right side you'll see we ran the request. I we will write different exploits and templates to test customers environments at scale. So this was the template I wrote and I just told it to cat Etsy password and then

I was presented the results of Etsy password. So from there we were able to basically get full code execution. We could run any system commands that we wanted to on the free PBX machine. So we kind of had full host compromise at that point. So once we found these vulnerabilities um they were some of the first ones that I reported. I haven't worked with free PBX in

the past but um talked with some other engineers on my team and they were just like yep you can just you know submit a report. So I kind of wrote up all the findings that I have. Usually it's just what is it that you enabled? What are the requests that you sent? Um what is the expected behavior and why is this bad? And so free PBX as

Chris said they have a public repo where you can submit re uh vulnerabilities. Um I worked specifically with Chris and K Gupta. So I just wanted to give them a shout out for you know doing the triage uh telling me how to update my free PBX to test their patches and everything and you know just working together and coordinating. So overall it was uh it was a

pretty successful um you know event just to work with them and all um some of the references if you are interested the full blog we have an attack research blog so a lot of our different attack engineers they will find vulnerabilities in other software uh we're known for forinet a lot we have a lot of exploits for forinet we've done things with Cisco um a lot of

different other software vendors that are big, especially anything that gives you a foothold into your network and infrastructure. So, if you're interested in ever trying Horizon 3 or um you know running these tests to make sure all your infrastructure is secure, you know, you can always check out our product. We have free trials and all. Um if you need to report a vulnerability, it's it's easy on

GitHub. And then uh there's just a link to that hacker news article as well. But yeah, that's all I have. If there's any questions, I'd be happy to answer them. >> Yeah, thank thanks so much, Noah. We got a question back here, Alex. Thank you. Um, this might be my lack of familiarity with Free PBX or my Naete, but you mentioned there's a public repo to submit

what sound like zero day exploits. So, how is the fact that they become publicly available handled? I I uh helped field a little bit of this one if that's okay. No. Yeah. So, uh the private uh the security reporting function of GitHub is a is a private area. Well, private to us and GitHub. Uh and uh we we loop in we try to loop in people who

I I'll talk about a little bit this more in my free PBX talk tomorrow. We try to loop in uh you know the reporter obviously and and keep them ab breast of it and then we also work internally in discussion about the solution and so the uh publication comes about later and that is just the description at the top of the project then all the other work

that's done and subsequent comments as we move along and private code forks remains private. I believe the fork is actually discarded when the issue is is published entirely, but the comment history is there. And so we move some of the uh critical sections of the PC uh out of that top level description and summarize it and shorten it for for users. So we have a few steps

that we kind of move through uh to to in this process. Um because we do have a policy and we um you know it's it's uh we tried a hue to uh 30-day analysis 30-day for a fix. I'll talk about a little bit more tomorrow. Not to steal any thunder from Miller, but it was you know great working with you. Thanks again. And so I if you

do have some more questions, I think you'll be around for a little bit uh today and hopefully we can um you know connect a few more people who have questions about this or have found issues and how we go about >> yeah. Yeah, for Sure. And I'm on LinkedIn, so if anybody wants to connect, feel free to reach out. Uh, always happy to answer any questions and

everything. So, >> yeah. Yeah, great to hear about the honeypot, too. We'll talk more about that as well tomorrow. So, all right. Thanks so much, uh, Noah King. And next up, Robert Keller to talk. This will be actually, I believe, our second AI talk. I forgot that Horizon A3 was.AI for the domain name. So, uh, two AI talks in a row at some level. and uh just

a a a brief minute here on on Robert. Um longtime contributor to Free PBX uh over over many years um has developed a number of different solutions. I I think we're going to uh you know really enjoy, you know, taking a look here at at what he's developed now for a little bit more natural way to perhaps configure your phone system. uh we talk about this uh

voice AI concept and and what that looks like practically for end users but it does seem to be that voice is a very common way to configure these systems and we can kind of extend you know that to configuring the phone system that you're itself that you're doing voice AI with. So just kind of touching on that talk a little bit um as we get Robert situated

here. So Hello. Hello. Hello. Hello. Hello. Can >> Robert Keller's talk on free PBX AI. >> All right. So, this isn't going to blow your minds or anything. I I go through a lot of thought processes. is I go what if right and so just the thought process here was I previously uh worked in support at Sangoma providing support to free PBX users and one of one

of my fantasies was always what if when they called in we could intercept them much like a chat might intercept a chat with a chatbot but in voice and allow them once they've authenticated to make common moves adds and changes before even reaching a a a support agent. So um a little bit about myself um 30 years of uh communication experience I started uh believe it or

not with um asterisk at home if anyone recalls that back I don't even know what year that was brought out but that was my first insight to it then free PBX and then onto my career at Sangoma and so on. Um I'm an army veteran and I'm an AI enthusiast. So again this isn't going to blow your mind. It's really just a thought. Hey, what if I

could do this? Um, and just as an aside, my workflow is I use cloud co-work develop in Python locally and then distribute the code once it's, you know, debug and distribute the code. Works pretty dang well. I'm pretty happy with that u that bit. So, why would I do this? Because a lot of like end users, they're not really super familiar with free PBX. Maybe somebody put

it in their ID department and now they got to do a change the name on an extension or something. So it's not comp exactly easy. You know it's a complex UI. There's config files uh slow iteration and then a lot of times not this crowd but a lot of times for users they need uh somebody an expert to go in and make these moves ads and changes.

In my opinion why not just type it? And just just as an aside, I've also I didn't have it ready to show you, but I've extended it to voice using 11 Labs. So I can just tell it to do these things as well. So for example, if I just want to add an extension 105 for John Smith, type it and it goes um and so on. There

are some limitations with GraphQL and and uh Free PBX, which I'll talk about in a moment, but but that's basically it. natural language uh commands that go to free PBX, make the change, apply the config, Bob's your uncle. Um, essentially this architecture for for this thought exercise is, you know, on my local machine, I've got the Python script and then that's where I do the input. Cloud

AI interprets what I'm saying, you know, what my sentence is and converts it into what needs to be done in free PBX. And then the free PBX and I know MySQL is Maria DB, but hey, uh the graph GraphQL API writes it all through an SSH tunnel. Um you know, it's pretty straightforward. I want to also point out at the very end of the end slide is

a couple of QR codes. I wrote up a guide on you can go home and do this yourself if you want. Um and so they'll be up there. I'm only going to leave them up for a month. So if it's of interest to you, grab the QR code, just download the HTML. So if you want it. So um here's the thing about GraphQL. Uh the the things

that do work is I can, you know, uh add and remove and edit extensions, ring groups, and I can read any object, but really I'm kind of stuck. Um I can't do any like IVRs or inbound routes or anything like that. Again, this is just a thought experiment. Although I have been given a few tips on how I might get there with the rest of the config.

We'll see it over time. Um I think this is a little demonstration. This is going to be a little tough to see, but the long and short of it in the upper right is just the free PBX CLI and the lower right is my commands uh local commands on my desktop. And here I'm going to add this is just a a video I recorded. uh add an

extension, you know, natural language, add an extension for whomever. Um there's other things I could say like, well, I do point with voicemail for example, right? And so it's going to do a a deal. It's reaching out to Claude. I I don't know if you can see it or not in the lower right, but Claude is interpreting, and this is real time. Uh just to give you

an example how long this takes in its current iteration. Um there it signifies okay um I'm doing the mutation in the upper right you can see the CLI is fixing to spru around a bit because it's applied the config and then if I go check free PBX Bob's your uncle extension 1000 was created not mind-blowing but it's I think it's kind of cool now in the lower

right I'm typing in I want to create a ring group um and I can decide at this point which extensions I just said all extensions ions for brevity of this, you know, presentation. It's analyzing my query. And then it's saying, do you want to really do this? And it's like, yep, go do it. And it's writing to free PBX. And you'll see in a moment the CLI

will spool about a little bit. And then if I go check the guey, um, we can see that, um, ring 300 is there. And if I was to dig into it, you'd see it had the all the extensions that I have on there. That's basically what it does. Again, um I'm I'm extending a little further. One of the fun things about this project is on on my

desktop, I've got a folder that has all the mechanisms, the SSH mechanism for connecting and to free PBX. And now I can just start writing projects that use that mechanism to connect and the APIs I've already, you know, connected and and do other things. Um so that's Oops, I don't want to do Um anyway, long story short is there's a little bit of safety net built in

there. You know, it does a difference preview at this is okay, you had this, now I'm going to do that. You know, I have to explicitly type in yes because you don't want it just willy-nilly doing things. Um something you don't see, but in the background, every time I'm doing changes, it's making a little backup for me, so I can go back if I screw up something.

Um, that's the roll back, auto backup and roll roll back and GraphQL validation and then auto reload. And by auto reload, I mean it basically does the apply config and free PBX and then page refreshes. Um, some of the commands that you can get away with today, you know, add add an extension, list extensions, delete extension, you know, what extensions do we have? I mean, it just

be natural language queries um into the free PBX system. Same thing with ring groups. Um I I work for a company called Techmetric. Um I left Sangoma to go to Techmetric and we provide uh shop management software for auto independent auto repair shops. And so my role at Techmetric with AI is to is empower our staff and also to bolster our ability to support our customers. So,

um, you know, right now, you know, we've used Intercom's Finn app to be our chatbot, which I got to tell you, I was pretty skeptical, but we're having great, great results. Next step up, um, a Gentech voice AI. That's the bit about when people are calling in, I want them to be able to do some things before they even reach a support agent. It frustrates some users,

but we're actually getting, believe it or not, pretty good response if they can just take care of a thing and not wait. Customers seem to like that. Um, lastly, um, you know, this will be voice max over the phone. Um, in our shop software, there's lots of little different, uh, settings you can tweak. I I will it probably won't be this year, but I will eventually have

it so that an authenticated user either by voice or chat can actually make these changes in their system. I saw a really cool Google Google demo of these things happening and then basically your, you know, your app in the web page, it's a SAS product. It goes to the page and allows you to make, you know, guides you through all the thing dynamically. And I'm going to

go there with it. Um, pretty cool stuff. Um the last thing is I mentioned that it's extensible now that I built this project to do natural language query. Well, within that folder structure has all the authentication mechanisms, API connections and so on. So I I can extend it. One of the things I did is built a hey, what's on this free PBX query? Now I know someone's

already done a visual dial plan. Here's the thing. What I'm not going to show you because you wouldn't be able to read it on the screen. And if I scroll down in this display, it has audited the free PBX system to see where the gaps are. Like if I made a ring group but didn't put any extensions in it or an inbound route that doesn't go anywhere,

whatever. It audits the entire free PBX system to determine where your gaps and where you know you may have setup to do. And I find this would be particularly useful if you're a free PBX support person and you got a new customer that has an existing system. Instead of digging through everything to find out what the dial plan is, run this query on it. It'll analyze the

entire system, visually outline what the existing dial plan is and where the gaps and problems are in it. Um, and I'm I'm just going to keep extending it from there. It's u I have too much fun in cla work to be honest. Um uh and then last thing as I said that's that that is the presentation. Here are some QR codes. Uh the one on the far

left over there that's the project itself. The one in the middle is that visualization thing I showed last. And just for shits and giggles techric is the last QR code in case you're curious where it is I work. So uh questions. >> Thanks Robert. No. Okay. Thank you very much. >> That was very informative. Uh great use of the several free PBX modules. Uh you're right. We

have about right on time noon an hour for lunch. So uh please um eat something. I think there's an apple if you you know just want to grab that on your way. Uh a few other beverages we'll have back uh after lunch. So, some after after lunch coffee tea availability. Uh, we'll get back started at 1. Uh, desperately seeking Susan uh an interesting talk about some interop

with the French government and for a project for the French government. So, that'll be at 1:00. We'll see you back here. Thanks so much. >> Okay. We're going to wait few more minutes for people to come. wait a couple more minutes to get back started here. So, if everybody uh some some cookies came in the back and some chips, uh refreshed coffee, and there's tea, so feel

free to grab a refreshment. some water as well back there. So, and then if you were curious, the restrooms, they are located through the double doors when you get out and make the left. I think those are closed now, but it's still back a little further that way. So, we'll get started in just a couple minutes. Alexand Alexand It's okay. >> All right, I think we're going

to get started now. F we can uh mosey on up. Thanks again, Astrocon 2026. Our next presentation after lunch from Alex. I'll give the courtesy of not trying to pronounce his last name. uh desperately seeking Susan fun asterisk project for translations on behalf I believe uh for a sponsored by the French government. So if everyone um can uh tune in over here uh that'd be great. And

again uh our live stream on YouTube if you can't see in the back you can hold up your phone and watch that from the that's the scale Linux live stream. There's a number of live streams for each room on YouTube. So, if you go to the scale website and then click on their YouTube, that's probably the quickest way to find the live streams. And afterwards, everything will

be published and cut up into little pieces so that you can watch it uh presentation by presentation little bit more focus on the slides. We have a couple separate feeds for slides and then for the speaker themselves. So, again, uh Alex, over to you. >> Thank you very much, Grace. Um first of all um who is older than 45 years old here in the audience. Okay. So

those one may know this movie 1984 where first movie with Madona and uh the Artet sister and um I would like to apologize because um this talk has nothing to do with the movie and you won't see nor Miss Artet or Miss Madonna on stage but it was about finding the right person at the right time. Um, who am I? I'm Alexander Moroni, but um, as Posan

said in the song, you can call me Hal. I'm 56 years old. I'm French, so excuse my English. I have um, three plus two kids with my wife. They are um, old now. It's a huge family, like a small business having five kids. This is my 10th Astricon in a row. First one was in 2016 Astri in Phoenix, Arizona. I'm an IT and telecom enthusiast since I'm

15. I was born with a ZX81 for those who remember it's small computer and I'm also a private pilot and sometime a singer, but that's another story. Don't ask me to sing a song today. Um, back to business. I'm the CEO and founder of Uris and Kibri. It's um, we established the company with my partner Elen in 1992. We are located in the south of Paris, 20

kilometers away from Paris. We have u more than 50 employees. If you come to Paris, feel free to ask me. We have some good coffee and our business is entering service. It's a specific part of call centers. I know that uh in America you have entering service company in France. It's a very specific niche. We uh we receive call mostly for uh doctors and clinics. Um and

we are operating on this industry um in two different ways. The first one um with the brand Uris. It's a call center. We have um around 40 to 50 employees on that brand. Uh and we are answering live calls, incoming live calls from doctors, clinics, small business businesses, uh customer care. We also call that BO. Um and uh since the beginning um as I said, I'm an

IT enthusiast. I was born with a computer in my hands and um we create softwares for the entering service industry. Basically, of course, we are our first clients, but we also provide our softwares to our competitors that make us in a very specific position on this uh market. Um if we talk about metrics uh so uh with Kibbri we are um we have up to 1,800 agents

human agents because now we have AI agent of course um we are handling 80 million calls a year mostly inbound um running over 150 a different asterisk I love asterisk um 20,000 uh different DIDs on French, German, and Belgium markets. And our system can go up to 2,500 uh concurrent call and up to 1,700 incoming call in just one minute thanks to uh Open Sips. so let

me talk about the project. We have one client um that on behalf of the French government well the French government changed the rules to ask for the politic uh refugee status. Uh you know that France is known as as the human rights country. Uh so everyone who ask for the refugee status can have it but he have to fill some forms and meet some people and blah

blah blah. um uh back in the past it was a long process and the French government asked to reduce this process and of course when um you come from Belgium but if you are Bel if you come from Belgium you don't ask for the political refugee status in France but if you come from a country where you don't speak French language it can be quite difficult to

fill the form so the point. One of our clients, which is a translation company, um was commissioned by the French government to be able to provide a translator in 125 different languages to um one government agent that having in front of him someone who asked for the uh political refugee status. We have 125 different languages covered by um more than I don't know 600 700 different translators

and that in just five minutes. So it means that if you go to this government agency and you ask uh to speak to someone and fill a form and you are coming from Sri Lanka for example and you speak only Tamil um the agent have five minutes to provide you a Tamil translator. So, let's play a little game and back to um when you were at school.

Um we have different languages right here. Can you tell me this country flag? >> Spanish. Who said Spanish? Who said Spanish? I heard Spanish. >> Yes. This is a candy for you. Okay. This one. This was a nice an easy one. This Ukraine. Great. Whoops. Sorry. I don't know if it was a good idea to put Ukraine and Russia in the same place. Um, Hindi, who said

India? India. Who is India? Okay, David. >> Yes, last year you were the one to send me some candies. I remember. Um, more difficult. Ask Mr. Trump. This is the Tajikistan. So people talk the Tajik. This one is for me. Of course. Maybe you know Cam. What? Nope. Nope. Cambodia. Who said Cambodia? Great. Thank you, sir. Okay, two more. A little bit difficult. This is Romania and

the the um this is Romania, sorry. And um sorry, the the one before it was the Philippine Island which have a lot of different languages. Uh and it's the Tagalog. Who ever heard about the Tagalog? Thank you, sir. Do you want a candy? No. And uh the last one of course Russia but we don't have a lot of people asking for the French uh citizenship coming from

Russia. the process. So as I said when uh government immigration immigration agency requests a specific language it calls uh the ID number it get in touch with an agent ask I want someone to translate Tagalog and the agent just ask a robot let's call it a robot to uh find a Tagalog uh after it qualifies of course the request This looks quite simple. Thank you to asterisk.

I will show you how it works Um, our two skills, our two subseries, the first one, the service company URI with the agent and the second one, the software company Kibri. All together we were able to faced our clients requests. At Uris, as I said, we have uh 50 uh agents working 44 64 hours a week. Um and we have an average queue duration before pickup uh

which is under 18 seconds. Nothing about uh call centers as we may know. You know, when you call and you have the music on hold for minutes. Um on the software side we were in need to build the UI for the agent which were part of a business. Uh setting up setting up the call flow I will demonstrate to you. And um just keep in mind that

for some uh specific languages Russian for example or uh Turkish we have more than 80 different translators that can speak that language. So how does it work? First when the agent ask for a Russian translator, we will send some batch calls. We will extract the 80s plus uh different translator. Then send the call in batch of 10. Wait for one of the translator or many translators to

answer the call. redirect and ask the translator to press one to accept the mission. And then we stop all other outgoing calls and put the and transfer the translator to the agent. So we can uh put the translator with the uh government agency. Quite simple. So we have the agent who send the call and of course if we have let's say 30 different translators and the first

10 one doesn't respond we stop them and try another batch of 10. It's quite simple. So when we have the first one to press one we kill all the other calls send the call back to the agent saying okay I have a mission for you. Okay. And put the code through. But it leads us to um different challenges. Well, first um what to involved in this um

mechanism? The first one obviously asterisk. Um we build a call controller that was that is in in charge of sending the call, manage the call, uh clear the call when someone get in the IVR and so We used AMI. I know Chris and Josh you told for years to stop using AMI in the benefit of ARI, but AMI is dead. long live to AMI as I said

um well we have a long expertise on AMI we use it a lot so it was um uh um common sense to use that kind of technology um we also used um asynchronous AGI for uh signaling during the IVR um we used an IVR of course for the mission confirmation with some specific and tricks. Um and mostly the group because um as we send a lot of

calls and um sometimes we had two or three different translator getting in the IVR at the exact same time and pressing one at the exact same time and getting to the agent which was only in need of one translator. So I will show you the dial plan after that. Um and how we used the group variable. So the call controller we have the agent uh the call

controller is exposing some APIs basic APIs that we created to create deate list get the status of a a job. And basically a job is what is uh requesting a language ID uh giving the device ID of the agent uh the number of simultaneous school we want to send and some different parameters. Um we have AMI to originate uh to um list the channels running uh and

to hang up the calls uh when u one translator accept the the mission. So, asterisk of course is sending the like on the previous chart when we get um the IVR send through um the a synchronous AGI uh the status is it answered uh does the uh translator press one um etc etc. So, let's get in the IVR. Um, sorry, I tried to put all that in

the same page. So, maybe it's a little bit uh small. Um, it's a classic. Who who made an IVR with the asterisk D plan? I guess a lot of people, right? Okay. It's quite basic to do. uh um and it's as basic as as it's a just one level uh IVI but as I said uh we use um a synchronous AGI to uh signaling the uh the

the call process in the IVR to the call controller the main problem we faced was um we we well first we had two problem the first one um is when you originate a call uh you know that imi is asynchronous so when you send uh when you ask to originate a call you don't get the answer the answer right now and you won't have the channel name

in response to the um uh originate and uh the problem is that sometimes we had some calls going out and at the same moment we had one translator pressing one But we didn't know which channel to clear. So that's the reason we used the uh the core channel. I don't remember the name. Um and um the second part the second uh uh challenge we faced in the

IVR was that two translator can come in the IVR at the exact same time as I said. So we have these uh variables. I put it in the bottom uh left. Uh it's the job ID, okay, that we have on the channel. And this job ID um we used the group variable with a name of the job ID and we checked how many channels are in uh

use this uh variable. So as you can see on the dial plan uh if we have account more than one it means that we already have someone in the IVR and we kill a call. Well we not we don't kill a call we said sorry we already have someone to fill this mission. uh one year after because we started this project one years ago um it was

at the beginning it was just one agency in France um there is 50 of them uh we had more than 1,000 requests um that was that were successfully uh handled uh and 99% of those requests were processed in less than five minutes. So mission accomplished. We have some uh future announcements that we plan. First uh instead of um hanging up brutally the call when someone when one

translator press one uh instead of uh hanging up all those other calls, uh we would like to redirect them to a specific sound file saying we are sorry you came too late. and for the moment we use uh our translator in a random basis. Um we and this is uh something we need to to uh uh figure out with the French government. It's how to do do

we need to ask if we have one translator that answer very fast do we need to priorit pre put it in the top of the list? Uh so um the uh um uh the draw point will be that it's always be the same translator uh or do we have to use a round robin? So we need to um enhance the rules to select the different translators. Um

and the last one yes sometimes because we send calls uh to some to some mobile phone and sometimes we get into the voicemail. Uh we have this wonderful module in asterisk uh AMD. Um this is uh entering machine detection I guess. Uh that works quite good. So maybe we will use it in in the next future. So that's all for me. Uh if you want to reach

me uh feel free to send me a mail or contact me in LinkedIn. Um and if you have a question I'm here to answer. And please uh ask your question slowly >> All right. Thanks very much, Al. Yeah, that was great. I uh appreciate the dial plane examples. Uh no no hard feelings on AMI. None at all. We'll have to ask Josh about that tomorrow. Might have

a little bit more to say, but uh personally big fan of comp files. I had a few in mine. And uh if you've looked at the free PBX source, there's an awful lot of that AMI going on. AGI as well. So, uh, there's a question back here. >> Well, it's more of a comment on the routing for the agents. One thing that we ran into and we

created in one of our systems was a priority and a weight. So, priority is like when you're at a deli and you have the lowest number, you always go first. But then if you have two people that have the same number, if you assign him a weight of 50 and him a weight of 1500, he's going to get far more calls. But in working with it in

call centers, you need to make sure that both mouths get fed, but we may want him to get fed twice as many times as he does based on whatever's in their history. And that brings me to my next question. Have what you're describing is a ping post lead generation system. Have you thought of doing something outside of asterisk in another system for whatever work they're doing so

that you can keep track of the agents route and they can claim it there without actually initiating phone calls? >> I'm not sure I get your point. Can you rephrase please? >> Um, let me help. Uh so you're asking about external application interaction >> for the agent to self- select their >> history >> with history. >> Well um it's not about the agent, it's about the translator.

The agent is getting a call. >> Yeah, I think I misspoke. I I think you meant the translator. The translator seizes the call. externally to the queueing mechanism. >> Exactly. Okay. So when when we send the >> well at the first place when we get the call from the uh government agency um it goes to an agent um among the 50 we have using a regular queue.

It's a homemade queue but it's quite the same that the one we have in asterisk. Right. This is basic. The the problem was to well the main goal of the mission was to find a translator in a very specific language in less that than five minutes uh an external one which is um a freelancer. So there are not employees waiting for missions to come. So that's and

five minutes. This is fast. This is very fast. So that's the reason >> just a can I ask a clarifying question? So the agents then are remote taking a call on a cell phone? >> They can be. Yeah. >> Oh, you you you talking about the translator? >> The translator. Excuse me. Yes. The >> Yes. Absolutely. We have we have translators all around the world. >> Oh,

very good. >> Yeah. And um we also have some uh that we call on soft phones or WebRLTC and >> Huh. So they have a variet maybe that's the issue then with with the option of the external it's a matter of they're not running a dedicated app. They're just answering the phone and translating not a standalone headset desk >> where that might be the that might be

the issue. Yeah absolutely sure. Well that's a great way to leverage the uh I I like it. >> That's another use of the PSTN. >> Yeah. Yeah. Uh anyone else have a question? uh for now. Oh, here we are. >> I have more of a human factors question. Uh given that you support 125 languages and you have a um you have translators supporting all those languages, how

does the agent know what language is being spoken to request a translator? >> Uh because we have a database. I I don't know if I get your question, but we have a database with different translators, the different languages and the skills for each. So, we know that Mr. uh Smith uh he speak Russian and uh um I don't know Indie and blah blah. >> I assume >>

Oh, just a second. Oh, so we can catch you on the mic. Yep. Is that all right? Here you are. >> It's the agents are the ones that f that get called by the French government. So I assume the French agent that is making that first initial call knows I need this language and they communicate that to the agent and then that goes to the database and

knows which >> absolutely that's correct. >> A couple couple humans involved in the >> yeah I I'm surprised that no one mentioned the AI >> uh well we just assume that these were AI agents. These were actual human beings. Al come on. >> Well that's a question. um uh to to replace the translator by a AI translation system of course uh but it's all a question about

liability. We are talking about refugees um about some government stuff. So that's the one of the main reason we still use some real human translator. >> Ah yeah confidentiality being a key component privacy very important functions of this kind of system. Uh excellent. Oh another question. Uh, you said about AI, like using AI to translate, but what about using AI to just replace all your agents? >>

Um, because I'm French. >> Moi, >> you are? >> I speak French. Yeah. >> Okay. And uh you know um maybe it's the main difference between uh Latin people, French, Spanish, Italian and um uh German, American, English people. We are very very inhuman relationship, empathy and so on. Um u my point is AI is um very good to enhance human. It's not here to replace human. >>

Okay. But your all your agent is doing is just routing the call to the the proper translator. >> The foreign government can go in an IVR be like I need language 16. Okay. That's Russian Indian bypass your agent or go an I or you know what I mean? >> Yeah. Yes. Absolutely. You're right. Um 99% or 97% of the calls are uh for new mission. Okay. Um

I have this Pakistan guy in front of me. I need a Pakistan translator. I called sometimes during the mission because um the translation mission can last 45 minutes, 1 hour, 1 hour and a half. It's it's very long calls, right? Um sometimes the call drop because of uh not PSTN but mobile uh uh and so the translator call back the agents and there is some specific treatments

of those calls. So we still need to have this human history uh to correctly all the mission um from the beginning to the end. But you're right that's one of the announcement we are working on. Great. Any other questions? Okay. Oh, hey, sorry. >> I love there is a lot of questions. I was not expecting that. >> Um, more of a question statement. So, I was just

curious. So, you do hangups, right, of all the translators that are not needed, but then you do batching, right? So, you do these like 10 at a time. Have you thought about like redirecting them back into the pool since you have their attention anyway? >> Um yeah, that's part of the announcement I talked about. Uh because it's a little bit brutal, right? Um when you get a

phone call and when you are ready to pick up the call and suddenly it hanging up. Uh so yes, we're working on that. Um the the very first point of this mission uh of our job of the project was to provide a translated five minutes. Um now we are trying to make things better >> like optimize the flow. Yeah. >> Yeah. That's it. >> Makes sense. >>

Great question. Anyone else? Oh, there we are. >> All right. Uh I'm going to assume the translators are being paid >> of course >> by the minute or whatever. So it's are you in charge of the billing for that? Like is it handled by your system? Is it handled by the government system? >> No, it's um well there is two sources. The first one it's there is

a cross-checking. First of all, we have the list of the call we transferred to the from the agency to the translator. uh and we know which translator how long it lasts and and on the other hand the agency is keeping track of all their requests on their side. So at the end our client which is the the translation company uh they do this cross check to uh

uh bill the uh French government and to give the money to the translators. They're they're not just there to to chat for the day, I guess. Yeah, volunteers. Yeah. Great. Uh good question. Anyone else have any questions for Al? Okay, that was a lot of great questions. Uh we'll get started in uh with our next speaker. Thanks again, Al. >> Thank you very much. >> We're going

to get uh Dan set up here. Hey, didn't rec Oh, nice. Nice beard. Uh yeah Dan uh to speak a little bit about ARRI uh just to you know we talked about uh Al talked about AMI the asterisk management interface that is a TCP connection typically over 5038 and has been around for a number of years in in asterisk uh it is one a great way to

interface spool calls return status of uh different channels in your system uh also mention then was a little bit of AGI the uh asterisk gateway interface which is similar to a CGI what you think of like an Apache web server if you're you know new to asterisk and so that was a little bit uh lower earlier I believe in the history of these interfaces and uh similar

though uh read write uh you know protocol that a lot of pearl clients and things of that sort uh were written and then moving on to the asterisk rest interface which is a little bit more web friendly GraphQL type uh solutions. So th this is uh sort of some of the evolution of of different ways to talk to your Astro system uh from other systems and there

have been a lot of improvements and and changes and things. So I I think that Dan is going to teach us about some of those in a minute. yeah, that's great. This is a good Yeah, this is great though. Um, yeah. I think we'll just uh make sure set up here. Looks good on the screen. >> Yeah, I think it's good. Huh? >> All right. Thanks a

lot again everyone coming to Astrocon 2026. Dan Boga joining here about real time billing using the new birectional ARI. Dan, over to you. Thank you very much. Thank you for having us here. So, just few words about company paying for my my trip here. So, we are a a company located in in Bavaria, Germany with back offices in Romania and Albania. So we first thing we we

choose a spot is the beer should be good and accessible price-wise. So all these uh offices they meet perfectly the goals. Then uh we we got over 19 years of experience now with architecting oldtime uh old but gold um serverside solutions in VOIPE environment and not only nowadays um we have um in the past plat platform implementation covering both wholesale as well as retail. Um by now

we understand what uh uh real time processing uh constraints are and how serious our life outages. thinking of all this together, putting all this together, this is how uh our billing engine came. Um we call it nowadays um real time enterprise billing suite because it's not anymore uh just a billing engine. It's a set of services or subsystems how we call them which help you with what

we think everything what is missing already uh in your um out of um concepts we put in is that it should be pluggable into existing So um you should not be forced to route your traffic to the the box we choose and you should not be forced to do things in the cloud by by not when you're not willing to. So you can either uh host it

on your side on premises or to the the hosting provider you prefer. Um it should also be able to accommodate um new components into uh whatever ISP ITSP uh network and um it should be non-intrusive. So the in the end the administrator should be the one choosing um the the architecture and choosing the demands of his network. Um it's all open-source software. Um was born back in

2010 and the first sources were published in 2012. Full sources is a are available on GitHub or on our um GitHub repository 100% Golang. So when we started in 2010, go is like it was like weekly released. So it was quite uh an adventure to start with and that makes us one of the first Golang uh projects around. Um we have today I think more than than

even 500,000 lines of code um and lots of considerations for community We got three branches indefinitely all supported. So uh v 010 which is uh stable conservatory release. U this working with telecom you know what what that means. U then we have the the master which is also used in production and then we have u but is more like um where where the new uh functionality comes

in and 1.0 zero where is which is our vision for the future. Uh we are performanceoriented as a billing engine. Uh most of the time it becomes the bottleneck of of of one's network. So we need to be fast. Uh for that we have built in our advanced caching system. So uh if people are using I don't know u NoSQL databases like radius for caching we use

radius as normal database. So uh we we have our own layer which is built in the in the process itself where the caching occurs. So that's written by us. So uh this is what it makes uh things go or work faster. uh it's it has some some uh interesting concepts built in. So LRU list record used. So we keep only hot data in the cache and uh

also TTL so you are able to to say how long time the data should stay cached. Then uh all the the processing it's asynchronous. So um if you know go you know that it's harder to synchronize the go routines what what they call in than uh putting them uh as synchronous so it's natively asynchronous and we have included an API load balancer if you have to grow

uh faster than uh what one engine can do which is something we have estimated about 7,000 to 10,000 requests per second per per and um of course it depends on on real life uh um and um we have um in memory database implementation our own. So if you want to uh replace radius although nowadays they came back to being open source um you can still do that

um with our own database and uh we also have same like radius on disk storage support it's very similar concepts um the architecture is modular it's cloud ready uh from from back the days microservices with rich set of RPC APIs. So all what we are doing it's API and it's easy to enhance by um rewriting specific components. So if you are unhappy with the licensing which is

uh AGPL the most modern version of GPL you know uh Radius went there uh due to the fighting with all these uh cloud providers. So we we learned from them before we we had the their problems and then we directly went to AGPL if you want to to rewrite one of the modules bounded to AGPL you can simply replace it and put your APIs because you all

it takes for per module there are like five APIs which you have to support. So write your own modules. You are not bounded anymore to GPL. Um testdriven development. Again, we usually start with test and then we write the the software and we have uh over 10,000 tests as part of the test suite. They all run with GitHub actions. So whenever we get a PR, we already

know if it's compatible or not with a stable software. So you can always see that before accepting the PR. Um in terms of functionality, um I I tend to sound like I'm a marketing guy, but I'm not at all. So I'm purely core developer of the software. Um the reason I'm I'm uh talking um uh about all this functionality and about the software is first of all

I'm very proud of what the software does and the second I'm trying to teach you what it can bring to you. So this is why why you see all all of these things um highlighted. So it's an um OCS. This was the the first target uh of the software. So online charging system. Uh we are popular in mobile networks. So including the latest with EIM stuff and

5G and all these uh big uh uh let's say trends. Then it's also an offline charging system. So you can uh charge you can bill your offline CDRs which you maybe receive from your suppliers from uh I don't know any any source of uh CSV files for example XML whatever you can charge them using CG rates it's it was multi-tenant from um day one so you can

do white labeling platforms you can uh uh implement stacks of of different billing um runs. Then um we support multiple databases. Then um it's uh we support real-time configuration reloads because you need to to do that when you are uh changing your mind regarding your your functionality. Uh it has a rating engine with derived charging. Uh this derived charging it's good because you can charge uh an

event like a CDR multiple times. So you can do it for your distributors, for your suppliers and for customer which is end customer of your distributor seller. Then um we support a number rating. Maybe here is not a big thing but in in Europe uh it's a a feature you must have as a as a carrier or as a wholesale Then um we support account balance balance

management with bundles and dynamic prepaid. So um like all these mobile networks to have bundles of data, bundles of SMS uh actually any kind of bundles. If you want to sell cars together with your uh telephon, it should be possible uh via these bundles. Dynamic prepaid creates accounts on the fly. So if you want that uh anybody could have a test account bounded to his CLI uh

of I don't know one euro or 5 or $5, you can configure the system. So if it doesn't recognize the CLI as being a local account, it should create the account but limit it for $5. So um this is what Dina prepaid does. Then you have sessions or event charging with balance reservation. So it's the the full prepaid mode like chargeable every 1 second to 5 seconds

whatever you have the heartbeat for u for your billing same like mobile world and uh you can do uh refunds because these are always uh charge in advance. So for example, if you're charging at 5-second interval and he's only talking 3 seconds, we will refund back in the end. Then we have what we we thought it's fun to implement back then, still shaken in the billing site.

Uh actually what it does is it it grabs the information from the VIP switch in this case. uh it compares it to steer shaken um hashes and all all this security stuff around it and it's able to to return back if it's uh authentificated the CLI or not and what it does also is that you can also use it to generate your own uh steel shaken header

so u it's unusual I know but back then when we have implemented most of the switches did not have it. Um now in the 1.0 we were still arguing whether we should keep it or not because not many are coming to the billing engine to implement the shaken but um there are still especially in the mobile world where the switches are pretty old. Uh there could be

still a nice enhancement to have during authorization phase because we need to anyway authorize the call. So the call it's coming to us before going to rooting it's coming to OCS. So it it feels like a native place where you can do uh steer shake and authentification and a native place where you can still reject the call if the CLI is spoofed. So I guess we will

still keep it because we don't pay for development anymore. And then uh CDR logging um as as normal CDR server. So I'll explain you how we create CDRs for um asterisk. Actually we steal the CDRs. Uh we can also rely on asteric CDRs but we can also generate CDRs via ARRI on the fly. And for that we have we had to use some tricks which is what

I will talk Then um we we also support uh rating cues and what are these rating cues is we can get the the CDRs in a real time push to us and we push like forward same in in real time um towards uh whatever uh message bus or database of your popular should be I don't know Kafka or something where the asteric CDRs can be received in

real times in Kafka already built. So it looks like they are coming from asterics because they finish by the end the when the call is finished few milliseconds later you have them in Kafka already built which is kind of cool because uh it it's um you pay almost no penalty other than configuring so hard to configure CG rates but you have no other penalties to get there

um in real time. So you don't you don't have to wait few uh minutes like batches of 15 minutes or or one day to get your CDRs to your customer. And um also um the the number of of uh interfaces for reader or exporter also offers you the possibility of having the CDRs from asterics distributed in many places in the same time. So you can send them

to your your distributor and you can also store them to to your own uh cues or and some examples you have here AMQP for rabbit MQ or uh SQS from Amazon SQL, CSV, XML whatever. So the the the normal ones then what we also do is doing billing it becomes again natural to do fraud detection because we work with money. So we know when something looks abnormal

and uh we also can do automatic mitigation. So if we detect fraud then we can shut down we can go so far to shut down the account until and maybe inform the knock. I don't know how many uh levels of escalation you have uh informed the knock that there was some uh probability of fraud and we have shut down the account so no can review it and

um unlock the account. So if things are happening in the night you are not anymore exposed. There can be some some customers shouting that they could not use the the system but better to have the car with brake first be before the engine because it could be that the damage is uh like uh more serious later. uh then we we also do again very um compatible to

building LCR because we have access so it's LCR with pricing it goes hand in hand then we do LCR with uh quality of service having access to CDRs you have access to all the the PDD or average call duration all this uh fancy old style which are still um better than looking on uh latency and jitter of your network because the your customer is the best tester

and this still is since always. Then we also can do LCR over bundles. uh there are suppliers offering particular number of minutes with special prices like 10,000 minutes to specific destinations with one cent and then they they go to I don't know two cents or whatever. So you need to know the the the QoS and we also support um pattern monitoring. So you can do also the

LCR based on um call duration and not only per minute because it's it's important if you have u uh between uh peak and off peak a call and prices are changing uh your average call duration it's three three and a half minutes and then you should get the proper feedback from the LCR system on LCR we also support number portabilities and all all these other stuff and

uh we also support dynamic pricing imports with templates uh diameter radius uh you know for for the mobile uh worlds all this this uh protocols are important DNS everybody thought DNS is dead but I know switches doing DNS um queries for their routing so enam routting uh SIP server we even have built-in SIP server where you can send your calls from your asterisk for example and get

redirects based on LCR. So this is another type of uh project. Then resource allocation controller and these are dynamic resources. So you can count the so-called virtual channels per destination, per time, per So you can uh limit things in the in the switch per uh via this resource allocation controller. So you you can have I don't know um 20 simultaneous channels to uh US mobile or to

premium or whatever. So um during this and this time so this uh virtual resources they kind of um give you the opportunity to to make this old static limitations of your switching and very useful in commercial uh like to monitoriize your uh business uh based on various tricks you want to to to do in your network. then API server which uh supports some special serialization of Golang

which is GOB then JSON JSON RPC then HTTP JSON um support then built-in high availability and dynamic partitioning uh API capturing and an analyst um um service and then clustering all this um as as soon as you go grow they are all uh in there. Then um of course we are developers and we are passionate about adding new Then um in terms of appliances, what can you

do with the software? So online offline charging system it's uh the the default one and uh original product. Then there you can implement very complex rating. So this kind of charge the the first minute as a as a whole round it with specific um um amount of of monitory and then if you want you can go to charging each nanocond differently in a call. Uh if anybody's

crazy enough because at some point everybody was just complaining that most of the switches were doing milliseconds and they wanted also to have milliseconds. So we offer nanconds and nobody wants to do it. So it's you know it's it's hard to to um to find the middle way of what are the limits because as soon as you you go more everybody realize that they want then um

sorry if I uh if I sometimes sound a bit um off sync where where I'm coming from it's like uh middle of the night more. So, it's 1:00 a.m. 2 a.m. So, I'm still trying to make sense where where I am now. Great. So, um the as part of online charging system uh we have the uh unlimited bundles so you can add as as many bundles as

you want. We have the CDR server and uh high number of interfaces all what I I I showed you before. Then as rooting server separate rooting server you have flexible rooting strategies. It's it's a very advanced rooting server. Then um as I told number portability and complex rating bundle support and routting based on QoS so you can choose and also fail over. So you you choose by

default to root on on uh best uh stats but also with limits on revenue for example. So you can combine and then uh load balance between multiple uh carriers and all this stuff. Then also limit the channels which you send to specific suppliers. So the engine has to to compute a lot of rules inside and it's capable it's perfectly capable of doing that. um fraud mitigation um

you you have access to multiple subsystems. So it's not anymore just the billing part or the bundles part. Um for example, if someone recharges too much money in short interval, you can also flag it as a fraud. Then if uh someone is hitting too high number of channels to a specific destination, it's again fraud. So uh statistics is again so if he if he does more um

average call duration than normal it can be uh fraud and so on. So you have so many places where you can default detect fraud is not anymore just and um you have also uh informative versus reactive. So we can react by shutting down as I told you the the accounts. um you have escalation procedures uh defined and um yeah it's um you you have to complain if

something is missing but I have the feeling only AI integration is missing um start server um you have uh the the difference of the of the start server since we have this advanced caching and inmemory database and stuff we could build all the the statistics in memory. So you don't have to press a button and wait like uh 15 minutes until your ASR for the last day

shows up. No, now it's real time and you can react on that. So the CDR comes in your stats are already updated and then you can react as soon as the ASR drops you can already send notification out and that is real time. Then um you have a lot of uh strategies most of of them which they are used in telecom nowadays and then you can also

have generic statistics. So you can say I want to calculate the sum of this column in my events. So you don't have to just go with all this ASR ACD PDD and which are coming from the history. Then uh resource utilization control um for virtual resource counters and the the one which I told you about virtual channels mediation server. So you can enhance your your events. Um,

an example of mediation is um the the emergency services where you have to map the user um the the closest uh available emergency service center or uh number portability which you have to again replace uh his um um number to um to to the real one which he he converts to and or the the the kind of LCR and um this LATA stuff which you have here.

This can be done also with the mediation server. Then billing assurance like parallel billing to what you have today already. This happens there are there are projects which they have to testify existing billing. So compare always billing from two different sources. And uh recently the the most craziest thing I could think about adding in a billing system is uh IP address management. Um everybody will say what

does IP address and DHCP have to to do with billing? And then I will tell you that the new eim services in the mobile world they need IP addressing and uh people who are able to statically control the assignment of the IP addresses they make a lot of money out of it because by default it's all DHCP. So mobile networks they offer by DHCP. If you are

able to give static IP addressing to your mobile headsets then you build you start building uh private VPNs between headsets of your big large corporations shell or whatever and then you enhance the the value added services for mobile world. So this is why we have IP address management incurates because we are part of the the mobile networks and then we have to offer enhanced services there. Um

this is some some sort of architecture I tried to build maybe it's it's too little uh font size. Um the agents are the inter sorry interfaces out. So um diameter, radius, HTTP, DNS, I I told you about SIP agent and here is also the asterisk agent. So we can talk directly at the edge of the of the sidurates infrastructure different dialects with different uh switches. We also

have free switch. We have kaleio. We have open sips and then we have something which we call ERS which is event reader service which is able to to map and read various databases of information uh various uh message buses and um CSVs and other uh uh supports for for CDRs. And then out of these events we push into a system which we call sessions. And from that

moment the single event which I'll show you coming for example from asterics it becomes a session. So we can make sense of billing per uh repetitive like in every few seconds. This is very useful because you can have uh concurrent uh accounts dialing in the same time and then the old type of emulated prepaid doesn't work anymore. So you have to regularly uh charge uh concurrently from

and then from sessions then you have access to all our subsystems what we call core subsystems which is rating CDRs attributes for mediation chargers for multiple charging rers rates and resources for for virtual I told you about routes stats thresholds trends trends is also interesting because it can build up trends. So you can compare the traffic of today with the traffic you had 24 hours before. Again,

a way of doing fraud detection because if the trend is not like if you if you get a sudden increase of uh 10% or more, you want to be notified because that's an again something abnormal or or comparing to the best to to the same day uh in the week last week. And then you have also ranking. So you can let's say tell me my best suppliers

where I'm sending my calls and stuff like that. And then you export outside and you can put it to to our own database or you can put to any kind of database outside of our system. Um session processing we have some states here which are important because I told you we get events. So for example from asterics we get events as well. So we have five different

states in the in this in our system. So one will be session authorization and that's before routting. So that's before dial plan before the dial application in in asterics. And here we can return maximum session duration. We can do resource authorization. We can do um um session uh properties adding session properties for example PayPal uh address if you want to keep it stored in the session whatever

and also rooting if you want and then we get the session start which marks us that we can start charging in real time if we if we want to session update is not used with with asterics it's used for example in mobile world with diameter updates and session terminate which tells us the the session was over. So we can stop the all the processes of realtime charging

and then session CDR which writes the CDR uh in the database or forwards it to the to the next place. Then um now coming to the hot topic asterisk and CG rates um asterisk integration um um it's it's actually uh done it was done quite some time ago. So we have done it in 2016. So we were directly advised back then from um by Matthew Fredson and

um we are not asterisk experts. So we we just know how to restart asterics and how to put things in the dial plan a bit but that's all we we know unfortunately. However um this works and it's it's used by by people um uh we know and we can call them customers. So um everything is done via ARRI also ARRI was pretty fresh. So it was AMI

before and we were advised okay if you're starting now start with something which is new. So ARRI was was the thing and um back then we could not find a good Golang library. So we had to to write our own uh Golang library which we call aringo. which it's quite um it it sounds like vacation time for me for example Mexico and Spanish but it's just ARI

in go language so um it's is maintained by us so uh we can freely uh and fast uh fix its problems um calls are not authorized if suates is not uh operational which again it's the break before the engine which I I always appreciated. And we also are able to generate CDRs out of events with um extra fields from uh stat stasis attributes which we can feed

in at the start of the call. I'll show you how. And we can also do asterics integration purely CDR driven. So offline charging out of CSVs for example um generated I I hope is still actual CDR CSV module um how how we have done it because there were tricks uh since we could not um find everything what we needed back then and I think it's it's uh

actual we needed to to apply certain uh logic from asterisk and certain logic inside sigurate to make them work. So this is why I also I'm sharing it. Maybe you are in the same position you want today to implement a billing real time for yourself. This is what we have done to make it work. So um when the the um channel uh when when the call starts

we were not able to feed in via dial plan um our extra information from CG rates. So we had to to to or or to send it to the module which we need. So we had to feed this via stasis application. So when calling stasis from dial plan, we are pushing extra variables which we need. For example, this is the CGR account and this call should be

prepaid or postpaid. So uh we'll send this via uh stasis. Then uh once this is inside CG rates we cache this information. So we we keep it like in parallel to what asterics has because when the CDR comes in in the channel uh destroy message we don't have all the the account information and all the the things which CG rates has set. So we can retrieve this

part from CG rates and uh it it works. So um also there is a two-way data exchange. So we are able to feed in additional information. For example, when doing routting uh we send with uh stasis start we send the request inside CG rates and then it will stay there forever until we send it back. Right? So from this point of view, it's secure. If it's not

authorized, it will not go through. It's it doesn't go to the next dial plan. Um but um it we can then also feed in the routting information. For example, we can specify the gateway where you should send the call. So gateway ID and then you capture this information because we feed it in via um post in channel variables. Nowadays with websocket this was the the change of

of um birectional ARRI. Before we used to set a post to an HTTP server of asterics which was part of ARRI but now we uh on the same um uh web soocket which is opened when the ARRI comes up uh we can feed in um uh instructions to asterics. So we will set uh or we will inject information like rooting and other information from the uh sig

rates or we we have gone through like using sig rates as a database if you want to um so we are feeding this information via when the call is disconnected from the CG rate side. So if we detect that the balance is finished, we will send a delete on a channels um a delete method again over the websocket and kill the call from CGA side. So if

fraud is detected or you shut down the account while the call is going through we are able to uh detect that because we do every 5 seconds uh debits and we will kill the call from the engine side by by just injecting over the websocket uh a delete message. Then um channel um is not sent after uh returning to dial plan and for that uh we use

um this um subscription. So we will be kept updated about what's happening to our uh channel. So the channel but only channels which we monitor for the billing side. So we subscribe to to information regarding uh what happens to the channels because otherwise we cannot do the billing also and um we will consider remember we have channel start incidurates when we need to start billing. So we

will have um channel state uh change received from um ARI uh and we need to have state up. So we know that the call started and then we can start charging and we will consider that the the call terminates when ARRI informs us with channel destroyed message and uh in channel destroyed we have uh CDR bill sec and this is the total call duration which we use

to bill for so um this I don't know if if it's um uh you are able to to uh read it from there otherwise I'll just share my slides and you use some glasses uh I also start having the problem nowadays so it's perfectly normal um so um on the left side you see this is part of the authorization so we get stasis start when when the

the call starts and then we will um send a rest request to the to the asterics to subscribe cribe for for news regarding this which is confirmed by asterics with rest response. Okay. Then we will send the rest requests to add channel variables as many as we want to the channel. And in the end we will send another rest request to to say um uh continue. So

we un uh we release the the call from stasis application and from there based on our subscription we just monitor we we don't interfere anymore. So you can route it in any kind uh you want we will just be notified that your call was started and then your call was ended. And of course as I told you we can also send delete instructions so we can kill

the call. And this is the the what we call accounting. So that's we receive channel state change with the status up saying that the the call started and then we we send uh later on we can send rest request delete the the the channel we get the channel hangup request and then we also get channel destroyed all the other events there will be a lot of other

events uh channel v set and whatever dial plan events and stuff we ignore. So um we still get them because we have subscribed to them. Uh we love to filter them out directly in asterics not to be sent to us but we don't know how. So and I don't know if it's that's possible. However, we ignore all the other um messages and it looks like complicated right?

Uh I I told you so so much information but the good news is that this is all you have to do uh in sigurates to get asterics uh connected inside su rates. Of course configuring sigates is another story. Um you enable an an asterisk agent. You set the its address user and password how many times it should attempt to connect and ARI websocket true. If you put

it on false, we will still work with the old uh asterisk version. So we will we work with both old and uh new way of doing ARRI. And you can also select here to create the CDR or you can still build a call out of u CSV. So you can do maybe rooting with sigurates and billing out of CSVs. You can select here CDR false. And in

asterisk as I told you um we initialize one channel variable named C CGR max session time which we later use here to to in the dial application in order to set the maximum call limit. And this is like a security timer for your call. Even if CG rates loses connectivity for some reason um and it doesn't detect the fraud or whatever your call is still limited to

initial security timer which is calculated when you start the call. So um yeah this these are uh things which you have to do. So uh here is this the stasis information. So you set some some um channel some some information which suates needs it. This is the exchange which I told you enabling also some of the modules of sigates. So you can select which functionality you want

from sig rates by setting these variables and then um you have here in the hangup handler uh what you what you do uh in when the call hangs up. And uh I'll just end up here with three minutes before. So this was really roughly calculated by me. I say I just put slides and let let me see David. I hope you don't hear that. Let me see

how much it goes. >> I think we have a uh a timer that you set for yourself just right. So to to end presentation, if anyone has any uh questions for uh Dan Christianos, uh happy to run the mic around. Oh, there >> Cool. Thank you. Um so for the real time part, you you mentioned that there's um security like where it'll reject the calls, right? If

the server CG rates down and it can't process. >> Yeah, actually it's not rejecting, but it will stay forever in statis. >> Okay, gotcha. So, is there an in between? Is there a hybrid where let's just say you have to take the server down or it crashes and then you want to continue the real time. Um, is if you start that back up, is the real time

now out of date? Can can it catch up based on CDR? Yeah. And backfill? >> Um, it no because um you have to be subscribed in order to get the channel updates. And if you if you were not subscribed when the call started, you have missed it. But you can also feed the the CDRs from CSV and then it will automatically process the ones which were not

processed by CGA. So you just dump everything what you have and most of them will say duplicated because we detect duplication based on the the channel ID and then we we reject it. >> Right. Okay. And then I had a part two of that communic real-time communication which was um with asterisk right are you are you directly listening to like dial events and things like that like

what about things that are asynchronous like originate um people doing click to call like would that still be able to run through like the engine >> anything which you can send to stasis so in your dial plan you call you need to call stasis in order for sigates to react otherwise we we don't see them >> and we We don't take responsibility for it. So today we

just do statis start I think they it's called. >> Yeah. So integrator would be responsible for their own like firing off CG rates when it's necessary. >> Yeah. And you you can do reload of your dial plan if you want to catch up uh and start sending. >> Gotcha. >> At some point. >> Yeah. Good. >> Great. Great question. Does anybody else have a question for Dan?

Okay. Well, um, we're right at 2:30 on schedule. Thanks again, Dan. >> Thank you very much. Thanks. >> We'll take about a 15minute break. Uh, looks like we still have some coffee, but snacks may have disappeared. So, unless they were gone, uh, that was your chance. I see one or two bags of chips left there. Some coffee and tea. Uh we'll get back started at 2:45. Uh

then we have a couple of talks and one uh remote presentation from Josh about performance improvement to wrap out the end of the day at 5:00. So if you mosy on back in 15 minutes, we'll get started for the last couple hours. Thank you. I think so. Okay, everybody. I think we're going to get started here in just a minute. You can find a seat. Still a

little bit of coffee left. I think the snacks are gone. There's water, too. So, feel free to got another couple hours here of uh presentations till 5. Um, again, my name advocate at Sangoma. Welcome back to Happy to have you here. We've got a number of talks about free PBX as well. Of course, one of the most open uh popular open source guies for asterisk. Been around

for a number of years. May have seen a few Tango the frog mascots out front. Uh be sure to get your picture. just just posed for some myself with the scale penguin and the asterisk uh kit the asteris mascot and tango the free PBX mascot all there to take your picture with smiling happily I am excited about this talk I've spent uh quite a lot of years

before uh working with asterisk and and some free PBX and a lot of more time with free PBX these days at Sangoma and come to find out that Rajan Patel has worked worked on some free PBX on Umbbuttu while we've been working on free PBX on Debian showing what a great option it can be for a variety of different operating systems. So I'm very excited about this

talk. Uh if you can uh bring your attention over to infrastructure as code for rapid free PBX deployments. Hand over to you Rajan. >> Thank you Chris. >> Can everyone hear me? Okay. All right. Mic seems like it's good. So uh I'm Rajin Patel. I'm an asterisk and free PBX evangelist and I love the project. I think more people should use it. And over the years I've

installed free PBX and asterisk a lot. My favorite place to install it is on Google Cloud because of the free E2 micro virtual machine that you can get with two shared vCPUs, one core and one GB of RAM. It costs zero dollars to run. And if you spin up a Debian image on an E2 micro instance and run the installation script, it will fail. But today, I

will show you how to get it going. So in 2023, uh people were trying to launch virtual machines on Oracle cloud to to use the uh extremely generous free ARM tier to run free PBX and asterisk. And Oracle does not provide Debian images. So users opted for Ubuntu 2204 to install free PBX and asterisk and there were some subtle differences between Debian and Ubuntu and I knew

where people would get stuck. Um so I authored some infrastructure as code deployment templates so people could launch free PBX and asterisk on Ubuntu on any architecture on any cloud and it's guaranteed to work even on the smallest VM offering anywhere without any friction. So, I've been maintaining these infrastructure as code templates since 2022. And the path that I see most people take when they want to

install free PBX and asterisk, they launch their favorite OS. They SSH in and then they follow the documentation to bang out a free PBX and asterisk installation. And then when they're done, they configure it and then think, I should bookmark this spot because if something goes sideways, I can come back to this moment and then pick up where I left off. So you take a disk snapshot.

So disk snapshots are bulky. They're impossible to reconfigure without redeploying and taking new snapshots. And from a security patching standpoint, like a golden image is going to become stale the day you make it. So if you have a stale snapshot, you can take a new snapshot. It's not elegant, but it works. You still have the same problem, and now you have a growing catalog of snapshots. So,

this is a bad workflow and I think it's rooted from the days of antiquity. If your free PBX went offline due to a vulnerability and you're using images or disk snapshots, you're relaunching a vulnerable system. That's awful. And what if you made five or six changes in Free PBX since your last snapshot was taken? Now, you have to replay those changes on the newly launched still vulnerable

system. So, that's double awful. And if you want to do this for many free PBX installations, this can become a full-time maintenance job. So with infrastructure as code, your business can sell to enterprise customers already using this technology. So in the world of disaster recovery, speed is your best friend and ambiguity is your worst enemy. So what if you could de declaratively configure the desired state of

your OS and all the software installed on it in one place and it would install and configure free PBX and asterisk on any future revision of your OS. So this solution is going to remove the guesswork from rebuilding Now these are some of the benefits you get with infrastructure as code and if you're not an expert on this topic already you will be in the next half

hour. So if the idea of infrastructure as code scares you, remember that most of this is just JSON and YAML. It is both machine and human readable. Cloudinit takes an initial configuration that you supply in a YAML file and it automatically applies those settings when the instance is created. It's it's rather like writing a to-do and then let Cloudinit deal with the list for you. You can

even add conditional logic with variables, if else statements and loops and then you can use Ginga templating to accomplish this. It is the industry standard for Linux instance initialization. It's used by every major cloud, every major virtualization platform and it's a canonical product since 2007 with active participation from the likes of Red Hat, VMware, Microsoft, Amazon and others. So this is my hello world equivalent for a

cloudinit file. The the pound pound template col ginger header tells cloud init to treat this as a dynamic script rather than a static file. And that curly brace percentage set that tag is a ginger statement and that creates a temporary variable during the rendering phase. So essentially cloud init processes the logic first strips out the ginger code and generates a clean custom YAML file that's tailored for

the specific instance before any configuration actually begins. So I like to set my variables first at the top of the cloudinit file and then I use those variables in the cloudinit file later on. So this is like a fill-in-the-blanks template because your configurations would change from one free PBX deployment to the next. And many people will store sensitive information in a vault. It's possible to retrieve secrets

from your cloud or hypervisor's vault if you were inclined. I didn't want to add that complexity to this uh cloudinit file here, but it is possible to do. So the first thing we'll configure is uh email. The server's cron tab can send email alerts. Free PBX needs to send voicemails and update notifications. So let's set the SMTP service provider credentials here. And then later we'll instruct Cloudinit

to use them when installing and configuring Postfix. Again, you could use Ginga to populate this variable with a secret from a vault, especially if you're versioning this file on GitHub. Now the host name and fully qualified domain name are needed to set networking and mail configurations. By setting the host name and FQDN as ginger variables here, you ensure that services like postfix are configured appropriately and you're

preventing mail delivery failures caused by domain name mismatches. You can use conditional logic in cloudinit. So here I'm using booleans to define which trunks should be enabled or disabled in free PVX and where I should download my backup tarball. So this allows Cloudinit to skip entire blocks of configuration for disabled providers, ensuring only what you intend to use is installed and configured. This will work in conjunction

with the free PBX backup and restore module. And we'll get to that in a moment. So you're going to want to use fail toban to block bad actors. So you can set IPs that require unrestricted access like your management subnet and SIP endpoints. This ensures that fail toban never blocks legitimate networks. If there was a human error and a device with the wrong username and password was

deployed, retries are going to keep reoccurring and then failtoband could block a large number of devices if this allow list isn't added. Later, we'll also itemize SIP provider IPs and add them to our failtoban configuration. Be a little silly if we blocked one of our Now, unless you need time to be tracked in UTC, it's a good idea to set your local time zone. reading time stamps

of scheduled tasks and call records. It makes more sense in your local time. Now consider when you want the maintenance window to occur. So for your automated security updates, uh the time specified here, it's going to be used to configure the unattended upgrades package and the needs restart packages. Optionally, you can disable unattended upgrades if you're using something like landscape to manage your estate. Now longunning free

PBX instances can use all the available disk space with comedic results. Everything goes offline. So this section defines the database cleanup policy for your CDR and CL records. And by setting these ginger variables, you're providing the parameters for a SQL command that will periodically run via cron. Consider your retention requirements and then adjust this accordingly. Now this is an Ubuntu specific configuration. If you're using Debian, you

don't need this. So, I'll use this variable as a version control toggle for PHP further down in the cloudinet file. Now, this is the end of our variable configurations. We're going to move on to the meat and potatoes of cloudinit. So, this is a list of tupils in Ginga. It's a type of dynamic data structure. The purpose of this is to map SIP trunk providers to their

enabled or disabled status and their network ranges. So by grouping the provider name and its activation flag like enable flow route and its signaling IPs together, you enable cloudinit to programmatically loop through this list and then the script can automatically build firewall rules and SIP trunk configurations only for the providers that you've marked as true, keeping your security clean and uh automated. So the for loop on

the next slide is going to use this list of tupils. So we're using ginger logic to programmatically build a security whitelist. It starts with your user IPs and then it loops through the providers list to append the signaling IPs only for the services that you previously marked as enabled. And finally it's using a ginga filter. If you notice that pipe join that flattens the list into a

single spaceepparated string and this is perfectly tailored for the ignore list for failtob. So all the SIP providers that you're using and then the users that you've configured for your free PBX install, they're all going to be prevented from being banned by fail to ban. Now this section uses Ginga string manipulation to dynamically extract a base domain from your fully qualified domain name by splitting the string

at the dots and grabbing the last two segments. So for example, if you've if you've got free PBX.subdomain.yoump.com, your company.com. It can extract your company.com and it programmatically identifies your root domain. Um, and finally, it's using ginger expressions, the uh curly brace, curly brace, um, and then the variable name that injects these processed values directly into the standard cloudinit host name and FQDN directives, ensuring the OS

identity is set correctly. Now, of course, you could skip these Ginga gymnastics and just write the host name and FQDN into those variables, but then you wouldn't have learned about this whole Ginger string manipulation thing. Um, so I'm just showing this as an example of what could be done. You probably would have other use cases. Now, Cloudinet has modules and these modules are run in a specified

order during startup. So, resist the urge to use the run command module, run cmd. That's where you can run arbitrary commands and you can do everything but the best practices are use the native modules as much as possible and then use the run command section as sparingly as possible. So this is the users module in cloud init. It's used to create the service account for asterisk. And

by defining the asterisk user with passwordless pseudo privileges and setting lockd password to true, you're ensuring the service has administrative permissions that it needs while strictly enforcing SSH keyonly access for better hardening. Now the cloudinit write files module can write files anywhere on Linux. I'm using it to create a systemd override for the apt daily upgrade timer and to set a security patching schedule of my choosing.

By injecting the security install time ginger variable directly into the oncal field, it forces automatic upgrades to run exactly when you specified in the variables earlier. This effectively replaces the default random schedule with a predictable maintenance window ensuring that your your PBX doesn't start updating during high call volume hours. This is a continuation of the write files module. Um, here we're writing the postfix credentials file and

by mapping your previously defined SMTP variables directly into the SAS format or SASL, however you're supposed to say that, um, Cloudinit creates a sensitive configuration file with restrictive 0400 permissions, which is readon for the root. And this ensures that your mail relay can securely authenticate with providers like Send Grid or Gmail if it's if it's if you're running like a super free version of this thing. Now

we also write a systemd service unit to manage the free PBX life cycle. This is quite standard as far as systemd goes. I can come back to this slide if you guys have questions. But by defining the after equals mariah mariab.service dependency, it ensures that free PVX only attempts to start once the database is ready using the fw console command to cleanly initialize or stop the PBX

services during system boots and shutdowns. Now this is still in the cloudinit write files module. This is an Ubuntu specific step and not necessary on Debian. But by creating the uh etsybcins.ini file here, you're building the bridge that allows asterisk to communicate with Maria DB. So here we're using cloudinit write files to set a custom log rotate configuration preventing asterisk and free PBX logs from consuming all

the available disk space. I've encountered that pleasant pleasantry as well. Um, I've had free PBX installations go sideways because the logs chewed up the entire disc. So, follow your retention policies and be mindful to use postrotate script um, a postrotate script for asterisk to to reload its logger. This is going to ensure that asterisk seamlessly switches to new log files without needing a service restart. Now, fail

to ban jails are defined through configuration files. So here we've created jails and set an allow list of IPs to ignore from our fancy loop earlier and specified where email notifications should go. Now I don't want email notifications for SSHD. I'd get emails constantly, but I am curious to get email notifications about brute force attacks on free PBX and asterisk. I'm looking at the asterisk security log

and the free PBX security log for three failed attempts within two minutes before I block IPs automatically for an hour. Now the percent uh parenthesis action_MW macro that ensures that whenever a block occurs an email notification is sent to the address that we defined earlier in our ginger Now fail to ban will fall over and cry if the log files for asterisk and free PVX don't exist.

So the cloudinit write files module to the rescue. We create empty log files with the appropriate permissions. And while failtoban can understand asterisk logs, we need to teach failtoban how to read free PBX logs. So we write a filter file with a custom reax pattern that fail toban uses on the free PBX log file. Now if you're reverse engineering the reax, you know that we're looking for

the string authentication failure and extracting uh the host which is the IP address. So it can be passed to the firewall for blocking. Now by using Ginga conditionals and variables, we write a cron tab ready file that manages three critical areas. We've got recording cleanup, voicemail rotation, and database optimization. So for recording cleanup, it's going to delete the directory structures for call recordings older than one year,

and it's going to prune all the empty or tiny ghost audio files. I have no idea what they are, but I see them in there. Then there's voicemail rotation. So let's automatically remove all old voicemail messages after 45 days to save storage. But uh again follow your retention policies. And then database optimization. So this uses the CDR retention days and the CL retention days variables that we

set earlier to purge old call logs and optimize the MARB tables. So this is going to reclaim disk space and keep the database queries fast. And later on using the cloudit run command module, this schedule.txt file that we have written, it's going to be installed into the appropriate chronab. Now the apt module in cloudinit can set apt configurations on your server. And here I'm configuring unattended upgrades

with the apt colon periodic settings. So these ensure that the OS automatically checks for updates and downloads security patches and cleans up the package cache every seven days. I can also configure repositories with conditional logic. Am I happy with PHP 8.3 or do I want PHP 8.2? Now, this is an Ubuntu concern. It's not something that you have to worry about on Debian. I'm using Ubuntu and

I want Canonicle to security maintain my PHP. So, I'll install 8.3 from Ubuntu's repository. But if you want the same version that is on Debian, you can install 8.2 from this Debian maintainers PPA. So using a ginga if block it adds the PPA repository only if you spec specifically chose PHP version 8.2. These are pretty easy. Package update. So that refreshes the local package index to ensure

you're pulling the latest versions of the software. It's the equivalent of running apt update. Package upgrade. Uh that upgrades all the installed packages to the latest available versions. It's the equivalent of running apt upgrade and package reboot if required. So this detects if there's a kernel update or a critical library change that requires a restart and if so cloudinit will automatically reboot the instance and then resume

where it left off to perform the remaining configurations. Now this section uses conditional ginger logic to optionally add the Ubuntu pro u token to your server. This is of course only relevant for Ubuntu users. If you're using Debian and you you know how if statements work, you can close your eyes. But if you provide a token, Cloudinit automatically attaches the machine to your to your canonical account

and enables three enterprisegrade security features. First is live patch, which is going to apply critical kernel patches in real time without requiring a system reboot. This is vital for kernel security between your normally scheduled security maintenance windows. The next is ESM infra which extends the life of your Ubuntu LTS beyond 5 years. We security maintain Ubuntu LTS for 15 years now and ESM apps which expands security

patching to include all the open source applications in the Ubuntu universe including asterisk and all its dependencies. So if you're deploying this to an environment that conforms to the US government hardening frameworks, then you can add FIPS updates to this list here as well. And that's how you're government ready uh hardened Ubuntu installation with with asterisk and free PBX on it. Now using Ginga conditionals, we can

validate the uh time zone variable before applying it to the system. Here we're just checking if a forward slash exists such as in America New York or Europelondon. That if condition ensures the value follows the AA time zone And this is the cloudinit packages module. It's a slim down list of everything required to run asterisk and free PBX. So cloudinit automatically handles the installation of these packages

and their dependencies. But it's worth noting that I'm installing asterisk as a deb package provided by the OS. It won't be the latest version, but on Ubuntu, it's the version that is security maintained by Canonicle. Now, this is a continuation of that list of packages that will be installed. We're using Ginger variable interpolation to dynamically install the PHP environment specified in the PHP version variable. The PHP

extensions will only work for their corresponding PHP runtime. The version numbers must align for both. A PHP 8.2 extension won't work for PHP 8.3. Now we're at the run command module. Everybody's favorite. It's executed last. Now I use run command to do a variety of things starting with failtoban configurations. So first I use curl to reach out to Amazon's check IP service and determine the public IPv4

address of the server and tell failtoban to never block itself. Now these commands are for Ubuntu only. In public cloud environments like AWS, Azure or GCP where Ubuntu Pro might already be pre-activated or attached at the image level. People won't be providing tokens in the cloud init file. We still want live patch and expanded security maintenance to be enabled. So these commands ensure that those entitlements are

turned on. Now by default, Ubuntu will never reboot itself after applying a security update. It waits for human interaction. Now, I specified a time that I would find this reboot to be acceptable. So, I'm making these changes to the default behavior of needs restart. It's a config change and then restart the needs restart service. That's a bit of a mouthful. Now, these changes to unattended upgrades reflect

my personal preferences. I want to set it and forget it. The system should update itself but at a predictable time and old unused kernels shouldn't be stored in perpetuity because my slashboot doesn't have unlimited space. I also enable automatic reboots and I used the security reboot time ginger variable to schedule those restarts at my specified Now this IP tables command performs a flush and save of the

current Linux firewall rules. when I say flush removes everything and then we save it. So it first checks if it if any active chains exist and if they do IP table-f clears them and net filter persist save ensures that the reset survives a reboot. Now, while this is specifically targeted at Oracle Cloud, which is known for deploying a highly restrictive default rule that blocks all traffic, this

clean slate approach is a safeguard for any custom or vendor specific image that might include preconfigured IP tables blocks. I think the firewall rules probably belong, you know, where where the uh public cloud is allowing you to configure them, not inside the VM itself. uh but it's going to be external. Now this is a silver bullet configuration for tiny virtual machines on public cloud. You don't get

swap space by default and 1 gigbyte of RAM is very small. But configuring a two gigabyte swap file provides a memory safety net for free PBX and by using focate MK swap and swap-on the script is going to create virtual RAM discs and then prevent the um asterisk or Mariah DB services from crashing when the physical memory becomes momentarily exhausted. The final GP and said commands ensure

this configuration is persistent automatically reenabling the swap space every time the server reboots. Now this block of commands configures postfix and then deletes the plain text sassel sasl password file for security. Now postfix is restarted at the end to apply the changes and your server can now send emails. So if you're enabling FIPS to run free PBX and asterisk in an environment regulated by the US government,

Ubuntu is probably your only sensible path forward, you need to have a FIPS kernel. You need to have FIPS enabled Node.js. You have to have all the correct cryptographic library versions and a number of other tweaks that can be applied by just adding three lines to this cloudet YAML file. But I'm happy to answer more questions about that if there is interest. Now by recursively changing the

ownership across the slvarlib, the slashvar log and the and the /var spool directories to the asterisk user and group it ensures the asterisk service has the permissions needed to manage configuration files, store voicemails and write logs. The said commands then modify the system defaults and the core asterisk configuration file to explicitly instruct the processes to run as a non-privileged user rather than root. So this hardens the

system against Now this is some sim link magic only relevant for Ubuntu if you're on Debian. Don't worry about this and more PHP version shenanigans. You can ignore this if you're on Debian, but I do recommend the changes to PHP.ini. This increases the upload limit in memory. Uh, this section also configures Apache to run as the asterisk user and enables mod rewrite and clears the default index

page to make way for free Now it's time to download and install free PBX. Uh, there's a race condition between asterisk start and the free PBX install instructions. So waiting for the co core show version to produce a valid response guarantees that your installation doesn't fail. On iuntu you must perform a database path correction within the ODBC.ini configuration. Remapping the socket path from the default varibql myql.sock

is necessary because Ubuntu uses run mysqldd mysqld.sock. Small change but without it it's not going to work. Now, these are some modules I've tested and used on Ubuntu. Uh, you can install Sangoma's commercial modules this exact same way. There may be a few additional licensing steps, but I'll defer to them in the Q&A about that. This command creates the necessary SIM link to ensure that free PBX

starts automatically every time the server Using chronab, we can mount the jobs we previously wrote to schedule.txt. And you can use conditional logic to decide which backup file is going to be used for your free PBX deployment. You can use your baseline configuration or perhaps you have a point in time backup that you're trying to restore. And restoring from a backup is easy. FW console backup-restore and

then the path to your backup file. Now, if you're anal about keeping things clean, you can run my SQL commands using FW console. So I use it to remove trunks that I may have restored from my backup but I don't plan on using. Again, purely optional. I think it's important to apply this T38 fax over IP setting in the database which sets the error correction to redundancy.

This is essential for reliable and accurate faxing and in my view should be the default setting. Uh the subsequent FW console commands serve as the cleanup crew. Chone fixes any remaining file permission issues. Reload generates the final configuration files from the database and tells asterisk to read them. And restart performs a full service bounce to ensure all modules including the freshly configured zip settings are running in

a clean state. So what now? I want you all to try this yourself and see how easy it is. It costs you nothing. Search for free PBX Ubuntu on Google and you will find this or you will find my article where I talk about faxing and ECC with asterisk and free PBX. Now the GitHub repo is linked from the article. Now if you're kicking the tires with

Cloudinit on Ubuntu, please use Ubuntu Pro. Personal users get free tokens and businesses kicking the tires can use these free tokens for evaluation purposes. Just don't spin up insecure software. On the current Ubuntu LTS, so 2404, Canonicle has published security updates for the following asterisk and free PBX dependencies. They're only installable if you have an Ubuntu Pro token attached to your machine. And I'm I'm busting the

misconception that Ubuntu Pro is only necessary after the five years of standard support. Um, it's relevant on day one of the Ubuntu LTS release because if you're installing any open source package from Universe, you want to get the security maintained versions of those packages that are available to Ubuntu Pro users. So now you're all cloudinit experts. Thank you. And if you guys have questions, I'll take them.

>> All right. Thanks very much, Chris. >> Yep. Uh, we've got some over here. Yeah. Thanks. Thanks Rajan. Great presentation. Come over here. >> Thank you very much. Um, have you uh looked at deploying on any of the uh BSDs like a FreeBSD open? >> I personally have not, but I'm sure if you were motivated enough to, you could figure out the corner cases and get it

>> Awesome. Yeah, that'd be fun. Uh, let's see. Yep. Right here. >> Hi. Have you uh made a container image from anything that you built this way? >> I haven't built a like an OCI container. >> Docker. >> Oh, a docker container. Yeah. No, I haven't built any containers with this, but you could. There's nothing stopping you from from creating a cloudnet template, >> spinning something up

and creating a creating a container from from the Um if you have if you're using LexD um then you can use cloud init to spin up LexD uh system containers if you're using multipass just to kick the tires on Ubuntu on any OS Mac OS uh Windows or or any other Linux then multipass will also let you launch VMs with a cloudinet YAML file. So wherever you're

spinning up a virtual machine and it doesn't have to be Ubuntu, you can supply a cloudinit YAML configuration file and it will be the thing that defines how that Linux is going to look once it's launched. >> we uh included some cloudmit in our fog ISO for free PBX. So it's a little friendlier now. We'll talk about that after, I guess. Uh any other questions uh for

Jean about uh installing free PBX on Umbutu Debian's favorite cousin maybe I suppose. Yeah. So all right um thanks so >> All right. Thank you all. So, I think we're going to have just a few minutes here before uh David's talk begins and let him get situated and also take a look at uh some contrast on our small screen. If you'd like to see closer, feel free

to mosy on up. Uh avoided making any serious money by geeking out over asterisk and playing around with it. I think we're in the same position with VCON. can get excited about creating them, but actually the real money will be made in the way that VCON are used because they can provide a memory of conversations. Now, if you're worried about the way that VCON will be used

in terms of data protection, authorization, uh, signing, and all that kind of stuff, there's a parallel standard called SKIT, and SKIT stands for supply chain integrity, transparency, and trust. And all of that is on the IETF as well. So, there's I'll give you another link at the end about that. So these are kind of parallel things that the IETF is dealing with. And when I say the

ITF, one of the prime movers is a guy called Thomas McCarthy How who's actually started a company called VConic recently about this. Now these AI tools, the so these are tools that could be using VCON. What could they do? Well, you know, obviously the world's your oyster. However, here's a few examples. Customer service insights, understanding the way customer service journeys work, optimizing them, making it better for

the customers, behavioral predictors like when people are most likely to buy in a sales cycle, sales growth and acceleration, customer delight and retention, bo, everybody loves a bit of that, and identifying new product and service opportunities as well. Really, the possibilities are totally limitless once you've got all of this data into the right place to be able to analyze it and work with it. And then you

can point your AI tool in any direction that you would care to. Now, how can we deal with or sorry, how can we create VCON in the world of asterisk? Well, you can do it entirely from within the dial plan. That's what I'm going to show you because that's what I experimented with first just to make sure I could do it. However, in production, you probably wouldn't

want to do that because there's a few moving parts that could actually if there was some poor connectivity or something like that that could slow up your call processing. You don't want to do that. You're much better off calling a script and then moving on. And that's the next version is calling a script. And then the last way is going to be looking at the carrier doing

it for you. So now this is when we actually jump into the Dell plan. And forgive my total lack of use of different colors here to highlight different things in the dial plan. But first of all, we're setting some global variables. And as you know, global variables can be used anywhere in any channel at any part of the dial plan. And we're just setting up a directory

for where our VCON are going to be stored and for where our call recordings that are associated with those VCOons are going to be stored. By the way, if at any time something comes up on the screen that you want to talk about or you want to ask questions about, just raise your hand and we'll we'll talk about it. I'm all for just in time questions and

stuff like that. Okay. Then we've got our demo extension. As you can see, it's very basic dial plan here. Uh I'm calling a noop just to tell everybody what I'm doing. And then I'm setting a variable called vcon start epoch and basing it on the Linux epoch that asterisk has pulled through there. Who can tell me what that double underscore at the beginning of the variable means?

Yes, Alexandra. >> Yes, I like what you're saying. The description I use is infinitely inheritable. You know, one underscore means it's going to be inheritable to the next channel. Two underscores at the beginning means it's infinitely inheritable. Doesn't matter how often you change channels, it's going to be there. Okay, so we're setting up an infinitely inheritable variable. Then we're setting a particular parameter of a function called

channel and it's the hangup handler push parameter and that's actually wanting um the uh direction of a go sub to go to. So what we're saying is the the sub routine we're going to go to is in a context called lab vcon hh for hangup handler extension s priority one. Then we're just using the system to call on Linux to make a couple of directories there. As

you can see, the VCON directory and the VCON record directory. And we just set those previously, didn't we, in the global variables. Everybody with me so far. >> Good. Okay. Love it. All right. Then we're setting some more inherited in infinitely inheritable variables um for the uh name of the the linked ID and the uh the the the actual recording there as well. And you can see

that we're using mix monitor. Good old mix monitor. Everybody loves a little bit of mix monitor. And for an extra chocolate, what does the B option mean in mix monitor? >> Go on, Mark. >> Record. >> Only record when it's bridge. So we don't get any rubbish in the recording. Did anybody else know that? Of course you did. You did know it, didn't you? Really? You really

didn't know. You all knew it. Really? Oh, sorry. Don't want to don't want to hurt anybody here. Okay, so that's uh making the recording. Then we're answering the call. Then we're waiting for 5 seconds. Then we're hanging up. So all pretty basic stuff. So now let's get to the hangup handler. And you can see then that we're setting the uh the end of the uh the end

epoch and able to get the duration as well by having one minus the other there. And then we've got dealing with the timestamps and we're setting the uh linked ID and the UU ID. And then this is where the meat is. So this is all still within the dial plan. You might think this is a bit of a waste of a D plan really, but what we're

doing here is we are very carefully using print f to create a complete file. So, do you remember this one? Hold on. Let's have a little look at it. It was that that's what we're creating here, isn't it? We're creating the JSON from within our dial plan. So, let's go back to where we were. Yes. So you can see there um standard notation the backslash for new

line and all that kind of stuff. We're creating the entire VCOM there and at the end of it we're putting it all in a file. Can you see the penultimate line there? We're putting it all in a file just called temp tm dollar tmp in the in there. And then we're using the move um to move it across into uh the right directory under the right ID

as a JSON file. Just while we're here, and since we are at scale, which is a Linux place, what's the difference between a move and a copy in the world of Linux? You've already answered the question, Mark. I see you primed there on the edge of your seat. >> Go ahead. Ex. >> Yes. Ex. Exactly. For for practical for practical purposes, a move is not going to

announce it to the file system until it's all there, isn't it? Where a copy, you could accidentally get a halfbaked thing if you're not careful. Good. Okay. Thank you. So then we're coming towards the end of the example. We've just got a little noop and then look, we've got a return because remember it was a go sub, wasn't it? We were calling a go sub as the

hang-up handler. And so that is the return of it I've just shown you what not to do. The reason I've shown you that is just so you can prove to yourself, and that's what I wanted to do first, just to have a little hack about to actually prove I could create a VCON that looked in the right format and had the right information in the right places

to uh be able to pass out to anybody appropriate. And it works works great. Asterisk will create the VCON for you. However, in production, really what you're going to do is call a script. And the many advantages of calling a script include number one is a very neat thing to do, isn't it? You're just going to call it could be calling it with an AGI or a

fast AGI. Uh, however you want to do that. Um, it's nonblocking in terms of the call traffic and of course you get to use a language of your choice. Um, I did it in shell script. Better people than me would have done it in Python or PHP or or something like that. Um, and you want two scripts really. You want the script to actually create the VCON

as we've seen and then you want a script to sign it for authentication purposes because you don't just want any person or anybody being able to get into a VCON because once that VCON is created it's immutable. You can't edit a VCON. Once it's created it's there. What you can do is add to it afterwards and it's like an appendage if you like to the VCON. But

what you can't do is change a VCON. So that's the scripting way and the scripting way I would suggest is the precise way you would like to do it um in production. Now I could have included an example to do that here but it would have been you know a long script and been difficult to show here. Suffice to say that if you'd like to see a

script I've got one in from my experimentation I'd be very happy to share it with you and you're welcome to ask me afterwards. However, as I'm sure all of you have probably already guessed and and maybe even experimented with, AI knows all about VCON. Now, I wouldn't call myself a power user, but I can tell you that in my extensive work with chat GPT, it knows all

about VCON. So, when I say write me a script to create a VCON, it can do it pretty well. And if it doesn't, and you probably had this experime experience yourself, if it doesn't work, you can go it don't work. And it goes, all right, let's fault find it together. Let's troubleshoot it together. So, you know, it's pretty straightforward thing to do. Now, the alternative to having

Asterisk create your VCON is to use a carrier that can create VCON for you. Well, number one, it saves you the work of actually doing it, doesn't it? If the carrier is going to create a VCON for you, whoever you're using for your SIP trunk provision. Um, and it gives you access to other things that might not be available at the time when asterisk would have wanted

to create the VCON. And those things could be a full transcription of the call that might not be immediately available. It could also be an AI based sentiment analysis of the call. There could be some keyword spotting. There could be all sorts of different things going on there. Uh, compliance scoring for contact centers. By the way, isn't it interesting that contact centers what they're going to look

like in the world of AI, agentic AI? We need never hear the old all of our representatives are busy at the moment. You know, you never need to hear that again. And of course, it turns out that actually the main problems with contact centers are people. training the people, managing the people, making sure the people don't take too long for lunch or go for smoke breaks when

they're not meant to. The all of the problems associated with contact centers pretty much a people. If you've got an agentic AI, there's going to be consistency. I'm not saying it should be doing people out of a job. It should be allowing them to then work on maybe the edge cases or as I've noticed, it's very popular here to say corner cases. Let's say both. can they

can work on the edge cases and the And if you insist on using asterisk to do it and as I said not ideal for production, but if you did then you can have asterisk create the um vcon and then you can append things to it afterwards. And you can see the the proper name for it is additional artifacts. So you've got a VCON of the basic information

that Asteris created or that the script created at the time and then you can add things to it afterwards. You can append artifacts to it, additional artifacts. Now the other thing about uh a carrier is of course when we all started the carriers were SIP carriers, weren't they? They did SIP stuff or maybe they did H323 stuff back in the day. Do you remember that in the

early 2000s? Bit like VHS and Betamax. weren't entirely sure whether it was going to be H323 or SIP. Personally, this is how much of a geek I am. I've still got a couple of H323 gateways and I still got O323 running in my asterisk and I just like to fuff about with H323 gateways every now and again just for fun. However, these days um your carrier is

perhaps not only going to support SIP, maybe they support WhatsApp integration as well. So now you've got VCON from WhatsApp conversations with AIS or WhatsApp conversations with personto person. Maybe it's text messaging. Maybe it's a WhatsApp call that you've got a VCON for as well. So if you're using a carrier that supports WhatsApp integration, now you've got a VCON for the SIP based calling part. You've got

a VCON for the WhatsApp part. Maybe you got it for Teams as well. If your carrier supports Teams integrations, maybe you got it for Zoom. In other words, you can have a multiplicity of channels, have all the VCON information, and with the right AI tools. Well, you can see everything, can't you? You can see everything. Not only from this bunch of conversations, but from that bunch of

conversations, you can probably root out the entire history from when a person first made contact to when they bought the automobile or or whatever it might be. And of course at the carrier level then generally speaking if a carrier is going to create your vicons for you they can probably allow other people access to them that are authorized. In fact I'd go so far as to say

that when a carrier creates a VCON they probably don't want to be responsible for storing it. They probably want to send it to where you want it and leave it for that because of course there's implications if they are the the conserver for this information. So using your uh carrier to generate VCONS is also a great thing as well. Now I've got a summary here, but before

I get to the summary, has anybody got any questions they'd like to ask about VCON? Go ahead, sir. Young Chris will run a microphone to you. >> Here you are. Yeah, I guess one thing I'm confused about is if if it's going to be signed at the time that it's created and in the future you you might want to change who has access to it or take

away access. >> Yes, absolutely. >> And in which case do you attach another piece to it? Is that and then sign that? >> I don't know, sir. >> Fair. Fair. I'm old enough to be honest and say I believe that within skit the supply chain integrity transparency and trust there's probably a mechanism for do that you you've got to be able to withdraw your right to be

part of it as a person as well as as you said all the other changes and I do know that there's mechanisms in there and that's I don't understand what mechanisms are but they're definitely there and that's one of the different things that before you got all these diverse things you don't really know what you're doing this is going to be a universal standard that allows everybody

to do it so the mechanism is there. I just can't tell >> great question. Anyone else for a >> question on fans? >> Is there a minimum version of asterisk that you need to be using? >> No, not not to do the experiment that I showed you because that's just very very basic dial plan stuff there. So whether you're running I wouldn't suggest 1.2 or 1.6, six,

but if you're on a 20 or thereabouts, uh, yeah, you'll be close >> Good question. >> That's one from Mark there. It's going to be a difficult one from Mark. >> Um, what major carriers are currently using VCON that you know of because you mentioned Zoom and Teams, but but are they actually using this yet? >> Okay, I can Yeah, again, I will tell you the things

I know. Simwood themselves have VCON in development. It's a Q1 2026 production goal and they're on it right now. Sansi, the people that do session border controllers are actually building VCON production into the silicon or the silicone I should say of their uh equipment. But more than that, I can't tell you. But let I will I will I'm going to point you to somewhere in a minute,

Mark, and when it comes up, I'll go Mark, this is for you where you can find that stuff out. How does that sound? >> Good. Okay. Any other questions before we move on? Got a few more things to go >> Okay, in that case, let's jump into the summary. So, VCON are going to be incredibly important at this intersection of real time communications and AI. And right

now, we're at the very start of this. So, imagine go back 20 odd years, maybe more than 20 years, and you're hearing about SIP for the first time. We're in that kind of space. And Jeff Pulver, who I mentioned, is very excited about VCON. If you follow him on LinkedIn and place like that, it's pretty much all he's talking about. And he believes he's he's quoted as

saying he believes that VCON will be as important, if not more so, for the telecoms industry than SIP was way back when. So, you know, he's setting a lot of store uh by this. Asterisk can create those simple vicons that I showed you from within the Dial plan. Feel free to have a play around, but it's probably not the best thing for production use. Um the best

practice is to have asterisk call a script or scripts to create your vicons and sign them properly. But I believe the real money will be made by those that do work with the vicons. But what that means for us, I mean maybe you can create something that works with the VCON, but what this means is just like you might be able to charge for call recordings, transcription,

sentiment analysis, you can charge for VCON production because they will be a very valuable source of data to the next layer up if you like. Okay. Further sources of information and right now Mark, this is for you. Look at the one at the very bottom. It says follow Jeff Paul for the VCON Foundation. The VCON Foundation has only recently been created, but there are a number of

industry players that have already joined up and I'll tell you the ones that I can remember. Grandstream, the phone and network gear manufacturer are a founding or a charter member of the VCON Foundation. Sansi that I just mentioned, I've run out of memory now. There's there's about 15 or 20 that are chartering members. And so if you go to the VCON Foundation and have a look that

will give you an idea as to the industry players that are already taking this very seriously. Is that okay? Good. All right. So s other sources of further information if you want to learn about VCON specifically then in addition to conserver IO there's the uh place that you can go to. The same for skit. Um all of the generative AIs know all about vcon. Personally, I've only

used Chat GPT, but it did a very good job in not only creating scripts for VCON, but analyzing them as well. And then, as I said, Jeff Pulver and the VCON Foundation, very much worth following. Now, I've got another thing that I'd like to talk to you about. There's a little QR code there if you'd like to know more about Simwood uh carrier. We support WhatsApp integration.

And we support agentic AI, conversational AI inside the carrier network as well. And in fact, if you'd like to see it, I've even got a demo where I can be messaging over WhatsApp to an AI agent. And when I say I want to speak to a doctor in this particular scenario, it goes, "All right, press the voice call button in this chat." And it roots the the

Simwood network routes that call through to a team's endpoint where my doctor is hanging out. So, I've got the WhatsApp messaging going one way and the WhatsApp voice calls going the other way, all without ever touching the PSTN. So, that's quite exciting. Anyway, there's a QR code if you'd like. The other thing I wanted to mention to you is, of course, we're here in the context of

scale, the wider Linux event, and I'm very uh excited to be a to be participating by delivering another talk on Sunday morning on the opensource careers day. So if you're a technical person that would like to further polish your ability to interact with non-technical people because you know geek to geek it's you know geek to geek always works doesn't it but when you get geek to non-geeek

especially when the non-geeeks are handling the money they've got the purse strings sometimes our communications don't always make the trip and so I'm doing a workshop on Sunday morning it's called humanto human connectivity It's all about the bandwidth. And so I'll be talking about seven different bandwidth boosters between geeks and non-geeeks. So the wider you can get that bandwidth, the more of your information can get across

to them and the more information you can see coming back from those people, whether it's buying signs or whether it's skepticism or whatever it is. Um, it's all based on a book I wrote called Be the Nerd That's Heard. Because one of the things that I saw in the Asterisk community is I saw so many really great people who knew exactly what they were talking about. They

really knew their stuff. But I just want to say this in closing that knowing your stuff is not It's the ability to present the right part of your stuff, not all of it. Because sometimes, you know what it's like when you get an electrician around, you ask for a quote and they go, he says, "Well, what I've got to do, I got to do this." You don't

really want to know everything he's got to do. You just want a price, don't you? And sometimes as geeks, we're very tempted to unload everything we know about something. But if you really want to communicate with full bandwidth to non-technical people, you need to find the relevant thing in what you know and to then communicate that in a way that can be easily received, digested, implemented, and

actioned by those nontechnical people. And that's what that workshop's all about. So, it's been a pleasure talking to you about VCON. It's been a pleasure going over some overview detail. We didn't go too deep into it, but I want the pleasure to come for you when you implement VCON and add another revenue stream to what you're already doing. >> Thanks, David. >> Was a great talk. do

love vcons and all that dial plan. Uh so last talk is a pre-recorded talk uh but we'll be live Q&A so it is from lead engineer of asterisk Josh Culp who we had some Q&A with this morning uh during the opening panel. he is uh it's getting pretty late for him, so we'll try to move this along. And Mike Berdine is going to connect and get the

uh Q&A situated as well. So, it's a little bit of talk about some performance improvements in Asterisk, which has been quite a focus recently. Uh we've been really polishing things up. Pretty excited to see uh you've got some improvements for CDRs. Seems relevant. uh got some improvements for AI of course uh you know back in the day it was speech to text now it's AI text to

speech now it's AI speech to speech that's definitely an AI thing right so unless we just mean talking to people but uh so we we've had some some areas that uh identified some bottlenecks and and had some hunches and there's been some experiments and some profiling that's really gone down and significantly improved things. So uh I'm going to give a few minutes here. We'll get started about

5 minutes the last talk of the day. So if you need one quick moment more to stretch your legs uh please please do this now and see you back here in quarter after. So 4 Hi everyone, welcome to my talk. Check, check, check. One, two, one, two. Yep. Good. Hello. Not that I could hear any of you. >> Josh, if you do talk, um, they will hear

you over the presentation. So, uh, you might want to mute until the video is Okay, I think we're going to get >> Can you hear me? Is that coming through? >> I mean, who are you talking to? Me or the room? >> Too loud. >> Was I too loud? >> You can't hear me. >> Very quiet. Too quiet. >> Check. Check. It's all right. Needs a little

louder. >> Ah, there we are. Are we ready? >> All right. How to y'all? Going to uh get started here in a minute. Back up again. better on the AV? We're good. Cool. Cool. All right. Thanks everybody. It's our last uh presentation of day one of at scale 23X. Thank you again for coming for the day. Appreciate it. Uh we have this talk from Josh pre-recorded. Mike's

gonna steer and it's about 39 minutes and then we'll have some time live with Q&A with Josh. So head of the engineering uh of the asterisk project. Mike is a senior engineer with the asterisk core project. Ben also in the audience another core engineer on asterisk. So, you know, feel free to chat us up about chat them up about other specifics that uh maybe Josh doesn't have

time to answer because it is getting late there. So, we will get started and watch this brief uh presentation on asterisk, a deep dive into performance improvements in >> Hi everyone, welcome to my talk under the hood. into asterisk performance. My name is Joshua Culp, also known as the Asteris guy, the hat guy, a director of engineering at Sangoma, the astros project lead, um an open source

guy at Sangoma. I have many different titles and things I do. Uh but in the end, when it comes to asterisk, I'm the person figuring out the direction of it, uh the road map within Sangoma for it, um and also making policy and decisions about it. Um, I wish I could be there with you guys in person, but circumstances did not allow it. So, I hope this

recording of a past version of me is uh sufficient to shed some light on performance when it comes to asterisk. So, let's go over what we're going to talk about. Um, going to ask the question, what is performance? Talk about the costs of execution when it comes to asterisk. Uh, what makes calls expensive? We're going to dive into some of the more commonly used things in asterisk

when it comes to um to areas that can have performance impact such as dial plan, AGI, PJ SIP, AMI, ARI, and a bit more. I'm going to talk about some things to monitor as well as configuration strategies, implementation strategies, and then we're going to try for a Q&A. So, I want to throw out the question. Um although because I'm recording this in the past, I cannot see

into the future. What is performance? Um that animation probably brought to you by uh Chris May because he loves throwing animations into uh different uh presentations and stuff. Uh but fundamentally the definition kind of differs between people. You have at a system level which is really easy to think about. So that's how much CPU is being used, how much memory is being used. Um how often storage

is used. Uh that's reading and writing and IOPS. Um as well not on here is network. VoIP itself is actually generally lots of small packets which have an impact on network interfaces particularly when you're in a virtualized environment. So from a system level there's a lot of aspects but for this presentation I'm not going to really go into too much about system level. I'm going to focus

on the asterisk side of things um also known as the application level. So from an application level you can think about performance as calls per second uh delays in call handling that can be um post style delay concurrent calls concurrent conference bridges concurrent transcoded calls um cues so maximum number of Q callers and agents in a queue because cues are extremely expensive in a way. um AMI

message throughput. AMI is very chatty. So that has a performance impact as well. And then latency for doing everything. So fundamentally when it comes down to it, this is kind of a copout in a way, but it depends on what the system is being used for. Performance characteristics of one system may not match that of another just based on usage patterns. But we can look at basic

aspects and break those down to provide a better insight into things overall. So calls per second um if you're using asterisk you generally have calls coming into the system unless you're a dialer in which case you probably have calls going out which we'll get to here soon. So from a highle how things work perspective incoming calls come into a channel driver to be processed. This could be

Chandi, EX2 or PJ SIP. In some cases, authentication is done. We then apply some sort of configuration so that we know how to, for example, route the call into the dial plan, what codecs to Uh once that is all done, we then look at sending the call into the dial plan for actual execution. So we can go through the dial plan and handle the So from a

SIP perspective, which is primarily what I'm going to talk about here, because PJ SIP is pretty much our number one channel driver at this point, um PJ SIP itself from performance perspective is actually pretty fast. So the PJIP stack, which is a third-party library known, um or rather as part of PJ project, which you can go to pjip.org and see, um has been around a long time.

it's been optimized. Uh it was actually originally or rather not originally designed for but a focus was put on making it applicable for embedded devices as well which have specific um requirements when it comes to CPU and memory. Uh though embedded devices of these days aren't exactly like embedded devices of old days. The CPU and memory is quite a bit faster than things used to be. Um,

but from a library and protocol implementation perspective, PJIP is plenty fast. Its parser can handle a ton of messages per second, including enforcing the specification itself. So, if invalid packets are received, then um it doesn't go too far into the stack. So uh from a PJ SIP library perspective, it's not something that we look at from performance perspective and it's not something I recommend people look at

if they are wanting to improve p uh performance. Now there is the second part of the PJ SIP which is our actual implementation of it. Um and that's also pretty fast. uh we try to keep things as minimal as we can and as fast as we can. Um there are some areas that it's just unavoidable that we have to do for example lookups in a container which

has a cost but overall PJ SIP is fairly fast and recently as of the last um set of releases uh last non-security releases it's actually gotten faster thanks to an API called task pool uh which I will go into a little bit later. So when it comes to PJ SIP, where I've seen performance issues is in connecting it to outside resources, specifically When you are connecting anything,

not just a asterisk, to a database, you um inherit some of the performance characteristics of the database that you are connecting to and the connection to the database itself. So a good example is if you are using real time and pjip you are not caching at all. That means we are doing a lot of queries to your database for uh endpoints coming in uh grabbing authentication details

if you have those configured. So if there's high latency there then that can have a knock-on effect for performance and it can really decrease the number of calls per second that you can actually handle. Something else to remember is that if the database is slow um depending on where we are actually um handling or doing a database query that may result in us not sending a SIP

response which means the remote side will generally retransmit resulting in more SIP traffic which is another knock-on effect. Um, starting the dial plan itself is slightly expensive due to creating a thread, but through recent investigation I did, I determined that it's actually pretty efficient at it. So, that's not an area that really um, we're looking at at this point. Um, call setup like this is kind of

like an attack on the system just because of how many resources need to be allocated and set up. So, overall it's expensive. Um but in uh isolation specific parts they're not that bad. So calls per second outgoing. Outgoing is simpler than incoming. Outgoing channel is requested from the channel driver. Channel driver allocates resources applies configuration does DNS if needed and establishes a call and then the outgoing

channel is given to what has requested it and then that handles it further. Um again outside access to resources database yet again uh DNS can also have an impact on things. So um if uh in the case of PJ SIP uh we do asynchronous DNS lookups which um helps in that regard but that does also consume a little bit of resources and if they're slow it can

become a poor experience. APIs as well. If you are before dialing doing an HTTP request in the dial plan for example, um that can also impact calls per second because we are now relying on your outside service. Uh if you're using the default configuration when it comes to PD SIP, um inbound SIP registrations are stored in a local SQLite 3 database that does cache data in memory

just because of the Linux kernel. If not, it can result in disk access. Um, which can be problematic. Um, if you're using a SAN, I've seen issues where the SAN disio latency is just so high that it blocks asterisk for an exceedingly long amount of time. Long enough that in some cases, I've had SIP registrations just expire by the time that it's actually persisted and handled the

inbound SIP re-registration. So always keep an eye on disio. Uh outgoing calls generally don't require starting a thread. So that impact is not as great as incoming. The only cases where that is not the case is if you're doing an originate from AMI ARI and it's having to go somewhere. Concurrent calls. This is an easy answer. It depends on what you do. Once a call is actually

established, the cost of maintaining that is relatively low from a um basic level. Uh it can increase somewhat depending upon your configuration choices such as codec and encryption. But what fundamentally matters is what you do with that call itself. A conference bridge is going to be infinitely more expensive than a file playback. though a file playback is not um without a cost itself because that generally will

have to go out to disk. Um now we're going to move on to more exciting stuff. So the dial plan uh most asterisk deployers use uh use extensions.com also known as the asterisk dial plan. There was an effort long ago to deprecate the dial plan. I don't see that ever happening. Dial plan is here to stay and it's what most people use. However, numerous years ago, it

was heavily optimized. So, dial plan execution itself is quite fast and I've never seen it really present any issue in performance. That being said, there is an area where the dynamic nature of the configuration can result in it being expensive, which is variable substitution. Realistically though, you'll probably never notice it, but if you can be smart about using it, then that may help you in the long

term. Dial plan hints themselves, which are mappings in the dial plan to devices. Um, so that's oh, extension 100 at users maps to pdsip Josh. Um, hints are used heavily in app Q. uh they go out over AMI and you can also subscribe to them from SIP devices. Uh those however are fairly expensive because they are stateful. It has to know who's actually wanting to know about

those hints. Um it's having to keep all that information in memory. When it gets device state, it's having to actually go through and figure out which of the hints it needs to talk to and notify and update. Um so hints themselves are expensive. Pattern matching hints is also even kind of more expensive because it requires a bit more handling during the reload process. Um asterisk has to

store that away and then restore it. Um and then from an operational perspective when it comes to hints uh hints with our current implementation the way it works is that the PBX core subscribes for every device state update even ones that a hint isn't actually interested in. So it gets a essentual essentially a flood of device state updates as they happen and then it has to go

through all of the hints and figure out which ones it's relevant to and then it goes through and gets the current device state for all of the devices on that hint and recalculates. So that's expensive currently. I am however currently working on improving it. Um, so hopefully in the future we'll be able to um knock quite a bit of CPU cost off of that and make it

more efficient. Um, we'll see where that lands. I'm trying to make it so it's applicable to all branches, but maintaining backwards compatibility is uh is complicated when it comes to that. Fundamentally though, the applications and dial plan functions are really where the bulk of the work happens when it comes to the dial plan and their cost is variable depending on their application. So my advice is don't

focus on the dial plan itself. So the do the execution through the dial plan. Um that's unlikely to really help you in the long run long run when it comes to performance. Um, unless it is truly somehow impacting you, but I doubt it, uh, you really need to focus on your actual usage of applications and dial plan functions instead. So, PJ SIP, uh, like I said, pretty

much our number one channel driver, I think, except for the diehard Chan SIP people, um, or people who are using Cisco phones with the patched Chanzip for that. PJ SIP is where everyone's pretty much being these days. Uh I'm going to talk a little bit about implementation details of PJ SIP just so um people get a better idea of how it's working internally which may um provide

some um insight into things in general. Uh so our PGS implementation uh excuse me for a sec. Yay water. Uh so our PJs implementation uses a dispatch model. Essentially this um we have a pool of uh well we have a pool of threads in a way which I'll get to um and also a pool of task cues essentially. So the way it works when it comes to

incoming SIP traffic is that we programmatically determine which of our work cues or task Q that packet should be handled in um and then pass it off to there. So we have one thread which is which is handling timers and sockets. It does minimal of work as it can and then hands it off to another thread or a task Q to be handled. When it comes to

um other tasks which are not like network oriented or rather incoming network oriented um everything goes into a task queue or uh whether it's programmatically determined or just one is randomly chosen because we don't care where it executes. So everything is generally passed off to another Qthread to be handled. Um I'm specifically saying Q and thread because they are not one to one. A Q does not

have a dedicated thread. There the number of threads that PJ SIP can use for this work. Um there's a minimum number and it can grow as needed. So um it's not a one onetoone mapping. Um this work or this approach previously used our threadpole implementation. Uh the thread pool is exactly what it says on the tin. It's allows you to have a pool of threads, it can

grow and shrink and um if you just want to throw something to a into the pool for a thread to execute, it can go there. Uh I had a hunch a while ago last year before I went on vacation that um the cost of thread pool was high and so I prototyped experimented um to determine that yes the cost of doing that is high and so I

came up with a new API which is better fitted to um the usage pattern of tasks which execute a short to medium length of time which I call task pool. Uh if you'd like to know more about task pool, I also have a blog post on asterisk.org talking about um talking about when I wrote it and some metrics about it and stuff. So it landed technically a

few releases ago uh for stasis and then as of the last major the last releases nonsecurity it landed for PJ Slip. So by moving from thread pool over to task pool uh we decreased CPU usage about 8 to 10%. Which is great. Uh stasis was more like 20 to 30% I think it was but every little bit helps. So like I said before PJIP stack underneath is

generally quite fast at operations and our higher level usage of it we try to keep simple. Um, like I said before, where things can fall apart is outside resources usage. Real time. I can't repeat this enough. Real time is not free. It has a cost. And people, I think, have been conditioned due to connecting websites and stuff to uh that they just from a user perspective, you

just have a different expectation when it comes to a website versus a phone call. If a website takes a few extra seconds to load, we can tolerate that. But from a SIP perspective and VOIP perspective, if things take long, then that has protocol ramifications as well as just user experience ramifications. So real time is synchronous. It has to be because it's going to get data that's needed

to process call. So it can block thread uh so it can block threads rather which then increases the handling delay. It can cause buffers to fill network buffers. It can cause packet retransmissions. It's generally not great. So if you're going to use real time, don't try to be clever and connect your asterisk to a remote database. That latency if it's large enough is just going to cause

you headaches. Another thing is that make sure that the network connection between asterisk and the database is clean. And by clean I mean ensure that there is nothing configured to actually do packet inspection and manipulation or blocking because that can have ramifications on the underlying database protocol which can really confuse the libraries for these databases. Good example is my SQL. If something happens to the connection there,

it can get into a really wonky We do have sourcing which is available to try to minimize and reduce this by caching data in memory. See the doc site docs.ashasters.org. It's very configurable. It can work great. However, there are cases such as when you're persisting or writing out data where that is not cached in memory and does have to persist out to the database. So, it's not

a magic a magic fix all bandaid. Uh configuration like I said can have an effect on CPU usage and um as well PJIP being a newer module can also react to task processor overloads. This is configurable in pjip.com if you need to. Um and it aims when the system when task processes are overloaded it assumes that the system is overloaded and so it tries to um reduce

the strain on the system by ignoring new calls. So you can configure that if you want to disable it for Now programming interfaces. So AGI that's split into two options. Executing an application AGI and establishing a TCP IP connection. Fast AGI. Um for large scale systems using fast AGI is vastly superior because executing an application using AGI can be expensive and this depends on the memory usage

of asterisk itself. When asterisk has to launch that application, it has to fork itself. If asterisk is used a lot of memory, that opture that operation can take a longer period of time and it can cause certain aspects of asterisk to pause. Um, asterisk itself doesn't exactly pause, but timers used by asterisk could do. So this manifests as audio blips or pauses during playback or in conference

bridges. If you use fast AGI, you don't experience that. However, once actually started, AGI is fairly lightweight. Um it's got a minimal of bespoke things inside of it that are specially coded for within AGI. Um it's mostly using common interfaces, dial plan applications, and dial plan functions. So the cost really comes from that initial launching of the applicationbased AGI. So if you've ever experienced that uh switch

to fast AGI uh AMI so AMI the asterisk manager interface uh it can produce a ton of messages because it's a bus of events that are happening with an asterisk. If an AMI application wants to actually receive them, it's got to be sent. If there's a ton of them, they may end up buffered. So the thing here is if you can reduce the number of messages your

application needs, the better specifically by doing the filtering on the asterisk side. If it still needs a large number of events, then use multiple AMI applications or multiple connections each receiving different um events. There is a consequence of that though in that um it it the events will not be serialized. So depending upon how you're using them, you do have to take that into account. And something

else is that the speed at which the AMI application reads or receives the events will have an impact on the aster side. Network buffering can only do so much. So one potential aspect is also to read and buffer in your application as well. So have a queue of work in the application itself too. ARRI master's rest interface um ARRI I just want to start by saying ARRI

is not a replacement for AMI. AM ARRI receives a subset of the events that AMI does. So, ARI fundamentally has less events and receives less events. That being said, it can still produce a decent number depending upon usage. Um, so the recommendation just like AMI there is to do server side filtering to reduce the message load to your ARRI application. So, if you're not interested in certain

events, tell us and we won't send them. that um that reduces the network traffic and also reduces the amount of processing you have to do and also helps on the Astra side some um I would also like to throw out that ARRI now provides outbound support so you can load balance your application uh you don't need to have one application receiving everything or try to do load

balancing using multiple applications in the dial plan um so you can just send it to an HTT HTTP load balancer that's websocket wearer and do it that way. Something else we added is the ability to do HTTP requests over the So you don't need to um figure out how to get HTTP requests back the normal um port 80 other connection way. You can just send them over

the websocket instead which also simplifies things. So um depending upon the scale of your application that can be a great approach for allowing it to scale better. Now some commonly used functionality CL and CDR uh so CEL or call event or channel event logging um is chatty. It's giving you not as many events as AMI, but it's still giving you a fair number of events that are

related to the channel as it progresses. And this is so you can interpret those events and create your own CDRs. Um, CDRs are fundamentally business I've never seen anyone truly agree on how CDRs should be because like I said, business logic, so everyone has their own idea or needs. uh we do provide CDRs. So CDRs produce much less records as a result. Generally it's a record per

call leg. Uh so if you're doing transfers more call legs but in comparison to the CL it's considerably less. Now going back to what I said about dial plan and variable substitution. These modules when it comes to certain um uh outputs or backend modules do use variable substitution. Uh so uh CEL custom um CDR custom they allow you to configure essentially the lines that you're going to

output using dial plan variables that get substituted in. It has a cost but we are working to add alternative configuration which will substantially decrease the cost of doing so. Um this is being done by George Joseph and in fact he's also come up with a way to allow it to automatically work for a good portion of configurations. So um that should decrease the cost there which will

be Going back to database I'm a broken record on this. I freely admit that directly connecting to a database it can be convenient but throughput of the particularly for CEL can be a problem. Uh so my recommendation is to do pro post post-processing instead of the records that asterisk is outputting and then shove them into the database. The nice thing about that is you can store them

in a um like locally on disk for example instead so that if something does happen to the database then they can still accumulate and then afterwards the database comes back which hopefully it should you can uh go ahead and bring those in. Transcoding. This is easy. It's CPU, CPU, CPU. Transcoding is pretty much CPUbound operation. If you reduce your transcoding, all the better. Store sound files in

the codecs that you use. For so if you're doing Ulaw G722, store two copies if you can. Disk should be relatively available. So you can do as well, not all codecs have the same cost for transcoding. You can check out core show translation for how long stuff takes. Transcoding G722 between Ulaw and ALOW is a lot simpler than something like Opus. Opus has gotten cheaper as things

have gone on and it's been optimized, but Opus itself is also a very expensive codec. Um, for example, uh, a few years ago, I don't know how many it would be, um, as phone manufacturers were starting to implement Opus, the CPUs and chipsets that they were actually able to get just wouldn't support two legs of Opus at a time, could only do one. And that's just because

of the complexity of Opus. things have improved there but opus is still expensive. Uh bridging. So two-party bridges are simple but going back to transcoding. If you're transcoding then that's where the cost will be when it comes to a two-party bridge. Multi-party bridges are just always expensive because media has to be transcoded to and from signed linear. Um it's always going to happen. So codec choices become

even more important So when it comes to bridging, bridging itself not bad. It's just transcoding that's the issue. So implementation comments uh I talked a bit before about thread pool when it came to PJ SIP. Um so I'll talk a little bit more about this a little bit more Um particularly the third point down if you saw task processor warnings on the asteris console about stasispool that

was the thread pool. Um so the way the thread pool works is that you give it work that work goes into a single Uh another task is cued to then wake up create uh wake up or create a thread if and then some work is given to that That single pool where all of the tasks would live could grow quite a lot depending upon how fast or

how many threads were actually available. So if you had warnings about stasispool, that should now be resolved as of the latest releases. Well, technically the two or three releases I think it should be resolved. So that should no longer Um so just putting that out there. So the number one thing to remember, everything has a cost. It's obvious. Um it's a kind of a throwaway kind of

sentence, but it's true. The less you do, the generally the more you can get out of the system. My recommendations reduce reliance or dependency on outside systems if you can reduce transcoding so minimize as much as you can reduce how much work any of your external applications so that's ARRI or AMI have to do keep things as simple as you can try to be clever don't try

to go fancy keep it simple if you if you have to do something then reduce it or optimize it if possible. Now, some things to monitor SIP response time to incoming requests. Uh so you can send in uh SIP options. SIP options will still flow through the dispatch model, which means you can see how fast stuff is being dispatched to handle. If that's creeping up more and

more, then that may indicate that if you're doing a database that something's up with your database, it could indicate that you are seeing an increase in SIP traffic and it's taking longer to handle Um, or that something internally in asterisk is finally reaching a point where it's impacting your performance in some way. uh SIP qualify time to endpoints. So if the um time to qualify stuff is

creeping up for essentially everyone, then that can indicate that you might be having network issues. There might be retransmissions going on. Um the dispatch model yet again could be having issues and is getting overloaded or blocked on something. uh you want to monitor the number of registered endpoints. That's whether it's performance or just in general that is a great thing to monitor because if your registered endpoints

are trending downwards or suddenly goes down by a large number then that can indicate that something in asterisk is blocked. Um one sec. I could pause this recording, but I'm not going to. Um, so for example, when it comes to the dispatch model, you can technically have one of the task cues blocked indefinitely. And because we use a programmatic approach to distribute tasks out, that won't manifest

as all of asterisk or all of PJ SIP blocking you. only a subset of it would that can increase if what it's blocking on is something used by the um the other task cues but it may not be. So monitoring registrated endpoints is a great approach for that as well. AMI response time if that's creeping up then you may have an increase in um events being generated.

uh your handling could be um not as fast as it normally is. You might be CPU starved on the task processors. So core show task in particular overloads and maximum cued task processors themselves are much to probably what people have said task processors themselves are incredibly efficient. It's all the users of task processors that can cause issues. So core show task processors can show you specific subsystems

that may be starved for CPU or maybe having issues. So if that's creeping up then that's something you want to keep an eye on. Uh standard stuff CPU usage, memory usage, disk access time and IOPS. Uh if you are if disk access is not what it should be then that's kind of a knock-on effect for other things particularly inbound SIP registrations those will start lagging nougat retransmissions

which will cascade and it's just not great and then like before outside services like your database monitor your database if you're using it uh as well as DNS so some configuration and implementation suggestions, endpoint configuration, restrict codecs. A lot of people just throw allow equals all. Don't do that. Restrict it down. Uh know what's in use. Be be specific about it. Real time. Yet again, I'm a

broken record. Avoid using an external database if you can. Doubles your scope of potential problems and can cause issues. If you must when it comes to PJ SIP, you can use source caching which will cache a ton of things in memory depending upon your configuration. Um on the read side, it will not cache writes. Writes will still go to your back end. So if writing is slow

or database having issues, then that can manifest itself. AMI and ARI filter it if you can or reduce your traffic in general. uh task pool that new API I mentioned uh monitor core show task processors for the task pool ones and adjust respective I think realistically it's unlikely you'll need to dial plan don't go overboard with variable substitution if you can avoid it and then ultimately test

test set up a test environment for your usage to understand how things react to it and get a better baseline for what is normal for you. Deployments and systems are not created equal. That even goes down to the CPU. So, you need to test and figure Overall though, we will continue to improve performance. It's a personal side project of mine. I want to improve performance for everything

I can. And with that, thank you. Uh, and we're now going to try to do a Q&A live over video conferencing. We'll see how that goes. Um, if it does not succeed, uh, you can find me on community.org, the forum. Feel free to ask questions there. I'll respond to them. I always generally do. Um, so I look forward to seeing everyone there, too. >> Am I up?

>> Have a great day. Check one, two. Oh, hey, get out of the way there. Um, thanks Josh. That was a great performance improvement overview. Uh, the, uh, stats look pretty good in a number of areas. Um, we are looking for some questions now at the end of your talk. If anyone has a question for remote Josh. Josh, are you there? >> I think I'm here. Can

anyone really be sure? >> I could be AI to be honest. You know, >> there we are. Yes, we hear you at least. >> I will say anyone thinks of questions later, as I said in the video, uh just post them on community.org. I'm >> okay. I guess that there's a request. Yeah, Josh, speak a little louder maybe into your mic. Uh We uh heard that. Yeah,

I was posted to the community forums or Jean back here. >> So Josh, you mentioned a list of things worth monitoring. I was curious if you had settled on some sort of an observability stack. What would you recommend to use to uh monitor those things? >> Um so can't say can't say I've settled I have no control over volume. So that's on your side. Um from an

observability perspective. Um haven't settled on anything specifically. Many people have different opinions on it. Um from a data perspective of what past people of asterisk um developers I would say settled on but implemented is um some stats d stuff that can be um that can be pulled out of asterisk. Um there's some Prometheus stuff as well. Um, but really we haven't defined anything of complete stack of

what to use. So honestly it's up to you whatever >> Um, I'll I'll throw in a quick uh plug maybe homer I I think there's a HP module that they've worked on. It's a larger observability uh across a number of different open source voy projects. So that it's worth it's worth a look. Um I I know they strive for some asterisk integration. It's a third party project.

So, you know, I guess can't really speak to it too uh too deeply, but um I think there's a few any other questions here for Josh. Remote Josh. Oh, hey. >> Hi. Um about AMI filtering, uh what is the cost uh on the server to put some especially reg x filter because it's um what's what are the benefits on the server um compared to uh what we

gain uh limiting the outputs and uh having some extra CPU use because of the filtering. So from a so the traditional um AMI filtering ends up being quite expensive. Um the rag base stuff the newer generation stuff um which is more intelligent and explicit is a lot less CPU. I think George did a blog post about it where he talked about it. Um so the newer stuff

is the way to go. The cost of doing it on the server um is fairly minimal. Um but it ultimately depends as well on how many messages you're generating too. Um the actual usage pattern of asterisk according to your use case. Um so it it depends but the newer generation stuff is considerably better and the penalty hit of doing it um server side using that is not

that high um in comparison to sending all of that down to an application and the application having to deal with it. um since in our experience trying to do it on the application side it just it's painful and it doesn't it doesn't scale and it doesn't work well. >> I think a minor uh shout out for George too on in regards to some of the uh some

of the tools I think for the performance monitoring he's process of releasing. Is that right? Have we we have some more repos? I think maybe I'm speaking out of turn here. >> Can't hear you. >> I know um you know spoke on some of the other uh you performance improvements uh early earlier in the year. Um um in in regards to some you know capacity improvements and

and things and so yeah so we've had a few internal tools hopefully we can release more of those soon follow on the questions um all right well we're a few minutes after five there's no uh further questions uh just uh you know say thanks Josh again so not sure if you heard your applause earlier but we can clap again I guess so >> and uh you know

another shout There's a few more uh asterisk engineers uh and some other uh Sangoma uh you know folks here too. So if you you know take a look at everyone's badge uh wave if you like. Um oh looking around for where they might be. I like it. I like it. Uh you know we've got a lot of uh great resources uh available um here at this meeting.

So thanks again for coming you know for the first day of Astrocon 2026. We'll get started again tomorrow morning, Friday at 9:00 am with some AI talks, uh, a couple of those and of course, and then, uh, we'll have some other, uh, discussions about, you know, more, um, asterisk improvements, or kind of an overview of of the project and where it's been, as well as a free

PBX talk. So, I I know there'll be a shout out for some community contributed uh, improvements there. Uh, and I think that this is uh, going to wrap up around 2 o'clock tomorrow. So, if you didn't grab your t-shirt yet, uh please do that on your way out. So, if you want to wear that tomorrow for the group photo at the end of the day, that'd be

great. Or I guess you could just change at lunchtime if you you don't think of it. Uh again, all the speakers, if you haven't grabbed a mug yet, please do so on your way out today uh as part of your uh you know, extra swag for coming to speak. Thanks again for doing that. Thanks all to attending and we'll see you tomorrow for day two of Astrocon

2026. Thanks all.