The Acme Packet SBC has an optional parameter that may be added under sip-config that allows the SBC to gracefully handle traffic in the event the SBC’s main processor reaches a certain threshold. There are several reasons as to why this may occur, but at the most basic level it’s a good idea to draw a line in the sand and define at what point to you want to start gracefully rejecting calls if the CPU reaches a certain threshold. Every SIP appliance should have this option but unfortunately most do not.

What separates the Acme Packet SBC from others is that when this threshold is reached the SBC will reply with a SIP 503 Service Unavailable message which tells the originator to try an alternate destination.  In most other SIP appliances once the CPU threshold reaches a certain point the traffic is disrupted by means of calls dropping, loss of RTP (if media is flowing through), or registrations becoming corrupted.

It is worth mentioning that the purpose of setting the load-limit is to avoid an unforeseen event that would cause any network device to experience excessive utilization and effect production traffic.  This is not a replacement for proper SROP (SIP Registration Overload Protection) and DDoS (Distributed Denial of Service) configuration on the SBC.  With the proper SROP and DDoS settings you are ensuring the Acme Packet SBC is running optimally and protecting your network while gracefully shedding unwanted or dangerous traffic.

The following is a shortened output of my sip-config with no options applied.  Adding the load-limit option is a matter of entering the following command.

At this point the Acme Packet SBC will gracefully reject incoming calls if the CPU reaches or exceeds 90%. Of course this value may be set higher or lower.  More than likely more options will be applied in your sip-config. If you follow the same process to add another option it will overwrite the option that already exists.  Prepending the + symbol in front of the option will add it in addition to any existing options.

In this example the load-limit is the first configured option. In addition, the max-udp-length is going to be set to 0 which allows the SBC to fragment UDP packets. Otherwise the maximum size a UDP packet may be is 1500 bytes before having to use SIP TCP.  Setting this in sip-config applies globally on the SBC but it is possible to configure this on a per sip-interface basis if desired.  The last option is reg-cache-mode=none which tells the SBC to retain the userinfo (post NAT) in the Contact. In most environments (such as Broadsoft BroadWorks) none is the value to use. This is also required when configuring SIP Registration Overload Protection (this will be a separate blog post in the near future).

Based on applying the load-limit to the sip-config this provides one additional line of defense that safeguards your platform from suffering a total outage. There have been many occasions noted in the past where other platforms do not employee this method and the end result is a complete and total network outage.  Once the outage starts it becomes very difficult to recover because endpoints will begin retransmitting. Using this feature in combination with alarm-thresholds plus the appropriate SROP and DDoS settings, there is always awareness of what’s occurring on the network while at the same time gracefully handling any overflow.

The Acme Packet 3800 and 4500 series Session Border Controllers come with an NIU (Network Interface Unit) that includes four Gigabit ports.  In rare cases one may need to support Media Aggregation where two of the Gigabit interfaces need to be look like they are “bonded” to accommodate a large number of calls (with media anchored).  For example, a wholesale SIP Peering Service Provider may need to handle upwards of 16,000 concurrent calls on one single Realm on the SBC.  If  calls are using the G.711 codec this would require 1.5GB of bandwidth.

The process used to support Media Aggregation is to assign the same Realm name to two different Steering Pools.  For example, on the realm Access it would have more than one steering pool which would reference two different network-interfaces.  The same is true for the Core.  In the following example NIU ports 0 and 1 are used for Access (public facing) and 2 and 3 are used for Core (private).  Note the NIU is considered slot 0 and the ports are 0, 1, 2, 3.

The next step is to verify you have your Access and Core realms configured.

The final step is to create the Steering Pools and define the network-interfaces for each.

At this point the SBC will utilize two ports assigned to Access and two ports assigned to Core to support media thus providing Media Aggregation. SIP signaling continues to only traverse on the sip-interfaces assigned to the Access and Core realms (s0p0 and s0p2) .

At this point the Acme Packet SBC is fully prepared to handle traffic that exceeds 1GB on a single Realm.

The Acme Packet SBC contains an optional parameter that may be added to the configuration which helps avoid a SIP avalanche from occurring.  One instance of a SIP avalanche is when a very large number of SIP endpoints consecutively send SIP registrations to an SBC which then forwards them to a SIP registrar. This can be dangerous for both the registrar and the SBC depending on how many endpoints are attempting to register. Assuming DDoS is applied to the SBC configuration, it will protect itself as well as the SIP registrar from being negatively impacted by the avalanche.

The following scenario shows how endpoints typically behave when they NAT through a firewall. Normally the firewall will NAT the layer 3 IP address but the layer 7 (SIP) address remains the private address. Upon receiving the NAT’d registration the SBC forwards the register message to a SIP register platform and after an exchange of a few messages between the registrar and endpoint a 200OK is sent from the registrar back to the endpoint with a register expires of 3600 seconds (default for Broadworks). The SBC knows the SIP endpoint is behind NAT and changes this timer from 3600 to something shorter (60 seconds is common).

Envision a scenario where the Internet is experiencing a route flap which is causing endpoints to lose connectivity to the SBC. If the endpoints should register every 60 seconds and they fail to do so, the SBC will delete them from the registration cache. If perhaps five minutes later the Internet is restored and the endpoints are able to communicate with the SBC again they would have to complete the entire registration process again. This would trigger a SIP avalanche that may negatively impact the core SIP network.

By implementing the option register-grace-timer parameter to the global SIP config and specifying a number of seconds, the SBC will retain the cached entries rather than let them expire even if the endpoint registration isn’t received. Once the endpoints come back after the Internet outage is resolved and they send a SIP registration to the SBC, the SBC will not forward this to the core because the previous registration remains valid in the SBC cache. This prevents the SBC from having to go through the entire registration process thus reducing the overhead involved on both the SBC and SIP registrar.

In this example the timer is set to 300 seconds. If endpoints are supposed to send their registration every 60 seconds but the SBC is not receiving them, it will retain the reg cache entry for another five minutes.

The diagram above demonstrates that even though the Internet is restored and endpoint registrations are reaching the SBC, there is no need for the SBC to forward these registrations to the SIP registrar since the reg-cache was retained during the outage. This prevents CPU cycles from being unnecessarily used on both the SBC and SIP registrar.

By default when a SIP Invite is sent from a SIP endpoint to the Acme Packet SBC, the default expires timer is used to determine when the Invite will timeout based on no response. Changing this globally may not produce the desired result as this impacts all Invites to all devices across all realms.  In the case of routing calls to a PSTN peering provider where there are two or more destinations provided in the Contact header, it may be desirable to shorten the timer to reduce post dial delay in the event the first Contact an Invite is sent to is experiencing an issue that is impacting call processing. This assume the device (session agent) is still responding to SIP OPTIONS PING messages and is therefore in-service, but unable to complete calls.

If the ITSP does not respond to the Invite the SBC will wait 32 seconds before the Invite will timeout. More than likely the calling party will abandon the call before waiting 32 seconds which means sending further Invites to secondary or tertiary gateways won’t do much good. In the event there is more than one Contact in the Contact header or the SBC is set to perform Load Balancing using a Session Agent Group (SAG), it is best to change the trans-expire timer on the SIP Interface communicating with the ITSP to a value that is more feasible.

In the above scenario the SIP Interface that peers with the ITSP has a trans-expire value of 5. This means once the Invite is sent to the first Contact, the SBC will wait for a response for 5 seconds. If no response is received a new Invite is sent to the next Contact address (or Session Agent in the SAG). This helps reduce post-dial delay and provides a more acceptable timer for failover to an secondary or tertiary destination. Setting the timer on the SIP Interface helps keep the behavior unique to a particular destination.

The Acme Packet SBC is now available in a Server Edition that runs on HP DL series servers. The platform is based on the same C-Series code that is used in the 3800 and 4500 appliances and runs on an Acme Packet optimized Linux kernel. There are some configuration elements that vary between the Server Edition and the appliances.

When configuring the PHY and NETWORK interfaces on the Server Edition it is important to verify the four physical Ethernet ports on the Server Edition correspond appropriately to wancom0, wancom1, s0p0, and s1p0. In most cases when installing Server Edition the Linux kernel will map wancom0 and wancom1 to the on-board Ethernet ports on the HP server appropriately. The Server Edition is equipped with an additional HP NC360T dual-port NIC providing two additional Ethernet ports that support s0p0 and s0p1. These are considered the Media interfaces which support SIP, H323, and RTP traffic. The wancom0 port supports platform management and wancom1 supports Active/Standby stateful synchronization when running two Server Editions as a High Availability pair.

To begin, the first step is to see what MAC address maps to which SBC interface.

In this case the MAC addresses identified in the HP server BIOS correctly match the respective wancom and media ports. When deploying in a simplex (non-HA) model, wancom1 is not going to be used since there is no Standby unit to replicate to. The Server Edition has the ability to support additional media interfaces if more physical Ethernet ports are added to the server. Also, for those familiar with the 3800/4500 and the use of wancom1 and wancom2 for redundant back-to-back connections between the Active and Standby nodes, the Server Edition does have the ability to support wancom1 and wancom2 for replication as well. It’s just a matter of adding more NIC ports. By default, with four physical ethernet ports on the server, only wancom1 is used for replication to the Standby which is still very adequate.

In the event the physical ports on the server do not map as desired, it is very easy to identify and reassign ports.

When entering the locate command the SBC will begin to rapidly blink the physical port that corresponds to s0p0 making it easy to indentify. To change the assignment just use the swap command.

Once this command is executed the s0p0 and s1p0 interface are reassigned.  To change the description use the label command.

The following is additional configuration output provided for the purpose of showing configuration completeness. Note in this case the Server Edition is connected to Cisco 3750 switches and is operating at 100MB Full Duplex.

Cisco Call Manager Express 8.5 (Unified Communications Manager Express) introduced support for the Cisco 9951 and 9971  phones but it was lacking support for the onboard camera and video calls. With CME 8.6 and newer, Cisco introduced support for the video camera on the 9951/9971 phones allowing for video calls as long as video and camera configuration elements are present in the voice global and register pools. The 99X1 phones only support SIP firmware and there are no known plans to support SCCP. Supporting SIP phones on CME is quite different than SCCP phones, especially if you want CME to automatically provision cnf files for each SIP endpoint.

I decided to create a minimal configuration on my lab router that shows how CME supports Cisco 9951 SIP phones. There are no SCCP phones in this configuration. I believe as Cisco continues to evolve towards SIP, the need for SCCP will slowly fade away into the sunset (this is just my opinion, but is a trend that is becoming more apparent from Cisco each year).

The first step is enabling SIP to SIP communication so phones may call each other as well as dial into CUE. The router should also act as a SIP registrar because we want our phones to register to CME. The min and max expires timers can be set to whatever value you choose, but it’s not a good idea to set it too low otherwise you will DDoS your own router for registration floods.

The next step is to enable voice register global so the router supports CME using SIP. This is done by using the mode cme command. Think of this process as the SIP version of telephony-service. In this case the 9951 phones are physically connected to an HWIC-4ESW-PoE module directly on the CME router and using the router’s IP on the voice VLAN as the SIP interface IP address.  The 9951 firmware was previously downloaded from Cisco’s web site and manually copied to the router’s Flash, hence the tftp-server flash: entry. The video and camera options enable support for video calls on supported phones. The authenticate register command is strongly encouraged for everyone to implement in their SIP configuration. When a SIP phone attempts to register to CME it will be challenged by CME for authentication credentials using MD5 hash. This prevents users from either simply hijacking a SIP identity by spoofing a MAC address. By default, when not using MD5 hash, CME compares the MAC address entry in the voice pool against its own ARP table entry to see if there is a match. It’s very easy to hack your way around this mechanism!

The voice register dn command is very similar to the ephone-dn command used for SCCP phones. As noted above, the extension for voicemail is 4009 which means if there is an incoming call to SIP phone extension 4001 or 4002 is busy or does not answer, the call will forward to voicemail.

The voice register pool command is similar to the ephone command used for SCCP. This is where the MAC address of the 9951 phones are listed, the type of phone is set, the phone number is assigned, and most importantly the pool tells CME to allow use of the 9951′s camera and allow video calls. Something I also consider best practice is to always assign a SIP authentication credential under each pool. This prevents others from hijacking another user’s extension. I would recommend using a stronger username and password. I made it simple for demonstration purposes.

In order for the 9951 to download firmware from Flash the tftp-server command should be set for all the related files.

The CNF files must be generated which provision the files the phones will download. CME will look at the MAC address under each voice register pool and generate the appropriate files. When the phones boot they will search for their configuration files and load the referenced firmware if needed.

At this point the CME configuration for support SIP phones is complete. This specific configuration includes support for the 9951 and video calls.  There are many additional operations and parameters to fine tune CME, but the referenced configuration is enough to enable CME supporting SIP-only phones.

In the event the phones are not loading as expected issue the following debugs commands:

router# debug tftp events

Assuming the 9951 is successfully on the network the output from the debug tftp events command should show the phone attempting to download the configuration files from Flash. If the phone is attempting to download files that end in sgn then Reset All Settings should be performed on the phone. There is a high probability the phone was previously deployed in a CUCM environment.

router# debug ccsip messages

If the phone successfully downloads the provisioned cnf files but fails to finish coming up on the network there is a good chance it is failing its SIP authentication. By executing debug ccsip messages the router will display the SIP dialog between the phone and CME. The response from CME should help determine what the root cause is. For example, if CME returns SIP 503 Service Unavailable then double check voice services voip settings.

Diagnosing a troubled SIP call has a tendency to be a real pain. Whether it’s running wireshark, tcpdump, or collecting debugs, having to sort through duplicate packets and attempting to merge different pcap files together does not provide a simple way to troubleshoot a single call while looking at both sides of the call in a single ladder diagram.

Fortunately the Acme Packet SBC now includes a free tool embedded in the code that once enabled, allows it turns the SBC into a SIP capture device. The distinct advantage here is seeing both sides of a call in a single ladder diagram. Even better, extra (and very useful) information is included in between each step of the ladder diagram referencing internal “logic decisions” as they occur as traffic passes through the SBC.  Finding a particular capture is easy using Search Filters which allow you to specify just about any criteria.

Pop-up context provides tool tips and additional information about a call depending on what area you are hovering over. It is also possible to export a capture locally so it may be emailed and viewed by others  rather than having a variety of users logging into the SBC’s web interface. Alternatively, captures may be exported as ASCII text files with proper and readable formatting of the call information.

There are three main parts to viewing a captured call. The first is the Session Summary view which contains information such as source and destination IP addresses, URI’s, Realms, etc..

The second viewing pane is SIP Message Details. This is the actual ladder diagram and SBC events.

The third pane is for viewing QoS statistics such as jitter, packet loss, delay, and MOS scores for the specified call.

In order to enable web browser viewing of SIP Monitoring and Tracing the web server must be set to the enabled state.

ATL# configure terminal
ATL(configure)# system
ATL(system)# web-server-config
ATL(web-server-config)# state enabled

The next step is to create one or more filters. In the following example there is a filter called hostedIpPbx and in the user portion of the filter any SIP messages containing the phone number digits 781801 will be captured.

The next step is to enable sip-monitoring and identify which monitoring filter should be used. Applying the filter here enabled the filter globally on the system. Usually filters are best applied to specific realms or session agents (under monitoring-filters) to capture only interesting traffic.

In this particular network there are IP phones behind a Cisco ASA firewall which NAT to the public internet and need to register to a SIP softswitch which is being “hidden” behind the SBC in a Service Provider core network.

IP_Phone——Cisco_ASA——-[APKT_SBC_Public<>APKT_SBC_Private]—-SIP_Registrar
<–Customer_Prem->NAT>           <–SIP Monitoring and Tracing–>
192.168.1.198    72.12.135.253    72.12.135.250      10.12.135.250       10.12.135.140

Click on the image below to see a full screenshot from the SBC’s SIP Monitoring and Tracing tool showing a SIP phone behind the ASA registering to a SIP Registrar server. As you can see SM&T captures both the ingress and the egress side and displays into one simple ladder diagram. The Session Summary provides additional useful information specific to the SBC that would normally not be seen in 3rd party capture tools.

Many Enterprises have migrated (or will soon migrate) to SIP Trunking for PSTN access. For an Enterprise with a single IP PBX platform and an SBC on the edge, call routing from the PSTN to the internal network is typically very straight forward because all DID’s point to one IP platform.

Example of a simple SIP Trunk model:

When another SIP communication platform is introduced into the Enterprise topology it raises the question of what IP platform is considered the call routing decision maker for calls coming from the PSTN SIP Trunk.

This is an example of a less-than-desirable implementation.

Usually one of these is the common way an Enterprise integrates their IP PBX with another IP platform (such as Lync):

• All calls go to the IP PBX and then fork to Lync
• All calls go to Lync and use Lync’s Simultaneous Ring feature to ring the IP PBX
• SBC forks the SIP Invite to the IP PBX and Lync simultaneously

Although many IP PBX’s deployed with Lync use one of the above methods, there is a much better way for this integration to occur.

The Acme Packet SBC supports Active Directory integration for call routing decisions. The beauty of this is in most organizations Active Directory is already used to store various phone numbers for users within the Enterprise.

When the local-policy is configured to support Active Directory (LDAP) and a SIP Invite is received from the PSTN, the SBC will query Active Directory to see if the phone number is assigned to a user.  If it is, the SBC will see if it’s part of the Active Directory field attribute telephoneNumber for a desk phone, or msRTCSIP for a Lync phone number. Other attributes are supported and are explained further in this discussion.

This is an example of an Active Directory query from the SBC to determine if the call should be sent to CUCM or Lync.

In a similar regard to the above diagram, if the dialed digits are the Lync phone number and the Active Directory query shows the dialed digits belong to the msRTCSIP attribute, the Acme Packet SBC knows the next hop should be the realm and session agent for Lync.

This is where the power of Active Directory integration in the SBC becomes very interesting..

The Acme Packet SBC’s local-policy may be configured in such a way that if the dialed digits +1-404-555-1000 match telephoneNumber in Active Directory, but msRTCSIP is also populated with a separate Lync phone number (in this case +1-404-555-9595) the SBC’s local-policy preference will route the call to Lync instead of the IP PBX. If msRTCSIP is not populated, the call will simply route the to the IP PBX. This is a very flexible way of handling call routing. What this comes down to in this particular case is an environment where some users are exclusively on CUCM and others are CUCM and Lync, but the Enterprise may want Lync users to always receive calls on Lync first regardless of the dialed digits coming from the PSTN.

Now for even more flexibility..

The Acme Packet SBC may also be configured in such a way that it recognizes both telephoneNumber and msRTCSIP being populated, each with their own phone numbers, but if the SIP Invite to the first platform experiences a SIP Invite timeout or failure, a SIP Invite may be sent to the next destination. You are not limited to just two destinations.

Finally, having calls reroute to the PSTN during an internal network outage is also possible by using the Active Directory attribute field mobile. If this field is populated and the SBC’s local-policy is setup to recognize the digits in the mobile field, calls that fail to setup internally (in this case, to Lync and the IP PBX) may route back out the PSTN SIP Trunk to the user’s mobile phone.

Configuration examples:

The two critical configuration elements to examine for Active Directory integration are local-policy and ldap-config.

First we will look at local-policy. This is very straight forward as the only difference when using Active Directory integration is the next-hop policy-attribute will reference ldap.

The next configuration element to talk about is ldap-config. This is where you define the Active Directory criteria and how call routing should be handled. You may have have more than one ldap-config on the SBC.

The example below is similar to our simple SIP Trunk diagram at the start of this blog post. There is a single IP PBX platform on the internal network. However, the mobile field in Active Directory is also populated with digits. The way this local-policy is configured, if a SIP Invite timeout occurs to CUCM, the policy will reroute the failed call back out the SIP Trunk to the Active Directory user’s mobile number.

In this next example the local-policy uses the attribute order. This means if the dialed digits match an IP PBX phone number (telephoneNumber) but the user also has a Lync phone number populated in Active Directory under msRTCSIP, send the SIP Invite to Lync.

As you can see from the above example one of the most useful elements in the ldap-cfg-attributes is the use of regular expressions for phone number formatting.

Other cool features in 6.4f1:

Support for RFC 6341 (SIP-based Media Recording – aka SIPREC) on the Linux version of the Acme Packet SBC.

In 6.4f1 the web interface may now be used for downloading logs.

Backup configs may now be restored from the web interface instead of only using the command line.

More 6.4f1 features coming in a future discussion.

The Acme Packet SBC includes support for Early Media Suppression. This allows you to decide what Realms can and cannot support Early Media and in what direction Early Media is allowed. Taking it one step further, the Acme Packet SBC also supports Selective Early Media Suppression. This means that even if a realm is configured to receive Early Media for all calls, the SBC can still be configured to deny Early Media from certain realms (even if those realms can send early media to other destinations) through the use of Realm Groups.

Early Media is defined by an endpoint sending RTP/RTCP packets before a session is established by a 200 OK. Otherwise, the expected behavior is media does not begin flowing between endpoints until 200 OK is received. Instances where early media may occur is when a user calls the PSTN and an announcement is immediately played or a custom ringtone (ie. music) is heard when calling a mobile phone.

There may be a variety of reasons why one does not want to allow Early Media under specific conditions. For example (real world) a Service Provider supporting Hosted IP PBX subscribers and SIP Trunking customers sends PSTN traffic from their network to multiple PSTN Media Gateways. One gateway supports outbound Local and Domestic Long Distance calls and the other media gateway supports outbound International calls. The Service Provider doesn’t want to allow Early Media from the International gateway to flow inbound to their Hosted IP PBX customer base, but they will allow Early Media to flow from the International media gateway to their SIP Trunking customer base. The Local and Domestic Long Distance media gateway is allowed to send Early Media to either customer base.

First, it’s important to cover the basics of implementing Early Media Suppression. This may be configured within a Realm or a Session Agent using the early-media-allow parameter. When a call egresses the SBC to a specific realm, the matching realm’s early-media-allow setting will be applied to either allow-all, block-all, or block one-way early media until a 200 OK is received. Media must be anchored to the SBC in order for Early Media settings to have any effect. In this example, calls are originated from the Access (Hosted IP PBX) realm to the PSTN (Access realm’s next-hop is the Core realm).

This example has early-media-allow set to none in the realm configuration. Early Media is not allowed in either direction. Therefore, if an Access subscriber calls the PSTN, no Early Media is permitted back to the Access realm, and vice-versa.

This example has early-media-allow set to reverse which in this case allows a Hosted IP PBX subscriber to place a call to BroadWorks or the PSTN and hear early media. However, the subscribers are not permitted to send Early Media in the other direction since early-media-allow is set to reverse and not both.

So far we have statically defined to always always allow Early Media from one direction. Now for a scenario that is a little more tricky. Lets say calls originating from the Access realm which route to the ITSP-1 realm/media gateway are not allowed to hear Early Media. However, calls originating from the Access realm which route to the ITSP-2 realm/media gateway are allowed to hear Early Media. However, ITSP-1 is allowed to send Early Media to other realms (ie. Acces-2, Acces-3, etc.). Only the “Access” realm cannot receive Early Media from ITSP-1.

The trick in this topology is all media gateways are allowed to send Early Media to any realm, and Access is allowed to receive Early Media from any realm. We want to selectively suppress only when the Early Media RTP traverses specifically from ITSP-1 to Access.

By using Realm Groups, we can essentially build a logical binding of source and destination realms together and customize Early Media settings just for calls flowing across those two realms.

Now only calls that are originated from the Access realm to the ITSP-1 realm will be prohibited from receiving early media from the ITSP-1 Media Gateway. If the same call routes to ITSP-2, Early Media is allowed.

BGP Route flaps, accidental fiber cuts, equipment failure, these are all things that trigger outages and cause traffic to behave erratically and unpredictably. In the moment of crisis, a five minute voice outage feels like an eternity. What many people do not realize is, just when the network begins to “normalize” itself, this will cause a massive SIP registration flood pouring into the network as potentially hundreds of thousands of endpoints (or more) will send their SIP registrations which would typically severely spike CPU on the Softswitch infrastructure, thus causing a second outage for the VoIP network.

There are ways to protect the core network with the Acme Packet SBC. The Acme Packet Net-SAFE architecture definitely warrants its own deep dive discussion and is the official word on SBC security. It’s not my intention to try and cover it here. However, having worked with similar topologies and dealing with short, bursty outages where endpoints disappear for anywhere from 3 to 15 minutes then come back in massive waves, there are a couple of simple steps one can take to keep the network healthy and avoid a second outage due to a SIP Registration Overload condition occurring after the network layer has resolved itself..

The following is a simple example of a Hosted IP PBX topology where SIP endpoint registration is targeted to an Acme Packet Session Load Balancer (SLB) which distributes the registrations to multiple SBC cluster members. Typically the cluster members are geographically located in different cities, states, or countries, supporting millions of subscribers (this scenario only shows three cluster members, but more are supported by the SLB).

Now lets say that all of a sudden the Internet experiences a BGP route flap. That’s at least 3 minutes of downtime while the routers converge. The SIP phones will lose the ability to communicate with the SBC and the cached SIP Registrations in the SBC will expire and be flushed out. In Hosted NAT Traversal environments it is common for the SIP registration “expires” timer in the Contact header to be set anywhere from 60 – 180 seconds by the SBC.  The reason for the low number is the SBC wants the Firewall to keep the UDP ports open. Firewalls tend to close them quick and since the SBC detects that the phone is behind NAT, it forces the registration to occur more frequently. Otherwise, if the Firewall close the UDP ports, the phone will not be able to receive calls.

Without utilizing any additional sip-config options in the SBC configuration, the SBC will forward all the registrations it receives from the SIP Endpoint to the core Softswitch when network service is back up and running. Note that if recommended DDoS security settings are in place (ie. Net-SAFE) the SBC will gracefully throttle the traffic entering the core so the Softswitch infrastructure does not get overwhelmed. The important things to note here is if a SIP registration cache entry expires, the SBC will forward the “new” registration to the core Softswitch once the network has normalized (as shown below). This could easily result in a flood of registrations happening all at once and put the Softswitch at risk.

By utilizing the register-grace-timer option under sip-config, the Acme Packet SBC will extend the life of the cached registration entries. Typical settings are between 600 and 900 seconds. The result is that when there is a network outage and SIP endpoints lose their connectivity to the SBC, the SBC will keep the binding of the endpoint in its cache. When the network is restored and all the SIP endpoints are able to communicate successfully with the SBC again (within the register-grace-timer period) there is no need for the SBC to forward all these registrations to the core Softswitch again. The Softswitch has no knowledge that an outage on the Public network occurred since the SBC is doing all the topology hiding and registration management that interfaces to the core.

Adding the following will enable the register-grace-timer value for 900 seconds.

So far the solution outlined above work great when SIP endpoints have a single ingress point into the network (Session Load Balancer) and the SBC cluster members are geographically dispersed in different cities, states, or countries.

The next example shows a topology where the Service Provider may have two different SBC pairs in two different geographic locations. In this case SIP Endpoints rely on DNS SRV records (or multiple static IP entries in the phone configuration files) to reach these SBC’s. The condition to consider here is intermittent network behavior. The network isn’t necessarily down hard where all endpoints would simply fail over to the second site. Instead, the primary site is flapping up and down, so phones are intermittently flapping their SIP registration between the primary and secondary sites.

The following is an example of a deployment where each site has its own independent High Availability pair of Acme Packet SBC’s. SIP Endpoints rely on DNS SRV records to support registration failover between Site 1 and Site 2.

The unfortunate and somewhat dysfunctional behavior in this topology is some SIP endpoints do not honor the expires= timer value sent by the SBC in a 200 OK response to their SIP registration. For example, if the SBC sends expires=180 seconds for the registration refresh, some endpoints reduce the value by as much as 50% and  send their registration at 90 seconds instead of 180 seconds. The law of SIP (RFC 3261) states that a SIP Endpoint should attempt the primary IP each time it tries to register. During an intermittent outage what will happen is the SIP endpoint is already registered with a cached entry at the primary site, then 90 seconds later fails to the secondary site, then 90 seconds later may fail back to the primary site again, then perhaps a fourth time to the secondary site again. Of course this leads to issues if the register-grace-timer is set since the registrations are cached and as a result will not be forwarded to the Softswitch again.

If the register-grace-timer is set to a value higher than 0 like the previous example where we used a value of 900, the SBC maintains the registration cache entry even though it has expired. For this particular topology, the register-grace-timer should be set to 0. This is critical when the SIP Endpoint flaps back and forth between the two sites, the registration will now be seen as a “new” entry anytime the SIP Endpoint is moving between sites. It’s not the ideal behavior we would like to see, but at minimum the SIP Endpoint is able to support Inbound and Outbound calls without completely failing.

Depending on which Softswitch vendor you are using, it may also be useful to add the force-unregistration option in sip-config. This way when the registration cache entry expires in the SBC, the SBC will also notify the Softswitch of the expired entry so it is removed and no SIP Invites are forked to both SBC’s when the subscriber receives a call.

The tradeoff of using force-unregistration is the core network becomes more chatty due to the SBC notifying the Softswitch about stale registrations, but the benefit is the SIP registrar’s location database is accurate and there won’t be any odd SIP messages flowing for incoming calls to a subscriber where location information is inaccurate.

In addition to the above sip-config options, the following commands are commonly used as throttling mechanisms. Actual values will vary based on the internal core Softswitch architecture, but this set of commands will guard the core SIP network when a large amount of SIP registrations come through so the Softswitch is not overwhelmed.

As mentioned previously, my goal is to cover the best way to recover from a network outage in a Hosted IP PBX environment supporting SIP Endpoints and Hosted NAT Traversal where SIP registrations are frequent. In the future I will provide a more in-depth look at the basics of Net-SAFE, DDoS, and SROP configuration.

Recently Cisco announced a blueprint and name change to the CCIE Voice certification. Beginning on November 21, 2013, the new Collaboration written exam will be available and replace the current Voice written exam. On February 14, 2014 (Valentine’s Day in the United States) the new CCIE Collaboration practical exam will be in full effect. Are you feeling the love yet?  If not, then the good news is all current CCIE Voice experts will be grandfathered into the Collaboration certification once passing the Collaboration Written exam (CCIE’s must retake their written exam every two years to maintain their CCIE status). For more information, read this excellent post by INE’s Mark Snow.

The new CCIE Collaboration blueprint include v9.x of CUCM, Unity Connection, UCCX, and CUPS. All of these are of course supported in VMWare ESXi but I have always preferred installing the Unified Communications platforms in VMWare Workstation. Currently I use CentOS 6.3 as the host operating system. It makes some general NTP, FTP, SFTP, and TFTP tasks easier when using VMWare this way. Another benefit is running Wireshark on the host operating system and capturing on “all interfaces” when things do not appear to be working as they should prior to doing practice labs.

My current setup is an HP xw8600 workstation with 48GB of memory and three Crucial SSD in a RAID configuration. The VM’s run incredibly fast even without pre-allocating hard drive space.

Installing Unified Communications 9.x in VMWare Workstation 9

• File > New Virtual Machine (Custom)
• Virtual Hardware Compatibility VMWare Workstation 9
• Guest Operating System Linux 5 32-bit
• Numer of Processors 1, Number of Cores per Processor 1
• Memory 2048MB (2GB)
• I/O Controller Type LSI Logic
• Virtual Disk Type SCSI
• Maximum Disk Size 80GB (160GB for Unity Connection)

Note the specifications above will work for CUCM, UCCX, and CUPS, but for Unity Connection the HD size must be 160GB.

When installing CUCM 9 you are required to use an ELM server for licensing. This may be a separate VM instance or the ELM that gets installed on the Publisher server (easiest). Unlike previous versions of CUCM, there are no free DLU’s for CUCM 9. The system will operate in demo mode for 60 days.  After 60 days you must rebuild your lab by reinstalling the Unified Communications applications. If you visit the Cisco Demo site there is a process for obtaining a 6 month demo license. Make sure to take VMWare Snapshots of your platforms after installing the demo license but before starting your practice labs. That way you can revert to a fresh system that is already licensed.

This is a  short update but should make a lot folks in the Service Provider world happy. Many people started using the web GUI based SIP Monitoring & Tracing feature in S-CX 6.3 noticed it was removed from S-CX 6.4. Note that S-CX is the Service Provider stream of code. SIP Monitoring & Tracing remained in the Enterprise code E-CX 6.4.

Last week (July 19th, 2013) Acme Packet (Oracle) released SCZ7.1.2 for both the 6300 and 4500 series SBC. SIP Monitoring & Tracing has returned!

The good news doesn’t stop there! ! If you are using the 4500 series (CPU2) and upgrade to SCZ7.1.2 this will also update the base operating systems from VxWorks to the Acme Packet Linux kernel (the 6300 was the first Acme Packet appliance to run the new Linux kernel). Without getting too “under the hood” lets just say the code is more optimized than ever and those SP’s grinding every last CPU cycle on their 4500′s will now see a an even bigger performance gain due to the optimized code within the new kernel. It’s definitely an upgrading worth pursuing in my opinion.

CLICK TO ENLARGE

Search, filter, customize displays, which helps to find calls quickly. QoS metrics and MOS scores are shown at the bottom. Captures may be downloaded as HTML files with a single click and emailed for local viewing exactly as they appear on the SBC. It is also possibly to download and view captures in an easy-to-read ASCII format for parsing.

Besides SIP Monitoring & Tracing there are other handy tools. By clicking on the System tab it is possible to download log files for troubleshooting, upload or download Local Route Table (LRT) files, SPL (plugin) files, playback media (for media injection), or to manage previously saved configurations.

To enable SIP Monitoring & Tracing the first step is to enable the web interface on the SBC. Once that has been completed it is necessary to create at least one monitoring filter. To capture traffic, apply the monitoring filter either globally under sip-monitoring, to a realm, or to a session-agent. In the following example the filter is applied to the Access realm which is the untrusted side of the SBC.

This short tip tends to slip through the cracks sometimes. If you prefer not to have your SBC output display all at once simply enable the more ACLI element.

Also, you can display just the specific show run element you are looking for. The example below will show whatever local-policy elements are configured on the system.

There is the ability to filter output based on specific criteria. Options will vary depending on which running-config element entered.

Either the entire output of the running-config or the sub-elements may be output to a file and retrieved through FTP via the management interface.

With the recent release of the Acme Packet E-CX 6.4m2 software the SBC now supports Remote Survivability. This is applicable in both a Service Provider Hosted IP PBX environment and Enterprises with large remote office using SIP phones that register to an IP PBX in a centralized Data Center model.

In this example I will focus primarily on the Service Provider model, although it’s just as relevant to the Enterprise model mentioned above since we are talking about remote phones registering over the WAN to a Softswitch or IP PBX. In the diagram below the SIP phones register from the Enterprise site over the Internet or MPLS network to the Service Provider core where a Broadsoft BroadWorks Softswitch resides. For those note familiar with BroadWorks, think of it as a very large software platform that has the ability to virtualize what looks like many IP PBX’s, each for dedicated customers. Most notably in large deployments it is common to see some sort of SIP enabled Firewall, ALG, or SBC (preferred) on the Enterprise premise. All SIP communication, including SIP registration and originated SIP calls, will egress the Enterprise phones through the SBC up into the Service Provider’s SBC and to BroadWorks. Even if calls are destined for another phone extension in the same Enterprise premise the call must go through the carrier core where CDR’s will be generated and any other Class 5 features may be provided for the call hair pinning back into the Enterprise to a different extension.

One of the biggest challenges is what happens if the SP core is unreachable. All registrations and call processing will fail, even for calls destined to another phone extension in the same Enterprise, since the SIP Invite must be able to reach the BroadWorks Softswitch or IP PBX to ring another extension in the same location.

If the Enterprise is using an Acme Packet SBC on premise the Survivability Mode will use either the P-Asserted Identity from the SIP endpoint or optionally the BroadWorks Survivability method using XML. When the SP Core or Enterprise data center becomes unreachable the Survivability Health Score degrades and the SBC knows to process registration and call traffic locally without sending the traffic to the SP Core.

In Survivability Mode phone A has the ability to call phone C even when the SP Core is unreachable. If the Enterprise normally uses abbreviated dialing such as 4-digit extensions, this will still continue to work, as well as dialing the full phone number.

Some Enterprises may want to have a local SIP Gateway (such as a Cisco ISR or Adtran Total Access IAD) on premise and utilize one or more PRI’s connected for PSTN access when the SP Core is unreachable. Depending on the Service Provider, the same SP used for SIP Services may also route to the PRI for overflow/outage scenarios so the Enterprise has both inbound and outbound redundancy on premise. The call originated from the SIP network the PSTN where there is no matching Address of Record for SIP phones, the SBC will route the calls to the SIP-PSTN Gateway.

Below is a basic Remote Survivability configuration.

Best-practice is to enable SIP Options Ping for each Session Agent. This way the SBC will pro-actively know when a Session Agent is in-service or out of service and degrade the health score appropriately.

The following has been a basic introduction to the Acme Packet Remote Survivability feature. There is more functionality and detail that may be applied and is well documented in the E-CX Maintenance Guide. Although a subset of this functionality has existed on other SIP ALG platforms in the past, Acme Packet has introduced many great new features relevant for Remote Survivability that do not exist elsewhere. All the resiliency and High-Availability that is inherent to the Acme Packet platform remains intact even when Remote Survivability is engaged.

The tail program is available on Unix operating systems and is used to view the tail end of  a text file or piped data. Once tail is running for a particular log file, and as the log file get populated with new data, the newly populated content is displayed on the screen real-time. This saves a step from having to use more or cat after the fact where a user would have to scroll all the way through the log file to the end in order to view the newly added data (and more data may continuously be added making it even more inconvenient).

In the case of the Acme Packet SBC, retrieving log files for viewing typically requires downloading them via FTP/SFTP. The Acme Packet SBC now supports tail from ACLI.

In the following example I want to see what dialog is happening when trying to register a SIP phone through the SBC.  It is possible to tail different log files.

NNOS-VM# notify sipd siplog

NNOS-VM# tail-logfile-open sipd sipmsg.log

waiting 1200 for response to request
completed

14:44.154 On [0:0]72.12.135.250:5060 received from 72.12.135.2:56883
REGISTER sip:acmepacket.local SIP/2.0
Via: SIP/2.0/UDP 172.20.10.4:56883;rport;branch=z9hG4bKPjjWHgcYsrr-T9bPbaZ1-hzFTE-U70g.F2
Route: <sip:acmepacket.local;lr>
Max-Forwards: 70
From: “Mark Holloway-8001″ <sip:7818018001@acmepacket.local>;tag=q.wbz5RZXd30uc7RwV2IP4GmSCGKDTwl
To: “Mark Holloway-8001″ <sip:7818018001@acmepacket.local>
Call-ID: VV5Yro3MQmi7fXX1CDeDax79zDnR.5jN
CSeq: 55540 REGISTER
User-Agent: Telephone 1.1.4
Contact: “Mark Holloway-8001″ <sip:7818018001@172.20.10.4:56883;ob>
Expires: 300
Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS
Content-Length: 0

—————————————-

14:44.158 On [0:0]72.12.135.250:5060 sent to 72.12.135.2:56883
SIP/2.0 404 Not Found
Via: SIP/2.0/UDP 172.20.10.4:56883;received=72.12.135.2;branch=z9hG4bKPjjWHgcYsrr-T9bPbaZ1-hzFTE-U70g.F2;rport=56883
From: “Mark Holloway-8001″ <sip:7818018001@acmepacket.local>;tag=q.wbz5RZXd30uc7RwV2IP4GmSCGKDTwl
To: “Mark Holloway-8001″ <sip:7818018001@acmepacket.local>;tag=aprqngfrt-ejkv9p0000m9f
Call-ID: VV5Yro3MQmi7fXX1CDeDax79zDnR.5jN
CSeq: 55540 REGISTER

—————————————-

NNOS-VM# notify sipd nosiplog

NNOS-VM# tail-logfile-close sipd sipmsg.log

The grep tool is a very popular Unix command line utility used to match a regular expression. Although “grep” doesn’t specifically exist on the Acme Packet SBC the find command serves a similar purpose.  This feature was introduced in ACLI release 6.4.

NNOS-SD# find [configuration | running-config | command]  [attribute]

NNOS-SD# find ?

<string>               find the specified string
command              find command within menus
configuration        find in editing configuration
running-config      find in running configuration

** My SBC has a realm called Access and I want to know where Access resides in the configuration **

NNOS-SD# find running-config Access

session-router -> local-policy [Access; *; *]
source-realm Access
description Access to Core

system -> network-interface [s0p0:0]
description Access

media-manager -> realm-config [Access]
identifier Access

session-router -> sip-interface [Access]
realm-id Access

media-manager -> steering-pool [72.12.135.250=16384+Access]
realm-id Access

Found 6 instances

NNOS-SD# find running-config Core

session-router -> local-policy [Access; *; *]
description Access to Core

session-router -> local-policy [Access; *; *] -> local-policy-attribute [Core; SAG:BW-APP-SERVERS]
realm Core
realm Core

system -> network-interface [s1p0:0]
description Core

media-manager -> realm-config [Core]
identifier Core

session-router -> session-agent [as1]
realm-id Core

session-router -> sip-config
home-realm-id Core

session-router -> sip-interface [Core]
realm-id Core

media-manager -> steering-pool [10.12.135.250=16384+Core]
realm-id Core

** I want to know what configuration elements support local-policy **

NNOS-SD# find local-policy
Command menu —————————–
(root) show configuration local-policy
(root) show running-config local-policy
(configure) session-router local-policy

Running configuration ——————–

Found 3 instances
NNOS-SD#

NNOS-SD# find force
Command menu —————————–
(root) halt force
(root) reboot force
(root) reboot fast force
(root) show configuration enforcement-profile
(root) show running-config enforcement-profile
(configure) media-manager realm-config enforcement-profile
(configure) session-router enforcement-profile
(configure) session-router session-agent enforcement-profile
(configure) session-router session-router force-report-trunk-info
(configure) session-router sip-config enforcement-profile
(configure) session-router sip-interface enforcement-profile

Running configuration ——————–

The Acme Packet SBC supports the ability to load balance SIP traffic several difference ways. Simply stated, a SIP invite will ingress to the SBC. The SBC will lookup the destination and if it results in a Session Agent Group (SAG) configuration with 2 or more Session Agents, the SBC will egress the SIP invite using one of the selected load balancing algorithms such as Least Busy.

The SBC also supports DNS SRV records for load balancing. For those not aware of what DNS SRV means, it’s a form of DNS that allows a single domain name (associated with a service and protocol) to resolve to many possible destinations. The destinations may be equal or weighted differently. The following is an example of a DNS SRV record for SIP over UDP looks like. In this case all records have an equal weight of 10.

The domain name sip.apkt.com will resolve to 5 different A records. The lookup must be a DNS SRV lookup for this to resolve correctly. In this case we are using SIP over UDP as the service which has been defined in Bind.

Here is how to enable DNS SRV load balancing on the SBC. Note this complies with RFC 3263 “Locating SIP Servers”. Once the config has been saved and actives the SBC will ping the address resolved by sip.apkt.com to see which IP’s are in service.

Now when a SIP Invite ingresses the SBC and matches a local-policy where the destination next-hop utilizes DNS SRC load balancing the traffic will be balanced across all nodes.

For those interested in the Acme Packet BCP documents, they are steadily reemerging as they are being updated with new content.

Previously they were available to customers who had access to the Acme Packet TAC portal. Fortunately they are now available to anyone who has an account on the Oracle-Acme Packet Community site located here.

Also worth noting is that all Acme Packet product documentation is publicly available here.  Enjoy!

The Acme Packet SBC is now on ACLI 7.3 code. Since the release of 7.x there have been a number of new enhancements. One of the most useful new features is Advanced Logging. This is like getting debug output in a log file without having to enable full debug. By setting a condition (or set of conditions) the SBC will capture the interesting traffic to a log file.

The Advanced Logging feature was created using Session Plugin Language (SPL). For those not familiar with SPL it is a plug-in language for the SBC based on the Lua scripting language. SPL provides an easy way to expand the SBC’s capabilities.

The first step is to enable the SPL and set the condition to match for logging. The below example sets advanced-logging for any call coming from phone number 4045551212.

By adding the additional line below only calls from 4045551212 to 7815551212 will be logged.

When finish it is recommended to disable advanced-logging using spl stop sip advanced-logging

It is possible to create different Advanced Logging profiles so they may be easily reused at another time without manually entering complex strings each time logging is enabled.

The following are the advanced-logging options which may be set.

Often times when deploying network monitoring probes they are passive devices tapped into the network using port spans, also known as port mirroring. Since the Acme Packet SBC is acting as a SIP b2bua and sees all SIP traffic it also has the ability to act as a probe without the need for port spans or mirroring.

Using IPFIX (IP Flow Information Export) the SBC will provide real-time statistics to the Oracle Communications Operations Monitor (formerly Palladion) Mediation Engine. The ME is able to provide real-time SIP ladder diagrams, call correlation, detect one-way RTP audio streams, provide alerting (fraud detection, call failures, emergency 911), display SBC configs, report on degraded MOS and RFactor scores, as well as show devices performing SIP Registration through the SBC and their associated endpoint make, model, and firmware information.

Currently Operations Monitor is the only platform I am aware of that can perform all of this real-time with no delay or lag. SIP ladder diagrams will display and continue to refresh in real-time whether a call is in progress or mid-call with intermediary dialog taking place.

All traffic traversing through the SBC north and southbound is reported using IPFIX 

The network-interface configuration element on the SBC determines which interface is used to send the IPFIX Export Data to the Mediation Engine. In most cases it will be the Management interface although it could be a slot0 or slot1 interface if desired.

Oracle Operations Monitor’s Mediation Engine supports an interactive multi-tenant web interface using HTML 5.

While the SBC is sending IPFIX Export Data the Operations Monitor platform will display SIP ladder diagrams on a per call basis showing a correlated call flow in real-time. 

Hover the mouse over a SIP message for more detail

Left click to lock multiple floating windows in place for additional visibility into multiple messages at the same time

Use the View option for more details such as CODEC, IP address, DTMF Events

View MOS scores in 10 second intervals for active calls for Called and Calling directions

Additional voice quality details are provided for each hop where media is anchored

SIP segments within a single call are broken down with additional details.

Note the PCAP export button for Wireshark compatibility (and PDF export)

See which Remote Workers or Hosted IP PBX endpoints are registering SIP through the SBC

Call correlation with five SIP hops 

View all calls traversing the network or call through a specific named “device” (SBC, IP PBX, Softswitch, Gateway, SIP Proxy, etc.)

Example of creating a new Alert when voice quality degrades

Create custom “apps” 

Use 200+ different Key Performance Indicators (KPI) or make custom KPI’s

Track calls by specific user

Create unique captures per user or device (SBC, Gateway, Softswitch, PBX)

To create a unique capture filter enter a telephone number or IP address. 

View trending voice quality metrics for the entire network. Notice there are some problematic RTP streams

Click on the red and see what device are in the RTP path effecting the MOS score