Wi-Fi System Status Update: December 2023

Spring and Summer 2023: Preparing for a Wi-Fi system overhaul

AUP has for many years provided fast, reliable internet connectivity to the campus community in all buildings.  In early 2023, our wireless system was still providing speeds that met and exceeded the requirements of the campus community.  However, the equipment being used on-campus was soon to be discontinued by its vendor, meaning that AUP would need to transition to a new Wi-Fi system by the end of the year to avoid security issues (possible security holes that would no longer be resolved by the Wi-Fi system's manufacturer) and reliability issues (lack of availability of replacement parts).  As such, IT Services initiated the process of searching among the available Wi-Fi solutions on the market to begin the process of finding a suitable replacement Wi-Fi system.
 

Defining our needs

Although the existing Wi-Fi system (based on what is known as "Wi-Fi 5" technology) was meeting the current needs of the community, we looked for opportunities to implement a cutting-edge system that would meet not only the current needs of the community, but that would meet the needs of the equipment and applications that would be used over the upcoming years.  In addition to the features that our current system offered, we therefore wished to add:

• Wi-Fi 6E compatibility
  • Wi-Fi 6E is a version of Wi-Fi 6 that uses an extra band of radio frequencies in the 6 GHz range, which would allow for much better performance than the existing system that used only the 5 GHz and 2.4 GHz bands.  This allows much faster speeds, and will be noticeable as more Wi-Fi 6E-compatible devices appear on our campus.
• Increased number of access points for better signal
  • Although IT Services has added extra Wi-Fi access points in the past when members of our community have had trouble connecting in certain areas, we proactively scouted the campus for areas with weaker-than-optimal signal, and determined how many access points we'd need to add to cover all of these zones.
  • In the end, we determined a need for just over 80 access points.  AUP's wireless deployment is relatively dense, with almost every public space and classroom having its own access point (or more than one, in the case of some lobby areas where demand is high).
• "Fast Roaming"
  • 802.11r "Fast Roaming" technology allows compatible devices to roam from one Wi-Fi access point to another with virtually no drop in the connection, which provides a major experience improvement over the older system where roaming between access points could cause a connection drop lasting several seconds.
     

Contacting vendors, and performing tests

Having determined the specifications we needed to provide our community with an excellent Wi-Fi experience, we contacted four market-leading vendors, detailed our needs, looked over the solutions that were offered.  We pursued conversations with three vendors and ended up getting equipment from two of those vendors that we could test.

Tests were performed in the busiest location on campus during the summer semester, the Amex Café, allowing us to see how the access points would behave with a high amount of users.  After working through some technical issues with some of the test devices, we found a solution that performed very well and suited our needs.
 

Late summer 2023: Replacing the Wi-Fi system

With the assistance of AUP's internal maintenance team, IT Services replaced the entire Wi-Fi system in mid-August, mounting over 80 access points throughout the campus's seven buildings.  After initial tests, everything seemed to be working very well: speeds and connection quality were excellent.
 

Fall 2023: Along with the crowds came the trouble

When students and faculty arrived on campus, we were thrilled to be able to offer a new, upgraded Wi-Fi experience, after having tested the new equipment in a busy location during the summer semester and having no problems before the arrival of students.  However, after the arrival of the entire student body on-campus, we encountered two issues:

• Four access points would randomly restart, taking 2-3 minutes to reboot.
• We received complaints of generally unreliable network connectivity on-campus, but were unable to pin down the origin of the problem, considering that the system had been functioning perfectly just a few days before - and we could not usually make the problem occur on-demand.
   

Access points restarting - what was the cause?

We opened a support request with the equipment supplier to ask why the access points were restarting.  After all, this was a brand-new system, and even a high amount of stress on the equipment shouldn't make the device restart.  The vendor was slow to respond, so we did investigations on our own, looking into what might cause this:

• Basic checks?
  • All cables were checked, power supplies were added, even network infrastructure was changed. Nothing we changed improved the situation, eliminating the possibility that it was a silly, obvious problem like a bad cable.
• Defective access points?
  • Next was the suspicion that the four access points being used were victims of a manufacturing defect. We replaced all four access points with spares, and they all continued to have the same issues.
• A bug in their firmware?
  • Originally, our Wi-Fi equipment had not worked well with some of our existing network equipment, but the issue was resolved by an update from the manufacturer.  We suspected that perhaps the bug hadn't been properly fixed, and moved around equipment in our wired network to test this hypothesis, but it didn't seem to make a difference.
• Location?
  • We swapped the locations of an in-service access point that did not suffer any issues with one that was suffering from random reboots... and the issue followed the installation location itself, not the access point.  With this information, we suspected that the access points were having an issue with something in the specific affected locations.

We provided ongoing updates to the vendor, but their support was not helpful.  Their support did confirm, after several weeks of data collection, that they discovered that the software in the access points was access points was crashing on their own, and that it was not a fault in our network that caused the issue.  Although this provided welcome reassurance that we had not incorrectly configured the system, it didn't fix this annoying problem.
 

General performance issues - or, "the difficulty in finding a problem that you can't recreate"

We received complaints of bad connectivity in many areas (mainly in the Quai d'Orsay and Combes buildings), and although we would occasionally see the issues ourselves when walking around the buildings, we could not reliably recreate them, and usually the students and staff who raised these issues with us could not recreate the issues on demand.  We noticed that issues mostly disappeared when few people were in the buildings, but that even with lots of people present, speeds were often excellent - when things worked.  We opened a second ticket with the Wi-Fi equipment vendor, but they also had difficulty discovering the origin of the problem, which seemed to pop up and disappear at any moment.  However, it was clear that there was an issue, due to the numerous complaints we received about the new system.

The vendor's support team investigated possible issues in our existing network backbone that might cause the problems about which we were hearing, but did not find anything wrong, leaving us in a situation where even the equipment's manufacturer confirmed that nothing was set up incorrectly yet could not find the origin of the issues.
 

Finally, progress!

After over a month of receiving disappointing non-responses on both of the support requests that we'd raised, the issue was finally escalated to someone working for the equipment vendor who was able to push their engineering team to work on identifying the issues.  After a few weeks, they were able to find what was causing the four wireless access points to crash (and we later found, what was also causing the general performance issues).
 

So, what was the problem?

Technical Background: IPv6

Devices on the internet need a way to speak with each other, which they do using the Internet Protocol version 4 (IPv4) and an accompanying "IP address".  Traditionally, each device was assigned a temporary or permanent IPv4 address consisting of four numbers ranging from 0-255, separated by decimals: for example, 193.49.37.66 is an IPv4 address that could be used by a server on the internet.  However, because IPv4 addresses by nature only allow for a maximum of just over 4 billion unique addresses, which seemed like a lot in the 1980s but is entirely insufficient for the growth of the modern Internet, a newer Internet Protocol called IPv6 was introduced, with a new, longer address.  An IPv6 address is represented as 32 hexadecimal characters (0-9, a-f) divided into groups of four that are separated by colons: 2001:067c:2ebc:abcd:face:1967:b33f:cafe, for example, would be an example of a valid IPv6 address. This new format, allowing for 16^32 (16 to the power of 32) addresses, or about 340 undecillion addresses, provides plenty of room for the internet to grow.  However, until almost every device on the internet can use IPv6, new devices with IPv6 addresses generally need to use both older IPv4 addresses and newer IPv6 addresses to communicate.  We are currently partway through a decades-long transition period where online services are made available using both IPv4 and IPv6, as Internet providers and corporate/education networks slowly activate IPv6, and eventually will deactivate IPv4.

Major services such as Google, Facebook, and LinkedIn have been available using both the older IPv4 addresses and the newer IPv6 addresses for over a decade.  Around 70 percent of French internet connections support IPv6 (the third-highest ranking in the world in the world), and more than two-thirds of Internet traffic on IPv6-enabled networks (where both IPv6 and IPv4 addresses are available) is routed using IPv6.  However, corporate and educational networks are notorious for being very slow to implement this new IPv6 connectivity and continue to rely only on the older IPv4 protocol, often due to a lack of resources or training.  As a result, equipment that is provided by major networking companies is often better-tested with IPv4 than with IPv6, even if the equipment is certified to work with both.
 

Context: IPv6 and AUP

AUP activated IPv6 connections on its network in the first part of 2023, providing every non-wireless device with both IPv4 and IPv6 addresses.  This allowed our Internet connection to be more reliable, decreased latency, and allowed us to double our available bandwidth to the campus.  IPv6 was not activated on our Wi-Fi network at the time, because our old Wi-Fi system did not support IPv6 addresses. When we installed the new Wi-Fi system, IPv6 became instantly available on Wi-Fi, allowing us to offer complete IPv6 and IPv4 connectivity to our community.  Over the course of the summer no issues were reported, and our tests showed that the system was working as expected.
 

The source of our problems: code in the Wi-Fi access points' firmware

After almost two months of applying pressure on the Wi-Fi equipment vendor, they notified us that they had made progress.  They noticed that something about IPv6 (reminder, a protocol that is not yet widely implemented with their customers) was causing the Wi-Fi access points to crash and have performance issues.  They then created a network in their lab that was similar to ours, and started transferring large amounts of data over it using the IPv6 protocol, until they found a configuration where their test equipment started crashing the way that ours did.  The culprit?  A piece of software in the device, tasked with converting IPv6 "multicast" packets to IPv6 "unicast" packets, was simply not able to keep up with the high IPv6 traffic level on the network that started when students arrived on campus for the Fall semester.  Given that this software is essential to our IPv6 network configuration and cannot be disabled, their suggestion was to temporarily block IPv6 traffic on the Wi-Fi network, to effectively work around the issue in their Wi-Fi system.
 

Temporary fix: disabling IPv6 on our Wi-Fi networks

After temporarily disabling IPv6 traffic on our Wi-Fi access points, as the vendor recommended:

• The access points that had been crashing started working reliably.
• Complaints about sporadic Wi-Fi connections stopped.
• The connection speed improved in many high-traffic locations.

Because virtually all online services are still offered using the old IPv4 protocol, we made the decision that until the vendor fixes their devices' handling of IPv6, we will leave IPv6 deactivated, offering a reliable, fast connection using only IPv4.  This is the current state of the network, and it has been reliable: an IPv4-only network, similar to what we had prior to summer of 2023, but with better performance and connection stability than our old Wi-Fi system that had been replaced.
 

Moving forward

Because implementing IPv6 has technical advantages, we continue to hold conversations with the vendor regarding the development of a permanent solution, and have had regular meetings with their US-based Director of Product Management to receive updates on their progress.  As the vendor works to resolve this issue, we will keep IPv6 connectivity turned off on our Wi-Fi networks, to maintain a high quality of service using the IPv4 protocol.  When the vendor provides us with a technical solution that will allow us to re-activate IPv6 on the Wi-Fi network, we will deploy it slowly, with extensive testing, until we are satisfied that they have fixed the issue in such a way that we can reactivate IPv6 campus-wide.

We would like to thank the AUP community for their patience as we worked to identify the problem and to resolve the issues that people have experienced.  We encourage all members of the community who have any questions or concerns relating to Wi-Fi at AUP to visit the IT Services office in room Q-A201, or to e-mail itservicesaup.edu.