Category Archives: Networking

CLI Command Cheat-sheet for Aruba OS Wi-Fi

I recently put together a cheat-sheet for Aruba OS CLI commands which would be useful for a network team operating a new Aruba Wi-Fi network that I deployed, and thought to share this out.

Feel free to print out or PDF this post, it’s useful if you don’t have access to Internet and need a few quick reminders of what to type like I do. Yes, damn-what-is-that-command-itis is a thing.

CLI Tips and Tricks

<cmd> | include <specific string>
• Filter to display only lines that include a specific string
• Can use comma as an OR operator. Useful to include output headers, for example “show user-table | include IP,—,aa:bb:cc:11:22:33” will show column headers as well as the output line for the specific client.

<cmd> | exclude <specific string>
• Filter to display lines without the specific string

In AOS 8.x (Not in AOS 6.x), It is possible to chain include and exclude filters, for example:
<cmd> | include <specific string A> | exclude <specific string B>
<cmd> | include <specific string A> | include <specific string B>
<cmd> | exclude <specific string A> | exclude <specific string B>
The first displays results for (A AND NOT B), the second (A AND B), and the third (NOT A AND NOT B)

<cmd> | begin <specific string>
• Filter to display only lines from the first occurrence of a specific string

<cmd> [tab]
• Auto-completion, will complete a command if there is only one choice available

<cmd> ?
• Provide a list of commands which match the initial part of the <cmd> string
• Provide a list of parameters usable for the command

no paging
• Disable page breaks, useful for getting a huge amount of output for logging without requiring the administrator to hit [enter]. For example show run, show tech-support etc.
• Return to usual operation by typing “paging”

For commands which generate a lot of output, for example “show run” which will have page breaks, you can type “/” to search for a specific word, and “n” to search for the next occurrence. Similar to Linux “less” command.

Generally Useful Commands

show ap database / show ap database long
• shows details on all APs that the controller is aware of
• “long” includes AP Wired MAC and Serial Number

show ap active
• Shows APs which are currently Actively terminated on the controller, and summary of RF operating parameters

show switches (On Master Controller, if using Master-Local architecture)
• Shows if all configuration has been successfully pushed down to the controllers
• Shows OS versions of all of the controllers

show database synchronize (From Master Controller, if using Master-Local architecture)
• Validate that Master has successfully replicated configuration and DBs to the Backup Master

show master-redundancy (From Master Controller, if using Master-Local architecture)
• Show current state of master redundancy, i.e. who’s Master and who’s Backup.

apboot <various parameters, use tab to expand>
• Reboot specific APs or a set of APs.
• Useful if you don’t have access to PoE settings of the switchport
• Applicable only on the controller where AP is terminated

User Diagnostic – To be run on controller where users are present

show user-table (option to add “| include <client MAC>” to drill down)
• Shows general connectivity of the client, including IP address. If a client did not receive DHCP IP address, this entry will NOT exist. Hence…

show station-table mac <client MAC>
• Shows if the client (802.11 parlance calls this a Station or STA) is even associated to the network. If it is associated, but there is no entry on user-table, investigate role policies (Is DHCP blocked?) and DHCP server.

show user mac <client MAC>
• Shows VERBOSE details about a connected client. Use “| include” to narrow down for example:
o show user mac <client MAC> | include VLAN
o show user mac <client MAC> | include ACL
o show user mac <client MAC> | include SNR
o show user mac <client MAC> | include IP
o show user mac <client MAC> | include DHCP

(config) # logging level debugging user-debug aa:bb:cc:11:22:33
• Turns on logging for a specific client in the global configuration mode
• If not specified, and no other user-debugs exist, “show auth-tracebuf” will show for all user entries – Mind that the log buffer is not very long and you could miss what you’re looking for
• If not specified, and other user-debugs exist, will not show output for what is not explicitly specified.
• Remember to remove this (and any other debug commands) at the end of the debug session.

show auth-tracebuf (option to “ | include <client-mac>”)
• Show auth logs for the client (refer logging level debugging user-debug).
• Shows EAP transactions and interaction between client, controller and RADIUS.
• First thing to check if clients cannot connect – Look for Rejects!
• Follow up by checking for client auth failure reason at ClearPass Tracker

show ap remote debug mgmt-frames ap-name <ap-name> client-mac <client-mac>
• Shows the 802.11 management frame exchanges between the client and AP
• Useful to see association/authentication exchanges in the air and complements “show auth-tracebuf” for troubleshooting EAP exchange problems
• Also shows explicit deauthentication/disconnection exchanges

show ap arm history ap-name <AP Name>
• Shows AP ARM history – including channel and power changes over time

show ap arm client-match history client-mac <client MAC>
• Shows Client Match history for a specific client – Answers whether client were moved by ClientMatch (change AP, change radio band), and for what reason.

Data Path / Security Diagnostic

show rights
• Shows summarized list of all user roles in existence

show rights <role name>
• Shows policies, VLANs associated for a specific role

show datapath session table <client IP>
• Shows all concurrent connections and associated flags
• Each session creates two entries – Ingress and Egress entries, with “C” flag indicating the client initiating the connection
• Look out for zero bytes entries, missing return state entries, or “D” flag which could indicate firewall blocking the connections

Miscellaneous

show vrrp
• Shows VRRP information

show ip interface brief
• Shows IP interfaces

show controller-ip
• Shows which is the primary IP used by the controller, usually used by management (Master, AirWave, SNMP, RADIUS etc, unless explicitly specified otherwise.)

vSphere Metro Storage Cluster Networking: Part 3

This post has been much delayed for a number of reasons, namely because some feasible solutions became End of Sale, while others, based on field experience were not practically seen or deployed. In the meantime, other newer solutions which can address some of the issues we discussed earlier have now become available, so here is Part 3.

So back in Part 1, I blogged about considerations for the L2 DCI link for a vSphere Metro Cluster. In Part 2, I covered the potential routing pitfalls of stretching L2 networks across sites.

In Part 3, I’m going to discuss the methods which can be used to workaround the some of issues which we talked about in Part 2. Just to recap, the issues with stretched networks were:

  • Asymmetrical traffic flow across DC sites
  • Inability of network services (eg firewalls) to handle asymmetric traffic flow
  • Lack of VM site-awareness for optimized routing
  • Inefficient use of the DCI

VMware NSX Distributed Firewall with Asymmetrical Traffic Flows

In Part 2, I mentioned that it is possible for a VM to move between sites, with the result being that traffic to the VM (ingress traffic) could come in on say DC1, while traffic from the VM (egress traffic) could exit on DC2. Such a situation would cause issues with traditional firewalls, since these need to see traffic flows in both directions in order to allow or deny traffic correctly.

vMSC Invalid Firewall State

Perimeter Firewalls do not see consistent flow state

In the diagram above, the firewall at DC1 sees the “in” state of the flow from both User 1 and User 2 to VM1, which happens to have vMotioned to DC2. Assuming we’ve tweaked the setup for local egress, the VM will send traffic out via the DC2 router. As a consequence, the firewall at DC2 sees only the “out” state of the flow. This means that firewalls at both sites would observe any or all of the following issues and start dropping traffic because of state inconsistencies:

  • Incomplete TCP handshake / termination
  • Inconsistent sequence numbers
  • Unidirectional traffic flow

With NSX for vSphere, it’s actually possible to deploy a stateful firewall at the VM level using the Distributed Firewall (DFW) feature. NSX DFW works by having security policy defined centrally via NSX, which is then pushed down to corresponding VMs for enforcement at the micro level. With this being the case, we’ve brought the firewall closer to the VM itself by enforcing policy at the vNIC level.

NSX DFW sees flow state

NSX Distributed Firewall sees full flow state

Looking at the diagram above, the network ingress and egress paths of traffic to the VM are still inconsistent. However, the firewall enforcement point is at the vNIC level, which is tied to the VM. At the vNIC level, the DFW will always observe all traffic entering and exiting the VM. The DFW filter will have full information on the network traffic flows of the VM, and be able to appropriately apply stateful firewall policies, regardless of where the VM is or moves to, or how traffic arrives and departs from it. We’ve effectively resolved the problem of stateful perimeter firewalls not working due to not seeing the full traffic flow, by moving the firewall to the VM vNIC.

Other Methods

It bears mentioning that there are/were other methods of addressing some of the other network considerations that come with stretching networks. When writing both Part 1 and 2, I  considered writing more on these methods, however it appears that they are not quite feasible in the real world. Here is just a summary of what might have been.

Locator ID Separation Protocol (LISP): As you may have realized, there doesn’t seem to be a solution which has VM site awareness, so there is no way to optimize ingress routing to VMs according to which site they are located on (potentially also reducing DCI traffic). The fact is, LISP was supposed to address this issue, by being able to insert granular routes to VMs depending on where they resided. The biggest challenge with utilizing LISP in order to optimize ingress routing to the VM is that it requires ISPs to support LISP within their infrastructure. It is quite rare to come across such ISPs in the real world. Also, LISP plays a lot with insertion of host routes, which is its own set of network black magic.

DNS Optimization with Cisco ACE Load Balancers: Cisco also developed an orchestration solution utilizing its global and local load balancers to dynamically update DNS A records to point to wherever a VM was vMotioned to. This would enable new connections to directly reach the VM at it’s new location, thus also ensuring new connections do not have to traverse the DCI. It’s really quite a creative hack, though unfortunately the Cisco ACE product line was EoS’ed not long after the solution was published.

vSphere Distributed Virtual Switch: Packet analysis using ERSPAN

Packet analysis is invaluable in troubleshooting network issues and network monitoring. While packet analysis used to be used only in the domain of physical networks, that is no longer the case.

The vSphere Distributed Virtual Switch is now able to produce dumps of specific virtual network traffic and transport using ERSPAN to packet monitoring consoles. Yes, that’s right, using the Distributed Virtual Switch you can monitor network traffic in the virtual realm even if the traffic doesn’t actually hit the physical wire.

I didn’t quite see much material covering this so far, so I thought I’d show how this would work. For this blog post, I used the following:

  • Distributed Virtual Switch (vSphere Enterprise Plus)
  • Wireshark installed in a monitoring console (my personal laptop)
  • A VM which we want to monitor (a Windows 7 VM which is my jump box VM)

Let’s start with setting up Wireshark for packet capturing on the monitoring console. Opening Wireshark, go to Capture -> Interfaces.

That should open up a list of interfaces which we can capture from. Now I’d like to capture using the “Local Area Connection”, though it’s probably a good idea to find out what the IP address for that interface is. We’ll need to set it as a receiver for ERSPAN captured traffic. Click on “Options”.

We look out again for the “Local Area Connection” and note the IP address associated with the chosen receiving interface. In the case, it’s 10.2.1.110. We’ll checkbox the interface, and then click on “Start”.

Just like that, Wireshark will start dumping out all the traffic it gets on the interface. In this case, we only want to monitor traffic capture via ERSPAN on the Distributed Virtual Switch. Since ERSPAN encapsulates traffic in GRE, that’s what we’ll filter for. Type in “gre” into the filter field and click on “apply”, which should immediately filter out all the “noise” packets.