Обнулить ошибки на порту cisco

Страница была создана 28.04.2022

Команда показывает статистику трафика и ошибок на определённом интерфейсе:

Switch#show interfaces имя_интерфейса

Пример вывода команды show interfaces, обратите внимание, на выделенный текст желтым цветом.

Switch#show interfaces gi0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
 Hardware is Gigabit Ethernet, address is 001e.1478.b7b1 (bia 001e.1478.b7b1)
 Description: SW-2
 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
  reliability 255/255, txload 1/255, rxload 1/255
 Encapsulation ARPA, loopback not set
 Keepalive set (10 sec)
 Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
 input flow-control is off, output flow-control is unsupported
 ARP type: ARPA, ARP Timeout 04:00:00
 Last input 00:00:00, output 00:00:00, output hang never
 Last clearing of «show interface» counters never
 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 42164
 Queueing strategy: fifo
 Output queue: 0/40 (size/max)
 5 minute input rate 781000 bits/sec, 122 packets/sec
 5 minute output rate 183000 bits/sec, 65 packets/sec
  75482 packets input, 104620499 bytes, 0 no buffer
  Received 6352 broadcasts (3951 multicasts)
  0 runts, 0 giants, 0 throttles
  105684 input errors, 103301 CRC, 0 frame, 0 overrun, 0 ignored
  0 watchdog, 3951 multicast, 0 pause input
  0 input packets with dribble condition detected
  39937001 packets output, 2917338077 bytes, 0 underruns
  0 output errors, 0 collisions, 4 interface resets
  10 unknown protocol drops
  0 babbles, 0 late collision, 0 deferred
  0 lost carrier, 0 no carrier, 0 pause output
  0 output buffer failures, 0 output buffers swapped out

После того, как вы устранили вероятную ошибку, нужно сбросить счётчики, чтобы убедиться, что ошибок больше нет.

Switch#clear counters gi0/1

После сброса, повторно проверяем счетчики, как видим счетчики обнулились, в примере я выделил их жёлтым цветом.

Switch#show interfaces gi0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
 Hardware is Gigabit Ethernet, address is 001e.1478.b7b1 (bia 001e.1478.b7b1)
 Description: SW-2
 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
  reliability 255/255, txload 1/255, rxload 1/255
 Encapsulation ARPA, loopback not set
 Keepalive set (10 sec)
 Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
 input flow-control is off, output flow-control is unsupported
 ARP type: ARPA, ARP Timeout 04:00:00
 Last input 00:00:00, output 00:00:00, output hang never
 Last clearing of «show interface» counters 00:00:08
 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
 Queueing strategy: fifo
 Output queue: 0/40 (size/max)
 5 minute input rate 1352000 bits/sec, 306 packets/sec
 5 minute output rate 313000 bits/sec, 91 packets/sec
  1274 packets input, 455165 bytes, 0 no buffer
  Received 199 broadcasts (118 multicasts)
  0 runts, 0 giants, 0 throttles
  0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
  0 watchdog, 118 multicast, 0 pause input
  0 input packets with dribble condition detected
  663 packets output, 312346 bytes, 0 underruns
  0 output errors, 0 collisions, 0 interface resets
  0 unknown protocol drops
  0 babbles, 0 late collision, 0 deferred
  0 lost carrier, 0 no carrier, 0 pause outputv
  0 output buffer failures, 0 output buffers swapped out

В таблице показаны некоторые значение и описания к ним.

Introduction

This document describes the errdisabled state, how to recover from it, and provides examples of errdisable recovery. This document uses the terms errdisable and error disable interchangeably. Customers often contact Cisco Technical Support when they notice that one or more of their switch ports have become error disabled, which means that the ports have a status of errdisabled. These customers want to know why the error disablement happened and how they can restore the ports to normal.

Note: The port status of err-disabled displays in the output of the show interfaces interface_number status command.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

To create the examples in this document, you need two Cisco Catalyst 4500/6500 Series Switches (or the equivalent) in a lab environment with cleared configurations. The switches must run Cisco IOS^® Software and each switch must have two Fast Ethernet ports that are capable of EtherChannel and PortFast.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Platforms That Use Errdisable

The errdisable feature is supported on these Catalyst switches:

• Catalyst switches that run Cisco IOS Software:

  • 2900XL / 3500XL

  • 2940 / 2950 / 2960 / 2970

  • 3550 / 3560 / 3560-E / 3750 / 3750-E

  • 3650 / 3850

  • 4500 / 4503 / 4506 / 4507 / 4510 / 4500-X

  • 6500 / 6503 / 6504 / 6506 / 6509

  • 9200 / 9300 / 9400 / 9500

The way in which errdisable is implemented varies between software platforms. This document specifically focuses on errdisable for switches that run Cisco IOS Software.

Errdisable

Function of Errdisable

If the configuration shows a port to be enabled, but software on the switch detects an error situation on the port, the software shuts down that port. In other words, the port is automatically disabled by the switch operating system software because of an error condition that is encountered on the port.

When a port is error disabled, it is effectively shut down and no traffic is sent or received on that port. The port LED is set to the color orange and, when you issue the show interfaces command, the port status shows err-disabled. Here is an example of what an error-disabled port looks like from the command-line interface (CLI) of the switch:

cat6knative#show interfaces gigabitethernet 4/1 status 

Port    Name       Status       Vlan       Duplex  Speed Type
Gi4/1              err-disabled 100          full   1000 1000BaseSX

Or, if the interface has been disabled because of an error condition, you can see messages that are similar to these in both the console and the syslog:

%SPANTREE-SP-2-BLOCK_BPDUGUARD: 
   Received BPDU on port GigabitEthernet4/1 with BPDU Guard enabled. Disabling port.
%PM-SP-4-ERR_DISABLE: 
   bpduguard error detected on Gi4/1, putting Gi4/1 in err-disable state

This example message displays when a host port receives the bridge protocol data unit (BPDU). The actual message depends on the reason for the error condition.

The error disable function serves two purposes:

• It lets the administrator know when and where there is a port problem.

• It eliminates the possibility that this port can cause other ports on the module (or the entire module) to fail.

  Such a failure can occur when a bad port monopolizes buffers or port error messages monopolize interprocess communications on the card, which can ultimately cause serious network issues. The error disable feature helps prevent these situations.

Causes of Errdisable

This feature was first implemented in order to handle special collision situations in which the switch detected excessive or late collisions on a port. Excessive collisions occur when a frame is dropped because the switch encounters 16 collisions in a row. Late collisions occur because every device on the wire did not recognize that the wire was in use. Possible causes of these types of errors include:

• A cable that is out of specification (either too long, the wrong type, or defective)

• A bad network interface card (NIC) card (with physical problems or driver problems)

• A port duplex misconfiguration

  A port duplex misconfiguration is a common cause of the errors because of failures to negotiate the speed and duplex properly between two directly connected devices (for example, a NIC that connects to a switch). Only half-duplex connections can ever have collisions in a LAN. Because of the carrier sense multiple access (CSMA) nature of Ethernet, collisions are normal for half duplex, as long as the collisions do not exceed a small percentage of traffic.

There are various reasons for the interface to go into errdisable. The reason can be:

• Duplex mismatch

• Port channel misconfiguration

• BPDU guard violation

• UniDirectional Link Detection (UDLD) condition

• Late-collision detection

• Link-flap detection

• Security violation

• Port Aggregation Protocol (PAgP) flap

• Layer 2 Tunneling Protocol (L2TP) guard

• DHCP snooping rate-limit

• Incorrect GBIC / Small Form-Factor Pluggable (SFP) module or cable

• Address Resolution Protocol (ARP) inspection

• Inline power

Note: Error-disable detection is enabled for all of these reasons by default. In order to disable error-disable detection, use the no errdisable detect cause command. The show errdisable detect command displays the error-disable detection status.

Determine If Ports Are in the Errdisabled State

You can determine if your port has been error disabled if you issue the show interfaces command.

Here is an example of an active port:

cat6knative#show interfaces gigabitethernet 4/1 status 

!--- Refer to show interfaces status for more information on the command.

Port    Name               Status       Vlan       Duplex  Speed Type
Gi4/1                      Connected    100          full   1000 1000BaseSX

Here is an example of the same port in the error disabled state:

cat6knative#show interfaces gigabitethernet 4/1 status 

!--- Refer to show interfaces status for more information on the command.

Port    Name               Status       Vlan       Duplex  Speed Type
Gi4/1                      err-disabled 100          full   1000 1000BaseSX

Note: When a port is error disabled, the LED on the front panel that is associated with the port is set to the color orange.

Determine the Reason for the Errdisabled State (Console Messages, Syslog, and the show errdisable recovery Command)

When the switch puts a port in the error-disabled state, the switch sends a message to the console that describes why it disabled the port. The example in this section provides two sample messages that show the reason for port disablement:

• One disablement is because of the PortFast BPDU guard feature.

• The other disablement is because of an EtherChannel configuration problem.

Note: You can also see these messages in the syslog if you issue the show log command.

Here are the sample messages:

%SPANTREE-SP-2-BLOCK_BPDUGUARD: 
   Received BPDU on port GigabitEthernet4/1 with BPDU Guard enabled. Disabling port.

%PM-SP-4-ERR_DISABLE: 
   bpduguard error detected on Gi4/1, putting Gi4/1 in err-disable state

 %SPANTREE-2-CHNMISCFG: STP loop - channel 11/1-2 is disabled in vlan 1

If you have enabled errdisable recovery, you can determine the reason for the errdisable status if you issue the show errdisable recovery command. Here is an example:

cat6knative#show errdisable recovery
ErrDisable Reason    Timer Status
-----------------    --------------
udld                 Enabled
bpduguard            Enabled
security-violatio    Enabled
channel-misconfig    Enabled
pagp-flap            Enabled
dtp-flap             Enabled
link-flap            Enabled
l2ptguard            Enabled
psecure-violation    Enabled
gbic-invalid         Enabled
dhcp-rate-limit      Enabled
mac-limit            Enabled
unicast-flood        Enabled
arp-inspection       Enabled

Timer interval: 300 seconds

Interfaces that will be enabled at the next timeout:

Interface      Errdisable reason      Time left(sec)
---------    ---------------------    --------------
  Fa2/4                bpduguard          273

Recover a Port from Errdisabled State

This section provides examples of how you can encounter an error-disabled port and how to fix it, as well as a brief discussion of a few additional reasons that a port can become error disabled. In order to recover a port from the errdisable state, first identify and correct the root problem, and then reenable the port. If you reenable the port before you fix the root problem, the ports just become error disabled again.

Correct the Root Problem

After you discover why the ports were disabled, fix the root problem. The fix depends on what triggered the problem. There are numerous things that can trigger the shutdown. This section discusses some of the most noticeable and common causes:

• EtherChannel misconfiguration

  In order for EtherChannel to work, the ports that are involved must have consistent configurations. The ports must have the same VLAN, the same trunk mode, the same speed, the same duplex, and so on. Most of the configuration differences within a switch are caught and reported when you create the channel. If one switch is configured for EtherChannel and the other switch is not configured for EtherChannel, the spanning tree process can shut down the channeled ports on the side that is configured for EtherChannel. The on mode of EtherChannel does not send PAgP packets to negotiate with the other side before channeling; it just assumes that the other side is channeling. In addition, this example does not turn on EtherChannel for the other switch, but leaves these ports as individual, unchanneled ports. If you leave the other switch in this state for a minute or so, Spanning Tree Protocol (STP) on the switch where the EtherChannel is turned on thinks that there is a loop. This puts the channeling ports in the errdisabled state.

  In this example, a loop was detected and the ports were disabled. The output of the show etherchannel summary command shows that the Number of channel-groups in use is 0. When you look at one of the ports that are involved, you can see that the status is err-disabled:

  %SPANTREE-2-CHNL_MISCFG: Detected loop due to etherchannel misconfiguration of Gi4/1
  
  cat6knative#show etherchannel summary
  
  !--- Refer to show etherchannel for more information on the command.
  
  Flags:  D - down        P - in port-channel
          I - stand-alone s - suspended
          H - Hot-standby (LACP only)
          R - Layer3      S - Layer2
          U - in use      f - failed to allocate aggregator
  
          u - unsuitable for bundling
  Number of channel-groups in use: 0
  Number of aggregators:           0
  
  Group  Port-channel  Protocol    Ports
  ------+-------------+-----------+-----------------------------------------------

  The EtherChannel was torn down because the ports were placed in errdisable on this switch.

  cat6knative#show interfaces gigabitethernet 4/1 status
  
  Port    Name               Status       Vlan       Duplex  Speed Type
  Gi4/1                      err-disabled 100          full   1000 1000BaseSX

  In order to determine what the problem was, look at the error message. The message indicates that the EtherChannel encountered a spanning tree loop. As this section explains, this problem can occur when one device (the switch, in this case) has EtherChannel turned on manually with use of the on mode (as opposed to desirable) and the other connected device (the other switch, in this case) does not have EtherChannel turned on at all. One way to fix the situation is to set the channel mode to desirable on both sides of the connection, and then reenable the ports. Then, each side forms a channel only if both sides agree to channel. If they do not agree to channel, both sides continue to function as normal ports.

  cat6knative(config-terminal)#interface gigabitethernet 4/1
  cat6knative(config-if)#channel-group 3 mode desirable non-silent
  

• Duplex mismatch

  Duplex mismatches are common because of failures to autonegotiate speed and duplex properly. Unlike a half duplex device, which must wait until there are no other devices that transmit on the same LAN segment, a full-duplex device transmits whenever the device has something to send, regardless of other devices. If this transmission occurs while the half-duplex device transmits, the half-duplex device considers this either a collision (during the slot time) or a late collision (after the slot time). Because the full-duplex side never expects collisions, this side never realizes that it must retransmit that dropped packet. A low percentage rate of collisions is normal with half duplex, but is not normal with full duplex. A switch port that receives many late collisions usually indicates a duplex mismatch problem. Be sure that the ports on both sides of the cable are set to the same speed and duplex. The show interfaces interface_number command tells you the speed and duplex for Catalyst switch ports. Later versions of Cisco Discovery Protocol (CDP) can warn you about a duplex mismatch before the port is put in the error-disabled state.

  In addition, there are settings on a NIC, such as autopolarity features, that can cause the problem. If you are in doubt, turn these settings off. If you have multiple NICs from a vendor and the NICs all appear to have the same problem, check the manufacturer website for the release notes and be sure that you have the latest drivers.

  Other causes of late collisions include:

  • A bad NIC (with physical problems, not just configuration problems)

  • A bad cable

  • A cable segment that is too long

• BPDU port guard

  A port that uses PortFast must only connect to an end station (such as a workstation or server) and not to devices that generate spanning tree BPDUs, such as switches, or bridges and routers that bridge. If the switch receives a spanning tree BPDU on a port that has spanning tree PortFast and spanning tree BPDU guard enabled, the switch puts the port in errdisabled mode in order to guard against potential loops. PortFast assumes that a port on a switch cannot generate a physical loop. Therefore, PortFast skips the initial spanning tree checks for that port, which avoids the timeout of end stations at bootup. The network administrator must carefully implement PortFast. On ports that have PortFast enabled, BPDU guard helps ensure that the LAN stays loop-free.

  This example shows how to turn on this feature. This example was chosen because creation of an error-disable situation is easy in this case:

  cat6knative(config-if)#spanning-tree bpduguard enable
  
  !--- Refer to spanning-tree bpduguard for more information on the command.
  
  

  In this example, a Catalyst 6509 switch is connected to another switch (a 6509). The 6500 sends BPDUs every 2 seconds (with use of the default spanning tree settings). When you enable PortFast on the 6509 switch port, the BPDU guard feature watches for BPDUs that come in on this port. When a BPDU comes into the port, which means that a device that is not an end device is detected on that port, the BPDU guard feature error disables the port in order to avoid the possibility of a spanning tree loop.

  cat6knative(config-if)#spanning-tree portfast enable
  
  !--- Refer to spanning-tree portfast (interface configuration mode) 
  !--- for more information on the command.
  
  
  Warning: Spantree port fast start can only be enabled on ports connected
  to a single host.  Connecting hubs, concentrators, switches, bridges, etc. to
  a fast start port can cause temporary spanning tree loops.
  
  %PM-SP-4-ERR_DISABLE: bpduguard error detected on Gi4/1, putting Gi4/1 in err-disable state.

  In this message, the switch indicates that it received a BPDU on a PortFast-enabled port, and so the switch shuts down port Gi4/1.

  cat6knative#show interfaces gigabitethernet 4/1 status
  
  Port    Name               Status       Vlan       Duplex  Speed Type
  Gi4/1                      err-disabled 100          full   1000 1000BaseSX

  You need to turn off the PortFast feature because this port is a port with an improper connection. The connection is improper because PortFast is enabled, and the switch connects to another switch. Remember that PortFast is only for use on ports that connect to end stations.

  cat6knative(config-if)#spanning-tree portfast disable
  

• UDLD

  The UDLD protocol allows devices that are connected through fiber-optic or copper Ethernet cables (for example, Category 5 cabling) to monitor the physical configuration of the cables and detect when a unidirectional link exists. When a unidirectional link is detected, UDLD shuts down the affected port and alerts the user. Unidirectional links can cause a variety of problems, which include spanning-tree topology loops.

  Note: UDLD exchanges protocol packets between the neighboring devices. Both devices on the link must support UDLD and have UDLD enabled on the respective ports. If you have UDLD enabled on only one port of a link, it can also leave the end configured with UDLD to go to errdisable state.

  Each switch port that is configured for UDLD sends UDLD protocol packets that contain the port device (or port ID) and the neighbor device (or port IDs) that are seen by UDLD on that port. The neighboring ports must see their own device or port ID (echo) in the packets that are received from the other side. If the port does not see its own device or port ID in the incoming UDLD packets for a specific duration of time, the link is considered unidirectional. Therefore, the respective port is disabled and a message that is similar to this is printed on the console:

  PM-SP-4-ERR_DISABLE: udld error detected on Gi4/1, putting Gi4/1 in err-disable state.

  For more information on UDLD operation, configuration, and commands, refer to the document Configuring UniDirectional Link Detection (UDLD).

• Link-flap error

  Link flap means that the interface continually goes up and down. The interface is put into the errdisabled state if it flaps more than five times in 10 seconds. The common cause of link flap is a Layer 1 issue such as a bad cable, duplex mismatch, or bad Gigabit Interface Converter (GBIC) card. Look at the console messages or the messages that were sent to the syslog server that state the reason for the port shutdown.

  %PM-4-ERR_DISABLE: link-flap error detected on Gi4/1, putting Gi4/1 in err-disable state

  Issue this command in order to view the flap values:

  cat6knative#show errdisable flap-values
  
  !--- Refer to show errdisable flap-values for more information on the command.
  
  ErrDisable Reason    Flaps    Time (sec)
  -----------------    ------   ----------
  pagp-flap              3       30
  dtp-flap               3       30
  link-flap              5       10

• Loopback error

  A loopback error occurs when the keepalive packet is looped back to the port that sent the keepalive. The switch sends keepalives out all the interfaces by default. A device can loop the packets back to the source interface, which usually occurs because there is a logical loop in the network that the spanning tree has not blocked. The source interface receives the keepalive packet that it sent out, and the switch disables the interface (errdisable). This message occurs because the keepalive packet is looped back to the port that sent the keepalive:

  %PM-4-ERR_DISABLE: loopback error detected on Gi4/1, putting Gi4/1 in err-disable state

  Keepalives are sent on all interfaces by default in Cisco IOS Software Release 12.1EA-based software. In Cisco IOS Software Release 12.2SE-based software and later, keepalives are not sent by default on fiber and uplink interfaces. For more information, refer to Cisco bug ID CSCea46385 (registered customers only) .

  The suggested workaround is to disable keepalives and upgrade to Cisco IOS Software Release 12.2SE or later.

• Port security violation

  You can use port security with dynamically learned and static MAC addresses in order to restrict the ingress traffic of a port. In order to restrict the traffic, you can limit the MAC addresses that are allowed to send traffic into the port. In order to configure the switch port to error disable if there is a security violation, issue this command:

  cat6knative(config-if)#switchport port-security violation shutdown
  

  A security violation occurs in either of these two situations:

  • When the maximum number of secure MAC addresses is reached on a secure port and the source MAC address of the ingress traffic differs from any of the identified secure MAC addresses

    In this case, port security applies the configured violation mode.

  • If traffic with a secure MAC address that is configured or learned on one secure port attempts to access another secure port in the same VLAN

    In this case, port security applies the shutdown violation mode.

• L2pt Guard

  When the Layer 2 PDUs enter the tunnel or access port on the inbound edge switch, the switch overwrites the customer PDU-destination MAC address with a well-known Cisco proprietary multicast address (01-00-0c-cd-cd-d0). If 802.1Q tunneling is enabled, packets are also double-tagged. The outer tag is the customer metro tag and the inner tag is the customer VLAN tag. The core switches ignore the inner tags and forward the packet to all trunk ports in the same metro VLAN. The edge switches on the outbound side restore the proper Layer 2 protocol and MAC address information and forward the packets to all tunnel or access ports in the same metro VLAN. Therefore, the Layer 2 PDUs are kept intact and delivered across the service-provider infrastructure to the other side of the customer network.

  Switch(config)#interface gigabitethernet 0/7
  l2protocol-tunnel {cdp | vtp | stp}
  

  The interface goes to errdisabled state. If an encapsulated PDU (with the proprietary destination MAC address) is received from a tunnel port or access port with Layer 2 tunneling enabled, the tunnel port is shut down to prevent loops. The port also shuts down when a configured shutdown threshold for the protocol is reached. You can manually reenable the port (issue a shutdown, no shutdown command sequence) or if errdisable recovery is enabled, the operation is retried after a specified time interval.

  To recover the interface from errdisable state, reenable the port with the command errdisable recovery cause l2ptguard. This command is used to configure the recovery mechanism from a Layer 2 maximum rate error so that the interface can be brought out of the disabled state and allowed to try again. You can also set the time interval. Errdisable recovery is disabled by default; when enabled, the default time interval is 300 seconds.

• Incorrect SFP cable

  Ports go into errdisable state with the %PHY-4-SFP_NOT_SUPPORTED error message when you connect Catalyst 3560 and Catalyst 3750 Switches and use an SFP Interconnect Cable.

  The Cisco Catalyst 3560 SFP Interconnect Cable (CAB-SFP-50CM=) provides for a low-cost, point-to-point, Gigabit Ethernet connection between Catalyst 3560 Series Switches. The 50-centimeter (cm) cable is an alternative to the SFP transceivers to interconnect Catalyst 3560 Series Switches through their SFP ports over a short distance. All Cisco Catalyst 3560 Series Switches support the SFP Interconnect Cable.

  When a Catalyst 3560 Switch is connected to a Catalyst 3750 or any other type of Catalyst switch model, you cannot use the CAB-SFP-50CM= cable. You can connect both switches with a copper cable with SFP (GLC-T) on both devices instead of a CAB-SFP-50CM= cable.

• 802.1X Security Violation

  DOT1X-SP-5-SECURITY_VIOLATION: Security violation on interface GigabitEthernet4/8, 
  New MAC address 0080.ad00.c2e4 is seen on the interface in Single host mode
  %PM-SP-4-ERR_DISABLE: security-violation error detected on Gi4/8, putting Gi4/8 in err-disable state

  This message indicates that the port on the specified interface is configured in single-host mode. Any new host that is detected on the interface is treated as a security violation. The port has been error disabled.

  Ensure that only one host is connected to the port. If you need to connect to an IP phone and a host behind it, configure Multidomain Authentication Mode on that switchport.

  The Multidomain authentication (MDA) mode allows an IP phone and a single host behind the IP phone to authenticate independently, with 802.1X, MAC authentication bypass (MAB), or (for the host only) web-based authentication. In this application, Multidomain refers to two domains — data and voice — and only two MAC addresses are allowed per port. The switch can place the host in the data VLAN and the IP phone in the voice VLAN, though they appear to be on the same switch port. The data VLAN assignment can be obtained from the vendor-specific attributes (VSAs) received from the AAA server within authentication.

  For more information, refer to the Multidomain Authentication Mode section of Configuring 802.1X Port-Based Authentication.

Reenable the Errdisabled Ports

After you fix the root problem, the ports are still disabled if you have not configured errdisable recovery on the switch. In this case, you must reenable the ports manually. Issue the shutdown command and then the no shutdown interface mode command on the associated interface in order to manually reenable the ports.

The errdisable recovery command allows you to choose the type of errors that automatically reenable the ports after a specified amount of time. The show errdisable recovery command shows the default error-disable recovery state for all the possible conditions.

cat6knative#show errdisable recovery
ErrDisable Reason    Timer Status
-----------------    --------------
udld                 Disabled
bpduguard            Disabled
security-violatio    Disabled
channel-misconfig    Disabled
pagp-flap            Disabled
dtp-flap             Disabled
link-flap            Disabled
l2ptguard            Disabled
psecure-violation    Disabled
gbic-invalid         Disabled
dhcp-rate-limit      Disabled
mac-limit            Disabled
unicast-flood        Disabled
arp-inspection       Disabled

Timer interval: 300 seconds

Interfaces that will be enabled at the next timeout:

Note: The default timeout interval is 300 seconds and, by default, the timeout feature is disabled.

In order to turn on errdisable recovery and choose the errdisable conditions, issue this command:

cat6knative#errdisable recovery cause ?
  all                 Enable timer to recover from all causes
  arp-inspection      Enable timer to recover from arp inspection error disable
                      state
  bpduguard           Enable timer to recover from BPDU Guard error disable
                      state
  channel-misconfig   Enable timer to recover from channel misconfig disable
                      state
  dhcp-rate-limit     Enable timer to recover from dhcp-rate-limit error
                      disable state
  dtp-flap            Enable timer to recover from dtp-flap error disable state
  gbic-invalid        Enable timer to recover from invalid GBIC error disable
                      state
  l2ptguard           Enable timer to recover from l2protocol-tunnel error
                      disable state
  link-flap           Enable timer to recover from link-flap error disable
                      state
  mac-limit           Enable timer to recover from mac limit disable state
  pagp-flap           Enable timer to recover from pagp-flap error disable
                      state
  psecure-violation   Enable timer to recover from psecure violation disable
                      state
  security-violation  Enable timer to recover from 802.1x violation disable
                      state
  udld                Enable timer to recover from udld error disable state
  unicast-flood       Enable timer to recover from unicast flood disable state

This example shows how to enable the BPDU guard errdisable recovery condition:

cat6knative(Config)#errdisable recovery cause bpduguard

A nice feature of this command is that, if you enable errdisable recovery, the command lists general reasons that the ports have been put into the error-disable state. In this example, notice that the BPDU guard feature was the reason for the shutdown of port 2/4:

cat6knative#show errdisable recovery
ErrDisable Reason    Timer Status
-----------------    --------------
udld                 Disabled
bpduguard            Enabled
security-violatio    Disabled
channel-misconfig    Disabled
pagp-flap            Disabled
dtp-flap             Disabled
link-flap            Disabled
l2ptguard            Disabled
psecure-violation    Disabled
gbic-invalid         Disabled
dhcp-rate-limit      Disabled
mac-limit            Disabled
unicast-flood        Disabled
arp-inspection       Disabled

Timer interval: 300 seconds

Interfaces that will be enabled at the next timeout:

Interface      Errdisable reason      Time left(sec)
---------    ---------------------    --------------
  Fa2/4                bpduguard          290

If any one of the errdisable recovery conditions is enabled, the ports with this condition are reenabled after 300 seconds. You can also change this default of 300 seconds if you issue this command:

cat6knative(Config)#errdisable recovery interval timer_interval_in_seconds

This example changes the errdisable recovery interval from 300 to 400 seconds:

cat6knative(Config)#errdisable recovery interval 400

Verify

• show version—Displays the version of the software that is used on the switch.

• show interfaces interface interface_number status—Shows the current status of the switch port.

• show errdisable detect—Displays the current settings of the errdisable timeout feature and, if any of the ports are currently error disabled, the reason that they are error disabled.

Troubleshoot

• show interfaces status err-disabled—Shows which local ports are involved in the errdisabled state.

• show etherchannel summary—Shows the current status of the EtherChannel.

• show errdisable recovery—Shows the time period after which the interfaces are enabled for errdisable conditions.

• show errdisable detect—Shows the reason for the errdisable status.

For more information on how to troubleshoot switchport issues, refer to Troubleshooting Switch Port and Interface Problems.

Related Information

• Interface Is in errdisable Status Troubleshooting Hardware and Common Issues on Catalyst 6500/6000 Series Switches Running Cisco IOS System Software
• Spanning Tree PortFast BPDU Guard Enhancement
• Understanding EtherChannel Inconsistency Detection
• Troubleshooting Switch Port and Interface Problems
• LAN Product Support
• LAN Switching Technology Support
• Technical Support — Cisco Systems

Introduction

This document describes how to determine why a port or interface experiences problems.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document applies to Catalyst switches that run on Cisco IOS® System Software.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Conventions

Refer toCisco Technical Tips Conventionsfor more information on document conventions.

Note: To access tools and websites, you must be a registered Cisco client.

Troubleshoot the Physical Layer 

Use the LEDs to Troubleshoot

If you have physical access to the switch, it can save time to look at the port LEDs which give you the link status or can indicate an error condition (if red or orange). The table describes the LED status indicators for Ethernet modules or fixed-configuration switches:

Ensure that both sides have a link. A single broken wire or one shutdown port can cause the problem where one side has a link light, but the other side does not.

A link light does not guarantee that the cable is fully functional. The cable can have encountered physical stress that causes it to be functional at a marginal level. Normally you can identify this situation if the port has many packet errors, or the port constantly flaps (loses and regains link).

Check the Cable and Both Sides of the Connection

If the link light for the port does not come on, you can consider these possibilities:

Ethernet Copper and Fiber Cables

Ensure that you have the correct cable for the type of connection you want to make. Category 3 copper cable can be used for 10 Mbps unshielded twisted pair (UTP) connections but must never be used for 10/100 or 10/100/1000Mbps UTP connections. Always use either Category 5, Category 5e, or Category 6 UTP for 10/100 or 10/100/1000Mbps connections.

Warning: Category 5e and Category 6 cables can store high levels of static electricity because of the dielectric properties of the materials used in their construction. Always ground the cables (especially in new cable runs) to a suitable and safe earth ground before you connect them to the module.

For fiber, make sure you have the correct cable for the distances involved and the type of fiber ports that are used. The two options are single mode fiber (SMF) or multimode fiber (MMF). Make sure the ports on the devices that are connected together are both SMF, or both are MMF ports.

Note: For fiber connections, make sure the transmit lead of one port is connected to the receive lead of the other port. Connections for transmit-to-transmit and receive-to-receive do not work.

Ethernet and Fast Ethernet Maximum Transmission Distances

For more details on the different types of cables/connectors, cable requirements, optical requirements (distance, type, patch cables, and so on.), how to connect the different cables, and which cables are used by most Cisco switches and modules, refer to Catalyst Switch Cable Guide .

Troubleshoot the Gigabit Ethernet

If you have device A connected to device B over a Gigabit link, and the link does not come up, perform this procedure.

Step-by-Step Procedure

1. Verify device A and B use the same GBIC, short wavelength (SX), long wavelength (LX), long haul (LH), extended wavelength (ZX), or copper UTP (TX). Both devices must use the same type of GBIC to establish link. An SX GBIC needs to connect with an SX GBIC. An SX GBIC does not link with an LX GBIC. Refer to Mode-Conditioning Patch Cord Installation Note for more information.

2. Verify distance and cable used per GBIC as defined in this table.

   1000BASE-T and 1000BASE-X Port Cabling Specifications

1. The numbers given for multimode fiber-optic cable refer to the core diameter. For single-mode fiber-optic cable, 8.3 microns refers to the core diameter. The 9-micron and 10-micron values refer to the mode-field diameter (MFD), which is the diameter of the portion of the fiber that is light-carrying. This area consists of the fiber core plus a small portion that covers the cladding. The MFD is a function of the core diameter, the wavelength of the laser, and the refractive index difference between the core and the cladding.

2. Distances are based on fiber loss. Multiple splices and substandard fiber-optic cable reduce the cable distances.

3. Use with MMF only.

4. When you use an LX/LH GBIC with 62.5-micron diameter MMF, you must install a mode-conditioning patch cord (CAB-GELX-625 or equivalent) between the GBIC and the MMF cable on both the transmit and receive ends of the link. The mode-conditioning patch cord is required for link distances less than 328 feet (100 m) or greater than 984 feet (300 m). The mode-conditioning patch cord prevents the over use of the receiver for short lengths of MMF and reduces differential mode delay for long lengths of MMF. Refer to Mode-Conditioning Patch Cord Installation Note for more information.

5. Use with SMF only.

6. Dispersion-shifted single-mode fiber-optic cable.

7. The minimum link distance for ZX GBICs is 6.2 miles (10 km) with an 8-dB attenuator installed at each end of the link. Without attenuators, the minimum link distance is 24.9 miles (40 km).

3. If either device has multiple Gigabit ports, connect the ports to each other. This tests each device and verifies that the Gigabit interface functions correctly. For example, you have a switch that has two Gigabit ports. Wire Gigabit port one to Gigabit port two. Does the link come up? If so, the port is good. STP blocks on the port and prevents any loops (port one receive (RX) goes to port two transmit (TX), and port one TX goes to port two RX).

4. If single connection or Step 3 fails with SC connectors, loop the port back to itself (port one RX goes to port one TX). Does the port come up? If not, contact the TAC, as this can be a faulty port.

5. If steps 3 and 4 are successful, but a connection between device A and B cannot be established, loop ports with the cable that adjoins the two devices. Verify that there is not a faulty cable.

6. Verify that each device supports 802.3z specification for Gigabit auto-negotiation. Gigabit Ethernet has an auto-negotiation procedure that is more extensive than the one used for 10/100 Ethernet (Gigabit auto-negotiation spec: IEEE Std 802.3z-1998). When you enable link negotiation, the system auto-negotiates flow control, duplex mode, and remote fault information. You must either enable or disable link negotiation on both ends of the link. Both ends of the link must be set to the same value or the link cannot connect. Problems have been seen when you connect to devices manufactured before the IEEE 802.3z standard was ratified. If either device does not support Gigabit auto-negotiation, disable the Gigabit auto-negotiation, and it forces the link up. It takes 300msec for the card firmware to notify the software that a 10/100/1000BASE-TX link/port is down. The 300msec default debounce timer comes from the firmware polling timer to the linecards, which occurs every 300 msec. If this link is run in 1G (1000BASE-TX) mode, Gigabit sync, which occurs every 10msec, must be able to detect the link down faster. There is a difference in the link failure detection times when you run GigabitEthenet on copper versus GigabitEthernet over fiber. This difference in detection time is based on the IEEE standards.

Warning: Disable auto-negotiation and this hides link drops or physical layer problems. This is only required if end-devices such as older Gigabit NICs are used which cannot support IEEE 802.3z. Do not disable auto-negotiation between switches unless absolutely required to do so, as physical layer problems can go undetected, which results in STP loops. The alternative is to contact the vendor for software/hardware upgrade for IEEE 802.3z Gigabit auto-negotiation support.

For GigabitEthernet system requirements as well as Gigabit Interface Converters (GBICs), Coarse Wavelength Division Multiplexing (CWDM), and Small Form-Factor Pluggable (SFP) system requirements, refer to these:

• System Requirements to Implement Gigabit Ethernet on Catalyst Switches

• Catalyst GigaStack Gigabit Interface Converter Switch Compatibility Matrix

• Cisco Gigabit Ethernet Transceiver Modules Compatibility Matrix

• Cisco 10-Gigabit Ethernet Transceiver Modules Compatibility Matrix

For general configuration information and additional information on how to troubleshoot, refer to Configuring and Troubleshooting Ethernet 10/100/1000 MB Half/Full Duplex Auto-Negotiation .

Connected vs Notconnected

Most Cisco switches have a port in the notconnect state. This means it is currently not connected to anything, but it can connect if it has a good connection to another operational device. If you connect a good cable to two switch ports in the notconnect state, the link light must become green for both ports, and the port status must indicate connected. This means that the port is up as far as Layer 1 (L1) is concerned.

For Cisco IOS, you can use the show interfaces command to verify whether the interface is up, line protocol is up (connected) . The first up refers to the physical layer status of the interface. The line protocol up message shows the data link layer status of the interface and says that the interface can send and receive keepalives.

Router#show interfaces fastEthernet 6/1
FastEthernet6/1 is down, line protocol is down (notconnect)

!--- The interface is down and line protocol is down.
!--- Reasons: In this case,
!--- 1) A cable is not properly connected or not connected at all to this port.
!--- 2) The connected cable is faulty.
!--- 3) Other end of the cable is not connected to an active port or device.

!--- Note: For gigabit connections, GBICs need to be matched on each
!--- side of the connection.
!--- There are different types of GBICs, depends on the cable and
!--- distances involved: short wavelength (SX),
!--- long-wavelength/long-haul (LX/LH) and extended distance (ZX).
!--- An SX GBIC needs to connect with an SX GBIC;
!--- an SX GBIC does not link with an LX GBIC. Also, some gigabit
!--- connections require conditioning cables,
!--- that depend on the lengths involved.

Router#show interfaces fastEthernet 6/1
FastEthernet6/1 is up, line protocol is down (notconnect)

!--- The interface is up (or not in a shutdown state), but line protocol down.
!--- Reason: In this case, the device on the other side of the wire is a
!--- CatOS switch with its port disabled.

Router#show interfaces fastEthernet 6/1 status
Port   Name    Status       Vlan    Duplex   Speed  Type
Fa6/1          notconnect    1       auto     auto   10/100BaseTX

Ifshow interfacesshows up/ line protocol up (connected) but you see errors increment in the output of either command, refer to the Common Port and Interface Problems section of this document for advice.

Troubleshoot the Most Common Port and Interface Commands for Cisco IOS

This table shows the most common commands used to troubleshoot the port or interface problems on switches that run Cisco IOS System Software on the Supervisor.

Note: The right hand column on the next table gives a brief description of what the command does and lists any exceptions to the use per platform.

If you have the output of the supported commands from your Cisco device, you can use Cisco CLI Analyzer to display potential issues and fixes.

Understand the Specific Port and Interface Counter Output for Cisco IOS

Most switches have some way to track the packets and errors that occur on a port or interface. The common commands used to find this type of information are described in the Most Common Port and Interface Troubleshooting Commands for Cisco IOS section of this document.

Note: There can be differences in the implementation of the counters across various platforms and releases. Although the values of the counters are largely accurate, they are not very precise by design. In order to pull the exact statistics of the traffic, it is suggested that you use a sniffer to monitor the necessary ingress and egress interfaces.

Excessive errors for certain counters usually indicate a problem. When you operate at half-duplex setup, some data link errors increment in Frame Check Sequence (FCS), alignment, runts, and collision counters are normal. Generally, a one percent ratio of errors to total traffic is acceptable for half-duplex connections. If the ratio of errors to input packets is greater than two or three percent, performance degradation can be noticed.

In half-duplex environments, it is possible for both the switch and the connected device to sense the wire and transmit at exactly the same time and result in a collision. Collisions can cause runts, FCS, and alignment errors due to the frame not completely copied to the wire, which results in fragmented frames.

When you operate at full-duplex, errors in FCS, Cyclic Redundancy Checks (CRC), alignment, and runt counters must be minimal. If the link operates at full-duplex, the collision counter is not active. If the FCS, CRC, alignment, or runt counters increment, check for a duplex mismatch. Duplex mismatch is a situation where the switch operates at full-duplex and the connected device operates at half-duplex, or vice versa. The results of a duplex mismatch are extremely slow performance, intermittent connectivity, and loss of connection. Other possible causes of data link errors at full-duplex are bad cables, faulty switch ports, or NIC software/hardware issues. See the Common Port and Interface Problems section of this document for more information.

Show Interfaces for Cisco IOS

The show interfaces card-type {slot/port}command is the used command for Cisco IOS on the Supervisor to display error counters and statistics. An alternative to this command (for Catalyst 6000, 4000, 3550, 2970 2950/2955, and 3750 series switches) is theshow interfacescard-type <slot/port>counters errors command which only displays the interface error counters. Refer to Table 1 for explanations of the error counter output.

Note: For 2900/3500XL Series switches use theshow interfacescard-type {slot/port}command with theshow controllers Ethernet-controllercommand.

Router#sh interfaces fastEthernet 6/1
FastEthernet6/1 is up, line protocol is up (connected)
   Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
   MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
      reliability 255/255, txload 1/255, rxload 1/255
   Encapsulation ARPA, loopback not set
   Full-duplex, 100Mb/s
   input flow-control is off, output flow-control is off
   ARP type: ARPA, ARP Timeout 04:00:00
   Last input 00:00:14, output 00:00:36, output hang never
   Last clearing of "show interface" counters never
   Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
   Queueing strategy: fifo
   Output queue :0/40 (size/max)
   5 minute input rate 0 bits/sec, 0 packets/sec
   5 minute output rate 0 bits/sec, 0 packets/sec

Theshow interfacescommand output up to this point is explained here (in order) :

• up, line protocol is up (connected) — The first up refers to the physical layer status of the interface. The line protocol up message shows the data link layer status of the interface and says that the interface can send and receive keepalives.

• MTU — The Maximum Transmission Unit (MTU) is 1500 bytes for Ethernet by default (for the max data portion of the frame).

• Full-duplex, 100Mb/s — Full-duplex and 100Mbps is the current speed and duplex setup of the interface. This does not tell you whether autoneg was used to achieve this. Use theshow interfaces fastEthernet 6/1 statuscommand to display this:

Router#show interfaces fastEthernet 6/1 status
Port    Name               Status       Vlan       Duplex  Speed Type
Fa6/1                      connected    1          a-full  a-100 10/100BaseTX

!--- Autonegotiation was used to achieve full-duplex and 100Mbps.

• Last input, output — The number of hours, minutes, and seconds since the last packet was successfully received or transmitted by the interface. This is useful to know when a dead interface failed.

• Last clearing of «show interface» counters — The last time the clear counters command was issued since the last time the switch was rebooted. The clear counters command is used to reset interface statistics.

Note: Variables that can affect routing (for example, load and reliability) are not cleared when the counters are cleared.

• Input queue — The number of packets in the input queue.Size/max/drops= the current number of frames in the queue / the max number of frames the queue can hold before it must start to drop frames / the actual number of frames dropped because the max queue size was exceeded.Flushesis used to count Selective Packet Discard (SPD) drops on the Catalyst 6000 Series that run Cisco IOS. (The flushes counter can be used but never increments on the Catalyst 4000 Series that run Cisco IOS.) SPD is a mechanism that quickly drops low priority packets when the CPU is overloaded in order to save some process capacity for high priority packets. The flushes counter in the show interface command output increments as part of selective packet discard (SPD), which implements a selective packet drop policy on the IP process queue of the router. Therefore, it applies to only process switched traffic.

  The purpose of SPD is to ensure that important control packets, such as routing updates and keepalives, are not dropped when the IP input queue is full. When the size of the IP input queue is between the minimum and maximum thresholds, normal IP packets are dropped based on a certain drop probability. These random drops are called SPD flushes.

• Total output drops — The number of packets dropped because the output queue is full. A common cause is traffic from a high bandwidth link that is switched to a lower bandwidth link or traffic from multiple inbound links that are switched to a single outbound link. For example, if a large amount of traffic flow comes in on a gigabit interface and is switched out to a 100Mbps interface, this can cause output drops to increment on the 100Mbps interface. This is because the output queue on that interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.

• Output queue — The number of packets in the output queue. Size/max means the current number of frames in the queue/the max number of frames the queue can hold before it is full and must start to drop the frames.

• 5 minute input/output rate — The average input and output rate seen by the interface in the last five minutes. Specify a shorter period of time to get an accurate read (to better detect traffic bursts for example and issue theload-interval <seconds>interface command.

    SeeTable 1for explanations of the error counter output.

!--- ...show interfaces command output continues.
     1117058 packets input, 78283238 bytes, 0 no buffer
      Received 1117035 broadcasts, 0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
      0 watchdog, 0 multicast, 0 pause input
      0 input packets with dribble condition detected
      285811 packets output, 27449284 bytes, 0 underruns
      0 output errors, 0 collisions, 2 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier
      0 output buffer failures, 0 output buffers swapped out

Note: There is a difference between the counter of show interface command output for a physical interface and a VLAN interface.The input packet counters increment in the output ofshow interfacefor a VLAN interface when that packet is Layer 3 (L3) processed by the CPU. Traffic that is Layer 2 (L2) switched never makes it to the CPU and is not counted in theshow interfacecounters for the VLAN interface. It would be counted on theshow interfaceoutput for the appropriate physical interface.

Theshow interfaces <card-type> <slot/port> counters errorscommand is used in Cisco IOS to display the output of the interface errors only. SeeTable 1for explanations of the error counter output.

Router#sh interfaces fastEthernet 6/1 counters errors

 Port        Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize OutDiscards
 Fa6/1               0          0          0          0         0           0

 Port      Single-Col Multi-Col  Late-Col Excess-Col Carri-Sen     Runts    Giants
 Fa6/1              0         0         0          0         0         0         0

Table 1. Cisco IOS error counter output forshow interfacesorshow interfaces<card-type> <x/y> counters errorsfor the Catalyst 6000 and 4000 Series.

Show Interfaces Counters for Cisco IOS

To monitor inbound and outbound traffic on the port as displayed by the next output, for unicast, multicast, and broadcast traffic. Theshow interfacescard-type {slot/port}counterscommand is used when you run Cisco IOS on the Supervisor.

Note: There is, an Out-Discard counter in the Cisco IOSshow interfaces counters errorscommand which is explained inTable 1.

Router#sh interfaces fas 6/1 counters

  Port            InOctets   InUcastPkts   InMcastPkts   InBcastPkts
  Fa6/1           47856076            23        673028           149

  Port           OutOctets  OutUcastPkts  OutMcastPkts  OutBcastPkts
  Fa6/1           22103793            17        255877          3280
  Router#
  
!--- Cisco IOS counters used to monitor inbound and outbound unicast, multicast
!--- and broadcast packets on the interface.

Show Counters Interface for Cisco IOS

Theshow counters interfacecard-type {slot/port}command was introduced in Cisco IOS software version 12.1(13)E for the Catalyst 6000 series only, it offers even more detailed statistics for ports and interfaces. This commanda display the 32-bit and 64-bit error counters per port or interface.

Show Controller Ethernet-Controller for Cisco IOS

For Catalyst 3750, 3550, 2970, 2950/2955, 2940, and 2900/3500XL switches use the command show controller ethernet-controller to display traffic counter and error counter output that is similar to theoutput for Catalyst 6000 series switches.

3550-1#show controller ethernet-controller fastEthernet 0/1
  !--- Output from a Catalyst 3550.

    Transmit FastEthernet0/1           Receive
          0 Bytes                        0 Bytes
          0 Unicast frames               0 Unicast frames
          0 Multicast frames             0 Multicast frames
          0 Broadcast frames             0 Broadcast frames
          0 Discarded frames             0 No dest, unicast
          0 Too old frames               0 No dest, multicast
          0 Deferred frames              0 No dest, broadcast
          0  1 collision frames
          0  2 collision frames          0 FCS errors
          0  3 collision frames          0 Oversize frames
          0  4 collision frames          0 Undersize frames
          0  5 collision frames          0 Collision fragments
          0  6 collision frames
          0  7 collision frames          0 Minimum size frames
          0  8 collision frames          0 65 to 127 byte frames
          0  9 collision frames          0 128 to 255 byte frames
          0 10 collision frames          0 256 to 511 byte frames
          0 11 collision frames          0 512 to 1023 byte frames
          0 12 collision frames          0 1024 to 1518 byte frames
          0 13 collision frames
          0 14 collision frames          0 Flooded frames
          0 15 collision frames          0 Overrun frames
          0 Excessive collisions         0 VLAN filtered frames
          0 Late collisions              0 Source routed frames
          0 Good (1 coll) frames         0 Valid oversize frames
          0 Good(>1 coll) frames         0 Pause frames
          0 Pause frames                 0 Symbol error frames
          0 VLAN discard frames          0 Invalid frames, too large
          0 Excess defer frames          0 Valid frames, too large
          0 Too large frames             0 Invalid frames, too small
          0 64 byte frames               0 Valid frames, too small
          0 127 byte frames
          0 255 byte frames
          0 511 byte frames
          0 1023 byte frames
          0 1518 byte frames

 3550-1#
 
!--- See the next table for additional counter output for 2900/3500XL Series switches.

Common System Error Messages

For the Cisco IOS system messages format, you can refer to theMessages and Recovery Procedures Guidefor the release of software you run. For example, you can look at theMessages and Recovery Proceduresfor Cisco IOS Releases.

%AMDP2_FE-3-UNDERFLO

This error message is caused when a frame is transmitted, and the local buffer of the controller chip local buffer receives insufficient data. The data cannot be transferred to the chip fast enough to keep pace with output rate. Normally, such a condition is temporary, dependent upon transient peak loads within the system. The issue occurs when an excessive amount of traffic is processed by the Fast Ethernet interface. The error message is received when the traffic level reaches about 2.5 Mb. This traffic level constrain is due to hardware limitation. Because of this, a chance exists for the device connected to the catalyst switch to drop packets.

The resolution is that ordinarily the system recovers automatically. No action is required. If the switch overwhelms the Ethernet interface, check the speed and duplex setup. Also, use a sniffer program to analyze packets that come in and out of the router fast Ethernet interface. In order to avoid packet drops on the device connected to the catalyst switch, issue theip cefcommand on the fast Ethernet interface of the device connected to the switch.

%INTR_MGR-DFC1-3-INTR: Queueing Engine (Blackwater) [1]: FIC Fabric-A Received Unexpected Control Code

The reason for this error message is the receipt of a packet from the switch fabric, where the CRC value in the fabric header on that packet did not match the CRC value calculated by the Fabric Interface Controller (FIC) subblock of the Blackwater ASIC. This indicates that a corruption of the packet occurred within transfer, and Blackwater received the corrupted packet.

Command Rejected: [Interface] not a Switching Port

In switches that support both L3 interfaces and L2 switchport, the message «Command rejected: [interface] not a switching port»displays when you try to enter a command related to layer 2 on a port that is configured as a layer 3 interface.

In order to convert the interface from layer 3 mode to layer 2 mode, issue the interface configuration commandswitchport. After you issue this command, configure the port for any layer 2 properties.

Common Port and Interface Problems

Port or Interface Status is Disable or Shutdown

An obvious but sometimes overlooked cause of port connectivity failure is an incorrect configuration on the switch. If a port has a solid orange light, this means the software inside the switch shut down the port, either by way of the user interface or by internal processes.

Note: Some port LEDs of the platform work differently in regard to STP. For example, the Catalyst 1900/2820 turns ports orange when they are in STP block mode. In this case, an orange light can indicate the normal functions of the STP. The Catalyst 6000/4000 does not turn the port light orange when it blocks for STP.

Make sure the port or module has not been disabled or powered down for some reason. If a port or module is manually shut down on one side of the link or the other, the link does not come up until you re-enable the port. Check the port status on both sides. Use theshow run interfacecommand and check to see if the interface is in ashutdownstate:

Switch#show run interface fastEthernet 4/2
!
interface FastEthernet4/2
 switchport trunk encapsulation dot1q
 switchport mode trunk
 shutdown
 duplex full
 speed 100
end

!--- Use the no shut command in config-if mode to re-enable this interface.  

If the port goes into shutdown mode immediately after a reboot of the switch, the probable cause is the port security setup. If unicast flooding is enabled on that port, it can cause the port to shut down after a reboot. Cisco recommends that you disable the unicast flooding because it also ensure that no flooding occurs on the port once the MAC address limit is reached.

Port or Interface Status is errDisable

By default, software processes inside the switch can shut down a port or interface if certain errors are detected.

When you look at show interfacecard-type {slot/port}statuscommand for Cisco IOS:

Router#show interface fastethernet 2/4 status

  Port    Name               Status       Vlan       Duplex  Speed Type
  Gi2/4                      err-disabled 1            full   1000 1000BaseSX

  !--- The show interfaces card-type {slot/port} status command for Cisco IOS
  !--- displays a status of errdisabled.
  !--- The show interfaces status errdisabled command shows all the interfaces
  !--- in this status.

Theshow loggingcommand for Cisco IOS also display the error messages (exact message format varies) that relate to the errdisable state.

Wheb ports or interlaces are shut down as a result of errdisable are referred to as causes in Cisco IOS. The causes for this range from EtherChannel misconfiguration that causes a PAgP flap, duplex mismatch, BPDU port-guard and portfast configured at the same time, UDLD that detects a one-way link, and so on.

You have to manually re-enable the port or interface to take it out the errdisable state unless you configure an errdisable recovery option. InCisco IOS software you have the ability to automatically re-enable a port after a configurable amount of time spent in the errdisable state. The bottom line is that even if you configure the interface to recover from errdisable the problem reoccurs until the root cause is determined.

Note: Use this Recover Errdisable Port State on Cisco IOS Platforms for more information on errdisable status on switches that run Cisco IOS.

This table shows an example of the commands used to configure verify and troubleshoot the errdisable status on switches. Navigate to the link for more information about the commands Recover Errdisable Port State on Cisco IOS Platforms:

Port or Interface Status is Inactive

One common cause of inactive ports on switches that run Cisco IOS is when the VLAN they belong to disappears. This can occur when interfaces are configured as layer 2 switchports that use theswitchportcommand.

Every port in a Layer 2 switch belongs to a VLAN. Every port on a Layer 3 switch configured to be a L2 switchport must also belong to a VLAN. If that VLAN is deleted, then the port or interface becomes inactive.

Note: Some switches show a steady orange (amber) light on each port when this happens.

Use theshow interfacescard-type {slot/port}switchportcommand along withshow vlanto verify.

Router#show interfaces fastEthernet 4/47 switchport
  Name: Fa4/47Switchport: Enabled
  Administrative Mode: static access
  Operational Mode: static access
  Administrative Trunking Encapsulation: negotiate
  Operational Trunking Encapsulation: native
  Negotiation of Trunking: Off
  Access Mode VLAN: 11 ((Inactive))

  !--- FastEth 4/47 is inactive.

 Router#show vlan

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 1    default                          active    Gi1/1, Gi2/1, Fa6/6
 10   UplinkToGSR's                    active    Gi1/2, Gi2/2
 
 !--- VLANs are displayed in order and VLAN 11 is not available.
 
 30   SDTsw-1ToSDTsw-2Link             active	Fa6/45

If the switch that deleted the VLAN is a VTP server for the VTP domain, every server and client switch in the domain has the VLAN removed from their VLAN table as well. When you add the VLAN back into the VLAN table from a VTP server switch, the ports of the switches in the domain that belong to that restored VLAN become active again. A port remembers what VLAN it is assigned to, even if the VLAN itself is deleted. Refer toUnderstanding and Configuring VLAN Trunk Protocol (VTP)for more information on VTP.

Note: If the output of theshow interface <interface> switchportcommand displays the port as a trunk port even after you configure the port as an access port with theswitchport access vlan <vlan>command, issue theswitchport mode accesscommand in order to make the port an access port.

Uplink Port or Interface Status is Inactive

On a Catalyst 4510R series switch, in order to enable both the 10-Gigabit Ethernet and the Gigabit Ethernet SFP uplink ports, there is an optional configuration. In order to enable the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces, issue thehw-module uplink select allcommand. After you issue the command, re-boot the switch or else the output of theshow interface status module <module number>command shows the uplink port as inactive.

Cisco IOS Software Release 12.2(25)SG supports the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces on Catalyst 4500 switches.

Note: On the Catalyst 4503, 4506, and 4507R series switches, this capability is automatically enabled.

Deferred Counter on the Catalyst Switch Interface Increments

The issue is because the traffic load destined for the switch is excessive and causes the frames to be discarded. Normally the deferred frames are the number of frames that have been transmitted successfully after waiting for the media, because the media was busy. This is usually seen in half-duplex environments where the carrier is already in use when it tries to transmit a frame. But in full duplex environments the issue occurs when the excessive load is destined for the switch.

This is the workaround:

• Hardcode both ends of the link to full duplex so that the negotiation mismatch can be avoided.

• Change the cable and patch panel cord to ensure that the cable and patch cords are not defective.

Note: If the Deferred Counter error increments on a GigabitEthernet of a Supervisor 720, turn on speed negotiation on the interface as a workaround.

Intermittent Failure to set timer [value] from vlan [vlan no]

The issue occurs when Encoded Address Recognition Logic (EARL) is unable to set the CAM aging time for the VLAN to the required number of seconds. Here, the VLAN aging time is already set to fast aging.

When the VLAN is already in fast aging, EARL cannot set the VLAN to fast aging, and aging timer set process is blocked. The default CAM aging time is five minutes, which means that the switch flushes the table of learned MAC addresses every five minutes. This ensures that the MAC address table (the CAM table) contains the most recent entries.

Fast aging temporarily sets the CAM aging time to the number of seconds that the user specifies, and is used in conjunction with the Topology Change Notification (TCN) process. The idea is that when a topology change occurs, this value is necessary to flush the CAM table faster, to compensate for the topology change.

Issue theshow cam agingcommand to check the CAM aging time on the switch. TCNs and fast aging are fairly rare. As a result, the message has a severity level of 3. If the VLANs are frequently in fast aging, check the reason for fast aging.

The most common reason for TCNs is client PCs connected directly to a switch. When you power up or down the PC, the switch port changes state, and the switch starts the TCN process. This is because the switch does not know that the connected device is a PC; the switch only knows that the port has changed the state.

In order to resolve this issue, Cisco has developed the PortFast feature for host ports. An advantage of PortFast is that this feature suppresses TCNs for a host port.

Note: PortFast also bypasses spanning-tree calculations on the port, and is therefore only suitable for use on a host port.

Trunking Mode Mismatch

Check the trunking mode on each side of the link. Make sure both sides are in the same mode (both trunking with the same method: ISL or 802.1q, or both not trunking). If you turn the trunking mode to on (as opposed to auto or desirable) for one port and the other port has the trunking mode set to off, they are not able to communicate. Trunking changes the formatting of the packet. The ports need to be in agreement as to what format they use on the link, or they do not understand each other.

For Cisco IOS, use theshow interfacescard-type {mod/port}trunkcommand to verify the trunking configuration and Native VLAN.

Router#sh interfaces fastEthernet 6/1 trunk

  Port      Mode         Encapsulation  Status        Native vlan
  Fa6/1     desirable    802.1q         trunking      1

  Port      Vlans allowed on trunk
  Fa6/1     1-4094
!--- Output truncated.

Refer to these documents for more information on the different trunking modes, guidelines, and restrictions:

• System Requirements to Implement Trunking

• Trunking Technology Support Page

Jumbos, Giants, and Baby Giants

The Maximum Transmission Unit (MTU) of the data portion of an ethernet frame is 1500 bytes by default. If the transmitted traffic MTU exceeds the supported MTU the switch does not forward the packet. Also, dependent upon the hardware and software, some switch platforms increment port and interface error counters as a result.

• Jumbo frames are not defined as part of the IEEE Ethernet standard and are vendor-dependent. They can be defined as any frame bigger than the standard ethernet frame of 1518 bytes (which includes the L2 header and Cyclic Redundancy Check (CRC)). Jumbos have larger frame sizes, typically > 9000 bytes.

• Giant frames are defined as any frame over the maximum size of an ethernet frame (larger than 1518 bytes) that has a bad FCS.

• Baby Giant frames are just slightly larger than the maximum size of an ethernet frame. Typically this means frames up to 1600 bytes in size.

Support for jumbo and baby giants on Catalyst switches varies by switch platform, sometimes even by modules within the switch. The software version is also a factor.

Refer toConfiguring Jumbo/Giant Frame Support on Catalyst Switchesfor more information on system requirements, configure and troubleshoot for jumbo and baby giant issues.

Cannot Ping End Device

Check the end device with a ping sent from the directly connected switch first, then work your way back port by port, interface by interface, trunk by trunk until you find the source of the connectivity issue. Make sure each switch can see the end device MAC address in its Content-Addressable Memory (CAM) table.

Use theshow mac address-table dynamiccommand or substitute theinterfacekeyword.

Router# show mac-address-table interface fastEthernet 6/3
Codes: * - primary entry

  vlan   mac address     type    learn qos            ports
------+----------------+--------+-----+---+--------------------------
*    2  0040.ca14.0ab1   dynamic  No    --  Fa6/3

!--- A workstation on VLAN 2 with MAC address 0040.ca14.0ab1 is directly connected
!--- to interface fastEthernet 6/3 on a switch running Cisco IOS.

Once you know the switch actually has the MAC address of the device in the CAM table, determine whether this device is on the same or different VLAN from where you try to ping.

If the end device is on a different VLAN from where you try to ping, a L3 switch or router must be configured to allow the devices to communicate. Make sure your L3 addressing on the end device and on the router/ L3 switch is correctly configured. Check the IP address, subnet mask, default gateway, dynamic routing protocol configuration, static routes, and so on.

Use of Switchport Host to Fix Startup Delays

If stations are not able to talk to their primary servers when they connect through the switch, the problem can involve delays on the switch port when it tries to become active after the physical layer link comes up. In some cases, these delays can be up to 50 seconds. Some workstations simply cannot wait this long to find their server and then they give up. These delays are caused by STP, trunking negotiations (DTP), and EtherChannel negotiations (PAgP). All of these protocols can be disabled for access ports where they are not needed, so the switch port or interface starts forwarding packets a few seconds after it establishes a link with its neighbor device.

In Cisco IOS, you can use theswitchport host command to disable channeling and to enable spanning-tree portfast and theswitchport nonegotiatecommand to turn off DTP negotiation packets. Use theinterface-range command to do this on multiple interfaces at once.

Router6k-1(config)#interface range fastEthernet 6/13 - 18
Router6k-1(config-if-range)#switchport
Router6k-1(config-if-range)#switchport host
switchport mode can be set to access
spanning-tree portfast can be enabled
channel group can be disabled
!--- Etherchannel is disabled and portfast is enabled on interfaces 6/13 - 6/18.
Router6k-1(config-if-range)#switchport nonegotiate
!--- Trunking negotiation is disabled on interfaces 6/13 - 6/18.
Router6k-1(config-if-range)#end
Router6k-1#

Cisco IOS has the option to use theglobal spanning-tree portfast defaultcommand to automatically apply portfast to any interface configured as a layer 2 access switchport. Check the Command Reference for your release of software to verify the availability of this command. You can also use thespanning-tree portfastcommand per interface, but this requires that you turn off trunking and etherchannel separately to help fix workstation startup delays.

Note: Refer toUsing Portfast and Other Commands to Fix Workstation Startup Connectivity Delaysfor more information how to fix startup delays.

Speed/Duplex, auto-negotiation, or NIC Issues

If you have a large amount of alignment errors, FCS errors, or late collisions, this can indicate one of these:

• Duplex Mismatch

• Bad or Damaged Cable

• NIC Card Issues

Duplex Mismatch

A common issue with speed/duplex is when the duplex setup are mismatched between two switches, between a switch and a router or between the switch and a workstation or server. This can occur when you manually hardcode the speed and duplex or from auto-negotiation issues between the two devices.

If the mismatch occurs between two Cisco devices with the Cisco Discovery Protocol (CDP) enabled, you see the CDP error messages on the console or in the logging buffer of both devices. CDP is useful to detect errors, as well as port and system statistics on nearby Cisco devices. CDP is Cisco proprietary and works when you send packets to a well-known MAC address 01-00-0C-CC-CC-CC.

The example shows the log messages that result from a duplex mismatch between two Catalyst 6000 series switches: one that runs CatOS, and the other that runs Cisco IOS. These messages generally tell you what the mismatch is and where it occurs.

2003 Jun 02 11:16:02 %CDP-4-DUPLEXMISMATCH:Full/half duplex mismatch detected on port 3/2
!--- CatOS switch sees duplex mismatch.

Jun  2 11:16:45 %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet6/2
(not half duplex), with TBA04251336 3/2 (half duplex).
!--- Cisco IOS switch sees duplex mismatch.

Use theshow cdp neighborscard-type <slot/port>detailcommand to display CDP information for Cisco neighbor devices.

Router#show cdp neighbors fastEthernet 6/1 detail
-------------------------
Device ID: TBA04251336
Entry address(es):
  IP address: 10.1.1.1
Platform: WS-C6006,  Capabilities: Trans-Bridge Switch IGMP
Interface: FastEthernet6/1,  Port ID (outgoing port): 3/1
Holdtime : 152 sec
Version :
WS-C6006 Software, Version McpSW: 6.3(3) NmpSW: 6.3(3)
Copyright (c) 1995-2001 by Cisco Systems
!--- Neighbor device to FastEth 6/1 is a Cisco Catalyst 6000 Switch
!--- on port 3/1 running CatOS.
advertisement version: 2
VTP Management Domain: 'test1'
Native VLAN: 1
Duplex: full
!--- Duplex is full.
Router#

setup auto speed/duplex on one side and 100/Full-duplex on the other side is also a misconfiguration and can result in a duplex mismatch. If the switch port receives a lot of late collisions, this usually indicates a duplex mismatch problem and can place the port in  an errdisable status in a result. The half-duplex side only expects packets at certain times, not at any time, and therefore counts packets received at the wrong time as collisions. There are other causes for late collisions besides duplex mismatch, but this is one of the most common reasons. Always set both sides of the connection to auto-negotiate speed/duplex or set the speed/duplex manually on both sides.

Use theshow interfaces <card-type> <slot/port>statuscommand to display speed and duplex setup as well as other information. Use thespeedandduplexcommands from interface configuration mode to hardcode both sides to 10 or 100 and half or full as necessary.

Router#show interfaces fasstEthernet 6/1 status
Port    Name               Status       Vlan       Duplex  Speed Type
Fa6/1                      connected    1          a-full  a-100 10/100BaseTX

If you use theshow interfacescommand without thestatusoption, you see a setup for speed and duplex, but you do not know whether this speed and duplex was achieved through auto-negotiation or not.

Router#sh int fas 6/1
FastEthernet6/1 is up, line protocol is up (connected)
  Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Full-duplex, 100Mb/s

!--- Full-duplex and 100Mbps does not tell you whether autoneg was used to achieve this.
!--- Use the sh interfaces fas 6/1 status command to display this.

Bad or damaged cable

Always check the cable for marginal damage or failure. A cable can be just good enough to connect at the physical layer, but it corrupts packets as a result of subtle damage to the wiring or connectors. Check or swap the copper or fiber cable. Swap the GBIC (if removable) for fiber connections. Rule out any bad patch panel connections or media convertors between source and destination. Try the cable in another port or interface if one is available and see if the problem continues.

Auto negotiation and NIC Card Issues

Problems sometimes occur between Cisco switches and certain third-party NIC cards. By default, Catalyst switch ports and interfaces are set to autonegotiate. It is common for devices like laptops or other devices to be set to autonegotiate as well, yet sometimes autonegotation issues occur.

In order to troubleshoot auto-negotiation problems it is often recommended to try and hardcode both sides. If neither auto-negotiation or hardcode setup seem to work, there can be a problem with the firmware or software on your NIC card. Upgrade the NIC card driver to the latest version available on the web site of the manufacture to resolve this.

Refer toConfiguring and Troubleshooting Ethernet 10/100/1000 MB Half/Full Duplex Auto-Negotiationfor details on how to resolve speed/duplex and auto-negotiation issues.

Refer toTroubleshooting Cisco Catalyst Switches to NIC Compatibility Issuesfor details on how to resolve third-party NIC issues.

Spanning Tree Loops

Spanning Tree Protocol (STP) loops can cause serious performance issues that masquerade as port or interface problems. In this situation, your bandwidth is used by the same frames over and over again, which leaves little room for legitimate traffic.

The STP loop guard feature provides additional protection against Layer 2 forwarding loops (STP loops). An STP loop is created when an STP block port in a redundant topology erroneously transitions to the forwarding state. This usually happens because one of the ports of a physically redundant topology (not necessarily the STP block port) no longer receives STP BPDUs. In its operation, STP relies on continuous reception or transmission of BPDUs based on the port role. The designated port transmits BPDUs, and the non-designated port receives BPDUs.

When one of the ports in a physically redundant topology no longer receives BPDUs, the STP conceives that the topology is loop free. Eventually, the block port from the alternate or backup port becomes designated and moves to a forwarding state. This situation creates a loop.

The loop guard feature makes additional checks. If BPDUs are not received on a non-designated port, and loop guard is enabled, that port is moved into the STP loop-inconsistent block state, instead of the listening / learning / forwarding state. Without the loop guard feature, the port assumes the designated port role. The port moves to the STP forwarding state and creates a loop. Refer toSpanning-Tree Protocol Enhancements using Loop Guard and BPDU Skew Detection Featuresfor more information on the loop guard feature.

This document covers reasons that STP can fail, what information to look for to identify the source of the problem, and what kind of design minimizes STP risks.

Loops can also be caused by a uni-directional link. For more information, refer to the UDLD: One-Way link problems section of this document.

UDLD: One-Way Link

A unidirectional link is a link where traffic goes out one way, but no traffic is received in the ingress direction. The switch does not know that the link ingress direction is bad (the port thinks that the link is up and works).

A broken fiber cable or other cabling/port issues can cause this one-way only communication. These partially functional links can cause problems such as STP loops when the switches involved do not know that the link is partially broken. UDLD can put a port in errdisable state when it detects a unidirectional link. The command udld aggressive-mode can be configured on switches that run Cisco IOS (check release notes for command availability) for point-to-point connections between switches where unidirectional links cannot be tolerated. The use of this feature can help you identify difficult to find unidirectional link problems

Refer toUnderstand and Configure the Unidirectional Link Detection Protocol (UDLD) Featurefor configuration information on UDLD.

Deferred Frames (Out-Lost or Out-Discard)

If you have a large number of deferred frames, or Out-Discard (also referred to as Out-Lost on some platforms), it means that the switch output buffers have filled up and the switch had to drop these packets. This can be a sign that this segment is run at an inferior speed and/or duplex, or there is too much traffic that goes through this port.

Use theshow interfaces counters errorcommand to look at OutDiscards.

Router#show interfaces counters error
Port        Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize OutDiscards
Fa7/47              0          0          0          0         0           0
Fa7/48              0          0          0          0         0     2871800
Fa8/1               0          0          0          0         0     2874203
Fa8/2             103          0          0        103         0     2878032
Fa8/3             147          0          0        185         0           0
Fa8/4             100          0          0        141         0     2876405
Fa8/5               0          0          0          0         0     2873671
Fa8/6               0          0          0          0         0           2
Fa8/7               0          0          0          0         0           0

!--- The show interfaces counters errors command shows certain interfaces
!--- that increment in large amounts OutDiscards while others run clean.

Investigate these common causes of output buffer failures:

Inferior Speed/Duplex for the Amount of Traffic

Your network can send too many packets through this port for the port to handle at its current speed/duplex setup. This can happen where you have multiple high-speed ports flowing to a single (usually slower) port. You can move the device that hangs off this port to faster media. For example, if the port is 10 Mbps, move this device to a 100 Mbps or Gigabit port. You can change the topology to route frames differently.

Congestion Issues: Segment Too Busy

If the segment is shared, other devices on this segment can transmit so much that the switch has no opportunity to transmit. Avoid daisy-chained hubs whenever possible. Congestion can lead to packet loss. Packet loss causes retransmissions at the transport layer which in turn causes users to experience latency at the application level. You can upgrade10Mbps links to 100Mbps or Gigabit Ethernet links when possible. You can remove some devices from crowded segments to other less populated segments. Make congestion avoidance a priority on your network.

Applications

At times the traffic transmission characteristics of the applications used can lead to output buffer problems. NFS file transfers that come from a Gigabit attached server that uses user datagram protocol (UDP) with a 32K window size is one example of an application setup that can bring out this type of problem. If you have checked or tried the other suggestions in this document (checked speed/duplex, no physical errors on the link, all the traffic is normal valid traffic, and so on), then reduce the unit size that is sent by the application which can help to alleviate this problem.

Software Problems

If you see behavior that can only be considered strange, you can isolate the behavior to a specific box, and you have looked at everything suggested so far, this can indicate software or hardware problems. It is usually easier to upgrade the software than it is to upgrade hardware. Change the software first.

Use theshow versioncommand to verify the current software version along with thedir flash: ordir bootflash: (dependent upon the platform) command to verify the available flash memory for the upgrade:

Router#show version
Cisco Internetwork Operating System Software
IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-IS-M), Version 12.1(13)EW, EA
RLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Fri 20-Dec-02 13:52 by eaarmas
Image text-base: 0x00000000, data-base: 0x00E638AC
ROM: 12.1(12r)EW
Dagobah Revision 71, Swamp Revision 24
trunk-4500 uptime is 2 weeks, 2 days, 6 hours, 27 minutes
System returned to ROM by redundancy reset
System image file is "bootflash:cat4000-is-mz.121-13.EW.bin"

!--- Typical Cisco IOS show version output.

Router#dir bootflash:
Directory of bootflash:/
1  -rw-     8620144   Mar 22 2002 08:26:21  cat4000-is-mz.121-13.EW.bin
61341696 bytes total (52721424 bytes free)

!--- Verify available flash memory on switch running Cisco IOS.

How to Upgrade Software

For information on how to upgrade software for your Cisco Switches, navigate to link, choose your platform and look at the Software Configuration section.

Hardware Software Incompatibility

There can be a situation where the software is not compatible with the hardware. This happens when new hardware comes out and requires special support from the software. For more information on software compatibility, use the Software Advisor tool.

Software Bugs

The operating system can have a bug. If you load a newer software version, it can often fix this. You can search known software bugs with the Software Bug Toolkit.

Corrupt Images

An image can have become corrupted. For information in regard to the recovery from corrupted images, choose your platform Switch and look at the Troubleshoot section.

Hardware Problems

Check the results ofshow modulefor Catalyst 6000 and 4000 series switches that run Cisco IOS.

Check the results of the POST results from the switch to see if there were any failures indicated for any part of the switch. Failures of any test of a module or port show an ‘F’ in the test results.

For Cisco IOS, on modular switches like the Cat6000, use the commandshow diagnostics. In order to see POST results per module, use theshow diagnostics module<module> command.

ecsj-6506-d2#sh diagnostic module 3
  Current Online Diagnostic Level = Minimal
  !--- The diagnostic level is set to minimal which is a shorter,
  !--- but also less thorough test result.
  !--- You may wish to configure diagnostic level complete to get more test results.
  Online Diagnostic Result for Module 3 : MINOR ERROR
  Online Diagnostic Level when Line Card came up = Minimal
  Test Results: (. = Pass, F = Fail, U = Unknown)
  1 . TestLoopback :
  Port  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
  ----------------------------------------------------------------------------
        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  F  F  F  F F  F
 
  !--- Notice the MINOR ERROR test result and failed loopback test which means
  !--- these ports are currently unusable.
  !--- Use the hw-module{mod}reset command or, if necessary, physically reseat the
  !--- module to try and fix this problem.

  !--- If these steps fail, open a case with Cisco Technical Support.

Note: For Catalyst 3750, 3550, 2970 , 2950/2955, and 2900/3500XL Series switches use theshow postcommand, which indicates a simple pass or fail for the hw status. Use the LEDs on these switches to help you understand the POST results.

For further information on how to troubleshoot hardware problems on Catalyst switches that run Cisco IOS, navigate to the Cisco Switches support pages, choose your platform and look at the Troubleshooting > Hardwaresection. For possible issues related to Field Notices, refer toField Noticesfor LAN and ATM Switches.

Input Errors on a Layer 3 Interface Connected to a Layer 2 Switchport

By default, all layer 2 ports are indynamic desirablemode, so the layer 2 port tries to form a trunk link and sends out DTP packets to the remote device. When a layer 3 interface is connected to a layer 2 switchport, it is not able to interpret these frames, which results in Input errors, WrongEncap errors, and Input queue drops.

In order to resolve this, change the mode of the switch port tostatic accessortrunkas per your requirement.

Switch2(config)#interface fastEthernet1/0/12
Switch2(config-if)#switchport mode access

Or

Switch2(config)#interface fastEthernet1/0/12
Switch2(config-if)#switchport trunk encapsulation dot1q
Switch2(config-if)#switchport mode trunk

Rapidly Increment Rx-No-Pkt-Buff Counter and Input Errors

The Rx-No-Pkt-Buff counter can increase on ports when it has blades, such as WS-X4448-GB-RJ45, WS-X4548-GB-RJ45, and WS-X4548-GB-RJ45V. Also, some packet drop incrementation is normal and is the result of traffic bursts traffic.

These types of errors increase rapidly, especially when the traffic that passes through that link is high or when it has devices such as servers connected to that interface. This high load of traffic oversubscribes the ports, which exhausts the input buffers and causes the Rx-No-Pkt-Buff counter and input errors to increase rapidly.

If a packet cannot be completely received because the switch is out of packet buffers, this counter is incremented once for every dropped packet. This counter indicates the internal state of the Switching ASICs on the Supervisor and does not necessarily indicate an error condition.

Pause Frames

When the receive part (Rx) of the port has its Rx FIFO queue filled and reaches the high water mark, the transmit part (Tx) of the port starts to generate pause frames with an interval value mentioned in it. The remote device is expected to stop / reduce the transmission of packets for the interval time mentioned in the pause frame.

If the Rx is able to clear the Rx queue or reach low water mark within this interval, Tx sends out a special pause frame that mentions the interval as zero (0x0). This enables the remote device to start to transmit packets.

If the Rx still works on the queue, once the interval time expires, the Tx sends a new pause frame again with a new interval value.

If Rx-No-Pkt-Buff is zero or does not increment and the TxPauseFrames counter increments, it indicates that our switch generates pause frames and the remote end obeys, hence Rx FIFO queue depletes.

If Rx-No-Pkt-Buff increments and TxPauseFrames also increments, it means that the remote end disregards the pause frames (does not support flow control) and continues to send traffic despite the pause frames. In order to overcome this situation, manually configure the speed and duplex, as well as disable the flow control, if required.

These types of errors on the interface are related to a traffic problem with the ports oversubscribed. The WS-X4448-GB-RJ45, WS-X4548-GB-RJ45, and WS-X4548-GB-RJ45V switching modules have 48 oversubscribed ports in six groups of eight ports each:

• Ports 1, 2, 3, 4, 5, 6, 7, 8

• Ports 9, 10, 11, 12, 13, 14, 15, 16

• Ports 17, 18, 19, 20, 21, 22, 23, 24

• Ports 25, 26, 27, 28, 29, 30, 31, 32

• Ports 33, 34, 35, 36, 37, 38, 39, 40

• Ports 41, 42, 43, 44, 45, 46, 47, 48

The eight ports within each group use common circuitry that effectively multiplexes the group into a single, non-block, full-duplex Gigabit Ethernet connection to the internal switch fabric. For each group of eight ports, the frames that are received are buffered and sent to the common Gigabit Ethernet link to the internal switch fabric. If the amount of data received for a port begins to exceed buffer capacity, flow control sends pause frames to the remote port to temporarily stop traffic and prevent frame loss.

If the frames received on any group exceeds the bandwidth of 1 Gbps, the device starts to drop the frames. These drops are not obvious as they are dropped at the internal ASIC rather than the actual interfaces. This can lead to slow throughput of packets across the device.

The Rx-No-Pkt-Buff does not depend on the total traffic rate. It depends on the amount of the packets that are stored in the Rx FIFO buffer of the module ASIC. The size of this buffer is only 16 KB. It is counted with short traffic bursts flow when some packets fill this buffer. Thus, Rx-No-Pkt-Buff on each port can be counted when the total traffic rate of this ASIC port group exceeds 1 Gbps, since WS-X4548-GB-RJ45 is 8:1 oversubscribed module.

When you have devices that need to carry a large amount of traffic through that interface, consider the use of one port of each group so that the common circuitry that shares a single group is not affected by this amount of traffic. When the Gigabit Ethernet switching module is not fully utilized, you can balancee the port connections across port groupings to maximize available bandwidth. For example, with the WS-X4448-GB-RJ45 10/100/1000 switching module, you can connect ports from different groups, such as ports 4, 12, 20, or 30 (in any order), before you connect ports from the same group, such as ports 1, 2, 3, 4, 5, 6, 7, and 8. If this does not solve the issue, you need to consider a module without any oversubscription of ports.

Understand Unknown Protocol Drops

Unknown protocol dropsis a counter on the interface. It is caused by protocols that are not understood by the router/switch. This example of theshow run interfacecommand shows the unknown protocol drops on the GigabitEthernet 0/1 interface.

Switch#show run interface GigabitEthernet0/1
GigabitEthernet0/1 is up, line protocol is up
  Hardware is BCM1125 Internal MAC, address is 0000.0000.0000 (via 0000.0000)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:05, output 00:00:03, output hang never
  Last clearing of "show interface" counters 16:47:42
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     3031 packets input, 488320 bytes, 0 no buffer
     Received 3023 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 63107 multicast, 0 pause input
     0 input packets with dribble condition detected
     7062 packets output, 756368 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     2015 unknown protocol drops
     4762 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Unknown protocol drops are normally dropped because the interface where these packets are received is not configured for this type of protocol, or it can be any protocol that the router does not recognize. For example, if you have two routers connected and you disable CDP on one router interface, this results in unknown protocol drops on that interface. The CDP packets are no longer recognized, and they are dropped.

Trunking between a Switch and a Router

Trunk links between a switch and a router can make the switchport go down. Trunk can come up after you disable and enable the switchport, but eventually the switchport can go down again.

In order to resolve this issue, complete these steps:

1. Make sure Cisco Discovery Protocol (CDP) runs between the switch and router and both can see each other.

2. Disable theKeepaliveson the interface of the router.

3. Reconfigure the trunk encapsulation on both devices.

When the keepalives are disabled, the CDP enables link to operate normally.

Connectivity Issues due to Oversubscription

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Refer toInterface/Module Connectivity Problemsfor more information on oversubscription.

Sub Interfaces in SPA Modules

In SPA modules, after you create a sub interface with 802.1Q, the same VLAN is not usable on the switch. Once you have encapsulation dot1q on a subinterface, you can no longer use that VLAN in the system because the 6500 or 7600 internally allocates the VLAN and makes that sub interface its only member. In order to resolve this issue, create trunk ports instead of sub interfaces. That way, the VLAN can be seen in all interfaces.

Troubleshoot Output Drops

Typically, the output drops can occur if QoS is configured and does not provide enough bandwidth to certain class of packets. It also occurs when the hardware hits an oversubscription.

For example, here you see a high amount of output drops on the interface GigabitEthernet 8/9 on a Catalyst 6500 Series Switch:

Switch#show interface GigabitEthernet8/9
GigabitEthernet8/9 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0013.8051.5950 (bia 0013.8051.5950)
  Description: Connection To Bedok_Core_R1 Ge0/1
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 18/255, rxload 23/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:28, output 00:00:10, output hang never
  Last clearing of "show interface" counters never
Input queue: 0/2000/3/0 (size/max/drops/flushes); Total output drops: 95523364
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 94024000 bits/sec, 25386 packets/sec
  5 minute output rate 71532000 bits/sec, 24672 packets/sec
     781388046974 packets input, 406568909591669 bytes, 0 no buffer
     Received 274483017 broadcasts (257355557 multicasts)
     0 runts, 0 giants, 0 throttles
     3 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     749074165531 packets output, 324748855514195 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

In order to analyze the problem, collect the output of these commands:

• show fabric utilization detail

• show fabric errors

• show platform hardware capacity

• show catalyst6000 traffic-meter

• show platform hardware capacity rewrite-engine drop

Last Input Never from the Output of Show interface Command

This example of the show interface command shows theLast input neveron the TenGigabitEthernet1/15 interface.

Switch#show interface TenGigabitEthernet1/15
TenGigabitEthernet1/15 is up, line protocol is up (connected)
  Hardware is C6k 10000Mb 802.3, address is 0025.84f0.ab16 (bia 0025.84f0.ab16)
  Description: lsnbuprod1 solaris
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:17, output hang never
  Last clearing of "show interface" counters 2d22h
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 46000 bits/sec, 32 packets/sec
     52499121 packets input, 3402971275 bytes, 0 no buffer
     Received 919 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     118762062 packets output, 172364893339 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

This shows the number of hours, minutes, and seconds since the last packet was successfully received by an interface and processed locally on the router. This is useful to know when a dead interface has failed. This counter is updated only when packets are process switched, not when packets are fast switched. Last input nevermeans there was no successful interface packet transfer to other end point or terminal. Usually this means there was no packet transfer relative to that entity.

Related Information

• Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues
• Using PortFast and Other Commands to Fix Workstation Startup Connectivity Delays
• Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation
• Upgrade Software Images and Working with Configuration Files on Catalyst Switches
• Technical Support & Documentation — Cisco Systems

Introduction

This document describes how to determine why a port or interface experiences problems.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document applies to Catalyst switches that run on Cisco IOS® System Software.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Conventions

Refer toCisco Technical Tips Conventionsfor more information on document conventions.

Note: To access tools and websites, you must be a registered Cisco client.

Troubleshoot the Physical Layer 

Use the LEDs to Troubleshoot

If you have physical access to the switch, it can save time to look at the port LEDs which give you the link status or can indicate an error condition (if red or orange). The table describes the LED status indicators for Ethernet modules or fixed-configuration switches:

Ensure that both sides have a link. A single broken wire or one shutdown port can cause the problem where one side has a link light, but the other side does not.

A link light does not guarantee that the cable is fully functional. The cable can have encountered physical stress that causes it to be functional at a marginal level. Normally you can identify this situation if the port has many packet errors, or the port constantly flaps (loses and regains link).

Check the Cable and Both Sides of the Connection

If the link light for the port does not come on, you can consider these possibilities:

Ethernet Copper and Fiber Cables

Ensure that you have the correct cable for the type of connection you want to make. Category 3 copper cable can be used for 10 Mbps unshielded twisted pair (UTP) connections but must never be used for 10/100 or 10/100/1000Mbps UTP connections. Always use either Category 5, Category 5e, or Category 6 UTP for 10/100 or 10/100/1000Mbps connections.

Warning: Category 5e and Category 6 cables can store high levels of static electricity because of the dielectric properties of the materials used in their construction. Always ground the cables (especially in new cable runs) to a suitable and safe earth ground before you connect them to the module.

For fiber, make sure you have the correct cable for the distances involved and the type of fiber ports that are used. The two options are single mode fiber (SMF) or multimode fiber (MMF). Make sure the ports on the devices that are connected together are both SMF, or both are MMF ports.

Note: For fiber connections, make sure the transmit lead of one port is connected to the receive lead of the other port. Connections for transmit-to-transmit and receive-to-receive do not work.

Ethernet and Fast Ethernet Maximum Transmission Distances

For more details on the different types of cables/connectors, cable requirements, optical requirements (distance, type, patch cables, and so on.), how to connect the different cables, and which cables are used by most Cisco switches and modules, refer to Catalyst Switch Cable Guide .

Troubleshoot the Gigabit Ethernet

If you have device A connected to device B over a Gigabit link, and the link does not come up, perform this procedure.

Step-by-Step Procedure

1. Verify device A and B use the same GBIC, short wavelength (SX), long wavelength (LX), long haul (LH), extended wavelength (ZX), or copper UTP (TX). Both devices must use the same type of GBIC to establish link. An SX GBIC needs to connect with an SX GBIC. An SX GBIC does not link with an LX GBIC. Refer to Mode-Conditioning Patch Cord Installation Note for more information.

2. Verify distance and cable used per GBIC as defined in this table.

   1000BASE-T and 1000BASE-X Port Cabling Specifications

1. The numbers given for multimode fiber-optic cable refer to the core diameter. For single-mode fiber-optic cable, 8.3 microns refers to the core diameter. The 9-micron and 10-micron values refer to the mode-field diameter (MFD), which is the diameter of the portion of the fiber that is light-carrying. This area consists of the fiber core plus a small portion that covers the cladding. The MFD is a function of the core diameter, the wavelength of the laser, and the refractive index difference between the core and the cladding.

2. Distances are based on fiber loss. Multiple splices and substandard fiber-optic cable reduce the cable distances.

3. Use with MMF only.

4. When you use an LX/LH GBIC with 62.5-micron diameter MMF, you must install a mode-conditioning patch cord (CAB-GELX-625 or equivalent) between the GBIC and the MMF cable on both the transmit and receive ends of the link. The mode-conditioning patch cord is required for link distances less than 328 feet (100 m) or greater than 984 feet (300 m). The mode-conditioning patch cord prevents the over use of the receiver for short lengths of MMF and reduces differential mode delay for long lengths of MMF. Refer to Mode-Conditioning Patch Cord Installation Note for more information.

5. Use with SMF only.

6. Dispersion-shifted single-mode fiber-optic cable.

7. The minimum link distance for ZX GBICs is 6.2 miles (10 km) with an 8-dB attenuator installed at each end of the link. Without attenuators, the minimum link distance is 24.9 miles (40 km).

3. If either device has multiple Gigabit ports, connect the ports to each other. This tests each device and verifies that the Gigabit interface functions correctly. For example, you have a switch that has two Gigabit ports. Wire Gigabit port one to Gigabit port two. Does the link come up? If so, the port is good. STP blocks on the port and prevents any loops (port one receive (RX) goes to port two transmit (TX), and port one TX goes to port two RX).

4. If single connection or Step 3 fails with SC connectors, loop the port back to itself (port one RX goes to port one TX). Does the port come up? If not, contact the TAC, as this can be a faulty port.

5. If steps 3 and 4 are successful, but a connection between device A and B cannot be established, loop ports with the cable that adjoins the two devices. Verify that there is not a faulty cable.

6. Verify that each device supports 802.3z specification for Gigabit auto-negotiation. Gigabit Ethernet has an auto-negotiation procedure that is more extensive than the one used for 10/100 Ethernet (Gigabit auto-negotiation spec: IEEE Std 802.3z-1998). When you enable link negotiation, the system auto-negotiates flow control, duplex mode, and remote fault information. You must either enable or disable link negotiation on both ends of the link. Both ends of the link must be set to the same value or the link cannot connect. Problems have been seen when you connect to devices manufactured before the IEEE 802.3z standard was ratified. If either device does not support Gigabit auto-negotiation, disable the Gigabit auto-negotiation, and it forces the link up. It takes 300msec for the card firmware to notify the software that a 10/100/1000BASE-TX link/port is down. The 300msec default debounce timer comes from the firmware polling timer to the linecards, which occurs every 300 msec. If this link is run in 1G (1000BASE-TX) mode, Gigabit sync, which occurs every 10msec, must be able to detect the link down faster. There is a difference in the link failure detection times when you run GigabitEthenet on copper versus GigabitEthernet over fiber. This difference in detection time is based on the IEEE standards.

Warning: Disable auto-negotiation and this hides link drops or physical layer problems. This is only required if end-devices such as older Gigabit NICs are used which cannot support IEEE 802.3z. Do not disable auto-negotiation between switches unless absolutely required to do so, as physical layer problems can go undetected, which results in STP loops. The alternative is to contact the vendor for software/hardware upgrade for IEEE 802.3z Gigabit auto-negotiation support.

For GigabitEthernet system requirements as well as Gigabit Interface Converters (GBICs), Coarse Wavelength Division Multiplexing (CWDM), and Small Form-Factor Pluggable (SFP) system requirements, refer to these:

• System Requirements to Implement Gigabit Ethernet on Catalyst Switches

• Catalyst GigaStack Gigabit Interface Converter Switch Compatibility Matrix

• Cisco Gigabit Ethernet Transceiver Modules Compatibility Matrix

• Cisco 10-Gigabit Ethernet Transceiver Modules Compatibility Matrix

For general configuration information and additional information on how to troubleshoot, refer to Configuring and Troubleshooting Ethernet 10/100/1000 MB Half/Full Duplex Auto-Negotiation .

Connected vs Notconnected

Most Cisco switches have a port in the notconnect state. This means it is currently not connected to anything, but it can connect if it has a good connection to another operational device. If you connect a good cable to two switch ports in the notconnect state, the link light must become green for both ports, and the port status must indicate connected. This means that the port is up as far as Layer 1 (L1) is concerned.

For Cisco IOS, you can use the show interfaces command to verify whether the interface is up, line protocol is up (connected) . The first up refers to the physical layer status of the interface. The line protocol up message shows the data link layer status of the interface and says that the interface can send and receive keepalives.

Router#show interfaces fastEthernet 6/1
FastEthernet6/1 is down, line protocol is down (notconnect)

!--- The interface is down and line protocol is down.
!--- Reasons: In this case,
!--- 1) A cable is not properly connected or not connected at all to this port.
!--- 2) The connected cable is faulty.
!--- 3) Other end of the cable is not connected to an active port or device.

!--- Note: For gigabit connections, GBICs need to be matched on each
!--- side of the connection.
!--- There are different types of GBICs, depends on the cable and
!--- distances involved: short wavelength (SX),
!--- long-wavelength/long-haul (LX/LH) and extended distance (ZX).
!--- An SX GBIC needs to connect with an SX GBIC;
!--- an SX GBIC does not link with an LX GBIC. Also, some gigabit
!--- connections require conditioning cables,
!--- that depend on the lengths involved.

Router#show interfaces fastEthernet 6/1
FastEthernet6/1 is up, line protocol is down (notconnect)

!--- The interface is up (or not in a shutdown state), but line protocol down.
!--- Reason: In this case, the device on the other side of the wire is a
!--- CatOS switch with its port disabled.

Router#show interfaces fastEthernet 6/1 status
Port   Name    Status       Vlan    Duplex   Speed  Type
Fa6/1          notconnect    1       auto     auto   10/100BaseTX

Ifshow interfacesshows up/ line protocol up (connected) but you see errors increment in the output of either command, refer to the Common Port and Interface Problems section of this document for advice.

Troubleshoot the Most Common Port and Interface Commands for Cisco IOS

This table shows the most common commands used to troubleshoot the port or interface problems on switches that run Cisco IOS System Software on the Supervisor.

Note: The right hand column on the next table gives a brief description of what the command does and lists any exceptions to the use per platform.

If you have the output of the supported commands from your Cisco device, you can use Cisco CLI Analyzer to display potential issues and fixes.

Understand the Specific Port and Interface Counter Output for Cisco IOS

Most switches have some way to track the packets and errors that occur on a port or interface. The common commands used to find this type of information are described in the Most Common Port and Interface Troubleshooting Commands for Cisco IOS section of this document.

Note: There can be differences in the implementation of the counters across various platforms and releases. Although the values of the counters are largely accurate, they are not very precise by design. In order to pull the exact statistics of the traffic, it is suggested that you use a sniffer to monitor the necessary ingress and egress interfaces.

Excessive errors for certain counters usually indicate a problem. When you operate at half-duplex setup, some data link errors increment in Frame Check Sequence (FCS), alignment, runts, and collision counters are normal. Generally, a one percent ratio of errors to total traffic is acceptable for half-duplex connections. If the ratio of errors to input packets is greater than two or three percent, performance degradation can be noticed.

In half-duplex environments, it is possible for both the switch and the connected device to sense the wire and transmit at exactly the same time and result in a collision. Collisions can cause runts, FCS, and alignment errors due to the frame not completely copied to the wire, which results in fragmented frames.

When you operate at full-duplex, errors in FCS, Cyclic Redundancy Checks (CRC), alignment, and runt counters must be minimal. If the link operates at full-duplex, the collision counter is not active. If the FCS, CRC, alignment, or runt counters increment, check for a duplex mismatch. Duplex mismatch is a situation where the switch operates at full-duplex and the connected device operates at half-duplex, or vice versa. The results of a duplex mismatch are extremely slow performance, intermittent connectivity, and loss of connection. Other possible causes of data link errors at full-duplex are bad cables, faulty switch ports, or NIC software/hardware issues. See the Common Port and Interface Problems section of this document for more information.

Show Interfaces for Cisco IOS

The show interfaces card-type {slot/port}command is the used command for Cisco IOS on the Supervisor to display error counters and statistics. An alternative to this command (for Catalyst 6000, 4000, 3550, 2970 2950/2955, and 3750 series switches) is theshow interfacescard-type <slot/port>counters errors command which only displays the interface error counters. Refer to Table 1 for explanations of the error counter output.

Note: For 2900/3500XL Series switches use theshow interfacescard-type {slot/port}command with theshow controllers Ethernet-controllercommand.

Router#sh interfaces fastEthernet 6/1
FastEthernet6/1 is up, line protocol is up (connected)
   Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
   MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
      reliability 255/255, txload 1/255, rxload 1/255
   Encapsulation ARPA, loopback not set
   Full-duplex, 100Mb/s
   input flow-control is off, output flow-control is off
   ARP type: ARPA, ARP Timeout 04:00:00
   Last input 00:00:14, output 00:00:36, output hang never
   Last clearing of "show interface" counters never
   Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
   Queueing strategy: fifo
   Output queue :0/40 (size/max)
   5 minute input rate 0 bits/sec, 0 packets/sec
   5 minute output rate 0 bits/sec, 0 packets/sec

Theshow interfacescommand output up to this point is explained here (in order) :

• up, line protocol is up (connected) — The first up refers to the physical layer status of the interface. The line protocol up message shows the data link layer status of the interface and says that the interface can send and receive keepalives.

• MTU — The Maximum Transmission Unit (MTU) is 1500 bytes for Ethernet by default (for the max data portion of the frame).

• Full-duplex, 100Mb/s — Full-duplex and 100Mbps is the current speed and duplex setup of the interface. This does not tell you whether autoneg was used to achieve this. Use theshow interfaces fastEthernet 6/1 statuscommand to display this:

Router#show interfaces fastEthernet 6/1 status
Port    Name               Status       Vlan       Duplex  Speed Type
Fa6/1                      connected    1          a-full  a-100 10/100BaseTX

!--- Autonegotiation was used to achieve full-duplex and 100Mbps.

• Last input, output — The number of hours, minutes, and seconds since the last packet was successfully received or transmitted by the interface. This is useful to know when a dead interface failed.

• Last clearing of «show interface» counters — The last time the clear counters command was issued since the last time the switch was rebooted. The clear counters command is used to reset interface statistics.

Note: Variables that can affect routing (for example, load and reliability) are not cleared when the counters are cleared.

• Input queue — The number of packets in the input queue.Size/max/drops= the current number of frames in the queue / the max number of frames the queue can hold before it must start to drop frames / the actual number of frames dropped because the max queue size was exceeded.Flushesis used to count Selective Packet Discard (SPD) drops on the Catalyst 6000 Series that run Cisco IOS. (The flushes counter can be used but never increments on the Catalyst 4000 Series that run Cisco IOS.) SPD is a mechanism that quickly drops low priority packets when the CPU is overloaded in order to save some process capacity for high priority packets. The flushes counter in the show interface command output increments as part of selective packet discard (SPD), which implements a selective packet drop policy on the IP process queue of the router. Therefore, it applies to only process switched traffic.

  The purpose of SPD is to ensure that important control packets, such as routing updates and keepalives, are not dropped when the IP input queue is full. When the size of the IP input queue is between the minimum and maximum thresholds, normal IP packets are dropped based on a certain drop probability. These random drops are called SPD flushes.

• Total output drops — The number of packets dropped because the output queue is full. A common cause is traffic from a high bandwidth link that is switched to a lower bandwidth link or traffic from multiple inbound links that are switched to a single outbound link. For example, if a large amount of traffic flow comes in on a gigabit interface and is switched out to a 100Mbps interface, this can cause output drops to increment on the 100Mbps interface. This is because the output queue on that interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.

• Output queue — The number of packets in the output queue. Size/max means the current number of frames in the queue/the max number of frames the queue can hold before it is full and must start to drop the frames.

• 5 minute input/output rate — The average input and output rate seen by the interface in the last five minutes. Specify a shorter period of time to get an accurate read (to better detect traffic bursts for example and issue theload-interval <seconds>interface command.

    SeeTable 1for explanations of the error counter output.

!--- ...show interfaces command output continues.
     1117058 packets input, 78283238 bytes, 0 no buffer
      Received 1117035 broadcasts, 0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
      0 watchdog, 0 multicast, 0 pause input
      0 input packets with dribble condition detected
      285811 packets output, 27449284 bytes, 0 underruns
      0 output errors, 0 collisions, 2 interface resets
      0 babbles, 0 late collision, 0 deferred
      0 lost carrier, 0 no carrier
      0 output buffer failures, 0 output buffers swapped out

Note: There is a difference between the counter of show interface command output for a physical interface and a VLAN interface.The input packet counters increment in the output ofshow interfacefor a VLAN interface when that packet is Layer 3 (L3) processed by the CPU. Traffic that is Layer 2 (L2) switched never makes it to the CPU and is not counted in theshow interfacecounters for the VLAN interface. It would be counted on theshow interfaceoutput for the appropriate physical interface.

Theshow interfaces <card-type> <slot/port> counters errorscommand is used in Cisco IOS to display the output of the interface errors only. SeeTable 1for explanations of the error counter output.

Router#sh interfaces fastEthernet 6/1 counters errors

 Port        Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize OutDiscards
 Fa6/1               0          0          0          0         0           0

 Port      Single-Col Multi-Col  Late-Col Excess-Col Carri-Sen     Runts    Giants
 Fa6/1              0         0         0          0         0         0         0

Table 1. Cisco IOS error counter output forshow interfacesorshow interfaces<card-type> <x/y> counters errorsfor the Catalyst 6000 and 4000 Series.

Show Interfaces Counters for Cisco IOS

To monitor inbound and outbound traffic on the port as displayed by the next output, for unicast, multicast, and broadcast traffic. Theshow interfacescard-type {slot/port}counterscommand is used when you run Cisco IOS on the Supervisor.

Note: There is, an Out-Discard counter in the Cisco IOSshow interfaces counters errorscommand which is explained inTable 1.

Router#sh interfaces fas 6/1 counters

  Port            InOctets   InUcastPkts   InMcastPkts   InBcastPkts
  Fa6/1           47856076            23        673028           149

  Port           OutOctets  OutUcastPkts  OutMcastPkts  OutBcastPkts
  Fa6/1           22103793            17        255877          3280
  Router#
  
!--- Cisco IOS counters used to monitor inbound and outbound unicast, multicast
!--- and broadcast packets on the interface.

Show Counters Interface for Cisco IOS

Theshow counters interfacecard-type {slot/port}command was introduced in Cisco IOS software version 12.1(13)E for the Catalyst 6000 series only, it offers even more detailed statistics for ports and interfaces. This commanda display the 32-bit and 64-bit error counters per port or interface.

Show Controller Ethernet-Controller for Cisco IOS

For Catalyst 3750, 3550, 2970, 2950/2955, 2940, and 2900/3500XL switches use the command show controller ethernet-controller to display traffic counter and error counter output that is similar to theoutput for Catalyst 6000 series switches.

3550-1#show controller ethernet-controller fastEthernet 0/1
  !--- Output from a Catalyst 3550.

    Transmit FastEthernet0/1           Receive
          0 Bytes                        0 Bytes
          0 Unicast frames               0 Unicast frames
          0 Multicast frames             0 Multicast frames
          0 Broadcast frames             0 Broadcast frames
          0 Discarded frames             0 No dest, unicast
          0 Too old frames               0 No dest, multicast
          0 Deferred frames              0 No dest, broadcast
          0  1 collision frames
          0  2 collision frames          0 FCS errors
          0  3 collision frames          0 Oversize frames
          0  4 collision frames          0 Undersize frames
          0  5 collision frames          0 Collision fragments
          0  6 collision frames
          0  7 collision frames          0 Minimum size frames
          0  8 collision frames          0 65 to 127 byte frames
          0  9 collision frames          0 128 to 255 byte frames
          0 10 collision frames          0 256 to 511 byte frames
          0 11 collision frames          0 512 to 1023 byte frames
          0 12 collision frames          0 1024 to 1518 byte frames
          0 13 collision frames
          0 14 collision frames          0 Flooded frames
          0 15 collision frames          0 Overrun frames
          0 Excessive collisions         0 VLAN filtered frames
          0 Late collisions              0 Source routed frames
          0 Good (1 coll) frames         0 Valid oversize frames
          0 Good(>1 coll) frames         0 Pause frames
          0 Pause frames                 0 Symbol error frames
          0 VLAN discard frames          0 Invalid frames, too large
          0 Excess defer frames          0 Valid frames, too large
          0 Too large frames             0 Invalid frames, too small
          0 64 byte frames               0 Valid frames, too small
          0 127 byte frames
          0 255 byte frames
          0 511 byte frames
          0 1023 byte frames
          0 1518 byte frames

 3550-1#
 
!--- See the next table for additional counter output for 2900/3500XL Series switches.

Common System Error Messages

For the Cisco IOS system messages format, you can refer to theMessages and Recovery Procedures Guidefor the release of software you run. For example, you can look at theMessages and Recovery Proceduresfor Cisco IOS Releases.

%AMDP2_FE-3-UNDERFLO

This error message is caused when a frame is transmitted, and the local buffer of the controller chip local buffer receives insufficient data. The data cannot be transferred to the chip fast enough to keep pace with output rate. Normally, such a condition is temporary, dependent upon transient peak loads within the system. The issue occurs when an excessive amount of traffic is processed by the Fast Ethernet interface. The error message is received when the traffic level reaches about 2.5 Mb. This traffic level constrain is due to hardware limitation. Because of this, a chance exists for the device connected to the catalyst switch to drop packets.

The resolution is that ordinarily the system recovers automatically. No action is required. If the switch overwhelms the Ethernet interface, check the speed and duplex setup. Also, use a sniffer program to analyze packets that come in and out of the router fast Ethernet interface. In order to avoid packet drops on the device connected to the catalyst switch, issue theip cefcommand on the fast Ethernet interface of the device connected to the switch.

%INTR_MGR-DFC1-3-INTR: Queueing Engine (Blackwater) [1]: FIC Fabric-A Received Unexpected Control Code

The reason for this error message is the receipt of a packet from the switch fabric, where the CRC value in the fabric header on that packet did not match the CRC value calculated by the Fabric Interface Controller (FIC) subblock of the Blackwater ASIC. This indicates that a corruption of the packet occurred within transfer, and Blackwater received the corrupted packet.

Command Rejected: [Interface] not a Switching Port

In switches that support both L3 interfaces and L2 switchport, the message «Command rejected: [interface] not a switching port»displays when you try to enter a command related to layer 2 on a port that is configured as a layer 3 interface.

In order to convert the interface from layer 3 mode to layer 2 mode, issue the interface configuration commandswitchport. After you issue this command, configure the port for any layer 2 properties.

Common Port and Interface Problems

Port or Interface Status is Disable or Shutdown

An obvious but sometimes overlooked cause of port connectivity failure is an incorrect configuration on the switch. If a port has a solid orange light, this means the software inside the switch shut down the port, either by way of the user interface or by internal processes.

Note: Some port LEDs of the platform work differently in regard to STP. For example, the Catalyst 1900/2820 turns ports orange when they are in STP block mode. In this case, an orange light can indicate the normal functions of the STP. The Catalyst 6000/4000 does not turn the port light orange when it blocks for STP.

Make sure the port or module has not been disabled or powered down for some reason. If a port or module is manually shut down on one side of the link or the other, the link does not come up until you re-enable the port. Check the port status on both sides. Use theshow run interfacecommand and check to see if the interface is in ashutdownstate:

Switch#show run interface fastEthernet 4/2
!
interface FastEthernet4/2
 switchport trunk encapsulation dot1q
 switchport mode trunk
 shutdown
 duplex full
 speed 100
end

!--- Use the no shut command in config-if mode to re-enable this interface.  

If the port goes into shutdown mode immediately after a reboot of the switch, the probable cause is the port security setup. If unicast flooding is enabled on that port, it can cause the port to shut down after a reboot. Cisco recommends that you disable the unicast flooding because it also ensure that no flooding occurs on the port once the MAC address limit is reached.

Port or Interface Status is errDisable

By default, software processes inside the switch can shut down a port or interface if certain errors are detected.

When you look at show interfacecard-type {slot/port}statuscommand for Cisco IOS:

Router#show interface fastethernet 2/4 status

  Port    Name               Status       Vlan       Duplex  Speed Type
  Gi2/4                      err-disabled 1            full   1000 1000BaseSX

  !--- The show interfaces card-type {slot/port} status command for Cisco IOS
  !--- displays a status of errdisabled.
  !--- The show interfaces status errdisabled command shows all the interfaces
  !--- in this status.

Theshow loggingcommand for Cisco IOS also display the error messages (exact message format varies) that relate to the errdisable state.

Wheb ports or interlaces are shut down as a result of errdisable are referred to as causes in Cisco IOS. The causes for this range from EtherChannel misconfiguration that causes a PAgP flap, duplex mismatch, BPDU port-guard and portfast configured at the same time, UDLD that detects a one-way link, and so on.

You have to manually re-enable the port or interface to take it out the errdisable state unless you configure an errdisable recovery option. InCisco IOS software you have the ability to automatically re-enable a port after a configurable amount of time spent in the errdisable state. The bottom line is that even if you configure the interface to recover from errdisable the problem reoccurs until the root cause is determined.

Note: Use this Recover Errdisable Port State on Cisco IOS Platforms for more information on errdisable status on switches that run Cisco IOS.

This table shows an example of the commands used to configure verify and troubleshoot the errdisable status on switches. Navigate to the link for more information about the commands Recover Errdisable Port State on Cisco IOS Platforms:

Port or Interface Status is Inactive

One common cause of inactive ports on switches that run Cisco IOS is when the VLAN they belong to disappears. This can occur when interfaces are configured as layer 2 switchports that use theswitchportcommand.

Every port in a Layer 2 switch belongs to a VLAN. Every port on a Layer 3 switch configured to be a L2 switchport must also belong to a VLAN. If that VLAN is deleted, then the port or interface becomes inactive.

Note: Some switches show a steady orange (amber) light on each port when this happens.

Use theshow interfacescard-type {slot/port}switchportcommand along withshow vlanto verify.

Router#show interfaces fastEthernet 4/47 switchport
  Name: Fa4/47Switchport: Enabled
  Administrative Mode: static access
  Operational Mode: static access
  Administrative Trunking Encapsulation: negotiate
  Operational Trunking Encapsulation: native
  Negotiation of Trunking: Off
  Access Mode VLAN: 11 ((Inactive))

  !--- FastEth 4/47 is inactive.

 Router#show vlan

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 1    default                          active    Gi1/1, Gi2/1, Fa6/6
 10   UplinkToGSR's                    active    Gi1/2, Gi2/2
 
 !--- VLANs are displayed in order and VLAN 11 is not available.
 
 30   SDTsw-1ToSDTsw-2Link             active	Fa6/45

If the switch that deleted the VLAN is a VTP server for the VTP domain, every server and client switch in the domain has the VLAN removed from their VLAN table as well. When you add the VLAN back into the VLAN table from a VTP server switch, the ports of the switches in the domain that belong to that restored VLAN become active again. A port remembers what VLAN it is assigned to, even if the VLAN itself is deleted. Refer toUnderstanding and Configuring VLAN Trunk Protocol (VTP)for more information on VTP.

Note: If the output of theshow interface <interface> switchportcommand displays the port as a trunk port even after you configure the port as an access port with theswitchport access vlan <vlan>command, issue theswitchport mode accesscommand in order to make the port an access port.

Uplink Port or Interface Status is Inactive

On a Catalyst 4510R series switch, in order to enable both the 10-Gigabit Ethernet and the Gigabit Ethernet SFP uplink ports, there is an optional configuration. In order to enable the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces, issue thehw-module uplink select allcommand. After you issue the command, re-boot the switch or else the output of theshow interface status module <module number>command shows the uplink port as inactive.

Cisco IOS Software Release 12.2(25)SG supports the simultaneous use of 10-Gigabit Ethernet and the Gigabit Ethernet SFP interfaces on Catalyst 4500 switches.

Note: On the Catalyst 4503, 4506, and 4507R series switches, this capability is automatically enabled.

Deferred Counter on the Catalyst Switch Interface Increments

The issue is because the traffic load destined for the switch is excessive and causes the frames to be discarded. Normally the deferred frames are the number of frames that have been transmitted successfully after waiting for the media, because the media was busy. This is usually seen in half-duplex environments where the carrier is already in use when it tries to transmit a frame. But in full duplex environments the issue occurs when the excessive load is destined for the switch.

This is the workaround:

• Hardcode both ends of the link to full duplex so that the negotiation mismatch can be avoided.

• Change the cable and patch panel cord to ensure that the cable and patch cords are not defective.

Note: If the Deferred Counter error increments on a GigabitEthernet of a Supervisor 720, turn on speed negotiation on the interface as a workaround.

Intermittent Failure to set timer [value] from vlan [vlan no]

The issue occurs when Encoded Address Recognition Logic (EARL) is unable to set the CAM aging time for the VLAN to the required number of seconds. Here, the VLAN aging time is already set to fast aging.

When the VLAN is already in fast aging, EARL cannot set the VLAN to fast aging, and aging timer set process is blocked. The default CAM aging time is five minutes, which means that the switch flushes the table of learned MAC addresses every five minutes. This ensures that the MAC address table (the CAM table) contains the most recent entries.

Fast aging temporarily sets the CAM aging time to the number of seconds that the user specifies, and is used in conjunction with the Topology Change Notification (TCN) process. The idea is that when a topology change occurs, this value is necessary to flush the CAM table faster, to compensate for the topology change.

Issue theshow cam agingcommand to check the CAM aging time on the switch. TCNs and fast aging are fairly rare. As a result, the message has a severity level of 3. If the VLANs are frequently in fast aging, check the reason for fast aging.

The most common reason for TCNs is client PCs connected directly to a switch. When you power up or down the PC, the switch port changes state, and the switch starts the TCN process. This is because the switch does not know that the connected device is a PC; the switch only knows that the port has changed the state.

In order to resolve this issue, Cisco has developed the PortFast feature for host ports. An advantage of PortFast is that this feature suppresses TCNs for a host port.

Note: PortFast also bypasses spanning-tree calculations on the port, and is therefore only suitable for use on a host port.

Trunking Mode Mismatch

Check the trunking mode on each side of the link. Make sure both sides are in the same mode (both trunking with the same method: ISL or 802.1q, or both not trunking). If you turn the trunking mode to on (as opposed to auto or desirable) for one port and the other port has the trunking mode set to off, they are not able to communicate. Trunking changes the formatting of the packet. The ports need to be in agreement as to what format they use on the link, or they do not understand each other.

For Cisco IOS, use theshow interfacescard-type {mod/port}trunkcommand to verify the trunking configuration and Native VLAN.

Router#sh interfaces fastEthernet 6/1 trunk

  Port      Mode         Encapsulation  Status        Native vlan
  Fa6/1     desirable    802.1q         trunking      1

  Port      Vlans allowed on trunk
  Fa6/1     1-4094
!--- Output truncated.

Refer to these documents for more information on the different trunking modes, guidelines, and restrictions:

• System Requirements to Implement Trunking

• Trunking Technology Support Page

Jumbos, Giants, and Baby Giants

The Maximum Transmission Unit (MTU) of the data portion of an ethernet frame is 1500 bytes by default. If the transmitted traffic MTU exceeds the supported MTU the switch does not forward the packet. Also, dependent upon the hardware and software, some switch platforms increment port and interface error counters as a result.

• Jumbo frames are not defined as part of the IEEE Ethernet standard and are vendor-dependent. They can be defined as any frame bigger than the standard ethernet frame of 1518 bytes (which includes the L2 header and Cyclic Redundancy Check (CRC)). Jumbos have larger frame sizes, typically > 9000 bytes.

• Giant frames are defined as any frame over the maximum size of an ethernet frame (larger than 1518 bytes) that has a bad FCS.

• Baby Giant frames are just slightly larger than the maximum size of an ethernet frame. Typically this means frames up to 1600 bytes in size.

Support for jumbo and baby giants on Catalyst switches varies by switch platform, sometimes even by modules within the switch. The software version is also a factor.

Refer toConfiguring Jumbo/Giant Frame Support on Catalyst Switchesfor more information on system requirements, configure and troubleshoot for jumbo and baby giant issues.

Cannot Ping End Device

Check the end device with a ping sent from the directly connected switch first, then work your way back port by port, interface by interface, trunk by trunk until you find the source of the connectivity issue. Make sure each switch can see the end device MAC address in its Content-Addressable Memory (CAM) table.

Use theshow mac address-table dynamiccommand or substitute theinterfacekeyword.

Router# show mac-address-table interface fastEthernet 6/3
Codes: * - primary entry

  vlan   mac address     type    learn qos            ports
------+----------------+--------+-----+---+--------------------------
*    2  0040.ca14.0ab1   dynamic  No    --  Fa6/3

!--- A workstation on VLAN 2 with MAC address 0040.ca14.0ab1 is directly connected
!--- to interface fastEthernet 6/3 on a switch running Cisco IOS.

Once you know the switch actually has the MAC address of the device in the CAM table, determine whether this device is on the same or different VLAN from where you try to ping.

If the end device is on a different VLAN from where you try to ping, a L3 switch or router must be configured to allow the devices to communicate. Make sure your L3 addressing on the end device and on the router/ L3 switch is correctly configured. Check the IP address, subnet mask, default gateway, dynamic routing protocol configuration, static routes, and so on.

Use of Switchport Host to Fix Startup Delays

If stations are not able to talk to their primary servers when they connect through the switch, the problem can involve delays on the switch port when it tries to become active after the physical layer link comes up. In some cases, these delays can be up to 50 seconds. Some workstations simply cannot wait this long to find their server and then they give up. These delays are caused by STP, trunking negotiations (DTP), and EtherChannel negotiations (PAgP). All of these protocols can be disabled for access ports where they are not needed, so the switch port or interface starts forwarding packets a few seconds after it establishes a link with its neighbor device.

In Cisco IOS, you can use theswitchport host command to disable channeling and to enable spanning-tree portfast and theswitchport nonegotiatecommand to turn off DTP negotiation packets. Use theinterface-range command to do this on multiple interfaces at once.

Router6k-1(config)#interface range fastEthernet 6/13 - 18
Router6k-1(config-if-range)#switchport
Router6k-1(config-if-range)#switchport host
switchport mode can be set to access
spanning-tree portfast can be enabled
channel group can be disabled
!--- Etherchannel is disabled and portfast is enabled on interfaces 6/13 - 6/18.
Router6k-1(config-if-range)#switchport nonegotiate
!--- Trunking negotiation is disabled on interfaces 6/13 - 6/18.
Router6k-1(config-if-range)#end
Router6k-1#

Cisco IOS has the option to use theglobal spanning-tree portfast defaultcommand to automatically apply portfast to any interface configured as a layer 2 access switchport. Check the Command Reference for your release of software to verify the availability of this command. You can also use thespanning-tree portfastcommand per interface, but this requires that you turn off trunking and etherchannel separately to help fix workstation startup delays.

Note: Refer toUsing Portfast and Other Commands to Fix Workstation Startup Connectivity Delaysfor more information how to fix startup delays.

Speed/Duplex, auto-negotiation, or NIC Issues

If you have a large amount of alignment errors, FCS errors, or late collisions, this can indicate one of these:

• Duplex Mismatch

• Bad or Damaged Cable

• NIC Card Issues

Duplex Mismatch

A common issue with speed/duplex is when the duplex setup are mismatched between two switches, between a switch and a router or between the switch and a workstation or server. This can occur when you manually hardcode the speed and duplex or from auto-negotiation issues between the two devices.

If the mismatch occurs between two Cisco devices with the Cisco Discovery Protocol (CDP) enabled, you see the CDP error messages on the console or in the logging buffer of both devices. CDP is useful to detect errors, as well as port and system statistics on nearby Cisco devices. CDP is Cisco proprietary and works when you send packets to a well-known MAC address 01-00-0C-CC-CC-CC.

The example shows the log messages that result from a duplex mismatch between two Catalyst 6000 series switches: one that runs CatOS, and the other that runs Cisco IOS. These messages generally tell you what the mismatch is and where it occurs.

2003 Jun 02 11:16:02 %CDP-4-DUPLEXMISMATCH:Full/half duplex mismatch detected on port 3/2
!--- CatOS switch sees duplex mismatch.

Jun  2 11:16:45 %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet6/2
(not half duplex), with TBA04251336 3/2 (half duplex).
!--- Cisco IOS switch sees duplex mismatch.

Use theshow cdp neighborscard-type <slot/port>detailcommand to display CDP information for Cisco neighbor devices.

Router#show cdp neighbors fastEthernet 6/1 detail
-------------------------
Device ID: TBA04251336
Entry address(es):
  IP address: 10.1.1.1
Platform: WS-C6006,  Capabilities: Trans-Bridge Switch IGMP
Interface: FastEthernet6/1,  Port ID (outgoing port): 3/1
Holdtime : 152 sec
Version :
WS-C6006 Software, Version McpSW: 6.3(3) NmpSW: 6.3(3)
Copyright (c) 1995-2001 by Cisco Systems
!--- Neighbor device to FastEth 6/1 is a Cisco Catalyst 6000 Switch
!--- on port 3/1 running CatOS.
advertisement version: 2
VTP Management Domain: 'test1'
Native VLAN: 1
Duplex: full
!--- Duplex is full.
Router#

setup auto speed/duplex on one side and 100/Full-duplex on the other side is also a misconfiguration and can result in a duplex mismatch. If the switch port receives a lot of late collisions, this usually indicates a duplex mismatch problem and can place the port in  an errdisable status in a result. The half-duplex side only expects packets at certain times, not at any time, and therefore counts packets received at the wrong time as collisions. There are other causes for late collisions besides duplex mismatch, but this is one of the most common reasons. Always set both sides of the connection to auto-negotiate speed/duplex or set the speed/duplex manually on both sides.

Use theshow interfaces <card-type> <slot/port>statuscommand to display speed and duplex setup as well as other information. Use thespeedandduplexcommands from interface configuration mode to hardcode both sides to 10 or 100 and half or full as necessary.

Router#show interfaces fasstEthernet 6/1 status
Port    Name               Status       Vlan       Duplex  Speed Type
Fa6/1                      connected    1          a-full  a-100 10/100BaseTX

If you use theshow interfacescommand without thestatusoption, you see a setup for speed and duplex, but you do not know whether this speed and duplex was achieved through auto-negotiation or not.

Router#sh int fas 6/1
FastEthernet6/1 is up, line protocol is up (connected)
  Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Full-duplex, 100Mb/s

!--- Full-duplex and 100Mbps does not tell you whether autoneg was used to achieve this.
!--- Use the sh interfaces fas 6/1 status command to display this.

Bad or damaged cable

Always check the cable for marginal damage or failure. A cable can be just good enough to connect at the physical layer, but it corrupts packets as a result of subtle damage to the wiring or connectors. Check or swap the copper or fiber cable. Swap the GBIC (if removable) for fiber connections. Rule out any bad patch panel connections or media convertors between source and destination. Try the cable in another port or interface if one is available and see if the problem continues.

Auto negotiation and NIC Card Issues

Problems sometimes occur between Cisco switches and certain third-party NIC cards. By default, Catalyst switch ports and interfaces are set to autonegotiate. It is common for devices like laptops or other devices to be set to autonegotiate as well, yet sometimes autonegotation issues occur.

In order to troubleshoot auto-negotiation problems it is often recommended to try and hardcode both sides. If neither auto-negotiation or hardcode setup seem to work, there can be a problem with the firmware or software on your NIC card. Upgrade the NIC card driver to the latest version available on the web site of the manufacture to resolve this.

Refer toConfiguring and Troubleshooting Ethernet 10/100/1000 MB Half/Full Duplex Auto-Negotiationfor details on how to resolve speed/duplex and auto-negotiation issues.

Refer toTroubleshooting Cisco Catalyst Switches to NIC Compatibility Issuesfor details on how to resolve third-party NIC issues.

Spanning Tree Loops

Spanning Tree Protocol (STP) loops can cause serious performance issues that masquerade as port or interface problems. In this situation, your bandwidth is used by the same frames over and over again, which leaves little room for legitimate traffic.

The STP loop guard feature provides additional protection against Layer 2 forwarding loops (STP loops). An STP loop is created when an STP block port in a redundant topology erroneously transitions to the forwarding state. This usually happens because one of the ports of a physically redundant topology (not necessarily the STP block port) no longer receives STP BPDUs. In its operation, STP relies on continuous reception or transmission of BPDUs based on the port role. The designated port transmits BPDUs, and the non-designated port receives BPDUs.

When one of the ports in a physically redundant topology no longer receives BPDUs, the STP conceives that the topology is loop free. Eventually, the block port from the alternate or backup port becomes designated and moves to a forwarding state. This situation creates a loop.

The loop guard feature makes additional checks. If BPDUs are not received on a non-designated port, and loop guard is enabled, that port is moved into the STP loop-inconsistent block state, instead of the listening / learning / forwarding state. Without the loop guard feature, the port assumes the designated port role. The port moves to the STP forwarding state and creates a loop. Refer toSpanning-Tree Protocol Enhancements using Loop Guard and BPDU Skew Detection Featuresfor more information on the loop guard feature.

This document covers reasons that STP can fail, what information to look for to identify the source of the problem, and what kind of design minimizes STP risks.

Loops can also be caused by a uni-directional link. For more information, refer to the UDLD: One-Way link problems section of this document.

UDLD: One-Way Link

A unidirectional link is a link where traffic goes out one way, but no traffic is received in the ingress direction. The switch does not know that the link ingress direction is bad (the port thinks that the link is up and works).

A broken fiber cable or other cabling/port issues can cause this one-way only communication. These partially functional links can cause problems such as STP loops when the switches involved do not know that the link is partially broken. UDLD can put a port in errdisable state when it detects a unidirectional link. The command udld aggressive-mode can be configured on switches that run Cisco IOS (check release notes for command availability) for point-to-point connections between switches where unidirectional links cannot be tolerated. The use of this feature can help you identify difficult to find unidirectional link problems

Refer toUnderstand and Configure the Unidirectional Link Detection Protocol (UDLD) Featurefor configuration information on UDLD.

Deferred Frames (Out-Lost or Out-Discard)

If you have a large number of deferred frames, or Out-Discard (also referred to as Out-Lost on some platforms), it means that the switch output buffers have filled up and the switch had to drop these packets. This can be a sign that this segment is run at an inferior speed and/or duplex, or there is too much traffic that goes through this port.

Use theshow interfaces counters errorcommand to look at OutDiscards.

Router#show interfaces counters error
Port        Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize OutDiscards
Fa7/47              0          0          0          0         0           0
Fa7/48              0          0          0          0         0     2871800
Fa8/1               0          0          0          0         0     2874203
Fa8/2             103          0          0        103         0     2878032
Fa8/3             147          0          0        185         0           0
Fa8/4             100          0          0        141         0     2876405
Fa8/5               0          0          0          0         0     2873671
Fa8/6               0          0          0          0         0           2
Fa8/7               0          0          0          0         0           0

!--- The show interfaces counters errors command shows certain interfaces
!--- that increment in large amounts OutDiscards while others run clean.

Investigate these common causes of output buffer failures:

Inferior Speed/Duplex for the Amount of Traffic

Your network can send too many packets through this port for the port to handle at its current speed/duplex setup. This can happen where you have multiple high-speed ports flowing to a single (usually slower) port. You can move the device that hangs off this port to faster media. For example, if the port is 10 Mbps, move this device to a 100 Mbps or Gigabit port. You can change the topology to route frames differently.

Congestion Issues: Segment Too Busy

If the segment is shared, other devices on this segment can transmit so much that the switch has no opportunity to transmit. Avoid daisy-chained hubs whenever possible. Congestion can lead to packet loss. Packet loss causes retransmissions at the transport layer which in turn causes users to experience latency at the application level. You can upgrade10Mbps links to 100Mbps or Gigabit Ethernet links when possible. You can remove some devices from crowded segments to other less populated segments. Make congestion avoidance a priority on your network.

Applications

At times the traffic transmission characteristics of the applications used can lead to output buffer problems. NFS file transfers that come from a Gigabit attached server that uses user datagram protocol (UDP) with a 32K window size is one example of an application setup that can bring out this type of problem. If you have checked or tried the other suggestions in this document (checked speed/duplex, no physical errors on the link, all the traffic is normal valid traffic, and so on), then reduce the unit size that is sent by the application which can help to alleviate this problem.

Software Problems

If you see behavior that can only be considered strange, you can isolate the behavior to a specific box, and you have looked at everything suggested so far, this can indicate software or hardware problems. It is usually easier to upgrade the software than it is to upgrade hardware. Change the software first.

Use theshow versioncommand to verify the current software version along with thedir flash: ordir bootflash: (dependent upon the platform) command to verify the available flash memory for the upgrade:

Router#show version
Cisco Internetwork Operating System Software
IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-IS-M), Version 12.1(13)EW, EA
RLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Fri 20-Dec-02 13:52 by eaarmas
Image text-base: 0x00000000, data-base: 0x00E638AC
ROM: 12.1(12r)EW
Dagobah Revision 71, Swamp Revision 24
trunk-4500 uptime is 2 weeks, 2 days, 6 hours, 27 minutes
System returned to ROM by redundancy reset
System image file is "bootflash:cat4000-is-mz.121-13.EW.bin"

!--- Typical Cisco IOS show version output.

Router#dir bootflash:
Directory of bootflash:/
1  -rw-     8620144   Mar 22 2002 08:26:21  cat4000-is-mz.121-13.EW.bin
61341696 bytes total (52721424 bytes free)

!--- Verify available flash memory on switch running Cisco IOS.

How to Upgrade Software

For information on how to upgrade software for your Cisco Switches, navigate to link, choose your platform and look at the Software Configuration section.

Hardware Software Incompatibility

There can be a situation where the software is not compatible with the hardware. This happens when new hardware comes out and requires special support from the software. For more information on software compatibility, use the Software Advisor tool.

Software Bugs

The operating system can have a bug. If you load a newer software version, it can often fix this. You can search known software bugs with the Software Bug Toolkit.

Corrupt Images

An image can have become corrupted. For information in regard to the recovery from corrupted images, choose your platform Switch and look at the Troubleshoot section.

Hardware Problems

Check the results ofshow modulefor Catalyst 6000 and 4000 series switches that run Cisco IOS.

Check the results of the POST results from the switch to see if there were any failures indicated for any part of the switch. Failures of any test of a module or port show an ‘F’ in the test results.

For Cisco IOS, on modular switches like the Cat6000, use the commandshow diagnostics. In order to see POST results per module, use theshow diagnostics module<module> command.

ecsj-6506-d2#sh diagnostic module 3
  Current Online Diagnostic Level = Minimal
  !--- The diagnostic level is set to minimal which is a shorter,
  !--- but also less thorough test result.
  !--- You may wish to configure diagnostic level complete to get more test results.
  Online Diagnostic Result for Module 3 : MINOR ERROR
  Online Diagnostic Level when Line Card came up = Minimal
  Test Results: (. = Pass, F = Fail, U = Unknown)
  1 . TestLoopback :
  Port  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
  ----------------------------------------------------------------------------
        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  F  F  F  F F  F
 
  !--- Notice the MINOR ERROR test result and failed loopback test which means
  !--- these ports are currently unusable.
  !--- Use the hw-module{mod}reset command or, if necessary, physically reseat the
  !--- module to try and fix this problem.

  !--- If these steps fail, open a case with Cisco Technical Support.

Note: For Catalyst 3750, 3550, 2970 , 2950/2955, and 2900/3500XL Series switches use theshow postcommand, which indicates a simple pass or fail for the hw status. Use the LEDs on these switches to help you understand the POST results.

For further information on how to troubleshoot hardware problems on Catalyst switches that run Cisco IOS, navigate to the Cisco Switches support pages, choose your platform and look at the Troubleshooting > Hardwaresection. For possible issues related to Field Notices, refer toField Noticesfor LAN and ATM Switches.

Input Errors on a Layer 3 Interface Connected to a Layer 2 Switchport

By default, all layer 2 ports are indynamic desirablemode, so the layer 2 port tries to form a trunk link and sends out DTP packets to the remote device. When a layer 3 interface is connected to a layer 2 switchport, it is not able to interpret these frames, which results in Input errors, WrongEncap errors, and Input queue drops.

In order to resolve this, change the mode of the switch port tostatic accessortrunkas per your requirement.

Switch2(config)#interface fastEthernet1/0/12
Switch2(config-if)#switchport mode access

Or

Switch2(config)#interface fastEthernet1/0/12
Switch2(config-if)#switchport trunk encapsulation dot1q
Switch2(config-if)#switchport mode trunk

Rapidly Increment Rx-No-Pkt-Buff Counter and Input Errors

The Rx-No-Pkt-Buff counter can increase on ports when it has blades, such as WS-X4448-GB-RJ45, WS-X4548-GB-RJ45, and WS-X4548-GB-RJ45V. Also, some packet drop incrementation is normal and is the result of traffic bursts traffic.

These types of errors increase rapidly, especially when the traffic that passes through that link is high or when it has devices such as servers connected to that interface. This high load of traffic oversubscribes the ports, which exhausts the input buffers and causes the Rx-No-Pkt-Buff counter and input errors to increase rapidly.

If a packet cannot be completely received because the switch is out of packet buffers, this counter is incremented once for every dropped packet. This counter indicates the internal state of the Switching ASICs on the Supervisor and does not necessarily indicate an error condition.

Pause Frames

When the receive part (Rx) of the port has its Rx FIFO queue filled and reaches the high water mark, the transmit part (Tx) of the port starts to generate pause frames with an interval value mentioned in it. The remote device is expected to stop / reduce the transmission of packets for the interval time mentioned in the pause frame.

If the Rx is able to clear the Rx queue or reach low water mark within this interval, Tx sends out a special pause frame that mentions the interval as zero (0x0). This enables the remote device to start to transmit packets.

If the Rx still works on the queue, once the interval time expires, the Tx sends a new pause frame again with a new interval value.

If Rx-No-Pkt-Buff is zero or does not increment and the TxPauseFrames counter increments, it indicates that our switch generates pause frames and the remote end obeys, hence Rx FIFO queue depletes.

If Rx-No-Pkt-Buff increments and TxPauseFrames also increments, it means that the remote end disregards the pause frames (does not support flow control) and continues to send traffic despite the pause frames. In order to overcome this situation, manually configure the speed and duplex, as well as disable the flow control, if required.

These types of errors on the interface are related to a traffic problem with the ports oversubscribed. The WS-X4448-GB-RJ45, WS-X4548-GB-RJ45, and WS-X4548-GB-RJ45V switching modules have 48 oversubscribed ports in six groups of eight ports each:

• Ports 1, 2, 3, 4, 5, 6, 7, 8

• Ports 9, 10, 11, 12, 13, 14, 15, 16

• Ports 17, 18, 19, 20, 21, 22, 23, 24

• Ports 25, 26, 27, 28, 29, 30, 31, 32

• Ports 33, 34, 35, 36, 37, 38, 39, 40

• Ports 41, 42, 43, 44, 45, 46, 47, 48

The eight ports within each group use common circuitry that effectively multiplexes the group into a single, non-block, full-duplex Gigabit Ethernet connection to the internal switch fabric. For each group of eight ports, the frames that are received are buffered and sent to the common Gigabit Ethernet link to the internal switch fabric. If the amount of data received for a port begins to exceed buffer capacity, flow control sends pause frames to the remote port to temporarily stop traffic and prevent frame loss.

If the frames received on any group exceeds the bandwidth of 1 Gbps, the device starts to drop the frames. These drops are not obvious as they are dropped at the internal ASIC rather than the actual interfaces. This can lead to slow throughput of packets across the device.

The Rx-No-Pkt-Buff does not depend on the total traffic rate. It depends on the amount of the packets that are stored in the Rx FIFO buffer of the module ASIC. The size of this buffer is only 16 KB. It is counted with short traffic bursts flow when some packets fill this buffer. Thus, Rx-No-Pkt-Buff on each port can be counted when the total traffic rate of this ASIC port group exceeds 1 Gbps, since WS-X4548-GB-RJ45 is 8:1 oversubscribed module.

When you have devices that need to carry a large amount of traffic through that interface, consider the use of one port of each group so that the common circuitry that shares a single group is not affected by this amount of traffic. When the Gigabit Ethernet switching module is not fully utilized, you can balancee the port connections across port groupings to maximize available bandwidth. For example, with the WS-X4448-GB-RJ45 10/100/1000 switching module, you can connect ports from different groups, such as ports 4, 12, 20, or 30 (in any order), before you connect ports from the same group, such as ports 1, 2, 3, 4, 5, 6, 7, and 8. If this does not solve the issue, you need to consider a module without any oversubscription of ports.

Understand Unknown Protocol Drops

Unknown protocol dropsis a counter on the interface. It is caused by protocols that are not understood by the router/switch. This example of theshow run interfacecommand shows the unknown protocol drops on the GigabitEthernet 0/1 interface.

Switch#show run interface GigabitEthernet0/1
GigabitEthernet0/1 is up, line protocol is up
  Hardware is BCM1125 Internal MAC, address is 0000.0000.0000 (via 0000.0000)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:05, output 00:00:03, output hang never
  Last clearing of "show interface" counters 16:47:42
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     3031 packets input, 488320 bytes, 0 no buffer
     Received 3023 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 63107 multicast, 0 pause input
     0 input packets with dribble condition detected
     7062 packets output, 756368 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     2015 unknown protocol drops
     4762 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Unknown protocol drops are normally dropped because the interface where these packets are received is not configured for this type of protocol, or it can be any protocol that the router does not recognize. For example, if you have two routers connected and you disable CDP on one router interface, this results in unknown protocol drops on that interface. The CDP packets are no longer recognized, and they are dropped.

Trunking between a Switch and a Router

Trunk links between a switch and a router can make the switchport go down. Trunk can come up after you disable and enable the switchport, but eventually the switchport can go down again.

In order to resolve this issue, complete these steps:

1. Make sure Cisco Discovery Protocol (CDP) runs between the switch and router and both can see each other.

2. Disable theKeepaliveson the interface of the router.

3. Reconfigure the trunk encapsulation on both devices.

When the keepalives are disabled, the CDP enables link to operate normally.

Connectivity Issues due to Oversubscription

When you use either the WS-X6548-GE-TX or WS-X6148-GE-TX modules, there is a possibility that individual port utilization can lead to connectivity problems or packet loss on the surrounding interfaces. Refer toInterface/Module Connectivity Problemsfor more information on oversubscription.

Sub Interfaces in SPA Modules

In SPA modules, after you create a sub interface with 802.1Q, the same VLAN is not usable on the switch. Once you have encapsulation dot1q on a subinterface, you can no longer use that VLAN in the system because the 6500 or 7600 internally allocates the VLAN and makes that sub interface its only member. In order to resolve this issue, create trunk ports instead of sub interfaces. That way, the VLAN can be seen in all interfaces.

Troubleshoot Output Drops

Typically, the output drops can occur if QoS is configured and does not provide enough bandwidth to certain class of packets. It also occurs when the hardware hits an oversubscription.

For example, here you see a high amount of output drops on the interface GigabitEthernet 8/9 on a Catalyst 6500 Series Switch:

Switch#show interface GigabitEthernet8/9
GigabitEthernet8/9 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is 0013.8051.5950 (bia 0013.8051.5950)
  Description: Connection To Bedok_Core_R1 Ge0/1
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 18/255, rxload 23/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is SX
  input flow-control is off, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:28, output 00:00:10, output hang never
  Last clearing of "show interface" counters never
Input queue: 0/2000/3/0 (size/max/drops/flushes); Total output drops: 95523364
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 94024000 bits/sec, 25386 packets/sec
  5 minute output rate 71532000 bits/sec, 24672 packets/sec
     781388046974 packets input, 406568909591669 bytes, 0 no buffer
     Received 274483017 broadcasts (257355557 multicasts)
     0 runts, 0 giants, 0 throttles
     3 input errors, 2 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     749074165531 packets output, 324748855514195 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

In order to analyze the problem, collect the output of these commands:

• show fabric utilization detail

• show fabric errors

• show platform hardware capacity

• show catalyst6000 traffic-meter

• show platform hardware capacity rewrite-engine drop

Last Input Never from the Output of Show interface Command

This example of the show interface command shows theLast input neveron the TenGigabitEthernet1/15 interface.

Switch#show interface TenGigabitEthernet1/15
TenGigabitEthernet1/15 is up, line protocol is up (connected)
  Hardware is C6k 10000Mb 802.3, address is 0025.84f0.ab16 (bia 0025.84f0.ab16)
  Description: lsnbuprod1 solaris
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:17, output hang never
  Last clearing of "show interface" counters 2d22h
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 46000 bits/sec, 32 packets/sec
     52499121 packets input, 3402971275 bytes, 0 no buffer
     Received 919 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     118762062 packets output, 172364893339 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

This shows the number of hours, minutes, and seconds since the last packet was successfully received by an interface and processed locally on the router. This is useful to know when a dead interface has failed. This counter is updated only when packets are process switched, not when packets are fast switched. Last input nevermeans there was no successful interface packet transfer to other end point or terminal. Usually this means there was no packet transfer relative to that entity.

Related Information

• Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues
• Using PortFast and Other Commands to Fix Workstation Startup Connectivity Delays
• Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation
• Upgrade Software Images and Working with Configuration Files on Catalyst Switches
• Technical Support & Documentation — Cisco Systems

Huron>show interfaces fa0/16
FastEthernet0/16 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 0009.e897.d290 (bia 0009.e897.d290)
  MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
     reliability 255/255, txload 19/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Mb/s, media type is 100BaseTX
  input flow-control is unsupported output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 2d17h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 24000 bits/sec, 40 packets/sec
  5 minute output rate 756000 bits/sec, 64 packets/sec
     46168 packets input, 4608074 bytes, 0 no buffer
     Received 1250 broadcasts (1161 multicast)
     0 runts, 0 giants, 0 throttles
     121 input errors, 16 CRC, 105 frame, 0 overrun, 0 ignored
     0 watchdog, 1161 multicast, 0 pause input
     0 input packets with dribble condition detected
     255151 packets output, 119141892 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Huron>

Last clearing of "show interface" counters 2d17h

Huron>enable
Password:
Huron#clear counters fa0/16
Clear "show interface" counters on this interface [confirm]y
Huron#show interface fa0/16
FastEthernet0/16 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 0009.e897.d290 (bia 0009.e897.d290)
  MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Mb/s, media type is 100BaseTX
  input flow-control is unsupported output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters 00:01:27
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 multicast)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     80 packets output, 7161 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Huron#

Huron>show interfaces fa0/16 counters

Port            InOctets   InUcastPkts   InMcastPkts   InBcastPkts
Fa0/16            386867          1624           294            21

Port           OutOctets  OutUcastPkts  OutMcastPkts  OutBcastPkts
Fa0/16           2527937          2352           671            39
Huron>enable
Password:
Huron#clear counters fa0/16
Clear "show interface" counters on this interface [confirm]y
Huron#show interfaces fa0/16 counters

Port            InOctets   InUcastPkts   InMcastPkts   InBcastPkts
Fa0/16                 0             0             0             0

Port           OutOctets  OutUcastPkts  OutMcastPkts  OutBcastPkts
Fa0/16               192             0             3             0
Huron#