Problem uploading Thawte issued certificate on the Cisco WLC....Certificate not properly chained.

Recently I came upon couple of scenarios where the Cisco WLC would not accept a web-auth server cert issued by Thawte (Known CA). This is because the later version of the Cisco WLC (I believe 7.6 and above) need to have a chained certificate before you can upload it on the WLC.

If you do further debugging on the WLC you will see the following error logs, which clearly points to the problem with the issuer certificate:


*TransferTask: Feb 12 12:26:05.987: Adding cert (7728 bytes) with certificate key password.
*TransferTask: Feb 12 12:26:06.015: sshpmCheckWebauthCert: Verification return code: 0
*TransferTask: Feb 12 12:26:06.015: Verification result text: unable to get issuer certificate
*TransferTask: Feb 12 12:26:06.015: Error at 2 depth: unable to get issuer certificate
*TransferTask: Feb 12 12:26:06.027: sshpmAddWebauthCert: Error decoding certificate, Deleting it.
*TransferTask: Feb 12 12:26:06.027: RESULT_STRING: Error installing certificate.
*TransferTask: Feb 12 12:26:06.027: RESULT_CODE:12
*TransferTask: Feb 12 12:26:06.027: Memory overcommit policy restored from 1 to 0
*emWeb: Feb 12 12:26:07.041: sshpmGetIdCertIndex: called to lookup cert >bsnSslWebauthCert<
*emWeb: Feb 12 12:26:07.041: sshpmGetIdCertIndex: found match in row 4
*emWeb: Feb 12 12:26:07.041: sshpmGetCID: called to evaluate <bsnSslWebauthCert>
*emWeb: Feb 12 12:26:07.041: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<



When you open the cert it does not appear to have any problem and the cert will look perfect. You OS will also not recognize it as invalid, this is because your laptop already has the Root and the Intermediate Certificate installed and even if the cert is not correctly chained it marks it as valid unlike the WLC.

Following is the mmc snapshot of the known Trusted CA on my laptop.

Now lets look at where the problem is:
==================================

One you open the certificate in a notepad you will see the following format:

Server Cert >>> Intermediate Cert >>> Root Cert (Generally the Root Cert should validate itself i.e the Root Cert is Root CA issuing itself a cert like below, where the issuer and the issued to is the same.)

The certificate looks something like this: (For security I have not shown the entire certificate).


Bag Attributes
    localKeyID: 3B DB 85 15 63 AF CA B7 57 27 4E A3 E5 0B 84 32 1D AC 06 18
subject=/C=XX/ST=XX/L=Sydney/O=XX/OU=XX/CN=XY.com.au
issuer=/C=US/O=thawte, Inc./CN=thawte SSL CA - G2

-----BEGIN CERTIFICATE-----
MIIE/TCCA+WgAwIBAgIQF//T50TPBQL4+/7Iqh7dsTANBgkqhkiG9w0BAQsFADBB
----------------Snipping------------------------------------
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMdGhhd3RlLCBJbmMuMRswGQYDVQQDExJ0
-----END CERTIFICATE-----

Bag Attributes: <No Attributes>
subject=/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
issuer=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

-----BEGIN CERTIFICATE-----
MIIEsjCCA5qgAwIBAgIQFofWiG3iMAaFIz2/Eb9llzANBgkqhkiG9w0BAQsFADCB
---------------------Snipping-------------------------------------
sjFuz4DliAc2UXu6Ya9tjSNbNKOVvKIxf/L157fo78S1JzLp955pxyvovrsMqufq
YBLqJop4
-----END CERTIFICATE-----

Bag Attributes: <No Attributes>
subject=/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
issuer=/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/emailAddress=premium-server@thawte.com

-----BEGIN CERTIFICATE-----
MIIERTCCA66gAwIBAgIQM2VQCHmtc+IwueAdDX+skTANBgkqhkiG9w0BAQUFADCB
------------------------Snipping------------------------------------
95OBBaqStB+3msAHF/XLxrRMDtdW3HEgdDjWdMbWj2uvi42gbCkLYeA=
-----END CERTIFICATE-----






So if we go through the certificate we see the certificate being issued to XY.com.au by thawte SSL CA - G2 (Intermediate CA).

Down the chain we see the Intermediate CA cert, cert issue to thawte SSL CA - G2 by thawte Primary Root CA (Can be a Root or another Intermediate CA).

Further down the chain we see thawte Primary Root CA being issued a cert by Thawte Premium Server CA and there is no other cert following this.

So the problem here is either thawte Primary Root CA can be a Root CA or an Intermediate CA. If it is a Root CA, the last cert is chain should had been for thawte Primary Root CA issued by thawte Primary Root CA itself.

If it is an intermediate CA, there should have been another cert down the chain, issued to Thawte Premium Server CA by itself, it being the Root.


In this scenario the WLC is looking for the Root Cert which is not there is the chain and thus marks the certificate as invalid.




How to fix this:
==============

Thawte do provide the Root CA and Intermediate CA cert on its website from where you can download the missing cert easily.


https://www.thawte.com/roots/index.html


So the fix would be either make thawte Primary Root CA as the Root CA and download the cert file for the Thawte website and replace the last cert in the chain, so that we have the cert for thawte Primary Root CA issued by thawte Primary Root CA.

Or, keep the same chain and download the Root CA cert for Thawte Premium Server CA and add it at the end of the chain so that the certificate chain is complete.

Once the chain is complete please follow the Cisco document to compile the cert along with the private key and get the final cert.


Please refer to the previous posts on certs:

http://lets-start-to-learn.blogspot.in/2014/10/creating-chained-certificate-from.html



Hope this was helpful.

CIMC Secure Page not opening in Firfox while works for Chrome and IE.

While working on my UCS box I came across a problem, where the GUI/CICM page would not load on my Firefox browser while it worked fine on Chrome and IE.

Doing some search I found that this has been already reported by Cisco under the Bug # CSCun04933.


Symptom:
============
Following error is observed while trying to access CIMC web page

SSL received a malformed Server Key Exchange handshake message. (Error code: ssl_error_rx_malformed_server_key_exch)


Conditions:
=========
When accessing CIMC web page with Firefox web browser version 27.0 and above.


Workaround:
===========
Change the max TLS version in Firefox.

1) Go to about:config
2) Search for 'tls'
3) Change '3' to '2'
4) Restart firefox

The workaround has worked for my issue seen on my Firefox version 35.0.1.

Double click on the Preference Name to change the value.

Hope this was helpful.

IPERF to measure throughput

Iperf is a handy tool to measure the bandwidth and the quality of a network link. It is a commonly used network testing tool that can create Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) data streams and measure the throughput of a network that is carrying them.

Iperf allows the users to vary various parameters that can be used for testing the network, or alternatively for optimizing and tuning a network. Iperf has a client and server functionality, and can measure the throughput between the two ends, either unidirectionally or bi-directionally.

Iperf can be installed very easily on any Linux or Microsoft Windows system, where one host can be configured as a client, the other one as server.

Setup required for running the iperf test:

1. Download the iperf setup, you can download it from: https://iperf.fr/
2. Copy the setup file on the two hosts you would be using to perform the test.
3. Set one host in the server mode and the other in the client mode with the following syntax:


To set the host in server mode use the command : iperf -s

C:\IOS Images\iperf-2.0.5-2-win32>iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------

To set the client in client mode use the command : iperf -c <server ip address>

C:\IOS Images\iperf-2.0.5-2-win32>iperf -c 192.168.1.5      // Where 192.168.1.5 is server ip address.

The other parameters available in iperf are:

C:\IOS Images\iperf-2.0.5-2-win32>iperf --help

Usage: iperf [-s|-c host] [options]

       iperf [-h|--help] [-v|--version]

Client/Server:

  -f, --format    [kmKM]   format to report: Kbits, Mbits, KBytes, MBytes

  -i, --interval  #        seconds between periodic bandwidth reports

  -l, --len       #[KM]    length of buffer to read or write (default 8 KB)

  -m, --print_mss          print TCP maximum segment size (MTU - TCP/IP header)

  -o, --output    <filename> output the report or error message to this specified file

  -p, --port      #        server port to listen on/connect to

  -u, --udp                use UDP rather than TCP

  -w, --window    #[KM]    TCP window size (socket buffer size)

  -B, --bind      <host>   bind to <host>, an interface or multicast address

  -C, --compatibility      for use with older versions does not sent extra msgs

  -M, --mss       #        set TCP maximum segment size (MTU - 40 bytes)

  -N, --nodelay            set TCP no delay, disabling Nagle's Algorithm

  -V, --IPv6Version        Set the domain to IPv6

Server specific:

  -s, --server             run in server mode

  -U, --single_udp         run in single threaded UDP mode

  -D, --daemon             run the server as a daemon

Client specific:

  -b, --bandwidth #[KM]    for UDP, bandwidth to send at in bits/sec

                           (default 1 Mbit/sec, implies -u)

  -c, --client    <host>   run in client mode, connecting to <host>

  -d, --dualtest           Do a bidirectional test simultaneously

  -n, --num       #[KM]    number of bytes to transmit (instead of -t)

  -r, --tradeoff           Do a bidirectional test individually

  -t, --time      #        time in seconds to transmit for (default 10 secs)

  -F, --fileinput <name>   input the data to be transmitted from a file

  -I, --stdin              input the data to be transmitted from stdin

  -L, --listenport #       port to receive bidirectional tests back on

  -P, --parallel  #        number of parallel client threads to run

  -T, --ttl       #        time-to-live, for multicast (default 1)

  -Z, --linux-congestion <algo>  set TCP congestion control algorithm (Linux only)

Miscellaneous:

  -x, --reportexclude [CDMSV]   exclude C(connection) D(data) M(multicast) S(settings) V(server) reports

  -y, --reportstyle C      report as a Comma-Separated Values

  -h, --help               print this message and quit

  -v, --version            print version information and quit

[KM] Indicates options that support a K or M suffix for kilo- or mega-

The TCP window size option can be set by the environment variable

TCP_WINDOW_SIZE. Most other options can be set by an environment variable

IPERF_<long option name>, such as IPERF_BANDWIDTH.

Report bugs to <iperf-users@lists.sourceforge.net>

C:\IOS Images\iperf-2.0.5-2-win32>

Server side:

=========

#iperf -s

------------------------------------------------------------ 

Server listening on TCP port 5001 

TCP window size: 8.00 KByte (default) 

------------------------------------------------------------ 

[852] local 10.1.1.1 port 5001 connected with 10.6.2.5 port 33453 

[ ID]   Interval          Transfer       Bandwidth 

[852]   0.0-10.6 sec   1.26 MBytes   1.03 Mbits/sec 

Client side:

=========

#iperf -c 10.1.1.1

------------------------------------------------------------ 

Client connecting to 10.1.1.1, TCP port 5001 

TCP window size: 16384 Byte (default) 

------------------------------------------------------------ 

[ 3] local 10.6.2.5 port 33453 connected with 10.1.1.1 port 5001 

[ 3]   0.0-10.2 sec   1.26 MBytes   1.05 Mbits/sec 

Another example:

Use the syntax with some additional parameters " iperf.exe – c  <IP address of the server>   -P 10  -w 1000k " (  -P refers to the number of parallel TCP streams and –w referes to the TCP window size  )

Hope this was helpful.

Using filters on Cisco WLC

The WLC outputs makes me crazy when you have to search for a specific entry in the logs. Recently I came across the filter option available on the Cisco WLCs. Not sure which code version it has been supported from, but it is awesome..

You can now use the ‘grep’ command to get the specific match. This is especially useful when the output of any commands is lengthy and you have to scroll down to get to the information that you are looking for. Lets take some examples and the related syntax.


(WLC-Primary) >grep ?

include        Include lines that match.
exclude        Exclude lines that match.

(WLC-Primary) >grep include ?

<pattern>      Pattern to be searched.

(WLC-Primary) >grep include uptime ?

<command>      Enter complete show command in double quotes.

Lets try to find the uptime of the WLC:

(WLC-Primary) >grep include ime "show sysinfo"

Press yes to continue(y)y

System Up Time................................... 0 days 5 hrs 47 mins 57 secs

System Timezone Location......................... (GMT +5:30) Colombo, New Delhi, Chennai, Kolkata

System Stats Realtime Interval................... 5

There are 3 lines matching the pattern ime

Since this is case sensitive I searched for the value "ime".

Lets see another example, let suppose I want to see all APs except a specific AP.

(WLC-Primary) >show ap summary

Number of APs.................................... 3

Global AP User Name.............................. gce-apac

Global AP Dot1x User Name........................ Not Configured

AP Name             Slots  AP Model              Ethernet MAC       Location          Country  IP Address       Clients

------------------  -----  --------------------  -----------------  ----------------  -------  ---------------  -------

L3500-3              2     AIR-CAP3502I-E-K9     40:55:39:ca:8a:99  default location  AE       10.105.132.249   0

TEST-AP-1            2     AIR-CAP2602E-A-K9     6c:41:6a:78:d8:32  default location  US       10.105.132.247   0

AP-3602AP-1          2     AIR-CAP3602I-A-K9     e4:d3:f1:c9:04:ca  default location  US       10.105.132.251   0

We will use the option exclude this time.

(WLC-Primary) >grep exclude "L3500-3" "show ap summary"

Press yes to continue(y)y

Number of APs.................................... 3

Global AP User Name.............................. gce-apac

Global AP Dot1x User Name........................ Not Configured

AP Name             Slots  AP Model              Ethernet MAC       Location          Country  IP Address       Clients

------------------  -----  --------------------  -----------------  ----------------  -------  ---------------  -------

TEST-AP-1            2     AIR-CAP2602E-A-K9     6c:41:6a:78:d8:32  default location  US       10.105.132.247   0

AP-3602AP-1          2     AIR-CAP3602I-A-K9     e4:d3:f1:c9:04:ca  default location  US       10.105.132.251   0

There are 12 lines not matching the pattern L3500-3

Hope this was informational.