// Hardware Hacking of Cisco Catalyst for HP Blade System

CBS-3020 Gigabit Ethernet switch offers stunning performance and many enterprise features for relatively cheap price, comparable with SOHO devices. As it was designed for HP's proprietary c-class Blade System, bringing it to operation at home requires some tinkering, but not much.

It has 8x (or 24x) ports, including 4 slots for SFP modules for optical transceivers. It is fully managed, has full support for L2 switching, and limited L3 routing at wire-speed. In addition, it weights less than 2kg, and the size is 200x270x30mm – less then most of rack devices (19” or even 10”).

Under the hood

Switch fabric is based on Cisco's Sasquatch ASICs, three biggest ICs on photo.

Four BCM5466SR chips on the right of ASICs provide connectivity with internal servers. BCM5466SR is a quad port 1G PHY. Broadcom doesn't provide any documentation for own chips without signing NDA, but according to Broadcom's Product Guide output standard can be one of SerDes/SGMII/RGMII. This standards use high speed differential lanes with separated clock signal in each direction. So each internal port uses 8 pins (2 sections of GbX connector).

Two chips with heatsinks attached are PHYs for external ports. It can be the same chip as above, but I haven't tried to remove heatsinks.

Six R8A20100BG chips are located around ASICs. This ICs are Renesas 1-Mbit TCAM memories.

Main CPU is located in top left corner. In the neighborhood, there is Flash memory for IOS code, and two DRAM chips.

 PCB

 Amphenol Gbx connector to midplane

 Power section on PCB

Device is controlled from chassis using HP BladeSystem ISMIC. ISMIC stores basic informations about switch, i.e. hostname, URL of management page, rack name, bay number. Some variables are set by switch, some by chassis manager. ISMIC controls power state of device, and a few LED on front panel. This subsystem use Philips/NXP LPC2131 ARM microcontroller. It is connected as I²C slave to main CPU. Its RXD, TXD and programming enable signals are routed to unpopulated J10 connector, and by 3-state buffer to GbX connector.

Power requirements analysis

Analysis was based on HP's publicly available specifications (very interesting detailed electrical part, even PCB routing is described). Additionally, I've reviewed datasheets of chips used on PCB is power supply section. Result - it is single voltage, 12V DC input. Great! Midplane's connector design reassured this assumption, it has only 4 high-current pins, connected in pairs, 2 for VCC, 2 for GND. Another ~46x2x2 differential lanes are for data bi-directional transmission.

Powering up - hardware hacking

First attempt

Using adjustable power supply with overcurrent protection - current measured = 84mA. No smoke, no heat, but didnt't start.

Close-up on power supply section, it is quite impressive, it has 5 independent power regulators. First is active all the time, it provides 3.3V for chips near to midplane connector. One of chips is LPC2131 - ARM microcontroller. This means that the powering up can be complicated, if some special initialization is needed.

Let's analyze the PCB. The biggest power and ground planes of copper are located around some small, BGA-like chip, manufactured by International Rectifier according to logo. Unfortunately, any phrase from chip marking is not recovered by Google. There was no similar chip on IRF.com, package is not typical, pin organization was not visible. Some time for research and mystery solved, it is version of iP1202 - dual output buck power controller, version manufactured for Cisco. According to datasheet, ~ENABLE pin is fortunately near the edge, so it's time for second attempt. ~ENABLE shorted to ground.

Second attempt

After power on, current was much higher, over 1.5A, but in relation to datasheet specification - 30A output - nothing abnormal. There is 1.50V and 1.20V on iP1202 outputs, but LEDs still didn't blink. Such low voltage is probably used only for powering 3 ASICs and CPU, and maybe 6 Broadcom PHYs.

Last, but not least power modules are ready-made switching power supplies, on separate small PCBs. These modules are based on ISL6540A - Intersil's single phase Buck PWM controller. After comparing pinout from datasheet and routes on PCB, I've located ENABLE signal. This signal is driven by MOSFET transistor, with its gate routed somewhere near to LPC microcontroller.

Due to inverted levels of iP1202's and ISLs' enable signals, both are routed independently, but reach the same IC, respectively input and output of NC7S04 inverter. Next mystery solved! Inverter's input is also routed to P0.6 pin on LPC, so this is a signal, that can power on the device.

Third attempt

Console connected using RJ45-to-DB9 cable, overcurrent protection raised to 2.8A. The switch seemed to work correctly, all LEDs were on, started blinking after a while. After few seconds, boot loader output appeared on console. Success!

Boot Sector Filesystem (bs) installed, fsid: 2
Base ethernet MAC Address: 00:1a:6c:2d:3c:80
Xmodem file system is available.
The password-recovery mechanism is enabled.
Initializing Flash...
flashfs[0]: 465 files, 8 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 32514048
flashfs[0]: Bytes used: 16619520
flashfs[0]: Bytes available: 15894528
flashfs[0]: flashfs fsck took 8 seconds.
...done Initializing Flash.
done.
Loading "flash:cbs30x0-ipbasek9-mz.122-50.SE4.bin"...@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "flash:cbs30x0-ipbasek9-mz.122-50.SE4.bin" uncompressed and installed, entry point: 0x3000
executing...

..

Cisco IOS Software, CBS30X0 Software (CBS30X0-IPBASEK9-M), Version 12.2(50)SE4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Fri 26-Mar-10 10:31 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x016C0000

Initializing flashfs...

flashfs[1]: 466 files, 8 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 32514048
flashfs[1]: Bytes used: 16620032
flashfs[1]: Bytes available: 15894016
flashfs[1]: flashfs fsck took 3 seconds.
flashfs[1]: Initialization complete....done Initializing flashfs.

Checking for Bootloader upgrade.. not neededError: in system startup, communication failure with on board micro-controller
Failed to version in row 0
Error: in system startup, communication failure with on board micro-controller
Failed to read GPIO data in row 1
Error: in system startup, communication failure with on board micro-controller
Failed to read ISR reason block row 2
Error: in system startup, communication failure with on board micro-controller
Failed to read the Bay Number in row 12

POST: CPU MIC register Tests : Begin
POST: CPU MIC register Tests : End, Status Passed

POST: PortASIC Memory Tests : Begin
POST: PortASIC Memory Tests : End, Status Passed

POST: CPU MIC interface Loopback Tests : Begin
POST: CPU MIC interface Loopback Tests : End, Status Passed

POST: PortASIC RingLoopback Tests : Begin
POST: PortASIC RingLoopback Tests : End, Status Passed

POST: PortASIC CAM Subsystem Tests : Begin
POST: PortASIC CAM Subsystem Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed

ERROR, reading GPIO Pin Map row from ismic, POST status not set

Waiting for Port download...Complete

..

cisco WS-CBS3020-HPQ (PowerPC405) processor (revision C0) with 131072K bytes of memory.
Processor board ID FOC1047T0HH
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
24 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
CTRL-A Z for help | 9600 8N1 | NOR | Minicom 2.7 | VT102 | Offline | ttyUSB0

Initialisation lasts quite long, first loading firmware from flash, which takes about 20-30sec, then several ASICs POST auto-test procedures must be passed.

Press RETURN to get started!

:01:19.247: %LINK-3-UPDOWN: Interface FastEthernet0, changed state to down
*Mar  1 00:01:20.253: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthern
et0, changed state to down
Use the enabled mode 'configure' command to modify this configuration.

switch>
switch>
CTRL-A Z for help | 9600 8N1 | NOR | Minicom 2.7 | VT102 | Offline | ttyUSB0

Due to unknown state of LPC's output configuration, it seems desirable idea to keep IC in reset state by shorting ~RESET to ground, to ensure high impedance state of P0.6, hardwired now to ground. Without this hack, it could be possible to destroy internal structures of chip when original firmware tries to set output into high state programatically.

Power consumption (with internal ports disabled) is 2.050A x 12V = 24W. ASICs are very hot just in few minutes and don't have any heatsinks, so switch can't be used without fans. Adding heatsink should help a bit.

I've mounted heatsinks for all ASICs, TCAMs and PHYs. Some small fan is still needed to provide airflow when top cover is assembled. I used small 25x25mm brushless DC 5V fan.

The next part describes onboard IC fuctional analysis and custom firmware rewritten from scratch.

Comments

Tung
No. 1 @ 28-01-2015 01:29

Can you instruct or post pictures about how to fire it up? Thanks.

Leave a comment…




ESI If you can't read the letters on the image, download this .wav file to get them read to you.
  • E-Mail address will not be published.
  • Formatting:
    //italic//  __underlined__
    **bold**  ''preformatted''
  • Links:
    [[http://example.com]]
    [[http://example.com|Link Text]]
  • Quotation:
    > This is a quote. Don't forget the space in front of the text: "> "
  • Code:
    <code>This is unspecific source code</code>
    <code [lang]>This is specifc [lang] code</code>
    <code php><?php echo 'example'; ?></code>
    Available: html, css, javascript, bash, cpp, …
  • Lists:
    Indent your text by two spaces and use a * for
    each unordered list item or a - for ordered ones.
Hello World!

Tomasz Głuch
Hi! I'm Tomasz Głuch, sysadmin - IT specialist and electronics enthusiast from Kraków, PL . You'll find here articles about electronics, Linux. Welcome to my site.

QR Code: URL of current page