Since Broadcom acquired VMware, I quickly learned that Broadcom not only makes the forwarding Chip-Sets (Trident/Tomahawk/…) that run within a lot of Networking Vendor’s switches, but that it also delivers an ‘Enterprise’ version of SONIC . SONIC (Software for Open Networking In the Cloud), part of the Linux Foundation, is an open source network operating system (NOS) based on Linux offering a full-suite of network functionality, that has been production-hardened in the data centers of some of the largest cloud-service providers, leveraging the collective strength of a large ecosystem and community. Broadcom is a premier member of the SONiC project in the Linux Foundation supporting the mission of enabling mass innovation through open source.

This obviously peeked my intrest, as in my view, SONIC delivers a similar promise than VMware NSX, as you can run SONIC on your choice of white-box hardware, delivering a Vendor independent Underlay NOS for Data Centers. Our VMware Cloud Foundation (VCF) technologies, which include VMware NSX, do not require specialised hardware neither, and SONIC is an excellent choice to provide the IP Transport Layer for both VMware NSX, the Network Virtualisation technology for VCF, as well as VMware VSAN, the Virtual Storage technology for VCF . Furthermore I was interested in finding out how SONIC would support interworking with VMware NSX, in the hand-off between the Virtual, and Physical Domain, as both solutions support BGP/EVPN.

Obviously I don’t have access to a lot of Hardware Switches in the VCF lab that I use, but then I learned about GNS3, an application which allows one to emulate network topologies interconnecting network devices, and you can run multiple NOS’es , including various Vendors like Cisco IOS and Juniper JUNOS, as well as SONIC. You can run GNS3 on ESXi, here is a blog on how to do this. In this way I was able to run a couple of SONIC switches inside GNS3, and use the GNS3 ‘Cloud’ object to connect it to other things running in my lab, like NSX.

My GNS 3 topology looked like this:

There are a couple of things I had to do to make things work though!

I installed the GNS3 VM using the vSpshere Client rather than the ESXi web interface, and gave the VM ample resources, 10 vCPU and 20 GB of memory, as I wanted to run 3 SONIC virtual switches inside of it. I created a dedicated PortGroup with the VLAN Type set to trunking, and configured the VLAN trunk range to only include the necessary VLANs (the VLAN to manage the VM itself, and the VLAN I created to connect NSX to the first SONIC leaf switch, Leaf-2), and assigned the GNS3 VM vNIC to it. Booted the VM and used the console to configure IP addressing and so on. I then installed the GNS3 application on a Windows machine I am using as my jump-host in the environment, and tried to point it to using the GNS3 VM as as remote server. It didn’t work.

I then quickly realised the linux OS on the GNS3 VM doesn’t support dot1q trunking out-of-the-box. After some googling I stumbled upon this, which did the trick. In summary this is what I had to do on the GNS3 VM:

First , enter the shell (you can get to this via the Console) , and install vlans on the GNS3 VM:

sudo apt-get install vlan

Load 802.1q into the kernel:

sudo modprobe 8021q

To load it permanently so it survives reloads of the VM:

sudo su -c 'echo "8021q" >> /etc/modules'

I modified the netplan yaml to use DHCP on VLAN 50 (using ‘sudo nano <yaml-file>’), where I have a DHCP Server, and has connectivity to the jump-host. (obviously static would also have worked).

gns3@gns3vm:~$ cd /etc/netplan
gns3@gns3vm:/etc/netplan$ more 00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    eth0:
      dhcp-identifier: mac
      dhcp4: true
  version: 2
  vlans:
    eth0.50:
      id: 50
      link: eth0
      dhcp-identifier: mac
      dhcp4: true

Then applied the yaml:

gns3@gns3vm:/etc/netplan$ sudo netplan apply

I did use a Distributed Port Group rather than a Standard Port Group, so I had to set the Portgroup Security Settings ‘Promiscuous mode’, ‘MAC address Changes’, & ‘Forged Transmits’ all to ‘Accept’ to make things work. Finally my GNS3 windows app on my jumphost could talk to the GNS3 VM. I configured an additional VLAN and its associated VLAN uplink on an NSX T0 Gateway, and made sure that VLAN was also configured on the Trunking Portgroup I had previously configured for the GNS3 VM. Then within GNS3 you configure a ‘Cloud’ object and connect that (I used the main eth0 interface) to the rest of the virtual network topology inside GNS3. The idea was to initially test this out with the built-in L2 switch, connecting a trunk interface to the Cloud object, and plugging a GNS3 PC object into the L2 switch, on the same VLAN as where I had the NSX T0 Uplink. This turned out to be a waste of time! I replaced the L2 switch with a a SONIC switch (Leaf-2 in above GNS3 topology), and finally packets started to flow between the GNS3 topology (PC1 in the GNS3 topology, which is in the same VLAN as the NSX T0 Gateway VLAN Uplink).

A thing to note is that the default username/password for SONIC switches is ‘admin/YourPaSsWoRd’, and that the first thing you want to do is disable Zero Touch Provisioning by entering ‘sudo config zip disable -y’ command. I also decided to use the ‘industry-standard’ CLI by executing the command ‘sonic-cli’

Following blogs will dive into more detail, but here is a drawing of the initial test topology, more of that in another blog!

I would like to thank Kamran Naqvi, now an esteemed colleague of mine at Broadcom, who introduced me to SONIC and helped me out a lot! He was able to get me a stable, feature-rich version of the virtual SONIC appliance image to run within GNS3. Thanks Kamran!

VMware NSX and Enterprise SONIC : Better Together

Inline EVPN between VMware NSX and SONIC

Who is Yves ?