Release schedule and upcoming support for Accton/Edgecore AS5916-54XKS
We want to give details of the upcoming release schedule of the DANOS project. The initial seed code contribution by AT&T to The Linux Foundation happened in November 2019. Our goal is to make three releases a year. In 2020, we expect to have one before the first half of the year, and then two before the end of the year.
The first release of 2020 will contain many new features that have been developed internally within AT&T since the 1908 release. In future blog posts, we will go into detail about the upcoming features, but there is one feature that we are particularly excited about and want to share now.
The next release of DANOS will run on the Accton/Edgecore Qumran-MX platform AS5916-54XKS 10G/100G Edge Router. The Accton/Edgecore AS5916-54XKS is an affordable Carrier Access/Aggregation data centre switch.
DANOS uses Broadcom’s Open Network Switch Layer (OpenNSL) libraries to support a hardware programming plugin architecture for merchant silicon-based forwarding. The support for the AS5916-54XKS device is now the second OpenNSL plugin (along with the initial UFiSpace S9500-30xs one) that will be available as part of DANOS.
Comments
Unfortunately not, for two reasons:
The AS5912-54X lacks a BMC and DANOS doesn’t currently have the functionality to monitor temperatures and control the fans accordingly. In addition, some of the SFP integration works differently on certain Accton platforms when a BMC option is present versus without and we’ve done the integration assuming the BMC is present as that is the case on the AS5916-54XKS.
The switch chip is different - BCM88370 on the AS5912-54X vs. BCM88375 on the AS5916-54XKS and the detection as to which Broadcom config to apply relies on this.
Even without those two reasons I wouldn’t be able to say confidently one way or the other without having the hardware specification for the AS5912-54X and without knowing that the serdes lanes are used in the same way, with the same parameters, etc.
Hello Scott/Robert,
We success to port DANOS 2005 to AS5912-54X hardware. We need to hack current release as follow:
Back port OpenNetworkLinux drivers and tools in order to supply the lack of BMC on this platform. Kernel drivers allow to use
/systo monitor temperature, fan, psu and sfp. For that purpose, we need the special version ofi2c_mux_pca954xprovided with the ufispace kernel module (to avoidi2c_ismttimeout).Add new python scripts to detect the platform and to provide
sfphelperfunctions.show hardware sensorcommand provides output for the various temperature sensors as well as fan speedAdd
config.bcmandrc.socfiles for the Qumran-MX (88370)
All is fine expected the libopennsl. It seems that the new version of libopennsl provided with Danos 2005 is restricted to some Broadcom chips. Indeed, replacing the libopennsl provided by Danos 2005 by the one provided by release 1909 or 1812 will do the job. Our Broadcom 88370 is correctly initialized, ports are detected and new interface created.
So, can you point us what have been change between Danos 1909 and Danos 2005 regarding the libopennsl (I suspect modification in compilation flags)? Can you revert back to a more generic libopennsl for the Danos 2009 release?
Finally, I have two questions:
How I could inspect
sfp? I don’t find a show transceivers command while I found avyatta-op-transceiversscripts in/opt/vyatta/bin? For the moment, we useonlpdcommand to show SFP status and decode eeprom.I think we miss a part of the Broadcom configuration. Indeed, if I could configure an IP address to the newly created interface which correspond to the port managed by the ASIC, a
show interface dataplane dpe0xe0provides an error message telling that the interface is not in the system. And more, dramatically, IP packets send to the Braodcom chips are not forwarded to the control plane nor to another port. Do we need a special action to activate a port like a no shutdown as theshow interfacecommand said that the configured port dp0xe0 is down.
Once last problem solve, can you tell us how to publish our modifications? Regarding Danos organisation on GitHub, this certainly need to create new repo.
Regards
Olivier
So, can you point us what have been change between Danos 1909 and Danos 2005 regarding the
libopennsl(I suspect modification in compilation flags)? Can you revert back to a more genericlibopennslfor the Danos 2009 release?
We added support for AS5916-54XKS in the DANOS 2005 release and the detection is rather basic - if it’s a BCM88370 chip then we assume it’s the AS5916-54XKS platform and use its configuration files. The configuration is embedded in libopennsl to comply with licensing conditions. I would suggest opening a DANOS feature ticket and we can work out how to proceed here, and with the other modifications.
How I could inspect
sfp? I don’t find a show transceivers command while I found avyatta-op-transceiversscripts in/opt/vyatta/bin? For the moment, we useonlpdcommand to show SFP status and decode eeprom.
If you’ve done the SFP helper integration then you can use vyatta-sfp-util --read-eeprom --port xe0 to verify that the platform integration works, substituting “xe0” for the actual port of course. Now the gotcha is that libfal-opennsl hasn’t implemented the translation between the DPDK module EEPROM APIs and the SFP helper daemon to be able to return this information through the system. If it did, then show interfaces dataplane dp0xe0 physical would give the decoded and nicely formatted version of the EEPROM data.
I think we miss a part of the Broadcom configuration. Indeed, if I could configure an IP address to the newly created interface which correspond to the port managed by the ASIC, a
show interface dataplane dpe0xe0provides an error message telling that the interface is not in the system. And more, dramatically, IP packets send to the Braodcom chips are not forwarded to the control plane nor to another port. Do we need a special action to activate a port like a no shutdown as theshow interfacecommand said that the configured port dp0xe0 is down.
If you’ve set the default platform type in your new Python platform type library to switch then it should bring the ports admin up at startup time. A platform type of router isn’t supported with libfal-opennsl as it lacks the VLAN and L3 interface management to do this currently. So if you’ve rebooted after adding the platform support then the interfaces should automatically be brought up. If they’re not then I guess a DANOS bug report would be the best place to investigate this further and try to work out what is going on.
Hello Robert,
Thanks for the useful information
We added support for AS5916-54XKS in the DANOS 2005 release and the detection is rather basic - if it’s a BCM88370 chip then we assume it’s the AS5916-54XKS platform and use its configuration files. The configuration is embedded in libopennsl to comply with licensing conditions. I would suggest opening a DANOS feature ticket and we can work out how to proceed here, and with the other modifications.
We add new files in new /usr/share/vyatta/platform/as5912-54x for that purpose. The main difference is that we need to specify the config.bcm and rc.soc file in platform.conf.in and use the libopennsl from Danos release 1812. The detection is perform by a new python script accton_as5912-54x.py that we added under /usr/lib/python3/dist-packages/vyatta/platform/type completed by a new SFP helper. In fact, we complete the libfal-opennsl1 debian package and once last bug fix, we could propose a PR on github for that purpose.
If you’ve done the SFP helper integration then you can use vyatta-sfp-util --read-eeprom --port xe0 to verify that the platform integration works, substituting “xe0” for the actual port of course. Now the gotcha is that libfal-opennsl hasn’t implemented the translation between the DPDK module EEPROM APIs and the SFP helper daemon to be able to return this information through the system. If it did, then show interfaces dataplane dp0xe0 physical would give the decoded and nicely formatted version of the EEPROM data.
vyatta-sfp-util --read-eeprom --port xe0 gives expected result i.e. raw data of eeprom with our dedicated sfp helper function. As we back port ONL to manage Fan speed regulation, we could also used the ONL tools to read eeprom (only few information). While waiting for the integration of eeprom decode in libfal-openssl we use oom tools https://github.com/opencomputeproject/oom which allow to monitor the SFP.
If you’ve set the default platform type in your new Python platform type library to switch then it should bring the ports admin up at startup time. A platform type of router isn’t supported with libfal-opennsl as it lacks the VLAN and L3 interface management to do this currently. So if you’ve rebooted after adding the platform support then the interfaces should automatically be brought up. If they’re not then I guess a DANOS bug report would be the best place to investigate this further and try to work out what is going on.
Yes. We setup platform to switch and all ports are detected (48 1/10g and 6 100g like for the AS5916 platform). Now, our problem is that we can’t received packets. A tcpdump on a neighbor router show us that the Danos platform send packets (ARP request or Ping is we configure manually arp table) and that the neighbor correctly reply to the Danos platform, but on Danos router, we saw nothing (just counter of interface that are incremented when sending L2 traffic lke IS-IS). In addition, packets sent by the Danos platform are raw IP on optical layer without an ehternet layer. It is why we have some doubt about our current configuration. And, we must set the speed for each port (SFP or QSFP) to have the port up e.g. set interface dataplane dp0xe0 speed 10g.
Of course we could fulfill a bug for that. What is it the better place ? Jira or Github ?
Regards
Olivier
Hello Oliver,
Are you still utilising DANOS in any shape or form? If so, I would like to know what the experience was on baremetal devices.
Hello Paul,
Our tests with Danos were successful. However, MPLS is not supported on baremetal devices and since Danos is no more developed, we stop using Danos and switch to SONiC now.
Regards
Olivier
Dear Scott/Robert,
We are trying to get the build environment ready for this hardware - AS5916-54XKS. We are refering to the following link for OpenNSL build - http://broadcom-switch.github.io/OpenNSL/doc/html/OPENNSL_BUILDING.html
We would like to understand if there is any documentation for DANOS build with Broadcom OpenNSL SDK.
Sunil,
In order to get support for the AS5916-54XKS platform then it is necessary to use our pre-built OpenNSL binaries, which are conveniently packages in Debian format. This can be done relatively easily in a Debian build environment by pointing apt at the repo listed on the https://danosproject.atlassian.net/wiki/spaces/DAN/pages/320634908 page and then using “apt-get build-dep <source-package>”, e.g. “apt-get build-dep libfal-opennsl”, and then using standard Debian build tools, e.g. “debuild -uc -us -b”. For more details, see the https://danosproject.atlassian.net/wiki/spaces/DAN/pages/786516 article.
Thank You for the guidance. We are able to build BCM dependencies and the modules like libfal-opennsl, opennsl-binary and bcm-linux-bde-moudule.
In bcm-linux-bde-module, it has a patch for disabling the KNET module(linux-bcm-knet.ko) build. As I understand, the KNET driver is not required for Danos Dataplane to communicate with BCM chipset. Please let me know if my understanding is correct.
You are correct - the punt path is via the software dataplane which has the advantage of being able to run features with high performance for certain use cases where the hardware doesn’t have support for those features.
CGNAT scenario using DANOS on AS5916-54XKS
We are considering use Edgecore AS5916-54XKS with DANOS to deploy CGNAT.
We had very good experiences using DANOS for CGNAT on X86/X64 with INTEL NIC and DPDK.
And we also have very good experiences with AS5916-54XKS, especially because of Qumran, using IP Infusion..
But I understand that the CGNAT scenario demands a very intensive Stateful Connection Tracking.
And I’m not sure how Qumran in AS5916-54XKS will deal with it.
So, here comes the questions:
a) Using AS5916-54XKS to deploy CGNAT would be feasible?
b) Would perform well?
c) Is possible to guess some limitations?
Hi Scott,
Great news. Do you think the AS5912-54X will be compatible also ? It the same hardware, but without the T-CAM chipset.
Olivier