SKALE Incentivized Testnet #1- Results & Community Update

The SKALE Network has an incentivized Testnet running for the last several weeks. SKALE Labs, the core developer of the SKALE Network, kicked it off on May 20th in conjunction with a number of the initial validators of the network. This initial version of the Testnet will continue to run for a few more weeks.

The goals of some of the initial phases of the Testnet are to use actual SKALE Network validators to (a) test and verify the setup of validator nodes, and (b) confirm the proper performance of validator and delegator operations. The Testnet also served as a way to familiarize many of the key initial validators with the unique capabilities of the network in advance of the launch of the SKALE Network Mainnet.

One note we want to make is that this testing was in addition to other forms of rigorous testing of the network. All the versions of the network go through intensive QA testing within a large validator pool. In addition, the smart contracts for the SKALE Network have successfully gone through two security audits – each performed by highly reputable and independent auditing firms. The purpose of this Testnet was to add more additional levels of operational testing and also to run through the validator onboarding process in a real-world situation, as well as familiarize validators with the inner workings of the network.

Testing Phases

The testing went through a number of phases. The process was to bring the validators along as a group via a series of steps and have validators as the various parties for testing multi-party transactions. For example, one validator might act as a validator while another might play the role of a delegator, not only testing validator/delegator operations but also seeing the process through a delegator’s point of view.

The specific phases of this initial version of the Testnet included:

• Validator Node Setup
• Delegation Operations
• Bounty Payouts
• SKALE Chain Creation and Node Rotation
• Testing Backup/Restore
  

We should note that the first few phases of testing was labeled Hike to Kilimanjaro. The meaning being that the initial setup and operations were in preparation for more involved testing steps and phases. Later phases will feature much more sophisticated operations as well as introduce elements of randomness and unexpected behaviors (making use of aspects of chaos engineering to ensure confidence in the network’s capability to withstand turbulent conditions).

In addition to the specific operations, we analyzed a number of data points  in great detail during various phases in order to identify performance issues,  isolate the cause of any downtime within the network, or spot any token payment anomalies. These data points included the following measurements:

• Histogram of the latency of data received by SKALE Manager
• Histogram of downtime received by the SKALE Manager
• Time Series of bounty payments made to Validators
  

Testing Participants

Twenty (20) validators have participated in this version of the Testnet. These validators ranged from experienced network operations teams to experienced crypto miners to experienced ETH developers.

It quickly became apparent during the setup phases that we were working with a great group of validators. They had a lot of experience across a wide set of domains. They had years of experience with complex infrastructure setup and orchestration as well as a solid history with Ethereum and EVMs. The feedback we received during the setup process alone helped us enhance the validator onboarding process while also having a hand in further hardening the security of the system.

All these insights and variations in experiences were quickly factored in by the network team as Testnet learning so that end-to-end setup of nodes can be as smooth as possible. (The overall goal of the SKALE Network is to be a truly decentralized network and so making node setup as turnkey as possible is a critical step in this process.)

While some of the node operators feel that validators need to earn their participation by running through setup hurdles as almost a rite of passage, the SKALE core engineering and solutions engineering teams feel differently and are working to get a foolproof setup process that takes less than 2 hours. (We should note that this threshold was by and large achieved for most of the validators in the setup phase even before the improvements.)

Testing of the SGX Wallet and ABBA-Based Consensus Layer

The node setup steps were followed by validator and node registration. All these operations were well understood by the validators as almost all had experience with other mining or validating operations. What did take a bit of time and where testing was particularly focusing on were the areas unique to the SKALE Network.

A few of these areas included the SGX Wallet and the ABBA-based consensus layer used within the SKALE EVM. The SGX Wallet aids validators and developers by helping them protect sensitive data, reducing their need to continually confirm transactions, and serving as a gateway for SKALE cryptographic operations such as BLS signatures and digital key generation. The ABBA-based consensus layer provides a secure and trustless consensus process while offering a highly performant mechanism for both high transaction throughput and low commit time latency.

Intel SGX

The SKALE Network makes use of the Intel SGX-based chipsets in order to offer enhanced security, increased data protections, and faster digital key generation. Intel SGX (Software Guard Extensions) let developers partition sensitive information into enclaves, which are areas of execution in memory with more security protection. A further benefit of utilizing Intel SGX-based servers is for use with BLS (Boneh–Lynn–Shacham) cryptography, which is featured heavily within the SKALE Network. The SKALE Network Validator FAQ contains much more detail on Intel SGX, BLS Signatures, and the roles they play in the SKALE Network.

ABBA-based Consensus

A variant of the Asynchronous Binary Byzantine Agreement (ABBA) protocol is used within the SKALE Network consensus model. The fundamental benefit of the ABBA protocol is that it is designed to exhibit robustness in the case of subnode downtime where each latent and/or down subnode is regarded as a slow link. The ABBA-based consensus layer replaces the default consensus model in the EVM (Ethereum Virtual Machine) that is included as a container within each node. It is this EVM that gets instantiated within each virtualized subnode that manages and maintains each SKALE chain. (Details on the ABBA-based protocol can be viewed here. Details on how containerization and virtualization works can be found in Containerization & The Future of Decentralized Infrastructure.)

Some of the validators were experienced with SGX while others needed to get up to speed on its setup and availability. Based on our learnings in working with the teams, we bumped a few features forward (SGX Wallet backup being one of them) as well as made more clear the differences between local SGX and network SGX (node clusters can run on native SGX servers or make use of a networked SGX server for Digital Key Generation and other SGX-dependent operations).

In the process, we also received a lot of great feedback on cloud providers and hardware setups. Whereas some validators ran their own dedicated SGX servers, others experimented with various cloud providers and analyzed the setup and performance. Some cloud providers  were easy to work with while others needed additional effort getting the right configuration. All in all, we were impressed with the work validators did to secure their nodes and take advantage of the benefits SGX-based chipsets provide.

Testing of Validator Delegation and Bounty Payouts

Another important testing area included all validator and/or delegator transaction flows. Delegation support is a key element within the SKALE Network. Given the enhanced delegation operations enabled by the  SKALE Token protocol, it was important to run validators through  significant tests in this area. The SGX Wallet plays a particularly important role in perfecting transactions within the SKALE Network as validators used validator wallets for receiving bounties and managing delegation operations. Bounties were paid out to validators, which in turn triggered payments to (validators acting as) delegators. Bounty withdrawal and fee withdrawals were also a part of testing phases.

Learnings to Date

No major issues were encountered in any of the initial phases of the incentivized Testnet. One of the nodes did run into a situation where they were making too many Ethereum calls from one of their containers, which resulted in the node running up against daily transaction limits. The issue has been identified and resolved.

Another issue we faced was traced to code within the SKALE Daemon and the interfaces to SGX signature generation where certain conditions affected performance. The team was able to identify and fix the algorithm so as to address this performance hit and eliminate the issue. We also fixed a few other cryptographic issues with Digital Key Generation and BLS public key signatures and have deployed all the changes to the SKALE Mainnet.

All the other issues that came up centered around documentation, requested features, and hardware configurations. Some validators got stuck on certain setup issues due to instructions unfairly assuming knowledge of certain topics around missing information. Other validators requested certain CLI commands to make node setup and operation more seamless. Still others identified issues with hardware configuration based on their cloud offerings or their bare metal setups. Almost all of these requests were or will be addressed in short order (aside from configuration issues that may be out of the network’s hands).

What’s Next

There are still a few more phases of testing to take place with this version of the Testnet. Specifically, SKALE Chains will be created and subnodes assigned to them on a random basis. Subnode rotation will take place along with block analysis and verification at each swap point. Unexpected conditions will also be introduced so as to allow validators to test node recovery, consensus pause and continue (with lossless transaction throughput), and other SKALE Chain-related operations.

Node replacement in the event of node failure will also be tested. If a node or subnode goes down, there’s no tech support to run to and say, “Come fix this node.” Instead, what happens is that every node is taking snapshots at specific points in time. If a node/subnode goes down, a new subnode gets swapped into the chain and the snapshot applied, thereby allowing this new subnode to catch up to the chain (and giving SKALE Chains a way to self heal). This self-healing capability has been heavily tested on the QA testnet and is the final component to be integrated into the master codebase of the SKALE Network. Testing of this node replacement capability is included in the next phases of the Testnet.

Final Thoughts

The overall experience working with the validators on the Testnet was an extremely rewarding one. The knowledge level of the validators is impressive and they helped identify a number of ways to improve node setup and operation. They asked the best questions and challenged our team to get them the right answers.

The validators also helped us realize that SKALE’s architecture is a generational shift from other networks,  specifically when it comes to containerization and virtualization. The flexibility and expandability that these two technologies provide also bring with them a learning curve when it comes to setup and operation. All this was certainly doable but it was a bit of a departure from their current network tech stacks and took a little time to fully adapt to the inner workings of the network infrastructure.

That aside, we think they’ll be far better off with containerization and virtualization in play, and feel our approach will be one that will be modeled more often than not given its throughput, operational flexibility, and other advantages. We’re grateful for the support of these validators and look forward to working with them in a live production environment.

About the SKALE Network

The SKALE Network is an open-source elastic blockchain network protocol. The mission is to make it quick and easy to set up cost-effective, high-performance sidechains that run full-state smart contracts. The SKALE Network aims to deliver a performant experience to developers that offers speed and functionality without giving up security or decentralization.

You can follow the SKALE Network on Telegram (@SkaleOfficial), Twitter (@SkaleNetwork), and Discord, visit the SKALE website, read developer documentation on SKALE Developer Portal (skale.network/docs), and see the code on Github.