A letter from a validator to the Hyperliquid team: You made a good product, but it can be better

Written by: Kam

Compiled by: Luffy, Foresight News

This letter is addressed to the Hyperliquid development team, hoping that the team can take the time to review this feedback regarding the governance of the Hyperliquid blockchain.

Key points:

• Validators face significant challenges due to closed code, lack of documentation, and reliance on centralized application programming interfaces (APIs), resulting in frequent jail time and unstable performance.
• Testnet incentives have spawned a black market for HYPE tokens, favoring deals with large holders rather than fair selection of validators.
• The validator rewards on the mainnet are too low to meet the high self-bonding requirements, and the degree of decentralization is limited because 81% of the staked shares are controlled by the foundation nodes.
• To compete with mainstream L1 blockchains, Hyperliquid must improve transparency, reduce staking centralization, implement a fair validator selection process, and strengthen interaction with external validators.

I started using Hyperliquid in December 2023 and it is an amazing crypto app. It is easy to use, has a great user experience, and offers some unique features like the vault and the famous HLP. Currently, HLP has over $350 million in assets under management, and anyone can participate in Hyperliquid in a passive way.

After seeing how great the platform is and knowing that Hyperliquid is running as its own L1 blockchain, I wanted Chorus One (Staking Solutions) to participate as an operator on the Hyperliquid chain. I am an employee of Chorus One, one of the largest node operators in the space. Chorus One has been active in the Proof of Stake industry since 2018.

Chorus One joined the Hyperliquid testnet after being whitelisted on October 17 last year. I wanted to share our overall experience on the testnet with the Hyperliquid engineering team, as we have not had the opportunity to interact with them even after joining the testnet for nearly 3 months.

During this time, we have witnessed one of the most successful token launches of 2024: the launch of the HYPE token. At the same time, we have also experienced an interesting and challenging testnet environment. I would like to mention some of the issues I observed, and I hope that these issues will be addressed in the coming days, weeks, or months.

Testnet Experience

The testnet experience has been extremely challenging so far. Operators have little knowledge of how to run a node, and the resources available are limited. In addition to this, we are basically exploring in the dark, and we have found multiple issues, including the following:

Frequent imprisonment for unknown reasons

At first, we were jailed multiple times, but we didn’t understand why. Since the code was not open source, we couldn’t accurately assess the reasons. The only thing we could do was to communicate with other validators on Discord and guess the reasons together. After talking to multiple validators, we found that everyone was jailed frequently, and they didn’t fully understand why.

Node location

We later discovered that the jailing issue was probably because we didn’t run a node in Tokyo. Moving the node to Tokyo might have helped. Unfortunately, the team never made this clear and we only discovered this after running into a lot of issues.

After moving the node to Tokyo, the situation improved. This is probably because many testnet nodes with a large amount of staked tokens are also located in Tokyo, so our node can miss fewer blocks and keep up with the pace. However, even after moving the node, we still continue to encounter jail time issues, and we still don’t know the exact reason. This problem is mainly due to the fact that the code is not open source.

Depends on the automatic jailbreak script

We realized that maintaining good uptime on the Hyperliquid testnet depended on how quickly the script could automatically jailbreak nodes. The only way to improve uptime was to rely on scripts that could automatically jailbreak nodes quickly. Validators could not fully understand or fix the underlying issue and could only automatically jailbreak nodes without a deep understanding.

Centralized Hyperliquid API as a single point of failure

On several occasions, our jailbreak attempts failed because the Hyperliquid API was down. If the API is down, validators cannot get out of jail on their own because they have to send a request to the Hyperliquid server to get out.

The team may be aware of this, but this design needs to be reconsidered because it makes the API a major single point of failure for the network. If the goal is to build a Byzantine fault-tolerant system, there should not be any nodes with special permissions, such as those that rely on a centralized API.

Mainnet Validator Selection

Hyperliquid recently selected about 16 validators in the process of decentralizing its validator set. Previously, Hyperliquid was managed by 4 validators from the core team, which attracted a lot of criticism. Recently, Hyperliquid took a major step and expanded the number of validators from 4 to 16.

Regarding the selection of validators, 4 validators were announced via the following post on Discord:

The four validators are Validao, Bharvest, Hypurrstake, and Prrposefulnode. They were selected based on uptime, and they have managed to maintain more than 90% uptime in the past 7 or 30 days.

This is a remarkable achievement for many reasons, primarily because validator performance is also impacted by external factors, such as glitches in the Hyperliquid API, jail issues, and persistent binary crashes, which all have a significant impact on performance.

In addition to these four validators selected based on testnet performance, five validators from the Hyperliquid Foundation are also running on the mainnet. In addition, seven other validators were selected for the mainnet, but the reasons for their selection have not been publicly disclosed.

Then, a black market for HYPE testnet tokens began to emerge.

The Hyperliquid testnet initially had 50 validators. Initially, specific entities were whitelisted to join the testnet, but on December 12, validators were fully open.

The conditions are simple: 10,000 HYPE testnet tokens are required to register as a validator. However, to become an active validator, you must also be in the top 50, otherwise the validator will be inactive.

This caused the price of the HYPE testnet token to surge. Initially, the price rose to more than 3,000 testnet USDC, and a few days later it even rose to more than 28,000 testnet USDC. At the time of writing this article, the current token price is about 700 testnet USDC.

Unfortunately, the faucet only releases 100 testnet USDC every 4 hours. To be among the top 50 active validators on the testnet, more than 528,747 HYPE testnet tokens are currently required. Assuming the token price is 700 testnet USDC per coin and only relying on the faucet to release USDC, the calculation is as follows:

Required days = (528747×700)÷(100×6) = 616871.5 days

This means that it would take approximately 616,871.5 days, or 1,690 years, to become an active validator on Hyperliquid, relying solely on the faucet to acquire the required HYPE testnet tokens.

However, those who received the HYPE airdrop on the mainnet also received an equivalent amount of tokens on the testnet. This creates an opportunity for validators to work with these community members and have them stake their testnet HYPE tokens to the validator, allowing the validator to take a seat in the active set.

At the same time, this situation also gave people who hold testnet HYPE tokens second thoughts. Given the fierce competition to join the testnet validator set, many validators are eager to obtain as many HYPE testnet tokens as possible. As a result, a black market came into being, and large holders of large testnet HYPE tokens began to sell tokens to validators in exchange for real USDC on the mainnet.

I have never seen such a mess. While the Hyperliquid team clearly disapprove of these practices, they are fully capable of solving this problem. One possible solution is to implement a proper validator selection process on the testnet.

In most other proof-of-stake networks, the core team typically shares a form that any validator can fill out to express their interest in running the chain. The team then reviews these applications and conducts an initial screening based on various criteria, such as the validator's experience running a node, past contributions, community involvement, or other factors.

This initial group of selected validators can contribute to the testnet, working closely with the engineering team to provide feedback and ensure everything runs smoothly. We have tried to provide feedback multiple times but have not been successful so far.

Mainnet and Decentralization

As mentioned previously, the current validator set on the Hyperliquid mainnet consists of 16 validators, which can be viewed here: https://app.hyperliquid.xyz/staking.

Five of them are from the Hyperliquid Foundation. Four were selected based on testnet performance, maintaining more than 90% uptime. Seven were selected by the Hyperliquid team themselves.

Of the 404,495,250 staked HYPE tokens, approximately 329,578,724 HYPE tokens are staked on foundation nodes, which is approximately 81.4% of the total staked shares. We don't know much about HyperBFT, but assuming it operates as a Byzantine Fault Tolerant system, the core assumption of most BFT systems is that no more than 33% of the voting power behaves maliciously. If a single entity controls one-third of the staked shares, they can stop the blockchain from functioning. If they control two-thirds of the staked shares, they can completely control the network.

The Hyperliquid Foundation initially staked 60 million HYPE tokens for each foundation node. However, many HYPE holders also chose to stake on foundation nodes, which is very detrimental to decentralization. The team should engage more with the community and encourage a more decentralized staking distribution.

There are three potential solutions:

1. Educate the community on the importance of staking with external validators to improve the security and decentralization of the chain.
2. The Foundation nodes implement a 100% commission rate to incentivize users to stake with external validators and promote decentralization.
3. Redistributing the foundation’s staked shares to external validators is the practice of most chains.

Spreading the stake to external validators also helps them become more economically sustainable. Hyperliquid is a blockchain focused on high throughput, and the infrastructure costs of running nodes, especially in Tokyo, can be high. Currently, validators at the bottom earn between $3,000 and $5,000 per year, which is not enough to cover their costs. What makes it particularly challenging is that they must stake 10,000 HYPE tokens on the mainnet (about $250,000 at current prices) to validate.

Currently, users interact with Hyperliquid by cross-chaining USDC from Arbitrum to the Hyperliquid chain. After reviewing the cross-chain bridge contract, it appears that the cross-chain bridge is still managed by 4 validators. These validators do not appear to be associated with the chain's consensus mechanism or the 16 validators on the mainnet.

Hyperliquid has a great product, but the team still needs to improve several aspects of its infrastructure to truly compete with mainstream L1 blockchains.

Some improvements are as follows:

• Listen to validators. While the team’s current approach of working on their own and not interacting with many outsiders has worked well in building their Perp product, validators are the backbone of L1 blockchains. Listening to them is just as important to ensure everything runs smoothly.
• Open source code. This will help validators better understand the issues they face when running nodes on the Hyperliquid chain and also help users trust the product. Open source code will also allow validators to learn more about the architecture and consensus algorithm. Currently, there is very limited information about HyperBFT, and open source can provide much-needed transparency and understanding.
• Create a proper validator selection process to eliminate black markets for HYPE testnet tokens. Selecting validators based on uptime is a fair approach, but getting good uptime should also be fair. It should not depend on external factors such as having connections to get testnet tokens, buying testnet tokens, or relying on the uptime of the Hyperliquid API.

Overall, Hyperliquid does not need to change drastically to compete with mainstream L1 blockchains. The main focus should be on engaging more with external parties and taking their opinions into account. I look forward to seeing how things develop in the coming weeks and months, and our team stands ready to help and give feedback. Wish the Hyperliquid team all the best!