How Concrete Uses Virtual TestNets for Appchain Development
"Using Virtual TestNets to have the current state of a production chain and manipulate the underpinnings of that state is really powerful. Those are the pieces that keep us moving forward."
Ryan Turner
VP of Engineering at Concrete
Concrete is a network that powers the creation and trading of DeFi derivatives. The chain is deployed as its own appchain through the OP Stack. Blueprint Finance, the operator of the Concrete Protocol, is a technology development company building a suite of decentralized finance products.
Concrete’s architecture is based on a hub-and-spoke model, with a novel cross-chain messaging system. The system was developed in-house and involves a combination of backend components and on-chain event monitoring. This structure also allows Concrete to manage both the state and data associated with the core product(s).
From a useability standpoint, the Concrete chain enables users to earn the highest risk-adjusted returns through ERC-4626 vaults running programmatic DeFi strategies. By depositing into these vaults, users also receive a liquid vault token, which can be used as margin in Concrete's Money Market. In addition, Concrete protects users from liquidation and prevents the accumulation of bad debt by issuing composable, automated credit facilities.
The challenges of building in public
Initially, the Concrete engineering teams used Foundry forked networks for unit and end-to-end testing. However, it quickly became apparent that this setup would become confined to a smart contract repository, limiting teams to working in isolated development environments.
There are deep limitations when using a network fork, including not having the up-to-date state of the production network. This lack of clarity leads to assumptions being made before deployment. Virtual TestNets eliminate this problem. – Ryan Turner, VP of Engineering at Concrete
The backend team then ran into other issues when looking to integrate with smart contracts. Concrete’s Solidity engineers explored deploying the contracts to Sepolia. However, since this is a public testnet, the team couldn’t use unverified code as this would prevent them from troubleshooting errors. This also made their code public in the early stages of development, something Ryan Turner, VP of Engineering at Concrete, was very uncomfortable with.
Additionally, Sepolia posed other problems, as it was difficult to find enough test funds to deploy and run basic executions, let alone more complex tests. The team even had to pay for Sepolia ETH.
It was undesirable to have to buy test tokens to manage multiple deployments and facilitate testing. – Ryan Turner, VP of Engineering at Concrete
Migrating to Virtual TestNets for seamless backend integration
Concrete’s backend is integral for the execution of smart contract code as most of the interactions with the Concrete smart contracts happen through its backend. For this reason, the backend integration on Virtual TestNets is an essential component for the team’s development processes.
The Concrete team uses Virtual TestNets for all of their contract deployments. Once their new features or product updates are ready, they deploy smart contracts to Virtual TestNets so the backend team can integrate them. This way, they’re able to test the integration against on-chain data while keeping their code private during the early stages of development.
In fact, the backend team has tiered Virtual TestNet instances to stage and test their integrations more efficiently. They use Virtual TestNets as dedicated development, staging, and production-level environments.
How Virtual TestNets boost the Concrete development workflows
The integration of Virtual TestNets opened new possibilities for the Concrete development teams:
- Seamless contract deployment for backend integration testing while maintaining privacy during early development stages.
- Syncing the latest production state for up-to-date on-chain data and realistic testing conditions.
- Manipulating the state of their environments with custom RPC methods for setting balances or jumping forward in time.
- An unlimited faucet for test tokens to provide the balance of any token to any address, ensuring that all of the team members have enough for testing.
- Efficient troubleshooting with a built-in explorer and debugging tools so the team can dive into code execution step-by-step.
- Quick redeployments whenever the smart contract code is changed, allowing teams to iterate quickly on their smart contracts.
Using Virtual TestNets to have the current state of a production chain and manipulate the underpinnings of that state is really powerful. Those are the pieces that keep us moving forward. – Ryan Turner, VP of Engineering at Concrete
How Concrete devs use Virtual TestNets for collaboration & debugging
The Concrete engineering teams have open and dynamic communication to ensure transparency and alignment throughout as they’re fully remote. This is particularly important for the smart contract and backend teams as they work closely together due to the nature of Concrete’s architecture.
Prior to Virtual TestNets, Concrete’s architecture brought about certain debugging challenges for the team as a single contract acted as the backend entry point into the protocol. The contract would then spread out across different protocol layers to execute transactions, which made troubleshooting difficult and unpredictable.
Now, with Virtual TestNets, they can collaborate and debug much more efficiently over a shared infrastructure. The Concrete smart contract and backend teams rely on the built-in virtual explorer and debugging tools to accelerate issue resolution.
Because of the way our architecture is set up, the entry point is completely disconnected from the end result, which can be a complete nightmare for debugging. Tenderly can trace the flow of transactions all the way through. Debugging with Tenderly is unmatched, there’s nothing I have found that comes close to it. – Ryan Turner, VP of Engineering at Concrete
With complete observability over transaction execution on Virtual TestNets, the teams can identify the specific line within a specific contract where a problem occurs. Once the backend team comes across an issue in execution, they share a link to a failed transaction with their smart contract team. The smart contract engineers then create a patch and deploy it to a Virtual TestNet so the backend team can continue their development and testing process.
Gaining confidence before deployment with Virtual TestNets
The Concrete team is in the process of automating deployments and testing on Virtual TestNet instances as a way to accelerate their development processes. By integrating Virtual TestNets into their CI/CD process, the team will be able to have an environment as close to production as possible.
As we integrate Virtual TestNets into our CI/CD pipeline, we can have confidence that when we deploy something to a real network, it executes exactly as we expect. – Ryan Turner, VP of Engineering at Concrete
As the team continues developing the Concrete network, they will integrate other Tenderly features. As part of their production infrastructure, the Concrete engineers will introduce Web3 Actions to eliminate backend dependencies, especially for their cross-chain messaging system.
