The Magical Land of Smart Contract Storage

What is this sorcery all about?

When doing any kind of development in Web3, it's not only about doing it fast to keep with the pace the space is growing at - it's also about having a range of flexibility when using development tools in order to optimize both time and other resources you are spending on publishing, debugging, re-publishing and testing your smart contracts as well as your product as a whole. In one of the previous blogs we visited and conquered the Dark Forest - now we are going into the magical land of smart contract storage and State Overrides.

One of the more useful aspects of this would be to enable anyone to change anything about their smart contract on the fly - including the source code, smart contracts parameters and even any of the states - so you can:

• Test bug fixes and improvement ideas
• React quickly in order to check what would be the effect of a hotfix
  (e.g. when a hack is imminent or when it already happened)
• Collaborate better and create robust CI/CD pipelines
• (Fork and) Simulate everything, either with actual on-chain data or by changing reality and substituting your own

So, let's get into how this magical stuff actually works.

Learning the Spells

The storage pattern used by Smart Contracts is in no way standard, and non-standard things often lead to confusion. We’ll delve a bit into demystifying the concept of Smart Contract Storage and further examine how we can leverage it to make our development process faster and easier.

Before going in-depth on storage slots themselves, let’s examine how the EVM stores data in general - every address that exists on a network has a State Object associated with it. A State Object is an object that contains the balance for the addresses, bytecode, nonce and storage.

Balance and bytecode are pretty self-explanatory, but storage gets a bit interesting. In effect, the storage of a State Object is a mapping of 32 byte keys to 32 byte values. The astute among you will notice that this is an astronomically large amount of possible keys (2^256 to be precise). Luckily, you don’t actually have to store all of the key/value pairs, since you can just treat absent keys as zero values.

Now that we have the basic structure in mind, let’s look at an example of a contract:

This contract has 2 state variables (2 values that will be stored in the storage section of the State Object for this contract). Let’s examine how this works exactly:

For uintStateVariable this is rather simple. This is the first storage variable so it’s index will be 0, meaning it will be stored at key 0 (the 32-byte representation of 0), so the storage slot associated with it would be 0x0000000000000000000000000000000000000000000000000000000000000000

For stringStateVariable the issue gets slightly more complex. Storage slots for strings (and bytes) are calculated a bit differently, and here we can observe 2 distinct cases:

• When the length of the string is smaller than 32 bytes both the string and the length of the string are stored in the same slot - in this case the slot is: 0x0000000000000000000000000000000000000000000000000000000000000001
• When the length of the string is larger than 31 bytes the length is stored in slot 0x0000000000000000000000000000000000000000000000000000000000000001 while the slot for the value itself is calculated using a hash function (sha256 in this case), so the actual slot of the value would be: hash(0x0000000000000000000000000000000000000000000000000000000000000001)

The kicker here is that you have to pay a gas fee every time you write something to the storage, so storing a string of length 32 is twice as expensive as storing a string of length 31. That said, there are techniques to optimize gas usage by packing together some values:

In this case the values of firstStateVariable and secondStateVariable would both be stored in the 0x0 storage slot, while the thirdStateVariable would be stored in slot 0x1. However, if we were to alter the ordering of the variables to the following...

... then the firstStateVariable and secondStateVariable can no longer be packed together, resulting is our contract needing 3 storage slots:

• firstStateVariable -> 0x0
• secondStateVariable -> 0x1
• thirdStateVariable -> 0x2

Note - You can read more about this on the official Solidity Docs.

Taming the Beast

Now that we have a basic overview of how Smart Contract Storage works lets see how we can use it to do some very useful stuff in Tenderly.

As you might have seen recently, the Simulator feature now supports State Overrides during simulations. This allows you to set any state variable to any value, run your transaction on top of that, and then inspect the outcome - all in great detail.

In the example above, Tenderly calculates the storage slots of desired state variables, then overwrites and sets them to the desired values. You can inspect the raw key/value pairs in the State Overrides tab:

Let’s delve a bit into how this is possible on Tenderly - first we have to understand how the EVM gets and uses the on-chain data during the execution of transactions.

As the aforementioned State Objects are stored by the nodes, the EVM needs to fetch the relevant data from them. The EVM maintains a state database during the execution to cache the data it fetches in order to avoid multiple calls to the nodes for the same thing. We inject our custom State Objects into this database before running the transaction itself, making the EVM use our data instead of real data. Because everything else is the same, all other features of Tenderly can be leveraged in an identical manner, as if you're working with actual on-chain data.

Editing contract source code

A useful (and very interesting) side effect of Tenderly being able to do this is that we can also overwrite the bytecode of the address. This enables our Custom Source feature, that allows you to make edits to the contract source code directly from the Simulator so that you can very quickly iterate and test potential bug-fixes - without re-publishing the contracts or spending money on gas.

By recompiling the contract with the edited source code, we can get the new bytecode to replace the existing on-chain bytecode. This allows us to assign an arbitrary source code to any address on Ethereum in order to test potential bug fixes, do some quick prototyping, discover bugs in testing with various parameter values and edge cases, and so much more. In tandem with the State Overrides described above, this gives us absolute flexibility to alter any on-chain data to suit our needs.

A quick example - Simulating a transfer we don’t have funds for

Firstly, we need to configure the simulation to invoke the proper function on the contract:

Then we’ll paste our address into the from field:

We’ll add the State Override that will set the balanceOf for our address to a value greater than the input to the function:

And we’re set! All that is left to do is run the simulation and see the result:

To conclude...

Tenderly is truly an all-in-one development platform for Web3 builders. We're ramping up our efforts to properly showcase everything that is possible with the current (and always-expanding) feature set that combines debugging tools with observability and blockchain infrastructure.

State Overrides are extremely useful in many situations - especially when you do a Fork, quickly change any (or all) of the states for any parameter of any contract and save it as a Branch as an easy access point for (repeated) testing. We also recently released Advanced Trace Search so you can instantly find anything you might need in the Execution Trace for any transaction.

We are also working on adding Native Balance Overrides so you can simulate transactions with any amount of funds - you will be timely updated about this feature coming to a Tenderly platform near you :)