Network Performance and Max Gas Discussion

Secret Governance
1

Hello Secret Community,

Last week we saw some network performance issues as a result of increased activity. In this post, we’ll outline:

1. Why these issues arose
2. What has been done
3. Why does this matter
4. What can we do to fix this

Why Did This Happen?

The reason for the impact on network performance is multi-layered. First, the network itself has some scalability issues that must be addressed long term - this has been known for some time, and the fixes required involve improving cosmwasm performance in the TEE. It’s a significant amount of work to do this, so while it’s in the roadmap, it has to compete with more pressing issues. As a result, the major changes required will not be feasible within the next six months, at least.

The second reason is that some legacy contracts are inherently inefficient in their use of gas. As an example, Shade multi-hop transactions of more than three hops tend to be computationally intense enough to impact block time and cause validators to miss blocks due to no fault of the validator. With respect to the relevant Shade contracts, the team told us the biggest issue is with the older StableSwap contract.

What’s Been Done?

Secret Labs worked with Shade to refactor the currently deployed contracts to make them more efficient. This sounds like a big step forward, until you take into account that making significant changes to a live production product is a challenging task. Sure, ETH 2.0 did it, and did it well, but it took them years longer than the original plan, because everything had to be carefully evaluated. You could summarize the situation by saying that both Secret Network and Shade Protocol have some tech debt, and neither is in a position to allocate resources to cleaning up tech debt at this exact moment, in the face of higher priorities with more upside.

Why Do We Care?

The challenge we face as an ecosystem is that if nothing is done, it is almost certain that increased usage of these legacy contracts without properly setting max gas limit will result in saturation of the network’s capacity, and ultimately could cause the network to perform poorly. This can lead to several undesirable outcomes:

  • Causing deposit / withdraw issues for SCRT on major exchanges
  • blocking SCRT holders from using the network
  • loss of confidence in Secret Network by token buyers
  • loss of ecosystem projects due to network unreliability

No one in the Secret ecosystem would choose these outcomes.

What Can We Do Today?

A research project was undertaken by Alex Saturn while at SNF to determine the optimal level of max gas that would permit the network to run smoothly while not impacting legacy contracts too heavily - proposing the best compromise. The number provided was around 5M per block.

We’ve also discussed at length with the Shade team how some of their older contracts work, and how many hops are required for certain types of transactions and users.

The tl;dr is that there is no ideal solution, but the highest priority is to keep the network running for longevity and sustainability.

It should be made clear that the proposed max gas of 4-5M is a theoretical number - it’s not possible to ascertain exactly what the impact will be without empirical evidence and traditionally, this parameter is changed on chain as needed.

Longer term, we will all need to undertake some work to improve performance. We don’t have a timeline for this today, but Secret Labs is committed to doing this over time.

Discuss!

This is an open thread for discussion by validators, users, and developers on their observations, possible next steps, open questions, and other comments. Please be respectful and constructive - our goal is to get Secret Network working at maximum capacity and allow for growth for all ecosystem builders and contributors.

Special note to all dapps building on secret: please let us know what gas usage you expect for your application deployment or usage during this discussion phase.

5 Likes
2

Strongly support lowering max gas limit to more realistic values. Also hope to see more great collaborations like this between core devs and builders on secret🙏🏾

1 Like
3

Our use case for NFPs would be quite limited by 5M gas, but considering that the CPU load it puts on the node is negligible, perhaps we could address it instead by adjusting the way that cost per byte read/written is metered.

1 Like
4

I have 2 questions on this.

  1. Whats the timeline for your NFP work launching? @supdoggie
  2. When might Secret Labs be able to look into changing how the metering is done? @alexz

Limiting use cases is frustrating, but in reality, the max gas limit or how metering is done has never been properly set on secret network. So this proposal isn’t adding new limits, it’s adjusting to the reality of an existing limit. At the end of the day, the network needs to be stable and the param can be changed multiple times along the way while other solutions are worked on.

5

Hi, I’m one of the shade developers.

I’ve got limited time today and hope to be able to elaborate more next week, but one point of caution when lowering gas is there are some contracts that will become undeployable that were previously deployable at a higher gas amount. In the case of new contracts it could be a pain to split logic into multiple contracts to work within this constraint, however that is something that is still workable (although as a side note, splitting contracts makes gas less efficient…). However in the case of existing contracts, if there is found to be a bug that needs to be fixed asap, the migrated contract code could become undeployable and that is a serious problem that becomes essentially unfixable without an emergency network change, which by the time you pull that together, it could be too late.

Just wanted to bring this up because there’s more to the gas needs than just optimizing block performance. You also want to be able to support deploying complex contracts.

2 Likes
6

Linking the old discussion here just to save some time on points brought up in the past

7

I wasn’t doing contracts at the time, but I remember one of my coworkers struggling with deployments right after that change. It was a side effect that was overlooked at the time. We ended up working through it and it wasn’t that bad of a deal, but did want to bring that up this time around because at some point we could hit a limit that is actually problematic.

I hope to share more details about this next week including size of our contracts. I’m not saying I’m opposed to the change, just wanted to bring it up as a point of caution.

1 Like
8

It looks like last time pmook asked all the teams how much gas they needed to deploy, and asked to deploy before the change essentially. Then the option of changing it as needed for storing new ones was mentioned. Generally though, no one wanted the network to cripple under high usage.

9

Is this the use case that will allow transactions to be pre-signed and strung together - i.e. the one you mentioned recently?

When do you expect that to be realistically ready to launch? Just asking because maybe core changes to gas costs per byte r/w could be implemented in the interim depending on timeline.

1 Like
10

Is there a specific period in degraded performance that brought up this discussion?

All we have really seen is full blocks but no delays on the processing of those.

We were potentially leaning more on the side of increasing the gas limit to see if we can get the chain to do more than 1 tx per second (which is the average we have seen during the 2 high volatility periods last 2 weeks with full blocks)

Side note on scrt deposits from exchanges and native related functions:
For this secret should really implement feemarkets, gas lanes and so forth using the skip module. This is possible with the latest sdk and shouldnt be too much of a lift as there are some standardized example implementations from chains like neutron and it doesnt necesarilly impact the VM and therefore TEE parts of the codebase. Reducing the gas limits will still see them full and therefore native functioms take longer than wanted.

Edit:

Seems we missed like a few % of blocks and others did as well, as long as this is not consistemt or puts people far below 95% we generally dont care, hence “didnt see much degraded performance”.
Many networks where vals mis consistently higher amount of blocks then this, not optimal but dont think its that big of a red flag.

The issues during past week imo were a blip om the radar. But frankly dont care enough what the chain does with lowering, i think this choice should be leaded by the big Dapps, slabs and Snf. The people that have seen the downstream user impact. If for now 5m is required and other validators vote for this then we will follow suit.

11

L5 was indeed one of the validators who experiences degraded performance when shade usage has an uptick. Please do not make a proposal to increase the gas limit, it could cause exchanges deposit / withdraws to not work whenever shade had a few (less than 10) users trying to consistently swap over longer periods of time. I know it’s been a while since they’ve had that happen, but we shouldn’t propose things that would cause issues for exchanges processing i/o’s because that damages user trust and likely damages the price.

Regarding quantifying the impact of missing blocks. The network will continue to run, even if every block is full of a shade transactions. Exchange withdraws and deposits pause when the exchanges nodes have issues and even with the network technically running, nodes can have enough issues for exchanges to pause.

12

I’d say if we are deciding to lower the gas limit, we probably need to update the network code to charge less gas for storing contracts. If time prohibits that, IMO it’d be a bad idea to go any lower than 5M since there are a decent number of contracts that need between 4-5M to store

2 Likes
13

Great idea and probably not very difficult to do.:ok_hand:

1 Like
14

Something we can definitely look into.

1 Like
15

I agree the network should be stable. Do we have evidence that confirms blocks with 5M-6M gas are the reason for missed blocks? If the root cause is something else then lowering the limit might not help. Who knows, maybe its what the contracts are doing and certain opcodes aren’t metered correctly? Without data one can only speculate.

16

The shade team performed some testing this week and our upcoming money market contract requires 5.9M gas to deploy.

Based on this testing we will be strongly against lowering the block limit to 5M unless there is something we can do to allow for larger contract deployments. I have heard from Alex that SLABS potentially has some options to address this.

17

That’s as simple as doing a proposal when you guys are actually ready to deploy that contract to increase it and Another proposal to lower it again. They can even be made 1 day apart so there is simply a 1 day window.

But risking exchange stability for longer bursts for the purpose of deploying a contract hardly seems like a good trade-off. Does shade have any issue with this? @AustinW

18

This is not an acceptable solution, because if an emergency migration was needed, we cannot wait 24+ hours to pull together the validators to allow us to redeploy.

19

Understood. In the future it would be ideal if all builders prioritized the L1 and it’s status on exchanges over other matters. It’s unfortunate to see that not be the priority for shade.

20

I never said that, I was just offering our perspective as app developers. We add a ton of risk to our users by making them use contracts that are not deployable.

1 Like