Testnet committee charter and Q2 funding proposal - Take 2

Secret Governance
#1

Hi everyone,

My name’s Taariq and I’m a long time Secret Network supporter;

  • Development and support of the ENG to SCRT token swap

  • Dev support in forums, discord, hackathons, docs, testing

  • Stable mainnet validator since genesis (quiet monkey mind)

As a software engineer of +20 years, I’m experienced in building and maintaining highly available systems.

In recent years I’ve worked on crypto projects for thousands of users who rely on multiple blockchains.

This experience taught me what’s important to developers going from their local machine to testnet.

It’s critical to provide a stable platform to prepare for mainnet, so of course this is the primary goal.
That includes upgrades to the testnet with zero/low downtime.

This requires some redundancy, so besides the 4 validators I’ll run, there are 4 additional backup servers used by the load balancer, which can be repurposed temporarily to test upgrades/forks.

There’ll also be a rate limiting API server to throttle requests to ensure availability.

Also important is documentation and support for developers and other users of the testnet, as well as validators who’d like to join the testnet at times, e.g. test upcoming features, or the upgrade process.

I’ll liaise with other committees, Secret Labs and other testnet users to maintain stability and keep an eye on the horizon.

Monitoring with Grafana and Prometheus is already available, so will use that out of the box.
Additionally, based on the many questions about which lcd/rpc/grpc endpoints to use, I’ll build a status webpage that queries these various endpoints and reports on their status.

Stake or die will provide alternate endpoints in case of DNS or any other issues with the primary testnet endpoints.

Existing testnet validators will continue and be compensated for expenses, and I encourage any other validators with experience on Secret Network, limited to 10 compensated validators.
Please reach out to me before the proposal goes on chain, assuming it does.

First Funding Proposal (2022-06-09 - 2022-09-09)

The testnet committee is seeking a budget for:

Lead Compensation: Taariq - $120/hr @ 30 hours/month = $10,800 per quarter

Validator Expenses: $250 per validator for Server Costs, up to 10 validators = $7500 per quarter

Committee validators: $250 per validator, 4 validators = $3000 per quarter

Committee backup nodes: $250 per node, 4 nodes = $3000 per quarter

Public Endpoint: $200 for server costs per month = $600 per quarter

Backup Public Endpoint: $800 for server costs per month = $2400 per quarter

Secret Node explorer support: $300 per month = $900 per quarter

Pulsar-2 team compensation back pay (One-time): Compensation for the work that the Pulsar-2 team has done so far ($6,000.00)

Total: $34,200

Total with 10% Buffer: $37,620

Carry Over
All remaining funds will roll over to the next quarter.

You can find the committee charter here

Thanks to DannyM for taking initiative with the first proposal, hopefully I’ve adjusted appropriately to meet the community’s expectations at this time, and look to build on this with your support.

7 Likes
Testnet committee charter and Q3 funding proposal
#4

Thanks for the feedback Ian.

Its not for perpetuity, it could be more or less next time, probably less given the community’s appetite for anything besides have a running testnet, or at least the voices you represent.
If I haven’t used the hours, I’d carry that over as mentioned.

I also want to do the work mentioned upfront, and if later in the quarter we have lots of maintenance/support (unlikely of course), I’m committed to that regardless of how many hours remain, that’s the #1 priority.

Here’s rinkeby for an example of a welcoming testnet.

7 Likes
#5

Thank you Taariq for your effort in putting this together. A much more realistic proposal!

I’m also happy to see the service level expectation for the 10 validator slots in the charter.

The dev hours seem to be the most controversial point. I think it falls within the expectation that you will have to spend some more time on it at the start, especially for front-end related matters. Looking forward to transparent communication on hours spend and on what. I believe that will be to the benefit of future support and continuity of this testnet solution (win-win).

For us, the dev hours, at a max monetary risk of $10k, in combination with your long history in Secret Network and running infrastructure, are very acceptable. Domerium Labs will vote ‘Yes’ when this goes to chain.

5 Likes
#7

Thanks so much, really appreciate the support

4 Likes
#9

Support this proposal, happy to see taariq step up.

3 Likes
#10

I’m happy to support your proposal @taariq!

4 Likes
#11

Thanks kindly folks, with your support I’m preparing to go on chain later, monday latest.

Please reach out if you’re already a mainnet validator, and keen to support the testnet with a validator as outlined, still a few slots open.

3 Likes
#12

hey @taariq please contact Gino or me to get access/info on the Documentation working group. We have a gitbook setup for the new docs and can get you integrated so you can work on the Testnet part of the documentation.

Hope we can see this testnet operating the CosmWasmV1 upgrade very soon :slight_smile:

3 Likes
#13

To get a headstart on the status page, I’ve deployed an initial version to Vercel for convenience, please open a PR or send your endpoints to me. Any other PRs also most welcome.
I’ll use a more appropriate domain in the coming days.

Currently supports only RPC, LCD and GRPC_WEB, whereas GRPC and WS haven’t been implemented, will circle back to those once the other tasks are completed. Include those endpoints anyway so I can test them when I get there.

The UI/layout will get a little more love, meanwhile please let me know if there are any issues or suggestions.

1 Like
#14

I will leave this clear from the start, that I have 0 experience running a testnet. But, having said that I have 14 years as a software developer and to me the weakest point of this whole proposal is that it relies solely on @taariq, no offense, and you’re a single human being.

I would much prefer that the biggest apps on the network, which also would benefit grately from a testnet to try upgrades to their apps, network upgrades, etc are the ones contributing to this, both with funding and resources instead. Hopefully with a team (even if small) instead of a single person.

What is also not me tioned here, which is critical IMHO is documentation about all the “development” of the testnet and all its accessory tools. Docs for users and validators is not enough.

As it stands, I would vote No (not that my vote would change the outcome much). Hopefully tuis perspective helps, I know the testnet is very important and in fact too inportant to rely on a single person

#15

Hi @votor133t , thanks for taking the time to share your feedback.

I’m fortunately not alone, I’ll just streamline some processes by taking sole responsibility, it remains in other validators’ and builders’ interests to also perform the upgrades and so forth.

The docs and scripts mentioned will help others run and scale to meet their own demands as needed.
If there’s any topics not covered please elaborate or mention those specifically and I’ll do my best, certainly for any work I’m contributing and more than happy to look at all testnet related docs, or missing docs.

Thanks again for your consideration, all feedback is greatly appreciated.

#16

Thanks for quickly replying to my feedback. I think I have the same concerns as others have expressed about the number of hours. Especially given how high the hourly fee is. This fee of $120/h would give $960 a day (assuming an 8h workday) this is quite high IMHO. Especially because the expenses for the 4 validators and backups are paid for separately in this proposal if I read it correctly

#17

Thanks for voting everyone, appreciate the support and will do my best.

Please note that I’ve updated the charter doc with a revised budget;
“At the time of proposal, SCRT price was $1.41, at the time of passing, the price was $0.97.
To avoid proposal fatigue by going back on chain, the period has been reduced to 2 months.”

While I’ve kept my rate the same, I’ll track the hours doing admin and docs etc as separate from dev/devops, and I expect I’ll end up doing more than 60 hours.

Thanks again, please don’t hesitate to reach out with any ideas or suggestions.

#18

Hey Taariq,

You have received $25.880 worth of SCRT which is less than the initial ask of $34.200 but more than 2/3 of the initial ask which is $22.800. This means you now have $2000 extra for this funding period, can i assume this goes to the Pulsar2 team (which was a set ask and not a monthly one) so that the committee budget is correct for exactly 2 months or will you ask for additional backpay in 2 months?

We have not seen anyone use this reduction in funding period yet (afaik) and with your proposal being mostly costs instead of man hours i think it makes sense. Goodluck with the work planned out.

#19

No, backpay will be settled in full with this ask.

I also don’t expect to require 3 months for the tasks I mentioned, given I can allocate the hours in spurts, with docs and initial setup being the priority, so the extra funds will either be utilized in providing the additional value proposed within these 2 months instead of 3, or carried over to the next funding period.
You can expect full transparency as mentioned.

Thanks, I also bounced it off Mook who referred me to governance proposal #81, and this seemed the best approach given current conditions.

3 Likes
#20

Secret Testnet progress report for Q2

Thanks to the community support, we’ve been able to set up the stable infrastructure described in the original request, and fulfilled all responsibilities, except the upgrade which is planned for September, or earlier on the testnet.
I’m already running a private testnet with the beta version, more details on this below.

As a reminder, these are the items committed to, with updates inline;

The Committee will hold itself to the following core responsibilities:

  1. Run and maintain stable testnet validators to enable new developers to join the network more easily, without having to deal with the pains of having to find a reliable testnet node.
    DONE (12 hours)

  2. Maintain documentation.
    DONE (23 hours)

The following information must be easily accessible and maintained for the current public testnet, and possibly any betanet config where validators can test network upgrades before applying to mainnet;

Similar code in SecretJS templates, simple UI connecting Keplr and interacting with the secret counter example.
https://github.com/scrtlabs/SecretJS-Templates/pull/30/commits/03760a79ccffc6e491bdfe230738c7001ce63d89
Same code used by Pulsar, https://connect.pulsar.scrttestnet.com/

  1. Script to run your own public testnet, using docker-compose to start the secret node, as well as an nginx server directing traffic from the internet to the correct container (initially only the node and faucet, in future there may be more services like stats or explorer) Letsencrypt will create the self-signed certs. This will allow devs collaborating to test shared contracts or bleeding edge features of an upcoming release in their own environment, without the risk of this committee introducing new bugs or downtime for upgrades while they’re relying on features already released in secret.

This script can also be used to spin up a staging env on demand, once the release is nearing completion, so as not to incur year-round costs for a short term need.

DONE (31 hours), see GitHub - levackt/private-scrt-testnet: Runs a private LocalSecret testnet, with essential services such as faucet and explorer

With this I can easily deploy and test contracts, upgrade the docker version, and retest my contracts.
You can see this in use to test the upcoming CosmWasm V1 on https://beta.scrttestnet.com/

  1. Test upcoming upgrades, then upgrade the main public testnet so that other testnet validators can confidently do so too. If there are any issues with the upgrade, the public testnet is unaffected.
    Coming up later this month, or September

  2. Monitoring, including a webpage showing the current status of any public endpoints.
    DONE (16 hours):
    Internal monitoring includes the standard Grafana dashboards, as well as PANIC to alert by telegram, email etc.
    Current status of any public endpoints is deployed on https://secretnetworknodes.com/

There you can see both testnet and mainnet endpoint status, and I’ve also been updating Cosmos Directory which has more features, but doesn’t yet support grpc web status for example. I may send them a PR to do so, meanwhile I’m still finding our own page helpful, hopefully others do too.

  1. Keep 3rd parties such as explorers Secret Nodes and Ping Pub informed of any changes, such as chain-id and endpoints
    DONE (1 hour)

Additional tasks:

  1. Griptape API cluster was shutdown, and Alex of Secret Saturn stepped up to run a backup cluster.
    This required many updates to documentation that referenced griptape’s endpoints, Keplr connections, explorers like Ping.Pub and cosmos directory.
    As well as these updates, I’ve also deployed an application that allows testnet users to reconnect/configure Keplr for pulsar-2, see https://connect.pulsar.scrttestnet.com/
    Total hours: 12

  2. While building the private testnet, I fixed a bug in local secret docker where a custom chain ID wasn’t possible. See https://github.com/scrtlabs/SecretNetwork/pull/970
    This allows us to run a custom testnet with it’s own chain-id, so Keplr etc has a unique connection for each environment.

Total hours: 4

  1. Also while building the private testnet, I ported the local secret faucet from Python to Node, see https://github.com/scrtlabs/SecretNetwork/pull/959
    Total hours: 10

  2. Support:
    There’s ongoing demand for support in discord. I’ve been available to promptly address any issues people think might be related to the testnet.
    Sometimes testnet support has some overlap with dev, or requires me to write some code to prove the testnet works and supports the features requested.

4.1 SecretJS Templates:
There’s been some big changes to SecretJS since the templates were last updated, so I’ve been updating the examples to work with the current beta version of Secret.JS.
This is still a work in progress, and so far has taken 12 hours to complete about 80% of the examples, the other 20% could take another 8 to 12 hours.

4.2 Showing how queries work ( 1 hour)
https://codesandbox.io/s/reverent-napier-r6ewh0?file=/src/App.js
4.3 Keplr example for balance (1 hour)
https://codesandbox.io/s/laughing-fog-b2rm36?file=/src/App.js
4.4 Contract example (1 hour)
https://codesandbox.io/s/headless-violet-uv2zow?file=/src/App.js
4.5 Multiple transactions in a block (1 hour)
Discord discussion over several days, with proofs from SecretJS templates to support the feature.

4.6 Faucet
Currently the faucet is down, so I’ll manually send tokens when people need it, but we should probably run another testnet faucet as either backup or the new primary faucet.
Estimate is about 4 to 8 hours to deploy on the same infrastructure used for the API server, so no additional hardware required.

Expenses for Q2:

  • Pulsar-2 validators backpay (Total $5000)
    Baedrik $1500
    Stake or die $1500
    Luigi $1500
    Jackal $500

  • Validators in Q2 (Total $3500)
    For the 7 validators @ $250/each over 2 months = $3500

We currently have 7 compensated validators, plus 4 of the committee, and 2 recently joined. I’ve reached out Ertemann who seems to run one node, and I don’t know who the other validator is (sgx-bad-boy)
As there’s room for 3 more, if you’re also a mainnet validator, I urge you to contact me and get looped into the testnet validator group, to stay informed of upgrades and get an equal share of the bonded tokens.

  • Backup endpoints
    Initial budget was for the same spec as validator nodes, we’ve bumped this up to compensate more fairly for the beefier hardware and extra overhead of archive nodes and the load balancer.
    Stake or die was hardly breaking even with their expenses and the accounting was a burden, so instead of $800/month, this is now $1200 monthly after comparing the expenses incurred by myself and Alex of Secret Saturn who took over from Stake or die.

$1200/month * 2 = $2400

  • Committee validators
    $250 * 4 validators * 2 months = $2000

  • Committee archive nodes and endpoints (total $2400)
    $400 * 3 nodes * 2 months

  • Secret nodes explorer $300

  • My hours:
    Estimated $120 * 90 = $10800
    Actual: $85 * 90 = $7650 (In reality I went over the 90, but this is the budget)

  • Small once off expenses like domain registration have been absorbed by me.

Grand total for Q2: $23,250

Surplus funds: $2,350

Q3 estimates:
Estimate for Q3:
My hours: 58 * $85 = $4930

  • New/Backup testnet faucet: 8 hours

  • Network upgrades: 20 hours

  • Support: 30 hours

  • Secret node explorer: $900

  • Validators and committee expenses: $20850

  • Surplus from Q2: $2350
    Total ask excluding buffer: $20850 (expenses) + $4930 (hours) - $2350 (surplus) = $23430

Transparency

Payments to validators:
I won’t link to validators’ wallets, but I’ll show amounts paid to each validator, and you can compare this on the explorer, and validators can object if they disagree.
Any of the validators receiving funding, can of course check the account they receive from to see if these transactions make sense.
If someone like pmueke | SecretSauce would like to check transactions on behalf of the community, I’m more than happy to comply.

Hours:
In terms of hours worked, many of you are experienced to know how long these tasks should take, and my activity on github is public and most of it is directly related to the Secret Network activities I committed to.
o9MKxJaF1aSx3aBfQ1sON7PFk8jmEv0KLbaQ-gmrpJXCucA9liA5AW_7A_zh7qOYEYqV3PGL2xpKumJVHbFVHJ1B5lNdAiAfjDKBnahn6TkM1kYJVtbC8ZW8M_i6yEC_B865kz2IBydv_vj5vdjEePA

Rate:
Initially I asked for the same rate as the other devs, a bit lower given I expected some easier admin work already well documented.
They have since reduced, and I’ll match the $85/hour retroactively and going forward.

Thanks for your support, it’s much appreciated.

3 Likes