Testnet Committee Charter and Q2 Funding Proposal

Secret Committees
#1

Hey All,

I’ve been an active participant in the network and have been the lead developer on Secret Dreamscape. I’m also the lead developer on the Digiline IDE for Secret Network (which received a grant from SCRT Labs, and we’re nearing the completion of the first phase of the project). I’m also running the Digiline Validator alongside Gino (one of the dev committee leads). It would be fair to say that I’m putting many of my eggs in the Secret Network basket.

With introductions out of the way, below is my proposal for a much-needed testnet committee:

The testnet committee is primarily responsible for running and maintaining three stable testnet validators, one in the United States, one in Central Europe, and one in Asia, with the country or region depending on data center availability. Furthermore, the committee will sponsor and support up to 10 additional testnet validators by paying the server costs and dividing 4000 USD/month amongst validators depending on their node’s stability.

*Note that validators run by the committee lead will not receive any rewards and will only have their costs covered.

First Funding Proposal (2022-05-18 - 2022-08-18)
The development committee is seeking a budget for:

Lead Compensation: Danny - $150/hr @ 20hr/week = $36,000 per quarter
Validator Runners Rewards: $175 per validator for Server Costs and an additional $4000/month to be split between all top validators = $18,825 per quarter
Public Endpoint: $175 per node for server costs per month = $5,250 per quarter
Public Endpoint Load balancer: $50 for server costs per month = $300 per quarter
Pulsar-2 team compensation back pay (One-time): Compensation for the work that the Pulsar-2 team has done so far ($6,900.00)

Total:
$67,275

Total with 10% Buffer:
$74,000

Carry Over
All remaining funds will roll over to the next quarter.

You can find the committee charter here

2 Likes
#2

I support this proposal. I encourage everyone to read the charter as it outlines the scope of the committee quite well.

2 Likes
#3

Questions

  1. What KPIs will you use to measure success of this committee? I need to know before I can for sure support this.
  2. Do you intend to supply the endpoints we (secretnodes.com) need to support this testnet?
3 Likes
#4

What are the 20 hours per week devoted to?

3 Likes
#5

Just curious, but isn’t 150$/h not a bit too high?

1 Like
#6

Nah, that’s consultant rates for a dev

1 Like
#7

I believe that in order to bring Secret to the same scale as ethereum, having a reliable testnet is crucial (ethereum has four public independent testnets, for example), so I want to make the stability of the testnet (and later betanet and staging net) a top priority, which means the KPIs the committee will track are as follows:

  1. The number of missed blocks throughout the testnet should be as low as possible. Ideally they should be as low as mainnet if not better and the uptime should be close to 99.99% (or at most 52.6 minutes downtime per year)
  2. The number of users asking questions about whether the testnet is down or is experiencing issues should decrease and ideally get close to zero. In the same spirit, the number of issues experienced in the testnet should be as close to zero as possible. This will require improving the tools and the documentation. Both of which will be done through extensive collaboration with the education committee and the development committee.
  3. The number of contracts deployed on testnet should increase; there will always be a need for localsecret (and I believe that this need will increase in the future with certain tools being developed), but I want to get to a point where every major developer will have a version of their application available on testnet. This will allow new users to join the Secret Network by seeing what it’s capable of, without having to spend money.

If you have any other questions about KPIs please do let me know and I will make sure to answer to the best of my abilities.

Yes, there will be two endpoints available, with 5 load balanced nodes each and run in separate data centres. The nodes will be provided by Stake or Die and Digiline.

I’m sorry I meant to write this in the initial post:

One of goal of the committee is it work and coordinate with all other major committees to start the necessary steps to improve the Secret Network as a whole. This committee will collaborate closely with the Dev Committee, the Education Committee and the Support Committee to make sure that the experience using the Secret Network is as good as possible, starting from the testnet.

I will help the Education Committee write documentation on topics such as:

  • Running and managing validators on the Secret Network
  • Creating a proper load-balanced setup for endpoints
  • Guides on how to find the right hardware and what to look for in a data center
  • Setting up a public-facing validator tracking website such as the one used by the Digiline Validator

I will collaborate with the Development Committee to write tools useful for the testnet and mainnet alike, such as:

  • The Snip-20 Faucet
  • The Snip-20 Transaction History Viewer
  • The testnet monitoring tool
  • Script to setup the previously mentioned front-facing validator tracking website
  • Scripts to quickly get a validator up and running in both testnet and mainnet
  • Scripts to manage a node more effectively and solve problems more quickly
  • Scripts to automatically handle the failover from one node to the next when a validator’s node goes down.
  • Writing necessary tools to get the Betanet and Stagingnet up and running

I will also work with the Support Committee in helping validators and node runners in the testnet and mainnet with problems they may be facing, whether it’s getting statesync to work or figuring out why a node suddenly crashed as is panic-ing when restarting.

In addition to that there will need to be some coordination with SCRT Labs to work on setting up the Betanet and later the Staging net. Betanet will probably need a lot of upkeep as it will be unstable. Staging net will need some work a few weeks before a major update, but will not have a lot of work most of the time.

Furthermore, I will be running three validators in three different regions. And I will be managing the Digiline front-facing endpoints for the network.

Frankly, some weeks 20 hours may not be enough :slightly_smiling_face:

No. This is on-par with the development committee and requires the same skills. As you can see there are a lot of development tasks that I will be taking on as part of the work I will be doing for this committee. This rate is on par with what top development firms pay most of their developers and is on-par with the rates that freelance developers are paid. Thank you for the question tho, I do understand your concerns

1 Like
#8

How about usage of the testnet in general? One of these things is measuring KPIs for node runners. I’d to see some better measurements but need to think and listen to hone in on better advice here.

This is good / we will need it if we are going to support it with the explorer.

Why are you going to be working on guides for node running? We already have people working on those docs, including how to find hardware, and I’m not personally comfortable paying for it via a testnet proposal which is unrelated.

I would prefer to see you work with existing tooling providers to get them to support the testnet vs supporting a new committee to create things that already exist like the snip20 history tooling, faucet, etc. I see a lot of fluff here and I’d rather see focus.

This seems like something I’d rather see from the support committee.

A testnet can be launched in 20 mins by an experienced person. We have one of the longest running load balancers on secret, and we do not put in anywhere near 20 hours a week on it. Load balancer would be similar amount of time. Forking existing faucet and launching, similar amount of time. I think I made a lot of assumptions about what a focused testnet committee would be, and I’m not seeing focus with the info you are sharing so far. For now I’m probably a no on this plan but I do think a focused testnet committee is important and I will keep an eye on the thread to see where the idea goes.

2 Likes
#9

Sure I can track testnet usage as another KPI!

Ertemann has specifically asked for the committee to have significant KPIs relating to the documentation. I decided to expand upon this by working alongside the Education Committee to write documentation on everything related to running a testnet and running nodes in general.

While I do agree with you with regards to the transaction history viewer being a relatively easy undertaking (I can just fork Trivium Node’s), I respectfully disagree about the rest.

  • The snip-20 faucet does not exist already to my knowledge, if you know of any please do let me know and I will work with the developers to bring it to testnet. So unless one exists, I will have to work on the frontend, backend and contract for it.
  • The testnet monitoring tool is something I discussed with the dev committee and it will entail having a web-based gui where you can see all validators on testnet, what their hardware looks like and what it’s usage is, what their stats are (numbers of missed blocks over time, number of blocks proposed, etc.). This is completely new development where I will have to do both the backend and the frontend development.
  • The script to setup a front-facing validator website isn’t “fluff” to me, but rather a useful tool to improve a validator’s visibility on the network, this is especially useful for smaller validators.
  • The script to get a validator up and running quickly and the script to manage a node are also not fluff to me, this will allow new and not-so-experienced validators to join the network
  • Similarly I wouldn’t call the failover script fluff, but rather a useful tool that every big validator will be able to use to save time when something goes wrong with their node (which judging by the node-support channel on discord and the validator’s chat on telegram happens very often)
  • Writing the tools for the betanet and stagingnet is not something that has been encouraged by the community but also absolutely necessary.

I can remove this from the proposal, but I think that in order to achieve the most stable testnet this may be necessary.

The 20 hours are NOT spent only to setup the node and the load balancer. A lot of the time will go towards tool developments and support and collaborations with other committees, as you can see from my answer above. I’m not sure why this was misunderstood, I may need to change the way the answer above is written to make it more clear.

#10

First and foremost, I 100% support the funding of a testnet committee, it is clear that there are still some nuances to work out with scope and KPIs but i am sure we will figure that out. To help with the scope i will explain my involvment and that of the education committee with the dev docs in the post below.

  1. The education committee is NOT in charge of the developer documentation, nor is developer education in scope of the education committee. The education committee focusus on making comprehendible information on the network, its dapps and tech stack available to end users and other interested parties. Although developers might use our info to learn about the tech stack (think of tech overviews on docs.scrt.network with lower depth) it will not create any guides on programming and developer tasks.
  2. My involvement in docs.scrt.network is separate from the education committee, i am doing this mostly on my own time and because not a lot of people are willing to step up to connect the dots among the several parties and other leads were too busy to take it on at that moment in time. My involvement includes getting people to keep connecting, gather feedback and info from community members, write non-developer tech and ecosystem overviews and helping thinking out a structure for both the actual docs and the parties/funding needed for that. I have been doing some of this work on hours from Proposal #68 but most just on my own time next to the hours i spend on education.
  3. I think developer documentation for the testnet is IN scope of the testnet committee! we want a scenario where every party involved in a certain piece of infrastructure, tooling or code in the network is also reponsible of the documentation on it. Having 1 party constantly do all documentation for everything in the network is NOT an option and not one we should even consider. This means this is OUT of scope for the edu committee.
    The edu committee however WILL work with the testnet committee on creating a guide on using the testnet (so not devving with it) and where to find info about the structure of it seeing as the user base of that content would be end users and members interested in the secret ecosystem.

Note: If people think more dev docs or at least the coordination of the dev docs should fall on the education committee then i am very open to discuss this. Our proposal is currently on the forum and if this would become in scope we would have to alter our prop and think about how many hours and resources that would take. currently only some of the content on docs.scrt.network can/will be in scope for the edu committee.

I think @dylanschultzie is working with Scrt labs on getting funding for rebuilding the documentation, for now all the work on this is just by pople who care and are willing to spend non/barely-funded hours on it. This proposal would only cover making the docs good enough again and the restructuring of it. keeping it up to date will eventually fall on the parties who handle the code and infrastructure of the network, highlighting again why testnet docs should fall on the testnet committee.

Note on the Prop:

  • Can we get a set of measurements which you will use to decide on the node providers payment? I am thinking of something similar to how shade higlights which nodes they include in stkd-scrt.
  • Seeing as you will be developing a lot of tools i think the 20 hours is reasonable. wether the same time would be needed next funding period is a different question.
1 Like
#11

Definitely. The most important thing I will consider is the stability of the validator; a validator that misses a lot of blocks or has a lot of downtime will not receive any rewards, but will still get their server costs covered (up to $175/month) upon showing an invoice. The metrics work as follows:

  • Any validator with 10% or more missed blocks will not receive any rewards
  • A validator with 5% or more missed blocks will receive half of the rewards
  • All missed rewards will be split equally amongst the three validators with the highest uptime

Yes, I think that in the next funding period the amount of hours will probably get a lot lower unless the community asks for more tools to be developed

#12

I’m still a no on this proposal until It becomes less convoluted and more focused but I am just 1 person and a yes on needing a focused testnet. From here I’m just going to listen and hope.

#13

You bring up important points and there is no reason why this committee and charter shouldn’t be modified to match the advice and consent of the community.
That having been said, I think it is important to look at this as a problem that needs to be solved and a willing and capable party who is committing time and resources to solving it. That has value above and beyond the baseline cost and time of executing the underlying tasks required.
I also think it is really important not to underestimate the actual work and cost required here to go from 0 to 100%. As technical people we tend to see something from the position of, I know how to do something and what it costs for me to do. The problem is that we generally underestimate on both fronts. If Slabs was to do this for instance, it would be way cheaper and less time consuming because of all the expertise and systems in place. But they aren’t raising their hand to do it. For Danny M to lead this he needs to build up from the scraps he has access to and do the work needed to glue them all together. He 100% should not rebuild tooling that already exists, but actually making that existing tooling work in testnet takes time and effort.
So, for instance, the docs available need to be tailored to work with testnet, in a way that is maintainable and not done as an after thought. In other words it needs someone to work with the maintainers to get testnet support and that should be compensated.
Kind of rambling, but at the end of the day we all want a testnet that just works. One that we don’t need to think about constantly and a place to point to when something goes wrong. Builders need to be freed to just build with confidence that a working app on testnet equals a working app on mainnet. That takes project management, coordination etc etc.

4 Likes
#14

I agree with @moonstash said on this.

Something just doesn’t fit with the proposed hours and hourly rate on this proposal. Most of the tooling that you describe is already available on mainnet. (xiphiar for the Snip-20 viewer, testnet monitoring is done by the testnet block explorer, scripts and tutorials are readily available and are being improved by other commitees).

Also crossfunding other part of the projects by using the testnet committee budget doesn’t make sense, you’re not doing testnet stuff then and whole point of having committees in the first place is a bit diluted.

As for the mainnet/testnet support I also asked around in this thread here if there was an interest in having a separate node support ((Feedback) Need for node/validator operator support - #5 by StoicNorth) and after ertemanns thoughts on this I agree with him that this should part of the support committee, docs and a helping hand from other testnet/mainnet node runners.

I think that the scope of this committee is bit too far and numbers a bit too jacked up, when that’s resolved then it’s a yes.

#15

The only one that’s on mainnet is the snip-20 viewer, which as I said is going to be extremely easy to move to testnet (I doubt it would take more than a few minutes). Everything else is not available and is new development.

The monitoring tool is not just a block explorer, in fact it’s not a block explorer at all. secretnodes.com can handle that part because it’s doing it very well already.

I’m not sure what you mean by this exactly, can you please clarify which projects you think are getting crossfunding?

Scripts and tutorials are out of scope for the other committees, as was mentioned above by ertemann from the education committee.

I’m not comfortable with lowering the numbers.

If I halve the time worked on this everything will take much longer than I expect it to and I would personally consider this a failure for the committee.

Similarly I’m not ok with lowering my rate, those are the standard rates for a developer, and other committees are getting the same rates for similar work.

#16

I have more of an issue with the hours than the rate per say.

Why create a monitoring tool at all? It would be way better to just create, update, or audit any existing guides for using prometheus / grafana with secretnetwork / tendermint nodes. The tools already exist and thats what most of the professional shops use (it seems) and that is certainly something that has more resources and people who can help with, than a custom tool you have.

Personally I think the best testnet proposal is going to focus on

  1. Maintaining a testnet
  2. Getting testnet support from existing tooling providers that have tools for mainnet.
  3. Slabs & community buy in
  4. Simple doc version that replaces things like binary links and mainet references to testnet in the existing guides. (Very easy stuff.)

None of this calls for 20 hours a weeks, every week.

You have a different vision and larger less focused scope than what I outlined which is your prerogative. I’d still much prefer a lighter footprint and more focused approach. Pitch that and I’m a yes.

2 Likes
#17

I think that there’s already a Prometheus server in every node

2 Likes
#18

Yes. But that does not report some of the system information it appears they want to track, you need more than that tendermint telemetry but still there are plenty of existing guides on how to do that and they can be applied to secret as multiple people have done already.

#19

By crossfunding I mean that you suggest a lot of stuff that would be outside of your scope for testnet and try to include it in your testnet committee funding, which instead should come out of other pools (education, development, documentation committees).

1 Like
#20

Sure, if this is what the community prefers I can change the proposal to only focus on this. This would lower the hours to around 4-5/week at the most. I personally don’t think this is the right decision but I’ll go with what the community wants :slight_smile:

1 Like