(Community Pool Proposal) DataHub - Scaling Secret Network Infrastructure

Secret Governance
1

Abstract

DataHub (DataHub: The Web 3 Gateway - Figment - Building Web 3) helps developers build applications on the Secret Network without having to manage blockchain infrastructure or develop custom middleware. We make sure they can focus on what they do best: developing amazing software.

DataHub will scale its Secret infrastructure to support more Secret DApps and prepare for the network’s growth.

Project Description

Figment is looking to scale its Secret Network infrastructure powering its DataHub services.

As part of the DataHub API suite, Figment hosts Gateway Access which allows developers to submit transactions and perform read queries on the Secret Network.

Developers are able to sign up and access live blockchain data and submit transactions without having to worry about maintaining blockchain infrastructure or doing custom integrations. We provide reliable access to the Tendermint RPC and the Secret REST API as well as providing the gateway to submit on-chain transactions. Both the latest mainnet and the latest testnet are supported, and an archive node option will be available to users as part of the subscription plans.

While Figment already operates the Secret DataHub service, we have been running into provisioning issues around Secret nodes which are more complex in nature than usual nodes given the need for SGX-enabled servers. This grant will allow us to provision enough servers to support Secret Network’s growth throughout the next year as to avoid any interruptions to the service for scaling startups.

Deliverables & Milestones

  1. Figment will triple its Secret node infrastructure over the year
  2. Figment will allocate part of the time of a DevOps engineer to the optimization of Secret APIs & infrastructure

The milestone unlocking the second half of the grant will require Secret DataHub to process over 10m requests/day for at least one day.

Ecosystem Value & Success Metrics

Figment DataHub already supports some of the leading organizations in the Secret ecosystem such as Keplr, Binance, and Enigma. With this project, Figment wants to increase developer adoption on the Secret Network.

To track the success of this project, Figment will track the following metrics:

  • of requests made via Secret Gateway Access
  • of Secret DataHub Subscribers

Budget

DataHub currently operates a mix of 12 bare metal servers and azure instances which cost about $6,000/month. Tripling that capacity will take us well over $200k/year in Secret node infrastructure. To support this growth, we are looking for a grant in SCRT tokens.

The grant includes the continued and scaled hosting of the Secret DataHub service:

  • $50,000 USD equivalent in $SCRT tokens upon acceptance of this proposal

About Figment

Figment’s mission is to build a better Internet by increasing usage of the next generation of Proof of Stake blockchains. These efficient blockchains give people greater control of their data, more privacy, and increase financial inclusion. By eliminating intermediaries they also limit the power of centralized data monopolies, rent seeking financial institutions and anti-social algorithms.

We bring our mission to reality by helping investors stake their tokens; earn yield and participate in securing the blockchain. Our Hubble and Prime applications as well as our governance expertise allow token holders to analyze, monitor and make informed decisions. We build Web 3 developer communities via our Learn education program and then make it simple for these developers to launch applications and manage smart contracts via our DataHub platform. And finally by building and operating The Graph based indexers we enable the efficient search and querying of blockchain data.

Together, we can build a better Internet.

We welcome feedback on this proposal and are looking forward to working with the Secret Community to accelerate the adoption of the protocol.

18 Likes
3

Figment has been the powerhouse behind most of the big apps on the network. Big thumbs up from me as well

10 Likes
4

This is awesome. I am in support of this one, this is intelligent.

5 Likes
5

Dear Figment Team,

We have been patiently awaiting this post.
This message is to signal our support for this proposal! :grinning:

Best Regards,
Dan

6 Likes
6

Full support, benefits all.

5 Likes
7

Thank you! Very excited for this.

2 Likes
8

I would be open to a higher grant, if figment does cluster in multiple regions( geographically )and with less downtime.

6 Likes
9

After seeing the Anons launch, I would say this is absolutely critical to have a professional team at the helm, who give out metrics for their success. Please upgrade the request to better support the network.

4 Likes
10

YES, Figment to the rescue!

11

Great work Figment/Yannick :+1:

1 Like
12

Given that every launch Secret has had so far has been a disaster, I’d like to hear if you guys had any larger plans in mind in exchange for an increased budget?

Tor mentioned in the other ongoing infrastructure post that we are not going to have unlimited opportunities to make a first impression - I couldn’t agree more. The Anon’s launch was a massive missed opportunity to make a good first impression on a brand new audience. We botched it.

If we can spend more now to ensure that this doesn’t happen again going forward, we should do it. How many more opportunities can we afford to lose due to poor preparation?

I’d be interested to hear if Figment have anything more that they could offer for an increased budget.

2 Likes
13

WhisperNode supports. Simple as that. Throw as much money at this problem as it takes.

4 Likes
14

Firstly. this is a great proposal, and I support it.

My question about this is if these endpoints will integrate automagicklty with the keplr wallet extension, or will the end user/dApp developer have to do some manual work to use your infra, instead of say XYZ’s infra, and potentially have the juggling left up to them.

Has there been any discussions around providing some manner of load balancing between them all to make it as close to invisible as possible for the end user ?

15

Hey, Figment’s API is currently used by the Keplr extension, so they wouldn’t need any further integrations to continue providing that support.

There have been discussions around this idea, and it’s still being played with. It would require some sort of authority to run said load balancing integration, on top of all (or whichever) API providers to buy into the load balancing system.

1 Like
16

Hi everyone, thank you for the overwhelming support! Based on the feedback we could bump our ask to $100k and significantly increase the capacity.

We are planning for a multi-region offering in the first half of 2022 for our top protocols and could include Secret in the initial rollout. I can’t promise an exact date but it’s one of our top priorities.

If this sounds good, I will up the ask.

Also note that we have added 6 bare metal nodes to our pool today and are now set up with a hardware provider who can increase our capacity in a matter of hours instead of weeks compared to now.

8 Likes
17

I support an increased ask. Figment has proven itself to be a pillar of the Secret Network infrastructure.

5 Likes
18

Done, multi-region support love <3

And higher capacity requested, is more of a network need. Also, optimization on load balancing so that API has higher uptimes is needed.

4 Likes
19

Would love to hear a more refined proposal at the $100k price point.

Timeframe & specs would be appreciated.

Looking forward to getting the ball rolling on increasing capacity!

2 Likes
20

Thank you for the additional information, great to hear of the improved logistics.

Some questions that came up during a discussion on Telegram:

  • Are you planning to scale both physical hardware and cloud? Or mainly physical and cloud when peaks are anticipated?
  • When are you planning to go on-chain and would bringing that forward contribute to more capacity sooner?
4 Likes
21

Thanks for your support everyone!

We are scaling mainly physical hardware in our data centers and only use Azure if we run into short-term scaling issues.

We are ready to move on-chain if we have consensus! We have pre-emptively doubled our hardware server count last week and will increase it again once the proposal passes to provide leeway for projects launching.

9 Likes