The Community Infrastructure Proposal

Hi all! I’m following up with a discussion from our infrastructure call today (Sep 8th 2020) regarding a plan to provide community seed and sentry nodes. Community sentry nodes would meaningfully increase the cost of an attack on any nodes that participated by adding more points that need to go down for the connected Secret Nodes to miss blocks. Community Seed Nodes would help to ensure anyone using the seed node could easily keep their address book up to date so they always know what peers to connect with.

What are Sentry Nodes?
On the Secret Network, a Secret Node can be attacked using the Distributed Denial of Service method. Secret Nodes have a fixed IP address and it opens a RESTful API port facing the Internet. Both the fixed IP and REST API are attack vectors that if overwhelmed can cause your Secret Node to miss blocks or otherwise be unable to communicate stably with the rest of the network.

What is a Seed Node?
Seeds are the first point of contact for a new node. They return a list of known active peers and then disconnect. Seed nodes are similar to regular full nodes except their only actual function is to gossip peer and maintain an up to date address book of nodes on the network to communicate with. Connecting to a seed node is a standard way to ensure your node maintains a healthy connection to the rest of the network by ensuring the address book remains up to date.

What would the Community Infrastructure Proposal offer?
The proposal would cover the costs of 3 SGX compliant nodes, expenses to colocate the nodes, and any time involved in maintaining them. Any community member would be able to request being added to the list of peers for the sentry nodes and in turn augment their infrastructure by adding another layer of sentry nodes, or be a resource to those who cannot run their sentry nodes. Furthermore at least one of the Sentry Nodes would also be in archival mode to ensure a complete copy of the blockchain (vs a pruned version) is maintained at all times for public use. These nodes would be maintained indefinitely (as long as / Secret LLC is sustainable) and no further requests for funding would be made by us in regards to this offering as long as the equipment works for the specified task.

Community Spend Amount
10,000 SCRT

Notes: While I do think this community infrastructure is important I want to make sure anyone interested in this knows that while we will maintain the nodes as a part of our core infrastructure, we still recommend and encourage node runners to have at least one of their own sentry nodes.


Thought I’d chime in with my 2 cents on this idea. I’ll start by saying that this opinion represents no official position by Enigma or anyone else. Just my personal feelings.

Anyways, I’m not a huge fan of something like this.

Protecting against DDoS isn’t an easy task. It’s usually a cost-benefit equation. i.e. how much would I have to pay to generate enough traffic to take down a target or disrupt a service. I feel that running centralized sentries would only increase resiliency linearly (e.g. you now need to DDoS 3 machines instead of 1), while creating an exponentially attractive target. Especially if 33%+ of the network grows to depend on it which could cause stability concerns for the entire network.

In my opinion investing in solid DDoS protection solutions, either from your cloud provider, ISP, or external device would provide a higher benefit than something like this.

Furthermore, if we’re trying to reduce node runner operating costs, I would suggest a more efficient use of these funds as bringing solutions like Iqlusion’s TMKMS to production to reduce the need to run the validator itself as a full node w/ SGX

1 Like

Thanks for chiming in cashmaney.

I think you are bringing up great points but I’d like to point out the infrastructure proposal as is, in my opinion, still would meaningfully help the network. It’s just one piece of the infrastructure puzzle, it’s not being positioned as a save all.

Re: single point of failure or attack issue can be addressed by whitelisting IPs who can use a sentry node, having that sentry node connect directly to another, and denying all non whitelisted traffic. Mitigating DDOS or the impact of legitimate traffic overwhelming a node is not trivial but with someone working on these things as a core responsibility for a team or person, I think the value add would be worth it. Also as a reminder it is normal And even best practice in cosmos based networks to have validators share or otherwise link their sentry nodes and this is essentially the same thing but would be maintained as public infrastructure.

Responding to the Iqlusion’s TMKMS comment, I think it’s a great idea. I’d like to see TMKMS and even threshold signers be available to everyone and they are 2 things that would make the network more robust. I would support an initiative from someone to build and maintain either of them (I approached you about threshold signers a while back), as long as they had ongoing updates.

I would like to see more mainnet validators comment on whether they would personally utilize such a service and if it would make a meaningful difference to their cost structure. Ideally this could be a difference-maker for node runner participation. Can you promote this thread a bit more heavily on the Secret Chat and elsewhere?

1 Like

I’m inclined to agree that centralized sentry nodes do not improve the resiliency of the network. A validator that connects to a sentry node over the public internet is exposing it’s IP address to that sentry node, and trusting that sentry node a) not to attack it and b) not to leak it’s IP address. If the validator’s public IP address becomes known, then the validator is vulnerable to a DDoS attack. These nodes also become a single point of failure for any validator that chooses to use them exclusively.

In order for the sentry architecture to have maximum value, validators need to be run in private IP address space, and communicate exclusively with trusted sentry nodes under the operator’s control. The resiliency of an operator can be improved with private links to other operator’s sentry nodes, ideally over private interconnects so that these paths are not vulnerable to DDoS.

The standard sentry architecture is a challenge here, due to the high cost off operating sentry nodes. For this reason, I see the appeal of community nodes available to all validators. However, I think the network’s resiliency would be better served with private relationships between operators to share sentry nodes. For example two operators who know and at least to an extent “trust” each other, could each set up one sentry node, and then each connect their validator to the other’s semi-trusted sentry. In this way, there is no central point to attack. Rather than a hub and spoke model, we would see a distributed web of trust. A network of bare validators operating without sentry nodes might actually be more resilient than a network of validators depending on a centralized cluster of sentry nodes.

I’m not sure how TMKMS has been brought up in this discussion. Validators need to be full validating nodes. TMKMS allow the validator’s private key material to be moved out of software onto an HSM. In order to use TMKMS with an HSM, the operator needs to be running on bare metal that they control, as the HSM needs to be physically plugged into a USB port on the TMKMS machine. TMKMS is excellent, but I think it’s orthogonal to this discussion.

1 Like

Just to be clear if the majority of people who comment here disapprove then I have no intention of trying to proceed with such a plan as I do already have the hardware I personally need for mainnet. Thanks for all the good feedback so far!

1 Like

Wouldn’t the Sentry nodes still provide some added level of security if smaller node operators use these sentries in addition to running 1 of their own? Instead of running 2 sentries they could run one of their own, plus the community run sentry? It was talked about on the call that it absolutely shouldn’t be used as the ONLY measure of protection, but could still be seen as an added layer?

Base benefits

  1. Operators who already have sentries could layer with this.

  2. Operators with only 1 sentry node could augment their setup.

  3. Operators who use a VPN to obscure their validators true IP could augment that setup by connecting to community sentry nodes.

There are clear undeniable benefits for anyone who used it properly. There are also risks people will misuse it and over depend on it.

Cc @Brendan

Exactly. This was my understanding form the call. It’s not meant to be something to be leaned on, but something to augment. This is why I was in favor of the idea, but I’m not super technical in terms of how the setup would work. 1 sentry, 1 backup, and 1 validator is a lot already, and I’d assume 2+ sentries would be best practice.

Since we are wanting people to have a sentry node of their own even if they are using the 3 proposed sentries, would the following be a good compromise? Instead of having a widely used 3-sentry hub that has the problems already mentioned, do we have people share their sentry info? Then the people who can only afford one sentry have the exact same costs, because they are still only setting up one sentry, but now instead of all of them using the same 3, they are all linked to all the other sentries of people who only could afford to set up one. So, same cost, but a whole network of sentries, so more decentralization, more points that would need to be attacked, but at the cost of trusting that the other sentries won’t be bad actors

Sounds like we’re getting closer to private peering.

From Figment’s disclosure

Our sentry nodes are operating in North America, Europe and Asia. In addition to public-facing sentry nodes, we operate a private peering node via AWS (Amazon Web Services) that peers with other capable private validator nodes via VPC (virtual private cloud) peering.

Here’s also some great info from the reviews of different validators, well it’s volunteered info mostly, but some nice insights into what some of them are doing. eg one has inbound and outbound sentries as usually, and additionally, outbound only sentries, with inbound traffic dropped at the firewall.

1 Like

Polychain’s Tenderseed may be relevant too

Tenderseed: A lightweight Tendermint Seed Node. Seed nodes maintain an address book of active peers on a tendermint p2p network. New nodes can dial known seeds and request lists of active peers for establishing p2p connections. This project’s lightweight node maintains an address book of active peers, but does not relay or store blocks or transactions.

1 Like

Being a total newb and non technical type, I would like to set up a personal validator without stakes till i have everything bedded down. A group of sentry nodes to link into at the start would assist me in the setup whilst i learn to develop sentry nodes and understand the environment. I think it would be a good idea, maybe practice to require at least 1 sentry when staking but the ability to list 4-5 sentry nodes that can be available to be added should give good fail over. Can we use a peer exchange reactor to propagate Sentry IP Addresses to the Validator nodes so if one gets attacked the Validators move to others till that one can come back online? (This is a thought bubble as i have only read enough to be dangerous :slight_smile: )

I decided to just pursue private relationships and pay for the hardware out of pocket instead of pursue this proposal. Thanks for the feedback.