Fungible Token Migration (Snip-20 --> Snip-22)

[the-dusky]

The Point - Why, What, and a little How:

The point of this post is to gauge interest in a community led initiative to migrate all SNIP-20 tokens to SNIP-22s. If there is enough interest I propose we pass a community spend proposal to compensate the execution of this migration.

Preface - Give me a break:

I may make some technical errors here in describing aspects of SNIP-20s and SNIP-22s. This is due to the immateriality of the details I may get wrong, as well as time constraints for fully research those details. Unless the error is relevant and material, I don’t want to focus on it for any discussion that follows.

SNIP-20 Background - The Problem:

The initial SNIP-20 standard made possible fungible tokens that have privacy. No one but the viewing key holder can see the balance or the history of transactions. This standard is the foundation on which much of what we think of as Secret Network has been built. But, SNIP-20s have a major flaw. The transaction history is incomplete, both in the sense of lacking certain transactions, and lacking critical information for the transactions that are included in the history. Essentially all you can see is movements of funds and the addresses involved, along with an ID that is an incrementing number added to each transaction. What is missing is any mint or burn transaction and, critically, neither the timestamp nor the block of the transaction are saved. The consequence of this lack of information is that unless you create a viewing key prior to a transactions, and save that viewing key, and all subsequent ones, if you ever create a new one, there is no way to recreate the historical events. (There are various ways in which you can come close, some involving community organization, that I can discuss with anyone interested, but they all require a lengthy explanation).

The viewing key is important here because it highlights the problem as I see it. You can use an archival node and preserved viewing key(s) to derive the block of each transfer. The problem is that A) it is resource intensive to do this, both computationally and time and effort-ly, and B) viewing keys were not “sold” to the general Secret Network user as something they absolutely needed to save.

I hope it is clear why I mentioned the possibility of immaterial errors in describing this stuff at the beginning. Just framing this issue is time consuming and risks confusing or seeming like a mere “technical issue” that is too in the weeds. But the wonkiness of the topic
belies its importance.

SNIP-22 Background - The Fix:

More recently the SNIP-20 standard was updated fixing the issues I outlined above. This means that a user of SNIP-22s can recreate their history precisely with minimal time and effort. SNIP-22 also offers a number of enhancements beyond this issue that make migration that much more valuable. But I won’t list them here, just trust that they are much better.

Back to the Point - Is this important?:

IMHO, migrating all SNIP-20 tokens is critical for Secret Network to move forward on a solid footing. To outline one reason for this, if we want large scale trading on Secret Swap we need professional traders to know they can use a tool to calculate their tax exposure, or just track their history for insights after the fact. Again IMO, Secret Network is about privacy, not about opacity. It is about control for the user, and that can only exist with information being available.

The problem with the solution - How hard is this?:

This is hard. Here is a list of implications of doing this:

• New contracts will need to be deployed for most bridge tokens, LP tokens, swap pairs, staking contracts
• We’ll need to write a nice generic migration contract that lets people deposit their coins in exchange for minting new ones on the new token (easy to write, but gotta get the configuration right because we have several dozen tokens)
• All token holders will need to be made aware of this, actively migrate all their base tokens (maybe we can provide a fancy LP migration contract? IDK)
• We will need to build a UI allowing people to easily migrate their tokens, make new viewing keys, migrate liquidity for LPs, etc
• The bridge will need to either continue supporting pulling out from older contracts, or offer the migration to holders of old tokens before allowing them to pull out of SN (latter might be annoying to people but probably a lot easier to implement)
• maybe more things that haven’t been considered yet, like token symbols and things like that.

Conclusion:

To summarize in reverse order:

• This is hard to do
• This is important
• SNIP-22s are private, but give the user their transaction history
• SNIP-20s are opaque even to the user
• I know the explanation is long and boring, and I may have made technical mistakes, but lets focus on the point

Next Steps:

If this gets enough interest, I want to move it to a signaling proposal on chain. It has been suggested to me that we should request that validators don’t vote on this. That we prove interest by asking only individual addresses to vote. I see the point here, but I worry that A) most people don’t vote, maybe never had B) the importance of this may be much greater than the interest in fixing it. If it is the difference between SCRT being a top 50 coin or not, but we don’t get enough votes because people aren’t paying enough attention, what then?
Anyway after that I would like to work with the leads and the foundation to identify parties who can work on the solution and then pass a community spend to pay them to do it.

Thoughts?

[the-dusky]

To clarify my viewing key remarks. New viewing keys can’t be used to query blocks prior to the block it was created on, they do not stop you from being able to see full history of transfers. The comment about needing to save viewing keys if you want to use an archival node to pinpoint transactions. The way you would do this is to search for the public contract execution logs of any snip-20 contract. You could then get your balance before and after that contract execution. The delta between those balances is the amount of the transaction. You can then match the datetime and block height info to the SNIP-20 transfer history. Even if you have consecutive transactions for the same amount you can use the order of transfers to derive which is which. BUT… but… This is a huge pain in the ass, and it requires you to have a viewing key that is valid for each block you want to query, and there are most certainly edge cases in which one algorithm wouldn’t work for all accounts. I have used this method and it sucks…

[Frank_L_Right]

You make a good point about the importance of transaction history. Could we test Snip-22s on testnet, including the process for looking up transaction history? And perhaps instead of migrating all the older SNIP-20s at once we might start by implementing them with newer tokens as they get listed on SecretSwap. We need to consider the userbase experience.

[reuven]

@Frank_L_Right We have already done so - All BSC bridge tokens, Some Ethereum bridge tokens, Cashback, and sXMR include implementations of the improved history features
You can see the information in the new interfaces by performing some operations with those tokens, and then using btn.group’s online contract interface to build the queries described in the SNIP-21 specification document.

(SNIP-22 is another extension on top of SNIP-21 but the distinction isn’t relevant to this discussion. You can find the full specifications in the SNIPs repository)

EDIT/NOTE: We can call this SNIP-22. The reference implementation implements both so an upgrade will include both. They’re different standards but for the purposes of this discussion they can be used interchangeably, and i like saying SNIP-22 more

[Brendan]

Sounds like this absolutely needs to occur if we’re serious about attracting larger traders, investment firms, etc. I would support this and understand it’s a huge undertaking. It’s a bit scary, but we’d be better doing it earlier than later.

[the-dusky]

I agree. As I worked on a SEFI staking history project, and then my own taxes, I realized how hard things are right now. And my volume is low.

[mumuse]

I don’t think it’s what is holding back larger traders from joining the ecosystem, but I also think this would be nice to have and agree in supporting this.

@the-dusky

It has been suggested to me that we should request that validators don’t vote on this

This just sounds like cope by people that have negligible amounts and want their vote to count as much as others who are more invested in the ecosystem and I hope it is not taken seriously.

@the-dusky

A) most people don’t vote, maybe never had B) the importance of this may be much greater than the interest in fixing it

Agree with this completely

[cow.level]

As time goes on and more people enter the ecosystem, it will only become more of a burden to migrate to the new contracts.

I think we should brainstorm what features might be missing from the SNIP-21/22 specs, and then push forward on this sooner than later.

[the-dusky]

@mumuse

This just sounds like cope by people that have negligible amounts and want their vote to count as much as others who are more invested in the ecosystem and I hope it is not taken seriously.

To be fair, I think the idea of validators sitting out is a way of attempting to gauge true interest / support vs just the support of the 50 highly engaged validators. Large holders would be able to have the normal outsized impact. But… I worry that it may have the affect of appearing to be less needed or wanted than is the reality

[reuven]

Just to put it out there, it could also just be framed as a product decision by the developers of various products, but it’s disruptive enough to ask whether or not it should be voted on first by the wider community. Maybe there’s a compromise to be made as well (e.g. make the decision between all committee leads (and validators?), which presumably would know what’s best for the chain)

[the-dusky]

To maybe help illustrate the point, here’s the info you currently get form sSCRT running a transfer_history query:

{
  "transfer_history": {
    "txs": [
      {
        "id": 32422,
        "from": "secret1pe96yhtmwqxxxxwaqcjkv4cylwcmn5kse5",
        "sender": "secret1pe96yhtmwqxxxxxxxwaqcjkv4cylwcmn5kse5",
        "receiver": "secret1t85jewrnlxxxxxxxfnfh5puzthd0lxzwp6ly",
        "coins": {
          "denom": "SSCRT",
          "amount": "13619424"
        }
      },
      {
        "id": 21363,
        "from": "secret1pe96yhtmwq5zxxxxwaqcjkv4cylwcmn5kse5",
        "sender": "secret14zv2fdsfwxxxxx2ushp4c4jr56ysyld5zcdf",
        "receiver": "secret14zv2fdsfxxxxxs2ushp4c4jr56ysyld5zcdf",
        "coins": {
          "denom": "SSCRT",
          "amount": "331380576"
        }
      },
      {
        "id": 8233,
        "from": "secret1pe96yhtmwq5ssssssmwaqcjkv4cylwcmn5kse5",
        "sender": "secret1pe96yhtmxxxxxxaqcjkv4cylwcmn5kse5",
        "receiver": "secret14zv2fdsfwqzxqt7sxxxxp4c4jr56ysyld5zcdf",
        "coins": {
          "denom": "SSCRT",
          "amount": "500000000"
        }
      }
    ]
  }
}

vs

{
  "transaction_history": {
    "txs": [
      {
        "id": 88730,
        "action": {
          "burn": {
            "burner": "secret12p4f38e0qajhrlzrexskskh3qajlydrj2mqa8s",
            "owner": "secret12p4f38e0qajhrlzrexskskh3qajlydrj2mqa8s"
          }
        },
        "coins": {
          "denom": "CSHBK",
          "amount": "37480982354"
        },
        "block_time": 1630424678,
        "block_height": 4966850
      },
      {
        "id": 58059,
        "action": {
          "mint": {
            "minter": "secret1tgagwaea268dkz7255mcau28z8qs08lnllgecm",
            "recipient": "secret12p4f38e0qajhrlzrexskskh3qajlydrj2mqa8s"
          }
        },
        "coins": {
          "denom": "CSHBK",
          "amount": "19830982354"
        },
        "block_time": 1626864849,
        "block_height": 4377104
      },
      {
        "id": 44345,
        "action": {
          "mint": {
            "minter": "secret1tgagwaea268dkz7255mcau28z8qs08lnllgecm",
            "recipient": "secret12p4f38e0qajhrlzrexskskh3qajlydrj2mqa8s"
          }
        },
        "coins": {
          "denom": "CSHBK",
          "amount": "17650000000"
        },
        "block_time": 1625236432,
        "block_height": 4105203
      }
    ],
    "total": 3
  }
}

[Frank_L_Right]

Perhaps don’t call it a vote, make it a poll. There’s no harm in that. Otherwise if the votes go different ways then there may be more conflict.

[Frank_L_Right]

Currently when you do a trade on SecretSwap, you get the pop-up, which colors are rather hard to read with white-colored font, and it disappears after a fairly short time. A short-term aid for users of SecretSwap before Snip-22s are implemented, would be to add a script where you can offer an optional receipt, either downloadable for the user or in a new window, or as a pop-up, to facilitate the recording of the transaction.

[the-dusky]

@Frank_L_Right

would be to add a script where you can offer an optional receipt, either downloadable for the user or in a new window, or as a pop-up, to facilitate the recording of the transaction.

This is a good idea. Not sure how hard it is to implement as you would want it for all txs

[Ewais001]

Is there any way this could be rushed over the next week or two while SecretSwap is nerfed due to the SEFI contract being inactive?

[dylanschultzie]

That makes a lot of sense to me.

@reuven if you need any dev support, I’m happy to contribute to the effort.

[Xiphiar]

Another reason to do this sooner than later, Keplr seems to have reset the transaction encryption keys for secret-2, so you can no longer decrypt old transactions to help you figure out what happened. I was planning to use this to help build a more complete history of user’s SNIP-20 tokens but that is no longer possible. This will probably happen again for future updates.

[oljo]

I’m still reading and maybe should not pip in without fully reading everything but, could this issue not be solved by a service ? If I need to know specific transaction history for say a taxable event , could I just use a dApp to figure it out?

I agree with an above comment about maybe this was done for a reason.

[oljo]

@the-dusky

The consequence of this lack of information is that unless you create a viewing key prior to a transactions, and save that viewing key, and all subsequent ones, if you ever create a new one, there is no way to recreate the historical events.

I think the key word there is “unless”

So I actually would not need to hire a service , I just need to be accountable and diligent with my blockchain events .

[mohammedpatla]

I think this should be taken into consideration as well. A new Implementation that’s moving away from Viewing Keys. This should be SNIP-24(or25) and might be a better point to migrate towards.

Query with a signed permit instead of a viewing key