What is the best way for importing a large data set into secret state?

Hi,

I’m trying to import a large data set (100000 rows and 20 columns integers/str) into the secret state. Does anyone have any better idea other than converting the data into text format and push it into “String” in order to pass it to the secret contract? Even in that idea, I need to split the original data set into several parts and to run the add-data contract for several times because “String” seems to have the maximum limit length.

2 Likes

Hey @KanaGold – Just as an FYI, we expect to have some additional documentation and FAQ for working with state efficiently up soon. To be candid though, we’re still testing and learning the limits of the system ourself. But I will raise the question about string max length limit and see if we can get any guidance for you.

4 Likes

@ainsley I have been testing the state capacity by myself, and have figured out several things:

1 serde_json does not work as a part of a secret contract.
2 even though setting the gas limit at sufficiently high, around 6*200 data set seem to be the hard limit in the current test environment.

fn add_data() {
        let mut index:U256 = U256::from(0);
        let mut datas = Self::get_datas();
        for i in 10..20 {
            let height:U256 = U256::from(i);
            for j in 1..9 {
                let age:U256 = U256::from(j);
                for k in 0..2 {
                    let sex:U256 = U256::from(k);
                    for m in 0..1 {
                        let rand10:U256 = U256::from(m);
                        let weight:U256 = U256::from((f64::from(height.as_u32()) * 0.3 + f64::from(age.as_u32()) * 0.1 + f64::from(sex.as_u32()) * 10.0 + f64::from(rand10.as_u32())).round() as u32);

                        index = U256::from(index.as_u32() + 1);
                        let name:String = "NAME".to_string() + &index.to_string();
                        let dataform = Dataform{index:index,name:name,sex:sex,weight:weight,height:height,age:age};
                        datas.push(dataform);
                    };
                };
            };
        };

        write_state!(DATAFORMS => datas);
    }
1 Like

Thanks @KanaGold. I’m going to flag this for our engineers focusing on the storage system and see if they have any feedback. We should also probably add your findings to the documentation somewhere cc @jlwaugh what do you think?

1 Like