The Data Committee’s goals are to provide data and analytics to other stakeholders in the ecosystem.
Our philosophy is to deliver the 80/20 solution, allowing us to work quickly, iteratively, and prioritize for impact. We write code, make pretty graphs, deliver data-driven insights, design ETL pipelines, build tools, automate stuff, maintain databases and APIs, and provide support.
The Data Committee is full-stack, covering analytics and data engineering; this covers the full pipeline of when a data point is created all the way to it ending up in the hands of end-users via whatever platform.
This involves working with people to make sure we’re capturing the right data, writing data transformation jobs to make sure the data is in a useful state, coding scripts and notebooks for analysis, presenting results, and following up to make sure our work is interpreted correctly and acted on.
Practically, this manifests as four continual deliverables:
- secretanalytics.xyz dashboard
- data requests (analysis / simulations / reports / providing datasets, etc.)
- data pipeline / tooling
- secretanalytics API and integrations (for CoinGecko, DefiLlama, devs, etc.)
Open membership. Lots of people from different backgrounds are interested in data and analytics, so all are welcome to participate as they’d like. This could be contributing to our work products, giving feedback on our dashboards, requesting new features or data to be tracked and visualized, etc. Several non-technical community members have contributed by giving feedback and brainstorming during our weekly calls.
Leads are expected to contribute to all parts of the stack as well as help scope and manage projects. I’d expect a lead to at a bare minimum have at least a few years experience working as a data scientist or data engineer and have proficiencies with: Python (SciPy stack), a major cloud platform (AWS/GCP/Azure/etc.) and basic devops, git, SQL and relational DBs (designing data warehouses and writing clean queries), writing ETL pipelines (custom packages or utilizing frameworks like dbt/apache beam/airflow/etc.), and some visualization/BI tools.
Weekly Discord calls - Thursday at 6:30PM UTC (i.e. 2:30PM EST, 1:30PM CT, 12:30PM MT, 11:30AM PT)
Committee calls involve sharing updates on recent projects, submitting data requests, talking about relevant news relating to data and analytics, prioritizing work for the following week(s), and working / brainstorming sessions.
Moving forward, all project management and tracking will be through the projects board on github.
Total: 45,000 SCRT
Lead pay: 15,000 SCRT over 3 months | ~15-20 hours per week
Me: superfish / gmail_user
- 7 years experience working as a data scientist / machine learning engineer
- Analytics Committee Lead for ~10 months
Discretionary Budget: 30,000 SCRT
- infrastructure costs (currently ~$1,100 per month)
- paying contributors (to date I’ve paid ~$9,000 out of my own pocket)
- I’d like to use this to pay community members that have been actively contributing and to attract external data experts to build and contribute
Any excess budget not used during the 3 month time frame will carry over into future months as we continue to develop the work product(s).
In the past, Secret Foundation has covered infrastructure costs and funded the initial work for the proof of concept of our data pipeline.
Within the next 3 months:
- Improvements to data platform
- Upgrading our ETL system
- Adding jobs for more in-depth analysis like cohort analysis
- Better data collection for social platforms
- Adding tracking for the Monero bridge and IBC
- Launch v3 Secret Analytics (soon!) and continual improvements
We will be moving towards open-sourcing our entire codebase, which is a goal for 2022.
Analytics starts with data engineering. This is a broader title that more accurately captures the full scope of work and activities.
Happy to hear any feedback and answer any questions