Meet the Validator AMA series - Blockpane


AMA session Akash Network with Blockpane


Block Pane was founded in 2018 by Todd Garrison, who has worked in the financial, technology, and blockchain industries as an information security engineer. Todd holds the AWS Certified Security - Specialty, and Offensive Security Certified Professional certifications.

In our Telegram chat, we spoke with Todd Garrison

1. Let’s start from the beginning, tell us more about yourself and the team behind Blockpane?

I’m Todd, and I’ve worked in defensive security since the late '90s. I spent most of that in the traditional finance space and worked in telecommunications. In '17, I made the jump into the blockchain space full-time. Presently I am the only person at Block Pane; I had a contractor helping for a while, but income from running validators is not enough to support additional staff.

So you are like one man orchestra. Doing all by yourself :slightly_smiling_face:
So what’s the story, how did you decide to become a validator, and does being a validator worth time and efforts?

Unfortunately for the time being, yes. In May I was planning on adding a full-time employee, and well I think we all felt that pain :sweat_smile:
I have a pretty long answer for that question, bear with me while I copy and paste.
Becoming a validator in the Cosmos ecosystem was a happy accident. I was already running a block producer for the FIO protocol (EOSIO-based chain) but had no intention of expanding beyond that. I was transitioning from my role at the FIO Foundation and was contracting with Halborn Security, where I performed security audits for several projects using the Cosmos-SDK.
I’m a Go-Nut (I picked up golang to help support some forensic work on Ethereum while I was at Ciphertrace,) so it was easy for me to dig into and understand the underlying parts of the code.
My work at Halborn involved auditing the setup and configuration of validator nodes. After setting several up to learn the process and identify where validators could make mistakes, I decided to run one of my own. Sifchain was the first Tendermint validator I brought online.
I never expected to transition into becoming a full-time node operator, but then Osmosis launched, and I found myself in the active validator set. I expanded the number of validators, eventually becoming a self-sustaining business.
Is it worth it?
The market downturn changed the incentives for validating—Block Pane engineers for high performance and availability, which is not cheap. Our validator operation is profitable but does not earn enough to support the business alone. I have been making up for this through various efforts, such as consulting and foundation grants for building tools. Being a validator is a big part of finding supplementary income, so it is worth it.

Wow, that’s a great story

Thanks, it’s been quite the journey!

You have to do what it takes to than thrive in a Bull market

Adaptation is key, but there are so many opportunities in the Cosmos space I’m not too worried about sticking around.

Moving on : what projects are you validating, and how are you contributing to them by being a validator?

I run validators on Akash, Junø, Kava, Osmosis, Sifchain, and Stargaze. I also am contracted by the Aardvark team to run their validators on Terra Classic and Terra2. Of course, I still also run a block producer for FIO. I was validating Kusama for a while, but it was not sustainable.
I try to help in many different ways. Many validators see this as a zero-sum game, and I don’t see it that way; I try to help my fellow validators when they run into trouble. Part of my work is out of public view; I often assist with incidents because of my forensics and incident-handling background.
I also write various programs for getting information out of chains. The tools range from node/validator performance monitoring software to forensics and visualization tools. An example of these tools is something that identifies the location of P2P nodes (not necessarily validator nodes); you can see the distribution of Akash nodes at the following URL:

**That’s quite a lot i have to say.I’m sure any project would love to have you as a validator. **
But from stakers perpective what are the benefits in staking with Blockpane Validator?

I always struggle with self-promotion because humility is essential in decentralized communities. If I had to pick the top reason, it would be that operationally you won’t find better-engineered and managed nodes.
Each validator is distributed across three different providers and uses 2/3 threshold signing (similar to a multisig but for signing blocks.) I exclusively use hardware to maximize performance and have extensive monitoring in place. Every aspect of the node’s configuration is handled through automation, ensuring consistency and the ability to redeploy a node quickly. I recently wrote a blog post about the architecture used: An overview of Block Pane’s validator architecture | by Todd G | Sep, 2022 | Medium.

This is supercool ! :sunglasses:

I tend to geek out on the details :slightly_smiling_face:

And last question from me: What’s in the pipeline for the next quarter?

I don’t expect to validate any new chains in the short term; instead, focusing on quality over quantity. Akash is a recent addition, and I’m delighted to be here. I use rigorous criteria for choosing a chain, and Akash has an excellent team and fantastic product.
I’ll be adding some features for “tenderduty,” a popular validator monitoring tool that will allow securely running on Akash for only a few dollars a month. I’ve been working on several new forensics-focused tools and plan to continue enhancing the software I have released.
I plan to continue working on various development, operations, and security consulting projects. I recently did some work with the Akash team, am currently working with the Skip Protocol, and will likely increase my contributions to Juno. My unique set of abilities means that I have a lot of opportunities available and ensures I’ll continue to focus on Tendermint and Cosmos.

So just a follow up question from reading all this. Is 24 hours a day is enough for you to do all that?

Well, you know having extensive automation and auditing actually makes running validators reasonable.

The operations aspects really don’t require much time, I am in a roundabout way, somewhat lazy. So I tend to build systems to be self-healing on failure, and never want to do setup manually.

Self-healing. Now this is something interesting.
But let’s give our community a chance to ask questions before we dive deeper

Can you explain the measures node placement follows to counter risk of single data centre or providers’ entire network failure?

My nodes are setup using a tool called Horcrux, this is a threshold signing provider for tendermint consensus. …
I split them up across multiple datacenters on rented hardware.
With a 2/3 signer setup, it allows for any particular datacenter to go offline and the validator will continue to sign blocks.
I use a wide range of datacenter providers, and build the systems so that no validator is on the same combination of regions and providers.
One of the situations I try to avoid is to have to deal with multiple validators being down simultaneously.
Even this morning I lost a datacenter provider for a while, and it only affected one validator node. Horcrux was able to recover relatively quickly.
It isn’t foolproof though, so I also have extensive monitoring and alerting.

I understand that Block Pane has build and operate of Tendermint validator nodes. Can you explain more about the progress of this works and what’s the biggest challenge?

This is a great question actually.
When I started out I was building everything by hand, and kept build logs of each step. Somewhere around running 20 servers this became un-manageable.
I’ve since then set everything up using a tool called “ansible”. It’s used for automating systems configuration.
The only aspect it doesn’t manage on the nodes is the initial OS install, and configuration of the various RAID/storage configuration.
For example, when there is a chain upgrade it only takes about 5 minutes to have everything staged in advance.

I understand that Blockpane is an experienced professional validator operating so many networks, including Akash Network.
The question is, why being so active in so many networks? What’s benefits for ecosystem in Blockpane it’s self?

This is an interesting question. I actually feel that the list of chains Block Pane is on is quite short. A lot of independent validators YOLO on every L1 that gets launched.
There are several types of organizations running validators. I fall into what I would call the independant category. A lot of the more recognizable validators are actually large operations and have deals with specific hedge funds and other VC type organizations.
But, getting back to the original point, I know a few independent validators running 50+ chains

Can one person run more than one node at a time (different locations obviously)?

This definitely happens, but running more than one validator on the same network is frowned upon by most other validators.
I do run a couple of what we call “white-label” validators.
But, I don’t run those on networks I already validate, and apply the same distributed configuration to those nodes.
Specifically, I run the Aardvark validator on Terra classic/2

Congratulations to the blockpane for being a validator for the AKASH network An unimportant question that needs classification, at least. Why do you think that the AKASH team has a great vision and is relevant to your logical framework and the team in the blockpane, especially regarding your mission as a validator?

Thanks :slightly_smiling_face: I’ve worked with the team a little bit. The accomplishment of what they have built is quite impressive IMO.
One of the things I like most about the project is that it actually solves a problem. It seems a lot of chains are launched that serve no purpose, so I’m careful about what chains I put effort into.
Having a competent team is key, and Greg and Boz are very impressive. I mean, how many projects have a CEO actively contributing code?

**Related with Blockpane validators monitoring. **
Where can we see the various blockchain nodes are located on? especially for AKASH? do you have your own network monitor tool for AKASH?

Sure, I actually publish the dashboard from one of the tools I wrote called “tenderduty”. It’s a node and uptime monitoring tool built specifically for tendermint nodes. Initially it was a very simple tool that only watched for missed blocks and sent alerts via pagerduty, but has grown into a more comprehensive monitoring tool for validators.
It’s become quite popular with other validators.
You can see my node uptimes here:
I’m a little embarassed actually because that outage this morning affected one of my signing clusters and I have quite a high number of missed blocks on my Juno node :scream:

What were the biggest challenges and difficulties that you faced while being a validator?

So far there have been a lot of different challenges. As a validator my first job is to sign blocks, that’s actually the easy part. Taking part in governance can often be challenging and can get intense sometimes.
Some of the various chains have had very, how to say this? Controversial proposals. Engaging with the community can be hard sometimes. Social media is quite unforgiving. I try very hard to remain humble and polite.

Ultimately my job is to protect the chain for its users and stake holders. Sometimes this makes my votes and opinions unpopular.

There is also some of the security issues and various bugs on cosmos networks. Those can always be a challenge. As a validator it’s kinda assumed that when something breaks you will dig in and.
So often this means long nights where there is an attack or chain halt. Digging through code, analyzing transactions and just helping coordinate the response.

Without proper marketing and capital infusion, project dies, how do you convince us you have adequate marketing power and capital to push this project to the top project?

Yes, I think this is both a good and tough question.
I’ve learned a lot about managing the finances of a validator over the last year. It’s simple during a bull run, but easy to make mistakes that wipe out a business during a bear market.
Right now, the validator side of the business doesn’t have very much income. It does cover hosting costs, and a small portion of the payroll.
The biggest challenge with this year has been taxes. A lot of validators, myself included, failed to take enough profits to cover all the liabilities.

I think a lot of people see validators cash out rewards and it makes them feel the validators are being greedy, but it’s more complex than just that.

US taxes account for about 40% of the rewards. Then there are legal fees, accountants, payroll, and the biggest cost is hosting and infrastructure.
Going forward, and to get back to the question, the Block Pane validator is self-sufficient enough to keep the lights on.
Over the last few months I’ve pivoted to providing more Cosmos-related software development, and contracting. I think most validators are doing a lot more than just validation.
I will admit, the marketing aspect of the company has been lacking. It’s really only over the last year I’ve started to engage more on social media and podcasts.

Long term, Block Pane isn’t going anywhere. The saying is “you build during a bear” and that’s what I’m doing.

Thanks everyone, a lot of great questions :slightly_smiling_face:

Thank you Todd, that was great!
Very informative AMA!

Thanks for having me, I really appreciate the opportunity to get to know the community better.

With that said, let’s conclude our AMA, i hope you enjoyed it guys!

Useful Links :

Medium :