IT Horror Stories

In this episode, Dr. Chase Cunningham, aka DrZeroTrust, joins us to shed light on what a horror story looks like from an adversarial perspective. In drawing on his extensive red teaming and NSA background, he explores why doing the basics and applying them intelligently does matter, why people should abandon the notion of perfect security, and what controls and practices organizations can adopt and follow to make it a bad day for bad actors.

---------

Key Quotes:

“ It's called a best practice for a reason. It's not because somebody woke up and was like, how can I just, you know, put things on a piece of paper? It's a best practice because it's a practice that's best.”

“ I think people should abandon the idea of perfect security because it doesn't exist. You know, you think if you're if you're postured up and you've spent enough money and you've got the right things in place that you're not going to have a breach. The odds of you not having a breach are really, really slim. It's just a matter of time. That doesn't mean that you stop trying, but it just means to be aware of when things go wrong, what is your plan, how will you isolate, how will you minimize the damage, how do you move or how do you stop lateral movement, etc.”

“There are some people that you just have to say like, Look, man, I gotta put some additional controls around you because you're, you're prone to clicking, bro. Like I don't know what to tell you, man, but I got to do something about you. And it's not because I don't like you and I don't like your hair color or whatever else, but you're a risk to my business.”

---------

Time stamps:

02:26 - About Chase

03:58 - What’s a classic attacker horror story?

08:49 - Red teamer incidents

12:35 - Lessons learned

26:00 - Advice for security leaders

---------

Links:

Find Chase on LinkedIn

Find Jonathan on LinkedIn

Learn more about NinjaOne

When COVID lockdowns loom in early 2020, it’s up to Matthews team to prepare hundreds of laptops for an employee base that’s about to start working from home.

Matthew Schuster, Information Security Analyst at a major sports team, describes the inevitability of IT problems, and the importance of learning how to solve them. His stance is that no questions or problems should be left unresolved.

Matthew Schuster has spent six years navigating the world of IT operations and information security within the fast-paced professional sports industry, steadily rising from an intern to various technician and analyst roles along the way. In addition to all things ITOps and security, he’s passionate about cars, motorcycles, baking, and 3D printing!

---------

Key Quotes:

“ During that time period where everyone was working from home, that gave us time to kind of think about and sit back of like, hey, we got through it.  If we had to do this again, which hopefully we don't, how can we make things better? Not only just adapting like that emergency plan. But also just improving  our day to day kind of stuff. And so, the changes that we made to that plan transitioned to our actual laptop refresh and desktop refresh strategy.”

“ We have a saying in the information security world of, you know, it's not if you get hacked, it's when you get hacked. So you got to be prepared for things and be proactive, but also learn how to fail and learn how to fail  quickly is the most important thing and learn from those mistakes.”

“A very, very common theme in IT is that at the end of the day, problems exist, and you gotta learn how to solve them. And, me personally, I don't like leaving a question or a problem unsolved. “

---------

Time stamps:

00:59 - Matthew’s role

03:50 - Oh, the horror

09:29 - IT in a stadium

10:21 - Breaking down the tasks

15:57 - Don’t be in the sequel

21:05 - Matthew’s survival kit

---------

Links:

Find Matthew on LinkedIn

Find Jonathan on LinkedIn

Learn more about NinjaOne

In this episode, Mike Anderson, CIO & CDO at Netskope, joins Jonathan to discuss several IT horror stories that have stuck with him throughout his career – including a fire suppression test gone wrong, how he helped his organization quickly recover from a ransomware attack in his first tenure as CIO, and now as a more seasoned CIO, how he thinks about taking calculated risks, investing in people, and depositing in relationship accounts to drive business forward.

---------

Key Quotes:

“ Security is everyone's responsibility in an organization. It's not just the CIO. It's not the CISO. It’s everyone in the organization, including the board of directors. And it's your job to make sure you're bringing forward the risks and having conversations about your risk appetite and how much you're willing to invest around the various controls you need to have and why certain controls are more important than others, depending on the business you're in and the risk if those business operations are disrupted.”

“ Things are going to happen. Things are going to go wrong. It's how you respond to it and how you recover. It's the resilience of your team.”

“ We have to make sure we're making calculated risks, but we also have to make sure we're creating a culture where people can experiment because that's how we're going to drive real innovation inside of our organizations.”

---------

Time stamps:

01:08 - Meet Mike

02:58 - The shutdown disaster

05:38 - Managing risk

06:06 - First CIO cyber incident

15:17 - What we’ve learned

29:23 - The fiber catastrophe

---------

Links:

Find Mike on LinkedIn

Find Jonathan on LinkedIn

Learn more about NinjaOne

When a corrupted virus definition file was distributed by a centralized management server, it was up to Jay Abbott to find a fix for the company-wide blue screen of death. Jay describes how the emergency escalated, how he mobilized an all-hands-on-deck response, and how he diagnosed the issue.

Tune in for a masterclass in incident response and disaster recovery as Jay, Director of Corporate IT at NinjaOne, tells his story to our host, Jonathan Crowe. 

---------

Key Quotes:

“It highlighted that nobody plans for this scale of a problem.”

“We tried to make it as fun as it could be. You have to be able to laugh at it.”

“Communicate with users. They're a lot nicer than you want to give them credit for as long as they understand what the solution is.”

“Don't let inaction paralyze you.”

---------

Time stamps:

00:34 - About Jay

04:04 - The incident

12:24 - Finding a solution

13:30 - Nailing prioritization

20:24 - Keeping morale up in a crisis

27:40 - Lessons learned

30:16 - Jay’s survival kit

32:45 - Advice for future leaders

---------

Links:

Find Jay on LinkedIn

Find Jonathan on LinkedIn

Learn more about NinjaOne

Nick Colisto, CIO of Avery-Dennison, describes the IT disaster he faced during Hurricane Sandy. When the storm caused a major power outage and a fuel shortage, his company’s primary data center had to rely on generator power. Tune in to hear how Nick prepared, adapted, and fought through the emergency.

Avery-Dennison is a global materials science and digital identification solutions company with locations in over 50 countries. They provide a wide range of branding and information solutions that optimize labor and supply chain efficiency, reduce waste, advance sustainability, circularity and transparency, and better connect brands and consumers.

---------

Key Quotes:

"Cloud based systems can reduce dependencies, of course, on physical data centers.”

“The event taught us that IT  resilience isn't just about technology, it's about people, about process, and it's also about partnerships and relationships that you had. You know, if we hadn't treated that fuel company well or We didn't have preparation for having our secondary site in place.”

“Solve problems that you  probably didn't anticipate, you're always going to learn something new from these events.”

---------

Time stamps:

01:03 - About Nick

02:21 - Setting the scene

03:46 - Nick’s career

18:25 - IT horror story survival pack

22:34 - Nick’s advice

---------

Links:

Find Nick on LinkedIn

Find Jonathan on LinkedIn

Learn more about NinjaOne

Subscribe on Apple Podcasts

Subscribe on Spotify

IT problems are a nightmare. And you’re not alone in losing sleep over them.

If you’ve worked in IT, you’ve probably got a few ghost stories. This show is dedicated to those stories and the IT heroes who survived to tell the tale. 

Join your host Jonathan Crowe, and some of the world’s top IT leaders, as they recount the most harrowing experiences of their careers. They’ll share how they found themselves in hot water, managed to escape disaster, and their best advice for making sure you don’t end up in the sequel. 

This is IT Horror Stories, where the call is coming from inside the office. 

Brought to you by NinjaOne, the leader in automated endpoint management.