Surprisingly, the majority of officials have never been competitive cyclists. Many don't even ride recreationally. Most got their start because friends or family were or are racers, and they got hooked on the pageantry of cycling, race day logistics puzzles, days in the sun amidst the most beautiful sport.
A key tenant I've been pushing over the last few years is that more racers need to become officials. There are many reasons for this, but one notion is that experienced racers are better equipped to evaluate safety factors. That's not to say by any means that racing experience is necessary to do so or be a good official, but it seems reasonable, modulus some overly gung-ho racers. Without that race experience it's often hard to gauge what racers need and what will work---how far out do they need to know a corner is controlled, how desperately will they avoid that gravel, can they see those lap cards, etc.
But, there are things I've learned that racing experience simply does not provide, and in fact may gloss over. In particular, racers don't generally think about external events and actors. Racers focus on the things they will encounter during their race: Potholes, bumps, sharp corners. They very rarely think about all the things they hopefully never encounter during a race---pedestrians and vehicles being the most troubling---precisely because so many people are working to ensure they don't encounter them.
Any racer can take a quick lap and settle on a corner to worry about that they can't pedal through or take at full speed. Only sharp observers think about mitigating external events, e.g. that doing lap after lap downhill through a blind double s-curve in a heavily trafficked area without total course control would be inviting that sort of external event to happen. Tough corners are problematic; someone may crash, someone may get hurt. That's the nature of the sport. External events are frequently fatal---for the racers if it's a vehicle, for the pedestrians if it's not. That's something to be avoided at all costs.
Unfortunately, those risks are significantly harder to recognize and judge. They're much less obvious than basic course "features" and the probabilities frequently pretty low. As a promoter, official, director, these are always the hardest decisions. No race can cover every possible contingency and no event can put out more than a finite number of marshals, each with only a very finite ability to control the world around them. Every time a road course crosses an unwatched driveway or tiny side street, every time a crit is held somewhere not in the middle of nowhere, we're all taking risks with gravely serious possibilities. Balancing course tradeoffs, evaluating probabilities versus outcomes, all of that is gut wrenching and definitely the deepest, darkest part of race promotion and officiating.