This brick retaining wall under a wood fence is falling apart. And it's too late to save it. The flaking surface is a classic example of “spalling,” a type of failure caused by moisture and the freeze/thaw cycle. Water from rainfall, melting snow or even wet soil saturated the brick at some point, probably often, and froze inside the brick when the temperature dropped. The slight expansion of freezing water fractured the brick. With repeated freezing and thawing, the fractures widened until the brick literally disintegrated.
To resist these forces, bricks used on the exterior are usually harder and denser (less moisture absorption). When you see an occasional brick in a wall that fails, it's often a softer brick that was mixed in the load by mistake. However, you have massive failure, indicating a severe water intrusion problem. The problem is probably not bad brick.
Exterior walls have to be detailed carefully to keep water out, particularly when they're part of a retaining wall. The most obvious element missing in your case is a coping or cap on the wall to keep rain out.
This consists of metal flashing, concrete or special bricks that are angled to direct water away from the top of the wall. The cap may have been there at one time, but it was removed or it wore away. In addition, the wood fence above probably catches and directs even more water down onto the wall.
If you have your brick wall rebuilt, make sure it's designed to minimize water intrusion. In addition to a cap on the top, it should have good draining fill (gravel) along the backside and weep holes along the bottom to relieve water pressure when the soil becomes saturated. Damp-proofing the backside of the wall will also help keep water out. And good drainage around the base will help keep water from entering from below.