If you get sandy mud in there it can shorten the life of your shoes by years. Leaving the mud most anyplace for any length of time usually turns into a rust trap and starts rot.
I pop the drums off, rinse off the brake parts and then scrub it down with hot water and a mild mix of dish washing soap. Rinse it again. Let it dry out have look behind the brake cylinder boot for seepage and button it up.
It actually works well and in the long run helps prevent rust.
On my 87, I actually drilled two large holes through the backing plate (with an antenna drill)l. Drilled the holes to fit a couple of rubber stoppers I had handy and regularly blew a power washer into the holes after mudding. It worked better than not cleaning them, but on my 88 I pop the drums off on a regular basis.