Oooohhhhhh.... besides my family, these are my two most enjoyable things, combine them and its even better!
Up along the Kern River Area is pretty good fishing, and there are some trails in that area as well... Nothing too challenging....
Go further up to Monache Meadows like Stephan said.... more of a summer time trip..
The Owens Valley all around the Owens river is lots of dirt trails with some excellent year round trout fishing.
Up near Mammoth is Laurel Lakes (High Clearance 4x4 - basically slightly taller than stock) is required. It is a small lake with very little pressure w/ primative camping.
I've been itching to go too.......
ScottyDog