I've seen some seriously shot out rifles that were still fairly accurate. As mentioned the first thing to check is anything touching the barrel like a sling swivel screw.
Second thing to check is the bullet casing, if the neck isn't sealing properly the excess gas bypassing the neck can cause grief. Not only does it cause strange harmonics, it can cause an almost unconscious flinch when your shooting.
Next thing I'd do is retrieve a shot bullet or ten. Got to be careful with this, water can and will ricochet (deflect) a bullet. A duffel bag with a trash bag liner, filled with water and hung from a tree (in a safe spot) will catch a bullet most times or at least slow it enough to retrieve. Compare the bullets, study the lands and grooves. If you have a machining failure in your barrel, I'd contact Remington and ask for a replacement. Also mark the barrel to receiver fit, see if your mark changes between shots (a tiny scratch will do).
I had one Heckler and Koch that had a machine flaw, Heckler isn't Remington but the replacement they sent me was absolutely a first pick pistol. For what it is, it is one of the most accurate I've ever owned.
When your shooting, rest the rifle up on three sand bags, two front, one rear, touch the rifle as little as possible, make sure it is as vertical as possible for each shot. Just let it touch your shoulder (the same every time) and squeeze it off. Let it cool down some between shots with the bolt open.
I had a Winchester XLR in 308. that would shoot outstanding groups, with a medium priced Tasco scope.
A rule of thumb is, the rifle is almost always more accurate than the shooter is.
Many rifles climb (or change) as they heat up, in real life the first shot is usually the only one that counts anyway. Scatter groups are most often trigger jerk. Subtle things that you may not even notice, like a loose chamber and powder squirting out of the bolt (it can sting a little) and you'll learn to flinch fast. You may not even consciously notice. Hold a piece of white paper up next to the bolt and squeeze one off, see if there is any powder spray leaking around the bolt.
Most well equipped Gunsmiths have a bore scope.
Last thought, I've seen the internals in scopes come loose. I've got one on the shelf now with a loose reticle prism.