Many make the mistake to key off the rider in front. Learn to ride your own race, pushing your personal limits.
You identified your mistake, but do not recognize your mistake. Learn to look ahead, trust me, you may think your looking ahead by keying off the rider in front of you, but your not looking ahead. Looking ahead one would look beyond and/or thru the rider your following. While he may be in your field of vision, the best riders do not focus/fixate on the rider in front. In there minds eye, they are already in front and ride by the rider as if he was not there.
When practicing, lead or fall back so the lead rider is out of view. Pursue and when the lead rider comes back in view, back off. Looking ahead slows everything down, the closer one looks, the faster one "feels". But as you note, when you get the lead, you slow. Your great at keying off riders, but you have not developed the skill to see the lines soon enough to make the best decisions to maintain the higher pace.
Another example is the rider who is riding along at his pace, in ones comfort zone. A faster rider approaches and the rider being overtaken, attempts to speed up, mistakes lead to more mistakes and each one presents a passing opportunity.
I learned a mental trick long ago racing MX. When you get the holeshot, in your minds eye, consider yourself last and that you need to ride your very best to catch up to the pack. Ride to unlap yourself, not to protect your lead.
I have ridden with many riders who can wheel suck. However, move over and follow them and the pace drops immediately, or they start making mistakes, as they try to maintain the pace of the other rider, but no longer have that rider to key off of.