The ultimate answer is to design in multiples of .48
I have been doing design that way for years now.  Yes, once you get over a few mm in thickness it will be less important.  But on the small stuff it can really help.

Unfortunately, even with the multiple of .48, it tries to do infill unless I tell it to do 0 infill. But that would mess up the other parts of the model that actually needs infill.

For starters, it really tried hard to do infill, doing a back and forth pattern across the 1mm walls. This resulted in a horrible looking print with gobs of plastic all over the sides. It was not nice and smooth like what I get with slic3r. Much to my amazement though, the walls are *much* closer to 1mm! I get measurements from .97-1.12 but it's hard to find good spots because of the horrible gobs of plastic all over it.  Where I can find good spots, they are all right at 1.00mm

So what happens when you specify 1 perimeter with no infill?  (Just as a test)
It should be a quick print.  I have no doubts that If I had a machine with a .40 nozzle in it and printed what my slicer generated, it would come out almost perfect.

Or alternately draw up the same rectangle with a .96 wall thickness and see what that print looks like.

Interesting, I made a similar model with .96 wall thickness all the way around. If I do *any* infill, it still shows a third path. Perhaps because of infill overlap? I have this currently set to 15%.  As a quick test though, I did try setting the infill to 0 with this model, confirmed there were only 2 paths, one for each perimeter.  The measurements I'm getting all the way around are almost exactly 1.00mm!  This is a neat trick, but it makes me think even more that I'm going to need to print something much thicker to get consistent results.  I also tried one at 1.5mm and it comes out to more like 1.8mm all the way around. I suspect as that thickness gets higher, I'll get better results, but that's not really what I'm shooting for here.

Or alternately draw up the same rectangle with a .96 wall thickness and see what that print looks like.

This is the result I get when I slice it at 50% and 100%.  What version of Slic3r are you running?

1.2.9 from Ubuntu Xenial. The three separate paths are easier to see when I look at the layers tab.

I'm using 2mm layer height, .4 nozzle diameter, 1 perimeter, 50% infill.

I'm not really sure what your image is showing me, but the reason I was thinking the infill would be an issue is that if the perimeter walls come out to .48mm, then there would be a .04mm gap between them unless it does *something*.  When I show the layers in slic3r, I distinctly see 3 paths at each layer - one for each perimeter and 1 in the middle to fill in the gap that would exist there.

If you get me your .stl (it didn't show up) and settings (just perimeters, infill and layer height) I will try and duplicate your print to help troubleshoot the cause of your problems.