It could be the h264 profile - MediaInfo will assist you to find that out. Anything over Level 4.1 will cause the file to be transcoded by the PS3 Profile in Serviio (and if playing it without transcoding may be too much for the PS3 CPU to decode natively). Post the MediaInfo detail of a troublesome file if you want and we can take a look at it.
Note that
all MKV's are remuxed by the PS3 profile (and have to be as the PS3 cannot play them natively). What I do is remux my MKV files to MP4 - this then allows them to be played natively. If you are a Windows user my tool here can assist with the remuxing:
viewtopic.php?f=17&t=12136 (shameless plug).
For the PS3 profile, the only [what I call] mainstream video type that gets transcoded is for h264 profiles above 4.1. Most other mainstream video types only get remuxed if they can't be played natively. Streaming natively doesn't take much processor grunt. Remuxing pushes the processor a bit harder (still not normally enough to cause stuttering) but video transcoding is an intense user of CPU - the more cores the better. Alternatively, you avoid transcoding by selecting files that can be played natively (for the PS3 - most MP4's and AVI's. If you remux your MKV's to MP4 then you pretty well have the mainstream covered).
You may want to test that its not something else causing the problem b4 outlaying the cash for a new rig. Can you get temporary access to a more powerful PC - perhaps install a copy of Serviio on it? The next likely candidate (after CPU due to transcoding) is network throughput. Is your PS3 connected wirelessly or via ethernet cable? Also. some users have reported issues that turned out to be their SSD - if possible try setting the transcode temporary folder (under Delivery) to a non-SSD location.
<edit> oh! and yes the extra for an i5 is certainly worth it. 2 extra cores, and for video encoding, cores count.