and the modem is connected to the computer via CAT 5 cable? Without the router, you basically have one less hop. Well I did think of a scenario that could potentially raise your pings. No router means you only have the windows firewall... which probably sucks pretty hard. You could be getting hit with all sorts of stuff trying to get access to your machine or you could have data being sent out, that your firewall would have normally blocked.
A few years back I had a web server running on a windows box that was in the "DMZ" which basically meant that the routers firewall did not apply to that machine. Would have been the same as if I had plugged straight into the modem. I checked the log files after a few days and I was getting spammed all day from different IP addresses trying to log into my machine.
Either way, I doubt you are getting spammed enough to give you a consistent 20+ ms higher ping, but it is possible. given my experience, I would say its very likely that you are at least getting pinged and seeing what ports you have open. Most routers have "respond to ping" turned off.