It took me a little while to figure this one out so I thought I would share:
If you have a blog built on the Domino 8 Blog template you can create categories to assign your blog entries to. To subscribe to the blog you use this url:
That is well documented. If you want to subscribe to a single category in your feed reader or on a website you can use this url:
http://domain.com/BlogDBName.nsf/catcontent.rss?OpenView&restricttocategory=categoryname