I'm the opposite, actually. I still have major urges to watch hentai and, while I'm not really proud of it, I still continue to watch it. I'm 37 days fap-free so far though, and I can say for sure it helped made me feel better; more stable and relaxed, psychologically.
I believe the reason people stress the need to avoid porn is that usually masturbation follows it for many men out there. It will also screw with the rewards circuitry in your brain, causing you to feel less attracted to other things in life (ex: real women, working on your projects, etc.) and making you end up easily distracted, much like people with their hand-held social media nowadays. Basically porn is just mental junk food which is best to avoid entirely if you can.
Fapping follows the same reasoning because your brain gets flooded with endorphins after a discharge and, just like a drug, your body will scream to get its fix since it's so used to it. Your symptoms of withdrawal can even be exacerbated if your habit of fapping is tied to certain habits and memory triggers (ex: sitting down at your computer and finding out you're alone…). So yes and no, fapping is not necessarily tied to porn/hentai, but yes it can be a trigger. The biggest trigger, actually.
As for the prostate cancer thing, I don't really believe in it. If that were true, where are all the reports of dead Buddhist monks that died from abstaining? You'd have better chance to avoid cancer by watching what you eat and drink, than from expelling all the man-juice which your body recycles anyway and sends back to the bloodstream along with all the good nutrients (like zinc) that's in it.