Personally, I hate this whole discussion (i.e. talking about this matter). And that's because it's entirely hypothetical.
First, effectively no-one can prove that any lolicon artist actually used child porn as a reference. You will say they can; but effectively no-one can prove either many other things considered 'proved' by public opinion, e.g. that humans are intelligent beings or that Barack Obama isn't a muslim terrorist. It's relatively easy to prove (though not to disprove) that some lolicon artist owned/watched child porn. It's relatively hard to tell whether he/she's actually used it as reference. This point is somewhat weak, but it's worth thinking about it.
Second, as it has already been pointed out in other threads here, there is no (logically explainable) causal relationship between watching CP and the abuse of the depicted child in this direction. That is, by watching CP you don't hurt that child again (though it's true that the availability of the material does 'perpetuate' the harm already done), at least not individually. Another weak point when it comes to real CP, but lolicon, if based on real CP at all, adds yet another layer of indirection that breaks the chain of causality.
Which leads to Third: After all, you never know what some artist had in mind when creating a certain piece of art. I'm not an expert on art, but I use the term in its broadest sense here so that any drawing can be considered art. And all of this is also true for pieces of music and so on. You never know what that person thought about when drawing/composing/… that drawing/music/… so why should the art be in any way linked to those thoughts? E.g. some say Lewis Carroll was a paedophile; while that cannot be proved anymore (and we don't know what he actually might have done to children), would you knowing that call Alice in Wonderland child porn or even a bad book? What about paintings created by murderers in prison? I'd rather say that the art's integrity doesn't depend on the circumstances of its creation or the integrity of the creator. (So how you perceive lolicon shouldn't be mixed with that, it should be based on the art 'as is'.)
From the separation of concerns viewpoint, I'd even say that it's entirely the artist's problem or responsibility, not that of the consumer. If you consume something that turns out to have been produced in an illegal way, you'd want the producer to be prosecuted, not yourself, wouldn't you? Current legislation works somewhat differently when it comes to CP, but in theory, that's what you'd expect. E.g. if you purchase some clothes and then learn that the production included child labour, you'd expect the company that produced the clothing to be prosecuted (if at all), not yourself. Many people say for moral reasons you shouldn't even buy clothes from companies known to employ child workers. Some people don't care. But even if you care, you'd expect the producing company to be responsible for designing their production in such a way that it's (morally and) legally unproblematic. You'd not expect yourself to be responsible for the company producing in an unproblematic way. (Especially if no company does that.)
It's the same with lolicon: It's the responsibility of the artist to create the art in a legally immaculate way, and it's the consumers' perfect right to expect a sound production (although the consumer doesn't need to trust the producer…). I exclude the term 'moral' here because we cannot always consider ourselves moral apostles.
Lolicon is considered fishy by the public anyway so if you're a consumer who cares about your reputation you should anyway be as careful as if you were watching actual CP. Apart from that you shouldn't suspect that every better artist used CP to train or create his/her works; such considerations just waste time. If your moral ideals require it, just stop consuming works by artists you came to know to be using CP as reference. Universal suspicion, however, isn't helpful.
If you're an artist, though, be aware it's your very responsibility towards your clients to create your art in a clean way. Don't moan about flat chests etc. being so hard to draw. Use your imagination! I'd expect artists to have plenty of that. Often it's not that important to meet reality but rather to meet the client's expectations.
TLDR:
Why care if you're a consumer? It's the artist's responsibility, not yours.