They are completely unrelated genres. The only direct connection is several newer metalcore take influence from nu metal (but back when nu metal was in it's hight metalcore was largly a reaction AGAINST nu metal), and some nu metal bands took influence from metal-influenced hardcore bands, but not really so much from either straight hardcore punk, or metalcore proper, mainly from the New York and DC hardcore scenes (which is odd as nu metal was born in Los Angels, which has it's own, more punk oriented hardcore scene) which kinda fill the gap between hardcore punk and metalcore.
Nu metal and metalcore also have a bit of common history in that they both take a lot from groove metal, but from there they pretty much went in opposite directions. I guess one way of thinking about that is metalcore is a form of extreme metal with an alternative touch, while nu metal is a form of alternative metal with an extreme touch. Beyond both utilizing at least some groovy or chuggy riffs, the similarities end. If you can differentiate either metalcore from groove metal, or groove metal from nu metal, it should be pretty easy to differentiate metalcore from nu metal. Like some groove metal, popular metalcore and nu metal often have more melodic vocals in the chorus, but it's not a defining characteristic of either genre, that stylistic choice actually originated in industrial metal. Really most popular metal nowadays uses that pattern in one way or another, regardless of subgenre.