Got some more feedback from Roger and Sarah today about the testing process. I had no doubts that it would be a complicated endeavor, but there’s a fair amount of studying I’ll need to do before I can really sink my teeth into the code of things.
As a side note, if you’re a rookie data scientist, programmer, statistician, etc. reading this, I hope you realize how much studying you’ll need to do to stay on top of your game within the industry. Tech changes FAST, and software changes even faster. That’s the double-edged sword of open-source science I suppose. There will be times where you spend a couple hours absolutely crushing 2,000 lines of code, and there will probably be many more times where you sit surrounded by research papers, textbooks, and Towards Data Science blogs, waiting 2 hours for a 10 line algorithm to finish its 30 minute runtime and trying to remember what all these things have to do with each other in the first place.
Anyways, let’s go over what I learned about metrics. Precision/recall is fairly intuitive, you’re just looking at the number of true positives, false positives, and false negatives in order to compare a predicted DAG model to its ground truth. SHD is a little trickier, but essentially it compares the number of edge changes a DAG would require in order to get back to the ground truth. I also found a metric called the Structural Interventional Distance, which is more focused on comparing levels of causality between models rather than just positional relationships. I’m hoping I’ll be able to use at least 2 of these to evaluate algorithm performance within the test cases.
Searching For a Cause 