Conditional Random Fields: A Beginner’s Survey
One interesting project that I am involved in these days involves certain problems in Intelligent Tutors. It turns out that perhaps one of the best ways to tackle them is by using Conditional Random Fields (CRFs). Many attempts to solving these problems still involve Hidden Markov Models (HMMs). Since I have never really been a Graphical Models guy (though I am always fascinated) so I found the going on studying CRFs quite difficult. Now that the survey is more or less over, here are my suggestions for beginners to go about learning them.
Tutorials and Theory
1. Log-Linear Models and Conditional Random Fields (Tutorial by Charles Elkan)
Log-linear Models and Conditional Random Fields
Charles
Elkan
6
videos: Click on Image above to view
Two directions of approaching CRFs are especially useful to get a good perspective on their use. One of these is considering CRFs as an alternate to Hidden Markov Models (HMMs) while another is to think of CRFs building over Logistic Regression.
This tutorial makes an approach from the second direction and is easily one of the most basic around. Most people interested in CRFs would ofcourse be familiar with ideas of maximum likelihood, logistic regression etc. This tutorial does a good job, starting with the absolute basics – talking about logistic regression (for a two class problem) to a more general multi-label machine learning problem with a structured output (outputs having a structure). I tried reading a few tutorials before this one, but found this to be the most comprehensive and the best place to start. It however seems that there is one lecture missing in this series which (going by the notes) covered more training algorithms.
2. Survey Papers on Relational Learning
These are not really tutorials on CRFs, but talk of sequential learning in general. For beginners, these surveys are useful to clarify the range of problems in which CRFs might be useful while also discussing other methods for the same briefly. I would recommend these two tutorials to help put CRFs in perspective in the broader machine learning sub-area of Relational Learning.
– Machine Learning for Sequential Learning: A Survey (Thomas Dietterich)
This is a very broad survey that talks of sequential learning, defines the problem and some of the most used methods.
– An Introduction to Structured Discriminative Learning (R Memisevic)
This tutorial is like the above, however focuses more on comparing CRFs with large margin methods such as SVM. Giving yet another interesting perspective in placing CRFs.
3. Comprehensive CRF Tutorial (Andrew McCallum and Charles Sutton)
This tutorial is the most compendious tutorial available for CRF. While it claims to start from the bare bone basics, I found it hard for a start and took it on third (after the above two). It is potentially the starting and ending point for a more advanced Graphical Models student. It is extensive (90 pages) and gives a feeling of comfort with CRFs when done. It is definitely the best tutorial available though by no means the most easiest point to start if you have never done any sequential learning before.
This might be considered an extension to this tutorial by McCallum et al : CRFs for Relational Learning (PDF)
4. Original CRF Paper (John
Lafferty et al.)
Though not necessary to learn CRFs given many better tutorials, this paper is still recommended, being the first on CRFs.
5. Derivations (Rahul Gupta)
This report is good for one to go through the detailed derivations of the equations in CRF.
___________________
Extensions to the CRF concept
There are a number of extensions to CRFs. The two that I have found most helpful in my work are (these are easy to follow given the above):
1. Hidden State Conditional Random Fields (H CRF)
2. Latent Dynamic Conditional Random Fields (LDCRF)
Both of these extensions work to include hidden variables in the CRF framework.
___________________
Software Packages
1. Kevin Murphy’s CRF toolbox (MATLAB)
2. MALLET (I haven’t used MALLET, it is Java based)
3. HCRF – LDCRF Library (MATLAB, C++, Python). As as the name suggests, this package is for HCRF and LDCRF, though can be used as a standalone package for CRF as well.