Metafiction is an elastic concept covering a wide range of fiction but in essence boils down to stories in which the book blurs the line between reality and fiction by drawing attention to itself in some shape or form. To boil it down even further, you could say that it is fiction about fiction.
William H. Gass is attributed with establishing the term metafiction in a 1970 essay titled "Philosophy and the Form of Fiction". Commenting on American fiction of the 1960s, he pointed out that a new description was needed for the emerging genre of experimental texts that openly broke with the tradition of literary realism still dominant in post-WWII American literature.
Some metafiction is like nesting dolls. For example, stories about readers reading books such as Elizabeth Kostova's The Historian (2005). Or writers writing books, such as Andrea Levy's The Long Song (2010), in which a woman is writing a book about Miss July, a slave. That woman turns out to be Miss July herself and she periodically comments on her experience of writing the story with her son looking over her shoulder as editor. Or the story might contain partial or complete stories within them such as David Mitchell's Cloud Atlas.
Stories in which characters are aware that they are part of the story are metafiction. For example Jasper Fforde's Thursday Next novels; or where the narrator is shown to be the author of the story, such as The Brief and Wondrous Life of Oscar Wao by Junot Diaz.
Books where the reader of the story can influence how the story develops are also considered to be metafiction. Children's picture books, where readers feel they influence the action, provide a rich vein of examples, such as Kenn Nesbitt's More Bears! In fact, there's likely a good case to make that the fast growing wealth of ebooks for children where the reader influences the course of the book by interacting with it are, in essence, examples of metafiction.