This project has moved and is read-only. For the latest updates, please go here.

How to regenerate TokenKind and tonkenizer

Apr 18, 2012 at 4:52 PM
Edited Apr 18, 2012 at 4:52 PM


I am playing with the syntax and I would like to add some keywords, how can I regenerate TokenKind and
the tokenizer's part that recognizes the keywords :



private Token ReadName() {
            #region Generated Python Keyword Lookup

            // *** BEGIN GENERATED CODE ***
            // generated by function: keyword_lookup_generator from:


Actually I don't care about TokenKind because I can add new keywords by hand but it's more difficult for ReadName().

I couldn't find in repository, only found it inside IronPython but didn't manage to make it work...

Apr 18, 2012 at 5:05 PM

This comes from the IronPython respository because PTVS's parser was originally based upon IronPython.  You could grab it from IronPython's github repo - it lives in Languages\IronPython\Scripts but it's likely out of date as PTVS has changed the parser so it'll probably need some updates.