ANTLR (ANother Tool for Language Recognition) is a parser generator.<\/p>\n
You might be designing your own programming\/scripting language or defining something a little simpler, such as query language for your application or even just something to parse user input (which is not necessarily simpler). <\/p>\n
ANTLR can be downloaded from here<\/a> and whilst the tool itself is written in Java it can generate parser code in Java, C#, Python, JavaScript and more. <\/p>\n Note: I’ve downloaded the “Complete ANTLR 4.8 Java binaries jar” hence the commands listed will all be in relation to that JAR.<\/em><\/p>\n Grammar files<\/strong><\/p>\n Before we can do anything meaningful with ANTLR we need to define a grammar for our lexer and parser to be generated from. These are text files with the extension .g or .g4 (for ANTLR 4 compatible grammars) and looks lot like BNF. Let’s create a grammar for a very simple query language that we’d like to incorporate into an application. <\/p>\n We start by creating a file named querylanguage.g4<\/p>\n I’m using Visual Studio Code to write the grammar and using the excellent ANTLR4 grammar syntax support<\/em> extension to aid in developing the grammar.<\/p>\n The first thing we add to the grammar is <\/p>\n The grammar name must match the name of your grammar file. It can be prefixed with lexer<\/em> or parser<\/em> if you want to write grammar specific to either a lexer or parser, without these we’re creating a combined lexer and parser grammar. <\/p>\n Here’s all the options for the grammar keyword.<\/p>\n Before we extend our grammar let’s talk about comments – single line comments can be created using \/\/ and multi line using \/* *\/.<\/p>\n We create our grammar using a combination of rules and tokens. Rules should be declared using camelCase whereas tokens should be all upper case. We’ll need an entry point (or start rule) so let’s extend our grammar within a start rule named query<\/em> and this will be made up of an expression rule. We’ll also add a token to handle string types.<\/p>\n We’ll look at the syntax of our rules etc. in a moment.<\/em><\/p>\n As you can see, a rule is made up of a name followed by a colon and then either another rule or token and finally a rule or token is terminated with a ;. Alternate rules or tokens are defined using the | operator. So in the example above we have two token rules, query and expression and three tokens rules, STRING, SIGN and NUMBER. <\/p>\n If we input “Hello World” then the ANTLR tokenize and parse this as a STRING. If we input 1234 then we’ll see this tokenize and parsed as a NUMBER. We can now start to build up our grammar from these basic building blocks, so let’s add some logic operators, AND and OR. We’ll also add a token rule to deal with whitespace characters (which we want to ignore). So change\/add the following to the existing grammar<\/p>\n WS is a special token rule which basically skips\/ignores the various whitespace characters.<\/p>\n Let’s now take a look at the rule syntax we’ve used. <\/p>\n See Grammars<\/a> for the rest of the Grammar syntax.<\/p>\n Another interesting keyword is fragment<\/em>, for example<\/p>\n A fragment is a modifier which doesn’t result in a token being visible to the parser but is more like an inline rule, in that it’s used “inline” within other rules, what if gives us is a way to create more reusable constructs and is useful if we want to share rules amongst other rules or to just make rules more readable.<\/p>\n Reserved keywords<\/strong><\/p>\n ANTLR4 includes the following reserved words<\/p>\n So obviously you cannot use these for your own rule names.<\/p>\n Debugging our grammar with the VS Code, ANTLR4 grammar syntax support extension<\/strong><\/p>\n This extension<\/a> is really useful when working with our grammars. First off we have syntax highlight and auto completion, which is always useful but it also includes a debugger and can display ANTLR exceptions when we’re not matching tokens or rules.<\/p>\n To debug your grammar in VS Code, either editor your launch.json or add a configuration via Run | Add Configuration, here’s the configuration for testing our grammar<\/p>\n You’re might want to change the “input” file to a named file of your choice, but basically this is just a text file where we put our test to be parsed via the grammar, so for example my sample.text file looks like this<\/p>\n The “startRule” is the top level rule we want to parse our input through, so in our example it’s the query<\/em> rule. Ofcourse we also need to tell the antrl-debug the “grammar” to use.<\/p>\n Now in VS code press the run button using this configuration and you’ll get to see the output of the parse tree, for example<\/p>\n There’s also a lovely parse tree so you can see how your grammar was parsed and which rules matched to which input.<\/p>\n Generating code<\/strong><\/p>\n Everything we’ve looked at is great, but ofcourse we’ll want to actually include our grammar within our application and ANTLR comes to help by allowing us to use the previously downloaded JAR to generate the code for our preferred (and ofcourse supported) language.<\/p>\n By default if you run<\/p>\n Then you’ll get a set of files produced which include Java source files for a Lexer, Parser and Listener. If you want to generate a Visitor then add the -visitor switch, like this<\/p>\n As I’m wanting to generate C# source files, we can simply add the -Dlanguage switch, for example<\/p>\n But sadly this doesn’t work<\/strong>. It seems that whilst the JAR supports generating C# code the ANTLR .NET package is not compatible. In another post we’ll look at how we can generate C# code from our grammar.<\/p>\n References<\/strong><\/p>\n ANTLR Tool Command Line Options<\/a> ANTLR (ANother Tool for Language Recognition) is a parser generator. You might be designing your own programming\/scripting language or defining something a little simpler, such as query language for your application or even just something to parse user input (which is not necessarily simpler). ANTLR can be downloaded from here and whilst the tool itself […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[310],"tags":[],"class_list":["post-8587","post","type-post","status-publish","format-standard","hentry","category-antlr"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts\/8587","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/comments?post=8587"}],"version-history":[{"count":5,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts\/8587\/revisions"}],"predecessor-version":[{"id":8606,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/posts\/8587\/revisions\/8606"}],"wp:attachment":[{"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/media?parent=8587"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/categories?post=8587"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/putridparrot.com\/blog\/wp-json\/wp\/v2\/tags?post=8587"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\r\ngrammar querylanguage;\r\n<\/pre>\n
\r\nlexer grammar querylanguage;\r\n\r\n\/\/ or\r\n\r\nparser grammar querylanguage;\r\n\r\n\/\/ or \r\n\r\ngrammar querylanguage;\r\n<\/pre>\n
\r\ngrammar querylanguage;\r\n\r\nquery\r\n : expression\r\n ;\r\n\r\nexpression\r\n : STRING\r\n | NUMBER\r\n ;\r\n\r\nSTRING : '"' .*? '"';\r\nSIGN\r\n : ('+' | '-')\r\n ;\r\nNUMBER \r\n : SIGN? ( [0-9]* '.' )? [0-9]+;\r\n<\/pre>\n\r\nexpression\r\n : STRING\r\n | NUMBER\r\n | expression 'AND' expression\r\n | expression 'OR' expression\r\n ;\r\n\r\nWS : (' '|'\\t'|'\\r'|'\\n')+ -> skip;\r\n<\/pre>\n\n
\r\nHexLiteral : '0' ('x'|'X') HexDigit+ ;\r\nfragment HexDigit : ('0'..'9'|'a'..'f'|'A'..'F') ;\r\n<\/pre>\n
\nimport, fragment, lexer, parser, grammar, returns,
\nlocals, throws, catch, finally, mode, options, tokens
\n<\/em><\/p>\n\r\n"version": "0.2.0",\r\n"configurations": [\r\n {\r\n "name": "Debug ANTLR4 grammar",\r\n "type": "antlr-debug",\r\n "request": "launch",\r\n "input": "sample.txt",\r\n "grammar": "querylanguage.g4",\r\n "startRule": "query",\r\n "printParseTree": true,\r\n "visualParseTree": true\r\n }\r\n]\r\n<\/pre>\n\r\n"HELLO" AND 123\r\n<\/pre>\n
\r\nParse Tree:\r\nquery (\r\n expression (\r\n expression (\r\n ""HELLO""\r\n )\r\n "AND"\r\n expression (\r\n "123"\r\n )\r\n )\r\n)\r\n<\/pre>\n
\r\njava -jar antlr-4.8-complete.jar .\\querylanguage.g4\r\n<\/pre>\n
\r\njava -jar antlr-4.8-complete.jar -visitor .\\querylanguage.g4 \r\n<\/pre>\n
\r\njava -jar antlr-4.8-complete.jar -visitor -Dlanguage=CSharp .\\querylanguage.g4\r\n<\/pre>\n
\nANTLR 4 Documentation<\/a>
\nvscode-antlr4<\/a>
\nSample grammars<\/a>
\nGrammars<\/a>
\nCheat Sheet<\/a>
\nLexer Rules<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"