encoders/term.rb By Rob Aldred (robaldred.co.uk) Based on idea by Nathan Weizenbaum (nex-3.com) MIT License (www.opensource.org/licenses/mit-license.php)
A CodeRay encoder that outputs code highlighted for a color terminal. Check out robaldred.co.uk
VERSION | = | '0.9.7' | Version: Major.Minor.Teeny[.Revision] Major: 0 for pre-stable, 1 for stable Minor: feature milestone Teeny: development state, 0 for pre-release Revision: Subversion Revision number (generated on rake gem:make) | |
DEF_NEW_STATE | = | WordList.new(:initial). add(%w(def), :def_expected). add(%w(import from), :include_expected). add(%w(class), :class_expected) | ||
DESCRIPTOR | = | / #{NAME} (?: \. #{NAME} )* | \* /x |
Encode a string.
This scans code with the the Scanner for lang and then encodes it with the Encoder for format. options will be passed to the Encoder.
See CodeRay::Encoder.encode
# File lib/coderay.rb, line 209 209: def encode code, lang, format, options = {} 210: encoder(format, options).encode code, lang, options 211: end
Encodes filename (a path to a code file) with the Scanner for lang.
See CodeRay.scan_file. Notice that the second argument is the output format, not the input language.
Example:
require 'coderay' page = CodeRay.encode_file 'some_c_code.c', :html
# File lib/coderay.rb, line 244 244: def encode_file filename, format, options = {} 245: tokens = scan_file filename, :auto, get_scanner_options(options) 246: encode_tokens tokens, format, options 247: end
Encode a string in Streaming mode.
This starts scanning code with the the Scanner for lang while encodes the output with the Encoder for format. options will be passed to the Encoder.
See CodeRay::Encoder.encode_stream
# File lib/coderay.rb, line 198 198: def encode_stream code, lang, format, options = {} 199: encoder(format, options).encode_stream code, lang, options 200: end
Encode pre-scanned Tokens. Use this together with CodeRay.scan:
require 'coderay' # Highlight a short Ruby code example in a HTML span tokens = CodeRay.scan '1 + 2', :ruby puts CodeRay.encode_tokens(tokens, :span)
# File lib/coderay.rb, line 232 232: def encode_tokens tokens, format, options = {} 233: encoder(format, options).encode_tokens tokens, options 234: end
Finds the Encoder class for format and creates an instance, passing options to it.
Example:
require 'coderay' stats = CodeRay.encoder(:statistic) stats.encode("puts 17 + 4\n", :ruby) puts '%d out of %d tokens have the kind :integer.' % [ stats.type_stats[:integer].count, stats.real_token_count ] #-> 2 out of 4 tokens have the kind :integer.
# File lib/coderay.rb, line 273 273: def encoder format, options = {} 274: Encoders[format].new options 275: end
Extract the options for the scanner from the options hash.
Returns an empty Hash if :scanner_options is not set.
This is used if a method like CodeRay.encode has to provide options for Encoder and scanner.
# File lib/coderay.rb, line 291 291: def get_scanner_options options 292: options.fetch :scanner_options, {} 293: end
Convenience method for plugin loading. The syntax used is:
CodeRay.require_plugin '<Host ID>/<Plugin ID>'
Returns the loaded plugin.
# File lib/coderay/helpers/plugin.rb, line 341 341: def self.require_plugin path 342: host_id, plugin_id = path.split '/', 2 343: host = PluginHost.host_by_id(host_id) 344: raise PluginHost::HostNotFound, 345: "No host for #{host_id.inspect} found." unless host 346: host.load plugin_id 347: end
Scans the given code (a String) with the Scanner for lang.
This is a simple way to use CodeRay. Example:
require 'coderay' page = CodeRay.scan("puts 'Hello, world!'", :ruby).html
See also demo/demo_simple.
# File lib/coderay.rb, line 156 156: def scan code, lang, options = {}, &block 157: scanner = Scanners[lang].new code, options, &block 158: scanner.tokenize 159: end
Scans filename (a path to a code file) with the Scanner for lang.
If lang is :auto or omitted, the CodeRay::FileType module is used to determine it. If it cannot find out what type it is, it uses CodeRay::Scanners::Plaintext.
Calls CodeRay.scan.
Example:
require 'coderay' page = CodeRay.scan_file('some_c_code.c').html
# File lib/coderay.rb, line 172 172: def scan_file filename, lang = :auto, options = {}, &block 173: file = IO.read filename 174: if lang == :auto 175: require 'coderay/helpers/file_type' 176: lang = FileType.fetch filename, :plaintext, true 177: end 178: scan file, lang, options = {}, &block 179: end
Scan the code (a string) with the scanner for lang.
Calls scan.
See CodeRay.scan.
# File lib/coderay.rb, line 186 186: def scan_stream code, lang, options = {}, &block 187: options[:stream] = true 188: scan code, lang, options, &block 189: end
Finds the Scanner class for lang and creates an instance, passing options to it.
See Scanner.new.
# File lib/coderay.rb, line 281 281: def scanner lang, options = {} 282: Scanners[lang].new '', options 283: end
# File lib/coderay/scanners/python.rb, line 97 97: def scan_tokens tokens, options 98: 99: state = :initial 100: string_delimiter = nil 101: string_raw = false 102: import_clause = class_name_follows = last_token_dot = false 103: unicode = string.respond_to?(:encoding) && string.encoding.name == 'UTF-8' 104: from_import_state = [] 105: 106: until eos? 107: 108: kind = nil 109: match = nil 110: 111: if state == :string 112: if scan(STRING_DELIMITER_REGEXP[string_delimiter]) 113: tokens << [matched, :delimiter] 114: tokens << [:close, :string] 115: state = :initial 116: next 117: elsif string_delimiter.size == 3 && scan(/\n/) 118: kind = :content 119: elsif scan(STRING_CONTENT_REGEXP[string_delimiter]) 120: kind = :content 121: elsif !string_raw && scan(/ \\ #{ESCAPE} /ox) 122: kind = :char 123: elsif scan(/ \\ #{UNICODE_ESCAPE} /ox) 124: kind = :char 125: elsif scan(/ \\ . /x) 126: kind = :content 127: elsif scan(/ \\ | $ /x) 128: tokens << [:close, :string] 129: kind = :error 130: state = :initial 131: else 132: raise_inspect "else case \" reached; %p not handled." % peek(1), tokens, state 133: end 134: 135: elsif match = scan(/ [ \t]+ | \\\n /x) 136: tokens << [match, :space] 137: next 138: 139: elsif match = scan(/\n/) 140: tokens << [match, :space] 141: state = :initial if state == :include_expected 142: next 143: 144: elsif match = scan(/ \# [^\n]* /mx) 145: tokens << [match, :comment] 146: next 147: 148: elsif state == :initial 149: 150: if scan(/#{OPERATOR}/o) 151: kind = :operator 152: 153: elsif match = scan(/(u?r?|b)?("""|"|'''|')/i) 154: tokens << [:open, :string] 155: string_delimiter = self[2] 156: string_raw = false 157: modifiers = self[1] 158: unless modifiers.empty? 159: string_raw = !!modifiers.index(?r) 160: tokens << [modifiers, :modifier] 161: match = string_delimiter 162: end 163: state = :string 164: kind = :delimiter 165: 166: # TODO: backticks 167: 168: elsif match = scan(unicode ? /#{NAME}/uo : /#{NAME}/o) 169: kind = IDENT_KIND[match] 170: # TODO: keyword arguments 171: kind = :ident if last_token_dot 172: if kind == :old_keyword 173: kind = check(/\(/) ? :ident : :keyword 174: elsif kind == :predefined && check(/ *=/) 175: kind = :ident 176: elsif kind == :keyword 177: state = DEF_NEW_STATE[match] 178: from_import_state << match.to_sym if state == :include_expected 179: end 180: 181: elsif scan(/@[a-zA-Z0-9_.]+[lL]?/) 182: kind = :decorator 183: 184: elsif scan(/0[xX][0-9A-Fa-f]+[lL]?/) 185: kind = :hex 186: 187: elsif scan(/0[bB][01]+[lL]?/) 188: kind = :bin 189: 190: elsif match = scan(/(?:\d*\.\d+|\d+\.\d*)(?:[eE][+-]?\d+)?|\d+[eE][+-]?\d+/) 191: kind = :float 192: if scan(/[jJ]/) 193: match << matched 194: kind = :imaginary 195: end 196: 197: elsif scan(/0[oO][0-7]+|0[0-7]+(?![89.eE])[lL]?/) 198: kind = :oct 199: 200: elsif match = scan(/\d+([lL])?/) 201: kind = :integer 202: if self[1] == nil && scan(/[jJ]/) 203: match << matched 204: kind = :imaginary 205: end 206: 207: else 208: getch 209: kind = :error 210: 211: end 212: 213: elsif state == :def_expected 214: state = :initial 215: if match = scan(unicode ? /#{NAME}/uo : /#{NAME}/o) 216: kind = :method 217: else 218: next 219: end 220: 221: elsif state == :class_expected 222: state = :initial 223: if match = scan(unicode ? /#{NAME}/uo : /#{NAME}/o) 224: kind = :class 225: else 226: next 227: end 228: 229: elsif state == :include_expected 230: if match = scan(unicode ? /#{DESCRIPTOR}/uo : /#{DESCRIPTOR}/o) 231: kind = :include 232: if match == 'as' 233: kind = :keyword 234: from_import_state << :as 235: elsif from_import_state.first == :from && match == 'import' 236: kind = :keyword 237: from_import_state << :import 238: elsif from_import_state.last == :as 239: # kind = match[0,1][unicode ? /[[:upper:]]/u : /[[:upper:]]/] ? :class : :method 240: kind = :ident 241: from_import_state.pop 242: elsif IDENT_KIND[match] == :keyword 243: unscan 244: match = nil 245: state = :initial 246: next 247: end 248: elsif match = scan(/,/) 249: from_import_state.pop if from_import_state.last == :as 250: kind = :operator 251: else 252: from_import_state = [] 253: state = :initial 254: next 255: end 256: 257: else 258: raise_inspect 'Unknown state', tokens, state 259: 260: end 261: 262: match ||= matched 263: if $CODERAY_DEBUG and not kind 264: raise_inspect 'Error token %p in line %d' % 265: [[match, kind], line], tokens, state 266: end 267: raise_inspect 'Empty token', tokens, state unless match 268: 269: last_token_dot = match == '.' 270: 271: tokens << [match, kind] 272: 273: end 274: 275: if state == :string 276: tokens << [:close, :string] 277: end 278: 279: tokens 280: end