Class CodeRay::TokenStream
In: lib/coderay/tokens.rb
Parent: Tokens

TokenStream

The TokenStream class is a fake Array without elements.

It redirects the method << to a block given at creation.

This allows scanners and Encoders to use streaming (no tokens are saved, the input is highlighted the same time it is scanned) with the same code.

See CodeRay.encode_stream and CodeRay.scan_stream

Methods

<<   dump   new   optimize   stream?   text_size  

Attributes

size  [R]  The Array is empty, but size counts the tokens given by <<.

Public Class methods

Creates a new TokenStream that calls block whenever its << method is called.

Example:

  require 'coderay'

  token_stream = CodeRay::TokenStream.new do |text, kind|
    puts 'kind: %s, text size: %d.' % [kind, text.size]
  end

  token_stream << ['/\d+/', :regexp]
  #-> kind: rexpexp, text size: 5.

[Source]

     # File lib/coderay/tokens.rb, line 313
313:     def initialize &block
314:       raise ArgumentError, 'Block expected for streaming.' unless block
315:       @callback = block
316:       @size = 0
317:     end

Public Instance methods

Calls block with token and increments size.

Returns self.

[Source]

     # File lib/coderay/tokens.rb, line 322
322:     def << token
323:       @callback.call(*token)
324:       @size += 1
325:       self
326:     end

A TokenStream cannot be dumped. Use Tokens.

[Source]

     # File lib/coderay/tokens.rb, line 335
335:     def dump
336:       raise NotImplementedError, 'A TokenStream cannot be dumped.'
337:     end

A TokenStream cannot be optimized. Use Tokens.

[Source]

     # File lib/coderay/tokens.rb, line 340
340:     def optimize
341:       raise NotImplementedError, 'A TokenStream cannot be optimized.'
342:     end

Whether the object is a TokenStream.

Returns true.

[Source]

     # File lib/coderay/tokens.rb, line 292
292:     def stream?
293:       true
294:     end

This method is not implemented due to speed reasons. Use Tokens.

[Source]

     # File lib/coderay/tokens.rb, line 329
329:     def text_size
330:       raise NotImplementedError,
331:         'This method is not implemented due to speed reasons.'
332:     end

[Validate]