Hi,  Priya  Good Afternoon

Your question is very interesting and useful to all Computer Science Students.  Kindly see the attachment.  I think, that attachment is very much useful for you.  I think so.

Thanks.

It is not too complicated to write

a tokenizer yourself. Attached is a

quick-and-dirty example. The idea is

to put read characters into a buffer.

(Today I'd prefer a StringBuilder over

adding characters to a string).

Whenever a new hypothesis about the

type of the token is made, the state

variable and the tokenType are set

accordingly; when the token is complete

(a separator is detected), the end

state is entered (in this case 1000)

and the loop is left... with the

new token set in the member variables

tokenType and tokenValue.

Regards,

Joachim