- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: strtok
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-24-2002 01:18 PM
10-24-2002 01:18 PM
strtok
Thanks,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-24-2002 01:41 PM
10-24-2002 01:41 PM
Re: strtok
strtok() works for simple tokenizing but for more complex tasks, you can use lex to generate a lexical engine and for really complex tasks, it's time to use yacc.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-24-2002 01:56 PM
10-24-2002 01:56 PM
Re: strtok
Here is what I try to do.
char *sep = "()*+,/ ";
char *str = "a * b + c/foo(x,y,z)";
strtok will give me the following tokens:
a
b
c
foo
x
y
z
But I want this:
a
*
b
+
c
/
foo
(
,
x
,
y
,
z
)
----------------------
In other words, I want to use a function where I know WHAT was the delimiting token.
Or must I write my own tokenizer.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-24-2002 02:38 PM
10-24-2002 02:38 PM
Re: strtok
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2002 02:08 AM
10-25-2002 02:08 AM
Re: strtok
If I had to do this now I would use regular expressiong.
They are not so hard to learn, and flexible enough.
man rexexec
man regcomp
Don't forget regfree at the end. ;)
Adam
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2002 03:30 AM
10-25-2002 03:30 AM
Re: strtok
I also don't think that you can do it using strtok only ... I also noticed that spaces have a special meaning. If you don't have time to invest, I wrote this which uses a "mirror" string.
Regards,
Jean-Louis.
#include
#include
main (argc, argv)
int argc;
char *argv[];
{
char *sep = "()*+,/ ";
char *str = "a * b + c/foo(x,y,z)";
char *v;
char *p;
char o[BUFSIZ];
int first;
v=str;
p=o;
while (*v)
{
if (*v != ' ')
{
*p=*v;
p++;
}
v++;
}
*p='\0';
p=o;
printf("string <%s>\ntokens <%s>\nnewstr<%s>\n\n", str, sep, o);
v=strtok(str,sep);
first=1;
do {
if (first)
{
first=0;
printf("<%s> first pass\n", v);
}
else
{
printf("<%s> token <%c>\n", v, *(p-1));
}
p+=strlen(v)+1;
v=strtok(NULL,sep);
} while(*v);
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2002 05:50 AM
10-25-2002 05:50 AM
Re: strtok
Back to my problem:
I was looking for something likes Java's StringTokenizer where there is an option to return the delims as tokens, that's all.
Regular Expressions.... I am very comfortable with regex, in fact in Perl I would do this:
@values = grep { length } split m!([ ()*+,/])!;
@values would get all non-zero length tokens, including the delims (notice the capturing parenthesis).
I have not used any regex in C, and I will certainly give it thought. For start, does C have a split function or better how would you write the above line in C?
In the mean time, for simplicity I think I would go with Jean's simple approach and do it in C.
Nevertheless, the regex route is cool:)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2002 08:18 AM
10-25-2002 08:18 AM
Re: strtok
As I think more about it I come to the conclusion that afterall you would end ap with the iterating loop, so in your case mayby simple Jean-Louis (the winner of the month!) approach would be more practical.
However if you were interested in regex in C check the
But one warning:
Don't remove spaces at the beginning. I think they should be ignored later. If you had two strings without operator they should be intepreted separately (and probably later considered semantical error). If you just remove spaces they will just concatenate into single string and you won't detect it.
Example:
"a + b + c d" -> "a+b+cd"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2002 08:29 AM
10-25-2002 08:29 AM
Re: strtok
As an academic, it would be interesting to see how many lines of C will be needed to do what the above Perl does in one line.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-28-2002 01:23 AM
10-28-2002 01:23 AM
Re: strtok
I was thinking about good regular expresions to be defined and I found few potential traps you can fall into, so I decided to write.
You have to treat operators differently from tre names. For the names good regex seems to be:
"[a-zA-Z_][a-zA-Z0-9_]*"
of course, if you think about C.
But for the operators its not so nice. Should they only be single characters? If so what about operators like "<=" (of course, if you respect them).
But you cannot collect all operators automatically, because in an expression like "a*(b+c)" the first operator you will get is "*(".
So, I'm affraid, operators should be defined more strictly, and for names I would use the abore regexp. The rest I would do in a loop with cases.
So, the conclusion is I would do this with regexp. Perl and (sorry) Jean-Louis sollutions could generate you errors.
Good luck
Adam
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-31-2002 12:29 PM
10-31-2002 12:29 PM
Re: strtok
Regards,
Leslie
#include
#include
#define TOKEN_DELIMITER 100
#define TOKEN_TEXT 200
char * my_strtok (char *text, char *sep)
{
static char
*p,
*this_token;
static short
next_token_type;
size_t
token_length;
static long
current_token_max_len = 50; /* Should be good for most tokens */
if ( this_token == NULL )
{
fprintf (stderr, "Info: Calling malloc\n");
if ( !(this_token = (char *) malloc ( sizeof (char) * current_token_max_len)) )
{
perror ("my_strtok() malloc failed");
return NULL;
}
}
if ( text != NULL )
{
p = text; /* Let 'p' point to the text given */
if ( strchr (sep, *p) )
{
next_token_type = TOKEN_DELIMITER;
}
else
{
next_token_type = TOKEN_TEXT;
}
}
if ( next_token_type == TOKEN_TEXT )
{
token_length = strcspn (p, sep);
next_token_type = TOKEN_DELIMITER;
}
else
{
token_length = strspn (p, sep);
if ( token_length == 1 )
{
next_token_type = TOKEN_TEXT;
}
else if ( token_length > 1 )
{
token_length = 1;
next_token_type = TOKEN_DELIMITER;
}
}
if ( token_length == 0 )
{
if (this_token) free (this_token);
return NULL;
}
if ( token_length >= current_token_max_len )
{
fprintf (stderr, "Info: Calling realloc\n");
if ( !(this_token = (char *) realloc (this_token, sizeof (char) * (token_length + 1))) )
{
perror ("my_strtok() realloc failed");
if (this_token) free (this_token);
return NULL;
}
}
strncpy (this_token, p, token_length);
*(this_token + token_length) = '\0';
p += token_length;
return this_token;
}
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-31-2002 12:31 PM
10-31-2002 12:31 PM
Re: strtok
If we could only edit our posts:(
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-04-2002 11:45 AM
11-04-2002 11:45 AM
Re: strtok
I'm affraid I don't like your code.
I don't see the point in the way hou handle dynamic memory.
Appart from it I see some bugs.
Actually I would do it from the scratch rather than remaster that code.
But I have to warn you: I like the things to mbe done precisely, so my code will be more complecated.
I'll think of it in the neares future if you still need it.
Adam