windows and unix file format using Samba 2.0.9

[Madhava Rao Lakkaniki]

Under windows, the text file look dos format correctly, but when I look the file under unix, it is having extra ^M character like unix format.

Why this ^M is showing in my file and how to stop it without running dos2ux command.

Is there any parameter required in smb.conf under global parameters.

I am using Samba(CIFS) 2.0.9 and HP-unix 11.0

Thanks in advance.

Madhav

[Rodney Hills]

The simple answer is no.

The complex answer is possibly. Samba has an option for each share called "magic script". This script is call on each file as it is closed (it runs on the host side).

Theoritically you might be able to generate a script that ran dos2ux automatically.

The documentation on "magic script" says it is experimental.

Good Luck

-- Rod Hills

[Rodney Hills]

In answer to why ^M show up.

Unix text files have ^J as line terminator.

Dos text files have ^M^J as line terminator.

The ^M is just another character to unix.

-- Rod Hills

[Wodisch_1]

Hi Madhav,

what you experience is an application problem, not an operating system problem.
If you need to process the same files (shared via SAMBA, e.g.) from both platforms, you will have to
- convert to and fro all the time
- use compatible(=interoperatible) applications on both platforms
The first would include the use of "dos2ux" and such, the second the use of applications like StarOffice or OpenOffice, which are available on both platforms, and are able to read and store their "data" files (i.e. XML, today) transparently.

Just my $0.02,
Wodisch

[Darren Prior]

Hi Madhav,

To add to Rodney's answer I don't believe you can achieve your aim with magic scripts - they only work for a nominated script for each share.

So, a user would need to open and close the magic script file whenever they made a change to a file from their PC. The magic script would then have to determine which file had been modified (using the file's datestamp) and run dos2ux on it.

The end result of this is that the files would then not be formatted correctly if another user tried to open it from a PC, which could cause confusion.

regards,

Darren.

[Madhava Rao Lakkaniki]

In the previous versions of samba, users did not get ^M problem and other unix application used to read the file and work with out any error. Now application is rejecting and giving error due to ^M at the of the line of this data text file. This problem has started after upgrade to Samba 2.0.9.

I can use dos2ux command, but everytime, user has to depend on me before running the other Unix application. User is not happy with this.

Any thoughts above this?

Regards

Madhav

[Rodney Hills]

For all the versions of samba I have used, the ^M issue has been around. Their have been forums discussing this issue and how to address it.

It usually boils down to since samba is doing block writes (not examining "records"), then it would cause more problems to automatically strip out the ^M. For instance a binary file like .zip should not be touched.

The only tool I know that strips the ^M during a network copy, is "ftp". Even in that situation, you tell ftp that the file being copied is ascii.

Any unix application we write, we add ^M if we know a PC app will be reading it, and when we read a file we know is from a PC, then we have the application strip it out.

What version of samba were you using before the upgrade? It doesn't make sense that any earlier version would behave differently.

-- Rod Hills

[Shannon Petry]

First, I find the ^J pretty funny. That is not why it happens at all. Understanding the problem may help you understand why you have it and can not solve it easily.

Unix uses a Carriage Return for a new line. Simple and Easy.
DOS uses a Carriage Return AND a Line feed. Since Any windows is still DOS this is how it saves files.

When you have a share between the two systems and users in Unix and Windows creating text files, your Unix users will see ^M(linefeed) on the end of each line of text made in DOS.

The Kicker here is that even though the Geniouses at MS made text files to have 2 character new lines, if they encounter a UNIX made file with simply a carriage return, they will automatically add the line feed to it to make it a DOS text file.

Samba is not capable of fixing this problem, nor is CIFS or any other file/print sharing software for that matter, and it has never been the case with Samba. I have been using Samba for 3 1/2 years and never seen it automagically know who is opening the file and translate new line's accordingly.

The simple answer is scenario based.
1. If all your users need the text files in windows, write your own cron job to run every file through unix2dos.
2. If you have users in both arena's that need the text files write your own cron job and run all files through dos2unix.
Scenario 2 will work, because like I said Notepad, Wordpad, and Word automatically add the DOS required linefeed character to plain text files. Unix is WYSIWYG.

Regards,
Shannon

[Rodney Hills]

Shannon,

Unix text files have ^J (hex 0a), as displayed by the "xd" command, as the end of line character.

You press the CR key at the keyboard, but unix writes the text line with a ^J in the file.

Try it-
xd somefile | more

You will find all text lines terminated with 0A.

-- Rod Hills

[harry d brown jr]

If the user isn't happy with it, then tell the user to quit using M$ products :-)

6 Feb 2000 - 9 Feb 2000 (21 posts) Archive Link: Using NT registry calls to solve CR-LF issue with text files.

Any old Unix hand will know all about line endings. MS-DOS text files are supposed to end their lines with the two ASCII characters CR and LF, whereas Unix text files usually just end lines with LF. Mac OS, like the earliest Apple II DOS before it, simply uses CR. Any user of a system that supports file access to/from multiple platforms will run into this incompatibility sooner or later; usually, text files must be converted manually from one format to another, since it is difficult for a computer to divine what is or isn't a "text file", as opposed to a binary file which must not be tampered with. Periodically, the issue comes up on the Samba lists of how to convert text files to and from the MS-DOS line-ending standard. One standard answer: on Windows, just use WORDPAD instead of NOTEPAD, since WORDPAD can work with Unix-style text files. Another common answer: use a preexec script to convert all the files in a share whenever a client mounts the share.

http://kt.linuxcare.com/samba/sm20000217_12.epl


live free or die
harry

[Shannon Petry]

The UNIX text files do not have a ^J on them, but have a control character for a LF. All unix Text is based on the ASCII character definitions.

Look up the ASCII equivalent Hex code to 0a and I'll bet id does not give ^J as the ASCII value but a LF.

Sorry, but the ASCII code set does not have ^J, nor ^M for that matter, but control characters represented by a terminal as ^J or ^M.

[Rodney Hills]

Shannon,

I guess I'm showing my age.

Pressing control-J on a old terminal generates hex 0A and transmits it to the host.

You are correct, that the ascii tables represent 0A as LF. I'm used to mapping all the control codes to their equivalent text code.

eg-
4A = J -- 0A = ^J (or LF)
4D = M -- 0D = ^M (or CR)
48 = H -- 08 = ^H (or BS)

On older terminals, that's what the control key would do. It would send the character with the bit 6 set to zero.

Either way, DOS and Unix text files are different...

-- Rod Hills

[Bill Hassell]

As mentioned, Samba is sharing the filesystems. There is no possible way for Samba to know whether a file is to be converted either direction (Notepad is unhappy with Unix files, vi is unhappy with DOS/Windows files).

Bottom line: DOS asnd Unix are unrelated and there is no reason to expect the file formats to be the same. A cron job could perform the translation but users would have to be careful not to include binary files which would be corrupted by dos2ux or ux2dos commands.

Perhaps a reexamination of the needs and process steps required to share these files is needed. For instance, a batch ftp script could transfer files either direction with the proper translation (the ftp ASCII command).

[Reinhard Burger]

Hi Madhav

I found the same situation with an application that has to parse ascii files content into ORACLE on a unix box. (shared by samba)

We handeled it the following way.
The application that has to read the files is getting the information if the file is Windoof (doof is german for silly ;-)) or if the file is unix format by reading a configration file.
If the file is windows we just skip the last character in the file and our application on UNIX is fine with that.

Maybe you can decide if you have windows or unix format just by reading the file extensions and let your application react on this.

example configfile for this

extension windowsflag
.doc Y
.ksh N
.xls Y


General question : Looking at your message i do not see if the files are really used by UNIX based applications. If you do not read or modify the files in UNIX then everything is AS IT SHOULD BE because of Windwos is seeing what it is expecting to see when opening a windows file on the samba share. So there is nothing wrong as long as UNIX applications do not need to touch the file.