2ny.netlify.com

NTFS-3G POSIX namespace incompatible with some embedded hw.

I have a USB multimedia hard disk enclosure, they're the kind that look more or less like any enclosure, but have a small embedded computer inside allowing them to act as a standalone player.
The enclosure's media firmware therefore supports reading off the hard drive's filesystem directly, and can read NTFS. When plugged into a PC it shows up like any other generic external disk.
Anyways, getting to the point. At first I tried writing to it with NTFS-3G, and it seemed to work for awhile, then suddenly after copying a massive amount of data (over 200G), some files started to appear 'invisible' to the enclosure's firmware, however, when you plugged it into a PC (windows or linux with ntfs-3g), all the files were present.
After much troubleshooting and reading, I read about how NTFS was designed with three namespaces, and that NTFS-3G supports all three, but writes files in POSIX by default, I figured this was a likely cause because it's conceivable that a product like this might not conform 100% to the official NTFS specifications.
I decided to test this by doing several experiments, I wrote data from windows to the enclosure, and all of it was readable, then I tried to write a lot of data from linux (ntfs-3g) and again this specific data showed up as invisible, in some cases the top level folder was visible, but anything inside was invisible. Again, on a PC everything is visible. Then I deleted that same data and rewrote it from windows, and it shows up. I did this several times, and also used ntfsinfo from ntfsprogs to check what the namespaces were on certain files.
There's more information I might be able to provide on the testing methodology I used, but for now, as a first post, I just want to get the basics on the table.
Now, I know this is not an NTFS-3G bug, however, we all know what hardware vendors are like. The fact is that NTFS-3G's behavior in regards to it's default namespace is different than Windows, and this means some devices out there could be incompatible, not just mine.
However, before I get flamed, let me say this, I do agree that POSIX should be the default namespace, since it makes sense for UNIX/LINUX users.
But what I think is that there should be a mount option to override this default behavior, to make NTFS-3G behave like Windows XP's driver, assigning both Win32 and DOS entries to each file created.
There could be mount options to specify them separately, such as:
-o ntspace=win32,dos
-o ntspace=win32
I've looked in the manpage and something like that doesn't seem to be present. This would potentially increase compatibility with all devices that test only against Windows and nothing else. It is unlikely that any hardware vendors actually take the POSIX namespace (or even the DOS one for that matter) into account when designing their products, so I would not be surprised if there were other hardware out there that has this problem.
Oh, and by the way, I did contact the vendor earlier this week and told them there is a problem with their NTFS implementation, however I did not have as much information about the specifics then as I do now. I also have suspicions that they likely licensed their implementation from some other vendor, meaning they probably can't do anything and would have to pass on my problem to *their* OEM for their firmware, and so on.
Anyways, that is my situation. All suggestions and discussion is welcome. Thank you in advance for taking the time to read this [quite lenghty] post.

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Thank you for the interesting description.
The vendor should fix their firmware. Windows SFU also creates files in the POSIX namespace, so they have the same problem with Windows too in that case. All namespaces are STANDARD, one can't support only one, or two. NTFS-3G supports all but writes only in the POSIX namespace to maximize interoperability. This 100% conforms to the NTFS specification what the flawless Window interoperability also proves.
The POSIX namespace was not only designed and used for UNIX/Linux by Microsoft, but for all operating systems, including Windows and the one your embedded hardware uses. This is a government regulation Microsoft had to comply to be able to run for $$$$$ governmental and other businesses.
May I ask what are these 'some embedded hw'? We would like to document and warn people they have a non-conformant NTFS driver. Thank you.
Regards, Szaka

Hello, thank you for your response. I appreciate it.
The particular embedded hardware I am referring to in this case is my MediaSonic Media Player HM2-SU2TV.
I understand that the proper fix would be for the vendor to change their firmware, but it is quite unlikely that they will. I also know that even Windows sometimes creates files in the POSIX namespace, I recall seeing some files that're longer than the Win32 character limit. However, even Windows itself has problems and incompatibilities with these files, I have noticed that sometimes if you try to delete such a file it will say things like 'file doesn't exist' or something: http://www.google.com/search?q=filename+too-long
In other words, just because the namespace is 100% valid, that doesn't mean that they treat it equally. And most if not all files that the user would be copying to this media enclosure (mostly your movies, music and images) are going to be created using the Win32 & DOS namespaces. They have no reason to comply with my request to fix their firmware.
The bottom line here, is that I have hardware that works in both Windows and Linux, but ONLY when I write data from Windows, can the media enclosure's firmware READ the data. Regardless of who's 'fault' it is, or who is to blame, the fact is that the driver on Linux behaves differently than the Windows NTFS driver, and therefore the results with some implementations out there will be different.
Like I said, the *default* can remain POSIX, but why would it be such a bad thing to have a mount *option* to change it? Free Software and Open Source are about choice, and taking control of your computer, I should be able to tell the driver to write in whatever namespace *I* chose.
Right now I am in a position of being forced to use Windows because I cannot tell the Linux driver to write files in the namespace that I want it to.
I want to make it clear, this is not a bug report, it is more of a feature request, I'm not accusing NTFS-3G of having the problem, I already noted that the problem is with the firmware. But adding this simple feature to the driver could easily solve the issue.
I'm sorry for the post being long again, but I like to be thorough and I got the feeling that I did not explain myself very well the first time. Thank you.

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: :

Yes. Unlike your vendor to who you paid, this is why we gave you the source code and the right to be able to implement yourself what you want.
But I sincerely hope you understand that you can't expect us to implement something brain-damaged (too long to explain) in our own free time so you can use your broken hardware because its vendor ignores you.
If you don't want to code yourself then I suggest return your hardware or sell it then use some other one which knows how to implement NTFS properly. There are plenty ones.

I understand your view, however I don't see how this is 'brain-damaged'.
The vfat driver in linux has to do the same sorts of things.
How is it brain-damaged to support a write mode that the Windows NTFS driver supports? The fact is, the Windows driver can do something that yours cannot. And Microsoft, as much as many people despise them, *are* the reference implementation for all their technologies.
Just like the Wine project has a tradition of doing everything 'the windows way' to maximize compatibility, any other project seeking to implement Microsoft's technology would do well to follow that principle at least to some extent if they wish to reach a proper level of compatibility with the various products available out there that are *only* tested against Windows.
However, I will not press the issue any longer, although I'd be interested in the 'long explanation' of why implementing something that exists in windows, as well as in the vfat linux driver, could be considered brain-damaged.
Again, I appreciate that you have taken the time to answer me, and I hope you haven't interpreted me as being hostile in any way, I am trying to be as polite as possible in voicing my concerns and opinions, with the greatest amount of respect.

Hi,
I own the same product which you've discussed here. I'm having a bit of problem with it and after reading your extensive post I thought you might be able to help me out here. This maybe a bit outta what you've discussed here.
I recently upgraded the firmware from using a download from the URL mentioned in the manual (http://www.usbex.com/media01/) After the upgrade everything seemed to go well until the restart. It starts up as before (when connected to the TV) but I could not navigate or do anything at all. I assumed it might be due to a remote control fault and bought a new remote as well. Didnt work.
So I was wondering whether you would know how to downgrade/revert or undo the firmware upgrade.
Please help me out here/
Thanks

Re: NTFS-3G POSIX namespace incompatible with some embedded hw.

I have the same problem with Packard Bell Studio 640 ( http://support.packardbell.com/it/item/ .. C052050001 )
Is there a workaround to this problem?
If I would try to patch the code, where should I begin? (in which part of the code?)
This hardware also claim FAT32 support, which is a pain, but it's better than nothing.. I'd prefer NTFS support anyway.