Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Make plzip compilable on MSYS2+MinGW-w64
Make plzip compilable on MSYS2+MinGW-w64.
It's Hannes Domani's plzip.patch taken from:
http://download.savannah.gnu.org/releases/lzip/plzip/plzip-1.1-w.zip
that was applied against plzip 1.7:
http://download.savannah.gnu.org/releases/lzip/plzip/plzip-1.7.tar.gz
changed to not artificially limit number of threads (the limit was 32),
and finally diff has been regenerated using output with unified context.
--- a/decompress.cc 2018-02-07 20:17:00.000000000 +0100
+++ b/decompress.cc 2018-02-23 23:14:21.644369300 +0100
@@ -37,6 +37,61 @@
#include "file_index.h"
+#ifdef _WIN32
+#include <windows.h>
+
+ssize_t pread(int fd, void *buf, size_t count, long long offset)
+{
+ OVERLAPPED o = {0,0,0,0,0};
+ HANDLE fh = (HANDLE)_get_osfhandle(fd);
+ uint64_t off = offset;
+ DWORD bytes;
+ BOOL ret;
+
+ if (fh == INVALID_HANDLE_VALUE) {
+ errno = EBADF;
+ return -1;
+ }
+
+ o.Offset = off & 0xffffffff;
+ o.OffsetHigh = (off >> 32) & 0xffffffff;
+
+ ret = ReadFile(fh, buf, (DWORD)count, &bytes, &o);
+ if (!ret) {
+ errno = EIO;
+ return -1;
+ }
+
+ return (ssize_t)bytes;
+}
+
+ssize_t pwrite(int fd, const void *buf, size_t count, long long offset)
+{
+ OVERLAPPED o = {0,0,0,0,0};
+ HANDLE fh = (HANDLE)_get_osfhandle(fd);
+ uint64_t off = offset;
+ DWORD bytes;
+ BOOL ret;
+
+ if (fh == INVALID_HANDLE_VALUE) {
+ errno = EBADF;
+ return -1;
+ }
+
+ o.Offset = off & 0xffffffff;
+ o.OffsetHigh = (off >> 32) & 0xffffffff;
+
+ ret = WriteFile(fh, buf, (DWORD)count, &bytes, &o);
+ if (!ret) {
+ errno = EIO;
+ return -1;
+ }
+
+ return (ssize_t)bytes;
+}
+#endif
+
+
// Returns the number of bytes really read.
// If (returned value < size) and (errno == 0), means EOF was reached.
//
--- a/main.cc 2018-02-07 20:17:00.000000000 +0100
+++ b/main.cc 2018-02-23 23:14:27.592984400 +0100
@@ -69,6 +69,31 @@
int verbosity = 0;
+#ifdef _WIN32
+#include <windows.h>
+
+#define _SC_NPROCESSORS_ONLN 1
+#define _SC_THREAD_THREADS_MAX 2
+long sysconf( int flag )
+{
+ switch( flag )
+ {
+ case _SC_NPROCESSORS_ONLN:
+ {
+ SYSTEM_INFO si;
+ GetSystemInfo( &si );
+ return si.dwNumberOfProcessors;
+ }
+
+ case _SC_THREAD_THREADS_MAX:
+ return -1;
+ }
+
+ return 0;
+}
+#endif
+
+
namespace {
const char * const Program_name = "Plzip";
@przemoc

This comment has been minimized.

Copy link
Owner Author

przemoc commented Apr 17, 2020

pread() and pwrite() implementation available here, i.e. revision 1 of plzip-msys2-mingw-w64.patch, is not POSIX conformant, so program may misbehave.
Check System Interfaces volume of POSIX, specifically read and write, where you can read:

The pread() function shall be equivalent to read(), except that it shall read from a given position in the file without changing the file offset.

The pwrite() function shall be equivalent to write(), except that it writes into a given position and does not change the file offset (regardless of whether O_APPEND is set).

The issue is that ReadFile and WriteFile used to implement them update file pointer (equivalent of file offset in POSIX world), regardless of OVERLAPPED structure being used to specify byte offset or not. As can be read in Synchronous and Asynchronous I/O:

File pointer position for a synchronous handle is maintained by the system as data is read or written and can also be updated using the SetFilePointer or SetFilePointerEx function.

Using SetFilePointer() to save current file pointer before pread()/pwrite() and restore it afterwards would be only acceptable in single-threaded applications. In multi-threaded applications, where pread()/pwrite() calls may be concurrent to read()/write() calls (relying on current file offset), we would have a window, when file offset is different than expected.

You can additionally check following 2 threads from mailing lists that touched this subject:

So how pread()/pwrite() could be implemented in Windows?

Basically additional file object is needed that would be exclusively used by pread()/pwrite(), thus file pointer changes would not affect read()/write() calls, as they would be using "original" file object. I do mean file object (equivalent of file description in POSIX world) and not just mere file handle (equivalent of file descriptor in POSIX worlds). Using DuplicateHandle won't suffice:

The duplicate handle refers to the same object as the original handle. Therefore, any changes to the object are reflected through both handles. For example, if you duplicate a file handle, the current file position is always the same for both handles. For file handles to have different file positions, use the CreateFile function to create file handles that share access to the same file.

It may be tempting to use NtQueryInformationFile with FileInternalInformation as FileInformationClass followed by NtCreateFile with FILE_OPEN_BY_FILE_ID in CreateOptions, but 64-bit file ID is not available, stable and unique on all file systems, as can be seen in below table. Therefore this approach cannot be used without any fallback solution if application is meant to work on non-NTFS file systems like FAT32 or exFAT.

64 bit file ID Generate Stable Unique
FAT Yes No No
EXFAT Yes No No
FAT32 Yes No No
Cdfs No n/a n/a
UDFS Yes Yes Yes
NTFS Yes Yes Yes
ReFS Yes Yes Yes

Table source:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/d4bc551b-7aaf-4b4f-ba0e-3a75e7c528f0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.