I have a small client application designed to run on windows machines. It is written in C/C++ (more in C). I want to keep it small and therefore I would prefer not to use external libraries and stick to WinAPI on client side.
On the other hand I have a backend server which is implemented in python3. Here I’m happy to use any existing lib. Now I want to add compression layer to increase transfer speed. The problem I’ve encountered is that it seems that WinAPI provides only:
// source: https://docs.microsoft.com/en-us/windows/win32/cmpapi/using-the-compression-api XPRESS XPRESS with Huffman encoding MSZIP LZMS
which seem to be a unique Microsoft compression algorithm implementation and I cannot find a way to decompress the data on the server side in python3.
Is there anything I am missing? I would love to hear some solutions:)
Thanks in advance
I’ve decided to use zlib https://zlib.net/ as suggested in the comment. The answer suggesting using ctypes was also very interesting, but unfortenately my backend is running on UNIX system.
As I am compiling my client part agains Multi-threaded CRT (and not DLL) I had some problems as zlib is being linked agains the Muli-threaded DLL. If anyone has enountered such issue, I have found a great and super simple solution here: https://yongweiwu.wordpress.com/2017/10/02/a-journey-of-purely-static-linking/ I will copy-paste it here:
zlib This part requires a small change to the build script (for version 1.2.11). I need to open win32Makefile.msc and change all occurrences of ‘-MD’ to ‘-MT’. Then these commands will work: nmake -f win32Makefile.msc zlib.lib
I found this Python script, and reverse-engineered it to produce a quick library to handle WinAPI Compression/Decompression. Basically, you can just use
ctypes and call the WinAPI from Python. Please keep in mind that I haven’t extensively tested this, but it should give you a pretty good place to start 🙂
EDIT: As requested, I’ve included the implementations of the
decompress functions in case the links ever go down.
def compress(UncompressedBuffer, UncompressedBufferSize, Format, Engine): CompressedBuffer = (ctypes.c_ubyte * UncompressedBufferSize)() CompressionFormatAndEngine = ctypes.c_uint16(Format.value | Engine.value) CompressBufferWorkSpaceSize = ctypes.c_uint32() CompressFragmentWorkSpaceSize = ctypes.c_uint32() WorkSpace = (CompressFragmentWorkSpaceSize.value * ctypes.c_ubyte)() FinalCompressedSize = ctypes.c_uint32() WinDLLCall(RtlGetCompressionWorkSpaceSize, CompressionFormatAndEngine, ctypes.byref(CompressBufferWorkSpaceSize), ctypes.byref(CompressFragmentWorkSpaceSize)) WinDLLCall(RtlCompressBuffer, CompressionFormatAndEngine, ctypes.byref(UncompressedBuffer), ctypes.c_uint32(UncompressedBufferSize), ctypes.byref(CompressedBuffer), ctypes.c_uint32(UncompressedBufferSize), UncompressedChunkSize, ctypes.byref(FinalCompressedSize), ctypes.byref(WorkSpace)) return CompressedBuffer, FinalCompressedSize def decompress(CompressedBuffer, CompressedBufferSize, UncompressedBufferSize, Format): UncompressedBuffer = (ctypes.c_ubyte * UncompressedBufferSize)() FinalUncompressedSize = ctypes.c_uint32() WinDLLCall(RtlDecompressBuffer, Format, ctypes.byref(UncompressedBuffer), ctypes.c_uint32(UncompressedBufferSize), ctypes.byref(CompressedBuffer), ctypes.c_uint32(CompressedBufferSize), ctypes.byref(FinalUncompressedSize)) return UncompressedBuffer, FinalUncompressedSize
The WinAPI functions are loaded with the following code:
import ctypes RtlDecompressBuffer = ctypes.windll.ntdll.RtlDecompressBuffer RtlCompressBuffer = ctypes.windll.ntdll.RtlCompressBuffer RtlGetCompressionWorkSpaceSize = ctypes.windll.ntdll.RtlGetCompressionWorkSpaceSize
Here are some other useful resources if you end up tinkering around:
- WinAPI Compress/Decompress Documentation: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-rtlcompressbuffer
- Header File: https://github.com/tpn/winsdk-10/blob/master/Include/10.0.14393.0/km/ntifs.h
- Status Codes: https://github.com/tpn/winsdk-10/blob/master/Include/10.0.10240.0/shared/ntstatus.h