Pickle data was truncated ошибка

I can not send my numpy array in socket. I use pickle but my client pickle crashes with this error: pickle data was truncated

My server :
I create a numpy array and I want to send in my client with pickle (it’s work)

import socket, pickle
import numpy as np
from PIL import ImageGrab
import cv2


while(True):
    HOST = 'localhost'
    PORT = 50007
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 4096)
    s.bind((HOST, PORT))
    s.listen(1)
    conn, addr = s.accept()
    print ('Connected by', addr)

    arr = np.array([[0, 1], [2, 3]])
    printscreen_pil=ImageGrab.grab(bbox=(10,10,500,500))
    img = np.array(printscreen_pil) ## Transform to Array
    
    data_string = pickle.dumps(img)
    conn.send(data_string)

    msg_recu = conn.recv(4096)
    print(msg_recu.decode())

    conn.close()

My client
He has my numpy array, but I can not load with pickle. I have this error.

import socket, pickle
import numpy as np

HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))

msg_a_envoyer = "hello".encode()
s.send(msg_a_envoyer)


while 1:
    data = s.recv(4096)
    if not data: break
    data_arr = pickle.loads(data)
    print (data_arr)
s.close()

🐛 Bug

I am trying to load my monodatasets for XLM and am stumped with this pickle data issue. I have attempted to pass various arguments including expression ascii, latin1 and utf-8
data = torch.load(path) File "\lib\site-packages\torch\serialization.py", line 358, in load return _load(f, map_location, pickle_module) File "\lib\site-packages\torch\serialization.py", line 532, in _load magic_number = pickle_module.load(f) _pickle.UnpicklingError: pickle data was truncated

To Reproduce

I am working with 0.4 pytorch on the recent (translation model)[https://github.com/facebookresearch/XLM/]

Environment

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.5
Is CUDA available: N/A
CUDA runtime version: 9.0
GPU models and configuration: GPU 0: GeForce GTX 960M
Nvidia driver version: 419.35
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip] numpy==1.16.2
[pip] torch==0.4.1
[conda] blas 1.0 mkl
[conda] cuda90 1.0 0 pytorch
[conda] mkl 2019.1 144
[conda] mkl 2019.0
[conda] mkl_fft 1.0.10 py36h14836fe_0
[conda] mkl_random 1.0.2 py36h343c172_0
[conda] pytorch 0.4.1 py36_cuda90_cudnn7he774522_1 pytorch

Additional context

published on Monday, December 21, 2020

You are probably aware that pickle.load can execute arbitrary code and
must not be used for untrusted data.

This post is not about that.

In fact, pickle.load can’t even really be trusted for trusted data. To
demonstrate the issue, consider this simple program:

import os, pickle, threading

message = '0' * 65537

recv_fd, send_fd = os.pipe()
read_end = os.fdopen(recv_fd, 'rb', 0)
write_end = os.fdopen(send_fd, 'wb', 0)

thread = threading.Thread(
    target=pickle.dump,
    args=(message, write_end, -1))
thread.start()

pickle.load(read_end)

This simply transmits a pickled message over a pipe over a pipe.
Looks innocuous enough, right?

Wrong! The program fails with the following traceback every time:

Traceback (most recent call last):
  File "<...>/example.py", line 16, in <module>
    pickle.load(read_end)
_pickle.UnpicklingError: pickle data was truncated

Worse: once you get this error, there is safe way to resume listening for
messages on this channel, because you don’t know how long the first message
really was, and hence, at which offset to resume reading. If you try this, you
invite evil into your home. A typical result of trying to continue reading
messages on the stream may be _pickle.UnpicklingError: unpickling stack
underflow, but I’ve even seen segfaults occur.

The reason that we get the error in the first place is of course that the
message size above the pipe capacity, which is 65,536 on my system. The
threshold at which you start getting errors may of course be different for
you. Try increasing the message size if you don’t see errors at first.

If you are using a channel other than os.pipe(), you might be safe – but I
can’t give any guarantees on that. I just can say that I wasn’t able to
reproduce the error on my system when exchanging the pipe for a socket or
regular file.

We used a thread here to send us the data, but it doesn’t matter if the remote
end is a thread or another process. Also, this is not limited to a specific
python version, or version of the pickle protocol. I could reproduce the same
error with several python versions up to python 3.9, and protocols 1-5.

- read_end = os.fdopen(recv_fd, 'rb', 0)
+ read_end = os.fdopen(recv_fd, 'rb')

import pickle
from struct import Struct

HEADER = Struct("!L")

def send(obj, file):
    """Send a pickled message over the given channel."""
    payload = pickle.dumps(data, -1)
    file.write(HEADER.pack(len(payload)))
    file.write(payload)

def recv(file):
    """Receive a pickled message over the given channel."""
    header = read_file(file, HEADER.size)
    payload = read_file(file, *HEADER.unpack(header))
    return pickle.loads(payload)

def read_file(file, size):
    """Read a fixed size buffer from the file."""
    parts = []
    while size > 0:
        part = file.read(size)
        if not part:
            raise EOFError
        parts.append(part)
        size -= len(part)
    return b''.join(parts)

This entry was tagged

bug,
coding and
python