Use a memoryview to wrap your bytearray:

buf = bytearray(toread)
view = memoryview(buf)
while toread:
    nbytes = sock.recv_into(view, toread)
    view = view[nbytes:] # slicing views is cheap
    toread -= nbytes