s3put对于大文件“对等连接重置”失败

我可以使用相同的命令很好地上传小文件,但每当我尝试上传一个5 GB的文件到S3然后s3脚本失败:

[backup]$ s3put --bucket john-murdoch-bucket --reduced --debug 2 --callback 100 data.tgz /usr/bin/s3put:45: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 err.message[len('No module named '):] + \ send: 'GET /?&max-keys=0 HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Tue, 11 Jun 2013 07:00:45 GMT\r\nContent-Length: 0\r\nAuthorization: AWS <snip>=\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: x-amz-id-2: <snip> header: x-amz-request-id: 1E11AD8B3C93E6B5 header: Date: Tue, 11 Jun 2013 07:00:46 GMT header: Content-Type: application/xml header: Transfer-Encoding: chunked header: Server: AmazonS3 Copying /backup/data.tgz to john-murdoch-bucket/backup/data.tgz send: 'PUT /backup/data.tgz HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nContent-Length: 5522978068\r\nExpect: 100-Continue\r\nx-amz-storage-class: REDUCED_REDUNDANCY\r\nDate: Tue, 11 Jun 2013 07:02:15 GMT\r\nContent-MD5: idp0nQ+9y5kshw0AjL9zMw==\r\nContent-Type: application/x-tar\r\nAuthorization: AWS <snip>=\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n \r\n' 0 bytes transferred / 5522978068 bytes total 0 bytes transferred / 5522978068 bytes total 0 bytes transferred / 5522978068 bytes total 0 bytes transferred / 5522978068 bytes total 0 bytes transferred / 5522978068 bytes total 0 bytes transferred / 5522978068 bytes total 0 bytes transferred / 5522978068 bytes total Traceback (most recent call last): File "/usr/bin/s3put", line 374, in <module> main() File "/usr/bin/s3put", line 371, in main headers=headers) File "/usr/bin/s3put", line 216, in singlepart_upload k.set_contents_from_filename(fullpath, *kargs, **kwargs) File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1241, in set_contents_from_filename encrypt_key=encrypt_key) File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1172, in set_contents_from_file chunked_transfer=chunked_transfer, size=size) File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 710, in send_file chunked_transfer=chunked_transfer, size=size) File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 882, in _send_file_internal query_args=query_args) File "/usr/lib/python2.6/site-packages/boto/s3/connection.py", line 544, in make_request override_num_retries=override_num_retries) File "/usr/lib/python2.6/site-packages/boto/connection.py", line 939, in make_request return self._mexe(http_request, sender, override_num_retries) File "/usr/lib/python2.6/site-packages/boto/connection.py", line 901, in _mexe raise e socket.error: [Errno 104] Connection reset by peer 

我错过了一些其他工具或命令行选项s3put会做多个重试,或上传在多个块,等等?

PS我只是做“sudo yum升级python – 博托”,但它并没有改变任何东西。

更新 – 尝试s3cmd 1.0.0:

 [ec2-user@ip-10-212-234-60 backup]$ s3cmd put data.tgz s3://john-murdoch-bucket/data.tgz data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1] 36864 of 5522978068 0% in 1s 20.31 kB/s failed WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe) WARNING: Retrying on lower speed (throttle=0.00) WARNING: Waiting 3 sec... data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1] 233472 of 5522978068 0% in 1s 144.89 kB/s failed WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe) WARNING: Retrying on lower speed (throttle=0.01) WARNING: Waiting 6 sec... data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1] 45056 of 5522978068 0% in 1s 27.88 kB/s failed WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe) WARNING: Retrying on lower speed (throttle=0.05) WARNING: Waiting 9 sec... data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1] 36864 of 5522978068 0% in 0s 45.52 kB/s failed WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe) WARNING: Retrying on lower speed (throttle=0.25) WARNING: Waiting 12 sec... data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1] 20480 of 5522978068 0% in 1s 19.90 kB/s failed WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe) WARNING: Retrying on lower speed (throttle=1.25) WARNING: Waiting 15 sec... data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1] 12288 of 5522978068 0% in 2s 4.63 kB/s failed ERROR: Upload of 'data.tgz' failed too many times. Skipping that file. 

你需要pip install filechunkio ,然后使用--multipart选项。