记一次Python操作HDFS的采坑

小鱼儿 2022-03-15 10:12 421阅读 0赞

最近在使用Python hdfs模块上传文件的时候，出现了一个错误：

Python3代码：

from hdfs.client import Client
    
    
    class Hdfs(object):
        """docstring for Hdfs"""
    
        def __init__(self, username):
            super(Hdfs, self).__init__()
            self.client = Client("http://192.168.125.130:50070")
    
    
        # 上传文件
        def uploadFile(self, hdfsPath, localPath):
            self.client.upload(hdfsPath, localPath, cleanup=True)
    
        # 列出目录下的文件
        def listDir(self, hdfsPath):
            return self.client.list(hdfsPath, status=True)
    
    
    if __name__ == "__main__":
        hdfsClient = Hdfs(username='root')
        hdfsClient.uploadFile('/yunpan/Ricky', 'settings.py')
        print(hdfsClient.listDir('/yunpan/'))

Traceback (most recent call last):
      File "D:\Python\PythonCode\Flask\yunpan\hadoop\hdfsUntil.py", line 35, in <module>
        hdfsClient.uploadFile('/yunpan/Ricky', 'settings.py')
      File "D:\Python\PythonCode\Flask\yunpan\hadoop\hdfsUntil.py", line 22, in uploadFile
        self.client.upload(hdfsPath, localPath, cleanup=True)
      File "D:\Python\lib\site-packages\hdfs\client.py", line 611, in upload
        raise err
      File "D:\Python\lib\site-packages\hdfs\client.py", line 600, in upload
        _upload(path_tuple)
      File "D:\Python\lib\site-packages\hdfs\client.py", line 530, in _upload
        self.write(_temp_path, wrap(reader, chunk_size, progress), **kwargs)
      File "D:\Python\lib\site-packages\hdfs\client.py", line 476, in write
        consumer(data)
      File "D:\Python\lib\site-packages\hdfs\client.py", line 468, in consumer
        data=(c.encode(encoding) for c in _data) if encoding else _data,
      File "D:\Python\lib\site-packages\hdfs\client.py", line 214, in _request
        **kwargs
      File "D:\Python\lib\site-packages\requests\sessions.py", line 512, in request
        resp = self.send(prep, **send_kwargs)
      File "D:\Python\lib\site-packages\requests\sessions.py", line 622, in send
        r = adapter.send(request, **kwargs)
      File "D:\Python\lib\site-packages\requests\adapters.py", line 463, in send
        low_conn.endheaders()
      File "D:\Python\lib\http\client.py", line 1234, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "D:\Python\lib\http\client.py", line 1026, in _send_output
        self.send(msg)
      File "D:\Python\lib\http\client.py", line 964, in send
        self.connect()
      File "D:\Python\lib\site-packages\urllib3\connection.py", line 196, in connect
        conn = self._new_conn()
      File "D:\Python\lib\site-packages\urllib3\connection.py", line 180, in _new_conn
        self, "Failed to establish a new connection: %s" % e)
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x000001F42B28C550>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed

提示上传文件失败。。。

在百度了很久之后，无果，最后在Google上找到了答案，还是Google大法好啊，百度是真的垃圾。。。

原因是Windows找不到Hadoop的hostname，需要修改Windows的host文件，在C:\\Windows\\System32\\drivers\\etc\\hosts添加如下内容：

192.168.125.130 master

其中192.168.125.130为Hadoop的节点IP，master为hostname，因为我是单节点的，所以只填了namenode的hostname和IP，如果有多个节点需要将这些节点的IP和hostname添加到hosts文件中。

最后，上面的代码在Linux系统上是可以成功运行的，因为在Linux上配置好了Haoop就已经配置好了hosts文件。

附上原文地址：[https://www.smwenku.com/a/5b892b272b71775d1ce055f3/zh-cn][https_www.smwenku.com_a_5b892b272b71775d1ce055f3_zh-cn]

[https_www.smwenku.com_a_5b892b272b71775d1ce055f3_zh-cn]: https://www.smwenku.com/a/5b892b272b71775d1ce055f3/zh-cn