linux + port in use + but no PID return from netstat to kill the PID

we have Hadoop cluster – the linux are rhel 7.5 machines

and namenode is using the port 50070

we can see from the log that port 50070 is in use

but the interesting thing is that when we do netstat -tulpn | grep 50070 , to find the PID , the its return nothing

netstat -tulpn | grep 50070   ( we not get output ) 

so how it can be ?

how to clear the port?

2020-07-18 21:26:22,753 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - NameNode metrics system shutdown complete.
2020-07-18 21:26:22,753 ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode.
java.net.BindException: Port in use: linux.gg.com:50070
        at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1001)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1023)
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1080)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:937)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:942)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:755)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
        at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:988)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1019)
        ... 9 more
2020-07-18 21:26:22,755 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2020-07-18 21:26:22,757 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at 
************************************************************/
[[email protected] hdfs]#
[[email protected] hdfs]#
[[email protected] hdfs]# netstat -tulpn | grep 50070 ( no PID number is returned ) 

Answer

The messages indicate the problem is with a HTTP server that belongs to Hadoop. Port 50070 is the default for HDFS web UI in Hadoop 2.7.1 and above, I think.

With netstat -tulpn, you are looking at ports listening for incoming TCP connections. And since the problem is with Hadoop’s HTTP server, you don’t need to look at UDP ports at all, since HTTP will only use a TCP port.

But since the port number is so high, it could be occupied by an outgoing connection instead. Try netstat -tapn | grep 50070 instead.

To see the range of ports that can be dynamically allocated for outgoing connections, run cat /proc/sys/net/ipv4/ip_local_port_range. You can use net.ipv4.ip_local_port_range = min_value max_value in /etc/sysctl.conf[.d] to adjust the range, but restricting the range on a busy server that has a lot of outgoing connections might not be a good idea. The default range on my Debian 10 is from port 32768 to 60999; enterprise distributions may use an expanded range by default.

Instead, you might want to choose a non-default port for this HDFS web UI, that is outside the range of ports used for outgoing connections. The property dfs.namenode.http-address in hdfs.xml has a default value of 0.0.0.0:50070 if it is not set. You could set that property with a value of 0.0.0.0:<some_other_port>.

In other words, to set the port number to e.g. 32070 instead, you could add this to your hdfs.xml:

<property>
    <name>dfs.namenode.http-address</name>
    <value>0.0.0.0:32070</value>
</property>

The 0.0.0.0 means “any IP address the system running the web UI has.” You could replace it with an IP address, if the system has multiple network connections with different addresses and you want the HDFS web UI to be reachable in a single IP address only.

Of course, you’ll also need to document that the HDFS web UI is now in a non-default port, so that the administrators who need the web UI functionality will be able to find it.

Leave a Reply

Your email address will not be published. Required fields are marked *