Hello Guys,
This issue is mysterious issue where NSX's local account gets locked out not for login into NSX GUI but for API operations. Creating this blog because I couldn't find the solution on web. VMware article was there explaining root cause but solution was not there too.
Symptoms
1) Any application or product which works with or uses NSX for network services, will not be able to use NSX to create, remove, update or deletion operations. However, running services will be running fine.
2) You will be able to login NSX portal but will not be able to use APIs. Not even through any other API tool like postman. You might get error like below
3) I uses VMware Cloud Director which uses NSX for networking and security service. On VMware Cloud Director, I could see error like below when you open any Edge Gateway or do any configuration change. Below are the VCD debug logs-
2021-07-12 16:18:52,955 | DEBUG | task-service-activity-pool-128 | NetworkSecurityErrorHandler | Response error: <!doctype html><html lang="en"><head><title>HTTP Status 403 – Forbidden</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 403 – Forbidden</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> This IP address has been blocked temporarily.</p><p><b>Description</b> The server understood the request but refuses to authorize it.</p><hr class="line" /></body></html> | requestId=18cbd3bc-2eaf-42af-b11e-4bce957bff9e,request=POST https://testvcd.com/api/admin/edgeGateway/509ed27c-724d-490b-b0c7-e25d19523017/action/redeploy,requestTime=1626106731682,remoteAddress=172.25.1.21:60592,userAgent=Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (...,accept=application/*+json;version 34.0 vcd=632554b9-0779-4ddc-8b62-be08d1c167f6,task=82c8a4fc-35ba-490a-a235-b4a65a25cace activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:82c8a4fc-35ba-490a-a235-b4a65a25cace)
4) I checked the NSX Manager logs and could see multiple events like below in /usr/appmgmt-webserver/logs/localhost_access_log.2021-07-13.txt.
Where localhost_access_log.2021-07-13.txt. This filename will change as per current date in logs.
172.25.1.239 - - [13/Jul/2021:23:21:38 +0200] "GET /api/2.0/services/ipset/ipset-29 HTTP/1.1" "https-jsse-nio-443-exec-2292" 403 649 2234
172.25.1.239 - - [13/Jul/2021:23:21:38 +0200] "GET /api/2.0/services/ipset/ipset-106 HTTP/1.1" "https-jsse-nio-443-exec-2334" 403 649 2241
172.25.1.239 - - [13/Jul/2021:23:21:38 +0200] "GET /api/versions HTTP/1.1" "https-jsse-nio-443-exec-2366" 403 649 2243
172.25.1.239 - - [13/Jul/2021:23:21:38 +0200] "GET /api/versions HTTP/1.1" "https-jsse-nio-443-exec-2325" 403 649 2224
Yellow - This is IP address which is trying to make a connection with NSX Manager which is failing
Red - This is error code 403 which means access forbidden
Above logs means, IP address (172.25.1.239) is trying connect with NSX Manager but NSX Manager is not allowing access to connect.
Root Cause
When any application (Monitoring or non-monitoring) is configured with incorrect username and password to connect with NSX Manager then as a security feature NSX Manager blacklist that username after certain invalid authentication attempts (by default it is 10 but can be modified). Any user too can do this because someone might be trying to guess the password! be aware of that too ;)
Solution
Before correcting it, you must identify which application or user is doing this. Because if post correction, if still invalid attempts are coming then it will again block the user account. How will you identify that's your headache but you can reach me out :)
Once identified then follow below steps
1. Login NSX Manager CLI with admin
2. enter into enable mode with commad
#enable
3. Enter into engineering mode with command↓
#st en
Press Y when asked and use password : IAmOnThePhoneWithTechSupport
4. then edit /home/secureall/secureall/sem/WEB-INF/spring/vsmconfig.properties and look for values as below in this file
#Denotes whether blacklisting is enabled
blacklist.enabed=true
You will find it true.
5. To resolve this issue, first you need to make it false that is from
blacklist.enabed=true
↓
blacklist.enabed=false
and then save the file
6. Reboot NSX Manager now with below command
#reboot
7. Once rebooted, you will find that NSX account which you used for integration with other application and which was locked out is now responding over API requests and will give you output as below now.
8. In step 5, you have changed the blacklisting configuration which can be a security vulnerability. As soon as this issue fix, then you need to revert it back to true so that future invalid attempts gets blocked.
9. Post making it true, reboot NSX Manager. Please note that everytime you change it, you have to reboot NSX manager to make it effective.
Don't forget to comment if it was useful for you. Cheers!
0 Comments:
Post a Comment