Imperva Cyber Community

Expand all | Collapse all

management server fails to start after changing the "system" (database) password

  • 1.  management server fails to start after changing the "system" (database) password

    Posted 11-25-2020 03:46
    After we changed the "system" (database) password and applied the changes, the management server fails to start.
    -----------------------------------------------------
    Pending settings
    P 'system' (database) password: <is-set>

    The changes are being applied ...

    impctl service stop --transient management-server ... [ OK ]
    impctl db config --encoded-password='XXX' ... [ OK ]
    impctl service start --transient management-server ...[ Failed ]

    Error: The transaction has failed. For more information, execute 'impctl show log' for details!, Press <Enter> to continue:

    -----------------------------------------------------

    The 'impctl show log' basically shows these:

    impctl_legacy --no-trace db start INFO
    impctl_legacy --no-trace db start INFO System altered.
    impctl_legacy --no-trace db start INFO
    impctl_legacy --no-trace db start INFO
    impctl_legacy --no-trace db start INFO System altered.
    impctl_legacy --no-trace db start INFO
    impctl_legacy --no-trace db start INFO Database closed.
    impctl_legacy --no-trace db start INFO Database dismounted.
    impctl_legacy --no-trace db start INFO ORACLE instance shut down.
    impctl_legacy --no-trace db start INFO ORACLE instance started.
    impctl_legacy --no-trace db start INFO
    impctl_legacy --no-trace db start INFO Total System Global Area 1603411968 bytes
    impctl_legacy --no-trace db start INFO Fixed Size 2228784 bytes
    impctl_legacy --no-trace db start INFO Variable Size 1493175760 bytes
    impctl_legacy --no-trace db start INFO Database Buffers 100663296 bytes
    impctl_legacy --no-trace db start INFO Redo Buffers 7344128 bytes
    impctl_legacy --no-trace db start INFO Database mounted.
    impctl_legacy --no-trace db start INFO Database opened.
    impctl_legacy --no-trace db start INFO
    impctl_legacy --no-trace db start INFO System altered.
    impctl_legacy --no-trace db start INFO
    impctl_legacy --no-trace db start INFO mutex_unlock: unlocked mutex_name="services/database-server", previously held by pid: 14234 for command: "impctl db start"
    impctl_legacy --no-trace db start INFO watchdog_add_service: added service "database-server".
    impctl_legacy --no-trace server status WARNING server_status: "http_get --timeout=30" returned status 4
    impctl_legacy --no-trace server status WARNING server_status: empty response
    impctl_legacy --no-trace server status WARNING server_status: "http_get --timeout=30" returned status 4
    impctl_legacy --no-trace server status WARNING server_status: empty response
    impctl_legacy --no-trace server start INFO watchdog_delete_service: deleted service "management-server".
    impctl_legacy --no-trace server status WARNING server_status: "http_get --timeout=30" returned status 4
    impctl_legacy --no-trace server status WARNING server_status: empty response
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response

    impctl_legacy server status --timeout=1 WARNING server_status: empty response
    impctl_legacy gateway status --timeout=1 WARNING gateway_state: no reply from "http_get --timeout=1 http://127.0.0.1:80/isalive".
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy server status --timeout=1 WARNING server_status: "http_get --timeout=1" returned status 8
    impctl_legacy server status --timeout=1 WARNING server_status: empty response
    impctl_legacy gateway status --timeout=1 WARNING gateway_state: no reply from "http_get --timeout=1 http://127.0.0.1:80/isalive".
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server stop INFO watchdog_delete_service: deleted service "management-server".
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server stop INFO mutex_trylock: FAILED to get services/management-server, rc = 102
    impctl_legacy --no-trace server stop FATAL [impctl/bin/server/stop:51 main] The "management-server" service is busy (mutex "services/management-server" is held by process id 2383 9 [alive] for command: "impctl server start"). (exit status: 7)
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace db stop INFO watchdog_delete_service: deleted service "database-server".
    impctl_legacy --no-trace db stop INFO db_stop: Stopping listener ...
    impctl_legacy --no-trace db stop DEBUG [impctl/lib/db.sh:58 db_stop]
    impctl_legacy --no-trace db stop DEBUG [impctl/lib/db.sh:58 db_stop] LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 25-NOV-2020 16:30:08
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace db stop DEBUG [impctl/lib/db.sh:58 db_stop]
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace db stop DEBUG [impctl/lib/db.sh:58 db_stop] Copyright (c) 1991, 2011, Oracle. All rights reserved.
    impctl_legacy --no-trace db stop DEBUG [impctl/lib/db.sh:58 db_stop]
    impctl_legacy --no-trace db stop DEBUG [impctl/lib/db.sh:58 db_stop] Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
    impctl_legacy --no-trace db stop DEBUG [impctl/lib/db.sh:58 db_stop] The command completed successfully
    impctl_legacy --no-trace db stop INFO db_stop: Stopping the database
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace db stop INFO mutex_unlock: unlocked mutex_name="services/database-server", previously held by pid: 10596 for command: "impctl db stop"
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8
    impctl_legacy --no-trace server start WARNING server_status: empty response
    impctl_legacy --no-trace server stop INFO mutex_trylock: FAILED to get services/management-server, rc = 102
    impctl_legacy --no-trace server stop FATAL [impctl/bin/server/stop:51 main] The "management-server" service is busy (mutex "services/management-server" is held by process id 2383 9 [alive] for command: "impctl server start"). (exit status: 7)
    impctl_legacy --no-trace server start WARNING server_status: "http_get --timeout=30" returned status 8



    Could anyone advise the solutions? Thanks.


    #On-PremisesWAF(formerlySecuresphere)

    ------------------------------
    Louis WAF
    ------------------------------


  • 2.  RE: management server fails to start after changing the "system" (database) password

    Community Manager
    Posted 11-30-2020 12:32
    @Louis WAF, thanks for asking in the community first. This really does require more analysis on your logs. In this instance, you will need to create a support ticket found here.  ​

    ------------------------------
    Christopher Detzel
    Community Manager
    Imperva
    ------------------------------



  • 3.  RE: management server fails to start after changing the "system" (database) password

    Posted 12-01-2020 20:31
    Hi Christopher,

      Thanks. A case was created and the issue resolved. 


    ------------------------------
    Louis WAF
    ------------------------------



  • 4.  RE: management server fails to start after changing the "system" (database) password

    Community Manager
    Posted 12-02-2020 09:06
    I'm glad it was resolved @Louis WAF. If it's possible to share that resolve here, that would be very helpful to the community folks in the future. Up to you, but would love to see the resolution. ​

    ------------------------------
    Christopher Detzel
    Community Manager
    Imperva
    ------------------------------



  • 5.  RE: management server fails to start after changing the "system" (database) password

    Posted 12-03-2020 02:16
    Hi Christopher,
     
     Sure. It was because the system" (database) password was first changed using the Oracle official syntax. The password was later changed using impctl command. The inconsistent passwords resulted in failure of management server to start. And the issue was resolved after using same password for both Oracle official syntax and impctl command. But the real crux is the password should have not be changed using the Oracle official syntax. The impctl command should only be used instead. 

    ------------------------------
    Louis WAF
    ------------------------------