Imperva Cyber Community

communities_1.jpg
 View Only

  • 1.  Cluster activation problem

    Posted 09-06-2022 16:59
    Hi everyone.

    I am configuring a DAM on-premise in version 14.7 on phisycal appliances.
    Is a very basic configuration: 2 MXs M170 (in mxha) and 2 GWs X6520.
    MXHA is working OK. GWs were configured against mxha-VIP. 
    I set the 2 GWs in a cluster (recommended by support team), so GW1 is Manager1 and MX2 is Manager2.
    I am using Dual-Separate Agent Nework: Cluster/Management in eth0. AgentsNetwork in eth1

    The issue is that the cluster fails to activate.
    Initially I use the v14.8 for installation, and the issue was the same.
    Someone in support told me that the problem could be the version, and recommended a downgrade to v14.7, I did it via USB.
    However the issue is the same. Right now the version is 14.7.1.31_0 in four appliances.
     
    Here the errors during cluster activation:
    06/09/2022 16:21:06.629997 [NOTIFICATION] DamGateway.h:72 [DamGateway] Parameters for setting IPU: Appliance model: X6520. Sonar only mode: false
    06/09/2022 16:21:06.630005 [warning] DamGateway.h:136 [DamGateway] Ignoring MAX IPU from MX (60000)
    06/09/2022 16:21:06.630010 [NOTIFICATION] DamGateway.h:139 [DamGateway] MAX IPU: 19400
    06/09/2022 16:21:06.630071 [warning] DacmSynchronizer.cpp:255 DacmSynchronizer::internalSynch - Synchronizer has not been activated
    06/09/2022 16:21:06.630142 [warning] DacmSynchronizer.cpp:255 DacmSynchronizer::internalSynch - Synchronizer has not been activated
    06/09/2022 16:21:06.630150 [NOTIFICATION] DacmSynchronizer.cpp:333 DacmSynchronizer::createControlAndFlow - activate DacmSynchronizer of GW ID num: P03NARDO (12253708515795469473)
    06/09/2022 16:21:06.630252 [NOTIFICATION] DamAgentsClusterManagerMaster.cpp:146 setActiveManagerAndActivate - Active manager Empty GW
    06/09/2022 16:21:07.307299 [warning] Thread.cpp:747 setCurrentThreadName: failed to set thread name 'Data Sync Worker #0' for thread 22925 - length is restricted to 16 characters
    06/09/2022 16:21:07.307346 [ERROR] Hades.cpp:200 Call to HADES_IOC_GENERIC failed with error -1 and errno EH_EXIST (1242)
    06/09/2022 16:21:07.307337 [warning] Thread.cpp:747 setCurrentThreadName: failed to set thread name 'Data Sync Thread' for thread 22926 - length is restricted to 16 characters
    06/09/2022 16:21:07.307353 [ERROR] Hades.cpp:2108 hadesObjectCtrl - IOCTL failed, object type 148 buff size 46
    06/09/2022 16:21:07.307363 [ERROR] DataSyncMember.cpp:437 start: start data sync failed
    06/09/2022 16:21:07.307418 [ERROR] GatewayClusterMember.cpp:1007 failed activate Data Sync - failed to start Data Sync Member
    06/09/2022 16:21:07.307435 [ERROR] GatewayClusterMember.cpp:262 doClusterActivate: failed to activate Data Sync
    06/09/2022 16:21:07.307456 [ERROR] GatewayClusterMember.cpp:292 Failed to activate cluster object
    06/09/2022 16:21:07.307461 [ERROR] GatewayClusterManager.cpp:567 cluster manager activate failed!
    06/09/2022 16:21:07.342293 [NOTIFICATION] ConfigManager.cpp:6957 Revision 766 was applied successfully in 1 sec
    06/09/2022 16:29:52.207903 [NOTIFICATION] ConfigurationDispatcher.cpp:72 Received configuration notification from server, fetching configuration...
    06/09/2022 16:29:52.254939 [NOTIFICATION] ConfigManager.cpp:6925 Started applying configuration revision 767 (incremental update)
    06/09/2022 16:29:52.255476 [NOTIFICATION] Gateway.cpp:7163 GatewayGroup configuration:

    Unfortunately I can't find documentation about HADES errors.
    Anyone have seen these errors before ?
    How did you solve them ?
    Please help. I need to finish this configuration this week

    Elfego
    #DatabaseActivityMonitoring

    ------------------------------
    Elfego
    ------------------------------


  • 2.  RE: Cluster activation problem

    Posted 09-07-2022 04:47
    Hi Elfego,

    I'll try to flag this with some of our DAM users. @Angfe Landagan, @Olgerta Prendi @Karol Gruszczynski @Cezmi Cal - do you have any insight here?

    I also wanted to draw your attention to this thread to see if @Robert Miller's post here is helpful in your query.

    Thanks,

    Sarah​​​​​​​

    ------------------------------
    Sarah Lamont
    Digital Community Manager
    ------------------------------

    Original Message


  • 3.  RE: Cluster activation problem

    Posted 09-07-2022 06:04
    Hi,


    First, I would check the license.
    Cluster Gateway it's a licensed product.
    Check it on Admin -> licensing



    ------------------------------
    Karol Gruszczynski
    IT Security Expert
    Trafford IT Sp. z o.o.
    Warsaw
    ------------------------------

    Original Message


  • 4.  RE: Cluster activation problem

    Posted 09-07-2022 09:39
    Thanks Karol for your answer..
    Licenses are OK. License was the first thing support reviewed.
    We only have 2 GWs
    Also, I already checked the requirements for cluster activation (2 mgrs, subnets, listeners, topology, etc), and everything looks fine.



    ------------------------------
    Elfego
    ------------------------------

    Original Message


  • 5.  RE: Cluster activation problem

    Posted 09-07-2022 07:30
    Hello,
    From the documentation, i would highly recommend to have the version to 14.5 after checking all the needed licenses (MX and GW).

    Couple of logs to look for at least, depending on your preference;
    1.) From the cli;
        impctl show log | grep ERROR   
       impctl show log | grep "Data Sync"

    2. Or automate the gateway registration;

    Automating Cluster Creation
    In order to automatically set up a Gateway and register it to a Cluster, add following lines to the automatic FTL script, after the FTL line:

    impctl stop --teardown --transient
    impctl gateway unregister
    impctl gateway cluster config --cluster-port=<port> --cluster-interface=eth0
    impctl platform config --staging-asset-tag=<Gateway model from auto-ftl>
    impctl sniffing config --delete-blocking-interface
    impctl gateway sniffing config --delete-blocking-interface
    impctl gateway register
    impctl service start --prepare --transient gateway

    Thanks,
    A
    Original Message


  • 6.  RE: Cluster activation problem

    Posted 09-07-2022 10:36
    Thanks Angfe for your tips..
    1. Logs say nothing: After trying activation (in Gws): 

    [root@P03TULIPAN ~]# impctl show log | grep ERROR
    [root@P03TULIPAN ~]# impctl show log | grep "Data Sync"
    [root@P03TULIPAN ~]#

    Running first command with option -i, I see some errors, but they are old. I think it was while I was restarting GWs services.

    root@P03TULIPAN ~]# impctl show log | grep -i error
    ...
    root@P03TULIPAN ~]# impctl show log | grep -i error
    impctl_legacy --no-trace gateway start --from-watchdog WARNING gateway_state: no reply from "http_get_trust --gw --timeout=10 /isalive". Error message: "04/09/2022 22:53:18.092322 [log_warning] /mnt/workspace/ci-dam_v14.7_P31-pipeline-dam/Gateway/src/argus/network/TcpConnection.cpp:424 Failed to connect to 10.3.3.99:443 error_num : 111 poll err :1
    impctl_legacy --no-trace gateway start --from-watchdog WARNING 04/09/2022 22:53:18.092437 [log_error] /mnt/workspace/ci-dam_v14.7_P31-pipeline-dam/Gateway/src/argus/network/SslConnection.cpp:986 Cannot connect. Error is 5
    impctl_legacy --no-trace gateway start --from-watchdog WARNING 04/09/2022 22:53:18.092464 [log_error] /mnt/workspace/ci-dam_v14.7_P31-pipeline-dam/Gateway/src/argus/baseutil/HttpFileClient.cpp:360 Cannot send/receive request/response. Error number is:1 Retried 1 times Max retries is 1
    impctl_legacy --no-trace gateway start --from-watchdog WARNING 04/09/2022 22:53:18.092495 [log_error] /mnt/workspace/ci-dam_v14.7_P31-pipeline-dam/Gateway/src/argus/gwlib/TrustUtils.cpp:1220 platformGetAction - failed to perform Http Get action to the Gateway on URL- /isalive.".""

    2.
    About automation of GW registration, I'm sorry, but I don't see what the benefit should be (automating it)
    Do you think running those commands manually, would be a good idea ? 

    Finally, I could downgrade to v14.5, however the appliances are in a collocation, and we need to get authorized for access.
    This would be the third time asking for access to downgrade. My only concern is that nothing guarantees tthat it will work.

    I initially installed v14.8 recommended by support. Then updated to v14.9, It didn't worked. After that, downgrade to v14.7 (current version). Same error.
    Now, another downgrade to 14.5..

    Best regards

    ------------------------------
    Elfego
    ------------------------------

    Original Message