Skip to main content

Appendix F: Detection in Kerberized Clusters

To execute detection task in spark, following properties need to be configured in spark.properties file available at the following location ‘.../HDFSAgent/expandedArchive/WEB-INF/classes’, once an IDP is deployed.

  1. spark.executor.instances
    This property specifies initial number of executors. The default value is set to 2g.

  2. spark.executor.cores
    This property specifies the number of cores to use on each executor. The default value is set to 2g.

  3. spark.executor.memory
    This property specifies the amount of memory to use per executor process. The default value is set to 1g.

  4. spark.driver.memory
    This property specifies the amount of memory to use for the driver process. The default value is set to 1g.

  5. spark.shuffle.service.enabled
    This property enables the external shuffle service. This service preserves shuffle files written by executors.

  6. spark.shuffle.service.port
    This property specifies the port number on which external shuffle service will run.

  7. spark.dynamicAllocation.enabled

    This property specifies whether to use dynamic resource allocation, which scales the number of executors registered with the application, up and down based on the workload.

  8. spark.submit.deployMode

    This property specifies the deployment mode of spark driver program, either ‘client’ or ‘cluster’, which means to launch driver program locally (client) or remotely (cluster) on one of the nodes, inside the cluster.

  9. spark.local.dir

    This property specifies the directory to use for scratch space in spark, including map output files and RDDs that get stored on the disk.

  10. spark.yarn.queue

    This property specifies the name of the yarn queue to which the application is submitted.

  11. spark.yarn.archive
    This property contains an archive of Spark jars for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory.

  12. spark.yarn.dist.jars
    This property contains the comma-separated list of jars to be placed in the working directory of each executor.

*Note: Uncomment the following properties to perform detection on Kerberized Clusters.

  1. spark.yarn.principal
    This property contains the principal to be used to login to Kerberos Domain Controller, while running on secure HDFS. This property is commented by default, which can be uncommented and configured when Kerberos authentication is enabled.

  2. spark.yarn.keytab
    This property contains full path to the file that contains the keytab for the principal specified above. This property is commented by default, which can be uncommented and configured when Kerberos authentication is enabled.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.