To execute detection task in spark, following properties need to be configured in spark.properties file available at the following location ‘.../HDFSAgent/expandedArchive/WEB-INF/classes’, once an IDP is deployed.
This property specifies initial number of executors. The default value is set to 2g.
This property specifies the number of cores to use on each executor. The default value is set to 2g.
This property specifies the amount of memory to use per executor process. The default value is set to 1g.
This property specifies the amount of memory to use for the driver process. The default value is set to 1g.
This property enables the external shuffle service. This service preserves shuffle files written by executors.
This property specifies the port number on which external shuffle service will run.
This property specifies whether to use dynamic resource allocation, which scales the number of executors registered with the application, up and down based on the workload.
This property specifies the deployment mode of spark driver program, either ‘client’ or ‘cluster’, which means to launch driver program locally (client) or remotely (cluster) on one of the nodes, inside the cluster.
This property specifies the directory to use for scratch space in spark, including map output files and RDDs that get stored on the disk.
This property specifies the name of the yarn queue to which the application is submitted.
This property contains an archive of Spark jars for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory.
This property contains the comma-separated list of jars to be placed in the working directory of each executor.
*Note: Uncomment the following properties to perform detection on Kerberized Clusters.
This property contains the principal to be used to login to Kerberos Domain Controller, while running on secure HDFS. This property is commented by default, which can be uncommented and configured when Kerberos authentication is enabled.
This property contains full path to the file that contains the keytab for the principal specified above. This property is commented by default, which can be uncommented and configured when Kerberos authentication is enabled.