Categories

Versions

You are viewing the RapidMiner Hub documentation for version 10.0 - Check here for latest version

Table of system settings for RapidMiner Job Agent

This table gives an overview over all properties which can be set via environment variables. This application is a Spring Boot application and therefore those property values can be also applied, e.g. SPRING_APPLICATION_NAME=Test would change the Spring's application name to Test.

Other properties not listed here should not be changed and have reasonable defaults.

Property Default Description
JOBAGENT_APP_ID A unique ID generated on first start and persisted in $job-agent-home/config/.id.properties. This ID file must be located on a persisted storage.
JOBAGENT_BASE_DIR /jobagent (docker image) The base directory of the Job Agent, other directory paths are derived from this base directory.
JOBAGENT_HOME_DIR $baseDir/home The Job Agent's home directory
JOBAGENT_LOG_DIR $baseDir/log The Job Agent's log directory
JOBAGENT_DATA_DIR $baseDir/data The Job Agent's data directory
JOBAGENT_JOBS_DIR $dataDir/jobs The Job Agent's jobs directory which stores information related to each job the Job Agent picks up
JOBAGENT_CONTAINERS_DIR $dataDir/containers The Job Agent's containers which stores information related to each spawned Job Container
JOBAGENT_QUEUE_ACTIVEMQ_USERNAME Define broker user
JOBAGENT_QUEUE_ACTIVEMQ_PASSWORD Define broker password
JOBAGENT_AUTH_REALM AIHub Realm to use for Keycloak
JOBAGENT_AUTH_AUTH_SERVER_URL URL to Keycloak
JOBAGENT_AUTH_SERVICE_CLIENT_SECRET The secret for aihub-jobagent Keycloak client
JOBAGENT_SYNC_ENABLED true If synchronization behavior (retrieving EXTENSIONS, JDBC, LIBS, LICENSES and the EXECUTION_CONTEXT from main application) is enabled.
JOBAGENT_SYNC_EXTENSIONS_ENABLED true If synchronization behavior is enabled for EXTENSIONS.
JOBAGENT_SYNC_EXTENSIONS_CLEAN_BEFORE true If synchronization behavior should clean local files for EXTENSIONS before fetching them from remote.
JOBAGENT_SYNC_JDBC_ENABLED true If synchronization behavior is enabled for JDBC.
JOBAGENT_SYNC_JDBC_CLEAN_BEFORE true If synchronization behavior should clean local files for JDBC before fetching them from remote.
JOBAGENT_SYNC_LIBS_ENABLED true If synchronization behavior is enabled for LIBS.
JOBAGENT_SYNC_LIBS_CLEAN_BEFORE true If synchronization behavior should clean local files for LIBS before fetching them from remote.
JOBAGENT_SYNC_LICENSES_ENABLED true If synchronization behavior is enabled for LICENSES.
JOBAGENT_SYNC_LICENSES_CLEAN_BEFORE true If synchronization behavior should clean local files for LICENSES before fetching them from remote.
JOBAGENT_SYNC_EXECUTION_CONTEXT_ENABLED true If synchronization behavior is enabled for EXECUTION_CONTEXT.
JOBAGENT_SYNC_EXECUTION_CONTEXT_CLEAN_BEFORE true If synchronization behavior should clean local files for EXECUTION_CONTEXT before fetching them from remote.
JOBAGENT_SYNC_BASE_URL sync/ The base URL for synchronization behavior.
JOBAGENT_SYNC_HEALTH_CHECK_URL $baseUrl/healthcheck The health check URL for synchronization behavior, derived from base URL.
JOBAGENT_SHUTDOWN_CHECK_DELAY 1000 Fixed delay in milliseconds for shutdown event checks
JOBAGENT_RESTART_POLICY NONE Defines when and how frequently underlying Job Containers, possible values are NONE, CRON and JOBCOUNT
JOBAGENT_RESTART_MAX_JOBS Amount of executed jobs before Job Container is restarted, related to JOBCOUNT
JOBAGENT_RESTART_CRON_EXPRESSION Cron expression to determine when Job Container is restarted in a scheduled manner, related to CRON
JOBAGENT_RESTART_TIMEOUT Timeout for restarting a container. Job container is killed when the value is reached.
JOBAGENT_CONTAINER_PROCESS_LOGGING_OVERRIDE_ENABLED false If overriding process log level during execution is enabled or not
JOBAGENT_CONTAINER_PROCESS_LOGGING_OVERRIDE_LEVEL ALL If process log level overriding is enabled, this defines the verbosity, e.g. ALL
JOBAGENT_CONTAINER_CONFIG_DIR $job-agent-home-dir/config Config location for the Job Container
JOBAGENT_CONTAINER_LIBS_DIR $job-agent-base-dir/lib/container Library location for the Job Container used to determine what is loaded into the classpath of spawned Job Containers
JOBAGENT_CONTAINER_JAR_DIR $job-agent-base-dir/lib/container/rapidminer-jobcontainer.jar JAR file location to the Job Container
JOBAGENT_CONTAINER_REPOSITORY_TYPE remote Type of the internally loaded repository (remote) for projects. Should not be changed.
JOBAGENT_CONTAINER_REPOSITORY_NAME _LOCAL Name of the internally registered repository. Should not be changed.
JOBAGENT_CONTAINER_REPOSITORY_CACHING_STRATEGY ROTATE Defines the caching strategy of already seen Projects inside the Job Container. Possible values are NONE (no caching), ROTATE (evict latest used), CRON (depending on a fixed schedule), IDLE (depending on an idle time).
JOBAGENT_CONTAINER_REPOSITORY_CACHING_ROTATE_COUNT 2 Used for ROTATE caching strategy and defines how many Projects are kept before the last used is evicted.
JOBAGENT_CONTAINER_REPOSITORY_CACHING_CRON_EXPRESSION Used for CRON caching strategy and defines when the cache is evicted.
JOBAGENT_CONTAINER_REPOSITORY_CACHING_IDLE_OFFSET 10000 Used for IDLE caching strategy and defines the offset how long a repository has not been used in milliseconds.
JOBAGENT_CONTAINER_REPOSITORY_CACHING_IDLE_INTERVAL 60 Used for IDLE caching strategy and defines how frequently the Projects are checked in minutes.
JOBAGENT_CONTAINER_REPOSITORY_CACHING_LOCK_BACKOFF [1000, 2000, 3000, 5000, 8000, 16000] List of retry attempts for initially cloning the Project in milliseconds.
JOBAGENT_CONTAINER_REPOSITORY_CACHING_USE_LFS_LIGHTWEIGHT true If files of a Project should only be requested when they're actually required during process execution.
JOBAGENT_CONTAINER_PLUGINS_DIR $job-agent-base-dir/lib/container/plugins The Job Container's plugins directory used for internal RapidMiner bundled extensions
JOBAGENT_CONTAINER_JDBC_DIR $job-agent-home-dir/resources/jdbc The Job Container's JDBC directory
JOBAGENT_CONTAINER_EXTENSIONS_DIR $job-agent-home-dir/resources/extensions The Job Container's extensions directory
JOBAGENT_CONTAINER_CUSTOM_LIBS_DIR $job-agent-home-dir/resources/libs The Job Container's custom library directory
JOBAGENT_CONTAINER_INIT_WITH_PROPERTIES_ENABLED false Enable to make the Job Container read a dedicated file for initialization.
JOBAGENT_CONTAINER_INIT_WITH_PROPERTIES_LOCATION $job-agent-home-dir/config/rapidminer/rapidminer-init.properties Properties file used for Job Container initialization.
JOBAGENT_CONTAINER_COUNT 1 The Job Containers amount the Job Agent will spawn in parallel
JOBAGENT_CONTAINER_MAX_ERROR_AMOUNT_BEFORE_SPAWN 5 Number to determine how often a Job Container can not respond before the Job Agent will forcefully restart it.
JOBAGENT_CONTAINER_MAX_TIME_BETWEEN_ERRORS 5000 Maximum time (in milliseconds) between errors during the health check
JOBAGENT_CONTAINER_LISTEN_SCHEME http Listen scheme of the Job Containers spawned by the Job Agent
JOBAGENT_CONTAINER_LISTEN_ADDRESS localhost Listen address of the Job Containers spawned by the Job Agent
JOBAGENT_CONTAINER_LISTEN_PORT_RANGE_START 10000 Listen port range of the Job Containers spawned by the Job Agent, are incremented by one for each additional Job Container, see COUNT property
JOBAGENT_CONTAINER_STARTUP_TIMEOUT 180000 Maximum time to wait (in milliseconds) before a Job Container is marked as not started successfully
JOBAGENT_CONTAINER_MEMORY_LIMIT 2048 Memory in MB limit for each Job Container
JOBAGENT_CONTAINER_USE_SYSTEM_TEMP_DIR false By default, the Job Container's temporary directory is located inside it's working directory and does not use the system's temporary folder
JOBAGENT_CONTAINER_JVM_PROPERTIES_DEFAULTS Xms521m,XX:+UseG1GC,XX:G1HeapRegionSize=32m,XX:-HeapDumpOnOutOfMemoryError,Djava.awt.headless=true,Dlog4j2.formatMsgNoLookups=true List of default JVM arguments passed to each Job Container. For additions, use JVM_CUSTOM_OPTIONS.
JOBAGENT_CONTAINER_JVM_CUSTOM_OPTIONS A simple line to add new JVM arguments passed to each Job Container, e.g. -Dmyproperty=value. It's recommended to use this property to transpose additional arguments to the Job Container.
JOBAGENT_CONTAINER_JVM_CUSTOM_PROPERTIES List of custom JVM arguments passed to each Job Container. It's recommended to use JVM_CUSTOM_OPTIONS instead.
JOBAGENT_CONTAINER_JVM_PROPERTIES Overwrite the default JVM properties
JOBAGENT_CONTAINER_KILL_FORCIBLY_TIMEOUT 5000 Timeout after a process is killed forcibly if it is not dead yet after receiving the destroy command
JOBAGENT_JOB_STATE_EVENTS_CLEANUP false If old events which have already been marked as sent to the main application, should be cleaned up.
JOBAGENT_JOB_STATE_EVENTS_INTERVAL false Interval in which candidates of already sent events are checked for deletion.
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_STARTING 1000 Time in milliseconds between invoking check for starting Job Containers
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_RUNNING 1000 Time in milliseconds between invoking check for running Job Containers
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_EXECUTING 1000 Time in milliseconds between invoking check for executing Job Containers
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_CAPACITY 1000 Time in milliseconds between invoking capacity check (available Job Containers)
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_UNREACHABLE 1000 Time in milliseconds between invoking check for unreachable Job Containers
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_SPAWN 1000 Time in milliseconds between invoking check for spawning new Job Containers
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_RESTART 1000 Time in milliseconds between invoking check for restarting Job Containers
JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_TOKEN 5000 Time in milliseconds between invoking check token validity of Job Containers
JOBAGENT_SCHEDULED_JOB_STATUS_UPDATE 2000 Time in milliseconds between invoking REST request to Job Containers to fetch the progress of execution
JOBAGENT_SCHEDULED_AGENT_STATE_UPDATE 1000 Time in milliseconds between invoking check of unsent events for jobs and to send them back to the main application
JOBAGENT_SCHEDULED_AGENT_STATUS_UPDATE 10000 Time in milliseconds between invoking check of Job Agent's state and sending it back to the main application (heartbeat)
JOBAGENT_SCHEDULED_AGENT_SYNC_REFETCH 5000 Time in milliseconds between invoking check of a newly arrived sync fetch message
JOBAGENT_SCHEDULED_JOB_TOKEN_UPDATE 240000 Time in milliseconds between invoking renewal of a token
JOBAGENT_SCHEDULED_JOB_STATE_UPDATE 2000 Time in milliseconds between invoking check of unsent events for jobs and to send them back to the main application
JOBAGENT_QUEUE_AGENT_COMMAND __agentCommand ActiveMQ queue for agent commands
JOBAGENT_QUEUE_AGENT_HEARTBEAT __agentHeartbeat ActiveMQ queue for agent heartbeats
JOBAGENT_QUEUE_JOB_REQUEST DEFAULT ActiveMQ queue for job requests
JOBAGENT_QUEUE_JOB_LOG __jobLog ActiveMQ queue for log requests
JOBAGENT_QUEUE_JOB_STATUS __jobUpdate?consumer.exclusive=true ActiveMQ queue for job status updated
JOBAGENT_QUEUE_SYNC_COMMAND __agentCommand ActiveMQ queue for sync commands
JOBAGENT_QUEUE_RECEIVE_TIMEOUT 5000 ActiveMQ queue receive timeout for JMS listeners
JOBAGENT_QUEUE_CACHE_LEVEL 4 ActiveMQ queue cache level for JMS listeners
JOBAGENT_QUEUE_TASK_EXECUTOR_COUNT 3 ActiveMQ queue JMS listener threads
JOBAGENT_QUEUE_ACTIVEMQ_MAXIMUM_REDELIVERIES 2 ActiveMQ queue JMS listener maximum redelivery attempts
JOBAGENT_QUEUE_ACTIVEMQ_MAXIMUM_JOB_REDELIVERIES 2 ActiveMQ queue JMS listener per job maximum redelivery attempts
JOBAGENT_AUDIT_ENABLED false If audit is enabled to see additional information about user changes
LICENSE_LICENSE Directly set a RapidMiner license string
LICENSE_LOCATION By default, points to the parent folder where licenses reside in, e.g. $appHome/resources/licenses inside the Docker container
LOGGING_... Set logging, e.g. LOGGING_LEVEL_COM_RAPIDMINER=DEBUG for debug logs of all RapidMiner classes
BROKER_ACTIVEMQ_HOST localhost Define broker host
BROKER_ACTIVEMQ_PORT 61616 Define broker port
BROKER_ACTIVEMQ_USERNAME Define broker user
BROKER_ACTIVEMQ_PASSWORD Define broker password
BROKER_ACTIVEMQ_URI Define broker URI
AIHUB_CONNECTION_PROTOCOL http Define AI Hub protocol for internal or external connectivity
AIHUB_CONNECTION_HOST localhost Define AI Hub host for internal or external connectivity
AIHUB_CONNECTION_PORT 8080 Define AI Hub port for internal or external connectivity
AIHUB_CONNECTION_CONTEXT_PATH api/v1/ Define AI Hub context path for internal or external connectivity

Attributes available during execution in a Job Agent / Job Container

rapidminer.aihub.connection.protocol=
rapidminer.aihub.connection.host=
rapidminer.aihub.connection.port=
rapidminer.aihub.connection.context-path=
rapidminer.aihub.process.location=
com.rapidminer.auth.jwt.token=
com.rapidminer.aihub.user_id=

Job Container

The Job Container properties are handled in a special way, because they are passed to the Job Container by the Job Agent during start.

All properties prefixed with JOBAGENT_CONTAINER_ are automatically transposed to the Job Container, some are transposed automatically. If you need to pass advanced properties not transposed by the Job Agent itself, please use JOBAGENT_CONTAINER_JVM_CUSTOM_OPTIONS with one of the following properties:

Property Default Description
jobcontainer.process.logging.override.enabled false Allows to override process logging
jobcontainer.process.logging.override.level ALL Set the override log level of the process which ignored the log_vebosity parameter defined in the process
jobcontainer.system-out-log.enabled false Allow to log System.out calls
jobcontainer.register-default-symmetric-encryption-context.enabled true Enabled registration of Project encryption context
jobcontainer.register-default-symmetric-encryption-context.key-location ${container-dir}/.RapidMiner/encryption/symmetric/default-local-context.rmek Default location of context's key
jobcontainer.jobs-cache-size 10 This amount of jobs will be kept in cache also after they finished
jobcontainer.progress-update 1000 Update process trace every second
jobcontainer.init-with-properties.enabled false If RapidMiner should be initialized with additional init properties on Job Container start
jobcontainer.init-with-properties.location Used properties file for init, when enabled
jobcontainer.remote.versioned.backoff [1000, 2000, 3000, 5000, 8000, 16000] Values in milliseconds how long to wait before each try to commit after a job finished
jobcontainer.remote.versioned.auto-commit-strategy OURS Possible values: OURS (keep changes done in the Job Container and merge them,NONE` to skip auto-commit and evict any change done locally (for testing)
jobcontainer.task-scheduler.pool-size 10 Job Container's scheduler used for periodic checks pool size
jobcontainer.task-scheduler.thread-priority 8 Job Container's scheduler used for periodic checks pool size
jobcontainer.memory-check-delay 5000 Time in milliseconds how frequently to check for out of memory of the Job Container
jobcontainer.memory-consumption-threshold 0.95 Threshold when to consider Job Container out of memory in decimal percent
jobcontainer.repository.type remote See Job Agent's explanation.
jobcontainer.repository.caching.strategy ROTATE See Job Agent's explanation.
jobcontainer.repository.caching.rotate-count 2 See Job Agent's explanation.
jobcontainer.repository.caching.lock-backoff [1000, 2000, 3000, 5000, 8000, 16000] See Job Agent's explanation.
audit.enabled false If audit is enabled. See Job Agent's explanation.
logging.* Set logging, e.g. logging.level.com.rapidminer.execution.jobcontainer=DEBUG for debug logs of the Job Container classes itself