config.txt
上传用户:quxuerui
上传日期:2018-01-08
资源大小:41811k
文件大小:7k
- HOD Configuration
- =================
- 1. Introduction:
- ================
- Configuration options for HOD are organized as sections and options
- within them. They can be specified in two ways: a configuration file
- in the INI format, and as command line options to the HOD shell,
- specified in the format --section.option[=value]. If the same option is
- specified in both places, the value specified on the command line
- overrides the value in the configuration file.
- To get a simple description of all configuration options, you can type
- hod --verbose-help
- This document explains some of the most important or commonly used
- configuration options in some more detail.
- 2. Sections:
- ============
- The following are the various sections in the HOD configuration:
- * hod: Options for the HOD client
- * resource_manager: Options for specifying which resource
- manager to use, and other parameters for
- using that resource manager
- * ringmaster: Options for the RingMaster process,
- * hodring: Options for the HodRing processes
- * gridservice-mapred: Options for the MapReduce daemons
- * gridservice-hdfs: Options for the HDFS daemons.
- The following are some of the important options in the HOD
- configuration:
- 3. Important / Commonly Used Configuration Options:
- ===================================================
- 3.1. Common configuration options:
- ----------------------------------
- Certain configuration options are defined in most of the sections of
- the HOD configuration. Options defined in a section, are used by the
- process for which that section applies. These options have the same
- meaning, but can have different values in each section.
- * temp-dir: Temporary directory for usage by the HOD processes. Make
- sure that the users who will run hod have rights to create
- directories under the directory specified here.
- * debug: A numeric value from 1-4. 4 produces the most log information,
- and 1 the least.
- * log-dir: Directory where log files are stored. By default, this is
- <install-location>/logs/. The restrictions and notes for the
- temp-dir variable apply here too.
- * xrs-port-range: A range of ports, among which an available port shall
- be picked for use to run an XML-RPC server.
- * http-port-range: A range of ports, among which an available port shall
- be picked for use to run an HTTP server.
- * java-home: Location of Java to be used by Hadoop.
- 3.2 hod options:
- ----------------
- * cluster: A descriptive name given to the cluster. For Torque, this is
- specified as a 'Node property' for every node in the cluster.
- HOD uses this value to compute the number of available nodes.
- * client-params: A comma-separated list of hadoop config parameters
- specified as key-value pairs. These will be used to
- generate a hadoop-site.xml on the submit node that
- should be used for running MapReduce jobs.
- 3.3 resource_manager options:
- -----------------------------
- * queue: Name of the queue configured in the resource manager to which
- jobs are to be submitted.
- * batch-home: Install directory to which 'bin' is appended and under
- which the executables of the resource manager can be
- found.
- * env-vars: This is a comma separated list of key-value pairs,
- expressed as key=value, which would be passed to the jobs
- launched on the compute nodes.
- For example, if the python installation is
- in a non-standard location, one can set the environment
- variable 'HOD_PYTHON_HOME' to the path to the python
- executable. The HOD processes launched on the compute nodes
- can then use this variable.
- 3.4 ringmaster options:
- -----------------------
- * work-dirs: These are a list of comma separated paths that will serve
- as the root for directories that HOD generates and passes
- to Hadoop for use to store DFS / MapReduce data. For e.g.
- this is where DFS data blocks will be stored. Typically,
- as many paths are specified as there are disks available
- to ensure all disks are being utilized. The restrictions
- and notes for the temp-dir variable apply here too.
- 3.5 gridservice-hdfs options:
- -----------------------------
- * external: If false, this indicates that a HDFS cluster must be
- bought up by the HOD system, on the nodes which it
- allocates via the allocate command. Note that in that case,
- when the cluster is de-allocated, it will bring down the
- HDFS cluster, and all the data will be lost.
- If true, it will try and connect to an externally configured
- HDFS system.
- Typically, because input for jobs are placed into HDFS
- before jobs are run, and also the output from jobs in HDFS
- is required to be persistent, an internal HDFS cluster is
- of little value in a production system. However, it allows
- for quick testing.
- * host: Hostname of the externally configured NameNode, if any
- * fs_port: Port to which NameNode RPC server is bound.
- * info_port: Port to which the NameNode web UI server is bound.
- * pkgs: Installation directory, under which bin/hadoop executable is
- located. This can be used to use a pre-installed version of
- Hadoop on the cluster.
- * server-params: A comma-separated list of hadoop config parameters
- specified key-value pairs. These will be used to
- generate a hadoop-site.xml that will be used by the
- NameNode and DataNodes.
- * final-server-params: Same as above, except they will be marked final.
- 3.6 gridservice-mapred options:
- -------------------------------
- * external: If false, this indicates that a MapReduce cluster must be
- bought up by the HOD system on the nodes which it allocates
- via the allocate command.
- If true, if will try and connect to an externally
- configured MapReduce system.
- * host: Hostname of the externally configured JobTracker, if any
- * tracker_port: Port to which the JobTracker RPC server is bound
- * info_port: Port to which the JobTracker web UI server is bound.
- * pkgs: Installation directory, under which bin/hadoop executable is
- located
- * server-params: A comma-separated list of hadoop config parameters
- specified key-value pairs. These will be used to
- generate a hadoop-site.xml that will be used by the
- JobTracker and TaskTrackers
- * final-server-params: Same as above, except they will be marked final.
- 4. Known Issues:
- ================
- HOD does not currently handle special characters such as space, comma
- and equals in configuration values.