Skip to content

Commit

Permalink
Use nproc to get total number of cores
Browse files Browse the repository at this point in the history
A next error occurred on ubuntu 14.04:

```
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/threading2/t2_base.py", line 331, in run
    self_run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ferry/http/httpapi.py", line 48, in _stack_worker
    _allocate_new_worker(payload["_uuid"], payload)
  File "/usr/local/lib/python2.7/dist-packages/ferry/http/httpapi.py", line 476, in _allocate_new_worker
    new_stack=True)
  File "/usr/local/lib/python2.7/dist-packages/ferry/http/httpapi.py", line 215, in _allocate_backend
    replace = replace)
  File "/usr/local/lib/python2.7/dist-packages/ferry/docker/manager.py", line 1201, in allocate_storage
    args)
  File "/usr/local/lib/python2.7/dist-packages/ferry/docker/configfactory.py", line 129, in generate_storage_configuration
    return self._generate_configuration(uuid, container_info, service)
  File "/usr/local/lib/python2.7/dist-packages/ferry/docker/configfactory.py", line 76, in _generate_configuration
    return config_factory.apply(config, container_info)
  File "/usr/local/lib/python2.7/dist-packages/ferry/config/hadoop/hadoopconfig.py", line 513, in apply
    hadoop_dirs, hadoop_entry = self._apply_hadoop(config, hadoop_containers)
  File "/usr/local/lib/python2.7/dist-packages/ferry/config/hadoop/hadoopconfig.py", line 387, in _apply_hadoop
    self._generate_yarn_site(yarn_master, new_config_dir)
  File "/usr/local/lib/python2.7/dist-packages/ferry/config/hadoop/hadoopconfig.py", line 249, in _generate_yarn_site
    cores = self.system.get_num_cores() / 2
  File "/usr/local/lib/python2.7/dist-packages/ferry/config/system/info.py", line 43, in get_num_cores
    return int(output.strip())
ValueError: invalid literal for int() with base 10: '2\n2'
```

On 2 core VM the output of the command is:
```
$ cat /proc/cpuinfo | grep cores | awk '{print $4}'
2
2
```

nproc is a part of coreutil and is available on most unix systems.
  • Loading branch information
petro-rudenko committed Oct 9, 2014
1 parent 6511bda commit dc2fc5c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion ferry/config/system/info.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,6 @@ def get_num_cores(self):
"""
Get total number of cores.
"""
cmd = "cat /proc/cpuinfo | grep cores | awk '{print $4}'"
cmd = "nproc"
output = Popen(cmd, stdout=PIPE, shell=True).stdout.read()
return int(output.strip())

0 comments on commit dc2fc5c

Please sign in to comment.