解析Linux iscsi multipath.conf为python嵌套字典

Question

我写了一个脚本，包括从标准multipath.conf配置文件中添加/删除多路径"对象"，示例如下:

# This is a basic configuration file with some examples, for device mapper
# multipath.
## Use user friendly names, instead of using WWIDs as names.
defaults {
                user_friendly_names yes
         }
##
devices {
        device {
                vendor "SolidFir"
                product "SSD SAN"
                path_grouping_policy multibus
                getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                path_selector "service-time 0"
                path_checker tur
                hardware_handler "0"
                failback immediate
                rr_weight uniform
                rr_min_io 1000
                rr_min_io_rq 1
                features "0"
                no_path_retry 24
                prio const
        }
}
multipaths {
        multipath {
                wwid 36f47acc1000000006167347a00000041
                alias dwqa-ora-fs
        }
        multipath {
                wwid 36f47acc1000000006167347a00000043
                alias dwqa-ora-grid
        }
        multipath {
                wwid 36f47acc1000000006167347a00000044
                alias dwqa-ora-dwqa1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000ae
                alias dwqa-ora-dwh2d10-1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000f9
                alias dwqa-ora-testdg-1
        }
}

所以我要做的是读取这个文件并将其存储在一个嵌套的python字典(或嵌套字典列表)中。我们现在可以忽略注释行(以#开头)。对于这个问题，我还没有一个清晰/简洁的解决方案。

这是我的部分解决方案(还没有给我期望的输出，但它接近)

def nonblank_lines(f):
    for l in f:
        line = l.rstrip()
        if line:
            yield line
def __parse_conf__(self):
    conf = []
    with open(self.conf_file_path) as f:
        for line in nonblank_lines(f):
            if line.strip().endswith("{"): # opening bracket, start of new list of dictionaries
                current_dictionary_key = line.split()[0]
                current_dictionary = { current_dictionary_key : None }
                conf.append(current_dictionary)
            elif line.strip().endswith("}"): # closing bracket, end of new dictionary
                pass
                # do nothing...
            elif not line.strip().startswith("#"):
                if current_dictionary.values() == [None]:
                    # New dictionary... we should be appending to this one
                    current_dictionary[current_dictionary_key] = [{}]
                    current_dictionary = current_dictionary[current_dictionary_key][0]
                key = line.strip().split()[0]
                val = " ".join(line.strip().split()[1:])
                current_dictionary[key] = val

这是结果字典(列表'conf'):

[{'defaults': [{'user_friendly_names': 'yes'}]},
 {'devices': None},
 {'device': [{'failback': 'immediate',
              'features': '"0"',
              'getuid_callout': '"/lib/udev/scsi_id --whitelisted --device=/dev/%n"',
              'hardware_handler': '"0"',
              'no_path_retry': '24',
              'path_checker': 'tur',
              'path_grouping_policy': 'multibus',
              'path_selector': '"service-time 0"',
              'prio': 'const',
              'product': '"SSD SAN"',
              'rr_min_io': '1000',
              'rr_min_io_rq': '1',
              'rr_weight': 'uniform',
              'vendor': '"SolidFir"'}]},
 {'multipaths': None},
 {'multipath': [{'alias': 'dwqa-ora-fs',
                 'wwid': '36f47acc1000000006167347a00000041'}]},
 {'multipath': [{'alias': 'dwqa-ora-grid',
                 'wwid': '36f47acc1000000006167347a00000043'}]},
 {'multipath': [{'alias': 'dwqa-ora-dwqa1',
                 'wwid': '36f47acc1000000006167347a00000044'}]},
 {'multipath': [{'alias': 'dwqa-ora-dwh2d10-1',
                 'wwid': '36f47acc1000000006167347a000000ae'}]},
 {'multipath': [{'alias': 'dwqa-ora-testdg-1',
                 'wwid': '36f47acc1000000006167347a000000f9'}]},
 {'multipath': [{'alias': 'dwqa-ora-testdp10-1',
                 'wwid': '"SSolidFirSSD SAN 6167347a00000123f47acc0100000000"'}]}]

显然，"None"应该用它下面的嵌套字典替换，但是我不能让这部分工作。

有什么建议吗?或者更好的解析该文件并将其存储在python数据结构中的方法?

Answer 1

试试这样:

def parse_conf(conf_lines):
    config = []
    # iterate on config lines
    for line in conf_lines:
        # remove left and right spaces
        line = line.rstrip().strip()
        if line.startswith('#'):
            # skip comment lines
            continue
        elif line.endswith('{'):
            # new dict (notice the recursion here)
            config.append({line.split()[0]: parse_conf(conf_lines)})
        else:
            # inside a dict
            if line.endswith('}'):
                # end of current dict
                break
            else:
                # parameter line
                line = line.split()
                if len(line) > 1:
                    config.append({line[0]: " ".join(line[1:])})
    return config

该函数将进入配置文件的嵌套层(由于递归和conf_lines对象是迭代器的事实)，并生成包含其他字典的字典列表。不幸的是，您必须再次将每个嵌套字典放入列表中，因为在示例文件中您展示了多路径如何重复，但在Python字典中键必须是唯一的。所以你列一个清单。

您可以使用示例配置文件进行测试，如下所示:

with open('multipath.conf','r') as conf_file:
    config = parse_conf(conf_file)
    # show multipath config lines as an example
    for item in config:
        if 'multipaths' in item:
            for multipath in item['multipaths']:
                print multipath
                # or do something more useful

输出将是:

{'multipath': [{'wwid': '36f47acc1000000006167347a00000041'}, {'alias': 'dwqa-ora-fs'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a00000043'}, {'alias': 'dwqa-ora-grid'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a00000044'}, {'alias': 'dwqa-ora-dwqa1'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a000000ae'}, {'alias': 'dwqa-ora-dwh2d10-1'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a000000f9'}, {'alias': 'dwqa-ora-testdg-1'}]}

Answer 2

如果你不使用递归，你将需要一些方法来跟踪你的关卡。但即便如此，为了添加数据，也很难引用父母或兄弟姐妹(我失败了)。下面是基于Daniele Barresi提到的可迭代输入的递归的另一种理解:

数据:

inp = """
# This is a basic configuration file with some examples, for device mapper
# multipath.
## Use user friendly names, instead of using WWIDs as names.
defaults {
                user_friendly_names yes
         }
##
devices {
        device {
                vendor "SolidFir"
                product "SSD SAN"
                path_grouping_policy multibus
                getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                path_selector "service-time 0"
                path_checker tur
                hardware_handler "0"
                failback immediate
                rr_weight uniform
                rr_min_io 1000
                rr_min_io_rq 1
                features "0"
                no_path_retry 24
                prio const
        }
}
multipaths {
        multipath {
                wwid 36f47acc1000000006167347a00000041
                alias dwqa-ora-fs
        }
        multipath {
                wwid 36f47acc1000000006167347a00000043
                alias dwqa-ora-grid
        }
        multipath {
                wwid 36f47acc1000000006167347a00000044
                alias dwqa-ora-dwqa1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000ae
                alias dwqa-ora-dwh2d10-1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000f9
                alias dwqa-ora-testdg-1
        }
}
"""

代码:

import re
level = 0
def recurse( data ):
    """ """
    global level
    out = []
    level += 1
    for line in data:
        l = line.strip()
        if l and not l.startswith('#'):
            match = re.search(r"s*(w+)s*(?:{|(?:"?s*([^"]+)"?)?)", l)
            if not match:
                if l == '}':
                    level -= 1
                    return out # recursion, up one level
            else:
                key, value = match.groups()
                if not value:
                    print( "  "*level, level, key )
                    value = recurse( data ) # recursion, down one level
                else:
                    print( "  "*level, level, key, value)
                out.append( [key,value] )
    return out  # once
result = recurse( iter(inp.split('n')) )
import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(result)

嵌套["key"， value]对的结果列表:

[   ['defaults', [['user_friendly_names', 'yes']]],
    [   'devices',
        [   [   'device',
                [   ['vendor', 'SolidFir'],
                    ['product', 'SSD SAN'],
                    ['path_grouping_policy', 'multibus'],
                    [   'getuid_callout',
                        '/lib/udev/scsi_id --whitelisted --device=/dev/%n'],
                    ['path_selector', 'service-time 0'],
                    ['path_checker', 'tur'],
                    ['hardware_handler', '0'],
                    ['failback', 'immediate'],
                    ['rr_weight', 'uniform'],
                    ['rr_min_io', '1000'],
                    ['rr_min_io_rq', '1'],
                    ['features', '0'],
                    ['no_path_retry', '24'],
                    ['prio', 'const']]]]],
    [   'multipaths',
        [   [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a00000041'],
                    ['alias', 'dwqa-ora-fs']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a00000043'],
                    ['alias', 'dwqa-ora-grid']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a00000044'],
                    ['alias', 'dwqa-ora-dwqa1']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a000000ae'],
                    ['alias', 'dwqa-ora-dwh2d10-1']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a000000f9'],
                    ['alias', 'dwqa-ora-testdg-1']]]]]]

Answer 3

Multipath conf有点难解析。这就是我使用的(最初基于daniele-barresi的答案)，输出比其他示例更容易处理。

def get_multipath_conf():
    def parse_conf(conf_lines, parent=None):
        config = {}
        for line in conf_lines:
            line = line.split('#',1)[0].strip()
            if line.endswith('{'):
                key = line.split('{', 1)[0].strip()
                value = parse_conf(conf_lines, parent=key)
                if key+'s' == parent:
                    if type(config) is dict:
                        config = []
                    config.append(value)
                else:
                    config[key] = value
            else:
                # inside a dict
                if line.endswith('}'):
                    # end of current dict
                    break
                else:
                    # parameter line
                    line = line.split(' ',1)
                    if len(line) > 1:
                        key = line[0]
                        value = line[1].strip().strip("'").strip('"')
                        config[key] = value
        return config
    return parse_conf(open('/etc/multipath.conf','r'))

输出:

{'blacklist': {'devnode': '^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda|sdb)[0-9]*$'},
 'defaults': {'find_multipaths': 'yes',
              'max_polling_interval': '4',
              'polling_interval': '2',
              'reservation_key': '0x1'},
 'devices': [{'detect_checker': 'no',
              'hardware_handler': '1 alua',
              'no_path_retry': '5',
              'path_checker': 'tur',
              'prio': 'alua',
              'product': 'iSCSI Volume',
              'user_friendly_names': 'yes',
              'vendor': 'StorMagic'}]}