Shell脚本:隔离多个文件

Question

我在本地目录中有一个：

Rep_{ReportType}_{Date}_{Seq}.csv
Rep_0001_20150102_0.csv
Rep_0001_20150102_1.csv
Rep_0102_20150102_0.csv
Rep_0503_20150102_0.csv
Rep_0503_20150102_0.csv

使用shell-script，

1. 如何从具有固定批次大小的本地目录中获取多个文件？

2. 如何按报告类型将文件隔离/将文件分组在一起（将0001文件分组在一起，将0102分组在一起，将0503分组在一起，等等。）

我将为每个组/报告类型生成一个序列文件（使用forqlift）。输出将为report0001.seq，report0102.seq，report0503.seq（3个序列文件）。我将保存到其他目录。

注意：在序列文件中，密钥是CSV（REP_0001_20150102.CSV）的文件名，该值是文件的内容。它被存储为[字符串，字节写]。

这是我的代码：

1  reportTypes=(0001 0102 8902)
2
3  # collect all files matching expression into an array
4  filesWithDir=(~/Report/Rep_[0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-1].csv)
5
6  # take only the first hundred
7  filesWithDir =( "${filesWithDir[@]:0:100}" )
8
9  # files="${filesWithDir[@]##*/}" #### commented out since forqlift cannot create sequence file without the path/to/file
10 # echo ${files[@]}
11
12 shopt -s nullglob
13
14 # Line 21 is commented out since it has a bug. It collects files in
15 # current directory when it should be filtering the "files array" created
16 # in line 7
17
18
19 for i in ${reportTypes[@]}; do
20   printf -v val '%04d' "$i"
21   # files=("Rep_${val}_"*.csv) 
     # solution to BUG: (filter files array)
     groupFiles=( $( for j in ${filesWithDir[@]} ; do echo $j ; done | grep ${val} ) )
22
23   # Generate sequence file for EACH Report Type
24   forqlift create --file="Report${val}.seq" "${groupFiles[@]}"
25 done

（注意：序列文件输出应在当前目录中，而不是〜/报告中）

Answer 1

只需取一个数组的子集：

# collect all files matching expression into an array
files=( ~/Report/Rep_[0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].csv )
# take only the first hundred
files=( "${files[@]:0:100}" )

第二部分更棘手：bash具有关联数组（"地图"），但是唯一可以存储在数组中的法律值是字符串 - 不是其他阵列 - 因此您无法将文件名列表存储为与单个条目关联的值（不序列化数组往返字符串 - 中等棘手的事情要安全地做，因为Unix中的文件路径可以包含Newlines以外的任何字符，包括Newlines）。

那么，最好只生成数组。

shopt -s nullglob # allow a glob to expand to zero arguments
for ((i=1; i<=1000; i++)); do
  printf -v val '%04d' "$i"     # pad digits: 12 -> 0012
  files=( "Rep_${val}_"*.csv )  # collect files that match
  ## emit NUL-separated list of files, if any were found
  #(( ${#files[@]} )) && printf '%s' "${files[@]}" >"Reports.$val.txt"
  # Create a sequence file with forqlift
  forqlift create --file="Reports-${val}.seq" "${files[@]}"
done

---

如果您真的不想这样做，那么我们可以将一些使用名称Vars进行重定向的内容：

#!/bin/bash
# This only works with bash 4.3
re='^REP_([[:digit:]]{4})_[[:digit:]]{8}.csv$'
counter=0
for f in *; do
  [[ $f =~ $re ]] || continue            # skip files not matching regex
  if ((++counter > 100)); then break; fi # stop after 100 files
  group=${BASH_REMATCH[1]}               # retrieve first regex group
  declare -g -a "array${group}"          # declare an array
  declare -n group_arr="array${group}"   # redirect group_arr to that array
  group_arr+=( "$f" )                    # append to the array
done
for varname in "${!array@}"; do
  declare -n group_arr="$varname"
  ## NUL-delimited form
  #printf '%s' "${group_arr[@]}" 
  #  >"collection${varname#array}"        # write to files named collection0001, etc.
  # forqlift sequence file form
  forqlift create --file="Reports-${varname#array}.seq" "${group_arr[@]}"
done

Answer 2

我会离开外壳脚本并开始向Perl转移。

#!/usr/bin/env perl
use strict;
use warnings;
my %groups; 
while ( my $filename = glob ( "~/Reports/Rep_*.csv" ) ) {
     my ( $group, $id ) = ( $filename =~ m,/Rep_(d{4})_(d{8}).csv$, ); 
     next unless $group; #undefined means it didn't match;
     #anything past 100 in a group is discarded:
     if ( @{$groups{$group}} < 100 ) { 
         push ( @{$groups{$group}}, $filename ); 
     }
}
foreach my $group ( keys %groups ) { 
   print "$group contains:n";
   print join ("n", @{$groups{$group});
}

Answer 3

另一种选择是将一些bash命令与regexp一起抓住。请参阅下面的实现

# Explanation:
# ls -p = List all files and directories in local directory by path
# grep -v / = ignore subdirectories
# grep "^Rep_d{4}_d{8}.csv$" = Look for files matching your regexp
# tail -100 = get 100 results
for file in $(ls -p | grep -v / | grep "^Rep_d{4}_d{8}.csv$" | tail -100);
    do echo $file;
    # Use reg exp to extract the desired sequence
    re="^Rep_([[:digit:]]{4})_([[:digit:]]{8}).csv$";
    if [[ $name =~ $re ]]; then
        sequence = ${BASH_REMATCH[1};
        # Didn't end up using date, but in case you want it
        # date = ${BASH_REMATCH[2]};
        # Just in case the sequence file doesn't exist
        if [ ! -f "$sequence" ] ; then
            touch "$sequence"
        fi
        # Output/Concat your filename to the sequence file, which you can
        # read in later to do whatever administrative tasks you wish to do    
        # to them
        echo "$file" >> "$sequence"
    fi
done;