Bug

[jas-]

I think I may have found a bug. If you look at the errors, inspected & warnings elements you will see a combination of a string and the expected JSON object in the array's. See the results here

This happens with both 4.4.20 & 5.1.12 versions of bash on both RHEL and Solaris.

{"id":"V0216246","title":"The audit system must produce records containing sufficient information to establish the identity of any user/subject associated with the event.","description":"Enabling the audit system will produce records with accurate time stamps, source, user, and activity information. Without this information malicious activity cannot be accurately tracked.","meta":{"date":"26-Jul-2023","rule_id":"SOL-11.1-010040","version":"SV-216246r603267_rule","severity":"CAT-II","classification":"UNCLASSIFIED","legacy_id":"V0047781","CCID":"CCI-001487","remediation":"true","outage_required":"false","stability":"stable"},"errors":[Elem:1,Item:2,Key:3,Value:4,{"Elem":"1"},{"Item":"2"},{"Key":"3"},{"Value":"4"}],"inspected":[Elem:one,Item:two,Key:three,Value:four,{"Elem":"one"},{"Item":"two"},{"Key":"three"},{"Value":"four"}],"warnings":[Elem:1,Item:2,Key:3,Value:4,{"Elem":"1"},{"Item":"2"},{"Key":"3"},{"Value":"4"}],"summary":{"inspected":"8","errors":"8","failed":"100.00%"},"metrics":{"start":"1721456521","stop":"1721456534","runtime":"13Sec."}}

The calling code example (uses the below functions)

$ reporting._load_dependencies
$ reporting.gen_stig 0 1721456521 1721456534 src/stigs/Solaris/11/V0216246.sh Elem:1,Item:2,Key:3,Value:4 Elem:one,Item:two,Key:three,Value:four Elem:1,Item:2,Key:3,Value:4

The code looks like the following

################################################
# @description Load reporting dependencies
#
# @noargs
#
# @example
#   $ reporting.load_dependencies
################################################
reporting._load_dependencies()
{
  source ./src/deps/json.bash/json.bash

  alias jb=json
  alias jb-array=json.array
}


################################################
# @description Generate report header
#
# @noargs
#
# @example
#   $ reporting.gen_header
#   {"hostname":"solaris","os":"Solaris","version":"11","kernel":"11.4.42.111.0","arch":"i386"}
#
# @stdout string
################################################
reporting._gen_header()
{
  local hostname os version kernel arch

  read -r hostname os version kernel arch <<< "$(env_set_env)"

  json @hostname @os @version @kernel @arch
}


################################################
# @description Generate report metrics
#
# @args $1 Integer UNIX EPOCH (start time)
# @args $2 Integer UNIX EPOCH (end time)
#
# @example
#   $ reporting._gen_metrics 1721456521 1721456534
#   {"start":"1721456521","stop":"1721456534","time":"13 Sec."}
#
# @stdout string
################################################
reporting._gen_metrics()
{
  local seconds start stop runtime

  start=${1}
  stop=${2}

  seconds=$(math.subtract ${start} ${stop})

  [ ${seconds:=1} -gt 60 ] &&
    runtime="$(math.divide ${seconds} 60)Min." ||
    runtime="${seconds}Sec."

  json @start @stop @runtime
}


################################################
# @description Generate report summary of module status
#
# @args $1 Integer Total number of STIG modules per OS and version
# @args $2 Integer Number of selected STIG modules for job run
# @args $3 Integer Number of modules that passed
# @args $4 Integer Number of modules that failed
#
# @example
#   $ reporting._gen_summary 219 88 48 40
#   {"modules":"219","selected":"88","passed":"48","failed":"40","failure_rate":"45.45%"}
#
# @stdout string
################################################
reporting._gen_summary()
{
  local modules selected passed failed failure_rate

  modules=${1}
  selected=${2}
  passed=${3}
  failed=${4}

  failure_rate=$(math.percent ${selected} ${failed})

  json @modules @selected @passed @failed @failure_rate
}


################################################
# @description Generate summary of inspected data per STIG modules
#
# @args $1 Integer Number of inspected items
# @args $2 Integer Number of errors found
# @args $3 Integer Number of warnings
#
# @example
#   $ reporting._gen_stig_summary 10 2
#   {"inspected":"10","errors":"2","failed":"20.00%"}
#   $ reporting._gen_stig_summary 10 2 5
#   {"inspected":"10","errors":"2","warnings":"5","failed":"20.00%"}
#
# @stdout string
################################################
reporting._gen_stig_summary()
{
  local inspected errors warnings failed

  inspected=${1}
  errors=${2}
  warnings=${3}

  failed="$(math.percent ${inspected} ${errors})%"

  json @inspected @errors @warnings:?? @failed
}


################################################
# @description Generate STIG header
#
# @args $1 String Path to STIG module
#
# @example
#   $ reporting._gen_stig_id src/stigs/Solaris/11/V0216246.sh
#   V0216246
#
# @stdout string
################################################
reporting._gen_stig_id()
{
  basename ${1} | sed "s|.sh||g"
}


################################################
# @description Generate STIG title
#
# @args $1 String Path to STIG module
#
# @example
#   $ reporting._gen_stig_title src/stigs/Solaris/11/V0216246.sh
#   Lorem ipsum dolor.
#
# @stdout string
################################################
reporting._gen_stig_title()
{
  awk '$0 ~ /^\# Title: /' ${1} |
    sed "s|\# Title: ||g"
}


################################################
# @description Generate STIG description
#
# @args $1 String Path to STIG module
#
# @example
#   $ reporting._gen_stig_description src/stigs/Solaris/11/V0216246.sh
#   Lorem ipsum dolor.
#
# @stdout string
################################################
reporting._gen_stig_description()
{
  awk '$0 ~ /^\# Description: /' ${1} |
    sed "s|\# Description: ||g"
}

################################################
# @description Generate STIG meta data
#
# @args $1 String Path to STIG module
#
# @example
#   $ reporting._gen_stig_meta src/stigs/Solaris/11/V0216246.sh
#   {"date":"26-Jul-2023","rule_id":"SOL-11.1-010040","version":"SV-216246r603267_rule","severity":"CAT-II","classification":"UNCLASSIFIED","legacy_id":"V0047781","CCID":"CCI-001487","remediation":"true","outage_required":"false","stability":"stable"}
#
# @stdout string
################################################
reporting._gen_stig_meta()
{
  local file date rule_id version severity classification legacy_id CCID \
        remediation_available outage_required stabilty blob

  file=${1}

  blob="$(sed -n '/^\# Date:/,/^\# Stability:/p' ${file} |
    sed "s|\"||g")"

  date="$(echo "${blob}" |
    awk '$0 ~ /^\# Date: /' |
    sed "s|\# Date: ||g")"

  rule_id="$(echo "${blob}" |
    awk '$0 ~ /^\# Rule_ID: /' |
    sed "s|\# Rule_ID: ||g")"
  
  version="$(echo "${blob}" |
    awk '$0 ~ /^\# STIG_Version: /' |
    sed "s|\# STIG_Version: ||g")"
  
  severity="$(echo "${blob}" |
    awk '$0 ~ /^\# Severity: /' |
    sed "s|\# Severity: ||g")"

  classification="$(echo "${blob}" |
    awk '$0 ~ /^\# Classification: /' |
    sed "s|\# Classification: ||g")"

  legacy_id="$(echo "${blob}" |
    awk '$0 ~ /^\# Legacy_STIG_ID: /' |
    sed "s|\# Legacy_STIG_ID: ||g")"

  CCID="$(echo "${blob}" |
    awk '$0 ~ /^\# CCI_IDS: /' |
    sed "s|\# CCI_IDS: ||g")"

  remediation="$(echo "${blob}" |
    awk '$0 ~ /^\# Remediation_Available: /' |
    sed "s|\# Remediation_Available: ||g")"

  outage_required="$(echo "${blob}" |
    awk '$0 ~ /^\# Outage_Required: /' |
    sed "s|\# Outage_Required: ||g")"

  stability="$(echo "${blob}" |
    awk '$0 ~ /^\# Stability: /' |
    sed "s|\# Stability: ||g")"

  json @date @rule_id @version @severity @classification @legacy_id:?? \
       @CCID:?? @remediation @outage_required @stability
}


################################################
# @description Generates object of inspected data; errors, warnings etc.
#
# @args $@ Array Data used to create an array of objects
#
# @example
#   $ reporting._gen_stig_object errors Elem:1,Item:2,Key:3,Value:4 Elem:one,Item:two,Key:three,Value:four
#   [{"Elem":"1","Item":"2","Key":"3","Value":"4"},{"Elem":"one","Item":"2","Key":"three","Value":"four"}]
#
# @stdout string
################################################
reporting._gen_stig_objects()
{
  local type
  local -a args obj

  args=(${@})

  type="${args[0]}"
  objs=(${args[@]:1})

  for obj in ${objs[@]}; do
    obj=${obj//:/=}
    out=${type} json ...:string{}@obj
  done

  json @${type}:raw[]
}


################################################
# @description Generate a per STIG module JSON object
#
# @args $0 Boolean True/False value associated with module passing or not
# @args $1 Integer STIG module start time
# @args $2 Integer STIG module stop time
# @args $3 String CSV of key/value items that are errors
# @args $4 String CSV of key/value items that were inspected (optional)
# @args $5 String CSV of key/value items that were warnings (optional)
#
# @example
#   $ reporting.gen_stig 0 1721456521 1721456534 \
#      src/stigs/Solaris/11/V0216246.sh \
#      item:1,item:2,item:3 \
#      foo:one,bar:baz \
#      key:val,test:result
#   
#
# @stdout string
################################################
reporting.gen_stig()
{
  local -a args errors inspected warnings
  local id title description meta err insp warn summary metrics result start stop file

  args=(${@})

  result=${args[0]}
  start=${args[1]}
  stop=${args[2]}

  file="${args[3]}"

  errors=( ${args[4]//,/ } )
  inspected=( ${args[5]//,/ } )
  warnings=( ${args[6]//,/ } )

  id="$(reporting._gen_stig_id ${file})"
  title="$(reporting._gen_stig_title ${file})"
  description="$(reporting._gen_stig_description ${file})"

  out=meta reporting._gen_stig_meta ${file}

  out=err reporting._gen_stig_objects errors ${errors[@]}
  out=insp reporting._gen_stig_objects inspected ${inspected[@]}
  out=warn reporting._gen_stig_objects warnings ${warnings[@]}

  out=metrics reporting._gen_metrics ${start} ${stop}
  out=summary reporting._gen_stig_summary ${#errors[@]} ${#inspected[@]}

  json @id @title @description @meta:raw @errors:raw[]?? @inspected:raw[]?? @warnings:raw[]?? @summary:raw @metrics:raw
}

[h4l]

I'm not able to run your program directly as It's referencing additional things,
but I think I see what's wrong from reading reporting.gen_stig.

You're calling that function with:

$ reporting.gen_stig 0 1721456521 1721456534 src/stigs/Solaris/11/V0216246.sh Elem:1,Item:2,Key:3,Value:4 Elem:one,Item:two,Key:three,Value:four Elem:1,Item:2,Key:3,Value:4

So args[4] is Elem:1,Item:2,Key:3,Value:4

$ args=(reporting.gen_stig 0 1721456521 1721456534 src/stigs/Solaris/11/V0216246.sh Elem:1,Item:2,Key:3,Value:4 Elem:one,Item:two,Key:three,Value:four Elem:1,Item:2,Key:3,Value:4)

# The function does this
$ errors=( ${args[4]//,/ } )
$ declare -p errors         
declare -a errors=([0]="Elem:1" [1]="Item:2" [2]="Key:3" [3]="Value:4")

$ # Then uses errors like this
$ json @errors:raw[]??
{"errors":[Elem:1,Item:2,Key:3,Value:4]}

So it's passing values that aren't JSON to a :raw value. :raw is assuming
the input is already valid JSON, so it just emits it as-is. If you do the same
with :json you'll get an error:

$ json @errors:json[]??
json.encode_json(): not all inputs are valid JSON: 'Elem:1' 'Item:2' 'Key:3' 'Value:4'
json(): Could not encode the value of argument '@errors:json[]??' as an array with 'json' values. Read from array-variable $errors.
␘

I think it would be a good idea to use :json when developing, and only switch to :raw at runtime once you know it's working in principle. (I guess it would make sense for json.bash to support an environment variable option something like JSON_BASH_DANGEROUSLY_DISABLE_VALIDATION to make :json act like :raw.)

---

To correct this problem depends on how you expect errors to be. I guess they should be objects, so you'd need to something like the for loop you have in reporting._gen_stig_objects to encode each entry as an object before outputting the errors array.

---

As an aside, I noticed you have json @${type}:raw[] in reporting._gen_stig_objects. I'd recommend against including a variable in the argument in this way, as it lets the $type var do unexpected things, like reference the absolute path of a file, or an unexpected environment variable. It's OK in principle if you take care to sanitise the value, but best to avoid it if you can. There's some advice on this in the README section on security which might be helpful.

I notice you've got a few places where variables aren't quoted, which may lead to unexpected splitting too. Personally I've adopted the practice of using ${var:?} syntax by default for variables in bash — this fails when a var is unset or empty. This way you have to consider whether you've verified something is set, and explicitly allow it to be empty, e.g. with ${var:-default}.

[h4l]

BTW, in #15 I've added a feature to use ggrep by default and let you choose a grep command. I don't know if you're able to ensure package providing GNU grep is available on the system's you're running your script on, but if you are then this could let you use :json rather than :raw.

[jas-]

Both comments will be useful. I think the problem I am seeing is from using multiple functions and the out=errors or other global variable.

func_one()
{
  local obj errors
  local -a obj
  
  objs=( ${@} )
  
  for obj in ${objs[@]}; do
    obj="${obj//:/=}"
    out=errors json ...:string{}@obj
  done
 
  json @errors:json[]
}

func_two()
{
  local -a args
  local errors

  args=( ${@} )
  
  out=errors func_one "${args[@]}"
  json @errors:json[]??
}

Looks like a local scope issue, because if I change errors to anything else in func_two it works.

Are there reserved names? I find it odd simply because I am using the local keyword in each function to change the scope. Do you have a better way to pass the JSON objects from function to function?

[h4l]

Thanks for this example! Just to make sure I'm seeing the same behaviour as you, this is what I'm seeing:

$ func_one foo:a,bar:b
{"errors":[{"foo":"a","bar":"b"}]}
$ func_two foo:a,bar:b
json(): Could not process argument '@errors:json[]??'. Its value references unbound variable $errors. (Use the '~' flag after the :type to treat a missing value as empty.)
␘

The problem here is that when func_two calls fun_one, the errors var in func_one is (correctly) masking the errors var in the outer func_two scope. The final json call in func_one sees out=errors and appends its output to the errors array var in func_one scope, but the errors var in func_two scope is unchanged, because it's masked.

The out=var return style relies on the named var not being accessible at the point of the call. To avoid this clashing, you need to make sure the var you use with out= is something that won't be used in the inner function call.

If you change func_two like this, it works:

func_two()
{
  local -a args
  local _f2_errors  

  args=( ${@} )

  out=_f2_errors func_one "${args[@]}"
  json errors:json[]??@_f2_errors
}

$ func_one foo:a,bar:b                                           
{"errors":[{"foo":"a","bar":"b"}]}
$ func_two foo:a,bar:b                                           
{"errors":[{"errors":[{"foo":"a","bar":"b"}]}]}

I guess you're not intending to have two layers of "errors" properties though. If you could share an example of the JSON data layout you're aiming for (and the source data) I could suggest how to adjust the funcs to generate it.

---

I don't think this out= pattern is a normal bash idiom. It's something I came up with when I was optimising the performance to avoid using a subshell to capture stdout. It's a bit odd, but once you realise it's just using bash's slightly odd dynamic scoping rules that allow functions to modify outer scopes, I think it starts to make more sense.

[jas-]

I really appreciate your responsiveness. The JSON object I am trying to recreate with your project can be found in the previous issue.

The biggest complexity is that the stigs array can have one or more of the errors, inspected & warnings array of objects. This is where I am running into issues generating the JSON objects from one function and passing to the next. Ultimately the report would be called from one function.

I am really pretty close. Just having some problems with the issues you mentioned about having two layers of "errors" and other potential objects per STIG module.

The end report ultimately has the following parts;

1. Global
   a. (Key/Values) A date & time stamp.
   b. (JSON object) A system object which includes the hostname, OS, version, kernel version and architecture.
   c. (JSON object) Summary of the results; total modules available, number of modules run, number of passed and failed and a percentage.
   d. (JSON object) Set of metrics, start, stop time and total time.
2. STIGs
   a. (Key/Values) The module ID, title and description
   b. (JSON object) A meta data object with a date the rule was released, the id and version of the rule, severity and classification etc.
   c. (Key/Values) The results and a description of the results
   d. (JSON object) An array of errors, inspected items or warning items (these can and may be empty)
   e. (JSON object) Summary of the results; total items inspected, number of passed and failed and a percentage.
   f. (JSON object) Set of metrics, start, stop time and total time specific to the module.

[h4l]

OK, yes. The approach to take seems to depend on whether the JSON data is being generated as part of the original scanning logic, or as a second pass to represent the output of the scanning process as JSON. I imagine this depends on the architecture you already have in place.

My intuition is that it would be more work to convert the existing output to JSON, as you need to also parse the output before representing it as JSON, whereas if you generate JSON at the source, you'll always have the source data to directly encode. The disadvantage of directly encoding JSON would be that you'd have a fixed output representation. (Although you could use a tool like jq to do a general transformation afterwards (e.g. if you needed XML, you could use an XSLT 3 processor which can consume JSON and generate XML).

(I say this because it seems like you're more on the side of parsing the output at the moment, e.g. passing data likeElem:1,Item:2,Key:3,Value:4 to the report function.)

Although bash data structures are quite limited, it should be possible to separate concerns of scanning/reporting and encoding as JSON by keeping the report generated by the scanner in memory. If the scanning/reporting phase can encode its results as a combination of bash arrays and associative arrays, a second report-encoding pass could walk the data tree generated by the scanning phase and encode it as JSON.

I've not had a chance to make a practical example, but I'll play with this a bit and get back to you. Unless I'm completely off target!

[jas-]

The current implementation is script a can and does call multiple scripts. Each (including the main script) generates a part of the report.

The called script(s) can have multiple arrays and associative arrays for errors, etc.

It seems to be a scoping issue.

I have the functions written to generate the report for each called script, but when multiple scripts are called only one prints because the objects seem to be globally scoped.

Pretty sure anyways, been working on it but need to do some more debugging to confirm

[h4l]

Sorry, been having a busy week! I thought I'd do an example of generating the whole depth of one of your objects from #11. I've used a kind of careful approach to naming variables with prefixes to avoid naming conflicts at lower levels. e.g. several of the functions use an "entries" var, but named with a different prefix to avoid clashing. It's not so pretty, but it works reliably, this is the pattern the json.bash code itself uses.

It doesn't generate every single property from your example, but includes an example of each different type of nested object I think. I hope it's somewhat similar structure to how you're generating report sections. And hopefully it should serve as an example of how to get around a clash, etc. Also just a few ways of representing data before emitting, e.g. the report_system using an implicit associative array to hold data. And merging together multiple objects into one in report_stig.V0216321.

source json.bash

function report_stig.V0216321.validated_files() {
  local -a _rvf_entries

  out=_rvf_entries json Name=/etc/default/passwd Option=MAXDAYS Expected=56 Current=Missing

  json Files:json[]@_rvf_entries
}

function report_stig.V0216321.validated_users() {
  local -a _rvu_entries

  out=_rvu_entries json Username=root Expected=56 Current=Missing
  out=_rvu_entries json Username=user1 Expected=56 Current=Missing
  out=_rvu_entries json Username=user2 Expected=56 Current=Missing

  json Users:json[]@_rvu_entries
}

function report_stig.V0216321.validated() {
  local -a _rv_entries

  out=_rv_entries report_stig.V0216321.validated_files
  out=_rv_entries report_stig.V0216321.validated_users

  json validated:json[]@_rv_entries
}

function report_stig.V0216321() {
  local -a _r_entries

  out=_r_entries json id=V0216321 \
    title="User passwords must be changed at least every 60 days." \
    meta:{}="date=26-Jul-2023,rule_id=SV-216321r646926_rule,version=SOL-11.1-040010"

  out=_r_entries report_stig.V0216321.validated

  out=_r_entries json summary:{}="inspected=4,errors=1,warnings=0,failed_rate=25.00%" \
    metrics:{}="start=1721456331,end=1721456344,time=13 Sec."

  # This is merging several individual JSON objects in the _r_entries array into
  # a single object. Can be useful if you need to build pieces separately.
  json ...:json{:json}@_r_entries
}

function report_stigs() {
  # do this twice to simulate having multiple
  report_stig.V0216321
  report_stig.V0216321
}

function report_system() {
  # could generate these dynamically
  system[hostname]=solaris
  system[kernel]=11.4.42.111.0
  system[OS]=Solaris
  system[version]=11
  system[architecture]=i386
}

function report() {
  local date=20240720-061533  # placeholder
  local -a _r_stigs
  local -A system

  report_system
  out=_r_stigs report_stigs

  json @date @system:{} stigs:json[]@_r_stigs
}

$ . example.sh
$ report | jq
{
  "date": "20240720-061533",
  "system": {
    "OS": "Solaris",
    "version": "11",
    "hostname": "solaris",
    "architecture": "i386",
    "kernel": "11.4.42.111.0"
  },
  "stigs": [
    {
      "id": "V0216321",
      "title": "User passwords must be changed at least every 60 days.",
      "meta": {
        "date": "26-Jul-2023",
        "rule_id": "SV-216321r646926_rule",
        "version": "SOL-11.1-040010"
      },
      "validated": [
        {
          "Files": [
            {
              "Name": "/etc/default/passwd",
              "Option": "MAXDAYS",
              "Expected": "56",
              "Current": "Missing"
            }
          ]
        },
        {
          "Users": [
            {
              "Username": "root",
              "Expected": "56",
              "Current": "Missing"
            },
            {
              "Username": "user1",
              "Expected": "56",
              "Current": "Missing"
            },
            {
              "Username": "user2",
              "Expected": "56",
              "Current": "Missing"
            }
          ]
        }
      ],
      "summary": {
        "inspected": "4",
        "errors": "1",
        "warnings": "0",
        "failed_rate": "25.00%"
      },
      "metrics": {
        "start": "1721456331",
        "end": "1721456344",
        "time": "13 Sec."
      }
    },
    {
      "id": "V0216321",
      "title": "User passwords must be changed at least every 60 days.",
      "meta": {
        "date": "26-Jul-2023",
        "rule_id": "SV-216321r646926_rule",
        "version": "SOL-11.1-040010"
      },
      "validated": [
        {
          "Files": [
            {
              "Name": "/etc/default/passwd",
              "Option": "MAXDAYS",
              "Expected": "56",
              "Current": "Missing"
            }
          ]
        },
        {
          "Users": [
            {
              "Username": "root",
              "Expected": "56",
              "Current": "Missing"
            },
            {
              "Username": "user1",
              "Expected": "56",
              "Current": "Missing"
            },
            {
              "Username": "user2",
              "Expected": "56",
              "Current": "Missing"
            }
          ]
        }
      ],
      "summary": {
        "inspected": "4",
        "errors": "1",
        "warnings": "0",
        "failed_rate": "25.00%"
      },
      "metrics": {
        "start": "1721456331",
        "end": "1721456344",
        "time": "13 Sec."
      }
    }
  ]
}

[jas-]

Thanks for this great example. I keep going back to the documentation and have gotten to a point where I am a bit lost as to what is happening.

I supply the following to function y() that conditionally passes $1 to function z() in order to handle an associated array data type. The arg string looks like key@Elem:1,Item:2,Key:3,Value:4+key@Elem:one,Item:two,Key:three,Value:four

The function currently looks like the following:

################################################
# @description Generates object of inspected data; errors, warnings etc.
#
# @args $@ Array Data used to create an array of objects
#
# @example
#   $ reporting._gen_stig_objects_associative key@Elem:1,Item:2,Key:3,Value:4+key@Elem:one,Item:two,Key:three,Value:four
#   {"key":[{"Elem":"1","Item":"2","Key":"3","Value":"4"},{"Elem":"one","Item":"2","Key":"three","Value":"four"}]}
#
# @stdout string
################################################
reporting._gen_stig_objects_associative()
{
  local key obj
  local -a keys objs tmp_objs results

  objs=( ${@//+/ } )

  keys=( $(echo "${objs[@]}" |
    tr ' ' '\n' | cut -d"@" -f1 |
    sort | uniq) )


  for key in ${keys[@]}; do

    tmp_objs=( $(echo "${objs[@]//=/^}" |
      tr ' ' '\n' | grep "^${key}@" |
      cut -d"@" -f2) )
#>&2 echo "1: ${key} -> ${tmp_objs[@]}"

    obj="${tmp_objs[@]//:/=}"
#>&2 echo "2: ${key} -> ${obj}"
    obj="${obj// /,}"
#>&2 echo "3: ${key} -> ${obj}"
    

    [ "${obj}" != "" ] &&
      results+=( $(json "${key}":{}="${obj}") )
  done

>&2 echo ${results[@]}
  (
    [ $(env._get_os_name) = "Solaris" ] &&
      json @results:raw[] ||
      json @results:json[]
  ) 2>/dev/null |
    sed "s|{\"results\":||g" |
    sed "s|]}$|]|g"
}

The output from stderr looks like this... which is what I want to see returned and correct.

reporting._gen_stig_objects_associative ernel@Option:/dev/mapper/rhel-swap+Kernel@Option:rd.lvm.lv=rhel/root+Kernel@Option:rd.lvm.lv=rhel/swap+Kernel@Option:rhgb+Kernel@Option:quiet+Kernel@Option:+Kernel@Option:fips=1,State:0+Mode@FIPS:completed.

{"Kernel":{"Option":"/dev/mapper/rhel-swap","Option":"rd.lvm.lv^rhel/root","Option":"rd.lvm.lv^rhel/swap","Option":"rhgb","Option":"quiet","Option":"","Option":"fips^1","State":"0"}}

The problem is once it calls json @results:json[] the results are getting stripped and look like this..

...
"errors": [
        {
          "Kernel": {
            "Option": "fips 1",
            "State": "missing"
          }
        },
        {
          "Packages": {
            "Name": "grub2",
            "State": "missing"
          }
        }
      ],
      "inspected": [
        {
          "Kernel": {
            "Option": "fips 1",
            "State": "0"
          }
        },
        {
          "Mode": {
            "FIPS": "disabled."
          }
        }
      ],
...

Not sure what is going on... any debugging tips?

[h4l]

Not sure what is going on... any debugging tips?

I find a good approach is to use print statement debugging with liberal use of declare -p var1 var2 .... Start at the beginning of a misbehaving function, insert some declare -p to check the initial state and add an intentional error to stop execution (e.g. return 99).

Then run the function interactively in a shell with different inputs to understand what it's doing. Then iteratively advance the error point and declare -p until you find a point that's not behaving as you expect.

Another approach is to use bats to write tests for functions, and iterate between the implementation and the tests. This is how I developed json.bash itself (see json.bats — no way I could have built it without writing tests as I wrote each fn.)

---

On your problem here — I can show you how to parse these + / @ delimited strings, but I feel like you'd be able to make your life easier if you could encode as JSON at the source. Presumably you have some code that's already got the split up data and you join it with + / @ to make these strings. At that point couldn't you encode it as JSON instead?

Anyway, here's what I came up with to parse these delimited strings and re-encode as JSON. It's rather gnarly, I can't say I'd recommend this if you can avoid parsing these +@ strings.

This is going to break if you have @/+ in your values by mistake.

$ group_named_objects key@Elem:1,Item:2,Key:3,Value:4+key@Elem:one,Item:two,Key:three,Value:four Files@Name:/etc/default/passwd,Option:MAXDAYS,Expected:56,Current:Missing Users@Username:root,Expected:56,Current:Missing+Files@Name:/etc/foo/bar,Option:MAXDAYS,Expected:12,Current:Missing+Users@Username:user1,Expected:56,Current:Missing Users@Username:user2,Expected:56,Current:Missing | jq
{
  "key": [
    {
      "Elem": "1",
      "Item": "2",
      "Key": "3",
      "Value": "4"
    },
    {
      "Elem": "one",
      "Item": "two",
      "Key": "three",
      "Value": "four"
    }
  ],
  "Files": [
    {
      "Name": "/etc/default/passwd",
      "Option": "MAXDAYS",
      "Expected": "56",
      "Current": "Missing"
    },
    {
      "Name": "/etc/foo/bar",
      "Option": "MAXDAYS",
      "Expected": "12",
      "Current": "Missing"
    }
  ],
  "Users": [
    {
      "Username": "root",
      "Expected": "56",
      "Current": "Missing"
    },
    {
      "Username": "user1",
      "Expected": "56",
      "Current": "Missing"
    },
    {
      "Username": "user2",
      "Expected": "56",
      "Current": "Missing"
    }
  ]
}

The group_named_objects function:

function group_named_objects() {
  local IFS
  IFS=+; local groups=($@)                     # [a@b+c@d e@f+g@h] -> [a@b c@d e@f g@h]
  local group_names=("${groups[@]/@*/}")       # [a@b c@d e@f g@h] -> [a c e g]
  IFS=@; local group_attr_pairs=(${groups[@]}) # [a@b c@d e@f g@h] -> [a b c d ...]

  # Map group names (a, c, e, g) to local array var names, like group_array_1, group_array_2 ...
  local -A group_arrays=()
  let i=0
  for group in "${group_names[@]}"; do
    # Skip if we've already defined a var for a group name
    if [[ "${group_arrays[${group?}]:-}" ]]; then continue; fi

    # Declare a local array variable with for each unique group name
    group_arrays["${group?}"]="group_array_$((i++))"  # generate the name
    local -a "${group_arrays["${group?}"]?}"          # declare the array using the name
  done

  # Encode each object as JSON and add it to the array of its group name
  for ((i=0; i < ${#group_attr_pairs[@]}; i+=2)); do
    local group_name=${group_attr_pairs[((i))]} \
      group_attrs=${group_attr_pairs[((i+1))]}

    # nameref group_array to point to the group's array var
    local -n group_array="${group_arrays["${group_name?}"]}"

    # Transform the foo:1,bar:2 syntax into foo=1,bar=2 attributes that
    # json.bash can parse.
    object_attrs=${group_attr_pairs[((i+1))]//:/=}  # replace : with =
    out=group_array json ...:{}@object_attrs
  done

  # Collect groups and encoded JSON arrays
  local group_entries=()
  for group_name in "${group_names[@]}"; do
    # nameref group_array to point to the group's array var
    local -n group_array="${group_arrays["${group_name?}"]}"

    out=group_entries json @group_name:json[]@group_array
  done

  json ...:json{:json}@group_entries
}

BTW sorry I didn't update from your example for this, it was easier for me to use bash level arrays/operations rather than command substitution ($())!

[jas-]

Well after trying a ton of different methods, the example is what I am using. I really appreciate your help.

[h4l]

I'm glad you got it working! Feel free to get in touch if you have another problem.