NodeJs comparing two arrays

For the record I am a relatively new programmer

I have the code working, but it seems clunky and slow if it were to sort through many items

granted, this node app does not need to be fast, i.e the process could take 5 min and it would be fine, but I was curious if there was a better way to do it…

I have this node app which is comparing two data sets… the goals of the program are as follows

  1. compare the csv file to an online api
  2. make sure all of the names in the csv file exist in the array
  3. throw an error (console.log()) message to the screen instead of finishing

right now this is the code

const fs = require("fs");
const csv = require("csv-parser");
const fetch = require("node-fetch");

const results = [];

fs.createReadStream("./customers.csv")
  .pipe(csv())
  .on("data", (data) => {
    results.push(data);
  })
  .on("end", () => {
    console.log("Getting Customer Data from Waze...");
    fetch("https://gql.waveapps.com/graphql/public", {
      method: "post",
      headers: {
        //prettier-ignore
        'Authorization': "Bearer MyAuth",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        query: `
          query {
            business(id: "MyBusinessId") {
              customers {
                edges {
                  node {
                    id
                    name
                  }
                }
              }
            }
          }
        `,
      }),
    })
      .then((res) => res.json())
      .then(({ data }) => {
        console.log("Filtering Data...");
        // this maps through the csv file
        results.map((csv) => {
          let array = [];
          name = "";
          data.business.customers.edges.map((customer) => {
            // push the results of the expression (true of false) to an array
            array.push(
              customer.node.name.toLowerCase() === csv.name.toLowerCase()
            );
            // push nonexistent name (if there is one) variable so error handling is clear
            if (customer.node.name.toLowerCase() !== csv.name.toLowerCase()) {
              name = csv.name;
            }
          });
          // if all elements in array are false, that means there is no matching name in the data.business.customers.edges array and error will be true, if there is a true field in the name, return false
          const error = !array.some((el) => {
            if (el) {
              return true;
            }
          });

          if (error) {
            return console.log(
              `Name: ${name} not found in Waze customer list, please check your spelling`
            );
          }
          // send http request here
        });
        console.log("Finished Sending Invoices");
      });
  });

the customer.csv file

"name","domain","expiration-date"
"bob","yahoo.com","7/2/2020"
"suzie","google.com","12/1/2020"

right now the graphql api returns data that looks like this…

[
  {
    node: {
      id: 'QnVzaW5lc3M6MzE4NmRmNDQtZDg4Zi00MzgxLTk5ZGEtYTQzMWRmYzhmMDk5O0N1c3RvbWVyOjQ3NTg0Mzc2',
      name: 'NOInvoice'
    }
  },
  {
    node: {
      id: 'QnVzaW5lc3M6MzE4NmRmNDQtZDg4Zi00MzgxLTk5ZGEtYTQzMWRmYzhmMDk5O0N1c3RvbWVyOjQ3NTg0MzU3',
      name: 'Suzie'
    }
  },
  {
    node: {
      id: 'QnVzaW5lc3M6MzE4NmRmNDQtZDg4Zi00MzgxLTk5ZGEtYTQzMWRmYzhmMDk5O0N1c3RvbWVyOjQ3NTgwODkx',
      name: 'Bob'
    }
  }
]

any help would be appreciated greatly

Answer

Nested maps = O(n*m) time complexity = poor performance

First create a hashmap of the names from the API, then scan the csv array and check each name against the hashmap to see if it exists or not.

Using a hashmap is a common way to improve the performance of nested loops. The result will be closer to O(n+m) time complexity, significantly more performant.

  // create hash of valid names from API
  const validNames = data.business.customers.edges.reduce(
    (names, customer) => { 
      names[customer.name] = customer;   /* or = true */
      return names; 
    }, 
    {}
  );

  // see if any of the names in the csv are not valid
  const err = results.reduce((err, csv) => validNames[csv.name] ? err: ++err, 0);
  if (arr > 0) {
    // have invalid names in CSV
  }

  // OR alternatively, find the invalid entries
  const invalid = results.reduce(
    (invalid, csv) => {
      if (!validNames[csv.name]) invalid.push(csv);
      return invalid;
    },
    []
  );

EDIT

  // OR shorter version of find the invalid entries
  const invalid = results.filter(csv => !validNames[csv.name]);
  if (invalid.length) {
    // have invalid names in CSV
  }

Leave a Reply

Your email address will not be published. Required fields are marked *