MongoDB

Definitions

MongoDB

MongoDB is a popular, open-source NoSQL database management system that stores data in JSON-like documents with dynamic schemas. It is designed to provide high performance, scalability, and flexibility for modern applications. MongoDB allows developers to work with structured, semi-structured, and unstructured data, making it suitable for a wide range of use cases. It supports powerful querying, indexing, and aggregation capabilities, enabling efficient data retrieval and analysis.

MongoDB Official Website

MongoDB Node.js Driver

The MongoDB Node.js driver is a powerful library that allows Node.js applications to interact with MongoDB databases. It provides a rich set of features, including CRUD operations (Create, Read, Update, Delete), indexing, aggregation, and support for various MongoDB data types. The driver follows an asynchronous, non-blocking approach, making it ideal for handling large amounts of data and concurrent operations.

Official MongoDB Node.js Driver Documentation

Implementation

MongoDB Connection

mongo.ts
The mongo.ts file contains the implementation for connecting to the MongoDB database using the official Node.js driver. It first imports the required modules and initializes the MongoDB client instance. The connection function (connect) connects to the MongoDB database, retrieves the desired collection (tms), and sets up an expiration index for automatic document deletion.

// Import necessary modules
import * as mongoDB from "mongodb";
require('dotenv').config();

// MongoDB client instance
const client: mongoDB.MongoClient = new mongoDB.MongoClient(process.env.DB_CONN_STRING);

// Collection objects
export const collections: { tms?: mongoDB.Collection } = {};

// Connect to MongoDB
export async function connect(): Promise<void> {
  try {
    await client.connect();
    const db: mongoDB.Db = client.db(process.env.DB_NAME || 'travis');
    const tmsCollection: mongoDB.Collection = db.collection(process.env.COLLECTION_NAME || 'tms');
    collections.tms = tmsCollection;
    // Set TTL index for automatic document deletion
    tmsCollection.createIndex({ expireAt: 1 }, { expireAfterSeconds: 0 });
    console.log(`[MongoDB] - Successfully connected to database: ${db.databaseName} and collection: ${tmsCollection.collectionName}`);
  } catch (error) {
    console.error("Database connection failed", error);
    process.exit(1);
  }
}

// Close the connection
export function disconnect(): void {
  client.close();
}

MongoDB Schemas and Models

tms_data_model.ts
The tms_data_model.ts file defines the MongoDB document schemas using TypeScript interfaces. It contains two interfaces: SensorValue and Station. These interfaces define the structure of the data to be stored in MongoDB documents.

// Define interfaces for data models
interface SensorValue {
  stationId: string;
  name: string;
  shortName: string;
  timeWindowStart: string;
  timeWindowEnd: string;
  measuredTime: string;
  unit: string;
  value: number;
}

interface Station {
  _id: string;
  tmsNumber: string;
  dataUpdatedTime: string;
  name: string;
  coordinates: [number, number];
  sensorValues: SensorValue[];
}

MongoDB Data Operations

fetch.ts
The fetch.ts file contains functions for fetching data from external APIs and inserting it into MongoDB. It utilizes Axios, a popular HTTP client for Node.js, to perform API requests.

// Fetch data from external API and insert it into MongoDB
export const fetchMongo = async (url: String) :Promise<StationData> => {
  try {
    // Fetch data from two different APIs
    const tmsData_response: AxiosResponse = await axios.get(url, axiosConf);
    const tmsStation_response: AxiosResponse = await axios.get(process.env.TMS_STATION_LIST_URL || "https://tie.digitraffic.fi/api/tms/v1/stations", axiosConf);

    // Process the retrieved data
    // ...

    // Combine and format the data
    // ...

    // Return the fetched and combined data
    return combinedData;
  } catch (error) {
    console.log(error);
    throw error;
  }
};

MongoDB Data Retrieval

saveToMongo.ts
The saveToMongo.ts file contains functions for querying data from MongoDB and performing insert and update operations. It also includes a function to store the last fetch time in the local storage.

// Check if MongoDB is empty
export const isMongoEmpty = async ()  : Promise<boolean>=> {
    const collectionCount = await collections.tms?.countDocuments({});
    return collectionCount === 0;
}

// Function to remove selected sensors from station data
function removeSelectedSensors(stationData: StationData, selectedSensors: string[]): StationData {
  // ...
  return { ...stationData, stations: updatedStations };
}

// Function to insert data into MongoDB
const insertDataToMongo = async (data: StationData)  : Promise<void>=> {
    // ...
}

// Function to update data in MongoDB
const updateDataToMongo = async (data: StationData) : Promise<void> => {
    // ...
}

// Save the last fetch time to local storage
export const storeFetch = async (data: StationData): Promise<mongoDB.InsertOneResult<mongoDB.Document> | unknown>  => {
    // ...
}

// Function to fetch data from external API and insert it into MongoDB
export async function mongoFetch(): Promise<void> {
    // ...
}

MongoDB Express API

tms_data.ts
The tms_data.ts file sets up an Express server and defines API endpoints to interact with MongoDB. It includes three endpoints: /stations, /feedback, and /station/:id.

// Express Router for TMS data API
export const tmsRouter = express.Router();

// Middleware to parse JSON data
tmsRouter.use(express.json());

// Endpoint to fetch stations data from external API and return it
tmsRouter.get("/stations", async (_req: Request, res: Response): Promise<void> => {
  // ...
});

// Endpoint to receive feedback data and process it
tmsRouter.post("/feedback", async (_req: Request, res: Response): Promise<Response> => {
  // ...
});

// Endpoint to fetch data for a specific station based on its ID
tmsRouter.get("/station/:id", async (_req: Request, res: Response): Promise<void> => {
  // ...
});

MongoDB for Data Analytics

MongoDB historical data for Analytics and Visualization

Conclusion

This documentation provides an overview of MongoDB and demonstrates how to implement MongoDB in a Node.js application. It covers MongoDB connection, data models, data operations (fetching and insertion), data retrieval (querying), and setting up an Express API to interact with MongoDB. With MongoDB's flexibility and scalability, it serves as an excellent choice for modern application development, especially when dealing with unstructured or semi-structured data.

Please note that some parts of the code have been omitted and replaced with comments to provide a clearer and more concise overview of the MongoDB implementation in a Node.js application. For the full code implementation, refer to the respective files provided in the code section.