"Using Mongoose in Express: A Step-by-Step Guide to Joining Collections with $lookup and populate"

1. populate Method: Simplifies the process of referencing documents from other collections.
2. $lookup Operator: An aggregation pipeline stage that performs a left outer join to another collection.

I'll walk you through both methods step-by-step with code examples below. We'll create a simple application with two models: User and Post. Each Post will reference a User.

Initialize the Project

mkdir express-mongoose-lookup-populate
cd express-mongoose-lookup-populate
npm init -y        

Install Dependencies

npm install express mongoose body-parser        

Create the Project Structure

mkdir models routes
touch server.js        

Defining Mongoose Schemas

1) models/User.js

// models/User.js
const mongoose = require('mongoose');

const UserSchema = new mongoose.Schema({
  name: {
    type: String,
    required: true,
  },
  email: {
    type: String,
    required: true,
    unique: true,
  },
});

module.exports = mongoose.model('User', UserSchema);        

2) models/Post.js

// models/Post.js
const mongoose = require('mongoose');

const PostSchema = new mongoose.Schema({
  title: {
    type: String,
    required: true,
  },
  content: {
    type: String,
    required: true,
  },
  author: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'User', // Reference to User model
    required: true,
  },
});

module.exports = mongoose.model('Post', PostSchema);        

Using populate to Populate References

The populate method replaces the specified path in the document with the actual document from the referenced collection.

Example: Get All Posts with Author Details

1. routes/posts.js

// routes/posts.js
const express = require('express');
const router = express.Router();
const Post = require('../models/Post');

// GET /posts - Retrieve all posts with populated author details
router.get('/', async (req, res) => {
  try {
    const posts = await Post.find().populate('author');
    res.json(posts);
  } catch (err) {
    res.status(500).json({ message: err.message });
  }
});

module.exports = router;        

2. Explanation

• Post.find() fetches all posts.
• .populate('author') replaces the author field (which contains the User ID) with the actual User document.

Using $lookup for Aggregation Join

The $lookup operator performs a left outer join to another collection in the same database to filter in documents from the "joined" collection.

Example: Get All Posts with Author Details Using Aggregation

1. routes/posts.js (Add a new route)

// routes/posts.js (continued)

// GET /posts/aggregate - Retrieve all posts with author details using $lookup
router.get('/aggregate', async (req, res) => {
  try {
    const posts = await Post.aggregate([
      {
        $lookup: {
          from: 'users', // The name of the User collection in MongoDB (usually lowercase and plural)
          localField: 'author', // Field in Post collection
          foreignField: '_id', // Field in User collection
          as: 'authorDetails', // Alias for the joined data
        },
      },
      {
        $unwind: '$authorDetails', // Optional: To deconstruct the array
      },
    ]);
    res.json(posts);
  } catch (err) {
    res.status(500).json({ message: err.message });
  }
});        

2. Explanation

• $lookup joins the Post collection with the User collection based on the author field in Post and _id in User.
• The as field specifies the name of the new array field to add to the Post documents.
• $unwind deconstructs the authorDetails array to a single object (since each post has one author).

Complete Code Example

Here's the complete code integrating all the steps above.

1. server.js

// server.js
const express = require('express');
const mongoose = require('mongoose');
const bodyParser = require('body-parser');

const app = express();
const PORT = process.env.PORT || 3000;

// Middleware
app.use(bodyParser.json());

// Connect to MongoDB
mongoose.connect('mongodb://localhost:27017/express_mongoose_lookup_populate', {
  useNewUrlParser: true,
  useUnifiedTopology: true,
})
  .then(() => console.log('MongoDB connected'))
  .catch(err => console.log(err));

// Routes
const postRoutes = require('./routes/posts');
app.use('/posts', postRoutes);

// Sample route to create users and posts
const User = require('./models/User');
const Post = require('./models/Post');

app.post('/users', async (req, res) => {
  const user = new User({
    name: req.body.name,
    email: req.body.email,
  });

  try {
    const newUser = await user.save();
    res.status(201).json(newUser);
  } catch (err) {
    res.status(400).json({ message: err.message });
  }
});

app.post('/posts', async (req, res) => {
  const post = new Post({
    title: req.body.title,
    content: req.body.content,
    author: req.body.author, // User ID
  });

  try {
    const newPost = await post.save();
    res.status(201).json(newPost);
  } catch (err) {
    res.status(400).json({ message: err.message });
  }
});

// Start the server
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});        

models/User.js and models/Post.js

(As defined earlier.)

routes/posts.js

(As defined earlier, including both / and /aggregate routes.)