November 14, 2023

Take NextJS to the next level: Create a GitHub stars monitor

CTO, Trigger.dev

Image for Take NextJS to the next level: Create a GitHub stars monitor

In this article, you will learn how to create a GitHub stars monitor to check your stars over months and how many stars you get daily.

Use the GitHub API to fetch the current number of stars received every day.
Draw a beautiful graph of stars per day on the screen.
Create a job to collect the new stars every day.

Jimmy

Your background job platform 🔌

Trigger.dev is an open source library that enables you to create and monitor long-running jobs for your app with NextJS, Remix, Astro, and so many more!

Please help us with a star 🥹. It would help us to create more articles like this 💖

Star the Trigger.dev repository ⭐️

Here is what you need to know 😻

Most of the work around getting the number of stars on GitHub will be done through the GitHub API.

GitHub API has some limits:

Maximum 100 stargazers per request
Max 100 concurrent requests
Max 60 requests per hour

The TriggerDev repository has more than 5000 stars, and it’s literally not possible to count all the stars in a reasonable amount of time (live).

So, we will do the same trick that GitHub Stars History does.

Fetch the total amount of stars (5,715) divided by 100 results per page = 58 pages
Set the maximum amount of requests we want (20 pages max) divided by 58 pages = 3 pages skip.
Fetch the stars from those pages (2000 stars) and then the stars left, and we will proportionally add to the other days (3715 stars).

It will draw us a nice graph with the bump in stars where needed.

When we fetch a new number of stars daily, it will be a lot easier. We will take the total number of stars we currently have minus the new number of stars from GitHub. We will not need to iterate the stargazers anymore.

Let’s set it up 🔥

Our application will consist of one page:

Add repositories you want to monitor.
See the list of repositories with their GitHub graph of stars.
Delete the ones you don’t want anymore.

StarsOverTime

💡 We will use NextJS new app router, please make sure you have a node version 18+ before installing the project.

Set up a new project with NextJS

npx create-next-app@latest

We will have to save all the stars into our database!

For our demo, we will use SQLite with Prisma.

It is super easy to install, but feel free to use any other database.

npm install prisma @prisma/client --save

Install Prisma in our project

npx prisma init --datasource-provider sqlite

Go to prisma/schema.prisma and replace it with the following schema:

generator client {
  provider = "prisma-client-js"
}
datasource db {
  provider = "sqlite"
  url      = env("DATABASE_URL")
}
model Repository {
    id        String   @id @default(uuid())
    month     Int
    year      Int
    day       Int
    name      String
    stars     Int
    @@unique([name, day, month, year])
}

And then run

npx prisma db push

We have basically created a new table in our SQLite database that’s called Repository:

month, year, day is a date.
name the name of the repository
stars and the number of stars for that specific date.

You can also see that we added a @@unique at the bottom, which means we can have a duplicate record of the name, month, year, day together. It will throw an error.

Let’s add our Prisma client.

Create a new folder called helper and add a new file called prisma.ts and the following code inside:

import {PrismaClient} from '@prisma/client';
export const prisma = new PrismaClient();

We can later use that prisma variable to question our database.

Application UI Skeleton 💀

We will need a few libraries to complete this tutorial:

Axios - to send requests to the server (feel free to use fetch if you feel more comfortable with it)
Dayjs - Great library to deal with dates. It’s an alternative to moment.js that’s not fully maintained anymore.
Lodash - Cool library to play with data structures.
react-hook-form - The best library to deal with forms (validation / values / etc.)
chart.js - My library of choosing to draw our GitHub stars charts.

Let’s install them:

npm install axios dayjs lodash @types/lodash chart.js react-hook-form react-chartjs-2 --save

Create a new folder called components and add a new file called main.tsx

Add the following code:

"use client";
import {useForm} from "react-hook-form";
import axios from "axios";
import {Repository} from "@prisma/client";
import {useCallback, useState} from "react";
export default function Main() {
    const [repositoryState, setRepositoryState] = useState([]);
    const {register, handleSubmit} = useForm();
    const submit = useCallback(async (data: any) => {
        const {data: repositoryResponse} = await axios.post('/api/repository', {todo: 'add', repository: data.name});
        setRepositoryState([...repositoryState, ...repositoryResponse]);
    }, [repositoryState])
    const deleteFromList = useCallback((val: List) => () => {
        axios.post('/api/repository', {todo: 'delete', repository: `https://github.com/${val.name}`});
        setRepositoryState(repositoryState.filter(v => v.name !== val.name));
    }, [repositoryState])
    return (
        <div className="w-full max-w-2xl mx-auto p-6 space-y-12">
            <form className="flex items-center space-x-4" onSubmit={handleSubmit(submit)}>
                <input className="flex-grow p-3 border border-black/20 rounded-xl" placeholder="Add Git repository" type="text" {...register('name', {required: 'true'})} />
                <button className="flex-shrink p-3 border border-black/20 rounded-xl" type="submit">
                    Add
                </button>
            </form>
            <div className="divide-y-2 divide-gray-300">
                {repositoryState.map(val => (
                    <div key={val.name} className="space-y-4">
                        <div className="flex justify-between items-center py-10">
                            <h2 className="text-xl font-bold">{val.name}</h2>
                            <button className="p-3 border border-black/20 rounded-xl bg-red-400" onClick={deleteFromList(val)}>Delete</button>
                        </div>
                        <div className="bg-white rounded-lg border p-10">
                            <div className="h-[300px]]">
                                {/* Charts Component */}
                            </div>
                        </div>
                    </div>
                ))}
            </div>
        </div>
    )
}

Super simple React component

Form that allows us to add a new GitHub library and send it to the server POST - /api/repository {todo: 'add'}
Delete repositories we don’t want POST - /api/repository {todo: 'delete'}
List of all the added libraries with their graph.

Let’s move to the complex part of the article, adding the new repository.

Counting stars

CountingStars

Inside of helper create a new file called all.stars.ts and add the following code:

import axios from "axios";
import dayjs from "dayjs";
import utc from 'dayjs/plugin/utc';
dayjs.extend(utc);
const requestAmount = 20;
export const getAllGithubStars = async (owner: string, name: string) => {
    // Get the amount of stars from GitHub
    const totalStars = (await axios.get(`https://api.github.com/repos/${owner}/${name}`)).data.stargazers_count;
    // get total pages
    const totalPages = Math.ceil(totalStars / 100);
    // How many pages to skip? We don't want to spam requests
    const pageSkips = totalPages < requestAmount ? requestAmount : Math.ceil(totalPages / requestAmount);
    // Send all the requests at the same time
    const starsDates = (await Promise.all([...new Array(requestAmount)].map(async (_, index) => {
        const getPage = (index * pageSkips) || 1;
        return (await axios.get(`https://api.github.com/repos/${owner}/${name}/stargazers?per_page=100&page=${getPage}`, {
            headers: {
                Accept: "application/vnd.github.v3.star+json",
            },
        })).data;
    }))).flatMap(p => p).reduce((acc: any, stars: any) => {
        const yearMonth = stars.starred_at.split('T')[0];
        acc[yearMonth] = (acc[yearMonth] || 0) + 1;
        return acc;
    }, {});
    // how many stars did we find from a total of `requestAmount` requests?
    const foundStars = Object.keys(starsDates).reduce((all, current) => all + starsDates[current], 0);
    // Find the earliest date
    const lowestMonthYear = Object.keys(starsDates).reduce((lowest, current) => {
        if (lowest.isAfter(dayjs.utc(current.split('T')[0]))) {
            return dayjs.utc(current.split('T')[0]);
        }
        return lowest;
    }, dayjs.utc());
    // Count dates until today
    const splitDate = dayjs.utc().diff(lowestMonthYear, 'day') + 1;
    // Create an array with the amount of stars we didn't find
    const array = [...new Array(totalStars - foundStars)];
    // Set the amount of value to add proportionally for each day
    let splitStars: any[][] = [];
    for (let i = splitDate; i > 0; i--) {
        splitStars.push(array.splice(0, Math.ceil(array.length / i)));
    }
    // Calculate the amount of stars for each day
    return [...new Array(splitDate)].map((_, index, arr) => {
        const yearMonthDay = lowestMonthYear.add(index, 'day').format('YYYY-MM-DD');
        const value = starsDates[yearMonthDay] || 0;
        return {
            stars: value + splitStars[index].length,
            date: {
                month: +dayjs.utc(yearMonthDay).format('M'),
                year: +dayjs.utc(yearMonthDay).format('YYYY'),
                day: +dayjs.utc(yearMonthDay).format('D'),
            }
        };
    });
}

So what’s going on here:

totalStars - We take the total amount of stars the library has.
totalPages - We calculate the number of pages (100 records per page)
pageSkips - Since we want a maximum of 20 requests, we check how many pages we must skip each time.
starsDates - We populate the number of stars for each date.
foundStars - Since we are skipping dates, we need to calculate the total number of stars we actually found.
lowestMonthYear - Finding the earliest date of stars we have.
splitDate - How many dates are there between the earliest date and today?
array - an empty array with splitDate amount of items.
splitStars - The number of stars we are missing and need to add each date proportionally.
Final return - The new array with the number of stars in each day since the beginning.

So, we have successfully created a function that can give us stars per day.

I have tried to display it like this, and it is chaos. You probably want to display the amount of stars for every month. Furthermore, you would probably want to accumulate stars instead of:

February - 300 stars
March - 200 stars
April - 400 stars

It would be nicer to have it like this:

February - 300 stars
March - 500 stars
April - 900 stars

Both options are valid. It depends on what you want to show!

So let’s go to our helper folder and create a new file called get.list.ts.

Here is the content of the file:

import {prisma} from "./prisma";
import {groupBy, sortBy} from "lodash";
import {Repository} from "@prisma/client";
function fixStars (arr: any[]): Array<{name: string, stars: number, month: number, year: number}> {
    return arr.map((current, index) => {
        return {
            ...current,
            stars: current.stars + arr.slice(index + 1, arr.length).reduce((acc, current) => acc + current.stars, 0),
        }
    }).reverse();
}
export const getList = async (data?: Repository[]) => {
    const repo = data || await prisma.repository.findMany();
    const uniqMonth = Object.values(
        groupBy(
            sortBy(
                Object.values(
                    groupBy(repo, (p) => p.name + '-' + p.year + '-' + p.month))
                    .map(current => {
                        const stars = current.reduce((acc, current) => acc + current.stars, 0);
                        return {
                            name: current[0].name,
                            stars,
                            month: current[0].month,
                            year: current[0].year
                        }
                    }),
            [(p: any) => -p.year, (p: any) => -p.month]
        ),p => p.name)
    );
    const fixMonthDesc = uniqMonth.map(p => fixStars(p));
    return fixMonthDesc.map(p => ({
        name: p[0].name,
        list: p
    }));
}

First, it converts all the stars by day to stars by month.

Later, we will accumulate the number of stars for every month.

One main thing to note here is that data?: Repository[] is optional.

We have made a simple logic: if we don’t pass the data, it will do it for all the repositories we have in our database.

If we pass the data, it will work only on it.

Why, you ask?

When we create a new repository, we need to work on the specific repository data after we add it to the database.
When we reload the page, we need to get the data for all of our data.

Now, let’s work on our stars create/delete route.

Go to src/app/api and create a new folder called repository. In that folder, create a new file called route.tsx.

Add the following code there:

import {getAllGithubStars} from "../../../../helper/all.stars";
import {prisma} from "../../../../helper/prisma";
import {Repository} from "@prisma/client";
import {getList} from "../../../../helper/get.list";
export async function POST(request: Request) {
    const body = await request.json();
    if (!body.repository) {
        return new Response(JSON.stringify({error: 'Repository is required'}), {status: 400});
    }
    const {owner, name} = body.repository.match(/github.com\/(?<owner>.*)\/(?<name>.*)/).groups;
    if (!owner || !name) {
        return new Response(JSON.stringify({error: 'Repository is invalid'}), {status: 400});
    }
    if (body.todo === 'delete') {
        await prisma.repository.deleteMany({
            where: {
                name: `${owner}/${name}`
            }
        });
        return new Response(JSON.stringify({deleted: true}), {status: 200});
    }
    const starsMonth = await getAllGithubStars(owner, name);
    const repo: Repository[] = [];
    for (const stars of starsMonth) {
        repo.push(
            await prisma.repository.upsert({
                where: {
                    name_day_month_year: {
                        name: `${owner}/${name}`,
                        month: stars.date.month,
                        year: stars.date.year,
                        day: stars.date.day,
                    },
                },
                update: {
                    stars: stars.stars,
                },
                create: {
                    name: `${owner}/${name}`,
                    month: stars.date.month,
                    year: stars.date.year,
                    day: stars.date.day,
                    stars: stars.stars,
                }
            })
        );
    }
    return new Response(JSON.stringify(await getList(repo)), {status: 200});
}

We are sharing both the DELETE and CREATE routes, which shouldn’t usually be used in production use, but we have done it for the article to make it easier for you.

We take the JSON from the request, check that the “repository” field exists, and that it’s a valid path for a GitHub repository.

If it’s a delete request, we use prisma to delete the repository from the database by the name of the repository and return the request.

If it’s a create, we use getAllGithubStars to get the data to save to our database.

💡 Since we have put a unique index on name,month,year and day we can use prisma upsert to update the data if the record already exists

Last, we return the newly accumulated data to the client.

The hard part finished 🍾

Main page population 💽

We haven’t created our main page component yet.

Let’s do it.

Go to the app folder create or edit page.tsx and add the following code:

"use server";
import Main from "@/components/main";
import {getList} from "../../helper/get.list";
export default async function Home() {
  const list: any[] = await getList();
  return (
      <Main list={list} />
  )
}

We use the same function of getList to get all data of all the repositories accumulated.

Let’s also modify the main component to support it.

Edit components/main.tsx and replace it with:

"use client";
import {useForm} from "react-hook-form";
import axios from "axios";
import {Repository} from "@prisma/client";
import {useCallback, useState} from "react";
interface List {
    name: string,
    list: Repository[]
}
export default function Main({list}: {list: List[]}) {
    const [repositoryState, setRepositoryState] = useState(list);
    const {register, handleSubmit} = useForm();
    const submit = useCallback(async (data: any) => {
        const {data: repositoryResponse} = await axios.post('/api/repository', {todo: 'add', repository: data.name});
        setRepositoryState([...repositoryState, ...repositoryResponse]);
    }, [repositoryState])
    const deleteFromList = useCallback((val: List) => () => {
        axios.post('/api/repository', {todo: 'delete', repository: `https://github.com/${val.name}`});
        setRepositoryState(repositoryState.filter(v => v.name !== val.name));
    }, [repositoryState])
    return (
        <div className="w-full max-w-2xl mx-auto p-6 space-y-12">
            <form className="flex items-center space-x-4" onSubmit={handleSubmit(submit)}>
                <input className="flex-grow p-3 border border-black/20 rounded-xl" placeholder="Add Git repository" type="text" {...register('name', {required: 'true'})} />
                <button className="flex-shrink p-3 border border-black/20 rounded-xl" type="submit">
                    Add
                </button>
            </form>
            <div className="divide-y-2 divide-gray-300">
                {repositoryState.map(val => (
                    <div key={val.name} className="space-y-4">
                        <div className="flex justify-between items-center py-10">
                            <h2 className="text-xl font-bold">{val.name}</h2>
                            <button className="p-3 border border-black/20 rounded-xl bg-red-400" onClick={deleteFromList(val)}>Delete</button>
                        </div>
                        <div className="bg-white rounded-lg border p-10">
                            <div className="h-[300px]]">
                                {/* Charts Components */}
                            </div>
                        </div>
                    </div>
                ))}
            </div>
        </div>
    )
}

Show Charts! 📈

Go to the components folder and add a new file called chart.tsx.

Add the following code:

"use client";
import {Repository} from "@prisma/client";
import {useMemo} from "react";
import React from 'react';
import {
    Chart as ChartJS,
    CategoryScale,
    LinearScale,
    PointElement,
    LineElement,
    Title,
    Tooltip,
    Legend,
} from 'chart.js';
import { Line } from 'react-chartjs-2';
ChartJS.register(
    CategoryScale,
    LinearScale,
    PointElement,
    LineElement,
    Title,
    Tooltip,
    Legend
);
export default function ChartComponent({repository}: {repository: Repository[]}) {
    const labels = useMemo(() => {
        return repository.map(r => `${r.year}/${r.month}`);
    }, [repository]);
    const data = useMemo(() => ({
        labels,
        datasets: [
            {
                label: repository[0].name,
                data: repository.map(p => p.stars),
                borderColor: 'rgb(255, 99, 132)',
                backgroundColor: 'rgba(255, 99, 132, 0.5)',
                tension: 0.2,
            },
        ],
    }), [repository]);
    return (
        <Line options={{
            responsive: true,
        }} data={data} />
    );
}

We use the chart.js library to draw a Line type of graph.

It’s pretty straightforward since we did all the data structuring on the server side.

Once big thing to note here is that we export default our ChartComponent. That’s because it uses Canvas. that’s unavailable on the server side, and we will need to lazy load this component.

Let’s modify our main.tsx:

"use client";
import {useForm} from "react-hook-form";
import axios from "axios";
import {Repository} from "@prisma/client";
import dynamic from "next/dynamic";
import {useCallback, useState} from "react";
const ChartComponent = dynamic(() => import('@/components/chart'), { ssr: false, })
interface List {
    name: string,
    list: Repository[]
}
export default function Main({list}: {list: List[]}) {
    const [repositoryState, setRepositoryState] = useState(list);
    const {register, handleSubmit} = useForm();
    const submit = useCallback(async (data: any) => {
        const {data: repositoryResponse} = await axios.post('/api/repository', {todo: 'add', repository: data.name});
        setRepositoryState([...repositoryState, ...repositoryResponse]);
    }, [repositoryState])
    const deleteFromList = useCallback((val: List) => () => {
        axios.post('/api/repository', {todo: 'delete', repository: `https://github.com/${val.name}`});
        setRepositoryState(repositoryState.filter(v => v.name !== val.name));
    }, [repositoryState])
    return (
        <div className="w-full max-w-2xl mx-auto p-6 space-y-12">
            <form className="flex items-center space-x-4" onSubmit={handleSubmit(submit)}>
                <input className="flex-grow p-3 border border-black/20 rounded-xl" placeholder="Add Git repository" type="text" {...register('name', {required: 'true'})} />
                <button className="flex-shrink p-3 border border-black/20 rounded-xl" type="submit">
                    Add
                </button>
            </form>
            <div className="divide-y-2 divide-gray-300">
                {repositoryState.map(val => (
                    <div key={val.name} className="space-y-4">
                        <div className="flex justify-between items-center py-10">
                            <h2 className="text-xl font-bold">{val.name}</h2>
                            <button className="p-3 border border-black/20 rounded-xl bg-red-400" onClick={deleteFromList(val)}>Delete</button>
                        </div>
                        <div className="bg-white rounded-lg border p-10">
                            <div className="h-[300px]]">
                                <ChartComponent repository={val.list} />
                            </div>
                        </div>
                    </div>
                ))}
            </div>
        </div>
    )
}

You can see that we use nextjs/dynamic to lazy load the component.

I hope in the future, NextJS will add something like "use lazy-load" for the client components 😺

But what about new stars? Meet Trigger.Dev!

The best way to add the new stars every day would be to run a cron request to check for the newly added stars and add them to our database.

Instead of using Vercel cron / GitHub actions or, god forbid, creating a new server for that.

We can use Trigger.DEV will work directly with our NextJS app.

So let’s set it up!

Once registered, create an organization and choose a project name for your job.

New Org

Select Next.js as your framework and follow the process for adding Trigger.dev to an existing Next.js project.

NextJS

Otherwise, click Environments & API Keys on the sidebar menu of your project dashboard.

Dev Key

Copy your DEV server API key and run the code snippet below to install Trigger.dev.

Follow the instructions carefully.

npx @trigger.dev/cli@latest init

Run the following code snippet in another terminal to establish a tunnel between Trigger.dev and your Next.js project.

npx @trigger.dev/cli@latest dev

Let's create our TriggerDev job!

You will see a newly created folder called jobs. Create a new file there called sync.stars.ts

Add the following code:

import { cronTrigger, invokeTrigger } from "@trigger.dev/sdk";
import { client } from "@/trigger";
import { prisma } from "../../helper/prisma";
import axios from "axios";
import { z } from "zod";
// Your first job
// This Job will be triggered by an event, log a joke to the console, and then wait 5 seconds before logging the punchline.
client.defineJob({
  id: "sync-stars",
  name: "Sync Stars Daily",
  version: "0.0.1",
  // Run a cron every day at 23:00 AM
  trigger: cronTrigger({
    cron: "0 23 * * *",
  }),
  run: async (payload, io, ctx) => {
    const repos = await io.runTask("get-stars", async () => {
      // get all libraries and current amount of stars
      return await prisma.repository.groupBy({
        by: ["name"],
        _sum: {
          stars: true,
        },
      });
    });
    //loop through all repos and invoke the Job that gets the latest stars
    for (const repo of repos) {
      getStars.invoke(repo.name, {
        name: repo.name,
        previousStarCount: repo?._sum?.stars || 0,
      });
    }
  },
});
const getStars = client.defineJob({
  id: "get-latest-stars",
  name: "Get latest stars",
  version: "0.0.1",
  // Run a cron every day at 23:00 AM
  trigger: invokeTrigger({
    schema: z.object({
      name: z.string(),
      previousStarCount: z.number(),
    }),
  }),
  run: async (payload, io, ctx) => {
    const stargazers_count = await io.runTask("get-stars", async () => {
      const { data } = await axios.get(
          `https://api.github.com/repos/${payload.name}`,
          {
            headers: {
              authorization: `token ${process.env.TOKEN}`,
            },
          }
      );
      return data.stargazers_count as number;
    });
    await prisma.repository.upsert({
      where: {
        name_day_month_year: {
          name: payload.name,
          month: new Date().getMonth() + 1,
          year: new Date().getFullYear(),
          day: new Date().getDate(),
        },
      },
      update: {
          stars: stargazers_count - payload.previousStarCount,
      },
      create: {
        name: payload.name,
        stars: stargazers_count - payload.previousStarCount,
        month: new Date().getMonth() + 1,
        year: new Date().getFullYear(),
        day: new Date().getDate(),
      },
    });
  },
});

We created a new job called “Sync Stars Daily” that will run every day at 23:00pm - The representation of it in the cron text is: 0 23 * * *

We get all our current repositories in our database, group them by their name, and sum the stars.

Since everything runs on Vercel serverless, we might get to a timeout going over all the repositories.

For that, we send each repository to a different job.

We use the invoke to create new jobs and then process them inside Get latest stars

We iterate over all the new repositories and get the current number of stars.

We remove the new number of stars by the old number of stars to get the today amount of stars.

We added it to the database with prisma. there is no simpler than that!

The last thing is to edit jobs/index.ts and replace the content with this:

export * from "./sync.stars";

And you are done 🥳

Let's connect! 🔌

As an open source developer, you're invited to join our community to contribute and engage with maintainers. Don't hesitate to visit our GitHub repository to contribute and create issues related to Trigger.dev.

The source for this tutorial is available here:

https://github.com/triggerdotdev/blog/tree/main/stars-monitor

Thank you for reading!

Jump to

Customer story

Real-time PDF conversion with Trigger.dev: Papermark’s success story

Marc Seitz

Oct 30, 2024

Ready to start building?

Build and deploy your first task in 3 minutes.

Get started now

Product

Trigger.dev Realtime

Concurrency & queues

Scheduled tasks

Observability & monitoring

Roadmap

Latest changelogs

Example tasks using Fal.ai and Realtime

The Trigger.dev Realtime beta is now available

Set a maximum duration for a task to run.

Latest blog posts

Real-time PDF conversion with Trigger.dev: Papermark’s success story

How to scrape a website using Browserbase, Puppeteer, OpenAI and Trigger.dev

Trigger anything from a database change using Supabase with Trigger.dev

Documentation

Guides

Take NextJS to the next level: Create a GitHub stars monitor

Your background job platform 🔌

Here is what you need to know 😻

Let’s set it up 🔥

Application UI Skeleton 💀

Counting stars

Main page population 💽

Show Charts! 📈

But what about new stars? Meet Trigger.Dev!

Let's connect! 🔌

Real-time PDF conversion with Trigger.dev: Papermark’s success story

Ready to start building?