Take NextJS to the next level: Create a GitHub stars monitor
CTO, Trigger.dev
In this article, you will learn how to create a GitHub stars monitor to check your stars over months and how many stars you get daily.
- Use the GitHub API to fetch the current number of stars received every day.
- Draw a beautiful graph of stars per day on the screen.
- Create a job to collect the new stars every day.
Your background job platform 🔌
Trigger.dev is an open source library that enables you to create and monitor long-running jobs for your app with NextJS, Remix, Astro, and so many more!
Please help us with a star 🥹. It would help us to create more articles like this 💖
Star the Trigger.dev repository ⭐️
Here is what you need to know 😻
Most of the work around getting the number of stars on GitHub will be done through the GitHub API.
GitHub API has some limits:
- Maximum 100 stargazers per request
- Max 100 concurrent requests
- Max 60 requests per hour
The TriggerDev repository has more than 5000 stars, and it’s literally not possible to count all the stars in a reasonable amount of time (live).
So, we will do the same trick that GitHub Stars History does.
- Fetch the total amount of stars (5,715) divided by 100 results per page = 58 pages
- Set the maximum amount of requests we want (20 pages max) divided by 58 pages = 3 pages skip.
- Fetch the stars from those pages (2000 stars) and then the stars left, and we will proportionally add to the other days (3715 stars).
It will draw us a nice graph with the bump in stars where needed.
When we fetch a new number of stars daily, it will be a lot easier. We will take the total number of stars we currently have minus the new number of stars from GitHub. We will not need to iterate the stargazers anymore.
Let’s set it up 🔥
Our application will consist of one page:
- Add repositories you want to monitor.
- See the list of repositories with their GitHub graph of stars.
- Delete the ones you don’t want anymore.
💡 We will use NextJS new app router, please make sure you have a node version 18+ before installing the project.
Set up a new project with NextJS
_10npx create-next-app@latest
We will have to save all the stars into our database!
For our demo, we will use SQLite with Prisma
.
It is super easy to install, but feel free to use any other database.
_10npm install prisma @prisma/client --save
Install Prisma in our project
_10npx prisma init --datasource-provider sqlite
Go to prisma/schema.prisma
and replace it with the following schema:
_18generator client {_18 provider = "prisma-client-js"_18}_18_18datasource db {_18 provider = "sqlite"_18 url = env("DATABASE_URL")_18}_18_18model Repository {_18 id String @id @default(uuid())_18 month Int_18 year Int_18 day Int_18 name String_18 stars Int_18 @@unique([name, day, month, year])_18}
And then run
_10npx prisma db push
We have basically created a new table in our SQLite database that’s called Repository
:
month
,year
,day
is a date.name
the name of the repositorystars
and the number of stars for that specific date.
You can also see that we added a @@unique
at the bottom, which means we can have a duplicate record of the name
, month
, year
, day
together. It will throw an error.
Let’s add our Prisma client.
Create a new folder called helper
and add a new file called prisma.ts
and the following code inside:
_10import {PrismaClient} from '@prisma/client';_10_10export const prisma = new PrismaClient();
We can later use that prisma
variable to question our database.
Application UI Skeleton 💀
We will need a few libraries to complete this tutorial:
- Axios - to send requests to the server (feel free to use fetch if you feel more comfortable with it)
- Dayjs - Great library to deal with dates. It’s an alternative to moment.js that’s not fully maintained anymore.
- Lodash - Cool library to play with data structures.
- react-hook-form - The best library to deal with forms (validation / values / etc.)
- chart.js - My library of choosing to draw our GitHub stars charts.
Let’s install them:
_10npm install axios dayjs lodash @types/lodash chart.js react-hook-form react-chartjs-2 --save
Create a new folder called components
and add a new file called main.tsx
Add the following code:
_46"use client";_46import {useForm} from "react-hook-form";_46import axios from "axios";_46import {Repository} from "@prisma/client";_46import {useCallback, useState} from "react";_46_46export default function Main() {_46 const [repositoryState, setRepositoryState] = useState([]);_46 const {register, handleSubmit} = useForm();_46_46 const submit = useCallback(async (data: any) => {_46 const {data: repositoryResponse} = await axios.post('/api/repository', {todo: 'add', repository: data.name});_46 setRepositoryState([...repositoryState, ...repositoryResponse]);_46 }, [repositoryState])_46_46 const deleteFromList = useCallback((val: List) => () => {_46 axios.post('/api/repository', {todo: 'delete', repository: `https://github.com/${val.name}`});_46 setRepositoryState(repositoryState.filter(v => v.name !== val.name));_46 }, [repositoryState])_46_46 return (_46 <div className="w-full max-w-2xl mx-auto p-6 space-y-12">_46 <form className="flex items-center space-x-4" onSubmit={handleSubmit(submit)}>_46 <input className="flex-grow p-3 border border-black/20 rounded-xl" placeholder="Add Git repository" type="text" {...register('name', {required: 'true'})} />_46 <button className="flex-shrink p-3 border border-black/20 rounded-xl" type="submit">_46 Add_46 </button>_46 </form>_46 <div className="divide-y-2 divide-gray-300">_46 {repositoryState.map(val => (_46 <div key={val.name} className="space-y-4">_46 <div className="flex justify-between items-center py-10">_46 <h2 className="text-xl font-bold">{val.name}</h2>_46 <button className="p-3 border border-black/20 rounded-xl bg-red-400" onClick={deleteFromList(val)}>Delete</button>_46 </div>_46 <div className="bg-white rounded-lg border p-10">_46 <div className="h-[300px]]">_46 {/* Charts Component */}_46 </div>_46 </div>_46 </div>_46 ))}_46 </div>_46 </div>_46 )_46}
Super simple React component
- Form that allows us to add a new GitHub library and send it to the server POST -
/api/repository
{todo: 'add'}
- Delete repositories we don’t want POST -
/api/repository
{todo: 'delete'}
- List of all the added libraries with their graph.
Let’s move to the complex part of the article, adding the new repository.
Counting stars
Inside of helper
create a new file called all.stars.ts
and add the following code:
_68import axios from "axios";_68import dayjs from "dayjs";_68import utc from 'dayjs/plugin/utc';_68dayjs.extend(utc);_68_68const requestAmount = 20;_68_68export const getAllGithubStars = async (owner: string, name: string) => {_68 // Get the amount of stars from GitHub_68 const totalStars = (await axios.get(`https://api.github.com/repos/${owner}/${name}`)).data.stargazers_count;_68_68 // get total pages_68 const totalPages = Math.ceil(totalStars / 100);_68_68 // How many pages to skip? We don't want to spam requests_68 const pageSkips = totalPages < requestAmount ? requestAmount : Math.ceil(totalPages / requestAmount);_68_68 // Send all the requests at the same time_68 const starsDates = (await Promise.all([...new Array(requestAmount)].map(async (_, index) => {_68 const getPage = (index * pageSkips) || 1;_68 return (await axios.get(`https://api.github.com/repos/${owner}/${name}/stargazers?per_page=100&page=${getPage}`, {_68 headers: {_68 Accept: "application/vnd.github.v3.star+json",_68 },_68 })).data;_68 }))).flatMap(p => p).reduce((acc: any, stars: any) => {_68 const yearMonth = stars.starred_at.split('T')[0];_68 acc[yearMonth] = (acc[yearMonth] || 0) + 1;_68 return acc;_68 }, {});_68_68 // how many stars did we find from a total of `requestAmount` requests?_68 const foundStars = Object.keys(starsDates).reduce((all, current) => all + starsDates[current], 0);_68_68 // Find the earliest date_68 const lowestMonthYear = Object.keys(starsDates).reduce((lowest, current) => {_68 if (lowest.isAfter(dayjs.utc(current.split('T')[0]))) {_68 return dayjs.utc(current.split('T')[0]);_68 }_68 return lowest;_68 }, dayjs.utc());_68_68 // Count dates until today_68 const splitDate = dayjs.utc().diff(lowestMonthYear, 'day') + 1;_68_68 // Create an array with the amount of stars we didn't find_68 const array = [...new Array(totalStars - foundStars)];_68_68 // Set the amount of value to add proportionally for each day_68 let splitStars: any[][] = [];_68 for (let i = splitDate; i > 0; i--) {_68 splitStars.push(array.splice(0, Math.ceil(array.length / i)));_68 }_68_68 // Calculate the amount of stars for each day_68 return [...new Array(splitDate)].map((_, index, arr) => {_68 const yearMonthDay = lowestMonthYear.add(index, 'day').format('YYYY-MM-DD');_68 const value = starsDates[yearMonthDay] || 0;_68 return {_68 stars: value + splitStars[index].length,_68 date: {_68 month: +dayjs.utc(yearMonthDay).format('M'),_68 year: +dayjs.utc(yearMonthDay).format('YYYY'),_68 day: +dayjs.utc(yearMonthDay).format('D'),_68 }_68 };_68 });_68}
So what’s going on here:
totalStars
- We take the total amount of stars the library has.totalPages
- We calculate the number of pages (100 records per page)pageSkips
- Since we want a maximum of 20 requests, we check how many pages we must skip each time.starsDates
- We populate the number of stars for each date.foundStars
- Since we are skipping dates, we need to calculate the total number of stars we actually found.lowestMonthYear
- Finding the earliest date of stars we have.splitDate
- How many dates are there between the earliest date and today?array
- an empty array withsplitDate
amount of items.splitStars
- The number of stars we are missing and need to add each date proportionally.- Final return - The new array with the number of stars in each day since the beginning.
So, we have successfully created a function that can give us stars per day.
I have tried to display it like this, and it is chaos. You probably want to display the amount of stars for every month. Furthermore, you would probably want to accumulate stars instead of:
- February - 300 stars
- March - 200 stars
- April - 400 stars
It would be nicer to have it like this:
- February - 300 stars
- March - 500 stars
- April - 900 stars
Both options are valid. It depends on what you want to show!
So let’s go to our helper folder and create a new file called get.list.ts
.
Here is the content of the file:
_40import {prisma} from "./prisma";_40import {groupBy, sortBy} from "lodash";_40import {Repository} from "@prisma/client";_40_40function fixStars (arr: any[]): Array<{name: string, stars: number, month: number, year: number}> {_40 return arr.map((current, index) => {_40 return {_40 ...current,_40 stars: current.stars + arr.slice(index + 1, arr.length).reduce((acc, current) => acc + current.stars, 0),_40 }_40 }).reverse();_40}_40_40export const getList = async (data?: Repository[]) => {_40 const repo = data || await prisma.repository.findMany();_40 const uniqMonth = Object.values(_40 groupBy(_40 sortBy(_40 Object.values(_40 groupBy(repo, (p) => p.name + '-' + p.year + '-' + p.month))_40 .map(current => {_40 const stars = current.reduce((acc, current) => acc + current.stars, 0);_40 return {_40 name: current[0].name,_40 stars,_40 month: current[0].month,_40 year: current[0].year_40 }_40 }),_40 [(p: any) => -p.year, (p: any) => -p.month]_40 ),p => p.name)_40 );_40_40 const fixMonthDesc = uniqMonth.map(p => fixStars(p));_40_40 return fixMonthDesc.map(p => ({_40 name: p[0].name,_40 list: p_40 }));_40}
First, it converts all the stars by day to stars by month.
Later, we will accumulate the number of stars for every month.
One main thing to note here is that data?: Repository[]
is optional.
We have made a simple logic: if we don’t pass the data, it will do it for all the repositories we have in our database.
If we pass the data, it will work only on it.
Why, you ask?
- When we create a new repository, we need to work on the specific repository data after we add it to the database.
- When we reload the page, we need to get the data for all of our data.
Now, let’s work on our stars create/delete route.
Go to src/app/api
and create a new folder called repository
. In that folder, create a new file called route.tsx
.
Add the following code there:
_54import {getAllGithubStars} from "../../../../helper/all.stars";_54import {prisma} from "../../../../helper/prisma";_54import {Repository} from "@prisma/client";_54import {getList} from "../../../../helper/get.list";_54_54export async function POST(request: Request) {_54 const body = await request.json();_54 if (!body.repository) {_54 return new Response(JSON.stringify({error: 'Repository is required'}), {status: 400});_54 }_54_54 const {owner, name} = body.repository.match(/github.com\/(?<owner>.*)\/(?<name>.*)/).groups;_54 if (!owner || !name) {_54 return new Response(JSON.stringify({error: 'Repository is invalid'}), {status: 400});_54 }_54_54 if (body.todo === 'delete') {_54 await prisma.repository.deleteMany({_54 where: {_54 name: `${owner}/${name}`_54 }_54 });_54_54 return new Response(JSON.stringify({deleted: true}), {status: 200});_54 }_54_54 const starsMonth = await getAllGithubStars(owner, name);_54 const repo: Repository[] = [];_54 for (const stars of starsMonth) {_54 repo.push(_54 await prisma.repository.upsert({_54 where: {_54 name_day_month_year: {_54 name: `${owner}/${name}`,_54 month: stars.date.month,_54 year: stars.date.year,_54 day: stars.date.day,_54 },_54 },_54 update: {_54 stars: stars.stars,_54 },_54 create: {_54 name: `${owner}/${name}`,_54 month: stars.date.month,_54 year: stars.date.year,_54 day: stars.date.day,_54 stars: stars.stars,_54 }_54 })_54 );_54 }_54 return new Response(JSON.stringify(await getList(repo)), {status: 200});_54}
We are sharing both the DELETE and CREATE routes, which shouldn’t usually be used in production use, but we have done it for the article to make it easier for you.
We take the JSON from the request, check that the “repository” field exists, and that it’s a valid path for a GitHub repository.
If it’s a delete request, we use prisma
to delete the repository from the database by the name of the repository and return the request.
If it’s a create, we use getAllGithubStars
to get the data to save to our database.
💡 Since we have put a unique index on
name
,month
,year
andday
we can useprisma
upsert
to update the data if the record already exists
Last, we return the newly accumulated data to the client.
The hard part finished 🍾
Main page population 💽
We haven’t created our main page component yet.
Let’s do it.
Go to the app
folder create or edit page.tsx
and add the following code:
_11"use server";_11_11import Main from "@/components/main";_11import {getList} from "../../helper/get.list";_11_11export default async function Home() {_11 const list: any[] = await getList();_11 return (_11 <Main list={list} />_11 )_11}
We use the same function of getList
to get all data of all the repositories accumulated.
Let’s also modify the main component to support it.
Edit components/main.tsx
and replace it with:
_51"use client";_51import {useForm} from "react-hook-form";_51import axios from "axios";_51import {Repository} from "@prisma/client";_51import {useCallback, useState} from "react";_51_51interface List {_51 name: string,_51 list: Repository[]_51}_51_51export default function Main({list}: {list: List[]}) {_51 const [repositoryState, setRepositoryState] = useState(list);_51 const {register, handleSubmit} = useForm();_51_51 const submit = useCallback(async (data: any) => {_51 const {data: repositoryResponse} = await axios.post('/api/repository', {todo: 'add', repository: data.name});_51 setRepositoryState([...repositoryState, ...repositoryResponse]);_51 }, [repositoryState])_51_51 const deleteFromList = useCallback((val: List) => () => {_51 axios.post('/api/repository', {todo: 'delete', repository: `https://github.com/${val.name}`});_51 setRepositoryState(repositoryState.filter(v => v.name !== val.name));_51 }, [repositoryState])_51_51 return (_51 <div className="w-full max-w-2xl mx-auto p-6 space-y-12">_51 <form className="flex items-center space-x-4" onSubmit={handleSubmit(submit)}>_51 <input className="flex-grow p-3 border border-black/20 rounded-xl" placeholder="Add Git repository" type="text" {...register('name', {required: 'true'})} />_51 <button className="flex-shrink p-3 border border-black/20 rounded-xl" type="submit">_51 Add_51 </button>_51 </form>_51 <div className="divide-y-2 divide-gray-300">_51 {repositoryState.map(val => (_51 <div key={val.name} className="space-y-4">_51 <div className="flex justify-between items-center py-10">_51 <h2 className="text-xl font-bold">{val.name}</h2>_51 <button className="p-3 border border-black/20 rounded-xl bg-red-400" onClick={deleteFromList(val)}>Delete</button>_51 </div>_51 <div className="bg-white rounded-lg border p-10">_51 <div className="h-[300px]]">_51 {/* Charts Components */}_51 </div>_51 </div>_51 </div>_51 ))}_51 </div>_51 </div>_51 )_51}
Show Charts! 📈
Go to the components
folder and add a new file called chart.tsx
.
Add the following code:
_50"use client";_50import {Repository} from "@prisma/client";_50import {useMemo} from "react";_50import React from 'react';_50import {_50 Chart as ChartJS,_50 CategoryScale,_50 LinearScale,_50 PointElement,_50 LineElement,_50 Title,_50 Tooltip,_50 Legend,_50} from 'chart.js';_50import { Line } from 'react-chartjs-2';_50_50ChartJS.register(_50 CategoryScale,_50 LinearScale,_50 PointElement,_50 LineElement,_50 Title,_50 Tooltip,_50 Legend_50);_50_50export default function ChartComponent({repository}: {repository: Repository[]}) {_50 const labels = useMemo(() => {_50 return repository.map(r => `${r.year}/${r.month}`);_50 }, [repository]);_50_50 const data = useMemo(() => ({_50 labels,_50 datasets: [_50 {_50 label: repository[0].name,_50 data: repository.map(p => p.stars),_50 borderColor: 'rgb(255, 99, 132)',_50 backgroundColor: 'rgba(255, 99, 132, 0.5)',_50 tension: 0.2,_50 },_50 ],_50 }), [repository]);_50_50 return (_50 <Line options={{_50 responsive: true,_50 }} data={data} />_50 );_50}
We use the chart.js
library to draw a Line
type of graph.
It’s pretty straightforward since we did all the data structuring on the server side.
Once big thing to note here is that we export default
our ChartComponent. That’s because it uses Canvas
. that’s unavailable on the server side, and we will need to lazy load this component.
Let’s modify our main.tsx
:
_53"use client";_53import {useForm} from "react-hook-form";_53import axios from "axios";_53import {Repository} from "@prisma/client";_53import dynamic from "next/dynamic";_53import {useCallback, useState} from "react";_53const ChartComponent = dynamic(() => import('@/components/chart'), { ssr: false, })_53_53interface List {_53 name: string,_53 list: Repository[]_53}_53_53export default function Main({list}: {list: List[]}) {_53 const [repositoryState, setRepositoryState] = useState(list);_53 const {register, handleSubmit} = useForm();_53_53 const submit = useCallback(async (data: any) => {_53 const {data: repositoryResponse} = await axios.post('/api/repository', {todo: 'add', repository: data.name});_53 setRepositoryState([...repositoryState, ...repositoryResponse]);_53 }, [repositoryState])_53_53 const deleteFromList = useCallback((val: List) => () => {_53 axios.post('/api/repository', {todo: 'delete', repository: `https://github.com/${val.name}`});_53 setRepositoryState(repositoryState.filter(v => v.name !== val.name));_53 }, [repositoryState])_53_53 return (_53 <div className="w-full max-w-2xl mx-auto p-6 space-y-12">_53 <form className="flex items-center space-x-4" onSubmit={handleSubmit(submit)}>_53 <input className="flex-grow p-3 border border-black/20 rounded-xl" placeholder="Add Git repository" type="text" {...register('name', {required: 'true'})} />_53 <button className="flex-shrink p-3 border border-black/20 rounded-xl" type="submit">_53 Add_53 </button>_53 </form>_53 <div className="divide-y-2 divide-gray-300">_53 {repositoryState.map(val => (_53 <div key={val.name} className="space-y-4">_53 <div className="flex justify-between items-center py-10">_53 <h2 className="text-xl font-bold">{val.name}</h2>_53 <button className="p-3 border border-black/20 rounded-xl bg-red-400" onClick={deleteFromList(val)}>Delete</button>_53 </div>_53 <div className="bg-white rounded-lg border p-10">_53 <div className="h-[300px]]">_53 <ChartComponent repository={val.list} />_53 </div>_53 </div>_53 </div>_53 ))}_53 </div>_53 </div>_53 )_53}
You can see that we use nextjs/dynamic
to lazy load the component.
I hope in the future, NextJS will add something like "use lazy-load"
for the client components 😺
But what about new stars? Meet Trigger.Dev!
The best way to add the new stars every day would be to run a cron request to check for the newly added stars and add them to our database.
Instead of using Vercel cron / GitHub actions or, god forbid, creating a new server for that.
We can use Trigger.DEV will work directly with our NextJS app.
So let’s set it up!
Sign up for a Trigger.dev account.
Once registered, create an organization and choose a project name for your job.
Select Next.js as your framework and follow the process for adding Trigger.dev to an existing Next.js project.
Otherwise, click Environments & API Keys
on the sidebar menu of your project dashboard.
Copy your DEV server API key and run the code snippet below to install Trigger.dev.
Follow the instructions carefully.
_10npx @trigger.dev/cli@latest init
Run the following code snippet in another terminal to establish a tunnel between Trigger.dev and your Next.js project.
_10npx @trigger.dev/cli@latest dev
Let's create our TriggerDev job!
You will see a newly created folder called jobs
.
Create a new file there called sync.stars.ts
Add the following code:
_83import { cronTrigger, invokeTrigger } from "@trigger.dev/sdk";_83import { client } from "@/trigger";_83import { prisma } from "../../helper/prisma";_83import axios from "axios";_83import { z } from "zod";_83_83// Your first job_83// This Job will be triggered by an event, log a joke to the console, and then wait 5 seconds before logging the punchline._83client.defineJob({_83 id: "sync-stars",_83 name: "Sync Stars Daily",_83 version: "0.0.1",_83 // Run a cron every day at 23:00 AM_83 trigger: cronTrigger({_83 cron: "0 23 * * *",_83 }),_83 run: async (payload, io, ctx) => {_83 const repos = await io.runTask("get-stars", async () => {_83 // get all libraries and current amount of stars_83 return await prisma.repository.groupBy({_83 by: ["name"],_83 _sum: {_83 stars: true,_83 },_83 });_83 });_83_83 //loop through all repos and invoke the Job that gets the latest stars_83 for (const repo of repos) {_83 getStars.invoke(repo.name, {_83 name: repo.name,_83 previousStarCount: repo?._sum?.stars || 0,_83 });_83 }_83 },_83});_83_83const getStars = client.defineJob({_83 id: "get-latest-stars",_83 name: "Get latest stars",_83 version: "0.0.1",_83 // Run a cron every day at 23:00 AM_83 trigger: invokeTrigger({_83 schema: z.object({_83 name: z.string(),_83 previousStarCount: z.number(),_83 }),_83 }),_83 run: async (payload, io, ctx) => {_83 const stargazers_count = await io.runTask("get-stars", async () => {_83 const { data } = await axios.get(_83 `https://api.github.com/repos/${payload.name}`,_83 {_83 headers: {_83 authorization: `token ${process.env.TOKEN}`,_83 },_83 }_83 );_83 return data.stargazers_count as number;_83 });_83_83 await prisma.repository.upsert({_83 where: {_83 name_day_month_year: {_83 name: payload.name,_83 month: new Date().getMonth() + 1,_83 year: new Date().getFullYear(),_83 day: new Date().getDate(),_83 },_83 },_83 update: {_83 stars: stargazers_count - payload.previousStarCount,_83 },_83 create: {_83 name: payload.name,_83 stars: stargazers_count - payload.previousStarCount,_83 month: new Date().getMonth() + 1,_83 year: new Date().getFullYear(),_83 day: new Date().getDate(),_83 },_83 });_83 },_83});
We created a new job called “Sync Stars Daily” that will run every day at 23:00pm - The representation of it in the cron text is: 0 23 * * *
We get all our current repositories in our database, group them by their name, and sum the stars.
Since everything runs on Vercel serverless, we might get to a timeout going over all the repositories.
For that, we send each repository to a different job.
We use the invoke
to create new jobs and then process them inside Get latest stars
We iterate over all the new repositories and get the current number of stars.
We remove the new number of stars by the old number of stars to get the today amount of stars.
We added it to the database with prisma
. there is no simpler than that!
The last thing is to edit jobs/index.ts
and replace the content with this:
_10export * from "./sync.stars";
And you are done 🥳
Let's connect! 🔌
As an open source developer, you're invited to join our community to contribute and engage with maintainers. Don't hesitate to visit our GitHub repository to contribute and create issues related to Trigger.dev.
The source for this tutorial is available here:
https://github.com/triggerdotdev/blog/tree/main/stars-monitor
Thank you for reading!