1 Commits

Author SHA1 Message Date
dependabot[bot]
e5a6192c31 Bump serve-static and express
Bumps [serve-static](https://github.com/expressjs/serve-static) to 1.16.2 and updates ancestor dependency [express](https://github.com/expressjs/express). These dependencies need to be updated together.


Updates `serve-static` from 1.15.0 to 1.16.2
- [Release notes](https://github.com/expressjs/serve-static/releases)
- [Changelog](https://github.com/expressjs/serve-static/blob/v1.16.2/HISTORY.md)
- [Commits](https://github.com/expressjs/serve-static/compare/v1.15.0...v1.16.2)

Updates `express` from 4.19.2 to 4.21.0
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/4.21.0/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.19.2...4.21.0)

---
updated-dependencies:
- dependency-name: serve-static
  dependency-type: indirect
- dependency-name: express
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-27 06:03:48 +00:00
16 changed files with 93 additions and 905 deletions

View File

@@ -50,27 +50,20 @@ HYBRID_NS_ACCESS=andkWEzbBWRR2K6iw4edRvfd6MmNNjRx
HYBRID_USER_AGENT=CSUX_blackdice
```
6. To start the service:
6. To start the service, run:
```
cd /usr/local/gcf1/
./sbin/gcf1 start
# Starting gcf1d services: ...succeeded
sbin/gcf1 start
Starting gcf1d services: ...succeeded
```
7. To download/update databases, run:
```
cd /usr/local/gcf1/
./bin/gcf1dbmng.sh etc urldb_download
# Success DB download .
./bin/gcf1dbmng.sh etc urldb_update
# Success DB update .
bin/gcf1dbmng.sh etc urldb_download
Success DB download .
bin/gcf1dbmng.sh etc urldb_update
Success DB update .
```
## UDP Server Setup
@@ -92,14 +85,14 @@ npm install
```
cp .env.example .env
# --------- example contents ---------
# UDP_PORT=33333
# HTTP_PORT=3333
# ------------------------------------
PORT=3000
```
## Digital Ocean Deployment details
1. The Node server is installed to `/opt/netstar-categorizer`
2. Rather than using `npm start` (which doesn't restart after a crash) the we use **[PM2](https://pm2.keymetrics.io/)**, which provides a more robust runtime
3. The full command to start the server using pm2 is `pm2 start npm --name "netstar-categorizer" --cwd /opt/netstar-categorizer/ -- start` - for convenience, there's a `run-me.sh` script in `/root` that will execute this command.
## Running the Server
To start the UDP server, use the following command:
```
npm start
```
This will run the server and bind it to the port specified in your .env file.

290
claude.md
View File

@@ -1,290 +0,0 @@
# Claude Development Guide - NetStar Categorizer
## Project Overview
**NetStar Categorizer** is a Node.js microservice that provides domain name/FQDN categorization using the inCompass NetStar SDK. It exposes both UDP and HTTP interfaces for real-time content classification with automatic daily database updates.
### Core Purpose
- Categorize domains into standardized categories using NetStar's content database
- Map raw NetStar category IDs to organization-specific category codes
- Provide both simple and detailed categorization results with reputation/age ratings
- Maintain updated categorization databases through automated cron jobs
## Key Architecture
### Technology Stack
- **Runtime**: Node.js v14+
- **Web Framework**: Express.js v4
- **External SDK**: inCompass NetStar v3.1.0-2 (C++ library for categorization)
- **Containerization**: Docker + Kubernetes
- **CI/CD**: CircleCI
- **Infrastructure**: Cloud-agnostic (DigitalOcean, Google Cloud, etc.)
### Core Services
1. **HTTP Server** (Port 3333) - REST API for categorization requests
2. **UDP Server** (Port 33333) - Legacy UDP interface for categorization
3. **Cron Service** - Daily database updates via Kubernetes CronJob
4. **Category Mapping** - NetStar ID → Organization Code conversion (singleton pattern)
## Project Structure
```
src/
├── server.js # Entry point: initializes UDP + HTTP servers
├── app.js # Core logic: orchestrates categorization flow
├── client.js # UDP test client
├── cron.js # Scheduled database update logic
├── use-cases/
│ ├── get-category-use-case.js # Executes NetStar gcf1check command
│ ├── category-converter-use-case.js # Maps NetStar IDs → org codes (singleton)
│ ├── parse-detailed-category-use-case.js # Parses detailed output with ratings
│ └── update-categories-use-case.js # Updates NetStar databases
└── etc/
└── categories-mapping.json # Mapping table: NetStar ID → Zvelo codes
deployment/
├── deployment.yaml # Kubernetes deployment manifest
└── staging/deployment.yaml # Staging-specific config
.circleci/config.yml # CI/CD pipeline (builds, tests, deploys)
Dockerfile # Container build specification
package.json # Node dependencies + scripts
makefile # Convenience commands
test-detailed.http # HTTP endpoint test file
```
## Language Standards
### Comments and Error Messages - ENGLISH ONLY
**All code comments, error messages, log statements, and documentation must be written in English.**
This includes:
- ✅ Code comments explaining logic
- ✅ Error messages and exception messages
- ✅ Console.log, console.error, and logging statements
- ✅ Variable and function names
- ✅ Commit messages
- ✅ Code review feedback
- ✅ Documentation strings (JSDoc, etc.)
**Examples:**
```javascript
// Correct: English comment
function mapCategoryId(netstarId) {
if (!netstarId) {
throw new Error('NetStar ID is required')
}
// Map to organization category code
return categoryConverter.convert(netstarId)
}
// Incorrect: Portuguese comment
function mapCategoryId(netstarId) {
if (!netstarId) {
throw new Error('ID do NetStar é obrigatório') // ❌ ERROR IN PORTUGUESE
}
// Mapear para código de categoria da organização // ❌ COMMENT IN PORTUGUESE
return categoryConverter.convert(netstarId)
}
```
## Coding Conventions
### Code Style
- Use **ES6 syntax** (const/let, arrow functions, template literals)
- **No semicolons** in new code (already established pattern)
- Functional/modular design - keep files focused on single responsibility
- Use **singleton pattern** for shared state (see: `CategoryConverterUseCase`)
- **All comments in English** - see Language Standards section
### Use Case Pattern
- Each business operation gets a dedicated use case class in `src/use-cases/`
- Use case classes should have a clear, single responsibility
- Example:
```javascript
class GetCategoryUseCase {
async execute(fqdn) {
// implementation
}
}
```
### Environment Variables
- Defined in `.env` (create from `.env.example`)
- `UDP_PORT=33333` - UDP server listen port
- `HTTP_PORT=3333` - HTTP server listen port
## HTTP API Endpoints
### `POST /` - Basic Categorization
- **Input**: `{"fqdn": "example.com"}`
- **Output**: `{"result": [10009, 10010]}` (array of category IDs)
- **Use Case**: Quick lookups when detailed info not needed
### `POST /detailed` - Detailed Categorization
- **Input**: `{"fqdn": "example.com"}`
- **Output**: Full category info with reputation score, age rating, primary/secondary categories, and human-readable names
- **Use Case**: Comprehensive categorization for security decisions
- **Includes**: `result` array matching the `/` endpoint format
### `POST /full` - Complete Raw Categorization Output
- **Input**: `{"fqdn": "example.com"}`
- **Output**: Complete detailed structure with all 35 fields from NetStar including:
- Primary, secondary, and security categories (with IDs, names, and mapped IDs)
- Reputation score and name
- Matching flags and their descriptions
- Age rating score and name
- All 9 category group classifications (Internet/Infrastructure, Malware/Security, Dangerous/Harmful, Adult, Business/Government, Personal, Computing/Technology, Social Media, Miscellaneous)
- Volume index
- Submitted URL
- **Use Case**: Complete diagnostic and analysis when all categorization data is needed
- **Includes**: `result` array matching the `/` endpoint format
### UDP Server (Port 33333)
- **Input**: Raw domain string (e.g., `"example.com"`)
- **Output**: JSON-formatted result (same as HTTP `/` endpoint)
- **Legacy Interface**: Maintained for backward compatibility
## Development Workflow
### Setup & Running Locally
```bash
# Install dependencies
npm install
# Development with auto-reload
npm run dev:server # Watch mode for server changes
npm run dev:client # Run UDP client for testing
# Production
npm start # Start both servers
# NetStar service commands (Linux system)
make gcf1-start # Start NetStar service
make gcf1-download # Download category databases
make gcf1-update # Update category databases
```
### Testing APIs
Use `test-detailed.http` in VS Code REST Client extension:
1. Open the file
2. Click "Send Request" on each endpoint
3. View responses in the side panel
### Git Workflow
- **Main Branch**: `main` - production stable code
- **Development Branch**: `development` - feature integration
- **Feature Branches**: Create from `development`, merge back via PR
- Recent commits show HTTP approach implementation and cron job additions
- **Commit Messages**: Must be in English
## Deployment
### Docker
- **Base Image**: Ubuntu 22.04
- **Includes**: Boost libraries, Node.js, NetStar SDK
- **Exposes**: Port 3000 (UDP)
- Build: `docker build -t netstar-categorizer .`
### Kubernetes (Production)
- **Namespace**: `blackdice`
- **Deployment**: Single replica in appropriate cluster
- **CronJob**: Daily database updates at 00:00 UTC
- **Ingress**: `netstar-cat-dev.blackdice.ai` (DNS varies by environment)
- **Branches to Environments**:
- `development` → development cluster
- `qa` → QA cluster
- `staging` → staging cluster
- `production` → production cluster
- `gke-staging`, `gke-pov` → specific GKE clusters
### CI/CD Pipeline (CircleCI)
- Automatically builds Docker image on push
- Tags image with commit SHA
- Deploys to appropriate Kubernetes cluster based on branch
- Release deployments via git tags
## Key Technical Details
### Category Mapping System
- **Source**: NetStar SDK returns numeric category IDs (e.g., 101, 102)
- **Mapping File**: `src/etc/categories-mapping.json`
- **Target**: Maps to organization's Zvelo pattern codes (e.g., 10075, 10078)
- **Singleton Implementation**: `CategoryConverterUseCase` maintains single instance across app
- **Example Mapping**:
- NetStar 101 (Illegal Activities) → Zvelo 10075
- NetStar 201 (Terrorism/Extremists) → Zvelo 10018
### NetStar SDK Integration
- **Command**: `gcf1check` - queries the NetStar database for domain categorization
- **Child Process**: Executed via Node.js `child_process` module
- **Output Parsing**: Raw output parsed into JSON structure
- **Detailed Mode**: Includes reputation scores and age ratings in output
### Automatic Database Updates
- **Mechanism**: Kubernetes CronJob at 0 0 * * * (daily at midnight UTC)
- **Fallback**: Manual update via `make gcf1-update`
- **Purpose**: Keeps categorization database current with latest NetStar classifications
## Common Tasks
### Adding a New Endpoint
1. Create a corresponding use case in `src/use-cases/`
2. Add route in `src/app.js` that calls the use case
3. Export and test in `test-detailed.http`
4. Update this guide if it's a significant feature
5. Ensure all error messages and comments are in English
### Updating Category Mappings
1. Modify `src/etc/categories-mapping.json` with new ID mappings
2. Restart the service (singleton will reload on next request)
3. Test with both HTTP and UDP interfaces
### Debugging
- **Server Logs**: Check Docker/Kubernetes logs for errors
- **Cron Logs**: View Kubernetes CronJob logs for database update issues
- **UDP Testing**: Use `npm run dev:client` to test directly
- **HTTP Testing**: Use `test-detailed.http` with VS Code REST Client
- **Error Messages**: All error logs must be in English
### Troubleshooting
- **NetStar Service Not Running**: Run `make gcf1-start`
- **Stale Categories**: Manually run `make gcf1-update` or wait for cron job
- **Port Conflicts**: Ensure ports 3333 (HTTP) and 33333 (UDP) are available
- **Docker Build Issues**: Check that Boost C++ libraries are installed correctly
## Current Development Status
### Recent Work
- ✅ HTTP server implementation (alongside UDP)
- ✅ Detailed categorization with reputation/age ratings
- ✅ Cron job for automated daily updates
- ✅ Singleton category converter pattern
- 🔄 Work in Progress:
- `playground.js` - experimental/testing code
- `parse-detailed-category-use-case.js` - new detailed parsing feature
- Enhanced `server.js` - expanded server capabilities
### Known Modified Files
- `playground.js` - development/testing (can be cleaned up)
- `src/server.js` - recent enhancements
- `makefile` - new convenience commands
- `test-detailed.http` - expanded test coverage
## Guidelines for Contributions
1. **Follow Existing Patterns**: Use use-case classes, follow module structure
2. **Test Before Committing**: Use `test-detailed.http` for API changes
3. **Update Mappings Properly**: Edit `categories-mapping.json`, not hardcode values
4. **Document Breaking Changes**: Update this guide if architecture changes
5. **Keep CircleCI Happy**: Ensure Docker build succeeds and K8s deployment configs are valid
6. **Don't Skip Steps**: Always test UDP and HTTP interfaces for categorization changes
7. **Language Standards**: All comments, error messages, and logs must be in English
## Resources & External Documentation
- **NetStar SDK**: Installed in Docker, documentation in inCompass SDK v3.1.0-2
- **Express.js**: https://expressjs.com
- **Node.js Child Process**: https://nodejs.org/api/child_process.html
- **Kubernetes**: https://kubernetes.io/docs
- **CircleCI**: Configuration at `.circleci/config.yml`

View File

@@ -1,8 +0,0 @@
gcf1-start:
cd /usr/local/gcf1 && sbin/gcf1 start
gcf1-download:
cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_download
gcf1-update:
cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_update

93
package-lock.json generated
View File

@@ -10,7 +10,7 @@
"license": "ISC",
"dependencies": {
"dotenv": "^16.4.5",
"express": "^4.19.2"
"express": "^4.21.0"
}
},
"node_modules/accepts": {
@@ -31,9 +31,9 @@
"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg=="
},
"node_modules/body-parser": {
"version": "1.20.2",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz",
"integrity": "sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
"version": "1.20.3",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.3.tgz",
"integrity": "sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g==",
"dependencies": {
"bytes": "3.1.2",
"content-type": "~1.0.5",
@@ -43,7 +43,7 @@
"http-errors": "2.0.0",
"iconv-lite": "0.4.24",
"on-finished": "2.4.1",
"qs": "6.11.0",
"qs": "6.13.0",
"raw-body": "2.5.2",
"type-is": "~1.6.18",
"unpipe": "1.0.0"
@@ -169,9 +169,9 @@
"integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
},
"node_modules/encodeurl": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
"integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
"integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
"engines": {
"node": ">= 0.8"
}
@@ -209,36 +209,36 @@
}
},
"node_modules/express": {
"version": "4.19.2",
"resolved": "https://registry.npmjs.org/express/-/express-4.19.2.tgz",
"integrity": "sha512-5T6nhjsT+EOMzuck8JjBHARTHfMht0POzlA60WV2pMD3gyXw2LZnZ+ueGdNxG+0calOJcWKbpFcuzLZ91YWq9Q==",
"version": "4.21.0",
"resolved": "https://registry.npmjs.org/express/-/express-4.21.0.tgz",
"integrity": "sha512-VqcNGcj/Id5ZT1LZ/cfihi3ttTn+NJmkli2eZADigjq29qTlWi/hAQ43t/VLPq8+UX06FCEx3ByOYet6ZFblng==",
"dependencies": {
"accepts": "~1.3.8",
"array-flatten": "1.1.1",
"body-parser": "1.20.2",
"body-parser": "1.20.3",
"content-disposition": "0.5.4",
"content-type": "~1.0.4",
"cookie": "0.6.0",
"cookie-signature": "1.0.6",
"debug": "2.6.9",
"depd": "2.0.0",
"encodeurl": "~1.0.2",
"encodeurl": "~2.0.0",
"escape-html": "~1.0.3",
"etag": "~1.8.1",
"finalhandler": "1.2.0",
"finalhandler": "1.3.1",
"fresh": "0.5.2",
"http-errors": "2.0.0",
"merge-descriptors": "1.0.1",
"merge-descriptors": "1.0.3",
"methods": "~1.1.2",
"on-finished": "2.4.1",
"parseurl": "~1.3.3",
"path-to-regexp": "0.1.7",
"path-to-regexp": "0.1.10",
"proxy-addr": "~2.0.7",
"qs": "6.11.0",
"qs": "6.13.0",
"range-parser": "~1.2.1",
"safe-buffer": "5.2.1",
"send": "0.18.0",
"serve-static": "1.15.0",
"send": "0.19.0",
"serve-static": "1.16.2",
"setprototypeof": "1.2.0",
"statuses": "2.0.1",
"type-is": "~1.6.18",
@@ -250,12 +250,12 @@
}
},
"node_modules/finalhandler": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.2.0.tgz",
"integrity": "sha512-5uXcUVftlQMFnWC9qu/svkWv3GTd2PfUhK/3PLkYNAe7FbqJMt3515HaxE6eRL74GdsriiwujiawdaB1BpEISg==",
"version": "1.3.1",
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.1.tgz",
"integrity": "sha512-6BN9trH7bp3qvnrRyzsBz+g3lZxTNZTbVO2EV1CS0WIcDbawYVdYvGflME/9QP0h0pYlCDBCTjYa9nZzMDpyxQ==",
"dependencies": {
"debug": "2.6.9",
"encodeurl": "~1.0.2",
"encodeurl": "~2.0.0",
"escape-html": "~1.0.3",
"on-finished": "2.4.1",
"parseurl": "~1.3.3",
@@ -411,9 +411,12 @@
}
},
"node_modules/merge-descriptors": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
"integrity": "sha512-cCi6g3/Zr1iqQi6ySbseM1Xvooa98N0w31jzUYrXPX2xqObmFGHJ0tQ5u74H3mVh7wLouTseZyYIq39g8cNp1w=="
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz",
"integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==",
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/methods": {
"version": "1.1.2",
@@ -497,9 +500,9 @@
}
},
"node_modules/path-to-regexp": {
"version": "0.1.7",
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
"integrity": "sha512-5DFkuoqlv1uYQKxy8omFBeJPQcdoE07Kv2sferDCrAq1ohOU+MSDswDIbnx3YAM60qIOnYa53wBhXW0EbMonrQ=="
"version": "0.1.10",
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.10.tgz",
"integrity": "sha512-7lf7qcQidTku0Gu3YDPc8DJ1q7OOucfa/BSsIwjuh56VU7katFvuM8hULfkwB3Fns/rsVF7PwPKVw1sl5KQS9w=="
},
"node_modules/proxy-addr": {
"version": "2.0.7",
@@ -514,11 +517,11 @@
}
},
"node_modules/qs": {
"version": "6.11.0",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.11.0.tgz",
"integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
"version": "6.13.0",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.13.0.tgz",
"integrity": "sha512-+38qI9SOr8tfZ4QmJNplMUxqjbe7LKvvZgWdExBOmd+egZTtjLB67Gu0HRX3u/XOq7UU2Nx6nsjvS16Z9uwfpg==",
"dependencies": {
"side-channel": "^1.0.4"
"side-channel": "^1.0.6"
},
"engines": {
"node": ">=0.6"
@@ -574,9 +577,9 @@
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
},
"node_modules/send": {
"version": "0.18.0",
"resolved": "https://registry.npmjs.org/send/-/send-0.18.0.tgz",
"integrity": "sha512-qqWzuOjSFOuqPjFe4NOsMLafToQQwBSOEpS+FwEt3A2V3vKubTquT3vmLTQpFgMXp8AlFWFuP1qKaJZOtPpVXg==",
"version": "0.19.0",
"resolved": "https://registry.npmjs.org/send/-/send-0.19.0.tgz",
"integrity": "sha512-dW41u5VfLXu8SJh5bwRmyYUbAoSB3c9uQh6L8h/KtsFREPWpbX1lrljJo186Jc4nmci/sGUZ9a0a0J2zgfq2hw==",
"dependencies": {
"debug": "2.6.9",
"depd": "2.0.0",
@@ -596,20 +599,28 @@
"node": ">= 0.8.0"
}
},
"node_modules/send/node_modules/encodeurl": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
"integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
"engines": {
"node": ">= 0.8"
}
},
"node_modules/send/node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
},
"node_modules/serve-static": {
"version": "1.15.0",
"resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.15.0.tgz",
"integrity": "sha512-XGuRDNjXUijsUL0vl6nSD7cwURuzEgglbOaFuZM9g3kwDXOWVTck0jLzjPzGD+TazWbboZYu52/9/XPdUgne9g==",
"version": "1.16.2",
"resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.2.tgz",
"integrity": "sha512-VqpjJZKadQB/PEbEwvFdO43Ax5dFBZ2UECszz8bQ7pi7wt//PWe1P6MN7eCnjsatYtBT6EuiClbjSWP2WrIoTw==",
"dependencies": {
"encodeurl": "~1.0.2",
"encodeurl": "~2.0.0",
"escape-html": "~1.0.3",
"parseurl": "~1.3.3",
"send": "0.18.0"
"send": "0.19.0"
},
"engines": {
"node": ">= 0.8.0"

View File

@@ -13,6 +13,6 @@
"description": "",
"dependencies": {
"dotenv": "^16.4.5",
"express": "^4.19.2"
"express": "^4.21.0"
}
}

View File

@@ -7,12 +7,12 @@ const { spawn, exec } = require("node:child_process")
// })
exec("echo 'li12.pages.dev' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
exec("echo '99999.incompass.netstar-inc.com' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
if (error) {
console.error(error)
return
}
console.log(stdout.trim().split(/\t|\s{2,}/))
console.log(stdout.trim())
})

View File

@@ -7,12 +7,10 @@ const getCategory = new GetCategoryUseCase();
const categoryConverter = new CategoryConverterUseCase({categoriesMapping});
module.exports = async function app(domain){
console.log(domain)
const category = await getCategory.execute(domain)
console.log(category)
const categoryConverted = await categoryConverter.execute(category)
console.log(categoryConverted)
return [categoryConverted]
return categoryConverted
}

View File

@@ -3,10 +3,8 @@ const { UpdateCategoriesUseCase } = require('./use-cases/update-categories-use-c
const updateCategories = new UpdateCategoriesUseCase()
module.exports = async function cron() {
updateCategories.execute()
const oneDay = 1000 * 60 * 60 * 24
setInterval(() => {
setImmediate(() => {
updateCategories.execute()
}, oneDay)
}

View File

@@ -1074,7 +1074,7 @@
"id": "20097",
"description": "VPN",
"related": [
"10460"
"10451"
]
},
{
@@ -1109,7 +1109,7 @@
"id": "20103",
"description": "Code Repositories",
"related": [
"10047"
"10429"
]
},
{
@@ -1462,4 +1462,4 @@
"10094"
]
}
]
]

View File

@@ -5,23 +5,14 @@ const app = require('./app')
const bodyParser = require('body-parser')
const cron = require('./cron')
const express = require('express')
const { ParseDetailedCategoryUseCase } = require('./use-cases/parse-detailed-category-use-case')
const { ParseFullCategoryUseCase } = require('./use-cases/parse-full-category-use-case')
const { CategoryConverterUseCase } = require('./use-cases/category-converter-use-case')
const categoriesMapping = require('./etc/categories-mapping.json')
const httpServer = express()
httpServer.use(bodyParser.json()) // for parsing application/json
httpServer.use(bodyParser.urlencoded({ extended: true })) // for parsing application/x-www-form-urlencoded
cron()
const UDP_PORT = process.env.UDP_PORT
const HTTP_PORT = process.env.HTTP_PORT
// Initialize use cases
const categoryConverter = new CategoryConverterUseCase({ categoriesMapping })
const parseDetailedCategory = new ParseDetailedCategoryUseCase({ categoryConverter })
const parseFullCategory = new ParseFullCategoryUseCase({ categoryConverter })
const UDP_PORT = process.env.UDP_PORT
const HTTP_PORT = process.env.HTTP_PORT
const server = dgram.createSocket('udp4');
@@ -59,55 +50,27 @@ server.on('listening', () => {
server.bind(UDP_PORT);
httpServer.post('/', async (req, res) => {
console.log("new request")
try {
const { fqdn } = req.body
const categories = await app(fqdn)
let result = {}
if (categories)
if (categories)
result = {
result: categories
}
res.status(200).json(JSON.stringify(result))
} catch (err) {
res.status(500).json(err)
}
})
httpServer.post('/detailed', async (req, res) => {
try {
const { fqdn } = req.body
const detailedResult = await parseDetailedCategory.execute(fqdn)
res.status(200).json(detailedResult)
} catch (err) {
console.error('Error in /detailed endpoint:', err)
res.status(500).json({ error: err.message || err })
}
})
httpServer.post('/full', async (req, res) => {
try {
const { fqdn } = req.body
const fullResult = await parseFullCategory.execute(fqdn)
res.status(200).json(fullResult)
} catch (err) {
console.error('Error in /full endpoint:', err)
res.status(500).json({ error: err.message || err })
}
})
httpServer.listen(HTTP_PORT, () => {
console.log('HTTP server listening 3333')
console.log('UDP server listening 3334')
})
// Response

View File

@@ -1,10 +0,0 @@
const app = require('./app');
(async () => {
try {
const categories = await app('www.facebook.com');
console.log(categories);
} catch (error) {
console.error('Error occurred:', error);
}
})();

View File

@@ -11,7 +11,7 @@ class CategoryConverterUseCase {
execute(category) {
const entry = this.categoriesMapping.find(item => item.id === category);
return entry ? entry.related[0] : null;
return entry ? entry.related[0].split(', ').map(str => str.trim()) : null;
}
}

View File

@@ -1,101 +0,0 @@
const { exec } = require("node:child_process")
class ParseDetailedCategoryUseCase {
constructor({ categoryConverter }) {
this.categoryConverter = categoryConverter;
}
execute(domain) {
return new Promise((resolve, reject) => {
exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
{ cwd: '/usr/local/gcf1' },
(error, stdout, stderr) => {
if (error) {
console.error(error);
reject(error);
return;
}
try {
const parsed = this.parseOutput(stdout);
resolve(parsed);
} catch (parseError) {
console.error('Parse error:', parseError);
reject(parseError);
}
});
});
}
parseOutput(output) {
// Extract all quoted strings in order
const quotedStrings = [];
const quoteRegex = /"([^"]*)"/g;
let match;
while ((match = quoteRegex.exec(output)) !== null) {
quotedStrings.push(match[1]);
}
// Split on tabs (primary separator in NetStar output)
const parts = output.trim().split('\t');
let quotedIndex = 0;
// Position mapping based on NetStar output structure:
// parts[0]: Categorized
// parts[1]: Matching Index
// parts[2]: Primary Category ID
// parts[3]: Primary Category Name (quoted)
// parts[4]: Secondary Category ID
// parts[5]: Secondary Category Name (quoted if secondary exists)
// parts[6]: Security Category ID
// parts[7]: Security Category Name
// parts[8]: Reputation Score ID
// parts[9]: Reputation Score Name (quoted)
// parts[10]: Matching Flag Value
// parts[11]: Matching Flag Names
// parts[12]: Age Rating ID
// parts[13]: Age Rating Name (quoted)
const primaryId = parseInt(parts[2]);
const primaryName = quotedStrings[quotedIndex++];
const primaryMapped = this.categoryConverter.execute(String(primaryId));
const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
let secondaryName = null;
let secondaryMapped = null;
if (secondaryId !== null) {
secondaryName = quotedStrings[quotedIndex++];
secondaryMapped = this.categoryConverter.execute(String(secondaryId));
} else {
secondaryName = parts[5] === '-' ? null : parts[5];
}
const reputationId = parseInt(parts[8]);
const reputationName = quotedStrings[quotedIndex++];
const ageRatingId = parseInt(parts[12]);
const ageRatingName = quotedStrings[quotedIndex++];
// Build result array in the same format as the / endpoint
const resultArray = [String(primaryMapped)];
if (secondaryMapped !== null) {
resultArray.push(String(secondaryMapped));
}
return {
result: resultArray,
primary: primaryMapped,
primary_name: primaryName,
secondary: secondaryMapped,
secondary_name: secondaryName,
reputation: reputationId,
reputation_name: reputationName,
age_rating: ageRatingId,
age_rating_name: ageRatingName,
raw_output: output.trim()
};
}
}
module.exports = { ParseDetailedCategoryUseCase }

View File

@@ -1,172 +0,0 @@
const { exec } = require("node:child_process")
class ParseFullCategoryUseCase {
constructor({ categoryConverter }) {
this.categoryConverter = categoryConverter;
}
execute(domain) {
return new Promise((resolve, reject) => {
exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
{ cwd: '/usr/local/gcf1' },
(error, stdout, stderr) => {
if (error) {
console.error(error);
reject(error);
return;
}
try {
const parsed = this.parseOutput(stdout);
resolve(parsed);
} catch (parseError) {
console.error('Parse error:', parseError);
reject(parseError);
}
});
});
}
parseOutput(output) {
const quotedStrings = [];
const quoteRegex = /"([^"]*)"/g;
let match;
while ((match = quoteRegex.exec(output)) !== null) {
quotedStrings.push(match[1]);
}
const parts = output.trim().split('\t');
let quotedIndex = 0;
// Extract primary category (parts[2], parts[3])
const primaryId = parseInt(parts[2]);
const primaryName = quotedStrings[quotedIndex++];
const primaryMapped = this.categoryConverter.execute(String(primaryId));
// Extract secondary category (parts[4], parts[5])
const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
let secondaryName = null;
let secondaryMapped = null;
if (secondaryId !== null && parts[5] !== '-') {
secondaryName = quotedStrings[quotedIndex++];
secondaryMapped = this.categoryConverter.execute(String(secondaryId));
} else {
secondaryName = parts[5] === '-' ? null : parts[5];
}
// Extract security category (parts[6], parts[7])
const securityId = parts[6] !== '0' && parts[6] !== '-' ? parseInt(parts[6]) : null;
let securityName = null;
let securityMapped = null;
if (securityId !== null && parts[7] !== '-') {
securityName = quotedStrings[quotedIndex++];
securityMapped = this.categoryConverter.execute(String(securityId));
} else {
securityName = parts[7] === '-' ? null : parts[7];
}
// Extract reputation (parts[8], parts[9])
const reputationId = parseInt(parts[8]);
const reputationName = quotedStrings[quotedIndex++];
// Extract matching flag (parts[10], parts[11])
const matchingFlagValue = parts[10];
const matchingFlagNames = parts[11] ? parts[11].split('|') : [];
// Extract age rating (parts[12], parts[13])
const ageRatingId = parseInt(parts[12]);
const ageRatingName = quotedStrings[quotedIndex++];
// Extract category groups starting from parts[16] (after 2 empty fields)
// Each group has: id (number), name (quoted string)
// Pattern: 0 "Internet/Infrastructure" 0 "Malware/Security" ...
const categoryGroups = {
internet_infrastructure: {
id: parseInt(parts[16]) || 0,
name: quotedStrings[quotedIndex++]
},
malware_security: {
id: parseInt(parts[18]) || 0,
name: quotedStrings[quotedIndex++]
},
dangerous_harmful: {
id: parseInt(parts[20]) || 0,
name: quotedStrings[quotedIndex++]
},
adult: {
id: parseInt(parts[22]) || 0,
name: quotedStrings[quotedIndex++]
},
business_government: {
id: parseInt(parts[24]) || 0,
name: quotedStrings[quotedIndex++]
},
personal: {
id: parseInt(parts[26]) || 0,
name: quotedStrings[quotedIndex++]
},
computing_technology: {
id: parseInt(parts[28]) || 0,
name: quotedStrings[quotedIndex++]
},
social_media: {
id: parseInt(parts[30]) || 0,
name: quotedStrings[quotedIndex++]
},
miscellaneous: {
id: parseInt(parts[32]) || 0,
name: quotedStrings[quotedIndex++]
}
};
// Extract volume index and submitted URL
const volumeIndex = parts[35];
const submittedUrl = parts[36];
const result = {
result_status: parts[0],
matching_index: parseInt(parts[1]),
primary_category: {
id: primaryId,
name: primaryName,
mapped_id: String(primaryMapped)
},
secondary_category: {
id: secondaryId,
name: secondaryName,
mapped_id: secondaryMapped ? String(secondaryMapped) : null
},
security_category: {
id: securityId,
name: securityName,
mapped_id: securityMapped ? String(securityMapped) : null
},
reputation: {
id: reputationId,
name: reputationName
},
matching_flag: {
value: matchingFlagValue,
names: matchingFlagNames
},
age_rating: {
id: ageRatingId,
name: ageRatingName
},
category_groups: categoryGroups,
volume_index: volumeIndex,
submitted_url: submittedUrl
};
// Add result array in the same format as / endpoint
const resultArray = [result.primary_category.mapped_id];
if (result.secondary_category.mapped_id !== null) {
resultArray.push(result.secondary_category.mapped_id);
}
result.result = resultArray;
return result;
}
}
module.exports = { ParseFullCategoryUseCase }

View File

@@ -1,26 +1,20 @@
const { exec } = require("node:child_process");
const { exec } = require("node:child_process")
class UpdateCategoriesUseCase {
execute() {
exec(`bin/gcf1dbmng.sh etc urldb_download`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
if (error) {
console.error('Erro no primeiro comando:', error);
return;
}
console.log('Saída do primeiro comando:', stdout);
// Executa o segundo comando após o primeiro ter sido concluído
exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error2, stdout2, stderr2) => {
if (error2) {
console.error('Erro no segundo comando:', error2);
return new Promise((_, reject) => {
exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
if (error) {
console.error(error);
reject(error);
return;
}
console.log('Saída do segundo comando:', stdout2);
console.log(stdout)
});
});
}
}
module.exports = { UpdateCategoriesUseCase };
module.exports = { UpdateCategoriesUseCase }

View File

@@ -1,188 +0,0 @@
@baseUrl = http://localhost:3333
# ============================================
# POST / - Basic Categorization Endpoint
# Returns: Simple result array of category IDs
# ============================================
### Basic endpoint - Facebook
POST {{baseUrl}}/
Content-Type: application/json
{
"fqdn": "facebook.com"
}
### Basic endpoint - Google
POST {{baseUrl}}/
Content-Type: application/json
{
"fqdn": "google.com"
}
### Basic endpoint - TikTok
POST {{baseUrl}}/
Content-Type: application/json
{
"fqdn": "tiktok.com"
}
### Basic endpoint - YouTube
POST {{baseUrl}}/
Content-Type: application/json
{
"fqdn": "youtube.com"
}
### Basic endpoint - Reddit
POST {{baseUrl}}/
Content-Type: application/json
{
"fqdn": "reddit.com"
}
### Basic endpoint - Twitter
POST {{baseUrl}}/
Content-Type: application/json
{
"fqdn": "twitter.com"
}
# ============================================
# POST /detailed - Detailed Categorization
# Returns: Categories with names, reputation, age rating
# ============================================
### Detailed endpoint - Facebook
POST {{baseUrl}}/detailed
Content-Type: application/json
{
"fqdn": "facebook.com"
}
### Detailed endpoint - Google
POST {{baseUrl}}/detailed
Content-Type: application/json
{
"fqdn": "google.com"
}
### Detailed endpoint - TikTok
POST {{baseUrl}}/detailed
Content-Type: application/json
{
"fqdn": "tiktok.com"
}
### Detailed endpoint - YouTube
POST {{baseUrl}}/detailed
Content-Type: application/json
{
"fqdn": "youtube.com"
}
### Detailed endpoint - Reddit
POST {{baseUrl}}/detailed
Content-Type: application/json
{
"fqdn": "reddit.com"
}
### Detailed endpoint - Twitter
POST {{baseUrl}}/detailed
Content-Type: application/json
{
"fqdn": "twitter.com"
}
# ============================================
# POST /full - Complete Raw Output
# Returns: All 35 fields from NetStar with category groups
# ============================================
### Full endpoint - Facebook
POST {{baseUrl}}/full
Content-Type: application/json
{
"fqdn": "facebook.com"
}
### Full endpoint - Google
POST {{baseUrl}}/full
Content-Type: application/json
{
"fqdn": "google.com"
}
### Full endpoint - TikTok
POST {{baseUrl}}/full
Content-Type: application/json
{
"fqdn": "tiktok.com"
}
### Full endpoint - YouTube
POST {{baseUrl}}/full
Content-Type: application/json
{
"fqdn": "youtube.com"
}
### Full endpoint - Reddit
POST {{baseUrl}}/full
Content-Type: application/json
{
"fqdn": "reddit.com"
}
### Full endpoint - Twitter
POST {{baseUrl}}/full
Content-Type: application/json
{
"fqdn": "twitter.com"
}
# ============================================
# Edge cases and special domains
# ============================================
### Basic endpoint - Unknown domain
POST {{baseUrl}}/
Content-Type: application/json
{
"fqdn": "unknowndomainexample12345.com"
}
### Detailed endpoint - Localhost
POST {{baseUrl}}/detailed
Content-Type: application/json
{
"fqdn": "localhost"
}
### Full endpoint - IP-like domain
POST {{baseUrl}}/full
Content-Type: application/json
{
"fqdn": "99999.incompass.netstar-inc.com"
}