Compare commits
17 Commits
dependabot
...
developmen
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
41358ea0ae | ||
|
|
69ea828968 | ||
|
|
cd15760e84 | ||
|
|
d0f4b58559 | ||
|
|
dfda9eac9a | ||
|
|
14d736c97e | ||
|
|
e630ca4868 | ||
|
|
3b259c4457 | ||
|
|
0bc1adb738 | ||
|
|
02c1df98d3 | ||
|
|
6f4057eb62 | ||
|
|
99f340e6a5 | ||
|
|
ce61dc25b4 | ||
|
|
4db9094584 | ||
|
|
947de9f2ea | ||
|
|
8cd0461088 | ||
|
|
7731c0fe27 |
39
README.md
39
README.md
@@ -50,20 +50,27 @@ HYBRID_NS_ACCESS=andkWEzbBWRR2K6iw4edRvfd6MmNNjRx
|
||||
HYBRID_USER_AGENT=CSUX_blackdice
|
||||
```
|
||||
|
||||
6. To start the service, run:
|
||||
6. To start the service:
|
||||
|
||||
```
|
||||
sbin/gcf1 start
|
||||
Starting gcf1d services: ...succeeded
|
||||
cd /usr/local/gcf1/
|
||||
|
||||
./sbin/gcf1 start
|
||||
|
||||
# Starting gcf1d services: ...succeeded
|
||||
```
|
||||
|
||||
7. To download/update databases, run:
|
||||
|
||||
```
|
||||
bin/gcf1dbmng.sh etc urldb_download
|
||||
Success DB download .
|
||||
bin/gcf1dbmng.sh etc urldb_update
|
||||
Success DB update .
|
||||
cd /usr/local/gcf1/
|
||||
|
||||
./bin/gcf1dbmng.sh etc urldb_download
|
||||
|
||||
# Success DB download .
|
||||
|
||||
./bin/gcf1dbmng.sh etc urldb_update
|
||||
# Success DB update .
|
||||
```
|
||||
|
||||
## UDP Server Setup
|
||||
@@ -85,14 +92,14 @@ npm install
|
||||
|
||||
```
|
||||
cp .env.example .env
|
||||
PORT=3000
|
||||
|
||||
# --------- example contents ---------
|
||||
# UDP_PORT=33333
|
||||
# HTTP_PORT=3333
|
||||
# ------------------------------------
|
||||
```
|
||||
|
||||
## Running the Server
|
||||
|
||||
To start the UDP server, use the following command:
|
||||
|
||||
```
|
||||
npm start
|
||||
```
|
||||
This will run the server and bind it to the port specified in your .env file.
|
||||
## Digital Ocean Deployment details
|
||||
1. The Node server is installed to `/opt/netstar-categorizer`
|
||||
2. Rather than using `npm start` (which doesn't restart after a crash) the we use **[PM2](https://pm2.keymetrics.io/)**, which provides a more robust runtime
|
||||
3. The full command to start the server using pm2 is `pm2 start npm --name "netstar-categorizer" --cwd /opt/netstar-categorizer/ -- start` - for convenience, there's a `run-me.sh` script in `/root` that will execute this command.
|
||||
|
||||
290
claude.md
Normal file
290
claude.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Claude Development Guide - NetStar Categorizer
|
||||
|
||||
## Project Overview
|
||||
|
||||
**NetStar Categorizer** is a Node.js microservice that provides domain name/FQDN categorization using the inCompass NetStar SDK. It exposes both UDP and HTTP interfaces for real-time content classification with automatic daily database updates.
|
||||
|
||||
### Core Purpose
|
||||
- Categorize domains into standardized categories using NetStar's content database
|
||||
- Map raw NetStar category IDs to organization-specific category codes
|
||||
- Provide both simple and detailed categorization results with reputation/age ratings
|
||||
- Maintain updated categorization databases through automated cron jobs
|
||||
|
||||
## Key Architecture
|
||||
|
||||
### Technology Stack
|
||||
- **Runtime**: Node.js v14+
|
||||
- **Web Framework**: Express.js v4
|
||||
- **External SDK**: inCompass NetStar v3.1.0-2 (C++ library for categorization)
|
||||
- **Containerization**: Docker + Kubernetes
|
||||
- **CI/CD**: CircleCI
|
||||
- **Infrastructure**: Cloud-agnostic (DigitalOcean, Google Cloud, etc.)
|
||||
|
||||
### Core Services
|
||||
1. **HTTP Server** (Port 3333) - REST API for categorization requests
|
||||
2. **UDP Server** (Port 33333) - Legacy UDP interface for categorization
|
||||
3. **Cron Service** - Daily database updates via Kubernetes CronJob
|
||||
4. **Category Mapping** - NetStar ID → Organization Code conversion (singleton pattern)
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── server.js # Entry point: initializes UDP + HTTP servers
|
||||
├── app.js # Core logic: orchestrates categorization flow
|
||||
├── client.js # UDP test client
|
||||
├── cron.js # Scheduled database update logic
|
||||
├── use-cases/
|
||||
│ ├── get-category-use-case.js # Executes NetStar gcf1check command
|
||||
│ ├── category-converter-use-case.js # Maps NetStar IDs → org codes (singleton)
|
||||
│ ├── parse-detailed-category-use-case.js # Parses detailed output with ratings
|
||||
│ └── update-categories-use-case.js # Updates NetStar databases
|
||||
└── etc/
|
||||
└── categories-mapping.json # Mapping table: NetStar ID → Zvelo codes
|
||||
|
||||
deployment/
|
||||
├── deployment.yaml # Kubernetes deployment manifest
|
||||
└── staging/deployment.yaml # Staging-specific config
|
||||
|
||||
.circleci/config.yml # CI/CD pipeline (builds, tests, deploys)
|
||||
Dockerfile # Container build specification
|
||||
package.json # Node dependencies + scripts
|
||||
makefile # Convenience commands
|
||||
test-detailed.http # HTTP endpoint test file
|
||||
```
|
||||
|
||||
## Language Standards
|
||||
|
||||
### Comments and Error Messages - ENGLISH ONLY
|
||||
**All code comments, error messages, log statements, and documentation must be written in English.**
|
||||
|
||||
This includes:
|
||||
- ✅ Code comments explaining logic
|
||||
- ✅ Error messages and exception messages
|
||||
- ✅ Console.log, console.error, and logging statements
|
||||
- ✅ Variable and function names
|
||||
- ✅ Commit messages
|
||||
- ✅ Code review feedback
|
||||
- ✅ Documentation strings (JSDoc, etc.)
|
||||
|
||||
**Examples:**
|
||||
```javascript
|
||||
// Correct: English comment
|
||||
function mapCategoryId(netstarId) {
|
||||
if (!netstarId) {
|
||||
throw new Error('NetStar ID is required')
|
||||
}
|
||||
// Map to organization category code
|
||||
return categoryConverter.convert(netstarId)
|
||||
}
|
||||
|
||||
// Incorrect: Portuguese comment
|
||||
function mapCategoryId(netstarId) {
|
||||
if (!netstarId) {
|
||||
throw new Error('ID do NetStar é obrigatório') // ❌ ERROR IN PORTUGUESE
|
||||
}
|
||||
// Mapear para código de categoria da organização // ❌ COMMENT IN PORTUGUESE
|
||||
return categoryConverter.convert(netstarId)
|
||||
}
|
||||
```
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
### Code Style
|
||||
- Use **ES6 syntax** (const/let, arrow functions, template literals)
|
||||
- **No semicolons** in new code (already established pattern)
|
||||
- Functional/modular design - keep files focused on single responsibility
|
||||
- Use **singleton pattern** for shared state (see: `CategoryConverterUseCase`)
|
||||
- **All comments in English** - see Language Standards section
|
||||
|
||||
### Use Case Pattern
|
||||
- Each business operation gets a dedicated use case class in `src/use-cases/`
|
||||
- Use case classes should have a clear, single responsibility
|
||||
- Example:
|
||||
```javascript
|
||||
class GetCategoryUseCase {
|
||||
async execute(fqdn) {
|
||||
// implementation
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
- Defined in `.env` (create from `.env.example`)
|
||||
- `UDP_PORT=33333` - UDP server listen port
|
||||
- `HTTP_PORT=3333` - HTTP server listen port
|
||||
|
||||
## HTTP API Endpoints
|
||||
|
||||
### `POST /` - Basic Categorization
|
||||
- **Input**: `{"fqdn": "example.com"}`
|
||||
- **Output**: `{"result": [10009, 10010]}` (array of category IDs)
|
||||
- **Use Case**: Quick lookups when detailed info not needed
|
||||
|
||||
### `POST /detailed` - Detailed Categorization
|
||||
- **Input**: `{"fqdn": "example.com"}`
|
||||
- **Output**: Full category info with reputation score, age rating, primary/secondary categories, and human-readable names
|
||||
- **Use Case**: Comprehensive categorization for security decisions
|
||||
- **Includes**: `result` array matching the `/` endpoint format
|
||||
|
||||
### `POST /full` - Complete Raw Categorization Output
|
||||
- **Input**: `{"fqdn": "example.com"}`
|
||||
- **Output**: Complete detailed structure with all 35 fields from NetStar including:
|
||||
- Primary, secondary, and security categories (with IDs, names, and mapped IDs)
|
||||
- Reputation score and name
|
||||
- Matching flags and their descriptions
|
||||
- Age rating score and name
|
||||
- All 9 category group classifications (Internet/Infrastructure, Malware/Security, Dangerous/Harmful, Adult, Business/Government, Personal, Computing/Technology, Social Media, Miscellaneous)
|
||||
- Volume index
|
||||
- Submitted URL
|
||||
- **Use Case**: Complete diagnostic and analysis when all categorization data is needed
|
||||
- **Includes**: `result` array matching the `/` endpoint format
|
||||
|
||||
### UDP Server (Port 33333)
|
||||
- **Input**: Raw domain string (e.g., `"example.com"`)
|
||||
- **Output**: JSON-formatted result (same as HTTP `/` endpoint)
|
||||
- **Legacy Interface**: Maintained for backward compatibility
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Setup & Running Locally
|
||||
```bash
|
||||
# Install dependencies
|
||||
npm install
|
||||
|
||||
# Development with auto-reload
|
||||
npm run dev:server # Watch mode for server changes
|
||||
npm run dev:client # Run UDP client for testing
|
||||
|
||||
# Production
|
||||
npm start # Start both servers
|
||||
|
||||
# NetStar service commands (Linux system)
|
||||
make gcf1-start # Start NetStar service
|
||||
make gcf1-download # Download category databases
|
||||
make gcf1-update # Update category databases
|
||||
```
|
||||
|
||||
### Testing APIs
|
||||
Use `test-detailed.http` in VS Code REST Client extension:
|
||||
1. Open the file
|
||||
2. Click "Send Request" on each endpoint
|
||||
3. View responses in the side panel
|
||||
|
||||
### Git Workflow
|
||||
- **Main Branch**: `main` - production stable code
|
||||
- **Development Branch**: `development` - feature integration
|
||||
- **Feature Branches**: Create from `development`, merge back via PR
|
||||
- Recent commits show HTTP approach implementation and cron job additions
|
||||
- **Commit Messages**: Must be in English
|
||||
|
||||
## Deployment
|
||||
|
||||
### Docker
|
||||
- **Base Image**: Ubuntu 22.04
|
||||
- **Includes**: Boost libraries, Node.js, NetStar SDK
|
||||
- **Exposes**: Port 3000 (UDP)
|
||||
- Build: `docker build -t netstar-categorizer .`
|
||||
|
||||
### Kubernetes (Production)
|
||||
- **Namespace**: `blackdice`
|
||||
- **Deployment**: Single replica in appropriate cluster
|
||||
- **CronJob**: Daily database updates at 00:00 UTC
|
||||
- **Ingress**: `netstar-cat-dev.blackdice.ai` (DNS varies by environment)
|
||||
- **Branches to Environments**:
|
||||
- `development` → development cluster
|
||||
- `qa` → QA cluster
|
||||
- `staging` → staging cluster
|
||||
- `production` → production cluster
|
||||
- `gke-staging`, `gke-pov` → specific GKE clusters
|
||||
|
||||
### CI/CD Pipeline (CircleCI)
|
||||
- Automatically builds Docker image on push
|
||||
- Tags image with commit SHA
|
||||
- Deploys to appropriate Kubernetes cluster based on branch
|
||||
- Release deployments via git tags
|
||||
|
||||
## Key Technical Details
|
||||
|
||||
### Category Mapping System
|
||||
- **Source**: NetStar SDK returns numeric category IDs (e.g., 101, 102)
|
||||
- **Mapping File**: `src/etc/categories-mapping.json`
|
||||
- **Target**: Maps to organization's Zvelo pattern codes (e.g., 10075, 10078)
|
||||
- **Singleton Implementation**: `CategoryConverterUseCase` maintains single instance across app
|
||||
- **Example Mapping**:
|
||||
- NetStar 101 (Illegal Activities) → Zvelo 10075
|
||||
- NetStar 201 (Terrorism/Extremists) → Zvelo 10018
|
||||
|
||||
### NetStar SDK Integration
|
||||
- **Command**: `gcf1check` - queries the NetStar database for domain categorization
|
||||
- **Child Process**: Executed via Node.js `child_process` module
|
||||
- **Output Parsing**: Raw output parsed into JSON structure
|
||||
- **Detailed Mode**: Includes reputation scores and age ratings in output
|
||||
|
||||
### Automatic Database Updates
|
||||
- **Mechanism**: Kubernetes CronJob at 0 0 * * * (daily at midnight UTC)
|
||||
- **Fallback**: Manual update via `make gcf1-update`
|
||||
- **Purpose**: Keeps categorization database current with latest NetStar classifications
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Adding a New Endpoint
|
||||
1. Create a corresponding use case in `src/use-cases/`
|
||||
2. Add route in `src/app.js` that calls the use case
|
||||
3. Export and test in `test-detailed.http`
|
||||
4. Update this guide if it's a significant feature
|
||||
5. Ensure all error messages and comments are in English
|
||||
|
||||
### Updating Category Mappings
|
||||
1. Modify `src/etc/categories-mapping.json` with new ID mappings
|
||||
2. Restart the service (singleton will reload on next request)
|
||||
3. Test with both HTTP and UDP interfaces
|
||||
|
||||
### Debugging
|
||||
- **Server Logs**: Check Docker/Kubernetes logs for errors
|
||||
- **Cron Logs**: View Kubernetes CronJob logs for database update issues
|
||||
- **UDP Testing**: Use `npm run dev:client` to test directly
|
||||
- **HTTP Testing**: Use `test-detailed.http` with VS Code REST Client
|
||||
- **Error Messages**: All error logs must be in English
|
||||
|
||||
### Troubleshooting
|
||||
- **NetStar Service Not Running**: Run `make gcf1-start`
|
||||
- **Stale Categories**: Manually run `make gcf1-update` or wait for cron job
|
||||
- **Port Conflicts**: Ensure ports 3333 (HTTP) and 33333 (UDP) are available
|
||||
- **Docker Build Issues**: Check that Boost C++ libraries are installed correctly
|
||||
|
||||
## Current Development Status
|
||||
|
||||
### Recent Work
|
||||
- ✅ HTTP server implementation (alongside UDP)
|
||||
- ✅ Detailed categorization with reputation/age ratings
|
||||
- ✅ Cron job for automated daily updates
|
||||
- ✅ Singleton category converter pattern
|
||||
- 🔄 Work in Progress:
|
||||
- `playground.js` - experimental/testing code
|
||||
- `parse-detailed-category-use-case.js` - new detailed parsing feature
|
||||
- Enhanced `server.js` - expanded server capabilities
|
||||
|
||||
### Known Modified Files
|
||||
- `playground.js` - development/testing (can be cleaned up)
|
||||
- `src/server.js` - recent enhancements
|
||||
- `makefile` - new convenience commands
|
||||
- `test-detailed.http` - expanded test coverage
|
||||
|
||||
## Guidelines for Contributions
|
||||
|
||||
1. **Follow Existing Patterns**: Use use-case classes, follow module structure
|
||||
2. **Test Before Committing**: Use `test-detailed.http` for API changes
|
||||
3. **Update Mappings Properly**: Edit `categories-mapping.json`, not hardcode values
|
||||
4. **Document Breaking Changes**: Update this guide if architecture changes
|
||||
5. **Keep CircleCI Happy**: Ensure Docker build succeeds and K8s deployment configs are valid
|
||||
6. **Don't Skip Steps**: Always test UDP and HTTP interfaces for categorization changes
|
||||
7. **Language Standards**: All comments, error messages, and logs must be in English
|
||||
|
||||
## Resources & External Documentation
|
||||
|
||||
- **NetStar SDK**: Installed in Docker, documentation in inCompass SDK v3.1.0-2
|
||||
- **Express.js**: https://expressjs.com
|
||||
- **Node.js Child Process**: https://nodejs.org/api/child_process.html
|
||||
- **Kubernetes**: https://kubernetes.io/docs
|
||||
- **CircleCI**: Configuration at `.circleci/config.yml`
|
||||
8
makefile
Normal file
8
makefile
Normal file
@@ -0,0 +1,8 @@
|
||||
gcf1-start:
|
||||
cd /usr/local/gcf1 && sbin/gcf1 start
|
||||
|
||||
gcf1-download:
|
||||
cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_download
|
||||
|
||||
gcf1-update:
|
||||
cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_update
|
||||
101
package-lock.json
generated
101
package-lock.json
generated
@@ -10,7 +10,7 @@
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"dotenv": "^16.4.5",
|
||||
"express": "^4.21.1"
|
||||
"express": "^4.19.2"
|
||||
}
|
||||
},
|
||||
"node_modules/accepts": {
|
||||
@@ -31,9 +31,9 @@
|
||||
"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg=="
|
||||
},
|
||||
"node_modules/body-parser": {
|
||||
"version": "1.20.3",
|
||||
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.3.tgz",
|
||||
"integrity": "sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g==",
|
||||
"version": "1.20.2",
|
||||
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz",
|
||||
"integrity": "sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
|
||||
"dependencies": {
|
||||
"bytes": "3.1.2",
|
||||
"content-type": "~1.0.5",
|
||||
@@ -43,7 +43,7 @@
|
||||
"http-errors": "2.0.0",
|
||||
"iconv-lite": "0.4.24",
|
||||
"on-finished": "2.4.1",
|
||||
"qs": "6.13.0",
|
||||
"qs": "6.11.0",
|
||||
"raw-body": "2.5.2",
|
||||
"type-is": "~1.6.18",
|
||||
"unpipe": "1.0.0"
|
||||
@@ -99,9 +99,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/cookie": {
|
||||
"version": "0.7.1",
|
||||
"resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.1.tgz",
|
||||
"integrity": "sha512-6DnInpx7SJ2AK3+CTUE/ZM0vWTUboZCegxhC2xiIydHR9jNuTAASBrfEpHhiGOZw/nX51bHt6YQl8jsGo4y/0w==",
|
||||
"version": "0.6.0",
|
||||
"resolved": "https://registry.npmjs.org/cookie/-/cookie-0.6.0.tgz",
|
||||
"integrity": "sha512-U71cyTamuh1CRNCfpGY6to28lxvNwPG4Guz/EVjgf3Jmzv0vlDp1atT9eS5dDjMYHucpHbWns6Lwf3BKz6svdw==",
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
@@ -169,9 +169,9 @@
|
||||
"integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
|
||||
},
|
||||
"node_modules/encodeurl": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
|
||||
"integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
|
||||
"integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
@@ -209,36 +209,36 @@
|
||||
}
|
||||
},
|
||||
"node_modules/express": {
|
||||
"version": "4.21.1",
|
||||
"resolved": "https://registry.npmjs.org/express/-/express-4.21.1.tgz",
|
||||
"integrity": "sha512-YSFlK1Ee0/GC8QaO91tHcDxJiE/X4FbpAyQWkxAvG6AXCuR65YzK8ua6D9hvi/TzUfZMpc+BwuM1IPw8fmQBiQ==",
|
||||
"version": "4.19.2",
|
||||
"resolved": "https://registry.npmjs.org/express/-/express-4.19.2.tgz",
|
||||
"integrity": "sha512-5T6nhjsT+EOMzuck8JjBHARTHfMht0POzlA60WV2pMD3gyXw2LZnZ+ueGdNxG+0calOJcWKbpFcuzLZ91YWq9Q==",
|
||||
"dependencies": {
|
||||
"accepts": "~1.3.8",
|
||||
"array-flatten": "1.1.1",
|
||||
"body-parser": "1.20.3",
|
||||
"body-parser": "1.20.2",
|
||||
"content-disposition": "0.5.4",
|
||||
"content-type": "~1.0.4",
|
||||
"cookie": "0.7.1",
|
||||
"cookie": "0.6.0",
|
||||
"cookie-signature": "1.0.6",
|
||||
"debug": "2.6.9",
|
||||
"depd": "2.0.0",
|
||||
"encodeurl": "~2.0.0",
|
||||
"encodeurl": "~1.0.2",
|
||||
"escape-html": "~1.0.3",
|
||||
"etag": "~1.8.1",
|
||||
"finalhandler": "1.3.1",
|
||||
"finalhandler": "1.2.0",
|
||||
"fresh": "0.5.2",
|
||||
"http-errors": "2.0.0",
|
||||
"merge-descriptors": "1.0.3",
|
||||
"merge-descriptors": "1.0.1",
|
||||
"methods": "~1.1.2",
|
||||
"on-finished": "2.4.1",
|
||||
"parseurl": "~1.3.3",
|
||||
"path-to-regexp": "0.1.10",
|
||||
"path-to-regexp": "0.1.7",
|
||||
"proxy-addr": "~2.0.7",
|
||||
"qs": "6.13.0",
|
||||
"qs": "6.11.0",
|
||||
"range-parser": "~1.2.1",
|
||||
"safe-buffer": "5.2.1",
|
||||
"send": "0.19.0",
|
||||
"serve-static": "1.16.2",
|
||||
"send": "0.18.0",
|
||||
"serve-static": "1.15.0",
|
||||
"setprototypeof": "1.2.0",
|
||||
"statuses": "2.0.1",
|
||||
"type-is": "~1.6.18",
|
||||
@@ -250,12 +250,12 @@
|
||||
}
|
||||
},
|
||||
"node_modules/finalhandler": {
|
||||
"version": "1.3.1",
|
||||
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.1.tgz",
|
||||
"integrity": "sha512-6BN9trH7bp3qvnrRyzsBz+g3lZxTNZTbVO2EV1CS0WIcDbawYVdYvGflME/9QP0h0pYlCDBCTjYa9nZzMDpyxQ==",
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.2.0.tgz",
|
||||
"integrity": "sha512-5uXcUVftlQMFnWC9qu/svkWv3GTd2PfUhK/3PLkYNAe7FbqJMt3515HaxE6eRL74GdsriiwujiawdaB1BpEISg==",
|
||||
"dependencies": {
|
||||
"debug": "2.6.9",
|
||||
"encodeurl": "~2.0.0",
|
||||
"encodeurl": "~1.0.2",
|
||||
"escape-html": "~1.0.3",
|
||||
"on-finished": "2.4.1",
|
||||
"parseurl": "~1.3.3",
|
||||
@@ -411,12 +411,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/merge-descriptors": {
|
||||
"version": "1.0.3",
|
||||
"resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz",
|
||||
"integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==",
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
|
||||
"integrity": "sha512-cCi6g3/Zr1iqQi6ySbseM1Xvooa98N0w31jzUYrXPX2xqObmFGHJ0tQ5u74H3mVh7wLouTseZyYIq39g8cNp1w=="
|
||||
},
|
||||
"node_modules/methods": {
|
||||
"version": "1.1.2",
|
||||
@@ -500,9 +497,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/path-to-regexp": {
|
||||
"version": "0.1.10",
|
||||
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.10.tgz",
|
||||
"integrity": "sha512-7lf7qcQidTku0Gu3YDPc8DJ1q7OOucfa/BSsIwjuh56VU7katFvuM8hULfkwB3Fns/rsVF7PwPKVw1sl5KQS9w=="
|
||||
"version": "0.1.7",
|
||||
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
|
||||
"integrity": "sha512-5DFkuoqlv1uYQKxy8omFBeJPQcdoE07Kv2sferDCrAq1ohOU+MSDswDIbnx3YAM60qIOnYa53wBhXW0EbMonrQ=="
|
||||
},
|
||||
"node_modules/proxy-addr": {
|
||||
"version": "2.0.7",
|
||||
@@ -517,11 +514,11 @@
|
||||
}
|
||||
},
|
||||
"node_modules/qs": {
|
||||
"version": "6.13.0",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.13.0.tgz",
|
||||
"integrity": "sha512-+38qI9SOr8tfZ4QmJNplMUxqjbe7LKvvZgWdExBOmd+egZTtjLB67Gu0HRX3u/XOq7UU2Nx6nsjvS16Z9uwfpg==",
|
||||
"version": "6.11.0",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.11.0.tgz",
|
||||
"integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
|
||||
"dependencies": {
|
||||
"side-channel": "^1.0.6"
|
||||
"side-channel": "^1.0.4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.6"
|
||||
@@ -577,9 +574,9 @@
|
||||
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
|
||||
},
|
||||
"node_modules/send": {
|
||||
"version": "0.19.0",
|
||||
"resolved": "https://registry.npmjs.org/send/-/send-0.19.0.tgz",
|
||||
"integrity": "sha512-dW41u5VfLXu8SJh5bwRmyYUbAoSB3c9uQh6L8h/KtsFREPWpbX1lrljJo186Jc4nmci/sGUZ9a0a0J2zgfq2hw==",
|
||||
"version": "0.18.0",
|
||||
"resolved": "https://registry.npmjs.org/send/-/send-0.18.0.tgz",
|
||||
"integrity": "sha512-qqWzuOjSFOuqPjFe4NOsMLafToQQwBSOEpS+FwEt3A2V3vKubTquT3vmLTQpFgMXp8AlFWFuP1qKaJZOtPpVXg==",
|
||||
"dependencies": {
|
||||
"debug": "2.6.9",
|
||||
"depd": "2.0.0",
|
||||
@@ -599,28 +596,20 @@
|
||||
"node": ">= 0.8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/send/node_modules/encodeurl": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
|
||||
"integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/send/node_modules/ms": {
|
||||
"version": "2.1.3",
|
||||
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
|
||||
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
|
||||
},
|
||||
"node_modules/serve-static": {
|
||||
"version": "1.16.2",
|
||||
"resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.2.tgz",
|
||||
"integrity": "sha512-VqpjJZKadQB/PEbEwvFdO43Ax5dFBZ2UECszz8bQ7pi7wt//PWe1P6MN7eCnjsatYtBT6EuiClbjSWP2WrIoTw==",
|
||||
"version": "1.15.0",
|
||||
"resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.15.0.tgz",
|
||||
"integrity": "sha512-XGuRDNjXUijsUL0vl6nSD7cwURuzEgglbOaFuZM9g3kwDXOWVTck0jLzjPzGD+TazWbboZYu52/9/XPdUgne9g==",
|
||||
"dependencies": {
|
||||
"encodeurl": "~2.0.0",
|
||||
"encodeurl": "~1.0.2",
|
||||
"escape-html": "~1.0.3",
|
||||
"parseurl": "~1.3.3",
|
||||
"send": "0.19.0"
|
||||
"send": "0.18.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.8.0"
|
||||
|
||||
@@ -13,6 +13,6 @@
|
||||
"description": "",
|
||||
"dependencies": {
|
||||
"dotenv": "^16.4.5",
|
||||
"express": "^4.21.1"
|
||||
"express": "^4.19.2"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,12 +7,12 @@ const { spawn, exec } = require("node:child_process")
|
||||
// })
|
||||
|
||||
|
||||
exec("echo '99999.incompass.netstar-inc.com' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
|
||||
exec("echo 'li12.pages.dev' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
|
||||
if (error) {
|
||||
console.error(error)
|
||||
return
|
||||
}
|
||||
|
||||
console.log(stdout.trim())
|
||||
console.log(stdout.trim().split(/\t|\s{2,}/))
|
||||
|
||||
})
|
||||
@@ -7,10 +7,12 @@ const getCategory = new GetCategoryUseCase();
|
||||
const categoryConverter = new CategoryConverterUseCase({categoriesMapping});
|
||||
|
||||
module.exports = async function app(domain){
|
||||
console.log(domain)
|
||||
const category = await getCategory.execute(domain)
|
||||
console.log(category)
|
||||
const categoryConverted = await categoryConverter.execute(category)
|
||||
console.log(categoryConverted)
|
||||
return categoryConverted
|
||||
return [categoryConverted]
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -3,8 +3,10 @@ const { UpdateCategoriesUseCase } = require('./use-cases/update-categories-use-c
|
||||
const updateCategories = new UpdateCategoriesUseCase()
|
||||
|
||||
module.exports = async function cron() {
|
||||
updateCategories.execute()
|
||||
|
||||
const oneDay = 1000 * 60 * 60 * 24
|
||||
setImmediate(() => {
|
||||
setInterval(() => {
|
||||
updateCategories.execute()
|
||||
}, oneDay)
|
||||
}
|
||||
|
||||
@@ -1074,7 +1074,7 @@
|
||||
"id": "20097",
|
||||
"description": "VPN",
|
||||
"related": [
|
||||
"10451"
|
||||
"10460"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -1109,7 +1109,7 @@
|
||||
"id": "20103",
|
||||
"description": "Code Repositories",
|
||||
"related": [
|
||||
"10429"
|
||||
"10047"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -5,6 +5,10 @@ const app = require('./app')
|
||||
const bodyParser = require('body-parser')
|
||||
const cron = require('./cron')
|
||||
const express = require('express')
|
||||
const { ParseDetailedCategoryUseCase } = require('./use-cases/parse-detailed-category-use-case')
|
||||
const { ParseFullCategoryUseCase } = require('./use-cases/parse-full-category-use-case')
|
||||
const { CategoryConverterUseCase } = require('./use-cases/category-converter-use-case')
|
||||
const categoriesMapping = require('./etc/categories-mapping.json')
|
||||
|
||||
const httpServer = express()
|
||||
httpServer.use(bodyParser.json()) // for parsing application/json
|
||||
@@ -14,6 +18,11 @@ cron()
|
||||
const UDP_PORT = process.env.UDP_PORT
|
||||
const HTTP_PORT = process.env.HTTP_PORT
|
||||
|
||||
// Initialize use cases
|
||||
const categoryConverter = new CategoryConverterUseCase({ categoriesMapping })
|
||||
const parseDetailedCategory = new ParseDetailedCategoryUseCase({ categoryConverter })
|
||||
const parseFullCategory = new ParseFullCategoryUseCase({ categoryConverter })
|
||||
|
||||
const server = dgram.createSocket('udp4');
|
||||
|
||||
|
||||
@@ -50,6 +59,7 @@ server.on('listening', () => {
|
||||
server.bind(UDP_PORT);
|
||||
|
||||
httpServer.post('/', async (req, res) => {
|
||||
console.log("new request")
|
||||
try {
|
||||
const { fqdn } = req.body
|
||||
|
||||
@@ -69,8 +79,35 @@ httpServer.post('/', async (req, res) => {
|
||||
|
||||
})
|
||||
|
||||
httpServer.post('/detailed', async (req, res) => {
|
||||
try {
|
||||
const { fqdn } = req.body
|
||||
|
||||
const detailedResult = await parseDetailedCategory.execute(fqdn)
|
||||
|
||||
res.status(200).json(detailedResult)
|
||||
} catch (err) {
|
||||
console.error('Error in /detailed endpoint:', err)
|
||||
res.status(500).json({ error: err.message || err })
|
||||
}
|
||||
})
|
||||
|
||||
httpServer.post('/full', async (req, res) => {
|
||||
try {
|
||||
const { fqdn } = req.body
|
||||
|
||||
const fullResult = await parseFullCategory.execute(fqdn)
|
||||
|
||||
res.status(200).json(fullResult)
|
||||
} catch (err) {
|
||||
console.error('Error in /full endpoint:', err)
|
||||
res.status(500).json({ error: err.message || err })
|
||||
}
|
||||
})
|
||||
|
||||
httpServer.listen(HTTP_PORT, () => {
|
||||
console.log('HTTP server listening 3333')
|
||||
console.log('UDP server listening 3334')
|
||||
})
|
||||
|
||||
// Response
|
||||
|
||||
10
src/test-app.js
Normal file
10
src/test-app.js
Normal file
@@ -0,0 +1,10 @@
|
||||
const app = require('./app');
|
||||
|
||||
(async () => {
|
||||
try {
|
||||
const categories = await app('www.facebook.com');
|
||||
console.log(categories);
|
||||
} catch (error) {
|
||||
console.error('Error occurred:', error);
|
||||
}
|
||||
})();
|
||||
@@ -11,7 +11,7 @@ class CategoryConverterUseCase {
|
||||
execute(category) {
|
||||
const entry = this.categoriesMapping.find(item => item.id === category);
|
||||
|
||||
return entry ? entry.related[0].split(', ').map(str => str.trim()) : null;
|
||||
return entry ? entry.related[0] : null;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
101
src/use-cases/parse-detailed-category-use-case.js
Normal file
101
src/use-cases/parse-detailed-category-use-case.js
Normal file
@@ -0,0 +1,101 @@
|
||||
const { exec } = require("node:child_process")
|
||||
|
||||
class ParseDetailedCategoryUseCase {
|
||||
constructor({ categoryConverter }) {
|
||||
this.categoryConverter = categoryConverter;
|
||||
}
|
||||
|
||||
execute(domain) {
|
||||
return new Promise((resolve, reject) => {
|
||||
exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
|
||||
{ cwd: '/usr/local/gcf1' },
|
||||
(error, stdout, stderr) => {
|
||||
if (error) {
|
||||
console.error(error);
|
||||
reject(error);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const parsed = this.parseOutput(stdout);
|
||||
resolve(parsed);
|
||||
} catch (parseError) {
|
||||
console.error('Parse error:', parseError);
|
||||
reject(parseError);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
parseOutput(output) {
|
||||
// Extract all quoted strings in order
|
||||
const quotedStrings = [];
|
||||
const quoteRegex = /"([^"]*)"/g;
|
||||
let match;
|
||||
while ((match = quoteRegex.exec(output)) !== null) {
|
||||
quotedStrings.push(match[1]);
|
||||
}
|
||||
|
||||
// Split on tabs (primary separator in NetStar output)
|
||||
const parts = output.trim().split('\t');
|
||||
let quotedIndex = 0;
|
||||
|
||||
// Position mapping based on NetStar output structure:
|
||||
// parts[0]: Categorized
|
||||
// parts[1]: Matching Index
|
||||
// parts[2]: Primary Category ID
|
||||
// parts[3]: Primary Category Name (quoted)
|
||||
// parts[4]: Secondary Category ID
|
||||
// parts[5]: Secondary Category Name (quoted if secondary exists)
|
||||
// parts[6]: Security Category ID
|
||||
// parts[7]: Security Category Name
|
||||
// parts[8]: Reputation Score ID
|
||||
// parts[9]: Reputation Score Name (quoted)
|
||||
// parts[10]: Matching Flag Value
|
||||
// parts[11]: Matching Flag Names
|
||||
// parts[12]: Age Rating ID
|
||||
// parts[13]: Age Rating Name (quoted)
|
||||
|
||||
const primaryId = parseInt(parts[2]);
|
||||
const primaryName = quotedStrings[quotedIndex++];
|
||||
const primaryMapped = this.categoryConverter.execute(String(primaryId));
|
||||
|
||||
const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
|
||||
let secondaryName = null;
|
||||
let secondaryMapped = null;
|
||||
|
||||
if (secondaryId !== null) {
|
||||
secondaryName = quotedStrings[quotedIndex++];
|
||||
secondaryMapped = this.categoryConverter.execute(String(secondaryId));
|
||||
} else {
|
||||
secondaryName = parts[5] === '-' ? null : parts[5];
|
||||
}
|
||||
|
||||
const reputationId = parseInt(parts[8]);
|
||||
const reputationName = quotedStrings[quotedIndex++];
|
||||
|
||||
const ageRatingId = parseInt(parts[12]);
|
||||
const ageRatingName = quotedStrings[quotedIndex++];
|
||||
|
||||
// Build result array in the same format as the / endpoint
|
||||
const resultArray = [String(primaryMapped)];
|
||||
if (secondaryMapped !== null) {
|
||||
resultArray.push(String(secondaryMapped));
|
||||
}
|
||||
|
||||
return {
|
||||
result: resultArray,
|
||||
primary: primaryMapped,
|
||||
primary_name: primaryName,
|
||||
secondary: secondaryMapped,
|
||||
secondary_name: secondaryName,
|
||||
reputation: reputationId,
|
||||
reputation_name: reputationName,
|
||||
age_rating: ageRatingId,
|
||||
age_rating_name: ageRatingName,
|
||||
raw_output: output.trim()
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { ParseDetailedCategoryUseCase }
|
||||
172
src/use-cases/parse-full-category-use-case.js
Normal file
172
src/use-cases/parse-full-category-use-case.js
Normal file
@@ -0,0 +1,172 @@
|
||||
const { exec } = require("node:child_process")
|
||||
|
||||
class ParseFullCategoryUseCase {
|
||||
constructor({ categoryConverter }) {
|
||||
this.categoryConverter = categoryConverter;
|
||||
}
|
||||
|
||||
execute(domain) {
|
||||
return new Promise((resolve, reject) => {
|
||||
exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
|
||||
{ cwd: '/usr/local/gcf1' },
|
||||
(error, stdout, stderr) => {
|
||||
if (error) {
|
||||
console.error(error);
|
||||
reject(error);
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const parsed = this.parseOutput(stdout);
|
||||
resolve(parsed);
|
||||
} catch (parseError) {
|
||||
console.error('Parse error:', parseError);
|
||||
reject(parseError);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
parseOutput(output) {
|
||||
const quotedStrings = [];
|
||||
const quoteRegex = /"([^"]*)"/g;
|
||||
let match;
|
||||
while ((match = quoteRegex.exec(output)) !== null) {
|
||||
quotedStrings.push(match[1]);
|
||||
}
|
||||
|
||||
const parts = output.trim().split('\t');
|
||||
let quotedIndex = 0;
|
||||
|
||||
// Extract primary category (parts[2], parts[3])
|
||||
const primaryId = parseInt(parts[2]);
|
||||
const primaryName = quotedStrings[quotedIndex++];
|
||||
const primaryMapped = this.categoryConverter.execute(String(primaryId));
|
||||
|
||||
// Extract secondary category (parts[4], parts[5])
|
||||
const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
|
||||
let secondaryName = null;
|
||||
let secondaryMapped = null;
|
||||
if (secondaryId !== null && parts[5] !== '-') {
|
||||
secondaryName = quotedStrings[quotedIndex++];
|
||||
secondaryMapped = this.categoryConverter.execute(String(secondaryId));
|
||||
} else {
|
||||
secondaryName = parts[5] === '-' ? null : parts[5];
|
||||
}
|
||||
|
||||
// Extract security category (parts[6], parts[7])
|
||||
const securityId = parts[6] !== '0' && parts[6] !== '-' ? parseInt(parts[6]) : null;
|
||||
let securityName = null;
|
||||
let securityMapped = null;
|
||||
if (securityId !== null && parts[7] !== '-') {
|
||||
securityName = quotedStrings[quotedIndex++];
|
||||
securityMapped = this.categoryConverter.execute(String(securityId));
|
||||
} else {
|
||||
securityName = parts[7] === '-' ? null : parts[7];
|
||||
}
|
||||
|
||||
// Extract reputation (parts[8], parts[9])
|
||||
const reputationId = parseInt(parts[8]);
|
||||
const reputationName = quotedStrings[quotedIndex++];
|
||||
|
||||
// Extract matching flag (parts[10], parts[11])
|
||||
const matchingFlagValue = parts[10];
|
||||
const matchingFlagNames = parts[11] ? parts[11].split('|') : [];
|
||||
|
||||
// Extract age rating (parts[12], parts[13])
|
||||
const ageRatingId = parseInt(parts[12]);
|
||||
const ageRatingName = quotedStrings[quotedIndex++];
|
||||
|
||||
// Extract category groups starting from parts[16] (after 2 empty fields)
|
||||
// Each group has: id (number), name (quoted string)
|
||||
// Pattern: 0 "Internet/Infrastructure" 0 "Malware/Security" ...
|
||||
const categoryGroups = {
|
||||
internet_infrastructure: {
|
||||
id: parseInt(parts[16]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
malware_security: {
|
||||
id: parseInt(parts[18]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
dangerous_harmful: {
|
||||
id: parseInt(parts[20]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
adult: {
|
||||
id: parseInt(parts[22]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
business_government: {
|
||||
id: parseInt(parts[24]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
personal: {
|
||||
id: parseInt(parts[26]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
computing_technology: {
|
||||
id: parseInt(parts[28]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
social_media: {
|
||||
id: parseInt(parts[30]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
},
|
||||
miscellaneous: {
|
||||
id: parseInt(parts[32]) || 0,
|
||||
name: quotedStrings[quotedIndex++]
|
||||
}
|
||||
};
|
||||
|
||||
// Extract volume index and submitted URL
|
||||
const volumeIndex = parts[35];
|
||||
const submittedUrl = parts[36];
|
||||
|
||||
const result = {
|
||||
result_status: parts[0],
|
||||
matching_index: parseInt(parts[1]),
|
||||
primary_category: {
|
||||
id: primaryId,
|
||||
name: primaryName,
|
||||
mapped_id: String(primaryMapped)
|
||||
},
|
||||
secondary_category: {
|
||||
id: secondaryId,
|
||||
name: secondaryName,
|
||||
mapped_id: secondaryMapped ? String(secondaryMapped) : null
|
||||
},
|
||||
security_category: {
|
||||
id: securityId,
|
||||
name: securityName,
|
||||
mapped_id: securityMapped ? String(securityMapped) : null
|
||||
},
|
||||
reputation: {
|
||||
id: reputationId,
|
||||
name: reputationName
|
||||
},
|
||||
matching_flag: {
|
||||
value: matchingFlagValue,
|
||||
names: matchingFlagNames
|
||||
},
|
||||
age_rating: {
|
||||
id: ageRatingId,
|
||||
name: ageRatingName
|
||||
},
|
||||
category_groups: categoryGroups,
|
||||
volume_index: volumeIndex,
|
||||
submitted_url: submittedUrl
|
||||
};
|
||||
|
||||
// Add result array in the same format as / endpoint
|
||||
const resultArray = [result.primary_category.mapped_id];
|
||||
if (result.secondary_category.mapped_id !== null) {
|
||||
resultArray.push(result.secondary_category.mapped_id);
|
||||
}
|
||||
result.result = resultArray;
|
||||
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { ParseFullCategoryUseCase }
|
||||
@@ -1,20 +1,26 @@
|
||||
const { exec } = require("node:child_process")
|
||||
const { exec } = require("node:child_process");
|
||||
|
||||
class UpdateCategoriesUseCase {
|
||||
execute() {
|
||||
return new Promise((_, reject) => {
|
||||
exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
|
||||
exec(`bin/gcf1dbmng.sh etc urldb_download`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
|
||||
if (error) {
|
||||
console.error(error);
|
||||
reject(error);
|
||||
console.error('Erro no primeiro comando:', error);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(stdout)
|
||||
console.log('Saída do primeiro comando:', stdout);
|
||||
|
||||
// Executa o segundo comando após o primeiro ter sido concluído
|
||||
exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error2, stdout2, stderr2) => {
|
||||
if (error2) {
|
||||
console.error('Erro no segundo comando:', error2);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log('Saída do segundo comando:', stdout2);
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { UpdateCategoriesUseCase }
|
||||
|
||||
module.exports = { UpdateCategoriesUseCase };
|
||||
188
test-detailed.http
Normal file
188
test-detailed.http
Normal file
@@ -0,0 +1,188 @@
|
||||
@baseUrl = http://localhost:3333
|
||||
|
||||
# ============================================
|
||||
# POST / - Basic Categorization Endpoint
|
||||
# Returns: Simple result array of category IDs
|
||||
# ============================================
|
||||
|
||||
### Basic endpoint - Facebook
|
||||
POST {{baseUrl}}/
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "facebook.com"
|
||||
}
|
||||
|
||||
### Basic endpoint - Google
|
||||
POST {{baseUrl}}/
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "google.com"
|
||||
}
|
||||
|
||||
### Basic endpoint - TikTok
|
||||
POST {{baseUrl}}/
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "tiktok.com"
|
||||
}
|
||||
|
||||
### Basic endpoint - YouTube
|
||||
POST {{baseUrl}}/
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "youtube.com"
|
||||
}
|
||||
|
||||
### Basic endpoint - Reddit
|
||||
POST {{baseUrl}}/
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "reddit.com"
|
||||
}
|
||||
|
||||
### Basic endpoint - Twitter
|
||||
POST {{baseUrl}}/
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "twitter.com"
|
||||
}
|
||||
|
||||
# ============================================
|
||||
# POST /detailed - Detailed Categorization
|
||||
# Returns: Categories with names, reputation, age rating
|
||||
# ============================================
|
||||
|
||||
### Detailed endpoint - Facebook
|
||||
POST {{baseUrl}}/detailed
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "facebook.com"
|
||||
}
|
||||
|
||||
### Detailed endpoint - Google
|
||||
POST {{baseUrl}}/detailed
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "google.com"
|
||||
}
|
||||
|
||||
### Detailed endpoint - TikTok
|
||||
POST {{baseUrl}}/detailed
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "tiktok.com"
|
||||
}
|
||||
|
||||
### Detailed endpoint - YouTube
|
||||
POST {{baseUrl}}/detailed
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "youtube.com"
|
||||
}
|
||||
|
||||
### Detailed endpoint - Reddit
|
||||
POST {{baseUrl}}/detailed
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "reddit.com"
|
||||
}
|
||||
|
||||
### Detailed endpoint - Twitter
|
||||
POST {{baseUrl}}/detailed
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "twitter.com"
|
||||
}
|
||||
|
||||
# ============================================
|
||||
# POST /full - Complete Raw Output
|
||||
# Returns: All 35 fields from NetStar with category groups
|
||||
# ============================================
|
||||
|
||||
### Full endpoint - Facebook
|
||||
POST {{baseUrl}}/full
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "facebook.com"
|
||||
}
|
||||
|
||||
### Full endpoint - Google
|
||||
POST {{baseUrl}}/full
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "google.com"
|
||||
}
|
||||
|
||||
### Full endpoint - TikTok
|
||||
POST {{baseUrl}}/full
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "tiktok.com"
|
||||
}
|
||||
|
||||
### Full endpoint - YouTube
|
||||
POST {{baseUrl}}/full
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "youtube.com"
|
||||
}
|
||||
|
||||
### Full endpoint - Reddit
|
||||
POST {{baseUrl}}/full
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "reddit.com"
|
||||
}
|
||||
|
||||
### Full endpoint - Twitter
|
||||
POST {{baseUrl}}/full
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "twitter.com"
|
||||
}
|
||||
|
||||
# ============================================
|
||||
# Edge cases and special domains
|
||||
# ============================================
|
||||
|
||||
### Basic endpoint - Unknown domain
|
||||
POST {{baseUrl}}/
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "unknowndomainexample12345.com"
|
||||
}
|
||||
|
||||
### Detailed endpoint - Localhost
|
||||
POST {{baseUrl}}/detailed
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "localhost"
|
||||
}
|
||||
|
||||
### Full endpoint - IP-like domain
|
||||
POST {{baseUrl}}/full
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"fqdn": "99999.incompass.netstar-inc.com"
|
||||
}
|
||||
Reference in New Issue
Block a user