Merge pull request #10 from blackdice-cyber/main

main to dev

README.md

@@ -50,20 +50,27 @@ HYBRID_NS_ACCESS=andkWEzbBWRR2K6iw4edRvfd6MmNNjRx
 HYBRID_USER_AGENT=CSUX_blackdice
 ```
 
-6. To start the service, run:
+6. To start the service:
 
 ```
-sbin/gcf1 start
+cd /usr/local/gcf1/
-Starting gcf1d services: ...succeeded
+
+./sbin/gcf1 start
+
+# Starting gcf1d services: ...succeeded
 ```
 
 7. To download/update databases, run:
 
 ```
-bin/gcf1dbmng.sh etc urldb_download
+cd /usr/local/gcf1/
-Success DB download .
+
-bin/gcf1dbmng.sh etc urldb_update
+./bin/gcf1dbmng.sh etc urldb_download
-Success DB update .
+
+# Success DB download .
+
+./bin/gcf1dbmng.sh etc urldb_update
+# Success DB update .
 ```
 
 ## UDP Server Setup
@@ -85,14 +92,14 @@ npm install
 
 ```
 cp .env.example .env
-PORT=3000
+
+# --------- example contents ---------
+# UDP_PORT=33333
+# HTTP_PORT=3333
+# ------------------------------------
 ```
 
-## Running the Server
+## Digital Ocean Deployment details
-
+1. The Node server is installed to `/opt/netstar-categorizer`
-To start the UDP server, use the following command:
+2. Rather than using `npm start` (which doesn't restart after a crash) the we use **[PM2](https://pm2.keymetrics.io/)**, which provides a more robust runtime
-
+3. The full command to start the server using pm2 is `pm2 start npm --name "netstar-categorizer" --cwd /opt/netstar-categorizer/ -- start` - for convenience, there's a `run-me.sh` script in `/root` that will execute this command. 
-```
-npm start
-```
-This will run the server and bind it to the port specified in your .env file.

claude.md

@@ -0,0 +1,290 @@
+# Claude Development Guide - NetStar Categorizer
+
+## Project Overview
+
+**NetStar Categorizer** is a Node.js microservice that provides domain name/FQDN categorization using the inCompass NetStar SDK. It exposes both UDP and HTTP interfaces for real-time content classification with automatic daily database updates.
+
+### Core Purpose
+- Categorize domains into standardized categories using NetStar's content database
+- Map raw NetStar category IDs to organization-specific category codes
+- Provide both simple and detailed categorization results with reputation/age ratings
+- Maintain updated categorization databases through automated cron jobs
+
+## Key Architecture
+
+### Technology Stack
+- **Runtime**: Node.js v14+
+- **Web Framework**: Express.js v4
+- **External SDK**: inCompass NetStar v3.1.0-2 (C++ library for categorization)
+- **Containerization**: Docker + Kubernetes
+- **CI/CD**: CircleCI
+- **Infrastructure**: Cloud-agnostic (DigitalOcean, Google Cloud, etc.)
+
+### Core Services
+1. **HTTP Server** (Port 3333) - REST API for categorization requests
+2. **UDP Server** (Port 33333) - Legacy UDP interface for categorization
+3. **Cron Service** - Daily database updates via Kubernetes CronJob
+4. **Category Mapping** - NetStar ID → Organization Code conversion (singleton pattern)
+
+## Project Structure
+
+```
+src/
+├── server.js                              # Entry point: initializes UDP + HTTP servers
+├── app.js                                 # Core logic: orchestrates categorization flow
+├── client.js                              # UDP test client
+├── cron.js                                # Scheduled database update logic
+├── use-cases/
+│   ├── get-category-use-case.js          # Executes NetStar gcf1check command
+│   ├── category-converter-use-case.js    # Maps NetStar IDs → org codes (singleton)
+│   ├── parse-detailed-category-use-case.js # Parses detailed output with ratings
+│   └── update-categories-use-case.js     # Updates NetStar databases
+└── etc/
+    └── categories-mapping.json           # Mapping table: NetStar ID → Zvelo codes
+
+deployment/
+├── deployment.yaml                       # Kubernetes deployment manifest
+└── staging/deployment.yaml               # Staging-specific config
+
+.circleci/config.yml                      # CI/CD pipeline (builds, tests, deploys)
+Dockerfile                                # Container build specification
+package.json                              # Node dependencies + scripts
+makefile                                  # Convenience commands
+test-detailed.http                        # HTTP endpoint test file
+```
+
+## Language Standards
+
+### Comments and Error Messages - ENGLISH ONLY
+**All code comments, error messages, log statements, and documentation must be written in English.**
+
+This includes:
+- ✅ Code comments explaining logic
+- ✅ Error messages and exception messages
+- ✅ Console.log, console.error, and logging statements
+- ✅ Variable and function names
+- ✅ Commit messages
+- ✅ Code review feedback
+- ✅ Documentation strings (JSDoc, etc.)
+
+**Examples:**
+```javascript
+// Correct: English comment
+function mapCategoryId(netstarId) {
+  if (!netstarId) {
+    throw new Error('NetStar ID is required')
+  }
+  // Map to organization category code
+  return categoryConverter.convert(netstarId)
+}
+
+// Incorrect: Portuguese comment
+function mapCategoryId(netstarId) {
+  if (!netstarId) {
+    throw new Error('ID do NetStar é obrigatório')  // ❌ ERROR IN PORTUGUESE
+  }
+  // Mapear para código de categoria da organização  // ❌ COMMENT IN PORTUGUESE
+  return categoryConverter.convert(netstarId)
+}
+```
+
+## Coding Conventions
+
+### Code Style
+- Use **ES6 syntax** (const/let, arrow functions, template literals)
+- **No semicolons** in new code (already established pattern)
+- Functional/modular design - keep files focused on single responsibility
+- Use **singleton pattern** for shared state (see: `CategoryConverterUseCase`)
+- **All comments in English** - see Language Standards section
+
+### Use Case Pattern
+- Each business operation gets a dedicated use case class in `src/use-cases/`
+- Use case classes should have a clear, single responsibility
+- Example:
+  ```javascript
+  class GetCategoryUseCase {
+    async execute(fqdn) {
+      // implementation
+    }
+  }
+  ```
+
+### Environment Variables
+- Defined in `.env` (create from `.env.example`)
+- `UDP_PORT=33333` - UDP server listen port
+- `HTTP_PORT=3333` - HTTP server listen port
+
+## HTTP API Endpoints
+
+### `POST /` - Basic Categorization
+- **Input**: `{"fqdn": "example.com"}`
+- **Output**: `{"result": [10009, 10010]}` (array of category IDs)
+- **Use Case**: Quick lookups when detailed info not needed
+
+### `POST /detailed` - Detailed Categorization
+- **Input**: `{"fqdn": "example.com"}`
+- **Output**: Full category info with reputation score, age rating, primary/secondary categories, and human-readable names
+- **Use Case**: Comprehensive categorization for security decisions
+- **Includes**: `result` array matching the `/` endpoint format
+
+### `POST /full` - Complete Raw Categorization Output
+- **Input**: `{"fqdn": "example.com"}`
+- **Output**: Complete detailed structure with all 35 fields from NetStar including:
+  - Primary, secondary, and security categories (with IDs, names, and mapped IDs)
+  - Reputation score and name
+  - Matching flags and their descriptions
+  - Age rating score and name
+  - All 9 category group classifications (Internet/Infrastructure, Malware/Security, Dangerous/Harmful, Adult, Business/Government, Personal, Computing/Technology, Social Media, Miscellaneous)
+  - Volume index
+  - Submitted URL
+- **Use Case**: Complete diagnostic and analysis when all categorization data is needed
+- **Includes**: `result` array matching the `/` endpoint format
+
+### UDP Server (Port 33333)
+- **Input**: Raw domain string (e.g., `"example.com"`)
+- **Output**: JSON-formatted result (same as HTTP `/` endpoint)
+- **Legacy Interface**: Maintained for backward compatibility
+
+## Development Workflow
+
+### Setup & Running Locally
+```bash
+# Install dependencies
+npm install
+
+# Development with auto-reload
+npm run dev:server    # Watch mode for server changes
+npm run dev:client    # Run UDP client for testing
+
+# Production
+npm start             # Start both servers
+
+# NetStar service commands (Linux system)
+make gcf1-start       # Start NetStar service
+make gcf1-download    # Download category databases
+make gcf1-update      # Update category databases
+```
+
+### Testing APIs
+Use `test-detailed.http` in VS Code REST Client extension:
+1. Open the file
+2. Click "Send Request" on each endpoint
+3. View responses in the side panel
+
+### Git Workflow
+- **Main Branch**: `main` - production stable code
+- **Development Branch**: `development` - feature integration
+- **Feature Branches**: Create from `development`, merge back via PR
+- Recent commits show HTTP approach implementation and cron job additions
+- **Commit Messages**: Must be in English
+
+## Deployment
+
+### Docker
+- **Base Image**: Ubuntu 22.04
+- **Includes**: Boost libraries, Node.js, NetStar SDK
+- **Exposes**: Port 3000 (UDP)
+- Build: `docker build -t netstar-categorizer .`
+
+### Kubernetes (Production)
+- **Namespace**: `blackdice`
+- **Deployment**: Single replica in appropriate cluster
+- **CronJob**: Daily database updates at 00:00 UTC
+- **Ingress**: `netstar-cat-dev.blackdice.ai` (DNS varies by environment)
+- **Branches to Environments**:
+  - `development` → development cluster
+  - `qa` → QA cluster
+  - `staging` → staging cluster
+  - `production` → production cluster
+  - `gke-staging`, `gke-pov` → specific GKE clusters
+
+### CI/CD Pipeline (CircleCI)
+- Automatically builds Docker image on push
+- Tags image with commit SHA
+- Deploys to appropriate Kubernetes cluster based on branch
+- Release deployments via git tags
+
+## Key Technical Details
+
+### Category Mapping System
+- **Source**: NetStar SDK returns numeric category IDs (e.g., 101, 102)
+- **Mapping File**: `src/etc/categories-mapping.json`
+- **Target**: Maps to organization's Zvelo pattern codes (e.g., 10075, 10078)
+- **Singleton Implementation**: `CategoryConverterUseCase` maintains single instance across app
+- **Example Mapping**:
+  - NetStar 101 (Illegal Activities) → Zvelo 10075
+  - NetStar 201 (Terrorism/Extremists) → Zvelo 10018
+
+### NetStar SDK Integration
+- **Command**: `gcf1check` - queries the NetStar database for domain categorization
+- **Child Process**: Executed via Node.js `child_process` module
+- **Output Parsing**: Raw output parsed into JSON structure
+- **Detailed Mode**: Includes reputation scores and age ratings in output
+
+### Automatic Database Updates
+- **Mechanism**: Kubernetes CronJob at 0 0 * * * (daily at midnight UTC)
+- **Fallback**: Manual update via `make gcf1-update`
+- **Purpose**: Keeps categorization database current with latest NetStar classifications
+
+## Common Tasks
+
+### Adding a New Endpoint
+1. Create a corresponding use case in `src/use-cases/`
+2. Add route in `src/app.js` that calls the use case
+3. Export and test in `test-detailed.http`
+4. Update this guide if it's a significant feature
+5. Ensure all error messages and comments are in English
+
+### Updating Category Mappings
+1. Modify `src/etc/categories-mapping.json` with new ID mappings
+2. Restart the service (singleton will reload on next request)
+3. Test with both HTTP and UDP interfaces
+
+### Debugging
+- **Server Logs**: Check Docker/Kubernetes logs for errors
+- **Cron Logs**: View Kubernetes CronJob logs for database update issues
+- **UDP Testing**: Use `npm run dev:client` to test directly
+- **HTTP Testing**: Use `test-detailed.http` with VS Code REST Client
+- **Error Messages**: All error logs must be in English
+
+### Troubleshooting
+- **NetStar Service Not Running**: Run `make gcf1-start`
+- **Stale Categories**: Manually run `make gcf1-update` or wait for cron job
+- **Port Conflicts**: Ensure ports 3333 (HTTP) and 33333 (UDP) are available
+- **Docker Build Issues**: Check that Boost C++ libraries are installed correctly
+
+## Current Development Status
+
+### Recent Work
+- ✅ HTTP server implementation (alongside UDP)
+- ✅ Detailed categorization with reputation/age ratings
+- ✅ Cron job for automated daily updates
+- ✅ Singleton category converter pattern
+- 🔄 Work in Progress:
+  - `playground.js` - experimental/testing code
+  - `parse-detailed-category-use-case.js` - new detailed parsing feature
+  - Enhanced `server.js` - expanded server capabilities
+
+### Known Modified Files
+- `playground.js` - development/testing (can be cleaned up)
+- `src/server.js` - recent enhancements
+- `makefile` - new convenience commands
+- `test-detailed.http` - expanded test coverage
+
+## Guidelines for Contributions
+
+1. **Follow Existing Patterns**: Use use-case classes, follow module structure
+2. **Test Before Committing**: Use `test-detailed.http` for API changes
+3. **Update Mappings Properly**: Edit `categories-mapping.json`, not hardcode values
+4. **Document Breaking Changes**: Update this guide if architecture changes
+5. **Keep CircleCI Happy**: Ensure Docker build succeeds and K8s deployment configs are valid
+6. **Don't Skip Steps**: Always test UDP and HTTP interfaces for categorization changes
+7. **Language Standards**: All comments, error messages, and logs must be in English
+
+## Resources & External Documentation
+
+- **NetStar SDK**: Installed in Docker, documentation in inCompass SDK v3.1.0-2
+- **Express.js**: https://expressjs.com
+- **Node.js Child Process**: https://nodejs.org/api/child_process.html
+- **Kubernetes**: https://kubernetes.io/docs
+- **CircleCI**: Configuration at `.circleci/config.yml`

makefile

@@ -0,0 +1,8 @@
+gcf1-start:
+	cd /usr/local/gcf1 && sbin/gcf1 start
+	
+gcf1-download:
+	cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_download
+
+gcf1-update:
+	cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_update

package-lock.json

@@ -10,7 +10,7 @@
       "license": "ISC",
       "dependencies": {
         "dotenv": "^16.4.5",
-        "express": "^4.21.1"
+        "express": "^4.19.2"
       }
     },
     "node_modules/accepts": {
@@ -31,9 +31,9 @@
       "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg=="
     },
     "node_modules/body-parser": {
-      "version": "1.20.3",
+      "version": "1.20.2",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.3.tgz",
+      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.2.tgz",
-      "integrity": "sha512-7rAxByjUMqQ3/bHJy7D6OGXvx/MMc4IqBn/X0fcM1QUcAItpZrBEYhWGem+tzXH90c+G01ypMcYJBO9Y30203g==",
+      "integrity": "sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA==",
       "dependencies": {
         "bytes": "3.1.2",
         "content-type": "~1.0.5",
@@ -43,7 +43,7 @@
         "http-errors": "2.0.0",
         "iconv-lite": "0.4.24",
         "on-finished": "2.4.1",
-        "qs": "6.13.0",
+        "qs": "6.11.0",
         "raw-body": "2.5.2",
         "type-is": "~1.6.18",
         "unpipe": "1.0.0"
@@ -99,9 +99,9 @@
       }
     },
     "node_modules/cookie": {
-      "version": "0.7.1",
+      "version": "0.6.0",
-      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.1.tgz",
+      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.6.0.tgz",
-      "integrity": "sha512-6DnInpx7SJ2AK3+CTUE/ZM0vWTUboZCegxhC2xiIydHR9jNuTAASBrfEpHhiGOZw/nX51bHt6YQl8jsGo4y/0w==",
+      "integrity": "sha512-U71cyTamuh1CRNCfpGY6to28lxvNwPG4Guz/EVjgf3Jmzv0vlDp1atT9eS5dDjMYHucpHbWns6Lwf3BKz6svdw==",
       "engines": {
         "node": ">= 0.6"
       }
@@ -169,9 +169,9 @@
       "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
     },
     "node_modules/encodeurl": {
-      "version": "2.0.0",
+      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
+      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
-      "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
+      "integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
       "engines": {
         "node": ">= 0.8"
       }
@@ -209,36 +209,36 @@
       }
     },
     "node_modules/express": {
-      "version": "4.21.1",
+      "version": "4.19.2",
-      "resolved": "https://registry.npmjs.org/express/-/express-4.21.1.tgz",
+      "resolved": "https://registry.npmjs.org/express/-/express-4.19.2.tgz",
-      "integrity": "sha512-YSFlK1Ee0/GC8QaO91tHcDxJiE/X4FbpAyQWkxAvG6AXCuR65YzK8ua6D9hvi/TzUfZMpc+BwuM1IPw8fmQBiQ==",
+      "integrity": "sha512-5T6nhjsT+EOMzuck8JjBHARTHfMht0POzlA60WV2pMD3gyXw2LZnZ+ueGdNxG+0calOJcWKbpFcuzLZ91YWq9Q==",
       "dependencies": {
         "accepts": "~1.3.8",
         "array-flatten": "1.1.1",
-        "body-parser": "1.20.3",
+        "body-parser": "1.20.2",
         "content-disposition": "0.5.4",
         "content-type": "~1.0.4",
-        "cookie": "0.7.1",
+        "cookie": "0.6.0",
         "cookie-signature": "1.0.6",
         "debug": "2.6.9",
         "depd": "2.0.0",
-        "encodeurl": "~2.0.0",
+        "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "etag": "~1.8.1",
-        "finalhandler": "1.3.1",
+        "finalhandler": "1.2.0",
         "fresh": "0.5.2",
         "http-errors": "2.0.0",
-        "merge-descriptors": "1.0.3",
+        "merge-descriptors": "1.0.1",
         "methods": "~1.1.2",
         "on-finished": "2.4.1",
         "parseurl": "~1.3.3",
-        "path-to-regexp": "0.1.10",
+        "path-to-regexp": "0.1.7",
         "proxy-addr": "~2.0.7",
-        "qs": "6.13.0",
+        "qs": "6.11.0",
         "range-parser": "~1.2.1",
         "safe-buffer": "5.2.1",
-        "send": "0.19.0",
+        "send": "0.18.0",
-        "serve-static": "1.16.2",
+        "serve-static": "1.15.0",
         "setprototypeof": "1.2.0",
         "statuses": "2.0.1",
         "type-is": "~1.6.18",
@@ -250,12 +250,12 @@
       }
     },
     "node_modules/finalhandler": {
-      "version": "1.3.1",
+      "version": "1.2.0",
-      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.1.tgz",
+      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.2.0.tgz",
-      "integrity": "sha512-6BN9trH7bp3qvnrRyzsBz+g3lZxTNZTbVO2EV1CS0WIcDbawYVdYvGflME/9QP0h0pYlCDBCTjYa9nZzMDpyxQ==",
+      "integrity": "sha512-5uXcUVftlQMFnWC9qu/svkWv3GTd2PfUhK/3PLkYNAe7FbqJMt3515HaxE6eRL74GdsriiwujiawdaB1BpEISg==",
       "dependencies": {
         "debug": "2.6.9",
-        "encodeurl": "~2.0.0",
+        "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "on-finished": "2.4.1",
         "parseurl": "~1.3.3",
@@ -411,12 +411,9 @@
       }
     },
     "node_modules/merge-descriptors": {
-      "version": "1.0.3",
+      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz",
+      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz",
-      "integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==",
+      "integrity": "sha512-cCi6g3/Zr1iqQi6ySbseM1Xvooa98N0w31jzUYrXPX2xqObmFGHJ0tQ5u74H3mVh7wLouTseZyYIq39g8cNp1w=="
-      "funding": {
-        "url": "https://github.com/sponsors/sindresorhus"
-      }
     },
     "node_modules/methods": {
       "version": "1.1.2",
@@ -500,9 +497,9 @@
       }
     },
     "node_modules/path-to-regexp": {
-      "version": "0.1.10",
+      "version": "0.1.7",
-      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.10.tgz",
+      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz",
-      "integrity": "sha512-7lf7qcQidTku0Gu3YDPc8DJ1q7OOucfa/BSsIwjuh56VU7katFvuM8hULfkwB3Fns/rsVF7PwPKVw1sl5KQS9w=="
+      "integrity": "sha512-5DFkuoqlv1uYQKxy8omFBeJPQcdoE07Kv2sferDCrAq1ohOU+MSDswDIbnx3YAM60qIOnYa53wBhXW0EbMonrQ=="
     },
     "node_modules/proxy-addr": {
       "version": "2.0.7",
@@ -517,11 +514,11 @@
       }
     },
     "node_modules/qs": {
-      "version": "6.13.0",
+      "version": "6.11.0",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.13.0.tgz",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.11.0.tgz",
-      "integrity": "sha512-+38qI9SOr8tfZ4QmJNplMUxqjbe7LKvvZgWdExBOmd+egZTtjLB67Gu0HRX3u/XOq7UU2Nx6nsjvS16Z9uwfpg==",
+      "integrity": "sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q==",
       "dependencies": {
-        "side-channel": "^1.0.6"
+        "side-channel": "^1.0.4"
       },
       "engines": {
         "node": ">=0.6"
@@ -577,9 +574,9 @@
       "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
     },
     "node_modules/send": {
-      "version": "0.19.0",
+      "version": "0.18.0",
-      "resolved": "https://registry.npmjs.org/send/-/send-0.19.0.tgz",
+      "resolved": "https://registry.npmjs.org/send/-/send-0.18.0.tgz",
-      "integrity": "sha512-dW41u5VfLXu8SJh5bwRmyYUbAoSB3c9uQh6L8h/KtsFREPWpbX1lrljJo186Jc4nmci/sGUZ9a0a0J2zgfq2hw==",
+      "integrity": "sha512-qqWzuOjSFOuqPjFe4NOsMLafToQQwBSOEpS+FwEt3A2V3vKubTquT3vmLTQpFgMXp8AlFWFuP1qKaJZOtPpVXg==",
       "dependencies": {
         "debug": "2.6.9",
         "depd": "2.0.0",
@@ -599,28 +596,20 @@
         "node": ">= 0.8.0"
       }
     },
-    "node_modules/send/node_modules/encodeurl": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz",
-      "integrity": "sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w==",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
     "node_modules/send/node_modules/ms": {
       "version": "2.1.3",
       "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
       "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="
     },
     "node_modules/serve-static": {
-      "version": "1.16.2",
+      "version": "1.15.0",
-      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.2.tgz",
+      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.15.0.tgz",
-      "integrity": "sha512-VqpjJZKadQB/PEbEwvFdO43Ax5dFBZ2UECszz8bQ7pi7wt//PWe1P6MN7eCnjsatYtBT6EuiClbjSWP2WrIoTw==",
+      "integrity": "sha512-XGuRDNjXUijsUL0vl6nSD7cwURuzEgglbOaFuZM9g3kwDXOWVTck0jLzjPzGD+TazWbboZYu52/9/XPdUgne9g==",
       "dependencies": {
-        "encodeurl": "~2.0.0",
+        "encodeurl": "~1.0.2",
         "escape-html": "~1.0.3",
         "parseurl": "~1.3.3",
-        "send": "0.19.0"
+        "send": "0.18.0"
       },
       "engines": {
         "node": ">= 0.8.0"

package.json

@@ -13,6 +13,6 @@
   "description": "",
   "dependencies": {
     "dotenv": "^16.4.5",
-    "express": "^4.21.1"
+    "express": "^4.19.2"
   }
 }

playground.js

@@ -7,12 +7,12 @@ const { spawn, exec } = require("node:child_process")
 // })
 
 
-exec("echo '99999.incompass.netstar-inc.com' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
+exec("echo 'li12.pages.dev' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
   if (error) {
     console.error(error)
     return
   }
 
-  console.log(stdout.trim())
+    console.log(stdout.trim().split(/\t|\s{2,}/))
 
 }) 

src/app.js

@@ -7,10 +7,12 @@ const getCategory = new GetCategoryUseCase();
 const categoryConverter = new CategoryConverterUseCase({categoriesMapping});
 
 module.exports = async function app(domain){
+  console.log(domain)
   const category = await getCategory.execute(domain)
+  console.log(category)
   const categoryConverted = await categoryConverter.execute(category)
   console.log(categoryConverted)
-  return categoryConverted
+  return [categoryConverted]
 }
 
 

src/cron.js

@@ -3,8 +3,10 @@ const { UpdateCategoriesUseCase } = require('./use-cases/update-categories-use-c
 const updateCategories = new UpdateCategoriesUseCase()
 
 module.exports = async function cron() {
+  updateCategories.execute()
+
   const oneDay = 1000 * 60 * 60 * 24
-  setImmediate(() => {
+  setInterval(() => {
     updateCategories.execute()
   }, oneDay)
 }

src/etc/categories-mapping.json

@@ -1074,7 +1074,7 @@
       "id": "20097",
       "description": "VPN",
       "related": [
-          "10451"
+          "10460"
       ]
   },
   {
@@ -1109,7 +1109,7 @@
       "id": "20103",
       "description": "Code Repositories",
       "related": [
-          "10429"
+          "10047"
       ]
   },
   {

src/server.js

@@ -5,6 +5,10 @@ const app = require('./app')
 const bodyParser = require('body-parser')
 const cron = require('./cron')
 const express = require('express')
+const { ParseDetailedCategoryUseCase } = require('./use-cases/parse-detailed-category-use-case')
+const { ParseFullCategoryUseCase } = require('./use-cases/parse-full-category-use-case')
+const { CategoryConverterUseCase } = require('./use-cases/category-converter-use-case')
+const categoriesMapping = require('./etc/categories-mapping.json')
 
 const httpServer = express()
 httpServer.use(bodyParser.json()) // for parsing application/json
@@ -14,6 +18,11 @@ cron()
 const UDP_PORT = process.env.UDP_PORT
 const HTTP_PORT = process.env.HTTP_PORT
 
+// Initialize use cases
+const categoryConverter = new CategoryConverterUseCase({ categoriesMapping })
+const parseDetailedCategory = new ParseDetailedCategoryUseCase({ categoryConverter })
+const parseFullCategory = new ParseFullCategoryUseCase({ categoryConverter }) 
+
 const server = dgram.createSocket('udp4');
 
 
@@ -50,6 +59,7 @@ server.on('listening', () => {
 server.bind(UDP_PORT);
 
 httpServer.post('/', async (req, res) => {
+  console.log("new request")
   try {
     const { fqdn } = req.body
 
@@ -69,8 +79,35 @@ httpServer.post('/', async (req, res) => {
 
 })
 
+httpServer.post('/detailed', async (req, res) => {
+  try {
+    const { fqdn } = req.body
+
+    const detailedResult = await parseDetailedCategory.execute(fqdn)
+
+    res.status(200).json(detailedResult)
+  } catch (err) {
+    console.error('Error in /detailed endpoint:', err)
+    res.status(500).json({ error: err.message || err })
+  }
+})
+
+httpServer.post('/full', async (req, res) => {
+  try {
+    const { fqdn } = req.body
+
+    const fullResult = await parseFullCategory.execute(fqdn)
+
+    res.status(200).json(fullResult)
+  } catch (err) {
+    console.error('Error in /full endpoint:', err)
+    res.status(500).json({ error: err.message || err })
+  }
+})
+
 httpServer.listen(HTTP_PORT, () => {
   console.log('HTTP server listening 3333')
+  console.log('UDP server listening 3334')
 })
 
 // Response

src/test-app.js

@@ -0,0 +1,10 @@
+const app = require('./app');
+
+(async () => {
+  try {
+    const categories = await app('www.facebook.com');
+    console.log(categories);
+  } catch (error) {
+    console.error('Error occurred:', error);
+  }
+})();

src/use-cases/category-converter-use-case.js

@@ -11,7 +11,7 @@ class CategoryConverterUseCase {
   execute(category) {
     const entry = this.categoriesMapping.find(item => item.id === category);
 
-    return entry ? entry.related[0].split(', ').map(str => str.trim()) : null;
+    return entry ? entry.related[0] : null;
   }
 }
 

src/use-cases/parse-detailed-category-use-case.js

@@ -0,0 +1,101 @@
+const { exec } = require("node:child_process")
+
+class ParseDetailedCategoryUseCase {
+  constructor({ categoryConverter }) {
+    this.categoryConverter = categoryConverter;
+  }
+
+  execute(domain) {
+    return new Promise((resolve, reject) => {
+      exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
+           { cwd: '/usr/local/gcf1' },
+           (error, stdout, stderr) => {
+             if (error) {
+               console.error(error);
+               reject(error);
+               return;
+             }
+
+             try {
+               const parsed = this.parseOutput(stdout);
+               resolve(parsed);
+             } catch (parseError) {
+               console.error('Parse error:', parseError);
+               reject(parseError);
+             }
+           });
+    });
+  }
+
+  parseOutput(output) {
+    // Extract all quoted strings in order
+    const quotedStrings = [];
+    const quoteRegex = /"([^"]*)"/g;
+    let match;
+    while ((match = quoteRegex.exec(output)) !== null) {
+      quotedStrings.push(match[1]);
+    }
+
+    // Split on tabs (primary separator in NetStar output)
+    const parts = output.trim().split('\t');
+    let quotedIndex = 0;
+
+    // Position mapping based on NetStar output structure:
+    // parts[0]: Categorized
+    // parts[1]: Matching Index
+    // parts[2]: Primary Category ID
+    // parts[3]: Primary Category Name (quoted)
+    // parts[4]: Secondary Category ID
+    // parts[5]: Secondary Category Name (quoted if secondary exists)
+    // parts[6]: Security Category ID
+    // parts[7]: Security Category Name
+    // parts[8]: Reputation Score ID
+    // parts[9]: Reputation Score Name (quoted)
+    // parts[10]: Matching Flag Value
+    // parts[11]: Matching Flag Names
+    // parts[12]: Age Rating ID
+    // parts[13]: Age Rating Name (quoted)
+
+    const primaryId = parseInt(parts[2]);
+    const primaryName = quotedStrings[quotedIndex++];
+    const primaryMapped = this.categoryConverter.execute(String(primaryId));
+
+    const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
+    let secondaryName = null;
+    let secondaryMapped = null;
+
+    if (secondaryId !== null) {
+      secondaryName = quotedStrings[quotedIndex++];
+      secondaryMapped = this.categoryConverter.execute(String(secondaryId));
+    } else {
+      secondaryName = parts[5] === '-' ? null : parts[5];
+    }
+
+    const reputationId = parseInt(parts[8]);
+    const reputationName = quotedStrings[quotedIndex++];
+
+    const ageRatingId = parseInt(parts[12]);
+    const ageRatingName = quotedStrings[quotedIndex++];
+
+    // Build result array in the same format as the / endpoint
+    const resultArray = [String(primaryMapped)];
+    if (secondaryMapped !== null) {
+      resultArray.push(String(secondaryMapped));
+    }
+
+    return {
+      result: resultArray,
+      primary: primaryMapped,
+      primary_name: primaryName,
+      secondary: secondaryMapped,
+      secondary_name: secondaryName,
+      reputation: reputationId,
+      reputation_name: reputationName,
+      age_rating: ageRatingId,
+      age_rating_name: ageRatingName,
+      raw_output: output.trim()
+    };
+  }
+}
+
+module.exports = { ParseDetailedCategoryUseCase }

src/use-cases/parse-full-category-use-case.js

@@ -0,0 +1,172 @@
+const { exec } = require("node:child_process")
+
+class ParseFullCategoryUseCase {
+  constructor({ categoryConverter }) {
+    this.categoryConverter = categoryConverter;
+  }
+
+  execute(domain) {
+    return new Promise((resolve, reject) => {
+      exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
+           { cwd: '/usr/local/gcf1' },
+           (error, stdout, stderr) => {
+             if (error) {
+               console.error(error);
+               reject(error);
+               return;
+             }
+
+             try {
+               const parsed = this.parseOutput(stdout);
+               resolve(parsed);
+             } catch (parseError) {
+               console.error('Parse error:', parseError);
+               reject(parseError);
+             }
+           });
+    });
+  }
+
+  parseOutput(output) {
+    const quotedStrings = [];
+    const quoteRegex = /"([^"]*)"/g;
+    let match;
+    while ((match = quoteRegex.exec(output)) !== null) {
+      quotedStrings.push(match[1]);
+    }
+
+    const parts = output.trim().split('\t');
+    let quotedIndex = 0;
+
+    // Extract primary category (parts[2], parts[3])
+    const primaryId = parseInt(parts[2]);
+    const primaryName = quotedStrings[quotedIndex++];
+    const primaryMapped = this.categoryConverter.execute(String(primaryId));
+
+    // Extract secondary category (parts[4], parts[5])
+    const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
+    let secondaryName = null;
+    let secondaryMapped = null;
+    if (secondaryId !== null && parts[5] !== '-') {
+      secondaryName = quotedStrings[quotedIndex++];
+      secondaryMapped = this.categoryConverter.execute(String(secondaryId));
+    } else {
+      secondaryName = parts[5] === '-' ? null : parts[5];
+    }
+
+    // Extract security category (parts[6], parts[7])
+    const securityId = parts[6] !== '0' && parts[6] !== '-' ? parseInt(parts[6]) : null;
+    let securityName = null;
+    let securityMapped = null;
+    if (securityId !== null && parts[7] !== '-') {
+      securityName = quotedStrings[quotedIndex++];
+      securityMapped = this.categoryConverter.execute(String(securityId));
+    } else {
+      securityName = parts[7] === '-' ? null : parts[7];
+    }
+
+    // Extract reputation (parts[8], parts[9])
+    const reputationId = parseInt(parts[8]);
+    const reputationName = quotedStrings[quotedIndex++];
+
+    // Extract matching flag (parts[10], parts[11])
+    const matchingFlagValue = parts[10];
+    const matchingFlagNames = parts[11] ? parts[11].split('|') : [];
+
+    // Extract age rating (parts[12], parts[13])
+    const ageRatingId = parseInt(parts[12]);
+    const ageRatingName = quotedStrings[quotedIndex++];
+
+    // Extract category groups starting from parts[16] (after 2 empty fields)
+    // Each group has: id (number), name (quoted string)
+    // Pattern: 0 "Internet/Infrastructure" 0 "Malware/Security" ...
+    const categoryGroups = {
+      internet_infrastructure: {
+        id: parseInt(parts[16]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      malware_security: {
+        id: parseInt(parts[18]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      dangerous_harmful: {
+        id: parseInt(parts[20]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      adult: {
+        id: parseInt(parts[22]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      business_government: {
+        id: parseInt(parts[24]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      personal: {
+        id: parseInt(parts[26]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      computing_technology: {
+        id: parseInt(parts[28]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      social_media: {
+        id: parseInt(parts[30]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      miscellaneous: {
+        id: parseInt(parts[32]) || 0,
+        name: quotedStrings[quotedIndex++]
+      }
+    };
+
+    // Extract volume index and submitted URL
+    const volumeIndex = parts[35];
+    const submittedUrl = parts[36];
+
+    const result = {
+      result_status: parts[0],
+      matching_index: parseInt(parts[1]),
+      primary_category: {
+        id: primaryId,
+        name: primaryName,
+        mapped_id: String(primaryMapped)
+      },
+      secondary_category: {
+        id: secondaryId,
+        name: secondaryName,
+        mapped_id: secondaryMapped ? String(secondaryMapped) : null
+      },
+      security_category: {
+        id: securityId,
+        name: securityName,
+        mapped_id: securityMapped ? String(securityMapped) : null
+      },
+      reputation: {
+        id: reputationId,
+        name: reputationName
+      },
+      matching_flag: {
+        value: matchingFlagValue,
+        names: matchingFlagNames
+      },
+      age_rating: {
+        id: ageRatingId,
+        name: ageRatingName
+      },
+      category_groups: categoryGroups,
+      volume_index: volumeIndex,
+      submitted_url: submittedUrl
+    };
+
+    // Add result array in the same format as / endpoint
+    const resultArray = [result.primary_category.mapped_id];
+    if (result.secondary_category.mapped_id !== null) {
+      resultArray.push(result.secondary_category.mapped_id);
+    }
+    result.result = resultArray;
+
+    return result;
+  }
+}
+
+module.exports = { ParseFullCategoryUseCase }

src/use-cases/update-categories-use-case.js

@@ -1,20 +1,26 @@
-const { exec } = require("node:child_process")
+const { exec } = require("node:child_process");
 
 class UpdateCategoriesUseCase {
   execute() {
-    return new Promise((_, reject) => {
+    exec(`bin/gcf1dbmng.sh etc urldb_download`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
-      exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
+      if (error) {
-        if (error) {
+        console.error('Erro no primeiro comando:', error);
-          console.error(error);
+        return;
-          reject(error);
+      }
+
+      console.log('Saída do primeiro comando:', stdout);
+
+      // Executa o segundo comando após o primeiro ter sido concluído
+      exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error2, stdout2, stderr2) => {
+        if (error2) {
+          console.error('Erro no segundo comando:', error2);
           return;
         }
 
-        console.log(stdout)
+        console.log('Saída do segundo comando:', stdout2);
       });
     });
   }
 }
 
-module.exports = { UpdateCategoriesUseCase }
+module.exports = { UpdateCategoriesUseCase };
-

test-detailed.http

@@ -0,0 +1,188 @@
+@baseUrl = http://localhost:3333
+
+# ============================================
+# POST / - Basic Categorization Endpoint
+# Returns: Simple result array of category IDs
+# ============================================
+
+### Basic endpoint - Facebook
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "facebook.com"
+}
+
+### Basic endpoint - Google
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "google.com"
+}
+
+### Basic endpoint - TikTok
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "tiktok.com"
+}
+
+### Basic endpoint - YouTube
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "youtube.com"
+}
+
+### Basic endpoint - Reddit
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "reddit.com"
+}
+
+### Basic endpoint - Twitter
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "twitter.com"
+}
+
+# ============================================
+# POST /detailed - Detailed Categorization
+# Returns: Categories with names, reputation, age rating
+# ============================================
+
+### Detailed endpoint - Facebook
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "facebook.com"
+}
+
+### Detailed endpoint - Google
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "google.com"
+}
+
+### Detailed endpoint - TikTok
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "tiktok.com"
+}
+
+### Detailed endpoint - YouTube
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "youtube.com"
+}
+
+### Detailed endpoint - Reddit
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "reddit.com"
+}
+
+### Detailed endpoint - Twitter
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "twitter.com"
+}
+
+# ============================================
+# POST /full - Complete Raw Output
+# Returns: All 35 fields from NetStar with category groups
+# ============================================
+
+### Full endpoint - Facebook
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "facebook.com"
+}
+
+### Full endpoint - Google
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "google.com"
+}
+
+### Full endpoint - TikTok
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "tiktok.com"
+}
+
+### Full endpoint - YouTube
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "youtube.com"
+}
+
+### Full endpoint - Reddit
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "reddit.com"
+}
+
+### Full endpoint - Twitter
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "twitter.com"
+}
+
+# ============================================
+# Edge cases and special domains
+# ============================================
+
+### Basic endpoint - Unknown domain
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "unknowndomainexample12345.com"
+}
+
+### Detailed endpoint - Localhost
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "localhost"
+}
+
+### Full endpoint - IP-like domain
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "99999.incompass.netstar-inc.com"
+}